Awaiting

When scraping the web, you may want to wait for various elements. WebScrapingAPI offers you the options to wait for almost everything related to the targeted website.

The wait_until parameter

Passing the wait_until parameter to your request tells the browser that a certain promise is still ongoing, hence preventing the service worker from terminating until the promise is resolved.

Your full GET request should then be sent to the following address:

https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=<TARGETED_URL>&wait_until=$promise

Available options

The currently supported options for the wait_until parameter are:

• load - load event is fired.

• domcontentloaded - DOMContentLoaded event is fired.

• networkidle0 - there are no more than 0 network connections for at least 500 ms.

• networkidle2 - there are no more than 2 network connections for at least 500 ms.

The wait_for parameter

This option comes in handy when dealing with websites that have a longer loading time. Passing the wait_for parameter will instruct WebScrapingAPI to wait for a certain amount of time, before closing the connection.

Your full GET request should then be sent to the following address:

https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=<TARGETED_URL>&wait_for=$milliseconds

The values passed to the wait_for parameter are expressed in milliseconds. We recommend that you use a value between 5000 and 10000.

The wait_for_css parameter

When you want to scrape a specific element of the page, passing the wait_for_css parameter will make WebScrapingAPI wait for that specific CSS selector to be available in DOM before returning the result.

If the CSS selector won't load within 10 seconds, the request is automatically marked as failed, and thus, you will not be charged for it.

Your full GET request should then be sent to the following address:

https://api.webscrapingapi.com/v1?api_key=<YOUR_API_KEY>&url=<TARGETED_URL>&wait_for_css=$cssSelector

We recommend that the $cssSelector value should be encoded or escaped in order for your request to pass successfully.