Proxy Mode

WebScrapingAPI offers a proxy front-end to the API, making the integration with third-party tools much easier.

Interacting with WebScrapingAPI via the proxy mode only changes the access method with our API; apart from that, full functionality is still available to the end user. Both costs, parameters and response format remain the same.

Note that our proxy mode supports GET, POST and PUT requests.

Accessing the API via Proxy Mode

The Proxy Mode is available via:

  • HTTPS at proxy.webscrapingapi.com:8000

  • HTTP at proxy.webscrapingapi.com:80

Authentication via proxy mode is done using the username:password@proxy.webscrapingapi.com:8000 method. However, the username part also holds the parameters you want to pass to your request. Hence, the configuration you need to pass to your request is:

  • Username: webscrapingapi, followed by the parameters you want to pass to your request (separated by dots)

  • Password: your personal API key

The most basic URL example for an authenticated request on our API via proxy mode is:

https://webscrapingapi.<parameter_1.parameter_2. ... .parameter_n>:<YOUR_API_KEY>@proxy.webscrapingapi.com:8000

Proxy Mode Requirements and Recommendations

If you decide to use the proxy mode, please make sure you understand and follow these requirements:

  1. Disable SSL certificates verification (-k switch in cURL, verify=false in Python etc.)

  2. If you use proxy mode with a browser automation library (e.g. Selenium, Puppeteer etc.), it is highly recommended to disable JavaScript rendering (render_js=0) or to limit the amount of resources requested by your browser.

Proxy Mode Integration Examples

#1: GET Requests

curl --proxy https://webscrapingapi.proxy_type=datacenter.device=desktop:<YOUR_API_KEY>@proxy.webscrapingapi.com:8000 -X GET http://httpbin.org/get -k
Response Example
{
  args: {},
  headers: {
    Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, br',
    Host: 'httpbin.org',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.80 Safari/537.36',
    'X-Amzn-Trace-Id': 'Root=1-632993e3-29380e237fe8c362301e8a57'
  },
  origin: '198.46.171.167',
  url: 'http://httpbin.org/get'
}

#2: POST Requests

curl --proxy https://webscrapingapi.proxy_type=datacenter.device=desktop:<YOUR_API_KEY>@proxy.webscrapingapi.com:8000 -X POST https://httpbin.org/post --data '{"foo":"bar"}' -k      
Response Example
{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "{\"foo\":\"bar\"}": ""
  },
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate, br",
    "Content-Length": "28",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36",
    "X-Amzn-Trace-Id": "Root=1-631edd38-2bbe74166ca70a5b438fa87f"
  },
  "json": null,
  "origin": "196.244.10.12",
  "url": "https://httpbin.org/post"
}

#3: PUT Requests

curl --proxy https://webscrapingapi.proxy_type=datacenter.device=desktop:<YOUR_API_KEY>@proxy.webscrapingapi.com:8000 -X PUT https://httpbin.org/put --data '{"foo":"bar"}' -k      
Response Example
{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "{\"foo\":\"bar\"}": ""
  },
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate, br",
    "Content-Length": "28",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36",
    "X-Amzn-Trace-Id": "Root=1-631edd38-2bbe74166ca70a5b438fa87f"
  },
  "json": null,
  "origin": "196.244.10.12",
  "url": "https://httpbin.org/put"
}

Last updated