API Documentation

The scrapestack API was built to offer a simple REST API interface for scraping web pages at scale without having to programatically deal with geolocations, IP blocks or CAPTCHAs. The API supports a series of features essential to web scraping, such as JavaScript rendering, custom HTTP headers, various geo-targets, POST/PUT requests and an option to use premium residential proxies instead of datacenter proxies.

In this documentation you will find detailed usage guides and code examples in different programming languages that will help you get up and running in the shortest time possible. If the articles below leave any questions unanswered, please feel free to contact our technical support team.

Getting Started

API Access Key & Authentication

After creating a scrapestack account, the account dashboard will reveal the unique API access key you can use to authenticate with the API. To do so, simply attach the access_key parameter to the API's base URL and set it to your API access key.

Sign Up to Run API Requesthttps://api.scrapestack.com/scrape
    ? access_key = YOUR_ACCESS_KEY

Secure your key: To prevent unauthorized access to your scrapestack account, please make sure to store your API access key in a secure location and never include it in any public scripts or files.

256-bit HTTPS Encryption Available on: Basic Plan and higher

Customers subscribed to the Basic Plan or higher may connect to the scrapestack API using industry-standard 256-bit HTTPS (SSL) encryption by appending an s to the HTTP protocol. Please find an illustration below.

Example API Request:

https://api.scrapestack.com/scrape

API Error Codes

If your request fails, the scrapestack API will return an error in JSON format. Find below an example error that occurs when the API failed scraping the requested URL.

Example Error:

{
    "success": false,
    "error": {
        "code": 105,
        "type": "scrape_request_failed"
    }
}

Common API Errors:

Code	Type	Info
`404`	`404_not_found`	User requested a resource which does not exist.
`101`	`missing_access_key`	User did not supply an access key.
`101`	`invalid_access_key`	User supplied an invalid access key.
`102`	`inactive_user`	User account is inactive or blocked.
`103`	`invalid_api_function`	User requested a non-existent API function.
`104`	`usage_limit_reached`	User has reached his subscription's monthly request allowance.
`105`	`function_access_restricted`	The user's current subscription does not support this API function.
`105`	`https_access_restricted`	The user's current subscription plan does not support HTTPS.
`210`	`missing_url`	User has not specified a valid URL to scrape.
`211`	`invalid_url`	User has specified an invalid value in the URL parameter.
`212`	`invalid_proxy_location`	User has specified an invalid or unsupported proxy location.
`213`	`scrape_request_failed`	The current scraping request failed due to a technical issue. If this error occurs, please report this to technical customer support.

API Features

Basic Request Available on: All plans

To scrape a web page using the scrapestack API, simply use the API's base endpoint and append the URL you would like to scrape as well as your API access key as GET parameters. There is also a series of optional parameters you can choose from. Below you will find an example request used to scrape the URL https://apple.com.

Example API Request:

Sign Up to Run API Requesthttps://api.scrapestack.com/scrape
    ? access_key = YOUR_ACCESS_KEY
    & url = https://apple.com

Request Parameters:

Object	Description
`access_key`	[Required] Specify your unique API access key to authenticate with the API. Your API access key can be found in your account dashboard.
`url`	[Required] Specify the URL of the web page you would like to scrape.
`render_js`	[optional] Set to `0` (off, default) or `1` (on) depending on whether or not to render JavaScript on the target web page. JavaScript rendering is done using a Google Chrome headless browser.
`keep_headers`	[optional] Set `0` (off, default) or `1` (on) depending on whether or not to send currently active HTTP headers to the target URL with your API request and have the API return these headers along with your API response.
`proxy_location`	[optional] Specify the 2-letter code of the country you would like to us as a proxy geolocation for your scraping API request. Supported countries differ by proxy type, please refer to the Proxy Locations section for details.
`premium_proxy`	[optional] Set `0` (off, default) or `1` (on) depending on whether or not to enable premium residential proxies for your scraping request. Please note that a single premium proxy API request is counted as 25 API requests.

Example API Response:

If your scraping request was successful, the API will respond with the raw HTML data of your target web page URL. If you have enabled HTTP headers, your API response will also contain the HTTP headers sent along with your original API request.

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-US" lang="en-US">

    <head>
    [...] // 44 lines skipped
    </head>
    
    <body>
    [...] // 394 lines skipped
    </body>

</html>

API response shortened: Please note that the API response above has been shortened for readability purposes. To see the entire API resonse, please click the "Run API Request" button in the Example API Request section or sign up for an access key.

JavaScript Rendering Available on: Basic Plan and higher

Some web pages render essential page elements using JavaScript, which means that some content is not present (and therefore not scrapable) with the initial page load. With the render_js parameter enabled, the scrapestack API is capable of accessing the target web using a headless browser (Google Chrome) and allow JavaScript page elements to render before delivering the final scraping result.

To enable JavaScript rendering, simply append the render_js HTTP GET parameter to your API request URL and set it to 1. By default, this parameter is set to 0 (off).

Example API Request:

Sign Up to Run API Requesthttps://api.scrapestack.com/scrape
    ? access_key = YOUR_ACCESS_KEY
    & url = https://apple.com
    & render_js = 1

Example API Response:

To see an API response, please click the "Run API Request" button above or sign up for an API access key.

HTTP Headers Available on: All plans

The scrapestack API will accept HTTP headers and pass them through to the target web page and the final API response if the keep_headers HTTP GET parameter is set to 1. By default, this parameter is set to 0 (off).

Below you will find an example API request (Bash using the "curl" command) that contains an HTTP X-Header. If this request is executed, the specified header will be sent to the target web page and returned in the final API response.

Example Bash (curl) Request with HTTP header:

curl --header "X-AnyHeader: Test" \
"https://api.scrapestack.com/scrape?access_key=YOUR_ACCESS_KEY&url=https://apple.com"

Unsupported HTTP Headers: Although most HTTP headers are supported by the API, there are some that cannot be processed. Please find a list of unsupported HTTP headers below:

content-encoding
content-length

Proxy Locations Available on: Basic Plan and higher

The scrapestack API is making use of a pool of 35+ million IP addresses worldwide. By default, the API will auto-rotate IP addresses in a way that the same IP address is never used twice in a row.

Across both standard and premium proxies, the scrapestack API supports more than 100 global geolocations your scraping request can be sent from. Using the API's proxy_location HTTP GET parameter you can choose a specific country by its 2-letter country code for your scraping request. Please find an example API request below, which specifies au (for Australia) as proxy location.

Example API Request:

Sign Up to Run API Requesthttps://api.scrapestack.com/scrape
    ? access_key = YOUR_ACCESS_KEY
    & url = https://apple.com
    & proxy_location = au

Example API Response:

To see an API response, please click the "Run API Request" button above or sign up for an API access key.

Standard Proxies - Supported Countries:

For standard (datacenter) proxies, the API currently supports a total of 77 global geolocations. You can download a full list of supported countries and 2-letter country codes using the following link: locations-standard-proxy.csv

Premium Proxies - Supported Countries:

For premium (residential) proxies, the API currently supports a total of 38 global geolocations. You can download a full list of supported countries and 2-letter country codes using the following link: locations-premium-proxy.csv

Premium Proxies Available on: Basic Plan and higher

By default, the scrapestack will always use standard (datacenter) proxies for scraping requests. While these proxies are the most common proxies used on the internet, they are also much more likely to get blocked on an IP-basis when attempting to scrape data.

For customers subscribed to the Professional Plan or higher, the scrapestack API allows access to premium (residential) proxies, which are associated with real residential addresses and therefore much less likely to get blocked while scraping data on the web. To perform your request using premium proxies, simply append the API's premium_proxy HTTP GET parameter and set it to 1.

Example API Request:

Sign Up to Run API Requesthttps://api.scrapestack.com/scrape
    ? access_key = YOUR_ACCESS_KEY
    & url = https://apple.com
    & premium_proxy = 1

Example API Response:

To see an API response, please click the "Run API Request" button above or sign up for an API access key.

Premium Proxy Requests: Please note that a single request using premium proxies will be charged as 25 API requests, while single requests using standard proxies are charged as only 1 API request.

HTTP POST/PUT Requests Available on: All plans

The scrapestack API also offers a way of scraping forms or API endpoints directly by supporting API requests via HTTP POST/PUT. Below you will find relevant examples that can be used both via POST and PUT.

Example Request - HTTP POST:

curl -d 'foo=bar' \
-X POST \
"https://api.scrapestack.com/scrape?access_key=YOUR_ACCESS_KEY&url=https://apple.com"

Example Request - HTTP POST with Form Data:

curl -H 'Content-Type: application/x-www-form-urlencoded' \
-F 'foo=bar' \
-X POST \
"https://api.scrapestack.com/scrape?access_key=YOUR_ACCESS_KEY&url=https://apple.com"

Code Examples

Below you will find sample scraping requests in the following programming languages: PHP, Python, Nodejs, jQuery, Go and Ruby.

Code Example - PHP

<?php

$queryString = http_build_query([
  'access_key' => 'YOUR_ACCESS_KEY',
  'url' => 'http://scrapestack.com',
]);

$ch = curl_init(sprintf('%s?%s', 'http://api.scrapestack.com/scrape', $queryString));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$website_content = curl_exec($ch);
curl_close($ch);

echo $website_content;

Code Example - Python

import requests

params = {
  'access_key': 'YOUR_ACCESS_KEY',
  'url': 'http://scrapestack.com'
}

api_result = requests.get('http://api.scrapestack.com/scrape', params)
website_content = api_result.content

print(website_content)

Code Example - Nodejs

const axios = require('axios');
const params = {
  access_key: 'YOUR_ACCESS_KEY',
  url: 'http://scrapestack.com'
}

axios.get('http://api.scrapestack.com/scrape', {params})
  .then(response => {
    const websiteContent = response.data;
    console.log(websiteContent);
  }).catch(error => {
    console.log(error);
  });

Code Example - jQuery

$.get('https://api.scrapestack.com/scrape',
  {
    access_key: 'YOUR_ACCESS_KEY',
    url: 'http://scrapestack.com'
  },
  function(websiteContent) {
    console.log(websiteContent);
  }
);

Code Example - Go

package main

import (
  "fmt"
  "io/ioutil"
  "net/http"
)

func main() {
  httpClient := http.Client{}

  req, err := http.NewRequest("GET", "http://api.scrapestack.com/scrape", nil)
  if err != nil {
    panic(err)
  }

  q := req.URL.Query()
  q.Add("access_key", "YOUR_ACCESS_KEY")
  q.Add("url", "http://scrapestack.com")
  req.URL.RawQuery = q.Encode()

  res, err := httpClient.Do(req)
  if err != nil {
    panic(err)
  }
  defer res.Body.Close()

  if res.StatusCode == http.StatusOK {
    bodyBytes, err := ioutil.ReadAll(res.Body)
    if err != nil {
        panic(err)
    }
    websiteContent := string(bodyBytes)
    fmt.Println(websiteContent)
  }
}

Code Example - Ruby

require 'net/http'
require 'json'

params = {
  :access_key => "YOUR_ACCESS_KEY",
  :url => "http://scrapestack.com"
}
uri = URI('http://api.scrapestack.com/scrape')
uri.query = URI.encode_www_form(params)

website_content = Net::HTTP.get(uri)
print(website_content)

Business Continuity - API Overages

Ensuring our customers achieve success is paramount to what we do at APILayer. For this reason, we will be rolling out our Business Continuity plan guaranteeing your end users will never see a drop in coverage. Every plan has a certain amount of API calls that you can make in the given month. However, we would never want to cut your traffic or impact user experience negatively for your website or application in case you get more traffic.

What is an overage?

An overage occurs when you go over a quota for your API plan. When you reach your API calls limit, we will charge you a small amount for each new API call so we can make sure there will be no disruption in the service we provide to you and your website or application can continue running smoothly.

Prices for additional API calls will vary based on your plan. See table below for prices per call and example of an overage billing.

Plan Name	Monthly Price	Number of Calls	Overage Price per call	Overage	Total price
Basic	$19.99	200,000	0.00011994	40,000	$24.79
Professional	$79.99	1,000,000	0.000095988	200,000	$99.19
Business	$199.99	3,000,000	0.000079996	600,000	$247.99

Why does APILayer have overage fees?

Overage fees allow developers to continue using an API once a quota limit is reached and give them time to upgrade their plan based on projected future use while ensuring API providers get paid for higher usage.

How do I know if I will be charged for overages?

When you are close to reaching your API calls limit for the month, you will receive an automatic notification (at 75%, 90% and 100% of your monthly quota). However, it is your responsibility to review and monitor for the plan’s usage limitations. You are required to keep track of your quota usage to prevent overages. You can do this by tracking the number of API calls you make and checking the dashboard for up-to-date usage statistics.

How will I be charged for my API subscription?

You will be charged for your monthly subscription plan, plus any overage fees applied. Your credit card will be billed after the billing period has ended.

What happens if I don’t have any overage fees?

In this case, there will be no change to your monthly invoice. Only billing cycles that incur overages will see any difference in monthly charges. The Business Continuity plan is an insurance plan to be used only if needed and guarantees your end users never see a drop in coverage from you.

What if I consistently have more API calls than my plan allows?

If your site consistently surpasses the set limits each month, you may face additional charges for the excess usage. Nevertheless, as your monthly usage reaches a certain threshold, it becomes more practical to consider upgrading to the next plan. By doing so, you ensure a smoother and more accommodating experience for your growing customer base.

I would like to upgrade my plan. How can I do that?

You can easily upgrade your plan by going to your Dashboard and selecting the new plan that would be more suitable for your business needs. Additionally, you may contact your Account Manager to discuss a custom plan if you expect a continuous increase in usage.

Introducing Platinum Support - Enterprise-grade support for APILayer

Upgrade your APIlayer subscription with our exclusive Platinum Support, an exceptional offering designed to enhance your business’ API management journey. With Platinum Support, you gain access to a host of premium features that take your support experience to a whole new level.

What does Platinum Support include?

	Standard Support	Platinum Support
General review on the issue
Access to knowledge base articles
Email support communication
Regular products updates and fixes
Dedicated account team
Priority Email Support with unlimited communication
Priority bug and review updates
Option for quarterly briefing call with product Management
Features requests as priority roadmap input into product

Priority Email Support: Experience unrivaled responsiveness with our priority email support. Rest assured that your inquiries receive top-priority attention, ensuring swift resolutions to any issues.

Unlimited Communication: Communication is key, and with Platinum Support, you enjoy unlimited access to our support team. No matter how complex your challenges are, our experts are here to assist you every step of the way.

Priority Bug Review and Fixes: Bugs can be a headache, but not with Platinum Support. Benefit from accelerated bug review and fixes, minimizing disruptions and maximizing your API performance.

Dedicated Account Team: We understand the value of personalized attention. That's why Platinum Support grants you a dedicated account team, ready to cater to your specific needs and provide tailored solutions.

Quarterly Briefing Call with Product Team: Stay in the loop with the latest updates and insights from our Product team. Engage in a quarterly briefing call to discuss new features, enhancements, and upcoming developments.

Priority Roadmap Input: Your input matters! As a Platinum Support subscriber, your feature requests receive top priority, shaping our product roadmap to align with your evolving requirements.

Don't settle for the standard when you can experience the exceptional. Upgrade to Platinum Support today and supercharge your APIlayer experience!

Upgrade now

3-Step Quickstart Guide

Code Examples