Scraper Requirements

Here is the basic requirements for Scraper plugin :

1CPU Core
25GB SSD Storage
1TB Transfer
40Gbps Network In
1000Mbps Network Out

We use Digital Ocean’s base server option for testing. The table above captured from’s base server.

If you use dedicated virtual cloud server for Scraper plugin, that will be the perfect choice scraping.

The main problem with scraping process, when you use a shared hosting, it will use shared resources. RAM, CPU or network can be used from other customers time to time. That can affect plugin’s performance.

Also these companies provides dynamic IP addresses that can be replaced with an another. And this availability can be useful in any ban or block cases.

So we simply suggest to you use virtual dedicated server. Even small ones will provide very good performances. We currently use Linode and DigitalOcean’s base plans for tests.

Also recommended:
PHP Memory Limit: 512M or more
Max Execution Time Limit: 300 seconds or more

Was This Article Helpful?

Related Articles

There are no comments yet

Leave a comment

Your email address will not be published.