wpBots Support

Helpdesk for Best Scraper Plugin

Search in:
Suggested Search: installation, demo data, icons

Request Timeout – HTTP Code : 500

wpBots Support – The best crawlers for WordPress › Forums › SCRAPER (after-sales) › General Issues › Request Timeout – HTTP Code : 500

Viewing 4 posts - 1 through 4 (of 4 total)
  • Author
    Posts
  • December 20, 2019 at 7:45 pm #983
    RoryRory
    Participant

    Hi,

    I am using the scraper and in the first couple of minutes its doing fine but after like 2 / 3 minutes i get the message:

    Request Timeout
This request takes too long to process, it is timed out by the server. If it should not be timed out, please contact administrator of this web site to increase 'Connection Timeout'.
HTTP Code : 500

    When i get the message the scraper made like 130 new posts but if just wait some time it will go up to 700 even when the message is showing. So whats is up and how can i get all the pages i want to scrape? Its from a XML file with around the 10582 pages inside it.

    And i asked my webhoster and i updated the “max_execution_time” but that didnt help so how can i fix this that i will get all the 10582 pages?

    December 22, 2019 at 4:50 pm #993
    Suman M.Suman M.
    Keymaster

    Hi, thanks for contacting us. That’s lots of posts. Please make sure that max_execution_time is 600 seconds or more, and PHP memory limit is 512M or more.
    Also, set “Loop Limit” value to 5 under Limits tab – https://www.screencast.com/t/hKcP1raVVFP9
    Let us know!

    December 23, 2019 at 8:34 pm #1005
    RoryRory
    Participant

    Hi Suman,

    I will proceed the 5 with no errors but if i make a task of it then it will not give any error also but it will also not add all posts. Now i have it on a limit of 15 and run the task every minute but when i close everything (browser) it looks like that the task stops then after a couple of minutes, its not staying alive?

    So what to do here? Is there not a url so i can trigger it with a cronjob or so?

    December 24, 2019 at 6:16 am #1008
    Suman M.Suman M.
    Keymaster

    Can you please let us know the exact source URL that you are fetching from? Also let us know your site’s max_execution_time and PHP memory limit values.
    Currently there is no direct url that can be used in crons.

  • Author
    Posts
Viewing 4 posts - 1 through 4 (of 4 total)

You must be logged in and have valid license to reply to this topic.

License required for the following item
Scraper - Automatic Content Crawler Plugin for WordPress
Login and Registration Log in · Register