wpBots Support

Helpdesk for Best Scraper Plugin

Search in:
Suggested Search: installation, demo data, icons

https url issue on wp scraper plugin

wpBots Support – The best crawlers for WordPress › Forums › SCRAPER (after-sales) › General Issues › https url issue on wp scraper plugin

Viewing 7 posts - 1 through 7 (of 7 total)
  • Author
    Posts
  • April 12, 2020 at 5:08 pm #1810
    Nav TomNav
    Participant

    First time installed this plugin and tried to add https URL for the new task but it redirects to HTTP URL. Is this intended or need some changes in plugin or server configuration. The website which I want to scrap doesn’t allow HTTP URL which giving me an access denied error.

    April 13, 2020 at 4:06 am #1816
    Suman M.Suman M.
    Keymaster

    Hi, is it that the https url when put in Page URL field (https://www.screencast.com/t/8K6o0lQffnL7) and hit Fetch button, https changes to http?
    Can you please let us know the exact source url from where you want to scrape? Thanks!

    April 13, 2020 at 7:02 am #1818
    Nav TomNav
    Participant

    Yup, that is the case, When I hit fetch button it changes to HTTP.
    This is the URL
    https://www.jagran.com/latest-news.html

    April 13, 2020 at 7:11 am #1820
    Suman M.Suman M.
    Keymaster

    I checked it and that’s not the actual issue. The mentioned site is not scrapable as it seems to have bot prevention system.

    April 13, 2020 at 7:36 am #1822
    Nav TomNav
    Participant

    It seems to be exactly that case.
    https://ibb.co/LSNftTP

    In the above screenshot is showing access denied error with URL showing HTTP instead of HTTPS.
    I am able to scrape the same site from another plugin, I don’t think that the case.

    April 14, 2020 at 12:37 pm #1834
    Nav TomNav
    Participant

    Any update?

    April 15, 2020 at 6:57 am #1839
    Suman M.Suman M.
    Keymaster

    In the above screenshot is showing access denied error with URL showing HTTP instead of HTTPS. I am able to scrape the same site from another plugin, I don’t think that the case.
    >> It’s not because of that. The above URL cannot be scraped via Scraper plugin. You can scrape from other URLs with https.
    We checked it using Scrape via Ajax feature and it could be scrapped – https://www.screencast.com/t/DxbwNj0cFC
    But this feature is not live yet and still under development phase.

  • Author
    Posts
Viewing 7 posts - 1 through 7 (of 7 total)

You must be logged in and have valid license to reply to this topic.

License required for the following item
Scraper - Automatic Content Crawler Plugin for WordPress
Login and Registration Log in · Register