Does oddsportal block scrapers somehow?

[owain]

I'm writing a python script using Requests and BeautifulSoup to scrape very select data from oddsportal; I'm only in the early stages, but it's functional on every website I've tried except for oddsportal.

The issue occurs during the HTML grabbing. Regardless of which specific oddsportal page I try to grab, whether it be the homepage or a league page, it returns the same oddsportal error page HTML which contains none of the relevant information, along with "Odds Portal: Page Not Found" and "The page you requested is not available", etc. I can post the entire HTML if necessary, but it really does contain nothing of interest.

I'm wondering if this is due to oddsportal blocking the HTML grab in some way, or whether it's due to javascript or something else that I'm missing. Any insight and/or remedies would be much appreciated.

[vampire assassin]

I'm writing a python script using Requests and BeautifulSoup to scrape very select data from oddsportal; I'm only in the early stages, but it's functional on every website I've tried except for oddsportal.

The issue occurs during the HTML grabbing. Regardless of which specific oddsportal page I try to grab, whether it be the homepage or a league page, it returns the same oddsportal error page HTML which contains none of the relevant information, along with "Odds Portal: Page Not Found" and "The page you requested is not available", etc. I can post the entire HTML if necessary, but it really does contain nothing of interest.

I'm wondering if this is due to oddsportal blocking the HTML grab in some way, or whether it's due to javascript or something else that I'm missing. Any insight and/or remedies would be much appreciated.

Oddsportal's site is formatted weirdly. I had trouble scraping it also. I'm no expert scraper, but the windows inside the display blew up my scraper.

[allnighter]

I'm also very interested on learning more about Python and web scraping but I'm, at best, a noob in coding.

You might try to ask this question on the stack overflow site. If it's a coding matter they might be able to help.
Cheers.

[A4K]

You guys would be much better served if you hire someone on Fiverr to do the scraping for you. Cost you a few bucks but it will save you a lot of time.

[allnighter]

You guys would be much better served if you hire someone on Fiverr to do the scraping for you. Cost you a few bucks but it will save you a lot of time.

Thanks for the info A4K.
I kinda of challenged myself to learn more coding and scraping but I think eventually will end up paying for that )

[Larkman]

They use javascript to populate the data tables using JSON data, so if you do a straight programmatic http request it will receive the html prior to this population so the odds data won't be there. Your options are to either open up your browser network console and try to decipher the various calls to get the JSON and then replicate that programmatically, which I tried doing and is a pain in the arse, or to use a headless browser like selenium/phantomjs, which will behave like a normal browser and allow you to scrape the html post data population, which is much easier but considerably slower when scraping a large number of pages.

I have a fully functional oddsportal scraper in C# so I'm happy to answer any more specific questions you have.

[Larkman]

Also now that I read your OP again, the reason you get a page not found error in the html is because as standard, programmatic http requests don't have a user-agent string set, and OddsPortal returns that page whenever there is no user-agent. Set it to "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" or some other valid string and you should at least not get the error anymore.

[turtledoves]

what url are you trying to scrape? try passing Referer header

"Referer: abc"

Selenium with chromedriver is most futureproof, it looks like a real browser is visiting. you might have to constantly change the scripts to fake the headers and play the cat and mouse game if you go the other route.

maybe find inspiration from this thread if its blocked in the future

https://github.com/seemethere/nba_py/issues/88

[owain]

Thanks for this information. I have to be honest here and admit that I have been getting some help from my son with this. He is abroad at the moment so I'll update him as soon as he gets back.
Thanks again.

[arwar]

interesting about headless browser!

[owain]

I took A4K's advice, and paid someone to do it.

[turtledoves]

https://developers.google.com/web/updates/2017/04/headless-chrome#drivers