Scraping data from Oddsportal

[strixee]

I'd like to scrape some odds from Oddsportal, but the data is stored in some XML database or what.
The Javascript code that manipulates it has to be this 0.5 MB monster

http://www.oddsportal.com/res/x/proto-1108031134.js

I don't think it's easy to acces the database directly, but at least gather the data using AJAX requests. Ha anyone worked with OP? Or do you have any advice how to research it? Should I start from functions such as

XMLHttpRequest, Ajax.Request

?

[uva3021]

screen scrape it, send an XML request to the web address and you'll get a response that you can then navigate the html file by accessing the properties

[strixee]

You mean using cURL or some already made application for the screen scraping?

[uva3021]

in vb

Set XMLHttpRequest = New MSXML2.XMLHTTP
XMLHttpRequest.Open "GET", URL, False
XMLHttpRequest.send

Set HTMLDoc = New HTMLDocument
HTMLDoc.body.innerHTML = XMLHttpRequest.responseText

[uva3021]

treat the HTML like an XML request

[vyomguy]

try curl.

[Pot luck]

In php

Get the HTML as a string using curl:

$url = "www.google.com";
$ch = curl_init(); // create curl resource
curl_setopt($ch, CURLOPT_URL, $url); // set url 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //return the transfer as a string 
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,2); // sets timeout to 2 seconds
$output = curl_exec($ch); 
curl_close($ch); // close curl resource to free up system resources 

Parse into a DOM object:

$doc = new domDocument;
$doc->loadHTML($output); 

[strixee]

Pot luck, this simple method works for non AJAX websites only.

[Pot luck]

Yeah true. I am scraping from the non-AJAX bit of oddsportal (eg http://www.oddsportal.com/matches/). What do you want to get from there? I emailed them a while back and asked how I could get this AJAX-fetched data but got a response along the lines of "huh, do what I dunno?".

Interested to see if you get further with this.

[strixee]

I want to get current odds of a few bookies, mainly SBO, 188 and Pinnacle.
The only easy thing is switching between the markets. For example for O/U 1st half you just need to add #over-under;3 to the URL.

[Pot luck]

Maybe hiring a freelancer would be the way to go.