Vfrdtyky

I would like to scrape all of the NFL betting lines/info from https://www.bovada.lv/sports/football/nfl using Python.

With help from the SO community I found the api for the NFL games: https://www.bovada.lv/services/sports/event/v2/events/A/description/football/nfl

You can check out some more info regarding the code on my pastebin:
https://pastebin.com/tmAenaBD

Here is my code so far:

#Just a heads up...The code loads slow since it is printing the entire source



import requests 



source = requests.get("https://www.bovada.lv/services/sports/event/v2/events/A/description/football/nfl").json()



print(source)

I can see the data in the code, however, I am unaware of how to parse it. Any suggestions?

The betting line info I am looking for includes the following for all games (13 games in total this week):

1) Date of game

2) Time of game

3) Teams playing (two teams per game)

4) The Spread (with odds)

5) The Win aka "the moneyline" (with odds)

6) The Total aka "the over/under" (with odds)

Here is a pic of the first game:

I would like all of the games if possible.

I am new to scraping with no prior experience coding. I appreciate any help I can get and I am grateful for your time and efforts.

You are pretty much there. You've hit he endpoint and transformed the source into a python object (or a collection of python objects), which is what .json() does.

So now you are looking for a systematic way to access the data that you want but you are unsure about the structure of the data... so the easiest way to address that is to look at it.

Python has a built in functionality to help you to find out what objects are. Your best friend is the function type(). Lets have a look at what source is:

>>> type(source)

<class 'list'>

Alright, so source is a list, therefore we can test for it's length:

>>> len(source)

1

OK, so it's a list with only one element inside. What does the list have in it?

>>> type(source[0])

<class 'dict'>

OK, so source is a list with a single dict inside. That dict has to be where all of the data is, lets isolate it:

>>> data = source[0]

The python dict has some handy functions that allow inspection as well. For starters we can have a look at what the keys of data are:

>>> data.keys()

dict_keys(['path', 'events'])

So only 2 keys, one called path and one called events. As you are looking for the betting info for every game, lets have a look at the events key, first we'll see what it is:

>>> type(data['events'])

<class 'list'>

>>> len(data['events'])

13

OK, presumably a list of NFL matches, what type are the contents of that list:

>>> set(type(e) for e in data['events'])

{<class 'dict'>}

So they are all dicts, are they all of the same structure?

>>> all(data['events'][0].keys() == e.keys() for e in data['events'][1:])

True

Yes, all the same structure. What is the structure?

>>> data['events'][0].keys()

dict_keys(['id', 'description', 'type', 'link', 'status', 'sport', 'startTime', 'live', 'awayTeamFirst', 'denySameGame', 'teaserAllowed', 'competitionId', 'notes', 'numMarkets', 'lastModified', 'competitors', 'displayGroups'])

Again, you have to know what everything actually is, before you can reason about it:

>>> for k, v in data['events'][0].items():

...     print(k, type(v))

...

id <class 'str'>

description <class 'str'>

type <class 'str'>

link <class 'str'>

status <class 'str'>

sport <class 'str'>

startTime <class 'int'>

live <class 'bool'>

awayTeamFirst <class 'bool'>

denySameGame <class 'bool'>

teaserAllowed <class 'bool'>

competitionId <class 'str'>

notes <class 'str'>

numMarkets <class 'int'>

lastModified <class 'int'>

competitors <class 'list'>

displayGroups <class 'list'>

Of those keys, only two hold collections as values, competitors and displayGroups. So any market data must be contained in either one of those.

I'm not going to do the whole job for you, but I hope you get the picture. When you are working with external data sources that don't have any documentation, inspect the object systematically so that know what you are dealing with.