Vfrdtyky

I'm trying to scrape the list of universities in the United States. I've tried looking around for hours but nothing is working (i.e. other methods just crash the console). Here's what I have so far.

The HTML is Formatted as follows:

<ol>

<a name="A"><b>A</b></a><br/>

<p>

<li><a href="http://www.acu.edu/">

    Abilene Christian University</a> (acu.edu)



<li><a href="http://www.adelphi.edu/">

    Adelphi University</a> (adelphi.edu)



<li><a href="http://www.scottlan.edu/">

        Agnes Scott College</a> (scottlan.edu)



<li><a href="http://www.afit.af.mil/">

    Air Force Institute of Technology</a> (afit.af.mil)

This is my code:

from bs4 import BeautifulSoup as soup

from urllib.request import urlopen as uReq



#Site for list scraping 

my_url = "http://doors.stanford.edu/~sr/universities.html"



#Open connection and grab the page

uClient = uReq(my_url)



#Save contents to variable

page_html = uClient.read()



#Close connection

uClient.close()



#Html parsing

page_soup = soup(page_html, "html.parser")



#Checking the list

page_soup.ol

I've tried page_soup.findChildren("li") as well as page_soup.find("li", {"class":"text"}) and countless others to no avail.

Help?

I just simply try page_soup.find_all("li") and I can get all the <li> tag.

Don't know why it's unable to get <li> inside the <ol> by "ol.getChildren()", there is also a post of it Unable to scrape <li> tag inside the <ol> tag using beautiful soup.

After looking at the documentation and experimenting I figured it out. It's kind of dirty though so you'll have to clean it.

#Get the list

listofuni = [li.text for li in page_soup.findAll('li')]