Vfrdtyky

I am using BeautifulSoup to extract all the links from this page: http://kern.humdrum.org/search?s=t&keyword=Haydn

I am getting all these links this way:

# -*- coding: utf-8 -*-



from urllib.request import urlopen as uReq

from bs4 import BeautifulSoup as soup



my_url = 'http://kern.humdrum.org/search?s=t&keyword=Haydn'



#opening up connecting, grabbing the page

uClient = uReq(my_url)



# put all the content in a variable

page_html = uClient.read()



#close the internet connection

uClient.close()



#It does my HTML parser

page_soup = soup(page_html, "html.parser")



# Grab all of the links

containers = page_soup.findAll('a', href=True)

#print(type(containers))



for container in containers:

    link = container

    #start_index = link.index('href="') 

    print(link)

    print("---")

    #print(start_index)

part of my output is:

Notice that it is returning several links but I really want all the ones with >Someting. (For example, ">Allegro" and "Allegro vivace" and so forth).

I am having a hard time getting the following type of output (example of the image):
"Allegro - http://kern.ccarh.org/cgi-bin/ksdata?location=users/craig/classical/beethoven/piano/sonata&file=sonata01-1.krn&format=info"

In other words, at this point, I have a bunch of anchor tags (+- 1000). From all these tags there are a bunch that are just "trash" and +- 350 of tags that I would like to extract. All these tags look almost the same but the only difference is that the tags that I need have a "> Somebody's name<a>" at the end. I would like to exctract only the link of all the anchor tags with this characteristic.

From what I can see in the image the ones with info have an href attribute containing format="info" so you could use an attribute=value CSS selector of [href*=format="info"] , where the * indicates contains; the attribute value contains the substring after the first equals.

import bs4 , requests



res = requests.get("http://kern.humdrum.org/search?s=t&keyword=Haydn")

soup = bs4.BeautifulSoup(res.text,"html.parser")

for link in soup.select('[href*=format="info"]'):

    print(link.getText(), link['href'])

The best and easiest way is using text attribute when printing the link. like this :
print link.text

Assuming you already have a list of the substrings you need to search for, you can do something like:

for link in containers:

    text = link.get_text().lower()

    if any(text.endswith(substr) for substr in substring_list):

        print(link)

        print('---')

you want to extract link with specified anchor text?

for container in containers:

    link = container

    # match exact

    #if 'Allegro di molto' == link.text:

    if 'Allegro' in link.text: # contain

        print(link)

        print("---")