Unable to parse data with BeautifulSoup: Python3











up vote
0
down vote

favorite












[Please click here to view the Tags][1]



The following is one of the tables of the website I am scrapping. Here, under 'tbody' I wish to click on the 'MS' button tag under both odd and even class which provides me a different table for further parsing it.
I am using Selenium and Python 3 to perform Web scrapping.
The current code only clicks on the 'MS' button in the first row. How can I create a for loop so that I can iterate through all the rows and click on 'MD' element in all the rows?



Thank you.



Following is the code:



table_0=table.find_element_by_tag_name('tbody')
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="odd"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="even"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()









share|improve this question




















  • 1




    Can you provide the URL? And please use the snippet tool available via edit to insert html so we can copy paste. And where is the MD text content element?
    – QHarr
    Nov 9 at 19:14












  • @QHarr The following is the URL: ibl.mdanderson.org/fasmic/#! I will edit the question so it will be easier to copy. Thanks
    – RRg
    Nov 9 at 19:16












  • @QHarr yes, you can just input an alphabet say, 'A', and from the drop-down you can select 'AKT1 (3 mutations)' . The go button doesn't work, so you will have to click enter or tab
    – RRg
    Nov 9 at 19:22










  • Are you trying to click all the buttons in all 3 rows?
    – QHarr
    Nov 9 at 19:27










  • @QHarr No, only the button with the text 'MS' in all 3 rows as that provides me a table of gene information of my interest
    – RRg
    Nov 9 at 19:33















up vote
0
down vote

favorite












[Please click here to view the Tags][1]



The following is one of the tables of the website I am scrapping. Here, under 'tbody' I wish to click on the 'MS' button tag under both odd and even class which provides me a different table for further parsing it.
I am using Selenium and Python 3 to perform Web scrapping.
The current code only clicks on the 'MS' button in the first row. How can I create a for loop so that I can iterate through all the rows and click on 'MD' element in all the rows?



Thank you.



Following is the code:



table_0=table.find_element_by_tag_name('tbody')
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="odd"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="even"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()









share|improve this question




















  • 1




    Can you provide the URL? And please use the snippet tool available via edit to insert html so we can copy paste. And where is the MD text content element?
    – QHarr
    Nov 9 at 19:14












  • @QHarr The following is the URL: ibl.mdanderson.org/fasmic/#! I will edit the question so it will be easier to copy. Thanks
    – RRg
    Nov 9 at 19:16












  • @QHarr yes, you can just input an alphabet say, 'A', and from the drop-down you can select 'AKT1 (3 mutations)' . The go button doesn't work, so you will have to click enter or tab
    – RRg
    Nov 9 at 19:22










  • Are you trying to click all the buttons in all 3 rows?
    – QHarr
    Nov 9 at 19:27










  • @QHarr No, only the button with the text 'MS' in all 3 rows as that provides me a table of gene information of my interest
    – RRg
    Nov 9 at 19:33













up vote
0
down vote

favorite









up vote
0
down vote

favorite











[Please click here to view the Tags][1]



The following is one of the tables of the website I am scrapping. Here, under 'tbody' I wish to click on the 'MS' button tag under both odd and even class which provides me a different table for further parsing it.
I am using Selenium and Python 3 to perform Web scrapping.
The current code only clicks on the 'MS' button in the first row. How can I create a for loop so that I can iterate through all the rows and click on 'MD' element in all the rows?



Thank you.



Following is the code:



table_0=table.find_element_by_tag_name('tbody')
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="odd"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="even"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()









share|improve this question















[Please click here to view the Tags][1]



The following is one of the tables of the website I am scrapping. Here, under 'tbody' I wish to click on the 'MS' button tag under both odd and even class which provides me a different table for further parsing it.
I am using Selenium and Python 3 to perform Web scrapping.
The current code only clicks on the 'MS' button in the first row. How can I create a for loop so that I can iterate through all the rows and click on 'MD' element in all the rows?



Thank you.



Following is the code:



table_0=table.find_element_by_tag_name('tbody')
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="odd"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="even"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()






python selenium-webdriver web-scraping beautifulsoup html-parsing






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 14 at 14:40

























asked Nov 9 at 19:04









RRg

105




105








  • 1




    Can you provide the URL? And please use the snippet tool available via edit to insert html so we can copy paste. And where is the MD text content element?
    – QHarr
    Nov 9 at 19:14












  • @QHarr The following is the URL: ibl.mdanderson.org/fasmic/#! I will edit the question so it will be easier to copy. Thanks
    – RRg
    Nov 9 at 19:16












  • @QHarr yes, you can just input an alphabet say, 'A', and from the drop-down you can select 'AKT1 (3 mutations)' . The go button doesn't work, so you will have to click enter or tab
    – RRg
    Nov 9 at 19:22










  • Are you trying to click all the buttons in all 3 rows?
    – QHarr
    Nov 9 at 19:27










  • @QHarr No, only the button with the text 'MS' in all 3 rows as that provides me a table of gene information of my interest
    – RRg
    Nov 9 at 19:33














  • 1




    Can you provide the URL? And please use the snippet tool available via edit to insert html so we can copy paste. And where is the MD text content element?
    – QHarr
    Nov 9 at 19:14












  • @QHarr The following is the URL: ibl.mdanderson.org/fasmic/#! I will edit the question so it will be easier to copy. Thanks
    – RRg
    Nov 9 at 19:16












  • @QHarr yes, you can just input an alphabet say, 'A', and from the drop-down you can select 'AKT1 (3 mutations)' . The go button doesn't work, so you will have to click enter or tab
    – RRg
    Nov 9 at 19:22










  • Are you trying to click all the buttons in all 3 rows?
    – QHarr
    Nov 9 at 19:27










  • @QHarr No, only the button with the text 'MS' in all 3 rows as that provides me a table of gene information of my interest
    – RRg
    Nov 9 at 19:33








1




1




Can you provide the URL? And please use the snippet tool available via edit to insert html so we can copy paste. And where is the MD text content element?
– QHarr
Nov 9 at 19:14






Can you provide the URL? And please use the snippet tool available via edit to insert html so we can copy paste. And where is the MD text content element?
– QHarr
Nov 9 at 19:14














@QHarr The following is the URL: ibl.mdanderson.org/fasmic/#! I will edit the question so it will be easier to copy. Thanks
– RRg
Nov 9 at 19:16






@QHarr The following is the URL: ibl.mdanderson.org/fasmic/#! I will edit the question so it will be easier to copy. Thanks
– RRg
Nov 9 at 19:16














@QHarr yes, you can just input an alphabet say, 'A', and from the drop-down you can select 'AKT1 (3 mutations)' . The go button doesn't work, so you will have to click enter or tab
– RRg
Nov 9 at 19:22




@QHarr yes, you can just input an alphabet say, 'A', and from the drop-down you can select 'AKT1 (3 mutations)' . The go button doesn't work, so you will have to click enter or tab
– RRg
Nov 9 at 19:22












Are you trying to click all the buttons in all 3 rows?
– QHarr
Nov 9 at 19:27




Are you trying to click all the buttons in all 3 rows?
– QHarr
Nov 9 at 19:27












@QHarr No, only the button with the text 'MS' in all 3 rows as that provides me a table of gene information of my interest
– RRg
Nov 9 at 19:33




@QHarr No, only the button with the text 'MS' in all 3 rows as that provides me a table of gene information of my interest
– RRg
Nov 9 at 19:33












1 Answer
1






active

oldest

votes

















up vote
0
down vote



accepted










You should be able to use a CSS selector to gather those for clicking



.btn-group.btn-group-xs button:first-child


The selector certainly works:





Not sure whether you need waits but maybe something like:



elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()




from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()

driver.get("https://ibl.mdanderson.org/fasmic/#!/")
driver.find_element_by_css_selector("input[type='text']").send_keys("AKT1 (3 mutations)")
driver.find_element_by_css_selector("input[type='text']").send_keys(Keys.RETURN)

elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()





share|improve this answer























  • I don’t see any change when performing manually so not sure what change should happen.
    – QHarr
    Nov 9 at 20:04










  • Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
    – RRg
    Nov 9 at 20:37











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53231864%2funable-to-parse-data-with-beautifulsoup-python3%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
0
down vote



accepted










You should be able to use a CSS selector to gather those for clicking



.btn-group.btn-group-xs button:first-child


The selector certainly works:





Not sure whether you need waits but maybe something like:



elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()




from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()

driver.get("https://ibl.mdanderson.org/fasmic/#!/")
driver.find_element_by_css_selector("input[type='text']").send_keys("AKT1 (3 mutations)")
driver.find_element_by_css_selector("input[type='text']").send_keys(Keys.RETURN)

elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()





share|improve this answer























  • I don’t see any change when performing manually so not sure what change should happen.
    – QHarr
    Nov 9 at 20:04










  • Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
    – RRg
    Nov 9 at 20:37















up vote
0
down vote



accepted










You should be able to use a CSS selector to gather those for clicking



.btn-group.btn-group-xs button:first-child


The selector certainly works:





Not sure whether you need waits but maybe something like:



elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()




from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()

driver.get("https://ibl.mdanderson.org/fasmic/#!/")
driver.find_element_by_css_selector("input[type='text']").send_keys("AKT1 (3 mutations)")
driver.find_element_by_css_selector("input[type='text']").send_keys(Keys.RETURN)

elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()





share|improve this answer























  • I don’t see any change when performing manually so not sure what change should happen.
    – QHarr
    Nov 9 at 20:04










  • Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
    – RRg
    Nov 9 at 20:37













up vote
0
down vote



accepted







up vote
0
down vote



accepted






You should be able to use a CSS selector to gather those for clicking



.btn-group.btn-group-xs button:first-child


The selector certainly works:





Not sure whether you need waits but maybe something like:



elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()




from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()

driver.get("https://ibl.mdanderson.org/fasmic/#!/")
driver.find_element_by_css_selector("input[type='text']").send_keys("AKT1 (3 mutations)")
driver.find_element_by_css_selector("input[type='text']").send_keys(Keys.RETURN)

elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()





share|improve this answer














You should be able to use a CSS selector to gather those for clicking



.btn-group.btn-group-xs button:first-child


The selector certainly works:





Not sure whether you need waits but maybe something like:



elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()




from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()

driver.get("https://ibl.mdanderson.org/fasmic/#!/")
driver.find_element_by_css_selector("input[type='text']").send_keys("AKT1 (3 mutations)")
driver.find_element_by_css_selector("input[type='text']").send_keys(Keys.RETURN)

elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()






share|improve this answer














share|improve this answer



share|improve this answer








edited Nov 9 at 19:56

























answered Nov 9 at 19:36









QHarr

27k81839




27k81839












  • I don’t see any change when performing manually so not sure what change should happen.
    – QHarr
    Nov 9 at 20:04










  • Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
    – RRg
    Nov 9 at 20:37


















  • I don’t see any change when performing manually so not sure what change should happen.
    – QHarr
    Nov 9 at 20:04










  • Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
    – RRg
    Nov 9 at 20:37
















I don’t see any change when performing manually so not sure what change should happen.
– QHarr
Nov 9 at 20:04




I don’t see any change when performing manually so not sure what change should happen.
– QHarr
Nov 9 at 20:04












Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
– RRg
Nov 9 at 20:37




Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
– RRg
Nov 9 at 20:37


















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53231864%2funable-to-parse-data-with-beautifulsoup-python3%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Xamarin.iOS Cant Deploy on Iphone

Glorious Revolution

Dulmage-Mendelsohn matrix decomposition in Python