Unable to parse data with BeautifulSoup: Python3
up vote
0
down vote
favorite
[Please click here to view the Tags][1]
The following is one of the tables of the website I am scrapping. Here, under 'tbody' I wish to click on the 'MS' button tag under both odd and even class which provides me a different table for further parsing it.
I am using Selenium and Python 3 to perform Web scrapping.
The current code only clicks on the 'MS' button in the first row. How can I create a for loop so that I can iterate through all the rows and click on 'MD' element in all the rows?
Thank you.
Following is the code:
table_0=table.find_element_by_tag_name('tbody')
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="odd"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="even"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
python selenium-webdriver web-scraping beautifulsoup html-parsing
add a comment |
up vote
0
down vote
favorite
[Please click here to view the Tags][1]
The following is one of the tables of the website I am scrapping. Here, under 'tbody' I wish to click on the 'MS' button tag under both odd and even class which provides me a different table for further parsing it.
I am using Selenium and Python 3 to perform Web scrapping.
The current code only clicks on the 'MS' button in the first row. How can I create a for loop so that I can iterate through all the rows and click on 'MD' element in all the rows?
Thank you.
Following is the code:
table_0=table.find_element_by_tag_name('tbody')
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="odd"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="even"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
python selenium-webdriver web-scraping beautifulsoup html-parsing
1
Can you provide the URL? And please use the snippet tool available via edit to insert html so we can copy paste. And where is the MD text content element?
– QHarr
Nov 9 at 19:14
@QHarr The following is the URL: ibl.mdanderson.org/fasmic/#! I will edit the question so it will be easier to copy. Thanks
– RRg
Nov 9 at 19:16
@QHarr yes, you can just input an alphabet say, 'A', and from the drop-down you can select 'AKT1 (3 mutations)' . The go button doesn't work, so you will have to click enter or tab
– RRg
Nov 9 at 19:22
Are you trying to click all the buttons in all 3 rows?
– QHarr
Nov 9 at 19:27
@QHarr No, only the button with the text 'MS' in all 3 rows as that provides me a table of gene information of my interest
– RRg
Nov 9 at 19:33
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
[Please click here to view the Tags][1]
The following is one of the tables of the website I am scrapping. Here, under 'tbody' I wish to click on the 'MS' button tag under both odd and even class which provides me a different table for further parsing it.
I am using Selenium and Python 3 to perform Web scrapping.
The current code only clicks on the 'MS' button in the first row. How can I create a for loop so that I can iterate through all the rows and click on 'MD' element in all the rows?
Thank you.
Following is the code:
table_0=table.find_element_by_tag_name('tbody')
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="odd"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="even"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
python selenium-webdriver web-scraping beautifulsoup html-parsing
[Please click here to view the Tags][1]
The following is one of the tables of the website I am scrapping. Here, under 'tbody' I wish to click on the 'MS' button tag under both odd and even class which provides me a different table for further parsing it.
I am using Selenium and Python 3 to perform Web scrapping.
The current code only clicks on the 'MS' button in the first row. How can I create a for loop so that I can iterate through all the rows and click on 'MD' element in all the rows?
Thank you.
Following is the code:
table_0=table.find_element_by_tag_name('tbody')
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="odd"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
for buttons in table_0.find_elements_by_tag_name("tr"):
buttons.find_elements_by_xpath('//tr[@class="even"]')
buttons.find_element_by_xpath('//button[text()="MS"]').click()
python selenium-webdriver web-scraping beautifulsoup html-parsing
python selenium-webdriver web-scraping beautifulsoup html-parsing
edited Nov 14 at 14:40
asked Nov 9 at 19:04
RRg
105
105
1
Can you provide the URL? And please use the snippet tool available via edit to insert html so we can copy paste. And where is the MD text content element?
– QHarr
Nov 9 at 19:14
@QHarr The following is the URL: ibl.mdanderson.org/fasmic/#! I will edit the question so it will be easier to copy. Thanks
– RRg
Nov 9 at 19:16
@QHarr yes, you can just input an alphabet say, 'A', and from the drop-down you can select 'AKT1 (3 mutations)' . The go button doesn't work, so you will have to click enter or tab
– RRg
Nov 9 at 19:22
Are you trying to click all the buttons in all 3 rows?
– QHarr
Nov 9 at 19:27
@QHarr No, only the button with the text 'MS' in all 3 rows as that provides me a table of gene information of my interest
– RRg
Nov 9 at 19:33
add a comment |
1
Can you provide the URL? And please use the snippet tool available via edit to insert html so we can copy paste. And where is the MD text content element?
– QHarr
Nov 9 at 19:14
@QHarr The following is the URL: ibl.mdanderson.org/fasmic/#! I will edit the question so it will be easier to copy. Thanks
– RRg
Nov 9 at 19:16
@QHarr yes, you can just input an alphabet say, 'A', and from the drop-down you can select 'AKT1 (3 mutations)' . The go button doesn't work, so you will have to click enter or tab
– RRg
Nov 9 at 19:22
Are you trying to click all the buttons in all 3 rows?
– QHarr
Nov 9 at 19:27
@QHarr No, only the button with the text 'MS' in all 3 rows as that provides me a table of gene information of my interest
– RRg
Nov 9 at 19:33
1
1
Can you provide the URL? And please use the snippet tool available via edit to insert html so we can copy paste. And where is the MD text content element?
– QHarr
Nov 9 at 19:14
Can you provide the URL? And please use the snippet tool available via edit to insert html so we can copy paste. And where is the MD text content element?
– QHarr
Nov 9 at 19:14
@QHarr The following is the URL: ibl.mdanderson.org/fasmic/#! I will edit the question so it will be easier to copy. Thanks
– RRg
Nov 9 at 19:16
@QHarr The following is the URL: ibl.mdanderson.org/fasmic/#! I will edit the question so it will be easier to copy. Thanks
– RRg
Nov 9 at 19:16
@QHarr yes, you can just input an alphabet say, 'A', and from the drop-down you can select 'AKT1 (3 mutations)' . The go button doesn't work, so you will have to click enter or tab
– RRg
Nov 9 at 19:22
@QHarr yes, you can just input an alphabet say, 'A', and from the drop-down you can select 'AKT1 (3 mutations)' . The go button doesn't work, so you will have to click enter or tab
– RRg
Nov 9 at 19:22
Are you trying to click all the buttons in all 3 rows?
– QHarr
Nov 9 at 19:27
Are you trying to click all the buttons in all 3 rows?
– QHarr
Nov 9 at 19:27
@QHarr No, only the button with the text 'MS' in all 3 rows as that provides me a table of gene information of my interest
– RRg
Nov 9 at 19:33
@QHarr No, only the button with the text 'MS' in all 3 rows as that provides me a table of gene information of my interest
– RRg
Nov 9 at 19:33
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
accepted
You should be able to use a CSS selector to gather those for clicking
.btn-group.btn-group-xs button:first-child
The selector certainly works:
Not sure whether you need waits but maybe something like:
elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
driver.get("https://ibl.mdanderson.org/fasmic/#!/")
driver.find_element_by_css_selector("input[type='text']").send_keys("AKT1 (3 mutations)")
driver.find_element_by_css_selector("input[type='text']").send_keys(Keys.RETURN)
elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()
I don’t see any change when performing manually so not sure what change should happen.
– QHarr
Nov 9 at 20:04
Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
– RRg
Nov 9 at 20:37
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
accepted
You should be able to use a CSS selector to gather those for clicking
.btn-group.btn-group-xs button:first-child
The selector certainly works:
Not sure whether you need waits but maybe something like:
elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
driver.get("https://ibl.mdanderson.org/fasmic/#!/")
driver.find_element_by_css_selector("input[type='text']").send_keys("AKT1 (3 mutations)")
driver.find_element_by_css_selector("input[type='text']").send_keys(Keys.RETURN)
elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()
I don’t see any change when performing manually so not sure what change should happen.
– QHarr
Nov 9 at 20:04
Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
– RRg
Nov 9 at 20:37
add a comment |
up vote
0
down vote
accepted
You should be able to use a CSS selector to gather those for clicking
.btn-group.btn-group-xs button:first-child
The selector certainly works:
Not sure whether you need waits but maybe something like:
elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
driver.get("https://ibl.mdanderson.org/fasmic/#!/")
driver.find_element_by_css_selector("input[type='text']").send_keys("AKT1 (3 mutations)")
driver.find_element_by_css_selector("input[type='text']").send_keys(Keys.RETURN)
elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()
I don’t see any change when performing manually so not sure what change should happen.
– QHarr
Nov 9 at 20:04
Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
– RRg
Nov 9 at 20:37
add a comment |
up vote
0
down vote
accepted
up vote
0
down vote
accepted
You should be able to use a CSS selector to gather those for clicking
.btn-group.btn-group-xs button:first-child
The selector certainly works:
Not sure whether you need waits but maybe something like:
elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
driver.get("https://ibl.mdanderson.org/fasmic/#!/")
driver.find_element_by_css_selector("input[type='text']").send_keys("AKT1 (3 mutations)")
driver.find_element_by_css_selector("input[type='text']").send_keys(Keys.RETURN)
elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()
You should be able to use a CSS selector to gather those for clicking
.btn-group.btn-group-xs button:first-child
The selector certainly works:
Not sure whether you need waits but maybe something like:
elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
driver.get("https://ibl.mdanderson.org/fasmic/#!/")
driver.find_element_by_css_selector("input[type='text']").send_keys("AKT1 (3 mutations)")
driver.find_element_by_css_selector("input[type='text']").send_keys(Keys.RETURN)
elements = driver.find_elements_by_css_selector(".btn-group.btn-group-xs button:first-child")
for element in elements:
element.click()
edited Nov 9 at 19:56
answered Nov 9 at 19:36
QHarr
27k81839
27k81839
I don’t see any change when performing manually so not sure what change should happen.
– QHarr
Nov 9 at 20:04
Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
– RRg
Nov 9 at 20:37
add a comment |
I don’t see any change when performing manually so not sure what change should happen.
– QHarr
Nov 9 at 20:04
Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
– RRg
Nov 9 at 20:37
I don’t see any change when performing manually so not sure what change should happen.
– QHarr
Nov 9 at 20:04
I don’t see any change when performing manually so not sure what change should happen.
– QHarr
Nov 9 at 20:04
Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
– RRg
Nov 9 at 20:37
Thanks a lot!!! This worked. I guess I was trying to implement using a logic based on the class.
– RRg
Nov 9 at 20:37
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53231864%2funable-to-parse-data-with-beautifulsoup-python3%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Can you provide the URL? And please use the snippet tool available via edit to insert html so we can copy paste. And where is the MD text content element?
– QHarr
Nov 9 at 19:14
@QHarr The following is the URL: ibl.mdanderson.org/fasmic/#! I will edit the question so it will be easier to copy. Thanks
– RRg
Nov 9 at 19:16
@QHarr yes, you can just input an alphabet say, 'A', and from the drop-down you can select 'AKT1 (3 mutations)' . The go button doesn't work, so you will have to click enter or tab
– RRg
Nov 9 at 19:22
Are you trying to click all the buttons in all 3 rows?
– QHarr
Nov 9 at 19:27
@QHarr No, only the button with the text 'MS' in all 3 rows as that provides me a table of gene information of my interest
– RRg
Nov 9 at 19:33