我正在嘗試抓取此網(wǎng)站上的排名表：https ://www.timeshighereducation.com/world-university-rankings/2021/world-ranking#!/page/0/length/25/sort_by/scores_overall/sort_order /asc/cols/分?jǐn)?shù)但我無法獲取數(shù)據(jù)，現(xiàn)在我有這個(gè)代碼：import scrapyfrom scrapy import Selectorfrom selenium import webdriverfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as ECfrom logzero import logfile, loggerclass ScrapeTableSpider(scrapy.Spider): name = "scrape-table" allowed_domains = ["toscrape.com"] start_urls = ['http://quotes.toscrape.com'] def start_requests(self): # headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0'} for url in self.start_urls: yield scrapy.Request(url=url, callback=self.parse) def parse(self, response): # driver = webdriver.Chrome() options = webdriver.ChromeOptions() options.add_argument("headless") desired_capabilities = options.to_capabilities() driver = webdriver.Chrome('C:/chromedriver', desired_capabilities=desired_capabilities) driver.get("https://www.timeshighereducation.com/world-university-rankings/2021/world-ranking#!/page/0/length/25/sort_by/scores_overall/sort_order/asc/cols/scores") driver.implicitly_wait(2) for table in driver.find_element_by_xpath('//*[contains(@id,"datatable-1")]//tr'): data = [item.text for item in table.find_elements_by_xpath(".//*[self::td or self::th]")] print(data)任何有關(guān)如何從表中獲取數(shù)據(jù)的見解將不勝感激。
2 回答
繁花不似錦
TA貢獻(xiàn)1851條經(jīng)驗(yàn) 獲得超4個(gè)贊
我真的不明白為什么你同時(shí)使用 scrapy 和 selenium，但我們可以說只是使用 selenium。要從表中獲取文本，您可以執(zhí)行以下非常簡(jiǎn)單的操作：
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("headless")
desired_capabilities = options.to_capabilities()
driver = webdriver.Chrome('C:/chromedriver', desired_capabilities=desired_capabilities)
driver.get("https://www.timeshighereducation.com/world-university-rankings/2021/world-ranking#!/page/0/length/25/sort_by/scores_overall/sort_order/asc/cols/scores")
driver.implicitly_wait(1)
table = driver.find_element_by_xpath('//*[@id="datatable-1"]')
print(table.text)
現(xiàn)在，如果您將表中的所有內(nèi)容分開，只需使用該find_element_by_xxx函數(shù)并通過 xpath 選擇其他部分即可。
慕慕森
TA貢獻(xiàn)1856條經(jīng)驗(yàn) 獲得超17個(gè)贊
如果您需要迭代結(jié)果，您應(yīng)該選擇 elements 而不是 element。更改您的代碼：
for table in driver.find_element_by_xpath('//*[contains(@id,"datatable-1")]//tr'):編碼：
for table in driver.find_elements_by_xpath('//*[contains(@id,"datatable-1")]//tr'):添加回答
舉報(bào)
0/150
提交
取消