我已經(jīng)編寫了基于 selenium 的函數(shù)，我希望它同時(shí)解析多個(gè)網(wǎng)頁(yè)。我有我傳遞給我想要同時(shí)抓取的函數(shù)的 url 列表，以節(jié)省時(shí)間。我創(chuàng)建了 scraper.py 文件，我在其中放置了 scraper 函數(shù)：def parser_od(url): price=[] url_of = url driver.get(url_of) try: price.append(browser.find_element_by_xpath("//*[@id='root']/article/header/div[2]/div[1]/div[2]").text.replace(" ","").replace("z?","").replace(",",".")) except NoSuchElementException: price.append("")multiprocessing現(xiàn)在我想使用該函數(shù)使用庫(kù)同時(shí)從我的 url 解析多個(gè) url ：from scraper import *url_list=['https://www.otodom.pl/oferta/2-duze-pokoje-we-wrzeszczu-do-zamieszania-ID42f6s', 'https://www.otodom.pl/oferta/mieszkanie-na-zamknietym-osiedlu-z-ogrodkiem-ID40ZxM', 'https://www.otodom.pl/oferta/zaciszna-nowe-mieszkanie-3-pokoje-0-ID41UaX', 'https://www.otodom.pl/oferta/dwupoziomowe-dewel-mieszkanie-101-m2-lebork-i-p-ID3JEcQ']driver = webdriver.Chrome(executable_path=r"C:\Users\Admin\chromedriver.exe")from multiprocessing import Poolwith Pool(4) as p: price = p.map(parser_od, url_list)但我收到以下錯(cuò)誤：NameError: name 'driver' is not defined這很奇怪，因?yàn)?chrome 被打開了。編輯：我需要在運(yùn)行此刮板時(shí)打開瀏覽器，以便在每次調(diào)用此函數(shù)時(shí)都打開驅(qū)動(dòng)程序。
1 回答
呼啦一陣風(fēng)
TA貢獻(xiàn)1802條經(jīng)驗(yàn) 獲得超6個(gè)贊
只是應(yīng)該將要處理 ino 4 個(gè)相等部分的 url 列表拆分，并driver為每個(gè)處理Pool.
def parser_od(urls, thread_index):
driver = webdriver.Chrome(executable_path=r"C:\Users\Admin\chromedriver.exe")
prices = []
for i in range(len(urls)):
url = urls[i]
if i % 4 == thread_index:
price=[]
url_of = url
driver.get(url_of)
try:
price.append(browser.find_element_by_xpath("//*[@id='root']/article/header/div[2]/div[1]/div[2]").text.replace(" ","").replace("z?","").replace(",","."))
except NoSuchElementException:
price.append("")
prices.append(price)
return prices
from multiprocessing import Pool
with Pool(4) as p:
price = p.map(lambda x: parser_od(x, url_list), list(range(len(url_list))))
添加回答
舉報(bào)
0/150
提交
取消