我正在嘗試從一些網(wǎng)站上抓取數(shù)據(jù)以進(jìn)行概念驗(yàn)證項(xiàng)目。目前使用 Python3 和 BS4 來(lái)收集所需的數(shù)據(jù)。我有一本來(lái)自三個(gè)站點(diǎn)的 URLS 字典。每個(gè)站點(diǎn)都需要不同的方法來(lái)收集數(shù)據(jù)，因?yàn)樗鼈兊?HTML 不同。我一直在使用“Try, If, Else, stack 但我一直遇到問(wèn)題，如果你能看看我的代碼并幫助我修復(fù)它那就太好了！當(dāng)我添加更多要抓取的站點(diǎn)時(shí)，我將無(wú)法使用“Try、If、Else”循環(huán)通過(guò)各種方法來(lái)找到抓取數(shù)據(jù)的正確方法，我如何才能讓這段代碼面向未來(lái)添加未來(lái)有多少網(wǎng)站并從其中包含的各種元素中抓取數(shù)據(jù)？# Scraping Script Here:def job():prices = { # LIVEPRICES "LIVEAUOZ": {"url": "https://www.gold.co.uk/", "trader": "Gold.co.uk", "metal": "Gold", "type": "LiveAUOz"}, # GOLD "GLDAU_BRITANNIA": {"url": "https://www.gold.co.uk/gold-coins/gold-britannia-coins/britannia-one-ounce-gold-coin-2020/", "trader": "Gold.co.uk", "metal": "Gold", "type": "Britannia"}, "GLDAU_PHILHARMONIC": {"url": "https://www.gold.co.uk/gold-coins/austrian-gold-philharmoinc-coins/austrian-gold-philharmonic-coin/", "trader": "Gold.co.uk", "metal": "Gold", "type": "Philharmonic"}, "GLDAU_MAPLE": {"url": "https://www.gold.co.uk/gold-coins/canadian-gold-maple-coins/canadian-gold-maple-coin/", "trader": "Gold.co.uk", "metal": "Gold", "type": "Maple"}, # SILVER "GLDAG_BRITANNIA": {"url": "https://www.gold.co.uk/silver-coins/silver-britannia-coins/britannia-one-ounce-silver-coin-2020/", "trader": "Gold.co.uk", "metal": "Silver", "type": "Britannia"}, "GLDAG_PHILHARMONIC": {"url": "https://www.gold.co.uk/silver-coins/austrian-silver-philharmonic-coins/silver-philharmonic-2020/", "trader": "Gold.co.uk", "metal": "Silver", "type": "Philharmonic"}}
1 回答
三國(guó)紛爭(zhēng)
TA貢獻(xiàn)1804條經(jīng)驗(yàn) 獲得超7個(gè)贊
為抓取添加一個(gè)配置，其中每個(gè)配置都是這樣的：
prices = {
"LIVEAUOZ": {
"url": "https://www.gold.co.uk/",
"trader": "Gold.co.uk",
"metal": "Gold",
"type": "LiveAUOz",
"price": {
"selector": '#id > div > table > tr',
"parser": lambda x: float(re.sub(r"[^0-9\.]", "", x))
}
}
}
使用 price 的選擇器部分獲取 HTML 的相關(guān)部分，然后使用解析器函數(shù)對(duì)其進(jìn)行解析。
例如
for key, config in prices.items():
response = requests.get(config['url'])
soup = BeautifulSoup(response.text, 'html.parser')
price_element = soup.find(config['price']['selector'])
if price_element:
AG_GRAM_SPOT = price_element.get_text()
# convert to float
AG_GRAM_SPOT = config['price']['parser'](AG_GRAM_SPOT)
# etc
您可以根據(jù)需要修改配置對(duì)象，但對(duì)于大多數(shù)站點(diǎn)來(lái)說(shuō)它可能非常相似。例如，文本解析很可能總是相同的，所以不用 lambda 函數(shù)，而是用 def 創(chuàng)建一個(gè)函數(shù)。
def textParser(text):
return float(re.sub(r"[^0-9\.]", "", text))
然后在配置中添加對(duì) textParser 的引用。
prices = {
"LIVEAUOZ": {
"url": "https://www.gold.co.uk/",
"trader": "Gold.co.uk",
"metal": "Gold",
"type": "LiveAUOz",
"price": {
"selector": '#id > div > table > tr',
"parser": textParser
}
}
}
這些步驟將允許您編寫通用代碼，保存所有那些嘗試異常。
添加回答
舉報(bào)
0/150
提交
取消