我正在從事一個大型網(wǎng)頁抓取項目，其中每個網(wǎng)頁的 HTML 結構都彼此不同。我想從網(wǎng)頁上抓取產(chǎn)品描述，并且我正在使用 BeautifulSoup 包。例如，我嘗試抓取的產(chǎn)品描述存儲在 HTML 結構中：<div class="product-description"> <p> "Title" </p> <p> "Some content" </p> <p> "Product description" </p></div><div class="product-description"> <p> "Title" </p> <p> "Product description" </p></div><div class="product-description"> <p> "Title" </p> <p> "Some content" </p> <p> "Some content" </p> <p> "Product description" </p></div><div class="product-description"> <p> "Title" </p> <p> "Some-content" </p> <p> "Some-content" </p> <p> "Some-content" </p> <p> "Product description" </p></div>我編寫了一個 for 循環(huán)，根據(jù)頁面結構從 div 類“產(chǎn)品描述”獲取數(shù)據(jù)。我的示例代碼片段：requests = (grequests.get(url) for url in urls)responses = grequests.imap(requests, grequests.Pool(1000))for response in responses: html_soup = BeautifulSoup(response.text, 'html.parser') if html_soup.find('div',class_='product_description').next_element.next_sibling.next_sibling.next_sibling.next_sibling: product_description = html_soup.find('div',class_='product_description').next_element.next_sibling.next_sibling.next_sibling.next_sibling.text elif html_soup.find('div', class_='product-description').next_element.next_sibling.next_sibling.next_sibling: product_description = html_soup.find( 'div', class_='product_description').next_element.next_sibling.next_sibling.next_sibling.text elif html_soup.find('div', class_='product-description').next_element.next_sibling.next_sibling: product_description = html_soup.find( 'div', class_='product_description').next_element.next_sibling.next_sibling.text else: product_description = html_soup.find( 'div', class_='product_description').next_element.next_sibling.text我期望 if 條件檢查當前 HTML 級別是否有同級，如果沒有則檢查后續(xù)條件。然而，經(jīng)過 3000 次迭代后，我得到了Attribute error一句話Nonetype object has no attribute next_sibling。下面附上截圖：我知道一定有其他更簡單的方法來處理這個動態(tài)頁面結構。任何幫助將非常感激。提前致謝！
1 回答
斯蒂芬大帝
TA貢獻1827條經(jīng)驗 獲得超8個贊
嘗試這個：
for i in soup.find_all('div',class_="product-description"):
try:
print(i.find_all('p')[-1].text)
except:
pass
這里的湯是：
<div class="product-description">
<p> "Title" </p>
<p> "Some content" </p>
<p> "Product description" </p>
</div>
<div class="product-description">
<p> "Title" </p>
<p> "Product description" </p>
</div>
<div class="product-description">
<p> "Title" </p>
<p> "Some content" </p>
<p> "Some content" </p>
<p> "Product description" </p>
</div>
<div class="product-description">
<p> "Title" </p>
<p> "Some-content" </p>
<p> "Some-content" </p>
<p> "Some-content" </p>
<p> "Product description" </p>
</div>
- 1 回答
- 0 關注
- 127 瀏覽
添加回答
舉報
0/150
提交
取消