我正在使用蟒蛇3.8.2和bs4美麗湯。我正在嘗試查找標(biāo)記的所有實例，并在結(jié)果集中列出每個實例，每行一個。但是，返回的結(jié)果集包含的行數(shù)多于網(wǎng)站的原始抓取。這是因為結(jié)果集的第一行包含標(biāo)記的所有實例。下一行包含除第一個實例之外的所有實例，第三行包含除第一個實例和第二個實例之外的所有實例，依此類推，以及結(jié)果集的其余部分。代碼如下：from bs4 import BeautifulSoupimport requestsurl = "https://www.sainsburys.co.uk/shop/gb/groceries/drinks/seeall"html_content = requests.get(url, timeout=5)soup = BeautifulSoup(html_content.text)test_1 = soup.find('ul',{"class": "productLister gridView"})test = test_1.find_all("li", attrs={"class": "gridItem"})我如何獲得它，以便每個實例僅列出本身，每行一個。<li class: "gridItem">
1 回答
UYOU
TA貢獻1878條經(jīng)驗 獲得超4個贊
網(wǎng)站加載了事件，該事件在頁面加載后動態(tài)呈現(xiàn)其數(shù)據(jù)。JavaScript
requests庫將無法動態(tài)渲染。所以你可以使用 或 .事實上，有很多模塊可以做到這一點。JavaScriptseleniumrequests_html
現(xiàn)在，我們在表上確實有另一個選項，可以跟蹤數(shù)據(jù)的呈現(xiàn)位置。我能夠找到XHR請求，該請求用于從中檢索數(shù)據(jù)并將其呈現(xiàn)給用戶端。back-endAPI
您可以通過打開開發(fā)人員工具來獲取請求，并檢查網(wǎng)絡(luò)并根據(jù)呼叫類型檢查發(fā)出的請求，例如
XHRXHR/JSfetch
您可以通過以下方式實現(xiàn)您的目標(biāo)：
請注意以下幾點：
website持有3068架item我增加了每頁要使用的項目
120parameter"pageSize": "120"所以 = 讓我們說 ，這意味著每頁 120 個項目，26 頁。
3068 / 12026因此，您將需要從哪個手段循環(huán)，依此類推，使用您將在循環(huán)下遞增的參數(shù)。
(0, 3120, 120)0 > 120 > 240"beginIndex": "0"for
在下面，您可以實現(xiàn)您的目標(biāo)，因為您沒有向我們提供您的最終目標(biāo)。但我相信你的目標(biāo)是或（網(wǎng)址，img）或其他什么。你會發(fā)現(xiàn)它。nameprice
import requests
from bs4 import BeautifulSoup
params = {
"langId": "44",
"storeId": "10151",
"catalogId": "10241",
"categoryId": "12192",
"parent_category_rn": "",
"top_category": "12192",
"pageSize": "120",
"orderBy": "FAVOURITES_FIRST",
"searchTerm": "",
"catSeeAll": "true",
"beginIndex": "0",
"categoryFacetId1": "12192",
"categoryFacetId2": "",
"requesttype": "ajax"
}
def main(url):
with requests.Session() as req:
r = req.post(url, params=params).json()
for item in r[5]['productLists']:
for nest in item['products']:
soup = BeautifulSoup(nest['result'], 'html.parser')
target = soup.find("div", class_="productNameAndPromotions")
name = target.h3.a.text.strip()
url = target.h3.a.get("href")
img = f"https"+target.h3.a.img.get("src")
price = soup.find(
"p", class_="pricePerUnit").get_text(strip=True)
print(name, price, img, url)
main("https://www.sainsburys.co.uk/webapp/wcs/stores/servlet/gb/groceries/drinks/AjaxApplyFilterSearchResultView")
名稱和價格的簡要輸出：
Sainsbury's British Semi Skimmed Milk 2.27L (4 pint) ￡1.10/unit
Sainsbury's British Semi Skimmed Milk 1.13L (2 pint) 80p/unit
Sainsbury's British Whole Milk 2.27L (4 pint) ￡1.10/unit
Cravendale Purefilter Semi Skimmed Milk 2L ￡1.90/unit
Sainsbury's British Skimmed Milk 2.27L (4 pint) ￡1.10/unit
Sainsbury's British Semi Skimmed Milk, SO Organic 2.27L (4 pint) ￡1.80/unit
Sainsbury's Sparkling Water, Basics 2L 25p/unit
Sainsbury's British Skimmed Milk 1.13L (2 pint) 80p/unit
Sainsbury's 100% Pure Squeezed Smooth Orange Juice, Not From Concentrate 1L ￡1.30/unit
Sainsbury's Water, Basics 2L 25p/unit
Sainsbury's British Whole Milk 1.13L (2 pint) 80p/unit
Sainsbury's Smooth Pure Orange Juice 1L 95p/unit
Pepsi Max 2L ￡1.90/unit
Sainsbury's Caledonian Still Water 4x2L ￡1.50/unit
Highland Spring Still Water 12x500ml ￡3.00/unit
Sainsbury's 100% Pressed Apple Juice, Not From Concentrate 1L ￡1.30/unit
Sainsbury's British Whole Milk, SO Organic 2.27L (4 Pint) ￡1.80/unit
Lactofree Semi Skimmed Lactose Free Fresh Dairy Drink 1L ￡1.50/unit
Diet Coke 8x330ml ￡4.00/unit
Alpro Roasted Almond Unsweetened UHT Drink 1L ￡1.80/unit
Robinsons Orange Squash No Added Sugar 1L ￡1.65/unit
Sainsbury's Soda Water 1L 60p/unit
Sainsbury's Caledonian Sparkling Water 4x2L ￡1.60/unit
Tropicana Smooth Orange Juice 950ml ￡2.45/unit
Sainsbury's Diet Indian Tonic Water 1L 60p/unit
Sainsbury's Pure Apple Juice 1L 95p/unit
Robinsons Apple & Blackcurrant Squash No Added Sugar 1L ￡1.65/unit
Sainsbury's Sparkling Flavoured Water, Lemon & Lime 1L 50p/unit
Sainsbury's Conegliano Prosecco, Taste the Difference 75cl ￡8.00/unit
Sainsbury's Unsweetened Soya Drink 1L 90p/unit
Sainsbury's British Semi Skimmed Milk, SO Organic 1.13L (2 pint) ￡1.15/unit
Sainsbury's Caledonian Sparkling Water 6x500ml ￡1.50/unit
Sainsbury's Apple & Blackcurrant Squash, No Added Sugar 1.5L ￡1.00/unit
Highland Spring Still Water 6x1.5L ￡3.00/unit
Alpro Roasted Almond Unsweetened Fresh Drink 1L ￡1.85/unit
Sainsbury's Semi Skimmed Long Life Milk 1L 90p/unit
Tropicana Smooth Orange Juice 1.6L ￡2.50/unit
Sainsbury's 100% Pure Squeezed Orange Juice with Bits, Not From Concentrate 1L ￡1.30/unit
Cravendale Purefilter Semi Skimmed Milk 1L ￡1.15/unit
Sainsbury's Caledonian Still Water Sports Cap 6x500ml ￡1.50/unit
Sainsbury's Double Strength Orange Squash, No Added Sugar 1.5L ￡1.00/unit
Diet Coke 18x330ml ￡7.00/unit
Sainsbury's Indian Tonic Water 1L 60p/unit
Sainsbury's Pure Orange Juice 1L 85p/unit
Sainsbury's Pure Apple Juice 6x200ml ￡1.50/unit
Buxton Still Natural Mineral Water 8x500ml ￡2.00/unit
Sainsbury's Whole Long Life Milk 1L ￡1.05/unit
Cravendale Purefilter Skimmed Milk 2L ￡1.90/unit
Sainsbury's Sparkling Flavoured Water, Blackcurrant & Cherry 1L 50p/unit
Innocent Smooth Orange Juice 1.35L ￡3.00/unit
Alpro Original Soya Fresh Drink 1L ￡1.55/unit
Sainsbury's Still Flavoured Water, Strawberry & Kiwi 1L 50p/unit
Sainsbury's British Filtered Semi Skimmed Milk 2L ￡1.35/unit
Sainsbury's Sparkling Flavoured Water, Mango & Passionfruit 1L 50p/unit
Sainsbury's Caledonian Still Water 5L ￡1.10/unit
McGuigan Estate Merlot 75cl ￡5.10/unit
Schweppes Slimline Tonic Water 1L ￡1.50/unit
PG tips Pyramid Tea Bags x240 696g ￡4.50/unit
Sainsbury's Sparkling Flavoured Water, Strawberry & Kiwi 1L 50p/unit
Sainsbury's Caledonian Sparkling Water 2L 55p/unit
Sainsbury's Sweetened Soya Drink 1L 90p/unit
Sainsbury's 100% Pure Squeezed Smooth Orange Juice, Not From Concentrate 1.75L ￡2.10/unit
Sainsbury's Diet Lemonade 2L 60p/unit
Sainsbury's Apple & Mango Juice, Not From Concentrate 1L ￡1.30/unit
Robinsons Summer Fruits Squash No Added Sugar 1L ￡1.65/unit
Sainsbury's 100% Pure Squeezed Pineapple Juice, Not From Concentrate 1L ￡1.30/unit
Clearsprings Sauvignon Blanc 75cl ￡5.50/unit
Phantom River Sauvignon Blanc 75cl ￡5.00/unit
Nestle Pure Life Still Spring Water 12x500ml ￡2.50/unit
Buxton Sparkling Natural Mineral Water 8x500ml ￡2.10/unit
Brancott Estate Sauvignon Blanc 75cl ￡6.75/unit
Schweppes Slimline Lemonade 2L ￡1.30/unit
McGuigan Estate South Australian Shiraz 75cl ￡5.10/unit
Coca-Cola Zero Sugar 8x330ml ￡4.00/unit
Villa Maria Private Bin Sauvignon Blanc 75cl ￡9.25/unit
Diet Coke Caffeine Free 8x330ml ￡4.00/unit
Sainsbury's British Skimmed Milk, SO Organic 1.13L (2 pint) ￡1.15/unit
Sainsbury's Kids Caledonian Still Water 6x300ml ￡1.10/unit
Canti Prosecco 75cl ￡7.50/unit
Oatly Enriched with Calcium Oat UHT Drink 1L ￡1.50/unit
Sainsbury's Pure Orange Juice 6x200ml ￡1.50/unit
Sainsbury's Still Flavoured Water, Lemon & Lime 1L 50p/unit
Valdo Prosecco Marca Oro 75cl ￡8.50/unit
Oyster Bay Sauvignon Blanc 75cl ￡8.00/unit
Ribena Blackcurrant Squash 850ml ￡2.30/unit
Volvic Mineral Water 6x1.5L ￡3.40/unit
Campo Viejo Rioja Tempranillo 75cl ￡6.75/unit
Nescafé Azera Americano Instant Coffee 100g ￡4.60/unit
Tropicana Orange Juice Original 950ml ￡2.45/unit
Sainsbury's Double Strength Orange & Mango Squash, No Added Sugar 1.5L ￡1.00/unit
Robinsons Lemon Squash No Added Sugar 1L ￡1.65/unit
Schweppes Lemonade 2L ￡1.30/unit
Robinsons Orange & Pineapple Squash No Added Sugar 1L ￡1.65/unit
Sainsbury's Diet Indian Tonic with Lime 1L 60p/unit
St Helen's Farm Semi Skimmed Goats Milk 1L ￡1.80/unit
Sainsbury's Double Strength Orange, Lemon & Pineapple Squash, No Added Sugar 1.5L ￡1.00/unit
Sainsbury's Double Strength Summerfruits Squash, No Added Sugar 1.5L ￡1.00/unit
Alpro Oat UHT Drink 1L ￡1.80/unit
Innocent Smooth Orange Juice 900ml ￡1.50/unit
Sainsbury's British Whole Milk, SO Organic 1.13L (2 pint) ￡1.15/unit
Sainsbury's Skimmed Long Life Milk 1L 80p/unit
Nescafé Gold Blend Instant Coffee 200g ￡7.00/unit
Highland Spring Still Water Sports Cap 12x330ml ￡3.00/unit
Sainsbury's Cava Brut 75cl ￡6.00/unit
Alpro Light Unsweetened Soya Fresh Drink 1L ￡1.55/unit
Sainsbury's Caledonian Still Water 2L 50p/unit
Koko Coconut UHT Drink 1L ￡1.50/unit
Sainsbury's House Pinot Grigio 75cl ￡4.50/unit
Sainsbury's Cola Zero 2L 45p/unit
St Helen's Farm Whole Goats Milk 1L ￡1.80/unit
Sainsbury's Double Strength Cherries & Berries Squash, No Added Sugar 1.5L ￡1.00/unit
Sainsbury's Lemonade 2L 60p/unit
Sainsbury's Pure Orange Juice With Bits 1L 85p/unit
Sainsbury's Pinot Grigio, Taste the Difference 75cl ￡6.00/unit
Schweppes Tonic Water 1L ￡1.50/unit
Sainsbury's Cranberry Juice Drink 1L 85p/unit
Nescafé Gold Blend Instant Coffee Refill 150g ￡3.50/unit
Sainsbury's Gold Roast Instant Coffee 200g ￡3.15/unit
Sainsbury's Pure Orange Juice with Juicy Bits 1L 95p/unit
Edizione 789 Di Mondelli Prosecco 75cl ￡6.25/unit
添加回答
舉報
0/150
提交
取消