我正在亞馬遜暢銷書頁面上測試 scrapy 蜘蛛(請參見下面的 URL),但它返回奇怪的價格數(shù)字或根本沒有輸出,正如您在最后的輸出中看到的那樣(我只共享了 1 頁的輸出) 。css 選擇器可能有問題,但我不確定。我希望蜘蛛將輸出保存在 JSON 文件中,這樣我就可以快速將其轉(zhuǎn)換為 pandas 數(shù)據(jù)幀進行一些分析。這是我在終端中編寫的用于運行蜘蛛的代碼: scrapycrawl amazon_booksUK -o somefilename.json我知道要查看的內(nèi)容有很多,但如果您有時間,這確實會對我有幫助!:)url = https://www.amazon.co.uk/s?i=stripbooks&bbn=266239&rh=n%3A266239%2Cp_72%3A184315031%2Cp_36%3A389028011&dc&page=1&fst=as%3Aoff&qid=1598942460&rnid=3890 22011 &ref=sr_pg_11.蜘蛛代碼:import scrapyfrom ..items import AmazonscrapeItemclass AmazonSpiderSpider(scrapy.Spider): page_number = 2 name = 'amazon_booksUK' start_urls = [ 'https://www.amazon.co.uk/s?i=stripbooks&bbn=266239&rh=n%3A266239%2Cp_72%3A184315031%2Cp_36%3A389028011&dc&page=1&fst=as%3Aoff&qid=1598942460&rnid=389022011&ref=sr_pg_1' ] def parse(self, response): items = AmazonscrapeItem() # if multiple classes --> .css("::text").extract() product_name = response.css('.a-color-base.a-text-normal::text').extract() product_author = response.css('.a-color-secondary .a-size-base.a-link-normal').css('::text').extract() product_nbr_reviews = response.css('.a-size-small .a-link-normal .a-size-base').css('::text').extract() product_type = response.css('.a-spacing-top-small .a-link-normal.a-text-bold').css('::text').extract() product_price = response.css('.a-spacing-top-small .a-price-whole').css('::text').extract() product_more_choice = response.css('.a-spacing-top-mini .a-color-secondary .a-link-normal').css('::text').extract() # this only selects the element that has the image --> need stuff inside src (source attr)
1 回答

胡子哥哥
TA貢獻1825條經(jīng)驗 獲得超6個贊
你只需要像這樣使用用戶代理:
USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'
就像真正的瀏覽器使用的東西
添加回答
舉報
0/150
提交
取消