我使用下面的代碼來爬取頁面上的多個鏈接，并從每個相應的鏈接中獲取數據列表：carspider.py:def parse_item(self, response): sel = Selector(response) item = CarscrapeItem() item['carType'] = sel.xpath('//div[@class="listing__section listing__section--key-details listing__key-details portable-one-whole push--bottom"]//span[@itemprop="manufacturer"]//text()').get() item['model'] = sel.xpath('//div[@class="listing__section listing__section--key-details listing__key-details portable-one-whole push--bottom"]//span[@itemprop="model"]//text()').get() item['variant'] = sel.xpath('//div[@class="listing__section listing__section--key-details listing__key-details portable-one-whole push--bottom"]//span[@class="float--right"]//text()')[3].get() item['year'] = sel.xpath('//div[@class="listing__section listing__section--key-details listing__key-details portable-one-whole push--bottom"]//span[@class="float--right"]//text()')[4].get() item['engineCapacity'] = sel.xpath('//div[@class="listing__section listing__section--key-details listing__key-details portable-one-whole push--bottom"]//span[@class="float--right"]//text()')[5].get() item['transmission'] = sel.xpath('//div[@class="listing__section listing__section--key-details listing__key-details portable-one-whole push--bottom"]//span[@class="float--right"]//text()')[6].get() item['seatCapacity'] = sel.xpath('//div[@class="listing__section listing__section--key-details listing__key-details portable-one-whole push--bottom"]//span[@class="float--right"]//text()')[7].get()我想刪除重復的汽車類型并將其余行值附加到現有汽車類型。我想這樣做一個推薦系統(tǒng)會更好。有可能用 Scrapy 做到這一點嗎？我搜索了與重復值相關的回復。大多數情況下，它們與重復過濾器有關，而其他過濾器對我不起作用。
1 回答
30秒到達戰(zhàn)場
TA貢獻1828條經驗 獲得超6個贊
僅供參考，字典鍵在 python 中必須是唯一的。所以你期望的輸出是不可能的。
建議：您可以通過以下方式存儲數據：
car = {
"Honda": [
{
"model": "Civic",
"variant": "TC VTEC Premium",
"year": "2020",
"engineCapacity": "1498cc",
"transmission": "Automatic",
"seatCapacity": "5"
},
{
"model": "Accord",
"variant": "TC",
"year": "2020",
"engineCapacity": "1498 cc",
"transmission": "Automatic",
"seatCapacity": "5"
}
],
"BMW": [
{
"model": "XYZ",
"year": "2020",
"transmission": "Automatic",
"seatCapacity": "5"
},
{
"model": "ABC",
"year": "2020",
"engineCapacity": "1498 cc",
"transmission": "Automatic",
"seatCapacity": "5"
}
]
}
您可以使用下面的部分代碼從文件中逐行讀取數據，您可以編寫自己的代碼以上述格式存儲數據
import json
with open('PATH_TO_FILE/FILE_NAME.json') as f:
data = f
for line in f:
line = json.loads(line)
# YOUR CODE HERE
添加回答
舉報
0/150
提交
取消