第七色在线视频,2021少妇久久久久久久久久,亚洲欧洲精品成人久久av18,亚洲国产精品特色大片观看完整版,孙宇晨将参加特朗普的晚宴

為了賬號安全，請及時綁定郵箱和手機立即綁定
已解決430363個問題，去搜搜看，總會有你想問的

如何根據(jù) rdd 中包含 pyspark 元組數(shù)組的第一個元素進行過濾？

如何根據(jù) rdd 中包含 pyspark 元組數(shù)組的第一個元素進行過濾？

幕布斯6054654 2023-09-26 16:40:39
我在從 rdd 中過濾元組列表時遇到問題。示例business.json{"business_id":"gnKjwL_1w79qoiV3IC_xQQ","state":"NC","postal_code":"28210","latitude":35.092564,"longitude":-80.859132,"stars":4.0},{"business_id":"xvX2CttrVhyG2z1dFg_0xw","state":"AZ","postal_code":"85338","latitude":33.4556129678,"longitude":-112.3955963552,"stars":5.0}from pyspark import SparkContextsc = SparkContext.getOrCreate()stars = "4.0"input_business_lines = sc.textFile('data/business.json') \    .map(lambda lines: json.loads(lines))business_ids = input_business_lines \    .map(lambda kv: (kv['business_id'], kv['stars'], kv['state'])) \    .filter(lambda kv: kv[1] >= float(stars)).map(lambda kv: (kv[0], kv[2])).collect()上面的代碼返回每個元組具有的元組列表(first element = business_id , second element = state)[('gnKjwL_1w79qoiV3IC_xQQ', 'NC'),('xvX2CttrVhyG2z1dFg_0xw', 'AZ'),...,('HhyxOkGAM07SRYtlQ4wMFQ', 'NC')]到目前為止一切都很好?，F(xiàn)在我需要與評論表進行聯(lián)接，并希望使用評論的 rdd 過濾所有匹配的業(yè)務 id。如果那是一個數(shù)據(jù)框那就容易多了。但對于元組，我不確定我們該怎么做。示例 review.json{"review_id": "c-6aA9Bd7JxpmMroRoas9A", "user_id": "bK4Y_GZUoAUTXIrmeEUGYw", "business_id": "gnKjwL_1w79qoiV3IC_xQQ", "stars": 4.0, "text": "Went there Saturday noon they open at 12pm but people were waiting outside before 12pm so you can tell it should be a good place. Nice Katsu & Eel with rice. Many Japanese go there.", "date": "2014-07-13 20:28:18"},
查看完整描述

2 回答

?
慕標琳琳

TA貢獻1830條經(jīng)驗 獲得超9個贊

import json

from pyspark import SparkContext


if __name__ == '__main__':

    input_review_json_path = "publicdata/review.json"

    input_business_json_path = "publicdata/business.json"

    output_csv_path = "outputs/user_state.csv"

    stars = "4.0"


    sc = SparkContext.getOrCreate()


    input_business_lines = sc.textFile(input_business_json_path) \

                             .map(lambda lines: json.loads(lines))


    business_ids = input_business_lines \

                        .map(lambda kv: (kv['business_id'], kv['stars'], kv['state'])) \

                        .filter(lambda kv: kv[1] >= float(stars)).map(lambda kv: (kv[0], kv[2]))


    input_review_lines = sc.textFile(input_review_json_path) \

                            .map(lambda lines: json.loads(lines))


    rew_ids_bus_ids = input_review_lines.map(lambda kv: (kv['business_id'], kv['user_id']))

    finalRdd = business_ids.join(rew_ids_bus_ids).map(lambda kv: (kv[0], kv[1][0]))


    review_rdd = finalRdd.collect()


查看完整回答
反對 回復 2023-09-26
?
慕俠2389804

TA貢獻1719條經(jīng)驗 獲得超6個贊

你可以加入那些rdd。


import json


stars = 4.0

input_business_lines = sc.textFile('test.json') \

    .map(lambda lines: json.loads(lines))


business_ids = input_business_lines \

    .filter(lambda kv: kv['stars'] >= stars) \

    .map(lambda kv: (kv['business_id'], kv['state']))


print(business_ids.collect())


input_review_lines = sc.textFile('test2.json') \

    .map(lambda lines: json.loads(lines))


rew_ids_bus_ids = input_review_lines \

    .map(lambda kv: (kv['business_id'], kv['user_id']))


joined = business_ids \

    .join(rew_ids_bus_ids)


print(joined.collect())



# [('gnKjwL_1w79qoiV3IC_xQQ', 'NC'), ('xvX2CttrVhyG2z1dFg_0xw', 'AZ')]

# [('gnKjwL_1w79qoiV3IC_xQQ', ('NC', 'bK4Y_GZUoAUTXIrmeEUGYw'))]


查看完整回答
反對 回復 2023-09-26
  • 2 回答
  • 0 關注
  • 131 瀏覽
慕課專欄
更多

添加回答

舉報

0/150
提交
取消
微信客服

購課補貼
聯(lián)系客服咨詢優(yōu)惠詳情

 幫助反饋 APP下載

慕課網(wǎng)APP
您的移動學習伙伴

 公眾號

掃描二維碼
關注慕課網(wǎng)微信公眾號