我是新手，我正嘗試創(chuàng)建一個(gè)字典，如下所示：{4: {'aenr': ['earn', 'rane'], 'aerr': ['rare', 'rear'], 'aenw': ['anew', 'wane', 'wean'], 'derw': ['drew']}本質(zhì)上這應(yīng)該是使用spark的結(jié)構(gòu){len(word): {sorted(word):[word1,word2,etc]}我有一個(gè)很大的文件，其中包含英語單詞，其結(jié)構(gòu)如下所示：{ "biennials": 0, "tripolitan": 0, "oblocutor": 0, "leucosyenite": 0, "chilitis": 0, "fabianist": 0, "diazeutic": 0, "alible": 0, "deciet":0}所以我想逐行讀取文件并創(chuàng)建一個(gè)rdd可以保存此內(nèi)容：{len(word): {sorted(word):[word1,word2,etc]}我已經(jīng)試過了： r = rdd.map(lambda x: {len(x):sorted(x)}) items = r.flatMap(lambda line: (line.items())) items.take(items.count()) groupedItems = items.groupByKey().mapValues(list) groupedItems.take(groupedItems.count())#j = filter2_rdd d = groupedItems.collectAsMap()但這會打印以下內(nèi)容：[{1: {u'{': [u'{']}},{9: {u'abeiilnns': [u' "biennials": 0, ']}}, {10: {u'aiilnoprtt': [u' "tripolitan": 0, ']}}, {9: {u'bclooortu': [u' "oblocutor": 0, ']}}, {12: {u'ceeeilnostuy': [u' "leucosyenite": 0, ']}}, {8: {u'chiiilst': [u' "chilitis": 0, ']}}, {9: {u'aabfiinst': [u' "fabianist": 0, ']}}, {9: {u'acdeiituz': [u' "diazeutic": 0, ']}}, {6: {u'abeill': [u' "alible": 0, ']}}, {6: {u'cdeeit': [u' "deciet":0,']}}, {5: {u'doosw': [u' "woods": 4601, ']}}, {14: {u'adeejmnnoprrtu': [u' "preadjournment": 0, ']}}, {7: {u'deiprss': [u' "spiders": 0, ']}}, {9: {u'aabfiimns': [u' "fabianism": 0, ']}}, {11: {u'cdgilnoostu': [u' "outscolding": 0, ']}}, {10: {u'eeilprrsty': [u' "sperrylite": 0, ']}}, {8: {u'agilnrtw': [u' "trawling": 0, ']}}, {13: {u'acdeimmoprrsu': [u' "cardiospermum": 0, ']}}, {10: {u'gghhiilttt': [u' "lighttight": 0, ']}}, {7: {u'deiprsy': [u' "spidery": 0, ']}}}我需要將它們按長度和列表中的所有單詞分組
1 回答
MYYA
TA貢獻(xiàn)1868條經(jīng)驗(yàn) 獲得超4個(gè)贊
您不能立即這樣map()做len()，sorted()因?yàn)槟チ俗畛醯膬r(jià)值。這是一種實(shí)現(xiàn)方法：
map創(chuàng)建密鑰sorted(x)groupByKey--sorted(x)map創(chuàng)建密鑰len(x)groupByKey--len(x)collectAsMap()
ResultIterable如果要打印出來，可能需要將s轉(zhuǎn)換為特定的python類型：
例如（假設(shè)您已將所有單詞并行化為rdd）：
In []:
(rdd
.map(lambda x: (''.join(sorted(x)), x))
.groupByKey()
.mapValues(lambda x: list(x))
.map(lambda x: (len(x[0]), x))
.groupByKey()
.mapValues(lambda x: dict(x))
.collectAsMap())
Out[]:
{6: {'abeill': ['alible'], 'cdeeit': ['deciet']},
8: {'chiiilst': ['chilitis']},
9: {'aabfiinst': ['fabianist'],
'abeiilnns': ['biennials'],
'acdeiituz': ['diazeutic'],
'bclooortu': ['oblocutor']},
10: {'aiilnoprtt': ['tripolitan']},
12: {'ceeeilnostuy': ['leucosyenite']}}