1 回答

TA貢獻1786條經(jīng)驗 獲得超11個贊
onehot = []
for groupi, group in df.groupby(df.index//1e5):
# encode each group separately
onehot.expand(group_onehot)
df = df.assign(onehot=onehot)
會給你 28 個小組單獨工作。
但是,查看您的代碼,該行:
codes_values = [int(''.join(r)) for r in columns.itertuples(index=False)]
integer正在創(chuàng)建一個可能長達 4k 位的字符串并嘗試在 10e4000 范圍內(nèi)創(chuàng)建一個字符串,這將導致溢出(請參閱https://numpy.org/devdocs/user/basics.types.html)
編輯
另一種編碼方法。從這個 df 開始:
df = pd.DataFrame({
'ClaimId': [1902659, 1902659, 1902663, 1902674, 1902674, 2563847, 2563883,
2564007, 2564007, 2564363],
'ServiceSubCodeKey': [183, 2088, 3274, 12, 23, 3109, 3109, 3626, 3628, 3109]
})
代碼:
scale = df.ServiceSubCodeKey.max() + 1
onehot = []
for claimid, ssc in df.groupby('ClaimId').ServiceSubCodeKey:
ssc_list = ssc.to_list()
onehot.append([claimid,
''.join(['1' if i in ssc_list else '0' for i in range(1, scale)])])
onehot = pd.DataFrame(onehot, columns=['ClaimId', 'onehot'])
print(onehot)
輸出
ClaimId onehot
0 1902659 0000000000000000000000000000000000000000000000...
1 1902663 0000000000000000000000000000000000000000000000...
2 1902674 0000000000010000000000100000000000000000000000...
3 2563847 0000000000000000000000000000000000000000000000...
4 2563883 0000000000000000000000000000000000000000000000...
5 2564007 0000000000000000000000000000000000000000000000...
6 2564363 0000000000000000000000000000000000000000000000...
這修復了您的方法中的溢出問題并避免調(diào)用pd.get_dummies()創(chuàng)建 600K x 4K 虛擬數(shù)據(jù)幀,具有迭代分組系列和在每個組上構(gòu)建列表理解的障礙(既不利用 pandas 的內(nèi)置 C 實現(xiàn)) .
從這里您可以:
推薦:繼續(xù)保持每個 one-hot 編碼的摘要ClaimId,或者
您要求的是:根據(jù)df需要合并,復制相同的編碼與ClaimId復制的次數(shù)一樣多df
和
df = df.merge(onehot, on='ClaimId')
輸出
ClaimId ServiceSubCodeKey onehot
0 1902659 183 0000000000000000000000000000000000000000000000...
1 1902659 2088 0000000000000000000000000000000000000000000000...
2 1902663 3274 0000000000000000000000000000000000000000000000...
3 1902674 12 0000000000010000000000100000000000000000000000...
4 1902674 23 0000000000010000000000100000000000000000000000...
5 2563847 3109 0000000000000000000000000000000000000000000000...
6 2563883 3109 0000000000000000000000000000000000000000000000...
7 2564007 3626 0000000000000000000000000000000000000000000000...
8 2564007 3628 0000000000000000000000000000000000000000000000...
9 2564363 3109 0000000000000000000000000000000000000000000000...
添加回答
舉報