我有一個(gè)超過 1GB 的大型文本文件（chat.txt），其格式如下：john|12-02-1999|hello#,there#,how#,are#,you#,tom$ tom|12-02-1999|hey#,john$,hows#, it#, goin#mary|12-03-1999|hello#,boys#,fancy#,meetin#,ya'll#,here#......john|12-02-2000|well#,its#,been#,nice#,catching#,up#,with#,you#,and#, mary$mary|12-03-2000|catch#,you#,on#,the#,flipside#,tom$,and#,john$我想處理此文本并分別總結(jié)每個(gè)用戶的某些關(guān)鍵字的字?jǐn)?shù)（例如 500 個(gè)字 - 你好，很好，喜歡......晚餐，不）。此過程還涉及從每個(gè)單詞中刪除所有尾隨特殊字符輸出看起來像user hello nice like ..... dinner No Tom 10000 500 300 ..... 6000 0John 6000 1200 200 ..... 3000 5Mary 23 9000 10000 ..... 100 9000 這是我當(dāng)前的 pythonic 解決方案：chat_data = pd.read_csv("chat.txt", sep="|", names =["user","date","words"])user_lst = chat_data.user.unique()user_grouped_data= pd.DataFrame(columns=["user","words"])user_grouped_data['user']=user_lstfor i,row in user_grouped_data.iterrows(): id = row["user"] temp = chat_data[chat_data["user"]==id] user_grouped_data.loc[i,"words"] = ",".join(temp["words"].tolist())result = pd.DataFrame(columns=[ "user", "hello", "nice", "like","...500 other keywords...", "dinner", "no"])result["user"]= user_lstfor i, row in result.iterrows(): id = row["user"] temp = user_grouped_data[user_grouped_data["user"]==id] words = temp.values.tolist()[0][1] word_lst = words.split(",") word_lst = [item[0:-1] for item in word_lst] t_dict = Counter(word_lst) keys = t_dict.keys() for word in keys: result.at[i,word]= t_dict.get(word)result.to_csv("user_word_counts.csv")這對于小數(shù)據(jù)來說效果很好，但是當(dāng)我的 chat_data 超過 1GB 時(shí)，這個(gè)解決方案變得非常慢并且無法使用。下面是否有任何我可以改進(jìn)的部分可以幫助我更快地處理數(shù)據(jù)？按用戶對文本數(shù)據(jù)進(jìn)行分組通過刪除尾隨特殊字符來清理每行中的文本數(shù)據(jù)計(jì)算單詞數(shù)并將單詞數(shù)分配給右列
3 回答
千萬里不及你
TA貢獻(xiàn)1784條經(jīng)驗(yàn) 獲得超9個(gè)贊
您可以將split逗號分隔的列轉(zhuǎn)換為列表，explode將列表列轉(zhuǎn)換為數(shù)據(jù)幀，groupby分解列表中的名稱和值，unstack或者將數(shù)據(jù)幀轉(zhuǎn)換為所需的格式，并使用, ,pivot_table對多索引列進(jìn)行最終清理ETC。droplevel()reset_index()
以下所有內(nèi)容都是矢量化的 pandas 方法，所以希望它很快。注意：當(dāng)我從剪貼板讀取并通過時(shí)，下面的代碼中的三列是 [0,1,2]headers=None
輸入：
df = pd.DataFrame({0: {0: 'john', 1: 'tom', 2: 'mary', 3: 'john', 4: 'mary'},
1: {0: '12-02-1999',
1: '12-02-1999',
2: '12-03-1999',
3: '12-02-2000',
4: '12-03-2000'},
2: {0: 'hello#,there#,how#,are#,you#,tom$ ',
1: 'hey#,john$,hows#, it#, goin#',
2: "hello#,boys#,fancy#,meetin#,ya'll#,here#",
3: 'well#,its#,been#,nice#,catching#,up#,with#,you#,and#, mary$',
4: 'catch#,you#,on#,the#,flipside#,tom$,and#,john$'}})
代碼：
df[2] = df[2].replace(['\#', '\$'],'', regex=True).str.split(',')
df = (df.explode(2)
.groupby([0, 2])[2].count()
.rename('Count')
.reset_index()
.set_index([0,2])
.unstack(1)
.fillna(0))
df.columns = df.columns.droplevel()
df = df.reset_index()
df
Out[1]:
2 0 goin it mary and are been boys catch catching ... on \
0 john 0.0 0.0 1.0 1.0 1.0 1.0 0.0 0.0 1.0 ... 0.0
1 mary 0.0 0.0 0.0 1.0 0.0 0.0 1.0 1.0 0.0 ... 1.0
2 tom 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0
2 the there tom tom up well with ya'll you
0 0.0 1.0 0.0 1.0 1.0 1.0 1.0 0.0 2.0
1 1.0 0.0 1.0 0.0 0.0 0.0 0.0 1.0 1.0
您還可以使用.pivot_table代替.unstack()，這樣可以節(jié)省這行代碼：df.columns = df.columns.droplevel()：
df[2] = df[2].replace(['\#', '\$'],'', regex=True).str.split(',')
df = (df.explode(2)
.groupby([0, 2])[2].count()
.rename('Count')
.reset_index()
.pivot_table(index=0, columns=2, values='Count')
.fillna(0)
.astype(int)
.reset_index())
df
Out[45]:
2 0 goin it mary and are been boys catch catching ... on \
0 john 0 0 1 1 1 1 0 0 1 ... 0
1 mary 0 0 0 1 0 0 1 1 0 ... 1
2 tom 1 1 0 0 0 0 0 0 0 ... 0
2 the there tom tom up well with ya'll you
0 0 1 0 1 1 1 1 0 2
1 1 0 1 0 0 0 0 1 1
2 0 0 0 0 0 0 0 0 0
[3 rows x 31 columns]
九州編程
TA貢獻(xiàn)1785條經(jīng)驗(yàn) 獲得超4個(gè)贊
如果你會(huì)使用scikit-learn，那就很容易了CountVectorizer
from sklearn.feature_extraction.text import CountVectorizer
s = df['words'].str.replace("#|\$|\s+", "")
model = CountVectorizer(tokenizer=lambda x: x.split(','))
df_final = pd.DataFrame(model.fit_transform(s).toarray(),
columns=model.get_feature_names(),
index=df.user).sum(level=0)
Out[279]:
and are been boys catch catching fancy flipside goin hello \
user
john 1 1 1 0 0 1 0 0 0 1
tom 0 0 0 0 0 0 0 0 1 0
mary 1 0 0 1 1 0 1 1 0 1
here hey how hows it its john mary meetin nice on the there \
user
john 0 0 1 0 0 1 0 1 0 1 0 0 1
tom 0 1 0 1 1 0 1 0 0 0 0 0 0
mary 1 0 0 0 0 0 1 0 1 0 1 1 0
tom up well with ya'll you
user
john 1 1 1 1 0 2
tom 0 0 0 0 0 0
mary 1 0 0 0 1 1
江戶川亂折騰
TA貢獻(xiàn)1851條經(jīng)驗(yàn) 獲得超5個(gè)贊
我不確定這種方法在大型 DataFrame 上的速度有多快，但您可以嘗試一下。首先，刪除特殊字符并將字符串拆分為單詞列表，從而形成另一列：
from itertools import chain
from collections import Counter
df['lists'] = df['words'].str.replace("#|\$", "").str.split(",")現(xiàn)在，按用戶分組），將列表收集到一個(gè)列表中，并使用以下命令計(jì)算出現(xiàn)次數(shù)Counter：
df.groupby('user')['lists'].apply(chain.from_iterable)\
.apply(Counter)\
.apply(pd.Series)\
.fillna(0).astype(int)添加回答
舉報(bào)
0/150
提交
取消