我正在嘗試計算一個tfidf沒有停用詞的矩陣。這是我的代碼：def removeStopWords(documents):? ? stop_words = set(stopwords.words('italian'))? ? english_stop_words = set(stopwords.words('english'))? ? stop_words.update(list(set(english_stop_words)))? ? for d in documents:? ? ? ? document = d['document']? ? ? ? word_tokens = word_tokenize(document)? ? ? ? ?filtered_sentence = ''? ? ? ? for w in word_tokens:? ? ? ? ? ? if not inStopwords(w, stop_words):? ? ? ? ? ? ? ? ?filtered_sentence = w + ' ' + filtered_sentence? ? ? ? d['document'] = filtered_sentence[:-1]? ? return calculateTFIDF(documents)def calculateTFIDF(corpus):? ? tfidf = TfidfVectorizer()? ? x = tfidf.fit_transform(corpus)? ? df_tfidf = pd.DataFrame(x.toarray(), columns=tfidf.get_feature_names())? ? return {c: s[s > 0] for c, s in zip(df_tfidf, df_tfidf.T.values)}但是當我返回矩陣（使用形式{word:value}）時，它還包含一些停用詞，例如whenor il。我該如何解決？謝謝
1 回答
一只萌萌小番薯
TA貢獻1795條經驗 獲得超7個贊
有更好的方法來刪除 TF-IDF 計算中的停用詞。有TfidfVectorizer一個參數(shù)stop_words，您可以在其中傳遞要排除的單詞集合。
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
documents = ['I went to the barbershop when my hair was long.', 'The barbershop was closed.']
# create set of stopwords to remove
stop_words = set(stopwords.words('italian'))
english_stop_words = set(stopwords.words('english'))
stop_words.update(english_stop_words)
# check if word in stop words
print('when' in stop_words) # True
print('il' in stop_words) # True
# else add word to the set
print('went' in stop_words) # False
stop_words.add('went')
# create tf-idf from original documents
tfidf = TfidfVectorizer(stop_words=stop_words)
x = tfidf.fit_transform(documents)
df_tfidf = pd.DataFrame(x.toarray(), columns=tfidf.get_feature_names())
print({c: s[s > 0] for c, s in zip(df_tfidf, df_tfidf.T.values)})
# {'barbershop': array([0.44943642, 0.57973867]), 'closed': array([0.81480247]), 'hair': array([0.6316672]), 'long': array([0.6316672])}
添加回答
舉報
0/150
提交
取消