更新 2020 年 10 月 21 日
我決定構(gòu)建一個 Python 模塊來處理我在這個答案中概述的任務(wù)。該模塊名為wordhoard ，可以從pypi下載
我曾嘗試在需要確定關(guān)鍵字（例如醫(yī)療保健）和關(guān)鍵字同義詞（例如健康計劃、預防醫(yī)學）的頻率的項目中使用Word2vec 和WordNet。我發(fā)現(xiàn)大多數(shù) NLP 庫無法生成我需要的結(jié)果，因此我決定使用自定義關(guān)鍵字和同義詞構(gòu)建自己的詞典。這種方法適用于多個項目中的文本分析和分類。
我確信精通 NLP 技術(shù)的人可能有更強大的解決方案，但下面的解決方案是一次又一次為我工作的類似解決方案。
我對答案進行了編碼以匹配您問題中的詞頻數(shù)據(jù)，但可以對其進行修改以使用任何關(guān)鍵字和同義詞數(shù)據(jù)集。
import string
# Python Dictionary
# I manually created these word relationship - primary_word:synonyms
word_relationship = {"father": ['dad', 'daddy', 'old man', 'pa', 'pappy', 'papa', 'pop'],
"mother": ["mamma", "momma", "mama", "mammy", "mummy", "mommy", "mom", "mum"]}
# This input text is from various poems about mothers and fathers
input_text = 'The hand that rocks the cradle also makes the house a home. It is the prayers of the mother ' \
'that keeps the family strong. When I think about my mum, I just cannot help but smile; The beauty of ' \
'her loving heart, the easy grace in her style. I will always need my mom, regardless of my age. She ' \
'has made me laugh, made me cry. Her love will never fade. If I could write a story, It would be the ' \
'greatest ever told. I would write about my daddy, For he had a heart of gold. For my father, my friend, ' \
'This to me you have always been. Through the good times and the bad, Your understanding I have had.'
# converts the input text to lowercase and splits the words based on empty space.
wordlist = input_text.lower().split()
# remove all punctuation from the wordlist
remove_punctuation = [''.join(ch for ch in s if ch not in string.punctuation)
for s in wordlist]
# list for word frequencies
wordfreq = []
# count the frequencies of a word
for w in remove_punctuation:
wordfreq.append(remove_punctuation.count(w))
word_frequencies = (dict(zip(remove_punctuation, wordfreq)))
word_matches = []
# loop through the dictionaries
for word, frequency in word_frequencies.items():
for keyword, synonym in word_relationship.items():
match = [x for x in synonym if word == x]
if word == keyword or match:
match = ' '.join(map(str, match))
# append the keywords (mother), synonyms(mom) and frequencies to a list
word_matches.append([keyword, match, frequency])
# used to hold the final keyword and frequencies
final_results = {}
# list comprehension to obtain the primary keyword and its frequencies
synonym_matches = [(keyword[0], keyword[2]) for keyword in word_matches]
# iterate synonym_matches and output total frequency count for a specific keyword
for item in synonym_matches:
if item[0] not in final_results.keys():
frequency_count = 0
frequency_count = frequency_count + item[1]
final_results[item[0]] = frequency_count
else:
frequency_count = frequency_count + item[1]
final_results[item[0]] = frequency_count
print(final_results)
# output
{'mother': 3, 'father': 2}
其他方法
以下是一些其他方法及其開箱即用的輸出。
NLTK字網(wǎng)
在此示例中，我查找了“母親”一詞的同義詞。請注意，WordNet 沒有與“mother”一詞相關(guān)的同義詞“mom”或“mum”。這兩個詞在我上面的示例文本中。另請注意，“父親”一詞被列為“母親”的同義詞。
from nltk.corpus import wordnet
synonyms = []
word = 'mother'
for synonym in wordnet.synsets(word):
for item in synonym.lemmas():
if word != synonym.name() and len(synonym.lemma_names()) > 1:
synonyms.append(item.name())
print(synonyms)
['mother', 'female_parent', 'mother', 'fuss', 'overprotect', 'beget', 'get', 'engender', 'father', 'mother', 'sire', 'generate', 'bring_forth']
Py字典
在此示例中，我使用 PyDictionary 查找“mother”一詞的同義詞，該詞典查詢synonym.com。此示例中的同義詞包括單詞“mom”和“mum”。此示例還包括 WordNet 未生成的其他同義詞。
但是，PyDictionary 還生成了“媽媽”的同義詞列表。這與“母親”這個詞無關(guān)。PyDictionary 似乎是從頁面的形容詞部分而不是名詞部分提取此列表。計算機很難區(qū)分形容詞媽媽和名詞媽媽。
from PyDictionary import PyDictionary
dictionary_mother = PyDictionary('mother')
print(dictionary_mother.getSynonyms())
# output
[{'mother': ['mother-in-law', 'female parent', 'supermom', 'mum', 'parent', 'mom', 'momma', 'para I', 'mama', 'mummy', 'quadripara', 'mommy', 'quintipara', 'ma', 'puerpera', 'surrogate mother', 'mater', 'primipara', 'mammy', 'mamma']}]
dictionary_mum = PyDictionary('mum')
print(dictionary_mum.getSynonyms())
# output
[{'mum': ['incommunicative', 'silent', 'uncommunicative']}]
其他一些可能的方法是使用牛津詞典 API 或查詢 thesaurus.com。這兩種方法也都有缺陷。例如，牛津詞典 API 需要 API 密鑰和基于查詢編號的付費訂閱。thesaurus.com 缺少可能對單詞分組有用的潛在同義詞。
https://www.thesaurus.com/browse/mother
synonyms: mom, parent, ancestor, creator, mommy, origin, predecessor, progenitor, source, child-bearer, forebearer, procreator
更新
為語料庫中的每個潛在單詞生成精確的同義詞列表很困難，并且需要多管齊下的方法。下面的代碼使用 WordNet 和 PyDictionary 創(chuàng)建同義詞的超集。與所有其他答案一樣，這種組合方法也會導致詞頻的過度計算。我一直在嘗試通過在最終的同義詞詞典中組合鍵和值對來減少這種過度計數(shù)。后一個問題比我預期的要困難得多，可能需要我提出自己的問題來解決。最后，我認為根據(jù)您的用例，您需要確定哪種方法最有效，并且可能需要結(jié)合多種方法。
感謝您提出這個問題，因為它使我能夠了解解決復雜問題的其他方法。
from string import punctuation
from nltk.corpus import stopwords
from nltk.corpus import wordnet
from PyDictionary import PyDictionary
input_text = """The hand that rocks the cradle also makes the house a home. It is the prayers of the mother
that keeps the family strong. When I think about my mum, I just cannot help but smile; The beauty of
her loving heart, the easy grace in her style. I will always need my mom, regardless of my age. She
has made me laugh, made me cry. Her love will never fade. If I could write a story, It would be the
greatest ever told. I would write about my daddy, For he had a heart of gold. For my father, my friend,
This to me you have always been. Through the good times and the bad, Your understanding I have had."""
def normalize_textual_information(text):
# split text into tokens by white space
token = text.split()
# remove punctuation from each token
table = str.maketrans('', '', punctuation)
token = [word.translate(table) for word in token]
# remove any tokens that are not alphabetic
token = [word.lower() for word in token if word.isalpha()]
# filter out English stop words
stop_words = set(stopwords.words('english'))
# you could add additional stops like this
stop_words.add('cannot')
stop_words.add('could')
stop_words.add('would')
token = [word for word in token if word not in stop_words]
# filter out any short tokens
token = [word for word in token if len(word) > 1]
return token
def generate_word_frequencies(words):
# list to hold word frequencies
word_frequencies = []
# loop through the tokens and generate a word count for each token
for word in words:
word_frequencies.append(words.count(word))
# aggregates the words and word_frequencies into tuples and coverts them into a dictionary
word_frequencies = (dict(zip(words, word_frequencies)))
# sort the frequency of the words from low to high
sorted_frequencies = {key: value for key, value in
sorted(word_frequencies.items(), key=lambda item: item[1])}
return sorted_frequencies
def get_synonyms_internet(word):
dictionary = PyDictionary(word)
synonym = dictionary.getSynonyms()
return synonym
words = normalize_textual_information(input_text)
all_synsets_1 = {}
for word in words:
for synonym in wordnet.synsets(word):
if word != synonym.name() and len(synonym.lemma_names()) > 1:
for item in synonym.lemmas():
if word != item.name():
all_synsets_1.setdefault(word, []).append(str(item.name()).lower())
all_synsets_2 = {}
for word in words:
word_synonyms = get_synonyms_internet(word)
for synonym in word_synonyms:
if word != synonym and synonym is not None:
all_synsets_2.update(synonym)
word_relationship = {**all_synsets_1, **all_synsets_2}
frequencies = generate_word_frequencies(words)
word_matches = []
word_set = {}
duplication_check = set()
for word, frequency in frequencies.items():
for keyword, synonym in word_relationship.items():
match = [x for x in synonym if word == x]
if word == keyword or match:
match = ' '.join(map(str, match))
if match not in word_set or match not in duplication_check or word not in duplication_check:
duplication_check.add(word)
duplication_check.add(match)
word_matches.append([keyword, match, frequency])
# used to hold the final keyword and frequencies
final_results = {}
# list comprehension to obtain the primary keyword and its frequencies
synonym_matches = [(keyword[0], keyword[2]) for keyword in word_matches]
# iterate synonym_matches and output total frequency count for a specific keyword
for item in synonym_matches:
if item[0] not in final_results.keys():
frequency_count = 0
frequency_count = frequency_count + item[1]
final_results[item[0]] = frequency_count
else:
frequency_count = frequency_count + item[1]
final_results[item[0]] = frequency_count
# do something with the final results
