我正在嘗試構(gòu)建一個名為“and_query”的函數(shù)，該函數(shù)將包含一個或多個單詞的單個字符串作為輸入，以便該函數(shù)返回文檔摘要中單詞的匹配文檔列表。首先，我將所有單詞放在倒排索引中，id 是文檔的 id，摘要是純文本。inverted_index = defaultdict(set)for (id, abstract) in Abstracts.items():for term in preprocess(tokenize(abstract)): inverted_index[term].add(id)然后，我編寫了一個查詢函數(shù)，其中 finals 是所有匹配文檔的列表。因?yàn)樗鼞?yīng)該只返回文檔中函數(shù)參數(shù)的每個單詞都匹配的文檔，所以我使用了設(shè)置操作“intersecton”。def and_query(tokens): documents=set() finals = [] terms = preprocess(tokenize(tokens)) for term in terms: for i in inverted_index[term]: documents.add(i) for term in terms: temporary_set= set() for i in inverted_index[term]: temporary_set.add(i) finals.extend(documents.intersection(temporary_set)) return finalsdef finals_print(finals): for final in finals: display_summary(final) finals_print(and_query("netherlands vaccine trial"))但是，該函數(shù)似乎仍在返回文檔摘要中只有 1 個單詞的文檔。有誰知道我在設(shè)置操作方面做錯了什么？（我認(rèn)為錯誤應(yīng)該出現(xiàn)在這部分代碼的任何地方）：for term in terms: temporary_set= set() for i in inverted_index[term]: temporary_set.add(i) finals.extend(documents.intersection(temporary_set))return finals 提前致謝基本上我想做的事情簡而言之：for word in words: id_set_for_one_word= set() for i in get_id_of that_word[word]: id_set_for_one_word.add(i)pseudo: id_set_for_one_word intersection (id_set_of_other_words)finals.extend( set of all intersections for all words)然后我需要所有這些詞的 id 集的交集，返回一個集合，其中 id 存在于詞中的每個詞。
3 回答
嗶嗶one
TA貢獻(xiàn)1854條經(jīng)驗(yàn) 獲得超8個贊
為了詳細(xì)說明我的代碼氣味注釋，這里是我之前為解決此類問題所做的工作的粗略草稿。
def tokenize(abstract):
#return <set of words in abstract>
set_ = .....
return set_
candidates = (id, abstract, tokenize(abstract)) for abstract in Abstracts.items():
all_criterias = "netherlands vaccine trial".split()
def searcher(candidates, criteria, match_on_found=True):
search_results = []
for cand in candidates:
#cand[2] has a set of tokens or somesuch... abstract.
if criteria in cand[2]:
if match_on_found:
search_results.append(cand)
else:
#that's a AND NOT if you wanted that
search_results.append(cand)
return search_results
for criteria in all_criterias:
#pass in the full list every time, but it gets progressively shrunk
candidates = searcher(candidates, criteria)
#whats left is what you want
answer = [(abs[0],abs[1]) for abs in candidates]
幕布斯6054654
TA貢獻(xiàn)1876條經(jīng)驗(yàn) 獲得超7個贊
最終自己找到了解決方案。替換
finals.extend(documents.intersection(id_set_for_one_word))
return finals
和
documents = (documents.intersection(id_set_for_one_word))
return documents
似乎在這里工作。
不過，謝謝大家的努力。
慕哥9229398
TA貢獻(xiàn)1877條經(jīng)驗(yàn) 獲得超6個贊
問題：返回文檔摘要中單詞的匹配文檔列表
該term用min的數(shù)量documents，保持始終result。
如果 aterm在 中不存在inverted_index，則根本不匹配。
為簡單起見，預(yù)定義數(shù)據(jù)：
Abstracts = {1: 'Lorem ipsum dolor sit amet,',
2: 'consetetur sadipscing elitr,',
3: 'sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat,',
4: 'sed diam voluptua.',
5: 'At vero eos et accusam et justo duo dolores et ea rebum.',
6: 'Stet clita kasd gubergren,',
7: 'no sea takimata sanctus est Lorem ipsum dolor sit amet.',
}
inverted_index = {'Stet': {6}, 'ipsum': {1, 7}, 'erat,': {3}, 'ut': {3}, 'dolores': {5}, 'gubergren,': {6}, 'kasd': {6}, 'ea': {5}, 'consetetur': {2}, 'sit': {1, 7}, 'nonumy': {3}, 'voluptua.': {4}, 'est': {7}, 'elitr,': {2}, 'At': {5}, 'rebum.': {5}, 'magna': {3}, 'sadipscing': {2}, 'diam': {3, 4}, 'dolore': {3}, 'sanctus': {7}, 'labore': {3}, 'sed': {3, 4}, 'takimata': {7}, 'Lorem': {1, 7}, 'invidunt': {3}, 'aliquyam': {3}, 'accusam': {5}, 'duo': {5}, 'amet.': {7}, 'et': {3, 5}, 'sea': {7}, 'dolor': {1, 7}, 'vero': {5}, 'no': {7}, 'eos': {5}, 'tempor': {3}, 'amet,': {1}, 'clita': {6}, 'justo': {5}, 'eirmod': {3}}
def and_query(tokens):
print("tokens:{}".format(tokens))
#terms = preprocess(tokenize(tokens))
terms = tokens.split()
term_min = None
for term in terms:
if term in inverted_index:
# Find min
if not term_min or term_min[0] > len(inverted_index[term]):
term_min = (len(inverted_index[term]), term)
else:
# Break early, if a term is not in inverted_index
return set()
finals = inverted_index[term_min[1]]
print("term_min:{} inverted_index:{}".format(term_min, finals))
return finals
def finals_print(finals):
if finals:
for final in finals:
print("Document [{}]:{}".format(final, Abstracts[final]))
else:
print("No matching Document found")
if __name__ == "__main__":
for tokens in ['sed diam voluptua.', 'Lorem ipsum dolor', 'Lorem ipsum dolor test']:
finals_print(and_query(tokens))
print()
輸出：
tokens:sed diam voluptua.
term_min:(1, 'voluptua.') inverted_index:{4}
Document [4]:sed diam voluptua.
tokens:Lorem ipsum dolor
term_min:(2, 'Lorem') inverted_index:{1, 7}
Document [1]:Lorem ipsum dolor sit amet,
Document [7]:no sea takimata sanctus est Lorem ipsum dolor sit amet.
tokens:Lorem ipsum dolor test
No matching Document found
用 Python 測試：3.4.2
添加回答
舉報
0/150
提交
取消