我有一句話的名單，我想創(chuàng)建skipgram(window size = 3)但我DONT希望跨句子柜臺(tái)跨度，因?yàn)樗麄兌紵o(wú)關(guān)。所以，如果我有以下句子：[["my name is John"] , ["This PC is black"]]三胞胎將是：[my name is][name is john][this PC is][PC is black]最好的方法是什么？
3 回答
一只斗牛犬
TA貢獻(xiàn)1784條經(jīng)驗(yàn) 獲得超2個(gè)贊
這是一個(gè)簡(jiǎn)單的功能來做到這一點(diǎn)。
def skipgram(corpus, window_size = 3):
sg = []
for sent in corpus:
sent = sent[0].split()
if len(sent) <= window_size:
sg.append(sent)
else:
for i in range(0, len(sent)-window_size+1):
sg.append(sent[i: i+window_size])
return sg
corpus = [["my name is John"] , ["This PC is black"]]
skipgram(corups)
人到中年有點(diǎn)甜
TA貢獻(xiàn)1895條經(jīng)驗(yàn) 獲得超7個(gè)贊
你并不是真的想要一個(gè)skipgram本身，但你想要一個(gè)按大小劃分的塊，試試這個(gè)：
from lazyme import per_chunk
tokens = "my name is John".split()
list(per_chunk(tokens, 2))
[出去]：
[('my', 'name'), ('is', 'John')]
如果你想要一個(gè)滾動(dòng)窗口，即ngrams：
from lazyme import per_window
tokens = "my name is John".split()
list(per_window(tokens, 2))
[出去]：
[('my', 'name'), ('name', 'is'), ('is', 'John')]
同樣在 ngrams 的 NLTK 中：
from nltk import ngrams
tokens = "my name is John".split()
list(ngrams(tokens, 2))
[出去]：
[('my', 'name'), ('name', 'is'), ('is', 'John')]
如果你想要實(shí)際的skipgrams，如何在python中計(jì)算skipgrams？
from nltk import skipgrams
tokens = "my name is John".split()
list(skipgrams(tokens, n=2, k=1))
[出去]：
[('my', 'name'),
('my', 'is'),
('name', 'is'),
('name', 'John'),
('is', 'John')]
慕村225694
TA貢獻(xiàn)1880條經(jīng)驗(yàn) 獲得超4個(gè)贊
嘗試這個(gè)！
from nltk import ngrams
def generate_ngrams(sentences,window_size =3):
for sentence in sentences:
yield from ngrams(sentence[0].split(), window_size)
sentences= [["my name is John"] , ["This PC is black"]]
for c in generate_ngrams(sentences,3):
print (c)
#output:
('my', 'name', 'is')
('name', 'is', 'John')
('This', 'PC', 'is')
('PC', 'is', 'black')
添加回答
舉報(bào)
0/150
提交
取消