主要總結一下簡單的工具：條件頻率分布、正規表示式、詞幹提取器和歸併器。
《自然語言學習》很多地方都用到了條件分布頻率，nltk提供了兩種常用的介面：簡單的freqdist和conditionalfreqdist。後面很多都會用到這兩種方法，特別是第二個。因為第二個更符合定義，會智慧型的找到條件。然後根據繪圖的庫，可以做出來很漂亮的圖形。
freqdist
函式接收list型別的引數後，會自動建立字典，生成對應的值為鍵值，而value就是元素的次數。
from nltk import *
tem = ['hello','world','hello','dear']
print(freqdist(tem))
out:
freqdist()
plot(topk,cumulative=true)和tabulate()可以繪製對應的折線圖和**（必須安裝matplotlib庫）
條件分布conditionalfreqdist
import nltk
from nltk.corpus import brown
cfd = nltk.conditionalfreqdist((genre,word) for genre in brown.categories() for word in brown.words(categories=genre))
print("conditions are:",cfd.conditions()) #檢視conditions
print(cfd['news'])
print(cfd['news']['could'])#類似字典查詢
out:
conditions are: ['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies', 'humor', 'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance', 'science_fiction']
86
cfd.tabulate(conditions=['news','romance'],samples=['could','can'])
cfd.tabulate(conditions=['news','romance'],samples=['could','can'],cumulative=true)
could can
news 86 93
romance 193 74
could can
news 86 179
romance 193 267
記錄正規表示式在自然語言中的應用。輸入法聯想提示（9宮格輸入法）
查詢類似於hole和golf序列（4653）的單詞。
import re
from nltk.corpus import words
wordlist = [w for w in words.words('en-basic') if w.islower()]
same = [w for w in wordlist if re.search(r'^[ghi][mno][jlk][def]$',w)]
print(same)
^[ghijklmno]+$等。像[^aeiouaeiou]就是匹配除母音外的所有字母。
尋找字元塊
查詢兩個或兩個以上的母音序列，並且確定相對頻率。
import nltk
wsj = sorted(set(nltk.corpus.treebank.words()))
fd = nltk.freqdist(vs for word in wsj for vs in re.findall(r'[aeiou]',word))
fd.items()
查詢詞幹
def stem(word):
for suffix in ['ing','ly','ed','ious','ies','ive','es','s','ment']:
if word.endswith(suffix):
return word[:-len(suffix)]
return none
re.findall(r'^(.*?)(ing|ly|ed|ious|ies|ive|es|s|ment)$',word)porterstemmer和lancasterstemmer兩個詞幹提取器，porter比較好，可以處理lying這樣的單詞。
porter = nltk.porterstemmer()
print(porter.stem('lying'))
wordnetlemmatizer
wnl = nltk.wordnetlemmatizer()
print(wnl.lemmatize('women'))
利用到nltk.index這個函式，nltk.index((word , i) for (i,word) in enumerate(['a','b','a']))
class indextext:
def __init__(self,stemmer,text):
self._text = text
self._stemmer = stemmer
self._index = nltk.index((self._stem(word),i) for (i,word) in enumerate(text))
def _stem(self,word):
return self._stemmer.stem(word).lower()
def concordance(self,word,width =40):
key = self._stem(word)
wc = width/4 #words of context
for i in self._index[key]:
lcontext = ' '.join(self._text[int(i-wc):int(i)])
rcontext = ' '.join(self._text[int(i):int(i+wc)])
ldisplay = '%*s' % (width,lcontext[-width:])
rdisplay = '%-*s' % (width,rcontext[:width])
print(ldisplay,rdisplay)
porter = nltk.porterstemmer()
grail = nltk.corpus.webtext.words('grail.txt')
text = indextext(porter,grail)
text.concordance('lie')
NLTK學習筆記
學習參考書 nltk.set proxy com 80 nltk.download 2.使用sents fileid 函式時候出現 resource tokenizers punkt english.pickle not found.please use the nltk to obtain the...
NLTK學習筆記
學習參考書 nltk.set proxy com 80 nltk.download 2.使用sents fileid 函式時候出現 resource tokenizers punkt english.pickle not found.please use the nltk to obtain the...
NLTK學習筆記
學習參考書 nltk.set proxy com 80 nltk.download 2.使用sents fileid 函式時候出現 resource tokenizers punkt english.pickle not found.please use the nltk to obtain the...