4.1 NLP之词条化与序列化
句子词条化
词条化就是设置分词器，把我们给定库中句子的单词进行编号。将单词编码为数字，得到一个词典。
from tensorflow.keras.preprocessing.text import Tokenizer
# 给定的库
sentences = [
'i love my dog',
'I, love my cat',
'You love my dog!'
]
# 设置一个分词器
tokenizer = Tokenizer(num_words = 100)
tokenizer.fit_on_texts(sentences)
word_index = tokenizer.word_index
print(word_index){'love': 1, 'my': 2, 'i': 3, 'dog': 4, 'cat': 5, 'you': 6}句子序列化
序列化就是将所有句子统一转化为长度相同的数字矩阵。
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
sentences = [
'I love my dog',
'I love my cat',
'You love my dog!',
'Do you think my dog is amazing?'
]
# 这里的oov_token="<OOV>"，是设置一个填充，当遇到不在库中的词时，都会用它代替
tokenizer = Tokenizer(num_words = 100, oov_token="<OOV>")
tokenizer.fit_on_texts(sentences)
word_index = tokenizer.word_index
sequences = tokenizer.texts_to_sequences(sentences)
# 把所有句子设置成统一的长度，不足就补0，默认前补0.超出就舍弃，默认也是舍弃前面。这两项都可以使用padding='post'设置为后面补0，或者舍弃后面。
padded = pad_sequences(sequences,maxlen = 8)
print("\nWord Index = " , word_index)
print("\nSequences = " , sequences)
print("\nPadded Sequences:")
print(padded)
# Try with words that the tokenizer wasn't fit to
test_data = [
'i really love my dog',
'my dog loves my manatee'
]
test_seq = tokenizer.texts_to_sequences(test_data)
print("\nTest Sequence = ", test_seq)
padded = pad_sequences(test_seq,maxlen = 2)
print("\nPadded Test Sequence: ")
print(padded)Word Index = {'<OOV>': 1, 'my': 2, 'love': 3, 'dog': 4, 'i': 5, 'you': 6, 'cat': 7, 'do': 8, 'think': 9, 'is': 10, 'amazing': 11}
Sequences = [[5, 3, 2, 4], [5, 3, 2, 7], [6, 3, 2, 4], [8, 6, 9, 2, 4, 10, 11]]
Padded Sequences:
[[ 0 0 0 0 5 3 2 4]
[ 0 0 0 0 5 3 2 7]
[ 0 0 0 0 6 3 2 4]
[ 0 8 6 9 2 4 10 11]]
Test Sequence = [[5, 1, 3, 2, 4], [2, 4, 1, 2, 1]]
Padded Test Sequence:
[[2 4]
[2 1]]讽刺数据集词条序列化
讽刺数据集是官方提供的数据集之一，它里面有文章的题目，文章的链接，文章是否为嘲讽性的这三个元素。
# !wget --no-check-certificate \
# https://storage.googleapis.com/laurencemoroney-blog.appspot.com/sarcasm.json \
# -O sarcasm.json
import json
with open("../../tensorflow_datasets/sarcasm.json", 'r') as f:
datastore = json.load(f)
sentences = []
labels = []
urls = []
for item in datastore:
sentences.append(item['headline'])
labels.append(item['is_sarcastic'])
urls.append(item['article_link'])
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
tokenizer = Tokenizer(oov_token="<OOV>")
tokenizer.fit_on_texts(sentences)
word_index = tokenizer.word_index
print(len(word_index))
print(word_index)
sequences = tokenizer.texts_to_sequences(sentences)
padded = pad_sequences(sequences, padding='post')
print(sentences[2])
print(padded[2])
print(padded.shape)这会让我们得到一个超大的词典，因为提供的句子量很大。
29657
{'<OOV>': 1, 'to': 2, 'of': 3, 'the': 4, 'in': 5, 'for': 6, 'a': 7, 'on': 8, 'and': 9, 'with': 10, 'is': 11, 'new': 12, 'trump': 13, 'man': 14, 'from': 15, 'at': 16, 'about': 17, 'you': 18, 'this': 19, 'by': 20, 'after': 21, 'up': 22, 'out': 23, 'be': 24,
......................
,'gonzalez': 29652, 'breached': 29653, "'basic'": 29654, 'hikes': 29655, 'gourmet': 29656, 'foodie': 29657}
mom starting to fear son's web series closest thing she will have to grandchild
[ 145 838 2 907 1749 2093 582 4719 221 143 39 46
2 10736 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0]
(26709, 40)
Comments NOTHING