Note: In this tutorial, we will be using Kafka's "The Metamorphosis" where tells the story of salesman Gregor Samsa who wakes one morning to find himself inexplicably transformed into a huge insect.
Link:
Open the file and delete the header and footer information and save the file as "metamorphosis_clean.txt"
Step 1. Sneak peek into the data
See structure, paragraph and punctuation of the data
Determine how much of this data is useful for us
Know your objective. For instance, if we are trying to develop a Kafka language model, we may want to keep the punctuation, quotes and cases
Step 2. White space/Punctuation/Normalize Case
< Manual cleaning >
2.1 Load data
# 1. Load data
filename = 'metamorphosis_clean.txt'
file = open(filename, 'rt')
text = file.read()
file.close()
2.2 Tokenization
Tokenization: Split paragraph -> sentences -> words. Commonly split by white spaces
Split by white space
Split by white space and remove punctuation
2.2.a Split by white space
Warning: We may end up with punctuation included. Eg: 'room,'
# 2. Tokenization
# 2.2.a. split by whitespace
words = text.split()
print(words[:100])
Apply mapping table over a list to translate/strip punctuation
# 2. Tokenization
# 2.2.b Split by whitespace and remove punctuation
Create a mapping table split by whitespace and remove punctuation
import string
words = text.split()
# list the characters in the punctuation constant
print(string.punctuation)
# create a mapping table
# where the 3rd parameter: list of characters to be removed
mappingTable = str.maketrans('', '', string.punctuation)
# apply mapping to remove punctuation
stripped = [w.translate(mappingTable) for w in words]
print(stripped[:100])
# 2b. Split by sentences
from nltk.tokenize import sent_tokenize
sentences = sent_tokenize(text)
print(sentences[:10])
Output:
['One morning, when Gregor Samsa woke from troubled dreams, he found\nhimself transformed in his bed into a horrible vermin.', 'He lay on\nhis armour-like back, and if he lifted his head a little he could\nsee his brown belly, slightly domed and divided by arches into stiff\nsections.', 'The bedding was hardly able to cover it and seemed ready\nto slide off any moment.', 'His many legs, pitifully thin compared\nwith the size of the rest of him, waved about helplessly as he\nlooked.', '"What\'s happened to me?"', 'he thought.', "It wasn't a dream.", 'His room,\na proper human room although a little too small, lay peacefully\nbetween its four familiar walls.', 'A collection of textile samples\nlay spread out on the table - Samsa was a travelling salesman - and\nabove it there hung a picture that he had recently cut out of an\nillustrated magazine and housed in a nice, gilded frame.', 'It showed\na lady fitted out with a fur hat and fur boa who sat upright,\nraising a heavy fur muff that covered the whole of her lower arm\ntowards the viewer.']
2.2.c Filter out punctuation
# 3. Filter out punctuation
tokens = word_tokenize(text)
# remove all tokens that are not alphabetic
words = [word for word in tokens if word.isalpha()]
print(words[:100])
# 2. Tokenize words + remove punctuation
tokens = word_tokenize(text)
words = [word for word in tokens if word.isalpha()]
stop_words = set(stopwords.words('english'))
clean_words = [w for w in words if not w in stop_words]
print(clean_words[:100])
# 1. Import stemming model: Porter and Snowball
from nltk.stem.porter import PorterStemmer
porter = PorterStemmer()
# 2. Tokenize words + remove punctuation
tokens = word_tokenize(text)
words = [word for word in tokens if word.isalpha()]
# 3. Stemming of words
# Note: stemmed words are automatically in lowercase
stemmed = [porter.stem(word) for word in words]
print(stemmed[:100])
