Posted By Jakub Nowacki, 18 September 2017

Pandas is a great tool for the analysis of tabular data via its DataFrame interface. Slightly less known are its capabilities for working with text data. In this post I’ll present them on some simple examples. As a comparison I’ll use my previous post about TF-IDF in Spark.

# Some used imports
%matplotlib inline
import pandas as pd
import numpy as np
import os
import glob
import matplotlib as mpl

# Just making the plots look better
mpl.style.use('ggplot')
mpl.rcParams['figure.figsize'] = (8,6)
mpl.rcParams['font.size'] = 12

First we load data, i.e. books previously downloaded from Project Gutenberg site. Here we use a handy glob module to walk over text files in a directory and read in files line by line into DataFrame.

books = glob.glob('data/*.txt')
d = list()
for book_file in books:
    with open(book_file, encoding='utf-8') as f:
        book = os.path.basename(book_file.split('.')[0])
        d.append(pd.DataFrame({'book': book, 'lines': f.readlines()}))
doc = pd.concat(d)
doc.head()

Since we’re now in Pandas, we can easily use its plotting capability to look at the number of lines in the books. Note that value_counts of books create Series indexed by the book names, thus, we don’t need to set index for plotting. Note that we can now choose plot styling; see demo page for available styles.

doc['book'].value_counts().plot.bar();

Now, we need to tokenize the sentences into words aka terms. While we can do it in a loop, we can take advantage of the split function in the text toolkit for Pandas’ Series; see this manual for all the functions. To get it we just invoke the strip function, which is a part of str, i.e. StringsMethods object. The argument is regular expression pattern, on which the string, in our case the sentence, will be split. In our case it is the regular expression set of everything that is not a letter (capital or not) or digit and underscore. This is because \W is the reverse of \w which is a set [A-Za-z0-9_], i.e. contains an underscore. In this case we want also to split on underscore so we had to add it to the splitting set. Note that this decision and other you’ll make while deciding on the splitting set, will influence the tokenization, and thus the result. Hence, select the splitting pattern carefully. If you’re not sure about your decision, use practicing pages, like regex101.com to check your patterns.

doc['words'] = doc.lines.str.strip().str.split('[\W_]+')
doc.head()

As a result we get a new column with array of words. Now we need to flatten the resulted DataFrame somehow. The best approach is to use iterators and create new DataFrame with words; see this StackOverflow post for the performance details of this solution. Note that this may take a few seconds depending on the machine; on mine it takes about 25s.

rows = list()
for row in doc[['book', 'words']].iterrows():
    r = row[1]
    for word in r.words:
        rows.append((r.book, word))

words = pd.DataFrame(rows, columns=['book', 'word'])
words.head()

Now we have DataFrame with words, but occasionally we get an empty string as a byproduct of the splitting. We should remove it as it would be counted as a term. To do that we can use function len as follows:

words = words[words.word.str.len() > 0]
words.head()

The new words DataFrame is now free of the empty strings.

If we want to calculate TF-IDF statistic we need to normalize the words by bringing them all to the same case. Again, we can use a Series string function for that.

words['word'] = words.word.str.lower()
words.head()

First, lets calculate counts of the terms per book. We can do it as follows:

counts = words.groupby('book')\
    .word.value_counts()\
    .to_frame()\
    .rename(columns={'word':'n_w'})
counts.head()

Note that as a result we are getting a hierarchical index aka MultiIndex; see Pandas manual for more details.

With this index we can now plot the results in much nicer way. E.e. we can get 5 words largest counts per book and plot it as shown below. Note that we need to reset, i.e. remove one level of index, as it doubles when we are using nlargest function. I’ve made the process into a function, as I will reuse it further.

def pretty_plot_top_n(series, top_n=5, index_level=0):
    r = series\
    .groupby(level=index_level)\
    .nlargest(top_n)\
    .reset_index(level=index_level, drop=True)
    r.plot.bar()
    return r.to_frame()


pretty_plot_top_n(counts['n_w'])

To finish the TF as shown in the previous post, we need the counts of the books to get the measure. As a reminder, below is the equation for TF:

Note that I’ve renamed the column as it inherits the name by default.

word_sum = counts.groupby(level=0)\
    .sum()\
    .rename(columns={'n_w': 'n_d'})
word_sum

Now we need to join both columns on book to get the sum of word per book into final DataFrame. Having both n_w and n_d, we can easily calculate TF as follows:

tf = counts.join(word_sum)

tf['tf'] = tf.n_w/tf.n_d

tf.head()

Again, lets look at the top 5 words with respect to TF.

pretty_plot_top_n(tf['tf'])

As before (and as expected), we’ve got the English stop-words.

Now we can do IDF; the remainder of the formula is given below:

First we need to get the document count. The simples solution is to use the Series’ method nunique, i.e. get size of set of unique elements in a series.

c_d = words.book.nunique()
c_d

6

Similarly, to get the number of unique books that every term appeared in, we can use the same method but on a grouped data. Again we need to rename the column. Note that sorting values is only for the presentation and it is not needed for the further computations.

idf = words.groupby('word')\
    .book\
    .nunique()\
    .to_frame()\
    .rename(columns={'book':'i_d'})\
    .sort_values('i_d')
idf.head()

Having all the components of the IDF formula, we can now calculate it as a new column.

idf['idf'] = np.log(c_d/idf.i_d.values)

idf.head()

To get the final DataFrame we need to join both DataFrames as follows:

tf_idf = tf.join(idf)

tf_idf.head()

Having now DataFrame with both TF and IDF values, we can calculate TF-IDF statistic.

tf_idf['tf_idf'] = tf_idf.tf * tf_idf.idf
tf_idf.head()

Again, lets see the top TF-IDF terms:

pretty_plot_top_n(tf_idf['tf_idf'])

As before we’ve got a set of important words for the given document.

Note that this more of a demo of Pandas text processing. If you’re considering using TF-IDF in a more production example, see some existing solutions like scikit-learn’s TfidfVectorizer.

Note that I’ve just scratched a surface with the Pandas’ text processing capabilietes. There are many other useful functions like the match function shown below:

r = words[words.word.str.match('^s')]\
    .groupby('word')\
    .count()\
    .rename(columns={'book': 'n'})\
    .nlargest(10, 'n')
r.plot.bar()
r

Again, I encourage you to check Pandas manual for more details.

Notebook with the above computations is available for download here.