Clustering of documents using the "bag of words" method

In this example, we will explore a way of presenting documents called a "bag of words" (BoW). This way of translating documents into numbers will allow us to translate each text into a vector representation. Despite the primitiveness of this vectorization method, it is possible to implement applied tasks based on it, provided that a suitable data set is available.

In this project, we will solve the clustering problem, which does not require training and test datasets, but simply divides the dataset into groups based on their coordinates in some space. First, we will study how to put texts into a homogeneous vector space, then how to saturate it with information, and finally how to see the result of clustering a set of texts.

We have prepared the environment and uploaded the necessary objects - a list of "stop words" that are skipped during the analysis and the simplest algorithm for detecting the root of a word (in order to get rid of word forms). Let's start downloading the texts.

Upload the texts

The data set that we will work with in this example will be a table with names and descriptions of examples from the Engee community. This is a set of small articles on various technical topics, from radio to economic calculations. The titles and short descriptions are written by the authors of the articles.

Let's see if we can use a primitive BoW to cluster these texts.

We should immediately note that the texts are quite short. With a sufficiently variable dictionary, the informative value of each text will be quite low.

Creating a case

The set of analyzed texts is called a corpus. They are both objects that fix the linguistic norm, and individual instances that can break out of it. First, we need to reduce the variability of words that occur. We will convert each text to lowercase (lowercase) and assemble the object. Corpus from objects StringDocument.

If we were to start working with this corpus now, each document could be represented by a vector of 5220 markers, each of which corresponds to the number of times a particular word occurs in the document. There are a lot more words than documents now. We need to simplify this space a bit. To do this, we will remove all numbers, spaces, and punctuation from the texts and reassemble the corpus.:

Let's remove stop words and words that are shorter than two characters from it.:

And the remaining words will be passed through a "stemmer" - an algorithm that cuts off typical word endings. Only cropped constructions will remain in the dictionary, which will sometimes match the root of the word. But we'll get rid of the word forms and be able to work with more unique constructions.

Each document is characterized by a certain set of words. But for our analysis, we won't need unique words that don't occur in any other document. Let's discard them by filtering out those words from the dictionary that do not occur more than once in any document.

Function dtm() collects a matrix in which a set of words from the dictionary is assigned to each document.

We have 850 documents, many of which are characterized by the same words (vertical lines in the matrix), so in the final analysis we will see clusters around uninformative words like "work" or "example".

This is not a typical result for document clustering, although it would also be interesting to analyze it. Perhaps these are clusters along the length of the documents, and the variance arises from the fact that we projected the data into a reduced-dimensional space, losing some information about their variability, although the documents were very scattered in the original multidimensional space.

We have arranged all the documents in a 1,458-dimensional space. Naturally, to see them on the graph, we need to reduce the dimension, and the PCA analysis of the main components of the vector space will help us in this. It finds the projection in which the data is most variable.

To highlight the outstanding words, we will continue to work with a sparse representation of the text, but we will try to highlight the subject of each text. We will not establish connections between words (dense text analysis), but will raise the weight of some words relative to others.

Noise reduction using TFIDF

The next method that we need to try to apply is the method of "weighing" words in our bag, which is called TFIDF (term frequency, inverse document frequency). We will add weights to the words so that not all of them have the same meaning for analysis.

In our vector space, the documents will no longer be located in the same place as the rest of the documents where the word "work" is located. Along the axes of uninformative words, documents will slide closer to zero, and these axes will become uninformative.

The most expressive coordinates will be the numbers that characterize the presence of words in the document that are rarely found in the entire corpus. The more often a word occurs in a document and the less frequently it occurs in the corpus, the further away our document will be from zero at the coordinate corresponding to this dictionary word.

A simple TFIDF document matrix looks the same as the frequency matrix after BoW, but we only see the presence or absence of a number, not its value. The most unique or specific words will gain more weight.

The problem is that we work with short texts. If there are from 2 to 23 words in a text (the average length is 13), then almost any words are rare and will look like noise. Nevertheless, let's see how our texts are arranged in space now.

Interestingly, many tests are located in the middle of the coordinate system (not representative), but there are several "rays" - groups of words that occur together and characterize their groups of documents. This is a typical picture for the thematic modeling task.

Nothing guarantees that these groups of words are related to each other, we have a fairly small data set and short texts, the grouping may be the result of a random distribution.

In addition, this is a grouping after projection, there might not have been any "rays" in the original space. But the texts were definitely located at different distances from the origin.

Let's perform clusterization for the number of clusters we set.

Cosine metric clustering, for centered data, allows us to cluster documents by their "spatial angle". We tried to make the individual rays fall into separate clusters.

Here are some of the most representative documents for each cluster.:

It's not that the differences are obvious. Let's look at the statistics:

Our documents don't lend themselves very well to this kind of clustering. The high sparsity of the matrix means that very few documents can be grouped by such a vector feature.

We also see that each document is characterized by an average of 1.8 unique words (our matrix has so many non-zero elements on average). This is very small, it is better to increase this figure to 5-10% of the vocabulary, but since our texts are very short, we will not be able to do this.

Having 1-2 significant words per document (after stemming), we get almost zero variance - all documents are very poor in information.

Conclusion

Sparse text analysis methods usually show good results in auxiliary tasks: spam filtering, identification of unique texts or symbols, comparison of documents with specific terminology, and basic clustering.

We have also studied the methods of text preprocessing that are used at every step, in every text analysis application project. As you can see, we can use a dozen functions to organize a relatively complex text processing chain.