Bag of Words

Clean the same kind of sentences bag of words works on: “he is a good boy”, “she is a good girl”, “boy girl good”, lowercase, stop words gone. Instead of one vector per word, bag of words builds one vector per document, counting how often each vocabulary word shows up in it.

The vocabulary is ordered by frequency across the whole corpus: good appears three times, boy and girl twice each, so the feature order is good, boy, girl. “good boy” becomes [1, 1, 0]; “good girl” becomes [1, 0, 1]; “boy girl good” becomes [1, 1, 1].

Counts aren't always useful signal, sometimes a word showing up once versus three times shouldn't matter, only whether it showed up at all. “good good boy” has a raw count vector of [2, 1, 0].

Binary bag of words caps every count at 1: that same sentence becomes [1, 1, 0], present or not, nothing else, the choice depends on whether repetition itself is meaningful for the task.

Scale this up to a real corpus and the vocabulary, and therefore every document's vector, still runs into the thousands, most of it zeros. The sparsity problem one-hot encoding had doesn't go away just because the entries are now counts instead of single 1s.

A word missing from the vocabulary still can't be represented, and it doesn't fail loudly. “good boy cat” with cat unseen during training just silently becomes the same vector as “good boy”, one word's worth of meaning quietly deleted.

Reorder a sentence and bag of words can't tell the difference: “good boy girl” and “boy girl good” count the exact same words the exact same number of times, so they produce the identical vector, [1, 1, 1]. Whatever the original order was meant to convey is gone.

Comparing these count vectors at all takes a specific tool: cosine similarity, the cosine of the angle between two vectors, 1 for pointing in exactly the same direction, 0 for pointing in completely unrelated directions. D1 and D2 here share only “good” and come out 0.5 similar; D3 shares two words with each of them and comes out around 0.82 similar to both, a real, principled number, just one that still has no idea D3's words were ever in a different order.