N-grams

An n-gram is just n consecutive words treated as one feature: a single word is a unigram, exactly what bag of words already counts. Two consecutive words are a bigram, three are a trigram, and so on.

Take the bigrams of “boy girl good”: the adjacent pairs are “boy girl” and “girl good”, not “good boy”, order decides which pairs exist. Across the corpus, four distinct bigrams show up, and each document lights up only the ones it actually contains: D1 “good boy” = [1,0,0,0], D2 “good girl” = [0,1,0,0], D3 “boy girl good” = [0,0,1,1].

The same three documents that were 50 to 82 percent similar under plain bag of words are now completely unrelated under bigrams, every pairwise cosine similarity drops to 0. Bigrams recovered the order information bag of words lost, at the cost of matching almost nothing unless the exact same pair of words shows up again.

That trade-off gets worse as n grows and as the vocabulary grows: the number of possible n-grams multiplies combinatorially, so the feature space explodes and ends up sparser than plain bag of words ever was. A word pair never seen together during training still can't be represented, an even sharper out-of-vocabulary problem than a single missing word. Fixing that without giving up on order is what dense word vectors, covered next, are built to do.