CBOW

A from-scratch CBOW (Continuous Bag of Words) language model in C++17, using Eigen for linear algebra. No ML framework — the forward pass, cross-entropy gradient, and SGD update are all written out by hand.

Given seq_length consecutive words, the model predicts the word that follows.

How it works

Two matrices:

	shape
embeddings	vocab_size x dims	random, frozen — one row per vocabulary word
theta	(seq_length * dims) x vocab_size	the trained weights

Per training sample:

1. Lookup — gather the seq_length context rows out of embeddings and concatenate them into one 1 x (seq_length * dims) row vector.
2. Linear — multiply by theta to get 1 x vocab_size logits.
3. Softmax — turn the logits into a probability over the vocabulary.
4. Backward — the cross-entropy gradient w.r.t. the logits is softmax - one_hot(target); the gradient w.r.t. theta is the outer product of the input vector with it.
5. Optimize — plain SGD, one sample at a time: theta -= lr * gradient.

Cross-entropy loss (-log p_target) is printed each epoch. For reference, with this corpus ln(vocab_size) ~ 5.19 is what a uniform random guess scores — anything below that is learning.

Layout

src/
  main.cpp      entry point — loads text, trains, predicts
  loader.hpp    text parsing, vocabulary, sample generation
  loader.cpp
  cbow.hpp      the model
  cbow.cpp
  common.hpp    hyperparameters (seq_length, stride, dims)
  text.txt      the training corpus

Build

Requires CMake 3.24+ and a C++17 compiler. Eigen is fetched automatically into third_party/ on the first configure — no manual install.

cmake -S . -B build
cmake --build build

Run

From the repository root, since the corpus path is relative:

./build/cbow

Running it from inside build/ fails with loader: cannot open src/text.txt.

Output is the mean loss per epoch:

epoch 0  loss=6.83315
epoch 1  loss=6.43517
...
epoch 199  loss=0.363341

Hyperparameters

Compile-time constants, so changing them means rebuilding.

src/common.hpp:

	default
seq_length	3	context words per sample
stride	1	step between sliding windows
dims	10	embedding dimensions per word

src/cbow.hpp:

	default
learning_rate	0.03	SGD step size
epoch	200	passes over the dataset

Notes

• Text is normalized aggressively: lowercased, every non-alphanumeric byte becomes a separator, and one-letter words are dropped. That last rule means a and i never enter the vocabulary, so the token stream differs from a plain [^a-z0-9]+ regex split.
• Embeddings are frozen. Only theta is trained, which is why convergence takes a few hundred epochs — the model is fitting a linear map on top of fixed random vectors. Backpropagating into embeddings reaches the same accuracy in far fewer epochs, and is what you'd do if you wanted the embeddings themselves as the output.
• Random is unseeded, so std::rand() starts from the same state and every run is identical. Call std::srand() if you want variation.