Tokenisation isn't lexing

When I started thinking about tokenisation for my TypeScript model, my mind went straight to lexers. Both take source text and split it into smaller units, so I assumed the ideas would carry over fairly neatly.

A lexer follows the language’s rules. It can recognise const as a keyword and customerId as an identifier. A language-model tokenizer has a different job: turn text into a sequence of IDs the model can work with. Depending on its vocabulary, customerId might be one token or several. It doesn’t need to know what an identifier is.

That distinction matters for how I’m approaching the training data. I can collect and clean TypeScript without committing to a tokenizer yet, and I don’t need to build a compiler frontend just to get started. I’ll tackle tokenisation properly once I have enough code to make informed decisions about the vocabulary.