Before the model comes the data

I’ve started building the dataset for my first language model, and it’s already changing how I think about training AI.

My goal is to train a model specifically for TypeScript code generation, entirely from scratch. Naturally, I assumed the interesting part would be building the transformer, implementing attention, and eventually watching the loss decrease.

But before any of that, I need data. Lots of it. And collecting code isn’t quite as simple as downloading a few thousand GitHub repositories. There’s licensing to consider, duplicate files to detect, generated code to exclude, and source information to preserve. Even extracting TypeScript from repositories containing several different languages requires some thought.

I’ve realised that a good training dataset is an engineering project in its own right. For now, I’m deliberately focusing on collecting and preparing the data before jumping into tokenisation.

The neural network might do the learning, but what it learns depends entirely on what we feed it. It’s another reminder that building AI is about much more than the model itself.