model deployment
Training in Python, serving in Rust
Most of my machine learning work so far has happened in Python. I’ve been writing forward passes, calculating gradients and training small networks, so I naturally pictured deployment as running the same Python code behind an API.
But training and inference ask different things of the system. Training needs gradients, optimiser state and repeated weight updates. Inference uses the learned weights to run a forward pass, with performance shaped by memory usage, batching and how efficiently the runtime uses the hardware. There’s no requirement for the HTTP server to use the language I trained in.
I’ve been exploring what serving ROMAN could look like with Rust and Axum, while keeping PyTorch for training. The model runtime is the important bridge between those two worlds. For the infrastructure itself, I’d rather begin with managed inference than spend my time maintaining GPU machines. It’s early research rather than an implementation, but I’m starting to see how much engineering sits between a checkpoint and a usable model.