CENTRAL CODE STATIONCentral Labs — Local AI & Computing
Local LLM Runtime
A self-hosted inference layer for running open-weight language models on our own GPUs, built to explore what stays possible without a cloud API in the loop.
Most of our AI client work runs on hosted models, for good reason. But we wanted a place to explore the other end of the spectrum: what a fully self-hosted AI stack looks like, with full control over data, latency, and cost, running on local GPUs in our own office.
How it works
A Rust-based scheduler queues inference requests across available GPUs, while a Python serving layer handles model loading and batching for a handful of open-weight models. A gRPC interface lets internal tools call into it the same way they'd call a hosted API.
What we learned
Running models ourselves surfaced operational questions — memory pressure, batching trade-offs, cold-start latency — that inform how we advise clients evaluating self-hosted versus hosted AI, well beyond this one project.