Self-hosted LLM deployment, from model choice to running it

Deciding to self-host is the easy part. Then come the questions that decide whether it works: which open model, at which size, quantized how far before quality drops, on which GPU, with what context length and how many requests at once, serverless or always-on. Get one wrong and the setup is slow, expensive or unstable, and the team finds out in production. After that, someone has to watch it, restart it when it fails and keep an eye on what it costs.

1. Evaluate. We pick the model, GPU type, quantization and configuration, and test them on your own data against a hosted API. That comparison is the point: if the hosted API wins for your work, we say so.
2. Build. We set it up in your cloud or GPU account: serving engine, model weights, the API your applications call, and the limits that keep it inside budget.
3. Run. We watch the setup, act on its alerts and track what it costs. How that cover works is below.

The cloud or GPU account is yours, in your name and billed to you. The code and configuration are yours, as with any build we hand over. You can take the setup elsewhere without asking us.

A hosted setup checks its own health, restarts a worker that has failed and alerts us when something looks wrong. We watch those alerts, act on them and keep track of what the setup costs.

Support is a scoped, paid option, and the level is agreed in the scope of work before anything starts. Standard cover runs in business hours. Round-the-clock cover, including emergencies, is available as a paid upgrade. In practice one founder works Indian hours and the other works US hours.

Evidence so far is one serverless test on RunPod. We make no claims beyond that: no production deployments, no uptime figures, no benchmark numbers and no cost savings.

• Light or uneven use. Idle GPUs cost money and serverless workers take time to start, so a hosted API is simpler (OpenAI).
• One person or a small team. A single machine with Ollama is easier to run (Ollama).
• Work that needs the strongest available model. Weigh the setups against each other before committing (hosted against self-hosted).
• Answers that must come from your own records. That is a retrieval problem first, whichever model you use (RAG).

• One open model serving many people at once, such as a company-wide assistant.
• Data that has to stay in an account you control.
• Load steady enough that GPUs stay busy.

Cold starts. A serverless GPU worker that has scaled to zero starts a container, fetches the model weights, loads them into GPU memory and warms up the engine before it answers. The first request after a quiet spell waits for all of that. Ways to shorten or avoid the wait:

• Keep weights close. Bake them into the container image or store them on an attached volume, rather than downloading them on each start.
• Keep a worker warm when users are active. One running worker through working hours removes cold starts then, at the cost of paying for idle time.
• Tune the idle timeout. A longer timeout keeps a worker alive between bursts; a shorter one saves money and brings cold starts back.
• Pick the smallest model that passes the tests. Smaller weights load faster.
• Measure cold and warm requests separately. An average hides the slow first request a user notices.
• Tell the user. When a cold start can’t be avoided, show that the answer is coming instead of a silent wait.

The rest of the setup.

• The exact GPU type matters. Partitioned GPU slices (MIG) can break model start-up in some setups even when memory looks sufficient. We test on the precise GPU type before committing.
• Failing workers can keep billing. A worker stuck restarting may still charge, so alerts and a way to stop the queue come before launch.
• Memory settings cap concurrency. --gpu-memory-utilization sets vLLM’s share of GPU memory; what is left after the weights becomes KV cache, which limits tokens in flight. --max-model-len caps each request’s length and --max-num-seqs caps concurrent requests. All three are checked before concurrency tests.
• Pin versions. Engine and model upgrades can change behaviour, so both are pinned and tested.

Pick the work you have in mind and the constraints around it, and the demo shows the choices we would make behind it. It runs no AI. Every result comes from a fixed rules table, and the page makes no network calls.

When the answers point to a hosted API instead, the card says so and recommends that rather than self-hosting.

1. Was the model tested on our own data against a hosted API, and what did that show?
2. Was it tested on exactly the GPU type it will run on?
3. Who watches it after launch, under what cover, and what stops the bill when a worker keeps failing?

• Hosted vs self-hosted models
• Local models on one machine · AI automation