Self-hosted AI
Every AI job in the house runs on one card, and a lot of them want it at the same time.
So I built the thing that decides who gets it, the lease that hands the whole card to one job at a time, and the ledger that rebuilds what each program spent.
The contention
One card, four things that all want it
All of the AI in my home lab runs through the one GPU on my workstation. The model that summarizes my sessions wants it, so does image generation, so does the agent that watches the machines, and so does whatever I am actually working on. Two of them on the card at the same time means both are slow and one usually falls over.
A proxy sits in front of the GPU and dispatches by priority, so a background job yields to something I am waiting on rather than racing it. Every caller is registered with a priority I can change at runtime, and a caller that is misbehaving can be paused without stopping anything else.
The lease
Handing the whole card to one job without losing it
Priority dispatch works while everyone is sharing. Some jobs can't share. A model that needs the full card has to have the others off it first, and the naive version of that is a lock, which is fine right up until the thing holding it dies. Then the card is gone and nothing else can have it.
So the lease carries a time limit. A caller acquires it, the proxy drains the running models off the card and parks the other callers, and the lease holder gets the card to itself. It can renew while it is still working, and if it stops renewing the lease expires on its own and the proxy brings everything back up. A session that dies holding the card doesn't keep it.
The same tool layer that hands out leases also drives the devices around the house, deploy and logs and restart, so a session can set up a test on a handheld and run it without me in the middle.
The hard ceiling on any lease
3,600 s
Renewing restarts the clock. A holder that goes silent loses the card on its own
Acquiring the card, and the three ways it refuses
PROXY_LEASE_MAX_TTL_SEC = int(os.environ.get("PROXY_LEASE_MAX_TTL_SEC", "3600"))
# The lease dict is REPLACED wholesale (never mutated in place) so the hot-path
# read in _gpu_lease_active() is lock-free safe (CPython atomic ref read).
_gpu_lease_lock = threading.Lock()
_gpu_lease: dict | None = None
def _gpu_lease_active() -> dict | None:
"""Return the active lease dict if one is held and not expired, else None."""
lease = _gpu_lease
if lease is None:
return None
if time.time() * 1000 >= lease["expires_ms"]:
return None # expired; the monitor loop finalizes the release (restart vLLM)
return lease
@app.post("/api/gpu/lease")
async def api_gpu_lease_acquire(request: Request):
"""Acquire an exclusive GPU lease: drain ALL VRAM (stop vLLM, evict Ollama,
free ComfyUI) and park non-PROTECTED LLM callers on the hold queue until
release or TTL expiry. Mode stays 'work' -- this is NOT a mode flip.
ttl_sec : seconds until auto-release, capped at PROXY_LEASE_MAX_TTL_SEC.
The lease ALWAYS expires.
"""
global _gpu_lease
ttl_sec = max(1, min(ttl_sec, PROXY_LEASE_MAX_TTL_SEC))
# Guard 1: one lease at a time.
if _gpu_lease_active() is not None:
return JSONResponse({"error": "already_leased"}, status_code=409)
# Guard 2: only from work mode (gaming/imagegen already own the card; a lease
# on top would restart vLLM on release while the mode still says drained).
if mode != "work":
return JSONResponse({"error": "mode_not_work"}, status_code=409)
# Guard 3: don't stomp an in-progress mode rollout.
if _active_rollout is not None:
return JSONResponse({"error": "rollout_active"}, status_code=409)
now_ms = time.time() * 1000
lease = {"id": f"lease-{int(now_ms)}-{secrets.token_hex(4)}",
"holder": holder, "reason": reason,
"acquired_ms": int(now_ms),
"expires_ms": int(now_ms + ttl_sec * 1000),
"ttl_sec": ttl_sec, "renew_count": 0}
with _gpu_lease_lock:
# Re-check under lock (TOCTOU with the active() check above).
if _gpu_lease is not None and time.time() * 1000 < _gpu_lease["expires_ms"]:
return JSONResponse({"error": "already_leased"}, status_code=409)
_gpu_lease = lease
_persist_lease_to_disk(lease) # a proxy restart mid-lease must not drop it
# Same drain routine game mode uses; the mode flag itself is untouched.
vram_drain = await _drain_vram_for_gaming()
return JSONResponse(payload)
The ledger
What each program actually spent
Local inference feels free because no invoice arrives, which makes it very easy to spend an evening of GPU time on something worth about four cents. The paid models are the opposite: the bill arrives monthly, long after the decision that caused it.
A dashboard rebuilds what each program spent from the raw request logs rather than from anything the programs report about themselves. It totals the day, breaks it down per caller, tracks the cache hit rate, and prices the local work against what the same tokens would have cost from a cloud model.
That last number settles arguments, because it is the difference between believing the local stack pays for itself and knowing what it saved.
Where it stands
Running daily. Local models take the first pass at everything, and a paid model only gets called when the local one can't finish.
priority dispatch · time-limited leases · per-program spend ledger