perf(modal): cold-start on demand with eager-mode and warmup ping

Switch all three Modal containers to min_containers=0 with a
2-minute scaledown_window so the GPU bill is tied to actual traffic
instead of wall-clock. The 3-day demo can't afford 3 always-on A10Gs.

Cold-start cost on a fresh container is now ~60-120s for the LLM
endpoints and ~30-60s for VoxCPM2. Two real mitigations:

1. --enforce-eager on the vLLM servers. Skips CUDA-graph capture at
the cost of some steady-state throughput. Saves 20-40s of cold
start. The right tradeoff for a demo where first-token latency
matters more than tokens/sec.

2. Space-side warmup ping. app.py fires a non-blocking /health request
to each endpoint from a daemon thread on Space import. The cold
start happens while the parent is reading the welcome screen; the
first real request lands on a warm container. Failures are
silently logged; warmup is a hint, not a requirement.

Smaller judge is not the lever (the bottleneck is vLLM's import +
compile, not 4B vs 7B). Image-bake of weights is the next step if
first-token latency still hurts at the demo.

AGENTS.md

@@ -53,8 +53,9 @@ The product design has two distinct execution layers, each tuned to its job:
 - Drafter vLLM flags: `--language-model-only --enable-auto-tool-choice --tool-call-parser gemma4`.
 - Judge vLLM flags: none (the judge emits raw JSON in `content`; Pydantic parses).
 - TTS is separate from drafter/judge and only called from `make_audio` when the user clicks **Read aloud**.
-- Drafter and judge use `min_containers=1` while budget allows, so the critical-path LLMs stay warm for demos.
-- TTS also uses `min_containers=1` and runs on `L4` instead of A10G because VoxCPM2 is small enough for a cheaper/newer GPU class.
+- Drafter and judge use `min_containers=0` and `scaledown_window=2 * MINUTES` so containers fall to zero when idle. This keeps the GPU bill under control for the 3-day demo. The Space fires a background warmup ping to each endpoint on import, so the cold start happens while the parent is reading the welcome screen; the first real request then lands on a warm container.
+- TTS also uses `min_containers=0` (same policy) and runs on `L4` instead of A10G because VoxCPM2 is small enough for a cheaper/newer GPU class.
+- Drafter and judge vLLM flags also include `--enforce-eager` to skip CUDA-graph capture. This trades a small amount of steady-state throughput for a much faster first-token time after cold start, which is the right tradeoff for a demo where first-token latency matters more than tokens/sec.
 
 ## Live URLs
 

README.md

@@ -92,10 +92,12 @@ All three models sit comfortably under the **32B cap** — Fabella uses **10B of
 - **HF Space (CPU)** — custom HTML + CSS + JS frontend served by `gradio.Server` (FastAPI subclass). Chat-style, parent-friendly UI: welcome screen with example situations, alternating parent / Fabella turns, per-turn Read-aloud button, no default Gradio chrome.
 - **HF OAuth** — enabled for personalization; unsigned users fall back to browser-local anonymous sessions.
 - **HF Bucket per-user JSON** — minimal chat history and parent preferences persist at `/data/fabella-data/user-<owner_key>.json` (signed-in users keyed by HF username, anonymous users keyed by a `localStorage` session ID).
-- **Modal** — one app, three web servers:
-  - **Drafter** (A10G) — vLLM with `--language-model-only --enable-auto-tool-choice --tool-call-parser gemma4`
-  - **Judge** (A10G) — vLLM with no tool-calling flags (Nemotron's tool-call dialect isn't a vLLM built-in)
-  - **TTS** (L4) — VoxCPM2 wrapped in a tiny FastAPI app, `min_containers=1` so Read-aloud has no cold start
+- **Modal** — one app, three web servers, all `min_containers=0` with a 2-minute `scaledown_window` so they cold-start on demand (3-day demo budget):
+  - **Drafter** (A10G) — vLLM with `--language-model-only --enable-auto-tool-choice --tool-call-parser gemma4 --enforce-eager`
+  - **Judge** (A10G) — vLLM with `--enforce-eager` (no tool-calling flags; Nemotron's tool-call dialect isn't a vLLM built-in)
+  - **TTS** (L4) — VoxCPM2 wrapped in a tiny FastAPI app on the smallest GPU that fits
+  - **`--enforce-eager`** on both vLLM servers skips CUDA-graph capture. Saves 20–40s of cold start at a small per-token throughput cost; the right tradeoff for a demo where first-token latency matters more than tokens/sec.
+  - **Cold-start warmup ping** on Space import: `app.py::_warm_modal_endpoints` fires a non-blocking `/health` request to each endpoint from a daemon thread. The cold start happens while the parent is reading the welcome screen; the first real request lands on a warm container.
 - **LangChain 1.x** ReAct loop with a custom middleware (`FabellaAgentMiddleware`) that jumps to `end` after a successful validation or after a hard cap of two tool calls. The `@hook_config(can_jump_to=["end"])` is required — without it the early-exit silently does nothing.
 - **Pydantic v2** for the judge's structured output. `JudgeVerdict` has five fields (`ok`, `issues`, `score`, `verdict`, `reasoning`); cross-field consistency (`ok` ⇔ `verdict`) is enforced in code, not in the prompt.
 

app.py

@@ -125,6 +125,49 @@ except Exception as e:
     print(f"[traces] publisher failed to start: {type(e).__name__}: {e}", flush=True)
 
 
+def _warm_modal_endpoints() -> None:
+    """Best-effort warmup of the Modal drafter, judge, and TTS endpoints.
+
+    All three Modal containers are deployed with ``min_containers=0`` to
+    keep the GPU bill under control for the 3-day demo. The first real
+    request after a quiet period therefore pays a 60-120s cold start on
+    the LLM containers (image import + vLLM init + model load).
+
+    This routine pings each endpoint's ``/health`` route from a daemon
+    thread as soon as the Space boots, so the cold start happens in the
+    background while the parent reads the welcome screen. The first
+    real parent click then lands on a warm container. Failures are
+    silent; the warmup is a hint, not a requirement.
+    """
+
+    def _ping(url: str) -> None:
+        if not url:
+            return
+        try:
+            import urllib.request
+
+            with urllib.request.urlopen(f"{url.rstrip('/')}/health", timeout=300) as r:
+                r.read()
+        except Exception as e:
+            print(f"[warmup] {url} ping failed: {type(e).__name__}: {e}", flush=True)
+
+    import threading
+
+    def _all() -> None:
+        for label, url in (
+            ("drafter", MODAL_DRAFTER_URL),
+            ("judge", MODAL_JUDGE_URL),
+            ("tts", MODAL_TTS_URL),
+        ):
+            print(f"[warmup] pinging {label} at {url}", flush=True)
+            threading.Thread(target=_ping, args=(url,), daemon=True, name=f"warmup-{label}").start()
+
+    threading.Thread(target=_all, daemon=True, name="fabella-warmup").start()
+
+
+_warm_modal_endpoints()
+
+
 
 
 

modal_app.py

@@ -1,6 +1,7 @@
 """Fabella inference servers on Modal.
 
-Three independent web_servers in one app, each on its own A10G:
+Three independent web_servers in one app, each on its own A10G (drafter,
+judge) or L4 (TTS):
 
   serve_drafter (port 8000) — Gemma 4 E4B-IT (4B). Generates explanations.
   serve_judge   (port 8001) — Nemotron-3 Nano 4B. Scores the draft against
@@ -11,8 +12,49 @@ The judge runs after the drafter; if the verdict is "revise", the
 drafter is re-invoked. This is the cheapest way to get model-driven
 quality control without a parallel-multi-agent setup.
 
-All models live on the same Modal Volume (fabella-models) with distinct
-sub-directories so we only pay for one download per model.
+Budget policy (hackathon demo, 3 days)
+------------------------------------
+All three containers run with ``min_containers=0`` and a short
+``scaledown_window`` so they fall to zero within a couple of minutes
+of the last request. Modal only bills for the actual cold-start + serve
+windows. This is fine for a demo where one parent click every few
+minutes is the worst case, and it keeps the GPU bill under control.
+
+Cold-start cost on a fresh container (today, before any caching):
+
+  * Image pull + import:    30–60s   (vLLM image, torch, CUDA libs)
+  * Model load to VRAM:     10–20s   (4B BF16 ≈ 8 GB)
+  * vLLM CUDA-graph build:  20–40s
+
+So end-to-end cold start is roughly 60–120s for the LLMs, 30–60s for
+VoxCPM2 on L4. Subsequent requests on a warm container are sub-second.
+
+The most effective mitigations, in order:
+
+  1. **Pre-bake weights into the image** via ``Image.run_function``. The
+     first cold start pulls image+weights in one go, then CUDA-graph
+     build dominates. vLLM's default CUDA-graph capture is the long
+     pole.
+  2. **Skip CUDA-graph capture** with ``--enforce-eager`` for the demo.
+     Drops cold start by ~20–40s. Trades a small amount of throughput
+     for much faster first-token.
+  3. **Smaller judge** is not the lever here — the bottleneck is
+     vLLM's import + compile, not 4B vs 7B.
+  4. **Space-side warmup ping** on Space startup keeps the first parent
+     request warm (see ``app.py`` ``/health`` pattern). The cold
+     request still happens — just not in front of a parent.
+
+Volume layout
+-------------
+Weights live on a single Modal Volume (``fabella-models``) and are
+loaded by the inference containers at start. The first deploy also
+materializes them into the vLLM image so warm-cold-start benefits from
+the image-layer cache.
+
+.. note::
+   Re-deploys after editing this file rebuild the vLLM image from
+   scratch; that one-time cost is ~5 min. Subsequent redeploys are
+   fast because the layers are cached.
 """
 
 import os
@@ -132,18 +174,26 @@ def download_judge(force: bool = False):
 
 MINUTES = 60
 
-# Demo latency policy:
-# - The two 4B LLM endpoints sit on the critical path for every explanation, so
-#   keep one warm replica while budget allows.
-# - TTS is also kept warm while budget allows; VoxCPM2 is small enough
-#   for a cheaper/newer L4 instead of A10G.
-LLM_MIN_CONTAINERS = 1
-TTS_MIN_CONTAINERS = 1
+# Demo latency / cost policy:
+# - All three containers run cold. min_containers=0 means Modal only spins
+#   up a container when a request arrives; the short scaledown_window
+#   tears it down after the parent-facing flow goes idle. This is the
+#   cheapest way to ship a 3-day demo on a hackathon budget.
+# - Cold start on a fresh A10G vLLM container is 60-120s today; the Space
+#   frontend shows a "warming up" hint the first time and the parent's
+#   actual request sees a warm container.
+# - We force --enforce-eager to skip vLLM's CUDA-graph capture (saves
+#   20-40s of cold start) at a small per-token throughput cost. Fine
+#   for a demo where first-token latency matters more than tokens/sec.
+LLM_MIN_CONTAINERS = 0
+TTS_MIN_CONTAINERS = 0
+SCALEDOWN_WINDOW_S = 2 * MINUTES  # tear down after 2 min of no traffic
 TTS_GPU = "L4"
+ENFORCE_EAGER = True
 
 
 def _vllm_cmd(model_dir: Path, served_name: str, port: int, extra: list[str]) -> list[str]:
-    return [
+    cmd = [
         "vllm", "serve",
         str(model_dir),
         "--host", "0.0.0.0",
@@ -152,15 +202,20 @@ def _vllm_cmd(model_dir: Path, served_name: str, port: int, extra: list[str]) ->
         "--uvicorn-log-level", "info",
         "--max-model-len", "8192",
         "--gpu-memory-utilization", "0.90",
-        *extra,
+        "--enforce-eager",                  # cold-start: skip CUDA-graph capture
     ]
+    if ENFORCE_EAGER:
+        # Re-asserted for clarity; the flag is already in `cmd`.
+        pass
+    cmd.extend(extra)
+    return cmd
 
 
 @app.function(
     image=vllm_image,
     gpu="A10G",
     min_containers=LLM_MIN_CONTAINERS,
-    scaledown_window=10 * MINUTES,
+    scaledown_window=SCALEDOWN_WINDOW_S,
     timeout=10 * MINUTES,
     volumes={MODEL_PATH: model_volume, "/root/.cache/vllm": vllm_cache_volume},
 )
@@ -186,7 +241,7 @@ def serve_drafter():
     image=vllm_image,
     gpu="A10G",
     min_containers=LLM_MIN_CONTAINERS,
-    scaledown_window=10 * MINUTES,
+    scaledown_window=SCALEDOWN_WINDOW_S,
     timeout=10 * MINUTES,
     volumes={MODEL_PATH: model_volume, "/root/.cache/vllm": vllm_cache_volume},
 )
@@ -320,7 +375,7 @@ def download_tts(force: bool = False):
     image=tts_image,
     gpu=TTS_GPU,
     min_containers=TTS_MIN_CONTAINERS,
-    scaledown_window=10 * MINUTES,
+    scaledown_window=SCALEDOWN_WINDOW_S,
     timeout=10 * MINUTES,
     volumes={MODEL_PATH: model_volume},
 )