Run SMG from source on your laptop and drive it against either a local toy model (Ollama) or a live engine in a Kubernetes cluster — a full development + testing environment for SMG features without deploying anything.
cargo run · :30000 · your code, your featureskubectl port-forwardThe engine doesn't care where SMG runs — SMG is just an HTTP/gRPC client to it. Responses flow back up the same path. That's the whole trick: borrow a live engine, bring your own gateway.
| Tool | Why | Check |
|---|---|---|
| Docker Desktop | Build/run the SMG image | docker version |
| Ollama (Path A only) | Tiny local OpenAI-compatible engine | ollama --version |
| Ollama shim (Path A only) | Fakes /health + /server_info so SMG can register Ollama as a worker | ls ./ollama_health_shim.py |
| kubectl + cluster context (Path B) | Discover & tunnel to live engine pods | kubectl config current-context |
| Rust toolchain (Path C only) | Fast code iteration without Docker rebuilds | cargo --version · needs protoc |
| HF_TOKEN (gRPC mode only) | Download tokenizers from private HuggingFace repos | echo $HF_TOKEN |
git clone https://github.com/smg-project/smg && cd smg docker build -f docker/Dockerfile -t smg:local .
python3 -m smg.launch_router, so router flags go after the image name in docker run.ollama pull llama3.2 ollama serve # OpenAI-compatible server on http://localhost:11434
SMG's worker-registration workflow probes endpoints that only SGLang/vLLM-style engines have. Ollama fails all three, in sequence:
| SMG probe | What it's for | Ollama says | Consequence |
|---|---|---|---|
GET /health | reachability gate (detect_connection_mode) | 404 | fatal — endless retries, Router ready |workers: [] |
GET /server_info (or /version) | backend identity (detect_backend) | 404 | ~27 min of retry spam (budget = --worker-startup-timeout-secs, default 1800) |
| metadata labels | the worker's model ID | n/a | registers as UNKNOWN_MODEL_ID → /v1/models empty, nothing routes |
The shim sits between SMG and Ollama on :11435: it answers /health and /server_info itself (model name fetched live from Ollama), and transparently proxies everything else.
python3 ./ollama_health_shim.py # listens :11435 → forwards :11434
Shim on :11435 -> Ollama on :11434 (adds /health + /server_info). Sanity check: curl localhost:11435/health → {"status": "ok"}, and curl localhost:11435/server_info shows your model name.docker run --rm -p 30000:30000 smg:local \ --host 0.0.0.0 --port 30000 \ --worker-urls http://host.docker.internal:11435 # ← shim port, NOT 11434
host.docker.internal = your Mac, from inside the container. On Linux use --network host + http://localhost:11435 instead.detect_connection_mode and detect_backend steps succeed (no retry spam), then register_workers / activate_workers. If you instead see Step failed … will_retry=true loops, the shim isn't up or you pointed at 11434.# 1. ALWAYS first: what model id did the worker register with? curl localhost:30000/v1/models # → {"data":[{"id":"llama3.2:latest","owned_by":"self_hosted",…}]} # 2. use that EXACT id in the chat call curl localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"llama3.2:latest","messages":[{"role":"user","content":"Explain SMG in one sentence."}]}'
"system_fingerprint":"fp_ollama" — proof the request rode client → your SMG → shim → Ollama → back. This exercises SMG as a smart proxy (HTTP mode): routing, retries, metrics, middleware.# second engine + its own shim (shim takes: listen_port upstream_port) OLLAMA_HOST=127.0.0.1:11436 ollama serve python3 ./ollama_health_shim.py 11437 11436 # restart SMG with two workers + a policy docker run --rm -p 30000:30000 smg:local \ --host 0.0.0.0 --port 30000 \ --worker-urls http://host.docker.internal:11435 http://host.docker.internal:11437 \ --policy cache_aware
Production model pods commonly expose two ports: an OpenAI-compatible HTTP port (often an in-pod SMG or similar gateway) and a raw engine gRPC port (SGLang/vLLM). The examples below use 8080 (HTTP) and 50051 (gRPC) — substitute whatever kubectl describe pod shows for your deployment.
# which cluster am I on? kubectl config current-context # find the model's pods kubectl get pods -o wide | grep <model-service-name> # confirm the ports the pod exposes kubectl describe pod <pod-name> | grep -iA8 "Ports" # (optional) the service in front of the pods kubectl get svc | grep <model-service-name>
# pick ONE pod (staging/canary ideally) — forwards its ports to localhost kubectl port-forward pod/<pod-name> 8080:8080 # in-pod gateway (OpenAI HTTP) # or, for the raw engine: kubectl port-forward pod/<pod-name> 50051:50051 # engine gRPC
http://<pod-ip>:8080 as the worker URL.kubectl port-forward is a private, authenticated tunnel through the k8s API server — nothing is exposed publicly, and only your kubeconfig identity can use it. It's safe for poking at prod-adjacent pods, but still: be gentle with pods serving real traffic.docker run --rm -p 30000:30000 smg:local \ --host 0.0.0.0 --port 30000 \ --worker-urls http://host.docker.internal:8080 # then hit YOUR gateway — request rides your SMG → pod gateway → engine → all the way back # (check curl localhost:30000/v1/models for the exact served model id first) curl localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"<served-model-id>","messages":[{"role":"user","content":"ping"}],"max_tokens":16}'
# needs: kubectl port-forward pod/<pod-name> 50051:50051 # needs: the model's tokenizer locally (HF_TOKEN for private repos) docker run --rm -p 30000:30000 \ -e HF_TOKEN=$HF_TOKEN \ smg:local \ --host 0.0.0.0 --port 30000 \ --worker-urls grpc://host.docker.internal:50051 \ --connection-mode grpc \ --model-path <hf-org>/<model-repo> \ --reasoning-parser deepseek_r1 \ --tool-call-parser llama \ --tokenizer-cache-enable-l0 --tokenizer-cache-enable-l1
--model-path must resolve to a tokenizer — a local directory (mount it with -v /path:/data:ro) or a HF repo id with HF_TOKEN. (2) Parser flags must match the model family, or reasoning/tool extraction silently won't trigger. (3) Large prompts: engines may cap gRPC message size — prod sets SGLANG_GRPC_MAX_MESSAGE_BYTES on the engine side.Docker rebuilds are slow. When you're changing SMG code (a parser, a policy, middleware), run the binary straight from the workspace and keep the same engine targets:
cd smg # your clone of github.com/smg-project/smg cargo run -p smg -- \ --worker-urls http://localhost:11435 \ # shim port (Ollama needs it here too) --policy cache_aware # or against the tunneled live engine, same as Path B: cargo run -p smg -- --worker-urls http://localhost:8080
cargo run again. Use Docker only to verify the final image before pushing.# quick correctness net before you test manually cargo test -p smg # gateway tests cargo clippy --workspace --all-targets # repo lint bar (unwrap/dbg are denied)
| If you're testing… | Do this | Proof it worked |
|---|---|---|
| Routing / LB policy | 2+ workers, same long prompt twice | SMG logs show the same worker chosen (cache_aware); smg_router_* metrics move |
| Reasoning parser | Thinking model + separate_reasoning: true | Response has reasoning_content split from content, also in SSE deltas |
| Tool-call parser | Ask for a tool call with tools set | Structured tool_calls in the response, not raw text |
| Tokenizer cache | Repeat a long prompt with L0/L1 enabled | Second request's prep stage is much faster (smg_router_stage_duration_seconds) |
| Middleware (auth/rate limit/WASM) | Request without a key / over quota | 401 / 429 before any routing; WASM hook visible in logs |
| Retries / circuit breaker | Kill the engine mid-test | Retry with backoff in logs; breaker opens; 5xx surfacing cleanly, not hangs |
| Metrics | curl localhost:<prometheus-port>/metrics (set --prometheus-port; the prod chart uses 9900) | smg_router_requests_total, per-stage durations, worker metrics present |
| Symptom | Likely cause | Fix |
|---|---|---|
| Container can't reach Ollama | localhost inside Docker ≠ your Mac | Use host.docker.internal (Mac/Win) or --network host (Linux) |
detect_connection_mode retries forever, Router ready |workers: [], 404 on /health | Ollama has no /health endpoint — SMG's registration probe requires a 2xx there | Run the shim: python3 ./ollama_health_shim.py, point SMG at http://host.docker.internal:11435 (live SGLang/vLLM engines serve /health natively — no shim needed) |
detect_backend retries for ~27 min (tried /v1/models, /version, /server_info) | Ollama isn't recognizable as sglang/vllm; retry budget comes from --worker-startup-timeout-secs (default 1800) | Same shim — it fakes /server_info so SMG detects "sglang" instantly. Worse: without it the worker registers as UNKNOWN_MODEL_ID and /v1/models stays empty |
model_not_found even though the worker is up | Model name in the request doesn't match what the worker registered | curl :30000/v1/models first, use that exact id (e.g. llama3.2:latest) — don't leave placeholders in |
curl to :30000 refused | SMG bound to 127.0.0.1 inside container | Pass --host 0.0.0.0 and keep -p 30000:30000 |
| gRPC mode: tokenizer errors | Private HF repo, no token | -e HF_TOKEN=$HF_TOKEN, or -v /local/tokenizer:/data:ro + --model-path /data |
| Reasoning/tools not parsed | Parser flag doesn't match model family | Match --reasoning-parser/--tool-call-parser to the model family (e.g. deepseek_r1, qwen3, llama) |
| port-forward dies | Pod restarted / rolled | Re-run kubectl port-forward against the new pod name |
| 429s from a live pod | You're hitting a pod under real load | Switch to a staging/canary pod — see the ground rule |
Build dies at smg-python with signal: 9 SIGKILL / ResourceExhausted: cannot allocate memory | Docker VM out of RAM during the fat-LTO link (default limit is ~8 GiB — not enough) | Docker Desktop → Settings → Resources → Memory → 12–16 GiB, then rebuild (cache resumes at the failed step). Or build with maturin --profile ci (thin LTO) |
| Build fails at maturin with compile errors | Stale build cache | docker build --no-cache (slow but clean) |
# build once git clone https://github.com/smg-project/smg && cd smg && docker build -f docker/Dockerfile -t smg:local . # A: toy local — ollama + shim + smg # (shim fakes /health + /server_info, which Ollama lacks and SMG requires) ollama pull llama3.2 && ollama serve python3 ./ollama_health_shim.py # :11435 -> :11434 docker run --rm -p 30000:30000 smg:local --host 0.0.0.0 --worker-urls http://host.docker.internal:11435 # B: live engine — discover, tunnel, run kubectl get pods -o wide | grep <model-service-name> kubectl port-forward pod/<pod> 8080:8080 # or 50051:50051 for raw gRPC docker run --rm -p 30000:30000 smg:local --host 0.0.0.0 --worker-urls http://host.docker.internal:8080 # C: dev loop cargo run -p smg -- --worker-urls http://localhost:8080 # always: verify — check /v1/models FIRST, then use that exact model id curl localhost:30000/v1/models curl localhost:30000/v1/chat/completions -H "Content-Type: application/json" \ -d '{"model":"llama3.2:latest","messages":[{"role":"user","content":"ping"}]}'