Master the Shepherd Model Gateway

The problem nobody warns you about

Congratulations — your team got GPUs. You fire up an open model on vLLM or SGLang, point your app at it, and everything works… on your laptop. Then reality shows up:

Every one of these is the same missing layer: something should sit between apps and engines, own routing + translation + reliability, and do it at line rate. That something is a model gateway. SMG is the best open-source one — written in Rust, born from the SGLang ecosystem, and battle-tested in front of fleets of hundreds of GPU replicas.

Think of it as a pizza chain

Forget computers for two minutes. SMG's whole design fits in one analogy — a pizza delivery business:

The cashier (SMG) does everything except cooking: applies the chat template, tokenizes your text, picks which oven to use, then detokenizes the output and wraps it as a clean OpenAI-style response. The oven (engine) does only the heavy math on GPUs.

And a real business has many shops. SMG knows every shop's menu, how busy each oven is, and — crucially — what dough each oven already has prepped. If your order looks like yesterday's, SMG sends you back to the same oven so it reuses the prep instead of starting from scratch. That's "cache-aware routing", and it's SMG's superpower (Chapter 6).

The 8 words you actually need

Everything in SMG's docs and configs reduces to these terms. Learn them now, skim the full glossary later.

One request, end to end — with a live walkthrough

Here is the entire system. Every request your app ever sends walks this exact path. Press Next step and watch it happen — 8 steps, ~40 seconds.

Notice the shape: one shared front half (server → middleware → brain), one of three paths through the middle, one shared back half (response processing → stream). Once you see this shape, every SMG config file and every source module becomes obvious.

Gateway = full server, smart proxy, or cloud router

The RouterManager's choice depends on what kind of worker serves the requested model. There are exactly three possibilities:

Seven stages, one conveyor belt

On the gRPC path, every request is assembled by a RequestPipeline — an ordered list of stages, each with one job. This is the heart of SMG; if you understand these 7 stages, you understand the codebase.

Cache-aware routing — the superpower, live

SMG ships 10 policies. Nine of them you already half-understand (round_robin, random, least_load, power_of_two = sample two workers pick the lighter, consistent_hashing = sticky by a key, prefix_hash, bucket, manual, passthrough). The one that makes SMG famous is cache_aware:

1. Tokenize the incoming request's prefix.
2. Search a radix tree per worker for the longest matching cached prefix.
3. If the match covers enough of the prompt → route to that worker (its KV cache will be reused → tiny TTFT).
4. Otherwise → route to the worker with the most free capacity / least load.
5. If the fleet gets imbalanced → fall back to least-loaded.

Don't read about it — play with it. Below are three workers with their current KV caches. Fire requests and watch the policy choose. Flip between cache_aware and round_robin and watch the "tokens recomputed" counter explode:

Which policy when?

Teaching one gateway to speak 20 model dialects

Modern models emit more than prose. Two extras matter most — and every model family formats them differently:

Before SMG, every application carried per-model if/else spaghetti for this. After SMG, the gateway absorbs dialects and your app sees one clean OpenAI-style shape regardless of which engine or model answered.

# what your app receives (separate_reasoning: true) — regardless of model family
{
  "choices": [{
    "message": {
      "role": "assistant",
      "reasoning_content": "Let me think step by step…",
      "content": "The answer is 42.",
      "tool_calls": [ … ]
    }
  }]
}

MCP, chat history, WASM plugins, multi-tenancy

Staying alive: HA mesh, resilience, observability

# a 3-node HA cluster is this boring to start (boring = good)
smg launch --enable-mesh --mesh-advertise-host 10.0.0.11 --worker-urls http://w1:8000
smg launch --enable-mesh --mesh-advertise-host 10.0.0.12 --mesh-peer-urls 10.0.0.11:39527 --worker-urls http://w2:8000
smg launch --enable-mesh --mesh-advertise-host 10.0.0.13 --mesh-peer-urls 10.0.0.11:39527 --worker-urls http://w3:8000
# then: curl http://any-node:30000/ha/status

Install it, run it, break it

1 · Install (pick one)

# Docker
docker pull lightseekorg/smg:latest
# Kubernetes
helm install smg oci://ghcr.io/smg-project/charts/smg
# Python (embeds the Rust binary)
pip install smg
# Rust (needs protoc)
cargo install smg

2 · Point it at workers

# one worker
smg launch --worker-urls http://localhost:8000
# a fleet, with the superpower on
smg launch --worker-urls http://gpu1:8000 http://gpu2:8000 --policy cache_aware

3 · Call it like it's OpenAI

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3","messages":[{"role":"user","content":"Hello!"}]}'

4 · The gRPC flagship (what production actually runs)

# engine side: expose SGLang/vLLM over gRPC using the bundled Python shim (grpc_servicer)
vllm serve meta-llama/Llama-3.1-8B --grpc
# or: sglang serve --model-path … --grpc-mode

# gateway side: full-server mode with parsers + tokenizer cache
smg launch --worker-urls grpc://worker:50051 \
  --reasoning-parser deepseek_r1 \
  --tool-call-parser llama \
  --tokenizer-cache-enable-l0 --tokenizer-cache-enable-l1 \
  --policy cache_aware

SMG in production Kubernetes — the two-tier pattern

Large serving platforms typically run SMG (or a fork of it) inside every model pod, with a separate cluster-level router tier in front. Here is the deployment shape you'll meet in the wild — and the part that confuses everyone:

Prove you own this

Eight questions. Instant feedback. 6+ correct = you understand SMG better than most people running it.

Mastery checklist

• I can explain why a gateway exists (Chapter 0) in one sentence.
• I can draw the request lifecycle: server → middleware → RouterManager → 3 paths → response processing.
• I know when the gateway is a "full server" vs a "smart proxy" — and who tokenizes in each.
• I can name the 7 gRPC pipeline stages in order.
• I can explain cache-aware routing with the radix tree and why it cuts TTFT.
• I know what reasoning/tool parsers do and why apps no longer need per-model code.
• I can start SMG against workers and hit it with curl.
• I can read a production deployment config: which process fronts the pod, which engine is inside, HTTP vs gRPC mode, and which tier dispatches across pods.

Glossary

Cheat sheet

# ── run ──
smg launch --worker-urls http://w1:8000 http://w2:8000 --policy cache_aware
smg launch --worker-urls grpc://w1:50051 --reasoning-parser deepseek_r1 --tool-call-parser llama
smg launch --enable-mesh --mesh-advertise-host 10.0.0.1 --mesh-peer-urls 10.0.0.2:39527

# ── call ──
curl localhost:30000/v1/chat/completions -d '{"model":"m","messages":[…]}'
curl localhost:30000/v1/models            # discovery
curl localhost:30000/ha/status            # mesh cluster state

# ── the four-line production-config reader ──
# pod command runs smg   → pod's front door is SMG
# --backend sglang       → engine = SGLang   |   --connection-mode grpc → full-server mode
# ingress/router tier    → who dispatches requests across pods

# ── the one-sentence summary, memorize it ──
# Axum HTTP → middleware → RouterManager picks gRPC/HTTP/external →
# gRPC path: template → tokenize(+cache) → policy pick → build → execute → detokenize+parse