aleksey
6fd2f0e689
LLM classify via LiteLLM gigachat-classifier when heuristic confidence is low. CLASSIFIER_MODE=heuristic|hybrid|llm. Metrics classifier_source.
134 lines
4.4 KiB
Markdown
134 lines
4.4 KiB
Markdown
# EventHub AI Router — Zed gateway + LiteLLM on IFT
|
||
|
||
OpenAI-compatible endpoint for [Zed](https://zed.dev) with automatic tier/lane routing (Novita AI), optional VLESS VPN, Grafana metrics, and LiteLLM Admin UI.
|
||
|
||
## URLs (IFT, Calentiq TLS)
|
||
|
||
| Service | URL |
|
||
|---------|-----|
|
||
| **Zed API** | `https://ai-router.ift.calentiq.com/v1` |
|
||
| **LiteLLM Admin UI** | `https://litellm.ift.calentiq.com/ui` (login: `admin` / `LITELLM_MASTER_KEY`) |
|
||
| **Grafana** | `https://grafana.ift.calentiq.com` |
|
||
| Legacy (self-signed) | `https://ai-router.ift.eventhub.local/v1` |
|
||
|
||
## Zed settings.json
|
||
|
||
```json
|
||
{
|
||
"openai": {
|
||
"api_url": "https://ai-router.ift.calentiq.com/v1",
|
||
"api_key": "<ROUTER_API_KEY>"
|
||
},
|
||
"assistant": {
|
||
"default_model": { "provider": "openai", "model": "smart-router" }
|
||
}
|
||
}
|
||
```
|
||
|
||
## Architecture
|
||
|
||
- **Gateway** (`router/`) — vision/OCR split, tier classification, lanes A/B/C orchestration, Redis session
|
||
- **LiteLLM** — named models `a-*` / `b-*` / `c-*`, Auto Router v2 fallback (`smart-router-internal`)
|
||
- **PostgreSQL** — LiteLLM Admin UI, spend logs
|
||
- **Redis** — response cache + gateway session context
|
||
- **VPN** — `vless-proxy` replicas=0 by default; `scripts/vpn-enable.sh`
|
||
|
||
Config sources:
|
||
|
||
- `config/routing_rules.yaml` — tier keywords
|
||
- `config/model_matrix.yaml` — tier × lane → Novita models
|
||
- `config/orchestration.yaml` — start lanes, escalation, budget caps
|
||
|
||
## Deploy on IFT
|
||
|
||
```bash
|
||
git clone git.sabilin.com/eventhub/EventHubAiRouter /opt/ai-router-stack
|
||
cd /opt/ai-router-stack
|
||
cp .env.example .env # fill NOVITA_API_KEY, keys, POSTGRES_PASSWORD, LITELLM_SALT_KEY
|
||
bash scripts/deploy.sh
|
||
bash scripts/smoke-test.sh
|
||
sudo bash scripts/install-audit-cron.sh # daily Novita audit 03:00 MSK
|
||
```
|
||
|
||
Prerequisites: Docker Swarm, external network `eventhub-ift-net`, Traefik routes in EventHubDevOps.
|
||
|
||
### Swarm secrets
|
||
|
||
Created automatically by `deploy.sh` from `.env`: `novita_api_key`, `litellm_master_key`, `litellm_salt_key`, `router_api_key`, `postgres_password`, optional `groq_api_key`, `gemini_api_key`, `vless_conf`.
|
||
|
||
**`LITELLM_SALT_KEY`** — generate once, never change after first deploy.
|
||
|
||
## VPN
|
||
|
||
```bash
|
||
bash scripts/vpn-enable.sh # scale vless + watchdog, set HTTP_PROXY on litellm
|
||
bash scripts/vpn-disable.sh
|
||
```
|
||
|
||
Novita always direct (`NO_PROXY=api.novita.ai`).
|
||
|
||
## Observability
|
||
|
||
- Prometheus scrape: `litellm:4000/metrics`, `ai-router:8000/metrics` (via `eventhub-ift-net`)
|
||
- Grafana: vendored LiteLLM v2 dashboard + custom `gateway-tier-lane.json` (EventHubDevOps)
|
||
- Uptime Kuma monitors — see table in plan / configure manually
|
||
|
||
## Scripts
|
||
|
||
| Script | Purpose |
|
||
|--------|---------|
|
||
| `deploy.sh` | secrets + build + stack deploy |
|
||
| `smoke-test.sh` | health, classify, chat max_tokens=16 |
|
||
| `sync-routing-config.sh` | regen `litellm_config.generated.yaml` |
|
||
| `audit-novita-pricing.sh` | daily model catalog check |
|
||
| `vendor-litellm-dashboards.sh` | fetch Grafana JSON → EventHubDevOps |
|
||
| `vpn-enable.sh` / `vpn-disable.sh` | VPN toggle |
|
||
|
||
## Quality modes
|
||
|
||
Header `X-AI-Quality: auto|economy|balanced|max` or `metadata.quality_mode` in request body. Default: `auto` (start lane A/B by tier, escalate on retry/5xx).
|
||
|
||
## LLM classifier (GigaChat, optional)
|
||
|
||
When `CLASSIFIER_MODE=hybrid` (default) and `GIGACHAT_CREDENTIALS` is set:
|
||
|
||
1. **Heuristic** classify first (0 cost)
|
||
2. If `confidence < 0.6` → one call to **GigaChat-2-Lite** via LiteLLM (`gigachat-classifier`)
|
||
3. Lane orchestration unchanged (A/B/C, Redis, budget)
|
||
|
||
| `CLASSIFIER_MODE` | Behavior |
|
||
|-------------------|----------|
|
||
| `heuristic` | Keywords only (no GigaChat) |
|
||
| `hybrid` | GigaChat only on low confidence |
|
||
| `llm` | Always GigaChat for text (except vision) |
|
||
|
||
Freemium GigaChat — для личного некомmercial теста; prod — юр. тариф Сбера.
|
||
|
||
Setup: [developers.sber.ru](https://developers.sber.ru/docs/ru/gigachat/quickstart/ind-create-project) → Authorization key → `.env` `GIGACHAT_CREDENTIALS`.
|
||
|
||
Response field: `x_router_meta.classifier_source` = `heuristic` | `gigachat` | `heuristic_fallback`.
|
||
|
||
## Response metadata
|
||
|
||
Each chat response includes `x_router_meta`:
|
||
|
||
```json
|
||
{
|
||
"tier": "MEDIUM_OPS",
|
||
"lane": "A",
|
||
"model": "a-medium-ops",
|
||
"escalation_level": 0,
|
||
"quality_mode": "auto"
|
||
}
|
||
```
|
||
|
||
## Local build
|
||
|
||
```bash
|
||
docker build -f router/Dockerfile -t ai-router-gateway:local .
|
||
```
|
||
|
||
## CI
|
||
|
||
Gitea Actions: `.gitea/workflows/ci.yml` — build gateway image, sync config check.
|