Suriyakumarvijayanayagam
dd3331ee9d
511 lines extracted from the code at release 0.4.1 / schema 013, and accurate for that point. The repository is nine releases and a schema past it. A stale document that states its own version reads as current to anyone skimming, which is the same failure this project keeps catching elsewhere: wrong in a way nobody can detect. So the top now lists what it predates by name - the motion gate, Gallery.health, tenantOnly, the password endpoint, customers and merge, the admin drill-down, sales and dashboard, migration 014, the macOS build - and points at API.md and CLAUDE.md, which are kept current. No credential values in it; the matches for password/secret/token are environment variable NAMES and package paths describing where secrets live, which is what a dossier should say. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KGcjxF1cNLcuwc3DAPcnfj
522 lines
36 KiB
Markdown
522 lines
36 KiB
Markdown
# Behavision — Technical Dossier
|
||
|
||
Everything below is extracted from the repository as of release 0.4.1 (engine 1.1.0, schema at migration 013). File paths, function names, thresholds, topics, ports and table definitions are the real ones. Where something lives outside the repository (the production host's proxy and container configuration) it is stated as such rather than invented.
|
||
|
||
> **Snapshot, not a live document.** Written against release 0.4.1 / schema
|
||
> 013, and the repository is past that. It predates at least: the engine's
|
||
> motion gate and one-thread detector (CPU 214% → 16%), `Gallery.health` and
|
||
> the stalled-camera state, the `tenantOnly` guard, `POST /api/auth/password`,
|
||
> `POST /api/customers` and the visitor merge, the admin console drill-down,
|
||
> `GET /api/sales` and `/api/dashboard/summary`, migration 014, and the macOS
|
||
> desktop build. Everything it *does* describe was extracted from the code and
|
||
> was true then; nothing here was invented. For the current surface read
|
||
> `API.md`, which is kept up to date, and `CLAUDE.md` for the decisions.
|
||
|
||
Companion documents: `API.md` (every route with request/response shapes), `docs/openapi.yaml` (generated), `docs/Behavision-Architecture.html` (diagrams).
|
||
|
||
---
|
||
|
||
## 1. System architecture
|
||
|
||
### 1.1 Repository layout
|
||
|
||
```
|
||
behavision/ Recognition engine (Python 3.10+)
|
||
__main__.py CLI: run | enroll | setup-models | calibrate | paths
|
||
api.py FastAPI on 127.0.0.1:8010 — dashboard, cameras, stream, stats
|
||
capture.py VideoSource: RTSP capture thread, latest-frame slot, probe_source
|
||
detection.py FaceDetector (YuNet), Detection
|
||
geometry.py umeyama, align_face, iou, clip_box
|
||
recognition.py ArcFaceEncoder (ONNX Runtime), face_quality
|
||
tracking.py Track, IouTracker
|
||
engine.py Engine, CameraWorker, PipelineStats — the per-camera pipeline
|
||
attributes.py AttributeEstimator (gender/age/emotion), aggregate
|
||
commission.py CommissionRun — placement check verdicts
|
||
gallery/store.py IdentityStore — SQLite (identities, embeddings, sightings)
|
||
gallery/index.py VectorIndex — FAISS IndexIDMap2(IndexFlatIP) / numpy fallback
|
||
gallery/service.py Gallery — resolve, enroll, reinforce, merge, duplicates
|
||
events.py EventBus + LogSink / WebhookSink / EmailSink
|
||
cameras.py CameraStore (cameras.json), protect/unprotect (DPAPI)
|
||
config.py pydantic Config, ${ENV} expansion, RTSP URL building
|
||
paths.py install_root / state_root / config_path resolution
|
||
static/dashboard.html Engine's own dashboard (no build step)
|
||
config/default.yaml Engine config (thresholds, cameras via ${ENV})
|
||
tests/ Engine tests (204), dependency-light: no camera, no models
|
||
|
||
agent/ Shop-PC agent (Go 1.22, module github.com/loyaly/behavision-agent)
|
||
main.go Headless agent binary
|
||
cmd/behavision-setup/ Installer: venv, wheel, models, config, smoke test, demo bundle
|
||
cmd/behavision-demo-pack/ Seals a camera list (AES-256-GCM) — build machine only
|
||
pkg/spool/ Durable queue: one file per event, bounded, ack by delete
|
||
pkg/mqtt/ paho adapter (client.go) + Pump + Waker (pump.go)
|
||
pkg/bridge/ Loopback webhook the engine posts to; derives event_id; queues
|
||
pkg/engine/ Supervisor (start/stop/restart/backoff), Health, ChildEnv
|
||
pkg/cameras/ Syncer: pull desired cameras, adopt local, run checks
|
||
pkg/enrol/ Redeem an installation code
|
||
pkg/config/ agent.json with DPAPI-protected secrets
|
||
pkg/paths/ StateRoot / InstallRoot — mirrors behavision/paths.py
|
||
pkg/demo/ Sealed bundle: NewCode, Seal, Open
|
||
|
||
desktop/ Shop-PC app (Wails v2.9.2, Go + React)
|
||
main.go wails.Run, HideWindowOnClose, tray start/stop
|
||
app.go Methods bound to the frontend; owns Supervisor, Bridge, Pump, Syncer
|
||
tray.go / icons.go fyne.io/systray; ICO rendered at runtime on Windows
|
||
stream_proxy.go Loopback relay for camera MJPEG (credential never in the page)
|
||
internal/local/ Client for the engine on 127.0.0.1:8010
|
||
internal/cloud/ Client for the platform API (sessions, refresh, images)
|
||
frontend/ React + Vite; src/bridge.js calls window.go.main.App.*
|
||
|
||
server/ Platform (Go, module github.com/loyaly/behavision-server)
|
||
cmd/behavision-server/main.go One binary: migrate → store → hub → ingest → API → web
|
||
Dockerfile Two-stage; static binary on alpine; EXPOSE 8080
|
||
migrations/001..013_*.sql go:embed'ed; applied at boot under an advisory lock
|
||
internal/api/ HTTP handlers, middleware, Hub (SSE doorbell), LiveHub (relay)
|
||
internal/store/ PostgreSQL access (pgx) — every query tenant-scoped
|
||
internal/ingest/ MQTT consumer: topic → site → visit/heartbeat → store
|
||
internal/auth/ Passwords (bcrypt 12), tokens, codes, Principal + role checks
|
||
internal/secret/ secret.Box — AES-256-GCM with AAD
|
||
internal/blob/ S3-compatible object storage (presign, private ACL check)
|
||
internal/assistant/ Claude tool loop; tools.go has no LLM import
|
||
internal/contract/ The MQTT wire contract: Visit, Heartbeat, ParseTopic
|
||
internal/migrate/ Migration runner (checksums, numeric order, baseline)
|
||
internal/provision/ CLI: provision key|client|site|user|token
|
||
internal/web/ go:embed of the built React console (web/dist → here)
|
||
|
||
web/ Head-office console (React + Vite); outDir → server/internal/web/dist
|
||
shared/cameraMakes.js Camera make → RTSP path table, imported by web AND desktop
|
||
installer/ build.ps1 (PyInstaller path), behavision.iss, INSTALL.txt, LAN launcher
|
||
run-local.sh Whole platform locally: Postgres + Mosquitto in Docker, server as binary
|
||
```
|
||
|
||
### 1.2 Processes and where they run
|
||
|
||
| Process | Language | Runs on | Listens | Talks to |
|
||
|---|---|---|---|---|
|
||
| Recognition engine | Python | shop PC | `127.0.0.1:8010` (Basic auth, generated) | cameras (RTSP), agent webhook (loopback) |
|
||
| Agent (inside the desktop app, or headless) | Go | shop PC | loopback webhook, port 0 | engine API, Mosquitto (TLS 8883), platform API (HTTPS) |
|
||
| Shop app | Go + webview | shop PC | loopback relay, port 0 | engine API, platform API |
|
||
| Mosquitto | C | cloud | `8883` TLS (agents), `1883` internal (server) | — |
|
||
| behavision-server | Go | cloud | `8080` (behind proxy) | PostgreSQL, Mosquitto (subscriber), object storage (optional), Anthropic API (optional) |
|
||
| PostgreSQL + pgvector | C | cloud | `5432` internal | — |
|
||
| Head-office console | React | browser | — | platform API |
|
||
| Mobile app | — | phone | — | platform API |
|
||
|
||
### 1.3 Network, domains, TLS
|
||
|
||
| Endpoint | Purpose | TLS |
|
||
|---|---|---|
|
||
| `https://platform.loyaly.ai` | Head-office console + API (`/api/*`) | Terminated at the reverse proxy (Traefik); the server listens plain HTTP on `LISTEN_ADDR` (default `:8080`) |
|
||
| `https://mcp.loyaly.ai/api/*` | Same API, the hostname the shop app defaults to (`BEHAVISION_CLOUD`) | Proxy |
|
||
| `tls://mcp.loyaly.ai:8883` | MQTT for agents (`AGENT_MQTT_URL` default) | Mosquitto's own listener; certificate must carry `DNS:mcp.loyaly.ai`; agents pin the issuing CA (`AGENT_CA_FILE` delivered at enrolment) |
|
||
| `tcp://behavision-mqtt:1883` | Server ↔ Mosquitto, internal network only (`MQTT_URL` default) | Plaintext on a private network |
|
||
|
||
Trust rules enforced in code:
|
||
- `X-Forwarded-For` is trusted for the login throttle **only because** nothing reaches the server port except through the proxy (`api/throttle.go`).
|
||
- The agent refuses `tcp://` to any non-loopback host unless `BEHAVISION_ALLOW_PLAINTEXT_MQTT=1` (`agent/pkg/mqtt/client.go`).
|
||
- No inbound route to a shop PC is ever required: agent → broker, agent → API, app → API are all outbound.
|
||
|
||
### 1.4 Server configuration (environment)
|
||
|
||
| Variable | Default | Purpose |
|
||
|---|---|---|
|
||
| `DATABASE_URL` | required | PostgreSQL DSN |
|
||
| `LISTEN_ADDR` | `:8080` | HTTP listener (behind proxy) |
|
||
| `MQTT_URL` / `MQTT_USERNAME` / `MQTT_PASSWORD` | `tcp://behavision-mqtt:1883` | Server's subscriber credential |
|
||
| `AGENT_MQTT_URL` | `tls://mcp.loyaly.ai:8883` | Broker URL handed to a PC at enrolment |
|
||
| `AGENT_CA_FILE` | — | CA PEM handed to a PC at enrolment (pinned) |
|
||
| `AGENT_MODELS_FILE` | — | Model manifest handed at enrolment |
|
||
| `BEHAVISION_SECRET_KEY` | — | 32-byte key for `secret.Box`; enrolment and camera passwords need it |
|
||
| `DO_SPACES_*` (`ENDPOINT`, `REGION`, `BUCKET`, `ACCESS_KEY`, `SECRET_KEY`, `PREFIX`) | prefix `behavision/v2` | Optional object storage; absent = images stored in Postgres |
|
||
| `ANTHROPIC_API_KEY` / `ANTHROPIC_WORKSPACE_ID` / `BEHAVISION_ASSISTANT_MODEL` | model `claude-sonnet-5` | Assistant; absent = `501 assistant_off` |
|
||
| `BEHAVISION_SKIP_MIGRATE` | — | Escape hatch; default applies migrations at boot |
|
||
|
||
### 1.5 Container / deployment shape
|
||
|
||
The repository ships `server/Dockerfile` (golang:1.25-alpine build → alpine:3.20 runtime, static binary, `EXPOSE 8080`, non-root user) and `run-local.sh`, which stands the whole platform up locally: `bv-pg` (pgvector/pgvector:pg16, port 55432), `bv-mqtt` (eclipse-mosquitto:2, port 51883, `passwd` + `acl` mounted), and the server as a local binary on 8088 with the console embedded.
|
||
|
||
Production host configuration (proxy routes, compose/unit files, certificate issuance) is **outside the repository**. What the code requires of it: a proxy terminating TLS for `platform.loyaly.ai` and forwarding to `LISTEN_ADDR`; Mosquitto with a TLS listener on 8883 whose certificate names `mcp.loyaly.ai`, a `passwd` file the `provision site` command adds to, and an ACL of the form `pattern write bv/%u/#`; PostgreSQL with the `vector` extension.
|
||
|
||
---
|
||
|
||
## 2. Recognition engine internals
|
||
|
||
### 2.1 Pipeline, function by function
|
||
|
||
```
|
||
RTSP ──▶ capture.VideoSource.run() thread per camera; cv2.VideoCapture(CAP_FFMPEG)
|
||
│ OPENCV_FFMPEG_CAPTURE_OPTIONS = rtsp_transport;tcp | stimeout;5000000 |
|
||
│ fflags;nobuffer | flags;low_delay | max_delay;200000
|
||
│ downscale to max_width (1280) with INTER_AREA
|
||
│ latest frame + timestamp in a lock-protected slot
|
||
▼
|
||
engine.CameraWorker.run() thread per camera; takes source.latest_since(ts)
|
||
│
|
||
├─ detection.FaceDetector.detect(frame) cv2.FaceDetectorYN (YuNet 2023mar)
|
||
│ score_threshold 0.82 · nms 0.3 · min_face_px 48 · max_faces 20
|
||
│ → Detection(box, kps[5], score)
|
||
│
|
||
├─ recognition.face_quality(frame, box, kps) weighted: sharpness .35 · size .25 · brightness .15 · frontality .25
|
||
│
|
||
├─ tracking.IouTracker.update(dets, ts) greedy IoU association, iou_threshold 0.3, max_misses 25
|
||
│ → active Track[], ended Track[] one Track == one person on camera
|
||
│
|
||
├─ for each active track, _should_identify(): hits ≥ 4 · quality ≥ min_quality_to_encode (0.35)
|
||
│ ≤ max_id_attempts (8) · spaced id_retry_interval_seconds (0.5)
|
||
│
|
||
├─ _identify(track, frame):
|
||
│ geometry.align_face(frame, kps) Umeyama similarity transform → 112×112 BGR chip
|
||
│ recognition.ArcFaceEncoder.encode() BGR→RGB, (x−127.5)/127.5, NCHW float32, L2-normalised 512-d
|
||
│ track.emb_sum += e; when emb_count ≥ min_embeddings_for_id (3): mean → normalise
|
||
│ gallery.Gallery.resolve(mean, quality, rcfg) → Resolution(kind, identity, similarity)
|
||
│
|
||
├─ Resolution.kind:
|
||
│ known sim ≥ match_threshold (0.42) → person.seen (+ reinforce if 0.32 ≤ sim < 0.55, q ≥ gate, < 5 stored)
|
||
│ ambiguous 0.32 ≤ sim < 0.42 → wait; retry on a later frame
|
||
│ new sim < enroll_threshold (0.32) → Gallery.enroll → "Visitor N" · person.new
|
||
│ skipped quality < min_enroll_quality (0.65) → counted as rejected_quality
|
||
│
|
||
├─ attributes.AttributeEstimator.estimate() genderage.onnx on a loose 1.5× crop; FER+ on the chip;
|
||
│ medianed over the track (attributes.aggregate)
|
||
│
|
||
├─ _finish_track(ended) PipelineStats.record(outcome) — exactly once per track
|
||
│
|
||
└─ _remember_tracks(active) boxes + labels for the live picture (no encode)
|
||
|
||
Live picture: CameraWorker.latest_jpeg_since(ts) — freshest CAPTURED frame + last boxes, encoded on demand.
|
||
Events: events.EventBus → LogSink · WebhookSink (→ agent bridge) · EmailSink
|
||
```
|
||
|
||
### 2.2 Models (`recognition.MODEL_CANDIDATES`, first loadable wins)
|
||
|
||
| Order | File | Role |
|
||
|---|---|---|
|
||
| 1–2 | `adaface_ir101.onnx`, `adaface_ir50.onnx` | wired, optional |
|
||
| **3** | **`w600k_r50.onnx`** (166 MB) | **in use** — IJB-C 97.25; same-person p05 0.719 on the office camera |
|
||
| 4 | `arcface_int8.onnx` | optional |
|
||
| 5 | `w600k_mbf.onnx` (13 MB) | always loads; MobileFaceNet fallback (95.02) |
|
||
| 6 | `arcface.onnx` (r100, 249 MB) | optional |
|
||
| — | `face_detection_yunet_2023mar.onnx` | detector |
|
||
| — | `genderage.onnx` (InsightFace buffalo_l) | attributes |
|
||
|
||
Every stored embedding is tagged with the model name; `IdentityStore.all_embeddings(model)` loads only same-model vectors into the index.
|
||
|
||
### 2.3 Local gallery
|
||
|
||
`gallery/store.py` — SQLite (WAL), single source of truth:
|
||
```
|
||
identities(id, label, kind auto|named, created_at, sighting_count)
|
||
embeddings(id, identity_id, model, vector BLOB, quality, created_at)
|
||
sightings(id, identity_id, camera_id, similarity, at)
|
||
```
|
||
`gallery/index.py` — `VectorIndex` over FAISS `IndexIDMap2(IndexFlatIP)` (exact inner product = cosine on L2-normalised vectors), rebuilt from SQLite at boot, −1 ids filtered, identical numpy fallback. Measured: 1k → 0.27 ms, 10k → 2.24 ms, 100k → 21.9 ms.
|
||
|
||
`gallery/service.py` — `Gallery.resolve` (three zones), `enroll`, `reinforce_identity` (refuses a view whose nearest neighbour is another identity), `merge_identities` (one transaction; human name outranks "Visitor N"; `sighting_count` recomputed; trimmed to 5 by quality), `duplicate_candidates` (k-NN across identities, O(n·k)).
|
||
|
||
### 2.4 What leaves the engine
|
||
|
||
`WebhookSink` POSTs each `person.seen` / `person.new` to the agent's loopback bridge with `identity_id`, `label`, `similarity`, `quality`, attributes, and optionally `image_path` (only when `app.store_faces: true`). The bridge fetches the identity's **best** stored embedding once per identity via `GET /api/identities/{id}/embedding`. `person.missed`, `camera.up/down` are diagnostics and never become visits.
|
||
|
||
---
|
||
|
||
## 3. Backend internals
|
||
|
||
### 3.1 Request path
|
||
|
||
```
|
||
proxy (TLS) ─▶ net/http mux (Go 1.22 patterns, method + path)
|
||
│
|
||
├─ s.authed(h) Bearer token → SHA-256 → sessions row → Principal{UserID, ClientID, Role}
|
||
│ token_expired vs unauthorized distinguished; last_used_at touched
|
||
├─ s.adminOnly(h) Principal.Role == "admin" AND ClientID == "" (both) → else 404
|
||
├─ s.agentAuthed(h) Agent token (hashed) → AgentPrincipal{ClientID, Client slug, SiteID, AgentID}
|
||
└─ no wrapper login, refresh, invitation preview, register, enrol
|
||
│
|
||
▼
|
||
handlers_*.go decode (unknown fields rejected) → validate → Store call → writeJSON
|
||
│ every Store call receives p.ClientID from the session, never the body
|
||
▼
|
||
store/*.go (pgx) SQL with client_id in every WHERE / INSERT
|
||
```
|
||
|
||
Login throttle (`api/throttle.go`): per-account 10 failures / 15 min and per-IP 60, in memory, pruned on read; success clears both. Unknown address is verified against `auth.DummyHash` so timing matches a wrong password.
|
||
|
||
### 3.2 Handler areas → store methods
|
||
|
||
| Area (file) | Routes | Store surface |
|
||
|---|---|---|
|
||
| `handlers_auth.go`, `handlers_sessions.go` | login, refresh, logout, me, sessions list/revoke | `UserByEmail`, `CreateSession`, `SessionByAccessHash`, `RotateSession`, `RevokeSession(s)` |
|
||
| `handlers_team.go` | team, members, password reset, invitations, register | `Team`, `UpdateTeamMember`, `CreateMember`, `ResetMemberPassword`, `CreateInvitation`, `RedeemInvitation`, `OwnerCount` |
|
||
| `handlers_admin.go` | admin/clients | `ListClients`, `CreateClientWithOwner` (one transaction) |
|
||
| `handlers_arrivals.go`, `hub.go` | visits, visits/stream | `Arrivals` (keyset by `seq`), `Hub.Notify` doorbell → SSE |
|
||
| `handlers_people.go` | visitors, history, profile, purchases, erasure | `SearchVisitors`, `VisitorHistory`, `SaveProfile`, `RecordPurchase`, `ForgetVisitor` |
|
||
| `handlers_images.go`, `handlers_faces.go` | visitor image, face bytes | `VisitorImageKey`, `FaceImage`; `imageFor(key)` decides presigned vs `auth:true` |
|
||
| `handlers_cameras.go`, `handlers_snapshots.go` | cameras CRUD, snapshot | `Cameras`, `CreateCamera`, `UpdateCamera`, `DeleteCamera` (tombstone), `Snapshot` |
|
||
| `handlers_checks.go` | camera check, site check | `RequestCheck`, `ClaimChecks`, `ReleaseStaleChecks`, `RecordCheck`, `SiteCheck` |
|
||
| `handlers_live.go`, `live.go` | cameras/{id}/live, agent live | `LiveHub` — one-slot buffer per viewer, on-demand upload |
|
||
| `handlers_reports.go` | footfall, conversion | `Footfall`, `Conversion` — unique vs visits, first-ever "new", single currency |
|
||
| `handlers_enrolment.go`, `handlers_agent.go` | enrol, agent cameras/checks/faces/upload-url | `RedeemEnrolment` (single-use via UPDATE), `AgentCameras`, `AgentReport`, `PutFace`, `UploadTarget` |
|
||
| `handlers_assistant.go` | assistant | `assistant.Client.Ask` with the Principal passed at the call site |
|
||
|
||
### 3.3 Server-side recognition (`store/store.go`, `RecordVisit`)
|
||
|
||
```
|
||
similarity = 1 - (embedding <=> $1::vector) -- pgvector cosine distance
|
||
ORDER BY embedding <=> $1::vector LIMIT 1 -- within client_id, same model
|
||
sim ≥ 0.42 → known visitor; reinforce if 0.32 ≤ sim < 0.55 AND quality ≥ floor AND < 5 stored
|
||
sim < 0.42 → new visitor: clients.visitor_seq += 1 RETURNING (row-locks the client), label "Visitor N"
|
||
INSERT visits ... ON CONFLICT (client_id, source_event_id) DO NOTHING -- idempotent
|
||
```
|
||
|
||
### 3.4 Single-process composition (`cmd/behavision-server/main.go`)
|
||
|
||
```
|
||
migrate.Apply(embedded FS) → store.Open → hub := api.NewHub()
|
||
ingest.Consumer{Store, Notify: hub.Notify} ← paho client, SetOrderMatters(true), subscribed bv/+/+
|
||
api.New(Store, Hub, LiveHub, Blob?, Assistant?) → web.Handler (embedded dist; /api/ keeps JSON 404)
|
||
http.Server{ReadTimeout, IdleTimeout, WriteTimeout: 0} -- zero: SSE streams must outlive any write deadline
|
||
```
|
||
|
||
---
|
||
|
||
## 4. MQTT architecture
|
||
|
||
### 4.1 Identity and topics
|
||
|
||
Broker username = `<client-slug>.<site-slug>` (e.g. `tenext-retail.chennai`). ACL: `pattern write bv/%u/#` — a site physically cannot publish under another site's prefix.
|
||
|
||
```
|
||
bv/<client>.<site>/visit Visit payload QoS 1 spooled, acked per event
|
||
bv/<client>.<site>/heartbeat Heartbeat payload QoS 1 never spooled — only meaningful now
|
||
bv/<client>.<site>/status reserved
|
||
bv/<client>.<site>/cmd/... reserved (server → site)
|
||
```
|
||
|
||
`contract.ParseTopic` → `Topic{Username, Client, Site, Kind, Rest}`; `Kind ∉ {visit, heartbeat, status, cmd}` is dropped as permanent.
|
||
|
||
### 4.2 Payloads (`server/internal/contract/contract.go`)
|
||
|
||
```json
|
||
// visit
|
||
{ "event_id": "tenext-retail.chennai|cam2|7|1757580000", // <site>|<camera>|<identity>|<unix second> — derived, never random
|
||
"occurred_at": "2026-09-11T05:20:00Z", "camera_id": "cam2",
|
||
"is_new": false, "similarity": 0.61, "quality": 0.70,
|
||
"local_visitor_id": 7, "embedding": [512 floats], "model": "w600k_r50",
|
||
"image_key": "behavision/v2/tenext-retail/chennai/2026/09/11/…jpg", // or "db:<uuid>", or absent
|
||
"attributes": { "gender": "Male", "age": 32, "emotion": "neutral" } }
|
||
|
||
// heartbeat
|
||
{ "sent_at": "…", "agent_version": "0.4.1", "engine_version": "1.1.0", "recognition_model": "w600k_r50",
|
||
"cameras": { "cam1": true, "cam2": true }, "queued": 0, "dropped": 0, "fraction_below_gate": 0.47 }
|
||
```
|
||
|
||
`Visit.Validate()` (permanent errors): `event_id` required ≤128; `occurred_at` required and not >24 h in the future; embedding must be exactly 512 and carry `model`.
|
||
|
||
### 4.3 Delivery semantics
|
||
|
||
| Stage | Component | Guarantee |
|
||
|---|---|---|
|
||
| Engine → agent | `WebhookSink` → `bridge.Bridge.Handle` | loopback HTTP; `event_id` derived from `<site>|<camera>|<identity>|<second>` (sighting cooldown is 30 s, so one person/camera cannot share a second) |
|
||
| Append | `spool.Spool.Append` | one file per event, fsync, bounded (`SpoolMax`, default 50,000); on overflow drops oldest and counts `Dropped()`; corrupt entry quarantined, not retried |
|
||
| Wake | `mqtt.Waker.Wake` **after** the append | a wake before durability is a drain that finds nothing |
|
||
| Publish | `mqtt.Pump.Run` → `Client.Publish` | QoS 1, `CleanSession(true)`, publish bounded by a timeout as well as context (half-open TCP otherwise stalls forever); **a failed publish stops the batch** (ordering per visitor) |
|
||
| Ack | `Spool.Ack(seq)` on PUBACK | per event, never per batch; file deleted only now |
|
||
| Consume | `ingest.Consumer.Handle` | paho `SetOrderMatters(true)`; permanent error → `drop()` + log (message is acked, never redelivered); transient error → returned → redelivered |
|
||
| Write | `store.RecordVisit` | `ON CONFLICT (client_id, source_event_id) DO NOTHING` — duplicates from at-least-once delivery are absorbed |
|
||
| Notify | `hub.Notify(clientID)` | only on a genuine insert; SSE streams re-query from their own cursor |
|
||
|
||
Backoff: reconnect 1 → 30 s exponential; supervisor backoff resets only after a run that stayed up 60 s. `describeStall` distinguishes *broker refused the credential* (TCP opens, connect never completes) from *broker unreachable*.
|
||
|
||
### 4.4 Failure paths
|
||
|
||
| Failure | Behaviour |
|
||
|---|---|
|
||
| Internet down | spool grows on disk; heartbeats stop; head office shows site offline after 3 missed beats; on reconnect the backlog drains in order |
|
||
| Broker rejects credential | pump logs "reachable but not accepted — re-link this PC"; spool retained |
|
||
| Server down, broker up | broker holds nothing (clean session); agent's PUBACKs still arrive from the broker, so events are acked at the broker — the server's own subscription reconnects and Mosquitto delivers what it queued for the persistent server session |
|
||
| Duplicate delivery | absorbed by `source_event_id` uniqueness |
|
||
| Malformed event | dropped with a log line naming the site and reason; never blocks the queue |
|
||
| Site clock wrong | `occurred_at` > 24 h ahead rejected as permanent; feed ordering uses server `seq`, so a wrong clock cannot hide a visit |
|
||
|
||
---
|
||
|
||
## 5. Database schema (PostgreSQL + pgvector, migrations 001–013)
|
||
|
||
### 5.1 Tables
|
||
|
||
```
|
||
clients id PK · slug UQ (immutable, = MQTT prefix) · name · active · visitor_seq bigint
|
||
sites id PK · client_id FK · slug (immutable, per client) · name · timezone · address · active
|
||
agents id PK · client_id FK · site_id FK · mqtt_username UQ · mqtt_password_enc bytea (sealed)
|
||
api_token_hash bytea · agent_version · engine_version · recognition_model
|
||
last_heartbeat_at · last_event_at · fraction_below_gate · cameras_total/up · spool_queued/dropped
|
||
app_users id PK · client_id FK (NULL = platform admin) · email (lower(email) UQ globally, 007)
|
||
password_hash (bcrypt 12) · full_name · role owner|manager|staff|admin · active · last_login_at
|
||
sessions id PK · user_id FK · client_id FK · access_hash UQ · refresh_hash UQ (SHA-256)
|
||
access_expires_at · refresh_expires_at · revoked_at · device · last_used_at
|
||
invitations id PK · client_id FK · email · full_name · role · code_hash UQ · invited_by FK
|
||
expires_at · used_at · used_by FK · revoked_at
|
||
site_enrolment_tokens id PK · client_id FK · site_id FK · token_hash UQ · label · expires_at · used_at · created_by
|
||
visitors id PK · client_id FK · number bigint (per-client, immutable → "V-42") · label
|
||
first_seen_at · last_seen_at · visit_count · deleted_at
|
||
visitor_embeddings id PK · visitor_id FK · client_id FK · model · embedding vector(512) · quality · source_site_id FK
|
||
visitor_profiles id PK · visitor_id FK · client_id FK · full_name · phone · email · gender · date_of_birth · notes
|
||
consents id PK · visitor_id FK · client_id FK · scope · method · granted_at · revoked_at · evidence jsonb
|
||
visits id PK · client_id FK · site_id FK · visitor_id FK (nullable) · source_event_id (UQ per client)
|
||
occurred_at · received_at · camera_id text · is_new_visitor · similarity · quality
|
||
attributes jsonb · image_key · image_deleted_at · seq bigserial (feed cursor)
|
||
purchases id PK · client_id FK · site_id FK · visitor_id FK · visit_id FK · amount numeric(14,2)
|
||
currency char(3) · items jsonb · source · external_ref · recorded_by · occurred_at
|
||
site_cameras id PK · client_id FK · site_id FK · camera_id text (immutable, UQ per site) · label
|
||
host · port · path · username · password_enc bytea (sealed, aad = site_id) · max_width
|
||
tuning jsonb · enabled · revision · connected (nullable) · last_seen_at · snapshot_key
|
||
snapshot_at · deleted_at (tombstone) · check_kind · check_* (requested/started/finished/result/image_key)
|
||
camera_snapshots camera_id PK FK · client_id FK · site_id FK · image bytea · bytes · captured_at
|
||
visit_faces id PK · client_id FK · site_id FK · image bytea · bytes · captured_at (one survives per visitor)
|
||
audit_log id bigserial PK · client_id FK (SET NULL) · actor_id · actor_kind · action · entity · entity_id · detail jsonb · at
|
||
schema_migrations version · checksum · applied_at · baselined
|
||
```
|
||
|
||
### 5.2 Relationships
|
||
|
||
```
|
||
clients ─┬─< sites ─┬─< agents
|
||
│ ├─< site_cameras ──< camera_snapshots (1:1, PK = camera_id)
|
||
│ ├─< site_enrolment_tokens
|
||
│ ├─< visits
|
||
│ ├─< purchases
|
||
│ └─< visit_faces
|
||
├─< app_users ─┬─< sessions
|
||
│ └─< invitations (invited_by, used_by)
|
||
├─< visitors ─┬─< visitor_embeddings
|
||
│ ├─< visitor_profiles
|
||
│ ├─< consents
|
||
│ ├─< visits
|
||
│ └─< purchases
|
||
└─< audit_log (SET NULL)
|
||
|
||
visits ──< purchases (visit_id, SET NULL)
|
||
```
|
||
|
||
Every FK onto `clients` is `ON DELETE CASCADE` except `audit_log` (`SET NULL`). Every tenant-owned table carries `client_id` directly, so no query needs a join to enforce tenancy.
|
||
|
||
### 5.3 Invariants enforced in the database
|
||
|
||
- `007` — `lower(email)` globally unique; the migration refuses to apply while duplicates exist and names them.
|
||
- `012` — `visitors.number` per-client sequence from `clients.visitor_seq` (`UPDATE … RETURNING`, row-locked); unique on `(client_id, number)`.
|
||
- `013` — triggers refuse changes to `clients.slug`, `sites.slug`, `site_cameras.camera_id`, `visitors.number` (`BEFORE UPDATE OF … WHEN OLD IS DISTINCT FROM NEW`). Display names are deliberately not frozen.
|
||
- `004` — `visits.seq bigserial`; the arrivals cursor is `v1:<seq>` base64, opaque to clients.
|
||
- `sites_slug_format` — `^[a-z0-9][a-z0-9-]{1,30}[a-z0-9]$`.
|
||
|
||
---
|
||
|
||
## 6. API specification
|
||
|
||
Full request/response shapes: `API.md`. Machine-readable: `docs/openapi.yaml`.
|
||
|
||
### 6.1 Authentication and session lifecycle
|
||
|
||
```
|
||
POST /api/auth/login {email,password,device}
|
||
→ {access_token (12 h), refresh_token (30 d), expires_at, user{id,email,full_name,role,client_id,client_name}}
|
||
tokens: 256-bit random; only SHA-256 stored; bcrypt cost 12 verify; DummyHash for unknown addresses
|
||
POST /api/auth/refresh {refresh_token,device}
|
||
→ same shape; BOTH rotate; old refresh invalid immediately; client must serialise and persist before use
|
||
401 {error:"token_expired"} → refresh once and retry
|
||
401 {error:"bad_credentials"} / {error:"unauthorized"} → sign in
|
||
GET/DELETE /api/auth/sessions[/{id}] · POST /api/auth/sessions/revoke-others
|
||
```
|
||
|
||
### 6.2 Route families and least role
|
||
|
||
| Family | Routes | Least role |
|
||
|---|---|---|
|
||
| Auth | login, refresh, logout, me, sessions | none / authed |
|
||
| Joining | `GET /api/auth/invitation?code=`, `POST /api/auth/register` | none |
|
||
| Team | `GET /api/team` | authed (tenant users) |
|
||
| | `POST /api/team/members`, `POST /api/team/{id}/password`, `PATCH /api/team/{id}`, `/api/team/invitations*` | manager |
|
||
| Arrivals | `GET /api/visits`, `GET /api/visits/stream` (SSE) | authed |
|
||
| Customers | `GET /api/visitors`, `/history`, `/image`, `GET /api/faces/{id}` | authed |
|
||
| | `PUT /api/visitors/{id}/profile`, `POST /api/purchases` | staff |
|
||
| | `DELETE /api/visitors/{id}` (erasure) | manager |
|
||
| Shops & cameras | `GET /api/sites`, `/check`, `GET /api/cameras`, `/snapshot.jpg`, `/live` (SSE) | authed |
|
||
| | `POST /api/sites/{site}/cameras`, `PATCH`/`DELETE /api/cameras/{id}`, `POST /api/cameras/{id}/check`, `POST /api/sites/{site}/enrolment-code` | manager |
|
||
| Reports | `GET /api/reports/footfall`, `/conversion` | authed |
|
||
| Assistant | `POST /api/assistant` | authed |
|
||
| Admin | `GET`/`POST /api/admin/clients` | platform admin (role admin AND no client) |
|
||
| Agent | `POST /api/agent/enrol` (none), then `/api/agent/{cameras, checks, faces, upload-url, live, cameras/{c}/snapshot, cameras/{c}/live}` | agent token |
|
||
|
||
Identifiers: any `{id}` or `site` accepts a uuid **or** the human reference (`V-42`, `chennai`, `cam1`). Unknown reference in a path → 404; in a query filter → 400. Another tenant's data → 404, never 403.
|
||
|
||
### 6.3 Assistant tools (`internal/assistant/tools.go`)
|
||
|
||
`list_sites`, `site_health`, `footfall`, `conversion`, `find_customer`, `customer_history`, `check_camera` (manager+; refuses staff in the tool, not the prompt). No tool takes a tenant id; the Principal is bound at the call site. Business tools only — never `execute_sql`. Loop bounded at 8 iterations; text produced alongside a tool call is discarded; failing tools return results, not errors.
|
||
|
||
---
|
||
|
||
## 7. Deployment topology
|
||
|
||
```
|
||
INTERNET
|
||
│
|
||
┌───────────────┼───────────────────┐
|
||
│ HTTPS 443 │ │ TLS 8883
|
||
┌────────▼─────────┐ │ ┌────────▼─────────┐
|
||
│ Traefik │ │ │ Mosquitto │
|
||
│ platform.loyaly │ │ │ mcp.loyaly.ai │
|
||
│ mcp.loyaly.ai │ │ │ passwd + ACL │
|
||
│ TLS termination │ │ │ pattern write │
|
||
└────────┬─────────┘ │ │ bv/%u/# │
|
||
│ :8080 plain │ └────────┬─────────┘
|
||
┌────────▼───────────────────────┐ │ :1883 internal
|
||
│ behavision-server (one binary) │◄───────────┘ subscribe bv/+/+
|
||
│ ├ migrate (boot) │
|
||
│ ├ ingest consumer → Hub │
|
||
│ ├ API (48 routes) → SSE │
|
||
│ ├ LiveHub (camera relay) │
|
||
│ ├ web (embedded React) │
|
||
│ └ assistant (optional) │
|
||
└────────┬───────────────┬───────┘
|
||
│ │ presigned PUT/GET (optional)
|
||
┌────────▼─────────┐ ┌──▼──────────────────┐
|
||
│ PostgreSQL 16 │ │ Object storage │
|
||
│ + pgvector │ │ (S3-compatible) │
|
||
│ 17 tables │ │ private ACL │
|
||
└──────────────────┘ └─────────────────────┘
|
||
|
||
════════════════════════ trust boundary: no inbound route ════════════════════════
|
||
|
||
SHOP NETWORK (one per shop, behind NAT)
|
||
┌──────────────────────────────────────────────────────────┐
|
||
│ Cameras ──RTSP/TCP──▶ Engine :8010 ──webhook──▶ Agent │
|
||
│ │ SQLite │ spool │
|
||
│ │ FAISS │ │
|
||
│ Shop app ◄─── relay ───────┘ │
|
||
│ │
|
||
│ outbound only: agent ──TLS 8883──▶ Mosquitto │
|
||
│ agent ──HTTPS────▶ /api/agent/* │
|
||
│ app ──HTTPS────▶ /api/* │
|
||
└──────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
**Failure domains.** A shop PC failing affects one shop; its footfall queues locally and nothing else notices except the heartbeat. Mosquitto failing stops delivery for all shops but loses nothing (every event is on a shop's disk). The server failing stops the console and API; Mosquitto retains the server's subscription backlog. PostgreSQL is the single stateful component in the cloud tier.
|
||
|
||
**Data residency.** Video never leaves the shop. Face templates leave the shop only as 512-float vectors inside visit events, over TLS, to the tenant's own prefix. Photographs leave only when `store_faces` is enabled, via presigned upload to a private object, or into Postgres when no bucket is configured. Every image read at head office is audited.
|
||
|
||
**Encrypted paths.** Camera credentials: DPAPI on the shop PC, AES-256-GCM (aad = site) in Postgres, plaintext only inside the agent's process and on the LAN RTSP connection to the camera. Broker password: sealed in `agent.json`, sealed in `agents.mqtt_password_enc`, hashed in Mosquitto's `passwd`. Sessions: SHA-256 at rest. User passwords: bcrypt 12.
|
||
|
||
---
|
||
|
||
## 8. Measured
|
||
|
||
| Metric | Value | Source |
|
||
|---|---|---|
|
||
| Identity stability | 103 tracks → 7 people, 44 re-recognitions, 5 min | engine `/api/stats`, office cam2 |
|
||
| Same-person similarity | p05 0.719 (frontal webcam, 18,528 pairs) | `calibrate` |
|
||
| Gallery search | 0.27 / 2.24 / 21.9 ms at 1k / 10k / 100k | `VectorIndex` benchmark |
|
||
| Delivery under burst | 120 of 120 simultaneous visits | live broker + Postgres |
|
||
| Camera → head office | ~3 s | end-to-end run |
|
||
| Head-office live view | 13 fps, 259 KB/s, 0 duplicates | relay measurement |
|
||
| Shop-PC live picture | 14.0 pictures/s vs 15 fps camera; engine CPU 90% → 62% | before/after, cam2 sub-stream |
|
||
| Clean-machine install | 10/10 steps, both cameras connected | fresh container |
|
||
| Server test suite | < 10 s (bcrypt cost lowered for tests only) | `go test ./...` |
|