Loyaly/Behavision
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 22 Wiki Activity

Behavision: face recognition for retail, edge to head office

Browse Source 
Five components that ship as one product:

- behavision/  the recognition engine. RTSP ingest, YuNet detection, IoU
               tracking, ArcFace embeddings, a FAISS/SQLite gallery, and a
               FastAPI dashboard. Identity is decided once per TRACK from an
               average of at least three embeddings, never per frame.
- agent/       the Go edge agent: supervises the engine, holds a durable
               spool, and drains it to MQTT. Nothing is acked before the
               broker confirms.
- desktop/     the shop PC application (Wails + React + tray).
- server/      the cloud API, MQTT consumer, reports and assistant.
- web/         platform.loyaly.ai, the head-office app, embedded in the
               server binary.

The gallery stores 512-float embeddings and timestamps - no images unless
`app.store_faces` is switched on. Those embeddings are biometric personal
data under GDPR and India's DPDP: template inversion reconstructs a
recognisable face from an ArcFace vector, so data/behavision.db is treated
as a biometric database and DELETE /api/visitors/{id} is a real erasure.

CLAUDE.md carries the reasoning behind every non-obvious decision here,
including the ones that were measured and the ones that were wrong first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HViLj9gYNRtSr7YVZmW5sn
This commit is contained in:
Suriyakumarvijayanayagam
2026-09-04 11:14:18 +05:30
commit dad04e8cda
216 changed files with 40473 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

171
server/internal/contract/contract.go Normal file
View File

@@ -0,0 +1,171 @@
// Package contract is the wire format between a store agent and the server.
//
// Written down rather than left to emerge from whichever struct happened to be
// convenient: the agent ships to customer sites and cannot be redeployed on
// demand, so the server must keep parsing what old agents send. Every field
// added here must be optional, and no field may change meaning.
//
// Topics
//
// bv/<client>.<site>/visit agent -> server, one recognised person
// bv/<client>.<site>/heartbeat agent -> server, "I am alive"
// bv/<client>.<site>/status agent -> server, health and pipeline stats
// bv/<client>.<site>/cmd/# server -> agent
//
// The prefix is the agent's MQTT username, which the broker enforces with
// `pattern write bv/%u/...`. A site therefore cannot publish under another
// site's prefix even if it tries, and the server does not have to trust the
// topic - it re-derives the tenant from the username it was published under.
package contract
import (
"errors"
"fmt"
"strings"
"time"
)
// EmbeddingDim is fixed by the encoder. A payload with any other length is
// rejected rather than stored: a wrong-length vector cannot be compared with
// anything and would sit in the gallery poisoning every future search.
const EmbeddingDim = 512
// Visit is one person seen at one site.
type Visit struct {
// EventID is generated by the agent and is the idempotency key. MQTT
// delivery is at-least-once by design, so a reconnect can redeliver; without
// this a store's footfall silently doubles, and footfall is the number the
// customer is paying for.
EventID string `json:"event_id"`
OccurredAt time.Time `json:"occurred_at"`
CameraID string `json:"camera_id"`
// IsNew is the agent's local verdict: did its own gallery already know this
// face. Advisory only — the server re-decides against the client-wide
// gallery, because a person new to this store may be known at another one.
IsNew bool `json:"is_new"`
Similarity float32 `json:"similarity"`
Quality float32 `json:"quality"`
// LocalVisitorID ties this event back to the identity in the agent's own
// SQLite, so a support question ("which local record is this?") is
// answerable without guessing.
LocalVisitorID int64 `json:"local_visitor_id,omitempty"`
// Embedding is biometric personal data. Optional: a site may be configured
// to keep templates local and send only counts.
Embedding []float32 `json:"embedding,omitempty"`
Model string `json:"model,omitempty"`
// ImageKey is an object-store KEY, never a URL. A stored URL is permanent
// and unrevocable; a key is presigned on read and expires.
ImageKey string `json:"image_key,omitempty"`
Attributes map[string]any `json:"attributes,omitempty"`
}
// Validate rejects what cannot be stored meaningfully.
//
// Returns two kinds of failure deliberately: a permanent one means the message
// will never become valid, so the consumer must drop it rather than retry
// forever — one bad message at the head of a queue must not stop every good one
// behind it.
func (v *Visit) Validate() error {
if strings.TrimSpace(v.EventID) == "" {
return fmt.Errorf("%w: event_id is required (idempotency key)", ErrPermanent)
}
if len(v.EventID) > 128 {
return fmt.Errorf("%w: event_id too long", ErrPermanent)
}
if v.OccurredAt.IsZero() {
return fmt.Errorf("%w: occurred_at is required", ErrPermanent)
}
// A clock skewed years into the future would park a visit at the top of
// every "recent" report forever. Reject rather than clamp: silently moving
// a timestamp makes the footfall report a lie that looks fine.
if v.OccurredAt.After(time.Now().Add(24 * time.Hour)) {
return fmt.Errorf("%w: occurred_at is more than a day in the future "+
"(%s) - check the site's clock", ErrPermanent, v.OccurredAt)
}
if len(v.Embedding) > 0 {
if len(v.Embedding) != EmbeddingDim {
return fmt.Errorf("%w: embedding has %d dimensions, want %d",
ErrPermanent, len(v.Embedding), EmbeddingDim)
}
if v.Model == "" {
// Vectors from different encoders occupy different spaces and must
// never be compared. An untagged one cannot be safely stored.
return fmt.Errorf("%w: embedding sent without a model tag", ErrPermanent)
}
}
return nil
}
// Heartbeat says a site is alive.
//
// Without it "this site is switched off" and "this shop had no customers" are
// the same row of zeroes on the customer's report, and only one of them is
// something to act on.
type Heartbeat struct {
SentAt time.Time `json:"sent_at"`
AgentVersion string `json:"agent_version,omitempty"`
EngineVersion string `json:"engine_version,omitempty"`
RecognitionModel string `json:"recognition_model,omitempty"`
Cameras map[string]bool `json:"cameras,omitempty"`
// Queued and Dropped come from the agent's spool. Dropped is non-zero only
// when a site was offline long enough to overflow its queue, which means
// that site genuinely lost footfall - it must be visible, not inferred.
Queued int `json:"queued,omitempty"`
Dropped uint64 `json:"dropped,omitempty"`
// FractionBelowGate is the share of faces this site's cameras saw that fell
// under the enrolment gate — the number that decides whether a footfall
// figure can be believed at all. It travels on the heartbeat rather than on
// each visit because it describes the SITE, and because the visits it is
// about are precisely the ones that never became visits.
//
// Optional: an old agent sends nothing and the server keeps the last value
// it had rather than recording a perfect zero it was never told.
FractionBelowGate float32 `json:"fraction_below_gate,omitempty"`
}
var (
// ErrPermanent means the message will never be valid. Drop it.
ErrPermanent = errors.New("permanent")
// ErrTransient means try again later.
ErrTransient = errors.New("transient")
)
// Topic is a parsed agent topic.
type Topic struct {
Username string // "<client>.<site>", as the broker authenticated it
Client string
Site string
Kind string // visit | heartbeat | status | cmd
Rest string
}
// ParseTopic splits bv/<client>.<site>/<kind>[/...].
func ParseTopic(topic string) (Topic, error) {
parts := strings.Split(strings.Trim(topic, "/"), "/")
if len(parts) < 3 || parts[0] != "bv" {
return Topic{}, fmt.Errorf("%w: unexpected topic %q", ErrPermanent, topic)
}
user := parts[1]
// Exactly one dot: "acme.store1". Splitting on the first dot would let
// "acme.store.1" through as client "acme", site "store.1", which is a
// different site than the one the broker authenticated.
dot := strings.Split(user, ".")
if len(dot) != 2 || dot[0] == "" || dot[1] == "" {
return Topic{}, fmt.Errorf("%w: topic %q has no <client>.<site> segment",
ErrPermanent, topic)
}
return Topic{
Username: user,
Client: dot[0],
Site: dot[1],
Kind: parts[2],
Rest: strings.Join(parts[3:], "/"),
}, nil
}

126
server/internal/contract/contract_test.go Normal file
View File

@@ -0,0 +1,126 @@
package contract
import (
"errors"
"strings"
"testing"
"time"
)
func validVisit() Visit {
return Visit{EventID: "evt-1", OccurredAt: time.Now(), CameraID: "entrance"}
}
func TestTopicMustCarryClientAndSite(t *testing.T) {
got, err := ParseTopic("bv/acme.store1/visit")
if err != nil {
t.Fatal(err)
}
if got.Client != "acme" || got.Site != "store1" || got.Kind != "visit" {
t.Fatalf("%+v", got)
}
if got.Username != "acme.store1" {
t.Fatalf("username %q must match what the broker authenticated", got.Username)
}
}
func TestTopicWithExtraDotsIsRejected(t *testing.T) {
// Splitting on the FIRST dot would read "acme.store.1" as client "acme",
// site "store.1" - a different site than the broker authenticated, and a
// way to write rows against a tenant you do not own.
for _, topic := range []string{
"bv/acme.store.1/visit",
"bv/acme/visit",
"bv/.store1/visit",
"bv/acme./visit",
"other/acme.store1/visit",
"bv/acme.store1",
} {
if _, err := ParseTopic(topic); err == nil {
t.Errorf("%q was accepted", topic)
}
}
}
func TestCommandSubtopicSurvivesParsing(t *testing.T) {
got, err := ParseTopic("bv/acme.store1/cmd/reload/now")
if err != nil {
t.Fatal(err)
}
if got.Kind != "cmd" || got.Rest != "reload/now" {
t.Fatalf("%+v", got)
}
}
func TestEventIDIsRequired(t *testing.T) {
// It is the idempotency key. Without it an at-least-once redelivery
// silently doubles a store's footfall - the one number they pay for.
v := validVisit()
v.EventID = " "
assertPermanent(t, v.Validate(), "event_id")
}
func TestAFutureTimestampIsRejectedNotClamped(t *testing.T) {
// A site with a skewed clock would otherwise park a visit at the top of
// every "recent" report forever. Clamping it silently makes the report a
// lie that looks fine.
v := validVisit()
v.OccurredAt = time.Now().Add(72 * time.Hour)
assertPermanent(t, v.Validate(), "clock")
}
func TestSlightlyFutureIsToleratedForClockSkew(t *testing.T) {
v := validVisit()
v.OccurredAt = time.Now().Add(30 * time.Minute)
if err := v.Validate(); err != nil {
t.Fatalf("ordinary clock skew rejected: %v", err)
}
}
func TestWrongLengthEmbeddingIsRejected(t *testing.T) {
// A wrong-length vector cannot be compared with anything and would sit in
// the gallery poisoning every future search.
v := validVisit()
v.Model = "w600k_r50.onnx"
v.Embedding = make([]float32, 128)
assertPermanent(t, v.Validate(), "dimensions")
}
func TestEmbeddingWithoutAModelTagIsRejected(t *testing.T) {
// Vectors from different encoders occupy different spaces. An untagged one
// cannot be stored safely because nothing later can tell what it is.
v := validVisit()
v.Embedding = make([]float32, EmbeddingDim)
assertPermanent(t, v.Validate(), "model tag")
}
func TestAVisitWithNoEmbeddingIsValid(t *testing.T) {
// A site may be configured to keep templates local and send only counts.
v := validVisit()
if err := v.Validate(); err != nil {
t.Fatalf("counts-only visit rejected: %v", err)
}
}
func TestValidationFailuresArePermanentSoTheQueueDrains(t *testing.T) {
// If a malformed message were retried forever, one bad payload at the head
// would stop every good one behind it - the same failure the agent's spool
// quarantine exists to prevent.
v := Visit{}
if !errors.Is(v.Validate(), ErrPermanent) {
t.Fatal("a malformed visit was not marked permanent")
}
}
func assertPermanent(t *testing.T, err error, want string) {
t.Helper()
if err == nil {
t.Fatalf("expected an error mentioning %q", want)
}
if !errors.Is(err, ErrPermanent) {
t.Fatalf("error is not permanent: %v", err)
}
if !strings.Contains(err.Error(), want) {
t.Fatalf("error %q does not mention %q", err, want)
}
}