Loyaly/Behavision
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 22 Wiki Activity

Three states that looked like health from outside

Browse Source 
Audited the engine for what it does when something goes wrong rather than
when it goes right. Each of these left the process healthy, the dashboard
green and the product not working.

A gallery the running encoder cannot read. Embeddings are model-tagged, so
when the fallback chain fires every vector the previous encoder wrote goes
invisible: the shop keeps its customer list and recognises nobody on it,
enrolling each regular a second time. Footfall stays correct, which is why
nothing looks wrong. The only evidence was an INFO line reading 'gallery
ready: 0 embeddings (model w600k_mbf) across 21 identities' - a sentence
that states the disaster and calls it ready. Gallery.health now warns with
the count of PEOPLE lost, not vectors, and carries the same numbers to
/api/stats and /api/health, because a log line on a shop PC is read by
nobody. Proved against the real 87-embedding gallery.

Connected, and sending nothing. 'connected' meant the socket opened, so a
stream that went quiet kept it true while last_frame_age_s climbed and the
heartbeat told head office the camera was up. OpenCV breaks a blocked read
at 30s, but a camera trickling a frame every 20s never trips that and never
recovers. streaming/stalled are reported beside connected and the dashboard
says live/stalled/offline - three states because offline sends you to the
network and stalled says the camera is answering and sending nothing.

The 5-second RTSP timeout that never existed. stimeout;5000000 carried a
comment claiming it bounded a dead camera. Measured on OpenCV 4.11 /
FFmpeg 7.1 against a socket that accepts and then says nothing: 30.0s with
stimeout, 30.0s with timeout, 30.3s with no option at all - identical, so
it was never honoured. stimeout became timeout in FFmpeg 5.0 and neither
reaches the RTSP protocol through this path; the real bound is OpenCV's own
interrupt constant. Replaced by the _tcp_reachable pre-flight probe_source
already used, in code we own: 30.3s -> 0.00-2.02s, each naming its cause.
That matters beyond speed - the VideoCapture constructor is not
interruptible, so stop() could not cut it short and a camera removed from
head office left a daemon thread holding a socket for half a minute.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KGcjxF1cNLcuwc3DAPcnfj
This commit is contained in:
Suriyakumarvijayanayagam
2026-09-24 14:15:54 +05:30
parent 2e60fbb57a
commit f61da2eeed
8 changed files with 439 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
behavision/api.py
View File

@@ -167,8 +167,13 @@ def create_app(engine: Engine) -> FastAPI:
@app.get("/api/health")
def health() -> dict:
from .paths import describe
# A gallery the running encoder cannot read is the failure most
# worth catching from outside: the process is healthy, the cameras
# are up, and the shop recognises nobody it already knows.
stranded = engine.gallery.health["stranded"]
return {"status": "ok" if engine.started_at else "starting",
"recognition_model": engine.encoder.model_name,
"gallery_unreadable_embeddings": stranded,
# "where is my database" must be answerable from the API: the
# tray, the installer and support all need it, and installed
# it is not next to the code.

71
behavision/capture.py
View File

@@ -20,7 +20,26 @@ log = logging.getLogger(__name__)
# Set before OpenCV loads ffmpeg, which reads this once.
#
# rtsp_transport=tcp: UDP is the default and silently drops frames on lossy
# Wi-Fi. stimeout: a 5s socket timeout so a dead camera is noticed.
# Wi-Fi.
#
# The timeout here is NOT what bounds a dead camera, and the comment that
# once said it did was wrong. Measured against OpenCV 4.11 / FFmpeg 7.1 on a
# socket that accepts the connection and then says nothing:
#
# stimeout;5000000 -> 30.0s timeout;5000000 -> 30.0s
# stimeout;2000000 -> 30.5s timeout;2000000 -> 30.4s
# no timeout option at all -> 30.3s
#
# Identical with the option absent, so it is not being honoured under either
# name through this path. `stimeout` was renamed `timeout` in FFmpeg 5.0, and
# neither reaches the RTSP protocol here. What actually bounds it is
# OpenCV's own interrupt callback (30s for open, 30s for read), which is a
# compile-time constant we do not control.
#
# Both names are still set, because on a build where they DO take effect the
# shorter bound is what we want and an unrecognised option is ignored. But
# nothing may depend on it: a wrong address is caught by _tcp_reachable
# below, in code we own, in under a second.
#
# fflags=nobuffer and flags=low_delay: without them ffmpeg's RTSP demuxer
# holds a comfortable queue of frames before handing over the first, which
@@ -30,10 +49,25 @@ log = logging.getLogger(__name__)
# reorder wait for the same reason.
os.environ.setdefault(
"OPENCV_FFMPEG_CAPTURE_OPTIONS",
"rtsp_transport;tcp|stimeout;5000000|fflags;nobuffer|flags;low_delay|max_delay;200000",
"rtsp_transport;tcp|stimeout;5000000|timeout;5000000"
"|fflags;nobuffer|flags;low_delay|max_delay;200000",
)
# A stream can stay open and stop delivering. OpenCV breaks a blocked read
# after 30s and we reconnect, but for those 30s `connected` is True and the
# camera is dead — and a stream that trickles a frame every 20s never trips
# that timeout at all, so it never reconnects and never recovers either.
#
# 10s is not a preference. The tracker gives up on a face after `max_misses`
# (25 frames, ~1.7s at 15 fps), so by 10s every track is long gone and 150
# frames are missing: whatever this is, it is not something recognition can
# work with. Reported separately from `connected` because the two need
# opposite actions — one says check the network, the other says the camera
# is answering but sending nothing.
STALL_AFTER_S = 10.0
def _tcp_reachable(source: "str | int", timeout: float
) -> "tuple[bool, str]":
"""Cheap pre-flight for an rtsp:// URL. Non-URL sources pass through."""
@@ -169,6 +203,10 @@ class VideoSource(threading.Thread):
self.frames_total = 0
self.reconnects = 0
self._ever_connected = False
# Why the last open failed, in the words an installer can act on.
# Without it a camera that never connects reports only `connected:
# false`, which cannot distinguish a wrong IP from a wrong password.
self.last_error = ""
# -- public ---------------------------------------------------------
def latest(self) -> "tuple[Optional[np.ndarray], float]":
@@ -199,11 +237,23 @@ class VideoSource(threading.Thread):
def stop(self) -> None:
self._stopping.set()
def stalled(self) -> bool:
"""Open, but not delivering. See STALL_AFTER_S."""
if not self.connected or not self._frame_ts:
return False
return (time.time() - self._frame_ts) > STALL_AFTER_S
def stats(self) -> dict:
return {
"camera_id": self.camera_id,
"url": self._display_url,
"connected": self.connected,
# Connected AND delivering. `connected` alone stays true through
# a stall, so it is the wrong thing for a dashboard to colour a
# camera green on.
"streaming": self.connected and not self.stalled(),
"stalled": self.stalled(),
"last_error": self.last_error,
"frames_total": self.frames_total,
"reconnects": self.reconnects,
"last_frame_age_s": round(time.time() - self._frame_ts, 1)
@@ -260,6 +310,19 @@ class VideoSource(threading.Thread):
log.info("[%s] capture stopped", self.camera_id)
def _open(self) -> Optional[cv2.VideoCapture]:
# Pre-flight the socket, exactly as probe_source does. Without it a
# camera that is off, moved or mistyped costs 30s per attempt inside
# the VideoCapture constructor (measured; it is OpenCV's interrupt
# timeout, not ours to shorten) — and the constructor is not
# interruptible, so stop() cannot cut it short and a removed camera
# leaves a daemon thread holding a socket for half a minute. A
# refused or unroutable address answers in well under a second, which
# is also what lets the backoff below mean what it says.
reachable, why = _tcp_reachable(self._source, 2.0)
if not reachable:
log.debug("[%s] %s", self.camera_id, why)
self.last_error = why
return None
try:
if isinstance(self._source, int):
cap = cv2.VideoCapture(self._source)
@@ -268,8 +331,12 @@ class VideoSource(threading.Thread):
cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
if not cap.isOpened():
cap.release()
self.last_error = ("reachable, but the stream would not open "
"- check the path and credentials")
return None
self.last_error = ""
return cap
except cv2.error:
log.exception("[%s] VideoCapture error", self.camera_id)
self.last_error = "VideoCapture error - see the engine log"
return None

21
behavision/engine.py
View File

@@ -18,6 +18,7 @@ import numpy as np
from .attributes import AttributeEstimator, aggregate as aggregate_attrs
from .cameras import CameraStore
from . import capture
from .capture import VideoSource
from .faces import FaceOutbox
from .commission import CommissionRun
@@ -169,6 +170,7 @@ class CameraWorker(threading.Thread):
self._overlay_ts = 0.0
self._last_frame_ts = 0.0
self._was_connected = False
self._was_stalled = False
self.frames_processed = 0
# Motion gate state: a 160x90 greyscale thumbnail of the last frame we
# actually searched, and how many frames we have skipped since.
@@ -346,6 +348,20 @@ class CameraWorker(threading.Thread):
type="camera.up" if connected else "camera.down",
camera_id=self.cam_cfg.id))
# A stall is not a disconnect and must not be reported as one: the
# socket is fine, the camera is answering, and nothing is arriving.
# Logged on the transition only — a per-frame warning would bury the
# one line that matters under thousands of copies of itself.
stalled = self.source.stalled()
if stalled != self._was_stalled:
self._was_stalled = stalled
if stalled:
log.warning("[%s] connected but no frame for over %.0fs - the "
"camera is answering and sending nothing",
self.cam_cfg.id, capture.STALL_AFTER_S)
else:
log.info("[%s] frames resumed", self.cam_cfg.id)
def _finish_track(self, track: Track, ts: float) -> None:
"""Record what became of a track, once, as it ends.
@@ -681,7 +697,10 @@ class Engine:
"age_model": ("genderage" if self.attributes is not None
and self.attributes.has_genderage else "caffe/none"),
},
"gallery": self.store.stats(),
# Counts, plus whether the running encoder can actually SEARCH
# them. A gallery of 21 identities that the loaded model cannot
# read is the silent version of an empty one.
"gallery": {**self.store.stats(), **self.gallery.health},
"cameras": [w.stats() for w in self.snapshot_workers()],
}

46
behavision/gallery/service.py
View File

@@ -53,10 +53,56 @@ class Gallery:
# vectors from a different model are numerically incompatible.
ids, vecs = store.all_embeddings(index.dim, model=model_name)
index.add(ids, vecs)
self.health = self._assess(len(ids))
if self.health["stranded"]:
# Not an INFO line. The encoder fallback chain exists so a
# memory-starved box still runs, and when it fires every vector
# written by the previous encoder becomes invisible: the shop
# keeps its customer list and recognises nobody on it, greeting
# every regular as new and enrolling them a second time. Footfall
# stays right, which is exactly why nothing looks wrong. The old
# message for that state was "gallery ready: 0 embeddings".
log.warning(
"gallery: %d of %d stored embeddings were written by a "
"DIFFERENT encoder (%s) and cannot be searched - %d known "
"%s unrecognisable under the running model '%s'. Either "
"restore that model or accept that these identities start "
"over.",
self.health["stranded"], self.health["stored"],
", ".join(sorted(self.health["other_models"])),
self.health["identities_stranded"],
"person is" if self.health["identities_stranded"] == 1
else "people are",
model_name)
log.info("gallery ready: %d embeddings (model '%s') across %d "
"identities", len(ids), model_name,
store.stats()["identities"])
def _assess(self, usable: int) -> dict:
"""What share of the gallery the running encoder can actually reach.
Reported rather than merely logged, because a log line on a shop PC
is read by nobody: this travels to head office the same way
`fraction_below_gate` does, beside the number it qualifies.
"""
counts = self.store.model_counts()
stored = sum(counts.values())
others = {m: n for m, n in counts.items() if m != self.model_name}
identities = self.store.stats()["identities"]
return {
"model": self.model_name,
"stored": stored,
"usable": usable,
"stranded": sum(others.values()),
"other_models": sorted(others),
"identities": identities,
"identities_usable": self.store.identities_with_model(
self.model_name),
"identities_stranded": max(
0, identities - self.store.identities_with_model(
self.model_name)),
}
def resolve(self, embedding: np.ndarray, quality: float, camera_id: str,
ts: "float | None" = None,
attributes: "dict | None" = None,

25
behavision/gallery/store.py
View File

@@ -281,6 +281,31 @@ class IdentityStore:
return None
return np.frombuffer(row["vector"], dtype=np.float32), float(row["quality"])
def model_counts(self) -> "dict[str, int]":
"""How many stored embeddings each encoder produced.
The gallery only ever searches vectors tagged with the *running*
encoder, so this is what says whether the rest of the gallery is
reachable at all. See `Gallery.health` for why that matters.
"""
with self._lock:
rows = self._db.execute(
"SELECT model, COUNT(*) AS n FROM embeddings "
"GROUP BY model").fetchall()
return {str(r["model"]): int(r["n"]) for r in rows}
def identities_with_model(self, model: str) -> int:
"""Identities holding at least one embedding from this encoder.
Not the same as the identity count: an identity whose only vectors
came from a previous encoder still exists, and is unrecognisable.
"""
with self._lock:
row = self._db.execute(
"SELECT COUNT(DISTINCT identity_id) AS n FROM embeddings "
"WHERE model=?", (model,)).fetchone()
return int(row["n"]) if row else 0
def embedding_owners(self, model: "str | None" = None) -> "dict[int, int]":
"""embedding_id -> identity_id, for turning index hits into identity
pairs without a round trip to SQLite per hit."""

17
behavision/static/dashboard.html
View File

@@ -514,11 +514,24 @@ async function refresh() {
]);
renderFeeds(camList);
renderCameras(camList);
// 'stalled' is its own word on purpose: connected and offline send you
// to the network, a camera that is answering and sending nothing does
// not. Three states, because two of them need opposite actions.
const cams = stats.cameras.map(c =>
`${c.camera_id}: ${c.connected ? 'live' : 'offline'}`).join(' · ');
`${c.camera_id}: ${c.streaming ? 'live' : c.connected ? 'stalled' : 'offline'}`
).join(' · ');
// The one failure that otherwise looks like perfect health: the encoder
// that loaded cannot read the embeddings already stored, so every known
// customer is a stranger. Counts stay right, which is why it needs saying.
const stranded = stats.gallery.stranded || 0;
const warn = stranded
? ` · ⚠ ${stats.gallery.identities_stranded} people unrecognisable `
+ `(${stranded} embeddings from ${stats.gallery.other_models.join(', ')}, `
+ `running ${stats.gallery.model})`
: '';
// textContent, not innerHTML — no escaping needed here.
document.getElementById('status').textContent =
`${cams} · ${stats.gallery.identities} people · ${stats.gallery.sightings} sightings`;
`${cams} · ${stats.gallery.identities} people · ${stats.gallery.sightings} sightings${warn}`;
document.getElementById('events').innerHTML = events.map(e => {
const cls = e.type === 'person.new' ? 'new'