Three states that looked like health from outside

Audited the engine for what it does when something goes wrong rather than
when it goes right. Each of these left the process healthy, the dashboard
green and the product not working.

A gallery the running encoder cannot read. Embeddings are model-tagged, so
when the fallback chain fires every vector the previous encoder wrote goes
invisible: the shop keeps its customer list and recognises nobody on it,
enrolling each regular a second time. Footfall stays correct, which is why
nothing looks wrong. The only evidence was an INFO line reading 'gallery
ready: 0 embeddings (model w600k_mbf) across 21 identities' - a sentence
that states the disaster and calls it ready. Gallery.health now warns with
the count of PEOPLE lost, not vectors, and carries the same numbers to
/api/stats and /api/health, because a log line on a shop PC is read by
nobody. Proved against the real 87-embedding gallery.

Connected, and sending nothing. 'connected' meant the socket opened, so a
stream that went quiet kept it true while last_frame_age_s climbed and the
heartbeat told head office the camera was up. OpenCV breaks a blocked read
at 30s, but a camera trickling a frame every 20s never trips that and never
recovers. streaming/stalled are reported beside connected and the dashboard
says live/stalled/offline - three states because offline sends you to the
network and stalled says the camera is answering and sending nothing.

The 5-second RTSP timeout that never existed. stimeout;5000000 carried a
comment claiming it bounded a dead camera. Measured on OpenCV 4.11 /
FFmpeg 7.1 against a socket that accepts and then says nothing: 30.0s with
stimeout, 30.0s with timeout, 30.3s with no option at all - identical, so
it was never honoured. stimeout became timeout in FFmpeg 5.0 and neither
reaches the RTSP protocol through this path; the real bound is OpenCV's own
interrupt constant. Replaced by the _tcp_reachable pre-flight probe_source
already used, in code we own: 30.3s -> 0.00-2.02s, each naming its cause.
That matters beyond speed - the VideoCapture constructor is not
interruptible, so stop() could not cut it short and a camera removed from
head office left a daemon thread holding a socket for half a minute.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KGcjxF1cNLcuwc3DAPcnfj

behavision/gallery/service.py

@@ -53,10 +53,56 @@ class Gallery:
         # vectors from a different model are numerically incompatible.
         ids, vecs = store.all_embeddings(index.dim, model=model_name)
         index.add(ids, vecs)
+        self.health = self._assess(len(ids))
+        if self.health["stranded"]:
+            # Not an INFO line. The encoder fallback chain exists so a
+            # memory-starved box still runs, and when it fires every vector
+            # written by the previous encoder becomes invisible: the shop
+            # keeps its customer list and recognises nobody on it, greeting
+            # every regular as new and enrolling them a second time. Footfall
+            # stays right, which is exactly why nothing looks wrong. The old
+            # message for that state was "gallery ready: 0 embeddings".
+            log.warning(
+                "gallery: %d of %d stored embeddings were written by a "
+                "DIFFERENT encoder (%s) and cannot be searched - %d known "
+                "%s unrecognisable under the running model '%s'. Either "
+                "restore that model or accept that these identities start "
+                "over.",
+                self.health["stranded"], self.health["stored"],
+                ", ".join(sorted(self.health["other_models"])),
+                self.health["identities_stranded"],
+                "person is" if self.health["identities_stranded"] == 1
+                else "people are",
+                model_name)
         log.info("gallery ready: %d embeddings (model '%s') across %d "
                  "identities", len(ids), model_name,
                  store.stats()["identities"])
 
+    def _assess(self, usable: int) -> dict:
+        """What share of the gallery the running encoder can actually reach.
+
+        Reported rather than merely logged, because a log line on a shop PC
+        is read by nobody: this travels to head office the same way
+        `fraction_below_gate` does, beside the number it qualifies.
+        """
+        counts = self.store.model_counts()
+        stored = sum(counts.values())
+        others = {m: n for m, n in counts.items() if m != self.model_name}
+        identities = self.store.stats()["identities"]
+        return {
+            "model": self.model_name,
+            "stored": stored,
+            "usable": usable,
+            "stranded": sum(others.values()),
+            "other_models": sorted(others),
+            "identities": identities,
+            "identities_usable": self.store.identities_with_model(
+                self.model_name),
+            "identities_stranded": max(
+                0, identities - self.store.identities_with_model(
+                    self.model_name)),
+        }
+
     def resolve(self, embedding: np.ndarray, quality: float, camera_id: str,
                 ts: "float | None" = None,
                 attributes: "dict | None" = None,