The engine stopped updating, and said "installed" every time

The SSL fix shipped and did not reach the machine it was written for. That
log said so, one line above the tick:

  behavision is already installed with the same version as the provided
  wheel. Use --force-reinstall to force an installation of the wheel.
   [ok] Engine and dependencies installed

and the traceback below it still pointed at model_assets.py line 59,
urllib.request.urlretrieve - code the fix had deleted.

Two frozen literals caused it: version = "1.1.0" in pyproject.toml and
__version__ = "1.0.0" in behavision/__init__.py. They disagreed with each
other and neither tracked a release, so every release built
behavision-1.1.0-py3-none-any.whl and pip install --upgrade on a machine that
already had 1.1.0 is a no-op. The comment beside that call claimed the
opposite.

The shape of the damage is what makes it bad. The Go binaries - app, agent,
setup tool - are rebuilt every release and updated normally. So a shop PC ran
a current app supervising an engine several releases old, and nothing said
which: /api/health reported the model, the paths, the cameras and the gallery,
and no version at all.

- One version, in the package, read by pyproject through
  [tool.setuptools.dynamic]. In a checkout it reads 0.0.0+dev: a
  plausible-looking number on a developer's /api/health is worse than none.
- pip install --force-reinstall --no-deps <wheel>, after the ordinary
  --upgrade. --upgrade settles the dependencies; the second call guarantees our
  own code is the code in the folder. --no-deps keeps it cheap - forcing the
  dependencies too would re-download ~300 MB every run. A rebuild at an
  unchanged version is the ordinary case while developing, so this must not
  rely on the version moving.
- release.sh stamps the tag: v0.5.6-demo -> 0.5.6+demo, valid PEP 440. Into a
  copy of the line, reverted in a trap, so the tree is never left dirty.

/api/health reports version now. Without it there is no way to tell a shop PC
three releases behind from a current one, which is how this survived several
releases.

Reproduced end to end before fixing, against real wheels on Python 3.12: two
builds of the same version, --upgrade leaves the old code in place and prints
the same sentence the colleague's Mac printed, --force-reinstall --no-deps
replaces it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KGcjxF1cNLcuwc3DAPcnfj

CLAUDE.md

@@ -3559,3 +3559,97 @@ one, and why installing into it would not survive a restart. Fixed by dragging
 the app to Applications; saying nothing leaves somebody re-running a setup tool
 that cannot win. The product is unsigned, so this is the *normal* first-run
 state on every Mac, not an edge case.
+
+### And then it could not download a 230 KB file
+
+With all of the above fixed the install succeeded on that Mac - Python 3.14
+chosen and accepted, numpy 2.5.3, onnxruntime 1.30, faiss 1.15.1, the engine
+itself - and setup died on the last step, fetching the YuNet model:
+
+```
+ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
+certificate verify failed: unable to get local issuer certificate
+```
+
+A python.org macOS build ships its **own OpenSSL with no trust store**, and
+populates one only when somebody double-clicks `Install Certificates.command`
+in the Python folder. Nobody installing face-recognition software has any
+reason to know that exists, and the failure is forty lines of traceback about
+`_ssl.c` at the end of a ten-minute install.
+
+`_urlopen` tries the default context first and retries with **certifi's**
+bundle on a verification failure. The order is the whole design:
+
+- Default first, because on Windows and on a system or Homebrew Python the
+  default context reads the machine's own certificate store - which is what
+  makes a corporate proxy with its own root CA work. Replacing it
+  unconditionally would break every site that has one in order to fix a
+  different platform.
+- certifi second, because it is already installed: `requests` is a hard
+  dependency and brings it.
+- A `URLError` is re-raised untouched. "No route to host" and "no trust store"
+  are different problems, and retrying the first with a different CA list only
+  delays the real message.
+
+`urlretrieve` had to go, since it offers no way to pass a context - and that is
+exactly the kind of rewrite that silently drops something. The
+`download: <label> <n>%` lines are a **contract**: `supervisor.go`'s
+`progressRe` parses them to put first-run progress in the tray, because the API
+is not up yet and a shop PC showing a stopped engine for five minutes after
+install looks broken. `tests/test_model_download.py` asserts them, and the
+rewritten fetch was checked against the real URL: 232,589 bytes, sha256
+identical to the model already on disk.
+
+## The engine stopped updating, and said "installed" every time
+
+The SSL fix above was released and **did not reach the machine it was written
+for**. Its log said so, one line above the tick:
+
+```
+behavision is already installed with the same version as the provided wheel.
+Use --force-reinstall to force an installation of the wheel.
+ [ok] Engine and dependencies installed
+```
+
+and the traceback that followed still pointed at `model_assets.py", line 59,
+in _fetch / urllib.request.urlretrieve` — code the fix had deleted.
+
+Two frozen literals caused it: `version = "1.1.0"` in `pyproject.toml` and
+`__version__ = "1.0.0"` in `behavision/__init__.py`. They disagreed with each
+other and neither tracked a release, so **every release built
+`behavision-1.1.0-py3-none-any.whl`** and `pip install --upgrade` on a machine
+that already had 1.1.0 is a no-op. The comment beside that call even claimed
+the opposite: *"`--upgrade` so re-running after a new release replaces the
+engine rather than leaving the old one in place and reporting success."*
+
+The shape of the damage is what makes it bad. The Go binaries — the app, the
+agent, the setup tool — are rebuilt every release and updated normally. So a
+shop PC ran a current app supervising an engine several releases old, and
+nothing anywhere said which: `/api/health` reported the model, the paths, the
+cameras and the gallery, and **no version at all**.
+
+Three changes, and the second is the one that does not depend on the first
+being remembered:
+
+- **One version, in the package**, read by `pyproject.toml` through
+  `[tool.setuptools.dynamic]`. Two literals in two files is how they came to
+  disagree. In a checkout it reads `0.0.0+dev`, because a plausible-looking
+  number on a developer's `/api/health` would be worse than none.
+- **`pip install --force-reinstall --no-deps <wheel>`, after the ordinary
+  `--upgrade`.** `--upgrade` settles the dependencies; the second call
+  guarantees our own code is the code in the folder. `--no-deps` is what keeps
+  it cheap — forcing the dependencies too would re-download ~300 MB of numpy,
+  OpenCV and onnxruntime on every run. A rebuild at an unchanged version is the
+  ordinary case while developing, so this must not rely on the version moving.
+- **`release.sh` stamps the tag**: `v0.5.6-demo` → `0.5.6+demo`, valid PEP 440
+  (a local segment takes alphanumerics and dots, never hyphens). Into a copy of
+  the line, reverted in a `trap`, so the tree is never left dirty.
+
+`/api/health` reports `version` now. Without it there is no way to tell a shop
+PC three releases behind from a current one, which is precisely how this
+survived several releases.
+
+Reproduced end to end before fixing, against real wheels on Python 3.12: two
+builds of the same version, `--upgrade` leaves the old code in place and prints
+the same sentence the colleague's Mac printed, `--force-reinstall --no-deps`
+replaces it.

agent/cmd/behavision-setup/main.go

@@ -539,7 +539,30 @@ func pipInstall(vpy, src string) error {
 	// 'behavision.egg-info': Read-only file system". Falling back to source
 	// copies it somewhere writable first, for the same reason.
 	if wheels, _ := filepath.Glob(filepath.Join(src, "behavision-*.whl")); len(wheels) > 0 {
-		return stream(exec.Command(vpy, "-m", "pip", "install", "--upgrade", wheels[0]),
+		// TWO calls, and the second is the one that matters.
+		//
+		// `--upgrade` alone is not an upgrade when the version has not moved:
+		// pip skips the wheel and says so, one line above this program
+		// printing "[ok] Engine and dependencies installed". The wheel version
+		// was a frozen literal for several releases, so every engine fix in
+		// them silently failed to reach any machine that had run setup once -
+		// while the Go binaries beside it, rebuilt every release, updated
+		// normally. Half the product current, half of it months old, and
+		// nothing saying which.
+		//
+		// release.sh stamps the tag into the version now, so the versions do
+		// differ. This does not rely on that: a rebuild at the same version is
+		// the ordinary case while developing, and "installed" has to mean the
+		// code in this folder either way.
+		if err := stream(exec.Command(vpy, "-m", "pip", "install", "--upgrade", wheels[0]),
+			"installing the engine"); err != nil {
+			return err
+		}
+		// --no-deps so this is our own package only: the call above has
+		// already settled the dependencies, and forcing those too would
+		// re-download ~300 MB of numpy, OpenCV and onnxruntime every run.
+		return stream(exec.Command(vpy, "-m", "pip", "install",
+			"--force-reinstall", "--no-deps", wheels[0]),
 			"installing the engine")
 	}
 	tmp, err := os.MkdirTemp("", "behavision-src-")

behavision/__init__.py

@@ -4,4 +4,15 @@ Pipeline: capture -> detect (YuNet) -> track (IoU) -> align + encode
 (ArcFace ONNX) -> match / auto-enroll (FAISS + SQLite) -> events + API.
 """
 
-__version__ = "1.0.0"
+# Stamped by release.sh from the git tag, into a copy of this line, so a
+# release always produces a wheel nobody has installed before. It was a frozen
+# "1.0.0" here and a frozen "1.1.0" in pyproject.toml - two literals that
+# disagreed with each other and tracked nothing - which is how every engine fix
+# for several releases silently failed to reach a machine that had run setup
+# once. pip skips a wheel whose version is already installed and says so, one
+# line above setup printing "[ok] Engine and dependencies installed".
+#
+# In a checkout it stays obviously a checkout: "0.0.0+dev" on /api/health is
+# the truth about a developer's machine, and a plausible-looking number there
+# would be worse than none.
+__version__ = "0.0.0+dev"

behavision/api.py

@@ -17,6 +17,7 @@ from fastapi.responses import HTMLResponse, Response, StreamingResponse
 from fastapi.security import HTTPBasic, HTTPBasicCredentials
 from pydantic import BaseModel, ValidationError
 
+from . import __version__
 from .config import ApiSection, CameraConfig, CameraTuning
 from .commission import CommissionRun
 from .events import Event
@@ -142,7 +143,7 @@ def _reencode(jpeg: bytes, width: int, quality: int) -> "bytes | None":
 
 
 def create_app(engine: Engine) -> FastAPI:
-    app = FastAPI(title="Behavision", version="1.0.0",
+    app = FastAPI(title="Behavision", version=__version__,
                   dependencies=_auth_dependencies(engine.cfg.api))
 
     def worker_or_404(camera_id: str):
@@ -172,6 +173,10 @@ def create_app(engine: Engine) -> FastAPI:
         # are up, and the shop recognises nobody it already knows.
         stranded = engine.gallery.health["stranded"]
         return {"status": "ok" if engine.started_at else "starting",
+                # Which build is actually running. Without it there was no way
+                # to tell a shop PC three releases behind from a current one -
+                # which is precisely how a silent install failure survived.
+                "version": __version__,
                 "recognition_model": engine.encoder.model_name,
                 "gallery_unreadable_embeddings": stranded,
                 # "where is my database" must be answerable from the API: the

behavision/model_assets.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import logging
 import shutil
+import ssl
 import urllib.request
 from pathlib import Path
 
@@ -35,6 +36,54 @@ _COPY_MAP = {
 }
 
 
+def _https_context() -> "ssl.SSLContext | None":
+    """The CA store to trust, or None to use whatever Python defaults to.
+
+    Returning None first is deliberate. On Windows and on a Homebrew or
+    system Python, the default context reads the machine's own certificate
+    store - which is what makes a corporate proxy with its own root CA work.
+    Replacing that with certifi's bundle unconditionally would break every
+    site that has one, in order to fix a different platform.
+
+    The platform this fixes is a python.org macOS build. It ships its own
+    OpenSSL with NO trust store, and populates one only when somebody
+    double-clicks `Install Certificates.command` in the Python folder -
+    which nobody installing face-recognition software has any reason to know
+    about. Every HTTPS request from that interpreter fails with:
+
+        ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
+        certificate verify failed: unable to get local issuer certificate
+
+    Measured on a colleague's Mac: the engine installed perfectly and then
+    could not download a 230 KB model file, ending setup in forty lines of
+    traceback about `_ssl.c`.
+    """
+    try:
+        import certifi
+    except ImportError:  # pragma: no cover - certifi ships with requests
+        return None
+    return ssl.create_default_context(cafile=certifi.where())
+
+
+def _urlopen(url: str, timeout: float = 60.0):
+    """Open a URL, falling back to certifi's CA bundle on a verify failure.
+
+    Default first, certifi second, so the fix is additive: a machine whose
+    own store works keeps using it, and one with no store at all gets a
+    bundle rather than a traceback. certifi is already here - `requests` is a
+    hard dependency and brings it.
+    """
+    try:
+        return urllib.request.urlopen(url, timeout=timeout)
+    except ssl.SSLCertVerificationError:
+        ctx = _https_context()
+        if ctx is None:
+            raise
+        log.info("the system certificate store could not verify %s; "
+                 "using the bundled CA list", url.split("/")[2])
+        return urllib.request.urlopen(url, timeout=timeout, context=ctx)
+
+
 def _fetch(url: str, dest: Path, label: str) -> None:
     """Download with progress on stdout the supervisor can read.
 
@@ -56,7 +105,22 @@ def _fetch(url: str, dest: Path, label: str) -> None:
             last = pct
             log.info("download: %s %d%%", label, pct)
 
-    urllib.request.urlretrieve(url, dest, hook)
+    # Streamed rather than urlretrieve, only because urlretrieve offers no way
+    # to pass an SSL context and the whole point here is choosing one. The
+    # `download: <label> <n>%` lines are a contract: the supervisor parses
+    # them (`progressRe`) to put first-run progress in the tray, and without
+    # them a shop PC shows a stopped engine for five minutes after install.
+    with _urlopen(url) as resp:
+        total = int(resp.headers.get("Content-Length") or 0)
+        blocks, block_size = 0, 64 * 1024
+        with open(dest, "wb") as out:
+            while True:
+                chunk = resp.read(block_size)
+                if not chunk:
+                    break
+                out.write(chunk)
+                blocks += 1
+                hook(blocks, block_size, total)
     log.info("download: %s 100%%", label)
 
 
@@ -91,7 +155,7 @@ def setup_models(models_dir: Path) -> "list[str]":
         import io
         import zipfile
 
-        with urllib.request.urlopen(BUFFALO_SC_URL) as resp:
+        with _urlopen(BUFFALO_SC_URL) as resp:
             payload = io.BytesIO(resp.read())
         with zipfile.ZipFile(payload) as zf, \
                 zf.open("w600k_mbf.onnx") as src, \

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "behavision"
-version = "1.1.0"
+dynamic = ["version"]
 description = "Production face recognition over RTSP"
 requires-python = ">=3.10"
 dependencies = [
@@ -32,3 +32,9 @@ behavision = ["static/*"]
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]
+
+# One version for the package and the wheel. Two literals in two files is how
+# they came to disagree (1.0.0 here, 1.1.0 there) and how neither tracked a
+# release.
+[tool.setuptools.dynamic]
+version = {attr = "behavision.__version__"}

release.sh

@@ -50,7 +50,28 @@ step "2. Agent and setup tool"
 step "3. Engine source and wheel"
 # The wheel is built with the checkout's own interpreter; requires-python is a
 # statement about the SHOP PC, which setup enforces when it finds Python there.
+# Stamp the tag into the wheel version. Without this every release built
+# behavision-1.1.0-py3-none-any.whl, and `pip install --upgrade` on a machine
+# that already had 1.1.0 is a no-op - so an engine fix reached nobody who had
+# ever run setup, while the Go binaries beside it updated normally. Nothing
+# reported a version either, so there was no way to tell a shop PC three
+# releases behind from a current one.
+#
+# v0.5.6-demo -> 0.5.6+demo, which is valid PEP 440: a local segment takes
+# alphanumerics and dots, never hyphens.
+TMPVER=$(mktemp)
+PEP440=$(printf '%s' "${TAG#v}" | sed 's/-/+/; s/[^0-9A-Za-z.+]/./g')
+trap 'git checkout -- behavision/__init__.py 2>/dev/null || true; rm -f "$TMPVER"' EXIT
+# A temp file rather than `sed -i`, whose argument handling differs between
+# BSD and GNU - this script is run from a Mac today and that is not a reason
+# to plant a portability trap in a release path.
+sed "s/^__version__ = .*/__version__ = \"$PEP440\"/" behavision/__init__.py > "$TMPVER"
+cat "$TMPVER" > behavision/__init__.py
 .venv/bin/python -m pip wheel --no-deps --ignore-requires-python -q -w "$STAGE/engine-src" . 2>&1 | grep -v "DEPRECATION\|WARNING: Ignoring" || true
+git checkout -- behavision/__init__.py
+rm -f "$TMPVER"
+trap - EXIT
+echo "   engine wheel version $PEP440"
 ls "$STAGE"/engine-src/behavision-*.whl >/dev/null || { echo "wheel was not built" >&2; exit 1; }
 cp pyproject.toml requirements.txt "$STAGE/engine-src/"
 mkdir -p "$STAGE/engine-src/config" && cp config/default.yaml "$STAGE/engine-src/config/"

tests/test_model_download.py

@@ -0,0 +1,105 @@
+"""Downloading the models is the last step of every fresh install, and it ran
+into the one macOS trap nothing else here does.
+
+A python.org macOS build ships its own OpenSSL with NO trust store, and
+populates one only when somebody double-clicks `Install Certificates.command`
+in the Python folder. Measured on a colleague's Mac: the engine installed
+perfectly - numpy, onnxruntime, faiss, all of it - and then could not fetch a
+230 KB model file, ending setup in forty lines of traceback about `_ssl.c`.
+"""
+import io
+import logging
+import ssl
+import urllib.request
+
+import pytest
+
+from behavision import model_assets
+
+
+class _Resp(io.BytesIO):
+    """Enough of an http response for _fetch: read() and .headers."""
+
+    def __init__(self, payload: bytes):
+        super().__init__(payload)
+        self.headers = {"Content-Length": str(len(payload))}
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *exc):
+        self.close()
+        return False
+
+
+def _verify_error():
+    return ssl.SSLCertVerificationError(
+        "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: "
+        "unable to get local issuer certificate")
+
+
+def test_a_machine_with_no_trust_store_falls_back_to_the_bundled_one(monkeypatch):
+    calls = []
+
+    def fake(url, timeout=None, context=None):
+        calls.append(context)
+        if context is None:
+            raise _verify_error()
+        return _Resp(b"ok")
+
+    monkeypatch.setattr(urllib.request, "urlopen", fake)
+    with model_assets._urlopen("https://example.invalid/m.onnx") as resp:
+        assert resp.read() == b"ok"
+
+    assert len(calls) == 2, f"expected a retry, got {calls}"
+    # Default FIRST, and that order is the point. On Windows and on a system
+    # Python the default context reads the machine's own certificate store,
+    # which is what makes a corporate proxy with its own root CA work.
+    # Replacing it unconditionally would break every site that has one in
+    # order to fix a different platform.
+    assert calls[0] is None
+    assert isinstance(calls[1], ssl.SSLContext)
+
+
+def test_a_working_trust_store_is_used_as_is(monkeypatch):
+    calls = []
+
+    def fake(url, timeout=None, context=None):
+        calls.append(context)
+        return _Resp(b"ok")
+
+    monkeypatch.setattr(urllib.request, "urlopen", fake)
+    model_assets._urlopen("https://example.invalid/m.onnx").close()
+    assert calls == [None], "the machine's own certificate store was bypassed"
+
+
+def test_a_real_network_failure_is_not_disguised_as_a_certificate_problem(monkeypatch):
+    def fake(url, timeout=None, context=None):
+        raise urllib.error.URLError("no route to host")
+
+    monkeypatch.setattr(urllib.request, "urlopen", fake)
+    with pytest.raises(urllib.error.URLError):
+        model_assets._urlopen("https://example.invalid/m.onnx")
+
+
+def test_progress_lines_survive_the_rewrite(tmp_path, monkeypatch, caplog):
+    """`download: <label> <n>%` is a CONTRACT, not logging.
+
+    The supervisor parses it (progressRe) to put first-run progress in the
+    tray and the window, because the API is not up yet and a shop PC showing
+    a stopped engine for five minutes after install looks broken. Switching
+    off urlretrieve - needed because it offers no way to pass an SSL context -
+    is exactly the kind of change that drops it silently.
+    """
+    payload = b"x" * (64 * 1024 * 8)
+    monkeypatch.setattr(urllib.request, "urlopen",
+                        lambda url, timeout=None, context=None: _Resp(payload))
+    dest = tmp_path / "model.onnx"
+    with caplog.at_level(logging.INFO, logger="behavision.model_assets"):
+        model_assets._fetch("https://example.invalid/m.onnx", dest, "face detector")
+
+    assert dest.read_bytes() == payload
+    lines = [r.getMessage() for r in caplog.records]
+    pct = [ln for ln in lines if ln.startswith("download: face detector ")]
+    assert pct, f"no progress lines at all: {lines}"
+    assert "download: face detector 100%" in pct, pct