urllib does not let SSLCertVerificationError out, and a fake said it did

The certificate fix shipped and failed on the machine it was written for, with
the exact traceback it was meant to prevent. The retry was written

    except ssl.SSLCertVerificationError:

and urllib never raises that from urlopen. It catches it and re-raises
urllib.error.URLError(err), carrying the original on .reason. So the except
matched nothing, ever, and the fallback could not fire.

The unit test passed throughout, because the stub it used raised the bare SSL
error - a shape real urllib never produces. That is the lesson: a fake that
agrees with the author is worse than no test, because it converts an untested
path into a tested-looking one. This file already says that about
UPDATE ... RETURNING and about the in-memory API fake, and it got written
again anyway.

_is_cert_failure checks the exception and its .reason, and the tests now raise
URLError(SSLCertVerificationError(...)) - what the traceback actually shows. A
plain URLError is re-raised untouched, and a test asserts no second attempt is
made for one.

Beside the stubs there is now a real reproduction, opt-in behind
BEHAVISION_NETWORK_TESTS=1. Python's default context honours SSL_CERT_FILE, so
an empty file gives a context that trusts nobody - the python.org condition
exactly - while certifi is loaded by path and is unaffected. It skips rather
than passes where it cannot reproduce that, and the difference is measured:

    macOS Command Line Tools  LibreSSL 2.8.3   128 CAs with an empty CA file
    python.org / pyenv build  OpenSSL 3.5.8      0 CAs -> reproduces it

Checked for teeth by putting the shipped except back: both the corrected stub
test and the live one fail, and pass again when it is restored.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KGcjxF1cNLcuwc3DAPcnfj

CLAUDE.md

@@ -3599,3 +3599,103 @@ is not up yet and a shop PC showing a stopped engine for five minutes after
 install looks broken. `tests/test_model_download.py` asserts them, and the
 rewritten fetch was checked against the real URL: 232,589 bytes, sha256
 identical to the model already on disk.
+
+## The engine stopped updating, and said "installed" every time
+
+The SSL fix above was released and **did not reach the machine it was written
+for**. Its log said so, one line above the tick:
+
+```
+behavision is already installed with the same version as the provided wheel.
+Use --force-reinstall to force an installation of the wheel.
+ [ok] Engine and dependencies installed
+```
+
+and the traceback that followed still pointed at `model_assets.py", line 59,
+in _fetch / urllib.request.urlretrieve` — code the fix had deleted.
+
+Two frozen literals caused it: `version = "1.1.0"` in `pyproject.toml` and
+`__version__ = "1.0.0"` in `behavision/__init__.py`. They disagreed with each
+other and neither tracked a release, so **every release built
+`behavision-1.1.0-py3-none-any.whl`** and `pip install --upgrade` on a machine
+that already had 1.1.0 is a no-op. The comment beside that call even claimed
+the opposite: *"`--upgrade` so re-running after a new release replaces the
+engine rather than leaving the old one in place and reporting success."*
+
+The shape of the damage is what makes it bad. The Go binaries — the app, the
+agent, the setup tool — are rebuilt every release and updated normally. So a
+shop PC ran a current app supervising an engine several releases old, and
+nothing anywhere said which: `/api/health` reported the model, the paths, the
+cameras and the gallery, and **no version at all**.
+
+Three changes, and the second is the one that does not depend on the first
+being remembered:
+
+- **One version, in the package**, read by `pyproject.toml` through
+  `[tool.setuptools.dynamic]`. Two literals in two files is how they came to
+  disagree. In a checkout it reads `0.0.0+dev`, because a plausible-looking
+  number on a developer's `/api/health` would be worse than none.
+- **`pip install --force-reinstall --no-deps <wheel>`, after the ordinary
+  `--upgrade`.** `--upgrade` settles the dependencies; the second call
+  guarantees our own code is the code in the folder. `--no-deps` is what keeps
+  it cheap — forcing the dependencies too would re-download ~300 MB of numpy,
+  OpenCV and onnxruntime on every run. A rebuild at an unchanged version is the
+  ordinary case while developing, so this must not rely on the version moving.
+- **`release.sh` stamps the tag**: `v0.5.6-demo` → `0.5.6+demo`, valid PEP 440
+  (a local segment takes alphanumerics and dots, never hyphens). Into a copy of
+  the line, reverted in a `trap`, so the tree is never left dirty.
+
+`/api/health` reports `version` now. Without it there is no way to tell a shop
+PC three releases behind from a current one, which is precisely how this
+survived several releases.
+
+Reproduced end to end before fixing, against real wheels on Python 3.12: two
+builds of the same version, `--upgrade` leaves the old code in place and prints
+the same sentence the colleague's Mac printed, `--force-reinstall --no-deps`
+replaces it.
+
+### `urllib` does not let `SSLCertVerificationError` out, and a fake said it did
+
+The certificate fix above shipped and **failed on the machine it was written
+for, with the exact traceback it was meant to prevent**. The retry was written
+
+```python
+except ssl.SSLCertVerificationError:
+```
+
+and urllib never raises that from `urlopen`. It catches it and re-raises
+`urllib.error.URLError(err)`, carrying the original on `.reason`. So the
+`except` matched nothing, ever, and the fallback could not fire.
+
+The unit test passed throughout, because the stub it used raised the bare SSL
+error — **a shape real urllib never produces**. That is the whole lesson: a
+fake that agrees with the author is worse than no test, because it converts an
+untested path into a tested-looking one. The same sentence is already in this
+file about `UPDATE ... RETURNING` and about the in-memory API fake, and it was
+written again here anyway.
+
+`_is_cert_failure` checks the exception **and** its `.reason`, and the tests
+now raise `URLError(SSLCertVerificationError(...))`, which is what his
+traceback shows. A plain `URLError` — "no route to host" — is re-raised
+untouched: retrying that with a different CA list changes nothing except how
+long the operator waits for the real message, and a test asserts the second
+attempt never happens.
+
+Beside the stubs there is now a **real** reproduction,
+`test_against_a_real_machine_with_no_trust_store`, opt-in behind
+`BEHAVISION_NETWORK_TESTS=1` because it reaches github.com. Python's default
+context honours `SSL_CERT_FILE`, so pointing it at an empty file gives a
+default context that trusts nobody — the python.org condition exactly — while
+certifi is loaded by path and is unaffected.
+
+It skips rather than passes where it cannot reproduce that, and the difference
+is measured rather than assumed:
+
+```
+macOS Command Line Tools  LibreSSL 2.8.3   128 CAs with SSL_CERT_FILE empty
+                                           (reads the system keychain)
+python.org / pyenv build  OpenSSL 3.5.8      0 CAs  -> reproduces it
+```
+
+Checked for teeth by putting the shipped `except` back: both the corrected stub
+test and the live one fail, and pass again when it is restored.

agent/cmd/behavision-setup/main.go

@@ -539,7 +539,30 @@ func pipInstall(vpy, src string) error {
 	// 'behavision.egg-info': Read-only file system". Falling back to source
 	// copies it somewhere writable first, for the same reason.
 	if wheels, _ := filepath.Glob(filepath.Join(src, "behavision-*.whl")); len(wheels) > 0 {
-		return stream(exec.Command(vpy, "-m", "pip", "install", "--upgrade", wheels[0]),
+		// TWO calls, and the second is the one that matters.
+		//
+		// `--upgrade` alone is not an upgrade when the version has not moved:
+		// pip skips the wheel and says so, one line above this program
+		// printing "[ok] Engine and dependencies installed". The wheel version
+		// was a frozen literal for several releases, so every engine fix in
+		// them silently failed to reach any machine that had run setup once -
+		// while the Go binaries beside it, rebuilt every release, updated
+		// normally. Half the product current, half of it months old, and
+		// nothing saying which.
+		//
+		// release.sh stamps the tag into the version now, so the versions do
+		// differ. This does not rely on that: a rebuild at the same version is
+		// the ordinary case while developing, and "installed" has to mean the
+		// code in this folder either way.
+		if err := stream(exec.Command(vpy, "-m", "pip", "install", "--upgrade", wheels[0]),
+			"installing the engine"); err != nil {
+			return err
+		}
+		// --no-deps so this is our own package only: the call above has
+		// already settled the dependencies, and forcing those too would
+		// re-download ~300 MB of numpy, OpenCV and onnxruntime every run.
+		return stream(exec.Command(vpy, "-m", "pip", "install",
+			"--force-reinstall", "--no-deps", wheels[0]),
 			"installing the engine")
 	}
 	tmp, err := os.MkdirTemp("", "behavision-src-")

behavision/__init__.py

@@ -4,4 +4,15 @@ Pipeline: capture -> detect (YuNet) -> track (IoU) -> align + encode
 (ArcFace ONNX) -> match / auto-enroll (FAISS + SQLite) -> events + API.
 """
 
-__version__ = "1.0.0"
+# Stamped by release.sh from the git tag, into a copy of this line, so a
+# release always produces a wheel nobody has installed before. It was a frozen
+# "1.0.0" here and a frozen "1.1.0" in pyproject.toml - two literals that
+# disagreed with each other and tracked nothing - which is how every engine fix
+# for several releases silently failed to reach a machine that had run setup
+# once. pip skips a wheel whose version is already installed and says so, one
+# line above setup printing "[ok] Engine and dependencies installed".
+#
+# In a checkout it stays obviously a checkout: "0.0.0+dev" on /api/health is
+# the truth about a developer's machine, and a plausible-looking number there
+# would be worse than none.
+__version__ = "0.0.0+dev"

behavision/api.py

@@ -17,6 +17,7 @@ from fastapi.responses import HTMLResponse, Response, StreamingResponse
 from fastapi.security import HTTPBasic, HTTPBasicCredentials
 from pydantic import BaseModel, ValidationError
 
+from . import __version__
 from .config import ApiSection, CameraConfig, CameraTuning
 from .commission import CommissionRun
 from .events import Event
@@ -142,7 +143,7 @@ def _reencode(jpeg: bytes, width: int, quality: int) -> "bytes | None":
 
 
 def create_app(engine: Engine) -> FastAPI:
-    app = FastAPI(title="Behavision", version="1.0.0",
+    app = FastAPI(title="Behavision", version=__version__,
                   dependencies=_auth_dependencies(engine.cfg.api))
 
     def worker_or_404(camera_id: str):
@@ -172,6 +173,10 @@ def create_app(engine: Engine) -> FastAPI:
         # are up, and the shop recognises nobody it already knows.
         stranded = engine.gallery.health["stranded"]
         return {"status": "ok" if engine.started_at else "starting",
+                # Which build is actually running. Without it there was no way
+                # to tell a shop PC three releases behind from a current one -
+                # which is precisely how a silent install failure survived.
+                "version": __version__,
                 "recognition_model": engine.encoder.model_name,
                 "gallery_unreadable_embeddings": stranded,
                 # "where is my database" must be answerable from the API: the

behavision/model_assets.py

@@ -5,6 +5,7 @@ from __future__ import annotations
 import logging
 import shutil
 import ssl
+import urllib.error
 import urllib.request
 from pathlib import Path
 
@@ -65,6 +66,27 @@ def _https_context() -> "ssl.SSLContext | None":
     return ssl.create_default_context(cafile=certifi.where())
 
 
+def _is_cert_failure(err: BaseException) -> bool:
+    """Is this a certificate-verification failure, however it is wrapped?
+
+    `urllib` does NOT let `ssl.SSLCertVerificationError` out. It catches it and
+    re-raises `urllib.error.URLError(err)`, carrying the original on `.reason`
+    - so `except ssl.SSLCertVerificationError` around `urlopen` matches
+    nothing, ever.
+
+    That is not a subtlety this file gets to record academically: the first
+    version of the fallback below was written exactly that way, shipped, and
+    failed on the machine it was written for with the very traceback it was
+    meant to prevent. The unit test passed throughout, because the fake it
+    used raised the bare SSL error - a shape real urllib never produces. A
+    stub that agrees with the author is worse than no test, and the test now
+    raises what urllib raises.
+    """
+    reason = getattr(err, "reason", None)
+    return isinstance(err, ssl.SSLCertVerificationError) or \
+        isinstance(reason, ssl.SSLCertVerificationError)
+
+
 def _urlopen(url: str, timeout: float = 60.0):
     """Open a URL, falling back to certifi's CA bundle on a verify failure.
 
@@ -75,7 +97,13 @@ def _urlopen(url: str, timeout: float = 60.0):
     """
     try:
         return urllib.request.urlopen(url, timeout=timeout)
-    except ssl.SSLCertVerificationError:
+    except (urllib.error.URLError, ssl.SSLCertVerificationError) as err:
+        # Only a certificate problem is worth a second attempt. "No route to
+        # host" and "connection refused" arrive as URLError too, and retrying
+        # those with a different CA list changes nothing except how long the
+        # operator waits for the real message.
+        if not _is_cert_failure(err):
+            raise
         ctx = _https_context()
         if ctx is None:
             raise

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "behavision"
-version = "1.1.0"
+dynamic = ["version"]
 description = "Production face recognition over RTSP"
 requires-python = ">=3.10"
 dependencies = [
@@ -32,3 +32,9 @@ behavision = ["static/*"]
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]
+
+# One version for the package and the wheel. Two literals in two files is how
+# they came to disagree (1.0.0 here, 1.1.0 there) and how neither tracked a
+# release.
+[tool.setuptools.dynamic]
+version = {attr = "behavision.__version__"}

release.sh

@@ -50,7 +50,28 @@ step "2. Agent and setup tool"
 step "3. Engine source and wheel"
 # The wheel is built with the checkout's own interpreter; requires-python is a
 # statement about the SHOP PC, which setup enforces when it finds Python there.
+# Stamp the tag into the wheel version. Without this every release built
+# behavision-1.1.0-py3-none-any.whl, and `pip install --upgrade` on a machine
+# that already had 1.1.0 is a no-op - so an engine fix reached nobody who had
+# ever run setup, while the Go binaries beside it updated normally. Nothing
+# reported a version either, so there was no way to tell a shop PC three
+# releases behind from a current one.
+#
+# v0.5.6-demo -> 0.5.6+demo, which is valid PEP 440: a local segment takes
+# alphanumerics and dots, never hyphens.
+TMPVER=$(mktemp)
+PEP440=$(printf '%s' "${TAG#v}" | sed 's/-/+/; s/[^0-9A-Za-z.+]/./g')
+trap 'git checkout -- behavision/__init__.py 2>/dev/null || true; rm -f "$TMPVER"' EXIT
+# A temp file rather than `sed -i`, whose argument handling differs between
+# BSD and GNU - this script is run from a Mac today and that is not a reason
+# to plant a portability trap in a release path.
+sed "s/^__version__ = .*/__version__ = \"$PEP440\"/" behavision/__init__.py > "$TMPVER"
+cat "$TMPVER" > behavision/__init__.py
 .venv/bin/python -m pip wheel --no-deps --ignore-requires-python -q -w "$STAGE/engine-src" . 2>&1 | grep -v "DEPRECATION\|WARNING: Ignoring" || true
+git checkout -- behavision/__init__.py
+rm -f "$TMPVER"
+trap - EXIT
+echo "   engine wheel version $PEP440"
 ls "$STAGE"/engine-src/behavision-*.whl >/dev/null || { echo "wheel was not built" >&2; exit 1; }
 cp pyproject.toml requirements.txt "$STAGE/engine-src/"
 mkdir -p "$STAGE/engine-src/config" && cp config/default.yaml "$STAGE/engine-src/config/"

tests/test_model_download.py

@@ -9,6 +9,7 @@ perfectly - numpy, onnxruntime, faiss, all of it - and then could not fetch a
 """
 import io
 import logging
+import os
 import ssl
 import urllib.request
 
@@ -33,9 +34,18 @@ class _Resp(io.BytesIO):
 
 
 def _verify_error():
-    return ssl.SSLCertVerificationError(
+    """What urlopen ACTUALLY raises, which is not what it looks like.
+
+    urllib catches ssl.SSLCertVerificationError and re-raises
+    urllib.error.URLError(err), carrying the original on `.reason`. The first
+    version of this test raised the bare SSL error - a shape real urllib never
+    produces - so it passed against a fallback that could never fire, and the
+    fix shipped and failed on the machine it was written for with the exact
+    traceback it was meant to prevent.
+    """
+    return urllib.error.URLError(ssl.SSLCertVerificationError(
         "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: "
-        "unable to get local issuer certificate")
+        "unable to get local issuer certificate"))
 
 
 def test_a_machine_with_no_trust_store_falls_back_to_the_bundled_one(monkeypatch):
@@ -74,12 +84,22 @@ def test_a_working_trust_store_is_used_as_is(monkeypatch):
 
 
 def test_a_real_network_failure_is_not_disguised_as_a_certificate_problem(monkeypatch):
+    """A URLError is not automatically a certificate problem.
+
+    "No route to host" and "connection refused" arrive as URLError too, and
+    retrying those with a different CA list changes nothing except how long
+    the operator waits for the real message.
+    """
+    calls = []
+
     def fake(url, timeout=None, context=None):
+        calls.append(context)
         raise urllib.error.URLError("no route to host")
 
     monkeypatch.setattr(urllib.request, "urlopen", fake)
     with pytest.raises(urllib.error.URLError):
         model_assets._urlopen("https://example.invalid/m.onnx")
+    assert calls == [None], f"a dead network was retried as a CA problem: {calls}"
 
 
 def test_progress_lines_survive_the_rewrite(tmp_path, monkeypatch, caplog):
@@ -103,3 +123,42 @@ def test_progress_lines_survive_the_rewrite(tmp_path, monkeypatch, caplog):
     pct = [ln for ln in lines if ln.startswith("download: face detector ")]
     assert pct, f"no progress lines at all: {lines}"
     assert "download: face detector 100%" in pct, pct
+
+
+@pytest.mark.skipif(not os.environ.get("BEHAVISION_NETWORK_TESTS"),
+                    reason="set BEHAVISION_NETWORK_TESTS=1 to reach github.com")
+def test_against_a_real_machine_with_no_trust_store(tmp_path, monkeypatch):
+    """The whole thing, over the real network, with the real failure.
+
+    Every stub above is a statement about what I believe urllib does, and the
+    first version of this file proved how much that is worth. Python's default
+    context honours SSL_CERT_FILE, so pointing it at an EMPTY file reproduces
+    a python.org macOS build exactly: a default context that trusts nobody.
+    certifi is loaded by path and is unaffected by the variable, so if the
+    fallback works here it works there.
+    """
+    empty = tmp_path / "no-cas.pem"
+    empty.write_text("")
+    monkeypatch.setenv("SSL_CERT_FILE", str(empty))
+    monkeypatch.setenv("SSL_CERT_DIR", str(tmp_path / "nothing"))
+
+    # Not every interpreter can be put into that state, and pretending
+    # otherwise would turn this into a test that passes by not running. A
+    # Python linked against LibreSSL - which is what macOS Command Line Tools
+    # ships - reads the system keychain and ignores SSL_CERT_FILE entirely:
+    # measured, 128 CAs loaded with the variable pointing at an empty file. A
+    # python.org build links OpenSSL and honours it: 0 CAs, which is the
+    # condition being reproduced.
+    if ssl.create_default_context().get_ca_certs():
+        pytest.skip("this interpreter ignores SSL_CERT_FILE (%s), so the "
+                    "no-trust-store condition cannot be reproduced here"
+                    % ssl.OPENSSL_VERSION)
+
+    # The premise: the default context really is broken in this process.
+    with pytest.raises(urllib.error.URLError) as caught:
+        urllib.request.urlopen(model_assets.YUNET_URL, timeout=30)
+    assert model_assets._is_cert_failure(caught.value), caught.value
+
+    dest = tmp_path / "yunet.onnx"
+    model_assets._fetch(model_assets.YUNET_URL, dest, "face detector")
+    assert dest.stat().st_size > 200_000