Compare commits
3 Commits
v0.5.4-dem
...
v0.5.7-dem
| Author | SHA1 | Date | |
|---|---|---|---|
| 0558344dc2 | |||
| 8f07dee048 | |||
| 3cddd9c2e1 |
140
CLAUDE.md
140
CLAUDE.md
@@ -3559,3 +3559,143 @@ one, and why installing into it would not survive a restart. Fixed by dragging
|
||||
the app to Applications; saying nothing leaves somebody re-running a setup tool
|
||||
that cannot win. The product is unsigned, so this is the *normal* first-run
|
||||
state on every Mac, not an edge case.
|
||||
|
||||
### And then it could not download a 230 KB file
|
||||
|
||||
With all of the above fixed the install succeeded on that Mac - Python 3.14
|
||||
chosen and accepted, numpy 2.5.3, onnxruntime 1.30, faiss 1.15.1, the engine
|
||||
itself - and setup died on the last step, fetching the YuNet model:
|
||||
|
||||
```
|
||||
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
|
||||
certificate verify failed: unable to get local issuer certificate
|
||||
```
|
||||
|
||||
A python.org macOS build ships its **own OpenSSL with no trust store**, and
|
||||
populates one only when somebody double-clicks `Install Certificates.command`
|
||||
in the Python folder. Nobody installing face-recognition software has any
|
||||
reason to know that exists, and the failure is forty lines of traceback about
|
||||
`_ssl.c` at the end of a ten-minute install.
|
||||
|
||||
`_urlopen` tries the default context first and retries with **certifi's**
|
||||
bundle on a verification failure. The order is the whole design:
|
||||
|
||||
- Default first, because on Windows and on a system or Homebrew Python the
|
||||
default context reads the machine's own certificate store - which is what
|
||||
makes a corporate proxy with its own root CA work. Replacing it
|
||||
unconditionally would break every site that has one in order to fix a
|
||||
different platform.
|
||||
- certifi second, because it is already installed: `requests` is a hard
|
||||
dependency and brings it.
|
||||
- A `URLError` is re-raised untouched. "No route to host" and "no trust store"
|
||||
are different problems, and retrying the first with a different CA list only
|
||||
delays the real message.
|
||||
|
||||
`urlretrieve` had to go, since it offers no way to pass a context - and that is
|
||||
exactly the kind of rewrite that silently drops something. The
|
||||
`download: <label> <n>%` lines are a **contract**: `supervisor.go`'s
|
||||
`progressRe` parses them to put first-run progress in the tray, because the API
|
||||
is not up yet and a shop PC showing a stopped engine for five minutes after
|
||||
install looks broken. `tests/test_model_download.py` asserts them, and the
|
||||
rewritten fetch was checked against the real URL: 232,589 bytes, sha256
|
||||
identical to the model already on disk.
|
||||
|
||||
## The engine stopped updating, and said "installed" every time
|
||||
|
||||
The SSL fix above was released and **did not reach the machine it was written
|
||||
for**. Its log said so, one line above the tick:
|
||||
|
||||
```
|
||||
behavision is already installed with the same version as the provided wheel.
|
||||
Use --force-reinstall to force an installation of the wheel.
|
||||
[ok] Engine and dependencies installed
|
||||
```
|
||||
|
||||
and the traceback that followed still pointed at `model_assets.py", line 59,
|
||||
in _fetch / urllib.request.urlretrieve` — code the fix had deleted.
|
||||
|
||||
Two frozen literals caused it: `version = "1.1.0"` in `pyproject.toml` and
|
||||
`__version__ = "1.0.0"` in `behavision/__init__.py`. They disagreed with each
|
||||
other and neither tracked a release, so **every release built
|
||||
`behavision-1.1.0-py3-none-any.whl`** and `pip install --upgrade` on a machine
|
||||
that already had 1.1.0 is a no-op. The comment beside that call even claimed
|
||||
the opposite: *"`--upgrade` so re-running after a new release replaces the
|
||||
engine rather than leaving the old one in place and reporting success."*
|
||||
|
||||
The shape of the damage is what makes it bad. The Go binaries — the app, the
|
||||
agent, the setup tool — are rebuilt every release and updated normally. So a
|
||||
shop PC ran a current app supervising an engine several releases old, and
|
||||
nothing anywhere said which: `/api/health` reported the model, the paths, the
|
||||
cameras and the gallery, and **no version at all**.
|
||||
|
||||
Three changes, and the second is the one that does not depend on the first
|
||||
being remembered:
|
||||
|
||||
- **One version, in the package**, read by `pyproject.toml` through
|
||||
`[tool.setuptools.dynamic]`. Two literals in two files is how they came to
|
||||
disagree. In a checkout it reads `0.0.0+dev`, because a plausible-looking
|
||||
number on a developer's `/api/health` would be worse than none.
|
||||
- **`pip install --force-reinstall --no-deps <wheel>`, after the ordinary
|
||||
`--upgrade`.** `--upgrade` settles the dependencies; the second call
|
||||
guarantees our own code is the code in the folder. `--no-deps` is what keeps
|
||||
it cheap — forcing the dependencies too would re-download ~300 MB of numpy,
|
||||
OpenCV and onnxruntime on every run. A rebuild at an unchanged version is the
|
||||
ordinary case while developing, so this must not rely on the version moving.
|
||||
- **`release.sh` stamps the tag**: `v0.5.6-demo` → `0.5.6+demo`, valid PEP 440
|
||||
(a local segment takes alphanumerics and dots, never hyphens). Into a copy of
|
||||
the line, reverted in a `trap`, so the tree is never left dirty.
|
||||
|
||||
`/api/health` reports `version` now. Without it there is no way to tell a shop
|
||||
PC three releases behind from a current one, which is precisely how this
|
||||
survived several releases.
|
||||
|
||||
Reproduced end to end before fixing, against real wheels on Python 3.12: two
|
||||
builds of the same version, `--upgrade` leaves the old code in place and prints
|
||||
the same sentence the colleague's Mac printed, `--force-reinstall --no-deps`
|
||||
replaces it.
|
||||
|
||||
### `urllib` does not let `SSLCertVerificationError` out, and a fake said it did
|
||||
|
||||
The certificate fix above shipped and **failed on the machine it was written
|
||||
for, with the exact traceback it was meant to prevent**. The retry was written
|
||||
|
||||
```python
|
||||
except ssl.SSLCertVerificationError:
|
||||
```
|
||||
|
||||
and urllib never raises that from `urlopen`. It catches it and re-raises
|
||||
`urllib.error.URLError(err)`, carrying the original on `.reason`. So the
|
||||
`except` matched nothing, ever, and the fallback could not fire.
|
||||
|
||||
The unit test passed throughout, because the stub it used raised the bare SSL
|
||||
error — **a shape real urllib never produces**. That is the whole lesson: a
|
||||
fake that agrees with the author is worse than no test, because it converts an
|
||||
untested path into a tested-looking one. The same sentence is already in this
|
||||
file about `UPDATE ... RETURNING` and about the in-memory API fake, and it was
|
||||
written again here anyway.
|
||||
|
||||
`_is_cert_failure` checks the exception **and** its `.reason`, and the tests
|
||||
now raise `URLError(SSLCertVerificationError(...))`, which is what his
|
||||
traceback shows. A plain `URLError` — "no route to host" — is re-raised
|
||||
untouched: retrying that with a different CA list changes nothing except how
|
||||
long the operator waits for the real message, and a test asserts the second
|
||||
attempt never happens.
|
||||
|
||||
Beside the stubs there is now a **real** reproduction,
|
||||
`test_against_a_real_machine_with_no_trust_store`, opt-in behind
|
||||
`BEHAVISION_NETWORK_TESTS=1` because it reaches github.com. Python's default
|
||||
context honours `SSL_CERT_FILE`, so pointing it at an empty file gives a
|
||||
default context that trusts nobody — the python.org condition exactly — while
|
||||
certifi is loaded by path and is unaffected.
|
||||
|
||||
It skips rather than passes where it cannot reproduce that, and the difference
|
||||
is measured rather than assumed:
|
||||
|
||||
```
|
||||
macOS Command Line Tools LibreSSL 2.8.3 128 CAs with SSL_CERT_FILE empty
|
||||
(reads the system keychain)
|
||||
python.org / pyenv build OpenSSL 3.5.8 0 CAs -> reproduces it
|
||||
```
|
||||
|
||||
Checked for teeth by putting the shipped `except` back: both the corrected stub
|
||||
test and the live one fail, and pass again when it is restored.
|
||||
|
||||
@@ -539,7 +539,30 @@ func pipInstall(vpy, src string) error {
|
||||
// 'behavision.egg-info': Read-only file system". Falling back to source
|
||||
// copies it somewhere writable first, for the same reason.
|
||||
if wheels, _ := filepath.Glob(filepath.Join(src, "behavision-*.whl")); len(wheels) > 0 {
|
||||
return stream(exec.Command(vpy, "-m", "pip", "install", "--upgrade", wheels[0]),
|
||||
// TWO calls, and the second is the one that matters.
|
||||
//
|
||||
// `--upgrade` alone is not an upgrade when the version has not moved:
|
||||
// pip skips the wheel and says so, one line above this program
|
||||
// printing "[ok] Engine and dependencies installed". The wheel version
|
||||
// was a frozen literal for several releases, so every engine fix in
|
||||
// them silently failed to reach any machine that had run setup once -
|
||||
// while the Go binaries beside it, rebuilt every release, updated
|
||||
// normally. Half the product current, half of it months old, and
|
||||
// nothing saying which.
|
||||
//
|
||||
// release.sh stamps the tag into the version now, so the versions do
|
||||
// differ. This does not rely on that: a rebuild at the same version is
|
||||
// the ordinary case while developing, and "installed" has to mean the
|
||||
// code in this folder either way.
|
||||
if err := stream(exec.Command(vpy, "-m", "pip", "install", "--upgrade", wheels[0]),
|
||||
"installing the engine"); err != nil {
|
||||
return err
|
||||
}
|
||||
// --no-deps so this is our own package only: the call above has
|
||||
// already settled the dependencies, and forcing those too would
|
||||
// re-download ~300 MB of numpy, OpenCV and onnxruntime every run.
|
||||
return stream(exec.Command(vpy, "-m", "pip", "install",
|
||||
"--force-reinstall", "--no-deps", wheels[0]),
|
||||
"installing the engine")
|
||||
}
|
||||
tmp, err := os.MkdirTemp("", "behavision-src-")
|
||||
|
||||
@@ -4,4 +4,15 @@ Pipeline: capture -> detect (YuNet) -> track (IoU) -> align + encode
|
||||
(ArcFace ONNX) -> match / auto-enroll (FAISS + SQLite) -> events + API.
|
||||
"""
|
||||
|
||||
__version__ = "1.0.0"
|
||||
# Stamped by release.sh from the git tag, into a copy of this line, so a
|
||||
# release always produces a wheel nobody has installed before. It was a frozen
|
||||
# "1.0.0" here and a frozen "1.1.0" in pyproject.toml - two literals that
|
||||
# disagreed with each other and tracked nothing - which is how every engine fix
|
||||
# for several releases silently failed to reach a machine that had run setup
|
||||
# once. pip skips a wheel whose version is already installed and says so, one
|
||||
# line above setup printing "[ok] Engine and dependencies installed".
|
||||
#
|
||||
# In a checkout it stays obviously a checkout: "0.0.0+dev" on /api/health is
|
||||
# the truth about a developer's machine, and a plausible-looking number there
|
||||
# would be worse than none.
|
||||
__version__ = "0.0.0+dev"
|
||||
|
||||
@@ -17,6 +17,7 @@ from fastapi.responses import HTMLResponse, Response, StreamingResponse
|
||||
from fastapi.security import HTTPBasic, HTTPBasicCredentials
|
||||
from pydantic import BaseModel, ValidationError
|
||||
|
||||
from . import __version__
|
||||
from .config import ApiSection, CameraConfig, CameraTuning
|
||||
from .commission import CommissionRun
|
||||
from .events import Event
|
||||
@@ -142,7 +143,7 @@ def _reencode(jpeg: bytes, width: int, quality: int) -> "bytes | None":
|
||||
|
||||
|
||||
def create_app(engine: Engine) -> FastAPI:
|
||||
app = FastAPI(title="Behavision", version="1.0.0",
|
||||
app = FastAPI(title="Behavision", version=__version__,
|
||||
dependencies=_auth_dependencies(engine.cfg.api))
|
||||
|
||||
def worker_or_404(camera_id: str):
|
||||
@@ -172,6 +173,10 @@ def create_app(engine: Engine) -> FastAPI:
|
||||
# are up, and the shop recognises nobody it already knows.
|
||||
stranded = engine.gallery.health["stranded"]
|
||||
return {"status": "ok" if engine.started_at else "starting",
|
||||
# Which build is actually running. Without it there was no way
|
||||
# to tell a shop PC three releases behind from a current one -
|
||||
# which is precisely how a silent install failure survived.
|
||||
"version": __version__,
|
||||
"recognition_model": engine.encoder.model_name,
|
||||
"gallery_unreadable_embeddings": stranded,
|
||||
# "where is my database" must be answerable from the API: the
|
||||
|
||||
@@ -4,6 +4,8 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import shutil
|
||||
import ssl
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
@@ -35,6 +37,81 @@ _COPY_MAP = {
|
||||
}
|
||||
|
||||
|
||||
def _https_context() -> "ssl.SSLContext | None":
|
||||
"""The CA store to trust, or None to use whatever Python defaults to.
|
||||
|
||||
Returning None first is deliberate. On Windows and on a Homebrew or
|
||||
system Python, the default context reads the machine's own certificate
|
||||
store - which is what makes a corporate proxy with its own root CA work.
|
||||
Replacing that with certifi's bundle unconditionally would break every
|
||||
site that has one, in order to fix a different platform.
|
||||
|
||||
The platform this fixes is a python.org macOS build. It ships its own
|
||||
OpenSSL with NO trust store, and populates one only when somebody
|
||||
double-clicks `Install Certificates.command` in the Python folder -
|
||||
which nobody installing face-recognition software has any reason to know
|
||||
about. Every HTTPS request from that interpreter fails with:
|
||||
|
||||
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
|
||||
certificate verify failed: unable to get local issuer certificate
|
||||
|
||||
Measured on a colleague's Mac: the engine installed perfectly and then
|
||||
could not download a 230 KB model file, ending setup in forty lines of
|
||||
traceback about `_ssl.c`.
|
||||
"""
|
||||
try:
|
||||
import certifi
|
||||
except ImportError: # pragma: no cover - certifi ships with requests
|
||||
return None
|
||||
return ssl.create_default_context(cafile=certifi.where())
|
||||
|
||||
|
||||
def _is_cert_failure(err: BaseException) -> bool:
|
||||
"""Is this a certificate-verification failure, however it is wrapped?
|
||||
|
||||
`urllib` does NOT let `ssl.SSLCertVerificationError` out. It catches it and
|
||||
re-raises `urllib.error.URLError(err)`, carrying the original on `.reason`
|
||||
- so `except ssl.SSLCertVerificationError` around `urlopen` matches
|
||||
nothing, ever.
|
||||
|
||||
That is not a subtlety this file gets to record academically: the first
|
||||
version of the fallback below was written exactly that way, shipped, and
|
||||
failed on the machine it was written for with the very traceback it was
|
||||
meant to prevent. The unit test passed throughout, because the fake it
|
||||
used raised the bare SSL error - a shape real urllib never produces. A
|
||||
stub that agrees with the author is worse than no test, and the test now
|
||||
raises what urllib raises.
|
||||
"""
|
||||
reason = getattr(err, "reason", None)
|
||||
return isinstance(err, ssl.SSLCertVerificationError) or \
|
||||
isinstance(reason, ssl.SSLCertVerificationError)
|
||||
|
||||
|
||||
def _urlopen(url: str, timeout: float = 60.0):
|
||||
"""Open a URL, falling back to certifi's CA bundle on a verify failure.
|
||||
|
||||
Default first, certifi second, so the fix is additive: a machine whose
|
||||
own store works keeps using it, and one with no store at all gets a
|
||||
bundle rather than a traceback. certifi is already here - `requests` is a
|
||||
hard dependency and brings it.
|
||||
"""
|
||||
try:
|
||||
return urllib.request.urlopen(url, timeout=timeout)
|
||||
except (urllib.error.URLError, ssl.SSLCertVerificationError) as err:
|
||||
# Only a certificate problem is worth a second attempt. "No route to
|
||||
# host" and "connection refused" arrive as URLError too, and retrying
|
||||
# those with a different CA list changes nothing except how long the
|
||||
# operator waits for the real message.
|
||||
if not _is_cert_failure(err):
|
||||
raise
|
||||
ctx = _https_context()
|
||||
if ctx is None:
|
||||
raise
|
||||
log.info("the system certificate store could not verify %s; "
|
||||
"using the bundled CA list", url.split("/")[2])
|
||||
return urllib.request.urlopen(url, timeout=timeout, context=ctx)
|
||||
|
||||
|
||||
def _fetch(url: str, dest: Path, label: str) -> None:
|
||||
"""Download with progress on stdout the supervisor can read.
|
||||
|
||||
@@ -56,7 +133,22 @@ def _fetch(url: str, dest: Path, label: str) -> None:
|
||||
last = pct
|
||||
log.info("download: %s %d%%", label, pct)
|
||||
|
||||
urllib.request.urlretrieve(url, dest, hook)
|
||||
# Streamed rather than urlretrieve, only because urlretrieve offers no way
|
||||
# to pass an SSL context and the whole point here is choosing one. The
|
||||
# `download: <label> <n>%` lines are a contract: the supervisor parses
|
||||
# them (`progressRe`) to put first-run progress in the tray, and without
|
||||
# them a shop PC shows a stopped engine for five minutes after install.
|
||||
with _urlopen(url) as resp:
|
||||
total = int(resp.headers.get("Content-Length") or 0)
|
||||
blocks, block_size = 0, 64 * 1024
|
||||
with open(dest, "wb") as out:
|
||||
while True:
|
||||
chunk = resp.read(block_size)
|
||||
if not chunk:
|
||||
break
|
||||
out.write(chunk)
|
||||
blocks += 1
|
||||
hook(blocks, block_size, total)
|
||||
log.info("download: %s 100%%", label)
|
||||
|
||||
|
||||
@@ -91,7 +183,7 @@ def setup_models(models_dir: Path) -> "list[str]":
|
||||
import io
|
||||
import zipfile
|
||||
|
||||
with urllib.request.urlopen(BUFFALO_SC_URL) as resp:
|
||||
with _urlopen(BUFFALO_SC_URL) as resp:
|
||||
payload = io.BytesIO(resp.read())
|
||||
with zipfile.ZipFile(payload) as zf, \
|
||||
zf.open("w600k_mbf.onnx") as src, \
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "behavision"
|
||||
version = "1.1.0"
|
||||
dynamic = ["version"]
|
||||
description = "Production face recognition over RTSP"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
@@ -32,3 +32,9 @@ behavision = ["static/*"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
|
||||
# One version for the package and the wheel. Two literals in two files is how
|
||||
# they came to disagree (1.0.0 here, 1.1.0 there) and how neither tracked a
|
||||
# release.
|
||||
[tool.setuptools.dynamic]
|
||||
version = {attr = "behavision.__version__"}
|
||||
|
||||
21
release.sh
21
release.sh
@@ -50,7 +50,28 @@ step "2. Agent and setup tool"
|
||||
step "3. Engine source and wheel"
|
||||
# The wheel is built with the checkout's own interpreter; requires-python is a
|
||||
# statement about the SHOP PC, which setup enforces when it finds Python there.
|
||||
# Stamp the tag into the wheel version. Without this every release built
|
||||
# behavision-1.1.0-py3-none-any.whl, and `pip install --upgrade` on a machine
|
||||
# that already had 1.1.0 is a no-op - so an engine fix reached nobody who had
|
||||
# ever run setup, while the Go binaries beside it updated normally. Nothing
|
||||
# reported a version either, so there was no way to tell a shop PC three
|
||||
# releases behind from a current one.
|
||||
#
|
||||
# v0.5.6-demo -> 0.5.6+demo, which is valid PEP 440: a local segment takes
|
||||
# alphanumerics and dots, never hyphens.
|
||||
TMPVER=$(mktemp)
|
||||
PEP440=$(printf '%s' "${TAG#v}" | sed 's/-/+/; s/[^0-9A-Za-z.+]/./g')
|
||||
trap 'git checkout -- behavision/__init__.py 2>/dev/null || true; rm -f "$TMPVER"' EXIT
|
||||
# A temp file rather than `sed -i`, whose argument handling differs between
|
||||
# BSD and GNU - this script is run from a Mac today and that is not a reason
|
||||
# to plant a portability trap in a release path.
|
||||
sed "s/^__version__ = .*/__version__ = \"$PEP440\"/" behavision/__init__.py > "$TMPVER"
|
||||
cat "$TMPVER" > behavision/__init__.py
|
||||
.venv/bin/python -m pip wheel --no-deps --ignore-requires-python -q -w "$STAGE/engine-src" . 2>&1 | grep -v "DEPRECATION\|WARNING: Ignoring" || true
|
||||
git checkout -- behavision/__init__.py
|
||||
rm -f "$TMPVER"
|
||||
trap - EXIT
|
||||
echo " engine wheel version $PEP440"
|
||||
ls "$STAGE"/engine-src/behavision-*.whl >/dev/null || { echo "wheel was not built" >&2; exit 1; }
|
||||
cp pyproject.toml requirements.txt "$STAGE/engine-src/"
|
||||
mkdir -p "$STAGE/engine-src/config" && cp config/default.yaml "$STAGE/engine-src/config/"
|
||||
|
||||
164
tests/test_model_download.py
Normal file
164
tests/test_model_download.py
Normal file
@@ -0,0 +1,164 @@
|
||||
"""Downloading the models is the last step of every fresh install, and it ran
|
||||
into the one macOS trap nothing else here does.
|
||||
|
||||
A python.org macOS build ships its own OpenSSL with NO trust store, and
|
||||
populates one only when somebody double-clicks `Install Certificates.command`
|
||||
in the Python folder. Measured on a colleague's Mac: the engine installed
|
||||
perfectly - numpy, onnxruntime, faiss, all of it - and then could not fetch a
|
||||
230 KB model file, ending setup in forty lines of traceback about `_ssl.c`.
|
||||
"""
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
import ssl
|
||||
import urllib.request
|
||||
|
||||
import pytest
|
||||
|
||||
from behavision import model_assets
|
||||
|
||||
|
||||
class _Resp(io.BytesIO):
|
||||
"""Enough of an http response for _fetch: read() and .headers."""
|
||||
|
||||
def __init__(self, payload: bytes):
|
||||
super().__init__(payload)
|
||||
self.headers = {"Content-Length": str(len(payload))}
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc):
|
||||
self.close()
|
||||
return False
|
||||
|
||||
|
||||
def _verify_error():
|
||||
"""What urlopen ACTUALLY raises, which is not what it looks like.
|
||||
|
||||
urllib catches ssl.SSLCertVerificationError and re-raises
|
||||
urllib.error.URLError(err), carrying the original on `.reason`. The first
|
||||
version of this test raised the bare SSL error - a shape real urllib never
|
||||
produces - so it passed against a fallback that could never fire, and the
|
||||
fix shipped and failed on the machine it was written for with the exact
|
||||
traceback it was meant to prevent.
|
||||
"""
|
||||
return urllib.error.URLError(ssl.SSLCertVerificationError(
|
||||
"[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: "
|
||||
"unable to get local issuer certificate"))
|
||||
|
||||
|
||||
def test_a_machine_with_no_trust_store_falls_back_to_the_bundled_one(monkeypatch):
|
||||
calls = []
|
||||
|
||||
def fake(url, timeout=None, context=None):
|
||||
calls.append(context)
|
||||
if context is None:
|
||||
raise _verify_error()
|
||||
return _Resp(b"ok")
|
||||
|
||||
monkeypatch.setattr(urllib.request, "urlopen", fake)
|
||||
with model_assets._urlopen("https://example.invalid/m.onnx") as resp:
|
||||
assert resp.read() == b"ok"
|
||||
|
||||
assert len(calls) == 2, f"expected a retry, got {calls}"
|
||||
# Default FIRST, and that order is the point. On Windows and on a system
|
||||
# Python the default context reads the machine's own certificate store,
|
||||
# which is what makes a corporate proxy with its own root CA work.
|
||||
# Replacing it unconditionally would break every site that has one in
|
||||
# order to fix a different platform.
|
||||
assert calls[0] is None
|
||||
assert isinstance(calls[1], ssl.SSLContext)
|
||||
|
||||
|
||||
def test_a_working_trust_store_is_used_as_is(monkeypatch):
|
||||
calls = []
|
||||
|
||||
def fake(url, timeout=None, context=None):
|
||||
calls.append(context)
|
||||
return _Resp(b"ok")
|
||||
|
||||
monkeypatch.setattr(urllib.request, "urlopen", fake)
|
||||
model_assets._urlopen("https://example.invalid/m.onnx").close()
|
||||
assert calls == [None], "the machine's own certificate store was bypassed"
|
||||
|
||||
|
||||
def test_a_real_network_failure_is_not_disguised_as_a_certificate_problem(monkeypatch):
|
||||
"""A URLError is not automatically a certificate problem.
|
||||
|
||||
"No route to host" and "connection refused" arrive as URLError too, and
|
||||
retrying those with a different CA list changes nothing except how long
|
||||
the operator waits for the real message.
|
||||
"""
|
||||
calls = []
|
||||
|
||||
def fake(url, timeout=None, context=None):
|
||||
calls.append(context)
|
||||
raise urllib.error.URLError("no route to host")
|
||||
|
||||
monkeypatch.setattr(urllib.request, "urlopen", fake)
|
||||
with pytest.raises(urllib.error.URLError):
|
||||
model_assets._urlopen("https://example.invalid/m.onnx")
|
||||
assert calls == [None], f"a dead network was retried as a CA problem: {calls}"
|
||||
|
||||
|
||||
def test_progress_lines_survive_the_rewrite(tmp_path, monkeypatch, caplog):
|
||||
"""`download: <label> <n>%` is a CONTRACT, not logging.
|
||||
|
||||
The supervisor parses it (progressRe) to put first-run progress in the
|
||||
tray and the window, because the API is not up yet and a shop PC showing
|
||||
a stopped engine for five minutes after install looks broken. Switching
|
||||
off urlretrieve - needed because it offers no way to pass an SSL context -
|
||||
is exactly the kind of change that drops it silently.
|
||||
"""
|
||||
payload = b"x" * (64 * 1024 * 8)
|
||||
monkeypatch.setattr(urllib.request, "urlopen",
|
||||
lambda url, timeout=None, context=None: _Resp(payload))
|
||||
dest = tmp_path / "model.onnx"
|
||||
with caplog.at_level(logging.INFO, logger="behavision.model_assets"):
|
||||
model_assets._fetch("https://example.invalid/m.onnx", dest, "face detector")
|
||||
|
||||
assert dest.read_bytes() == payload
|
||||
lines = [r.getMessage() for r in caplog.records]
|
||||
pct = [ln for ln in lines if ln.startswith("download: face detector ")]
|
||||
assert pct, f"no progress lines at all: {lines}"
|
||||
assert "download: face detector 100%" in pct, pct
|
||||
|
||||
|
||||
@pytest.mark.skipif(not os.environ.get("BEHAVISION_NETWORK_TESTS"),
|
||||
reason="set BEHAVISION_NETWORK_TESTS=1 to reach github.com")
|
||||
def test_against_a_real_machine_with_no_trust_store(tmp_path, monkeypatch):
|
||||
"""The whole thing, over the real network, with the real failure.
|
||||
|
||||
Every stub above is a statement about what I believe urllib does, and the
|
||||
first version of this file proved how much that is worth. Python's default
|
||||
context honours SSL_CERT_FILE, so pointing it at an EMPTY file reproduces
|
||||
a python.org macOS build exactly: a default context that trusts nobody.
|
||||
certifi is loaded by path and is unaffected by the variable, so if the
|
||||
fallback works here it works there.
|
||||
"""
|
||||
empty = tmp_path / "no-cas.pem"
|
||||
empty.write_text("")
|
||||
monkeypatch.setenv("SSL_CERT_FILE", str(empty))
|
||||
monkeypatch.setenv("SSL_CERT_DIR", str(tmp_path / "nothing"))
|
||||
|
||||
# Not every interpreter can be put into that state, and pretending
|
||||
# otherwise would turn this into a test that passes by not running. A
|
||||
# Python linked against LibreSSL - which is what macOS Command Line Tools
|
||||
# ships - reads the system keychain and ignores SSL_CERT_FILE entirely:
|
||||
# measured, 128 CAs loaded with the variable pointing at an empty file. A
|
||||
# python.org build links OpenSSL and honours it: 0 CAs, which is the
|
||||
# condition being reproduced.
|
||||
if ssl.create_default_context().get_ca_certs():
|
||||
pytest.skip("this interpreter ignores SSL_CERT_FILE (%s), so the "
|
||||
"no-trust-store condition cannot be reproduced here"
|
||||
% ssl.OPENSSL_VERSION)
|
||||
|
||||
# The premise: the default context really is broken in this process.
|
||||
with pytest.raises(urllib.error.URLError) as caught:
|
||||
urllib.request.urlopen(model_assets.YUNET_URL, timeout=30)
|
||||
assert model_assets._is_cert_failure(caught.value), caught.value
|
||||
|
||||
dest = tmp_path / "yunet.onnx"
|
||||
model_assets._fetch(model_assets.YUNET_URL, dest, "face detector")
|
||||
assert dest.stat().st_size > 200_000
|
||||
Reference in New Issue
Block a user