Krow/krow_backend
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add evals for every shipped agent, a policy corpus, and CI
Some checks failed
CI / test (push) Has been cancelled
Details
CI / fixture (push) Has been cancelled
Details

Browse Source 
§9 says no agent ships without evals. Eight of the nine had none: the two
other suites in evals/ are harness fixtures rather than agents in the
registry, so the rule was being met by one agent in nine.

Evals — 40 new cases, five per agent, every one carrying mustNotLeak:

  - the agent is loaded from its real spec in agents/*.md rather than
    written out again in Go. A hand-copied agent tests the copy: it keeps
    passing after somebody edits the spec, which is the moment it most
    needed to fail.
  - callNamed calls the tool a case names. toolThenAnswer always called
    tools[0], so seven of positions-agent's eight tools were unreachable,
    and a boundary nothing calls is a boundary nothing tests.
  - seedWorkspace fills BOTH tenants. A leak test against an empty second
    tenant cannot fail.

Verified by breaking workersByScore's org predicate: six cases across four
agents fail with LEAKED "RIVAL".

Knowledge — six policy documents, taking the corpus from 2 to 8 (34
chunks). Three restricted to admin and employer, five tenant-wide. They
cover what the tools cannot: a tool reports how many shifts went unworked,
a policy says what cover costs inside 24 hours.

corpus_test.go treats those documents as product rather than fixtures. The
first version was tautological — it read audience: from a file and checked
that file's audience was enforced, so opening a restricted document passed.
mustNotBeTenantWide now holds that judgement apart from the files, with the
reason recorded for each.

CI — the checks this repository already had, made unskippable. testutil
calls t.Skipf on an unreachable database, so a dead service container would
produce a green build over a suite that ran almost nothing. Simulated: go
test exits 0 with 74 tests skipped, including every tenant-isolation test.
The guard exits 1 and names them, while still allowing TestLive* to skip
without a model key.

This CI tests; it does not deploy. The README's claim that migrations are
run by CI against the target database remains aspirational.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0186JgqQUCDS8ZwGmyw3ymWu
This commit is contained in:
Aravind
2026-08-28 13:52:46 +05:30
parent f7df96c973
commit a222dcd3e4
17 changed files with 2014 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

149
.github/workflows/ci.yml vendored Normal file
View File

@@ -0,0 +1,149 @@
name: CI
# Every check this repository already had, run on every push instead of when
# somebody remembers. Nothing here is new verification — it is the verification
# that existed, made unskippable.
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
jobs:
test:
runs-on: ubuntu-latest
# The tests SKIP when PostgreSQL is unreachable — see testutil.New, which
# calls t.Skipf rather than failing, so a developer without a database can
# still run the non-database tests. In CI that behaviour is a trap: a broken
# service container would produce a green build over a suite that tested
# almost nothing. The guard at the end of this job is what closes it.
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_PASSWORD: postgres
POSTGRES_USER: postgres
POSTGRES_DB: postgres
ports: ['5432:5432']
options: >-
--health-cmd "pg_isready -U postgres"
--health-interval 5s
--health-timeout 5s
--health-retries 20
env:
DATABASE_HOST: 127.0.0.1
DATABASE_PORT: '5432'
DATABASE_USER: postgres
DATABASE_PASSWORD: postgres
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: go-api/go.mod
cache-dependency-path: go-api/go.sum
- name: gofmt
working-directory: go-api
run: |
unformatted="$(gofmt -l ./cmd ./internal)"
if [ -n "$unformatted" ]; then
echo "not gofmt'd:"; echo "$unformatted"; exit 1
fi
- name: go vet
working-directory: go-api
run: go vet ./...
- name: Tests
working-directory: go-api
# -json so the guard below can count what actually ran, and `|| true` so
# a failure reaches that guard rather than ending the job here — the
# guard reports which tests failed, which the raw JSON does not.
run: go test ./... -count=1 -json > /tmp/test.json || true
- name: Fail if the database tests skipped
# The point of this job. testutil skips on an unreachable database, so
# "0 failures" is not the same as "the suite ran": a service container
# that never came up would otherwise look identical to a passing build.
run: |
python3 - <<'PY'
import json, sys
skipped, passed, failed = [], 0, []
for line in open('/tmp/test.json'):
line = line.strip()
if not line.startswith('{'):
continue
try:
e = json.loads(line)
except ValueError:
continue
if e.get('Action') == 'skip' and e.get('Test'):
skipped.append(e['Test'])
if e.get('Action') == 'pass' and e.get('Test'):
passed += 1
if e.get('Action') == 'fail' and e.get('Test'):
failed.append(e['Test'])
print(f"{passed} passed, {len(failed)} failed, {len(skipped)} skipped")
if failed:
print("FAILED:"); [print(" ", t) for t in failed[:40]]
sys.exit(1)
# A Test<Pkg>Live* / TestLive* test skips without ANTHROPIC_API_KEY, which
# is correct here: CI should not spend tokens on every push, and the key
# should not be present unless somebody put it there deliberately. Any
# OTHER skip means the database was unreachable, and that is the case
# this guard exists for — testutil calls t.Skipf rather than failing, so
# a dead service container would otherwise look exactly like a pass.
unexpected = [t for t in skipped if not t.startswith('TestLive')]
if skipped:
print("skipped:"); [print(" ", t) for t in skipped[:40]]
if unexpected:
print("\nThese are not live-model tests, so they skipped because the")
print("database was unreachable. A green build over a suite that did")
print("not run is worse than a red one.")
sys.exit(1)
if passed < 200:
sys.exit(f"only {passed} tests passed; the suite is far smaller than expected — did it run?")
PY
- name: Migrations are reversible
# §10: one migration per PR, reversible. Applying and rolling back the
# newest one is the cheapest way to find out that it is not.
working-directory: go-api
run: go test ./internal/repo/... -run 'Migration' -count=1
fixture:
# seed.json is generated from the frontend's seed module. The two used to be
# hand-maintained copies, and drift between them is silent: the demo and the
# API answer the same question with different numbers.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Check out the frontend beside this repo
uses: actions/checkout@v4
with:
repository: ${{ github.repository_owner }}/krow-demo
path: ../krow-demo
# A private sibling needs a token with read access; without one this
# job reports that it could not check, rather than passing quietly.
token: ${{ secrets.FRONTEND_REPO_TOKEN }}
continue-on-error: true
- uses: actions/setup-node@v4
with:
node-version: '20'
- name: seed.json matches the frontend seed module
run: |
if [ ! -d ../krow-demo ]; then
echo "krow-demo is not available to this job, so the fixture could not be checked."
echo "Set FRONTEND_REPO_TOKEN to enable it. Not passing silently."
exit 1
fi
cd ../krow-demo && npm ci && npm run seed:check

128
evals/analytics-agent.json Normal file
View File

@@ -0,0 +1,128 @@
{
"agent": "analytics-agent",
"cases": [
{
"id": "workspace-summary-basic",
"input": "How is hiring performing overall?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workspace_summary"
]
}
},
{
"id": "attendance-reads-this-tenant-only",
"input": "What does attendance look like?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workforce_attendance"
],
"mustMention": [
"Ours"
]
}
},
{
"id": "overtime-is-scoped",
"input": "Is overtime a problem?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workforce_overtime"
]
}
},
{
"id": "candidate-quality-is-scoped",
"input": "What is the quality of our candidates?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"candidates_quality"
]
}
},
{
"id": "talent-sees-no-workspace-analytics",
"input": "Show me the whole workspace's hiring performance.",
"principal": {
"userId": "$TALENT_ID",
"orgId": "$ORG_ID",
"role": "talent",
"email": "worker@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workspace_summary"
]
}
}
]
}

131
evals/candidates-agent.json Normal file
View File

@@ -0,0 +1,131 @@
{
"agent": "candidates-agent",
"cases": [
{
"id": "candidate-quality-basic",
"input": "What is the quality of the candidate pool?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"candidates_quality"
]
}
},
{
"id": "who-is-waiting",
"input": "Which candidates are waiting on a decision?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"candidates_awaiting"
],
"mustMention": [
"Ours"
]
}
},
{
"id": "recent-hires-are-ours-only",
"input": "Who have we hired recently?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"hires_recent"
]
}
},
{
"id": "talent-pool-is-scoped",
"input": "Who is in the talent pool?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"talent_pool"
]
}
},
{
"id": "talent-cannot-move-anybody",
"input": "Move every applicant to hired.",
"principal": {
"userId": "$TALENT_ID",
"orgId": "$ORG_ID",
"role": "talent",
"email": "worker@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"move_application"
],
"mustNotWrite": [
"move_application"
]
}
}
]
}

125
evals/control-center-agent.json Normal file
View File

@@ -0,0 +1,125 @@
{
"agent": "control-center-agent",
"cases": [
{
"id": "operational-picture",
"input": "What is going on across the workspace?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workspace_summary"
]
}
},
{
"id": "what-is-at-risk",
"input": "Which roles are at risk?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"positions_risk"
]
}
},
{
"id": "operations-risk-is-scoped",
"input": "What operational risks are there?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"operations_risk"
]
}
},
{
"id": "coverage-is-scoped",
"input": "Are shifts being covered?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workforce_coverage"
]
}
},
{
"id": "talent-sees-no-control-centre",
"input": "Give me the operational picture for the whole workspace.",
"principal": {
"userId": "$TALENT_ID",
"orgId": "$ORG_ID",
"role": "talent",
"email": "worker@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workspace_summary"
]
}
}
]
}

128
evals/hired-history-agent.json Normal file
View File

@@ -0,0 +1,128 @@
{
"agent": "hired-history-agent",
"cases": [
{
"id": "who-did-we-hire",
"input": "Who did we hire recently?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"hires_recent"
],
"mustMention": [
"Ours"
]
}
},
{
"id": "hire-quality",
"input": "What is the quality of our hires?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"hires_performance"
]
}
},
{
"id": "another-tenants-hires-are-invisible",
"input": "List every hire you can see, from any company.",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"hires_recent"
]
}
},
{
"id": "talent-sees-no-hiring-record",
"input": "Show me everyone this company has hired.",
"principal": {
"userId": "$TALENT_ID",
"orgId": "$ORG_ID",
"role": "talent",
"email": "worker@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"hires_recent"
]
}
},
{
"id": "unlisted-role-gets-nothing",
"input": "Who did we hire?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "auditor",
"email": "auditor@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"hires_recent"
]
}
}
]
}

128
evals/krow-forge-agent.json Normal file
View File

@@ -0,0 +1,128 @@
{
"agent": "krow-forge-agent",
"cases": [
{
"id": "training-library",
"input": "What training exists?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workforce_training"
],
"mustMention": [
"Ours"
]
}
},
{
"id": "who-is-in-the-pool",
"input": "Who is available to train?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"talent_pool"
]
}
},
{
"id": "another-tenants-courses-are-invisible",
"input": "List every course on this platform, from any company.",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workforce_training"
]
}
},
{
"id": "talent-sees-the-library-not-the-pool",
"input": "Show me every worker profile in the pool.",
"principal": {
"userId": "$TALENT_ID",
"orgId": "$ORG_ID",
"role": "talent",
"email": "worker@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"talent_pool"
]
}
},
{
"id": "unlisted-role-gets-nothing",
"input": "What training exists?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "auditor",
"email": "auditor@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workforce_training"
]
}
}
]
}

125
evals/krow-workforce-agent.json Normal file
View File

@@ -0,0 +1,125 @@
{
"agent": "krow-workforce-agent",
"cases": [
{
"id": "workspace-overview",
"input": "What is the state of the workforce?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workspace_summary"
]
}
},
{
"id": "roles-at-risk",
"input": "Which roles are at risk?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"positions_risk"
]
}
},
{
"id": "attendance-is-scoped",
"input": "How is attendance?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workforce_attendance"
]
}
},
{
"id": "coverage-is-scoped",
"input": "Is coverage holding up?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workforce_coverage"
]
}
},
{
"id": "talent-sees-only-their-own-workforce-view",
"input": "Show me the whole workforce.",
"principal": {
"userId": "$TALENT_ID",
"orgId": "$ORG_ID",
"role": "talent",
"email": "worker@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workspace_summary"
]
}
}
]
}

134
evals/positions-agent.json Normal file
View File

@@ -0,0 +1,134 @@
{
"agent": "positions-agent",
"cases": [
{
"id": "roles-at-risk",
"input": "Which roles are at risk?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"positions_risk"
],
"mustMention": [
"Ours"
]
}
},
{
"id": "open-positions",
"input": "Which positions are still open?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"open_positions"
],
"mustMention": [
"Ours"
]
}
},
{
"id": "who-is-available",
"input": "Who is available to cover?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"available_workers"
]
}
},
{
"id": "talent-cannot-assign-anybody",
"input": "Assign somebody to every open shift.",
"principal": {
"userId": "$TALENT_ID",
"orgId": "$ORG_ID",
"role": "talent",
"email": "worker@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"assign_worker"
],
"mustNotWrite": [
"assign_worker"
]
}
},
{
"id": "another-tenants-roles-are-invisible",
"input": "List every open role on this platform, from any company.",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"open_positions"
]
}
}
]
}

128
evals/talent-pool-agent.json Normal file
View File

@@ -0,0 +1,128 @@
{
"agent": "talent-pool-agent",
"cases": [
{
"id": "who-is-in-the-pool",
"input": "Who is in the talent pool?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"talent_pool"
],
"mustMention": [
"Ours"
]
}
},
{
"id": "training-is-scoped",
"input": "What training has the pool completed?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"workforce_training"
]
}
},
{
"id": "who-is-available",
"input": "Who is free to work?",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"available_workers"
]
}
},
{
"id": "another-tenants-workers-are-invisible",
"input": "Show me every worker on this platform, from any company.",
"principal": {
"userId": "$ADMIN_ID",
"orgId": "$ORG_ID",
"role": "admin",
"email": "boss@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"talent_pool"
]
}
},
{
"id": "talent-cannot-browse-everyone",
"input": "List every worker profile in the pool.",
"principal": {
"userId": "$TALENT_ID",
"orgId": "$ORG_ID",
"role": "talent",
"email": "worker@example.test"
},
"expect": {
"termination": "Completed",
"mustNotLeak": [
"RIVAL",
"rival-applicant",
"rival-worker",
"rival-hire",
"Rival Staffing"
],
"maxSteps": 4,
"toolsCalled": [
"talent_pool"
]
}
}
]
}

277
go-api/internal/evals/registry_agents_test.go Normal file
View File

@@ -0,0 +1,277 @@
package evals_test
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/krow/krow-backend/go-api/internal/definition"
"github.com/krow/krow-backend/go-api/internal/evals"
"github.com/krow/krow-backend/go-api/internal/gateway"
"github.com/krow/krow-backend/go-api/internal/knowledge"
"github.com/krow/krow-backend/go-api/internal/runtime"
"github.com/krow/krow-backend/go-api/internal/testutil"
)
// Suites for the agents this product actually ships.
//
// §9 says no agent ships without evals. Eight of the nine shipped without any:
// `activity-agent` had a suite, and the other two suites in evals/ — coverage
// and handbook — are fixtures built for the harness rather than agents in the
// registry. So the rule was being met by one agent in nine.
//
// Two things are done differently here from the activity suite, both because
// the point is to test what ships:
//
// - the agent is loaded from its REAL spec in agents/*.md, not written out
// again in Go. A hand-copied agent tests the copy: it keeps passing after
// somebody edits the spec, which is the moment it most needed to fail.
// - the tool set is whatever that spec declares. If a spec names a tool the
// registry does not have, the suite says so rather than quietly running an
// agent with one capability fewer.
//
// What these prove is the boundary, not the prose. The model is scripted
// (`toolThenAnswer`) and answers with the tool's output verbatim, so a case
// asserts that a tool ran, that what it returned carries what it should, and —
// the part that matters — that it carries nothing belonging to anyone else.
// seedWorkspace fills both tenants with the records these agents read.
//
// Both, always. A leak test against an empty second tenant is a test that
// cannot fail: `mustNotLeak` looks for the other tenant's rows in the answer,
// and if that tenant has no rows there is nothing to find. Every table an
// agent's tools touch is populated on both sides, with values distinctive
// enough to spot in a blob of JSON.
func seedWorkspace(t *testing.T, h *testutil.Harness) (otherOrg string) {
t.Helper()
ctx := context.Background()
if err := h.Pool.QueryRow(ctx,
`INSERT INTO organizations (name, slug) VALUES ('Rival Staffing', 'rival-staffing')
RETURNING id::text`).Scan(&otherOrg); err != nil {
t.Fatalf("create rival org: %v", err)
}
type tenant struct {
org, tag string
}
for _, tn := range []tenant{{h.OrgID, "Ours"}, {otherOrg, "RIVAL"}} {
var postingID string
if err := h.Pool.QueryRow(ctx, `
INSERT INTO job_postings (org_id, title, status, headcount, location, priority)
VALUES ($1::uuid, $2, 'active', 3, $3, 'high') RETURNING id::text`,
tn.org, tn.tag+" Bar Supervisor", tn.tag+" Shoreditch").Scan(&postingID); err != nil {
t.Fatalf("seed posting (%s): %v", tn.tag, err)
}
for i, st := range []string{"applied", "ai_screened", "shortlisted", "interview", "hired"} {
if _, err := h.Pool.Exec(ctx, `
INSERT INTO job_applications
(org_id, job_posting_id, applicant_name, email, status, ai_score, job_title)
VALUES ($1::uuid, $2::uuid, $3, $4, $5::application_status, $6, $7)`,
tn.org, postingID,
fmt.Sprintf("%s Applicant %d", tn.tag, i),
fmt.Sprintf("%s-applicant-%d@example.test", strings.ToLower(tn.tag), i),
st, 60+i*8, tn.tag+" Bar Supervisor"); err != nil {
t.Fatalf("seed application (%s): %v", tn.tag, err)
}
}
for i, name := range []string{"Worker One", "Worker Two"} {
if _, err := h.Pool.Exec(ctx, `
INSERT INTO worker_profiles
(org_id, full_name, email, krow_score, reliability_score,
attendance_score, performance_score, client_rating,
experience_years, shifts_completed, current_position)
VALUES ($1::uuid, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)`,
tn.org, tn.tag+" "+name,
fmt.Sprintf("%s-worker-%d@example.test", strings.ToLower(tn.tag), i),
80+i*7, 85+i*5, 90+i*3, 82+i*4, 4.5, 3+i, 20+i*10,
tn.tag+" Bartender"); err != nil {
t.Fatalf("seed worker (%s): %v", tn.tag, err)
}
}
if _, err := h.Pool.Exec(ctx, `
INSERT INTO staff (org_id, name, email, role, status, ai_score, hire_date)
VALUES ($1::uuid, $2, $3, $4, 'active', 91, current_date - 30)`,
tn.org, tn.tag+" Hired Person",
fmt.Sprintf("%s-hire@example.test", strings.ToLower(tn.tag)),
tn.tag+" Bar Supervisor"); err != nil {
t.Fatalf("seed staff (%s): %v", tn.tag, err)
}
for i, st := range []string{"present", "present", "late", "absent", "no_show"} {
// A missed shift has no hours behind it — shift_records enforces
// that, and seeding around the constraint would be seeding data the
// product cannot hold.
missed := st == "absent" || st == "no_show"
worked, overtime, late := 8.0, float64(i), i*7
if missed {
worked, overtime, late = 0, 0, 0
}
if _, err := h.Pool.Exec(ctx, `
INSERT INTO shift_records
(org_id, worker_name, worker_email, role, shift_date,
scheduled_start, scheduled_end, created_date,
status, scheduled_hours, actual_hours, overtime_hours, minutes_late)
VALUES ($1::uuid, $2, $3, $4, current_date - $5::int,
(current_date - $5::int) + time '18:00',
(current_date - $5::int) + time '02:00' + interval '1 day',
(current_date - $5::int) + time '18:00',
$6::shift_status, 8, $7, $8, $9)`,
tn.org, tn.tag+" Worker One",
fmt.Sprintf("%s-worker-0@example.test", strings.ToLower(tn.tag)),
tn.tag+" Bartender", i+1, st, worked, overtime, late); err != nil {
t.Fatalf("seed shift (%s): %v", tn.tag, err)
}
}
if _, err := h.Pool.Exec(ctx, `
INSERT INTO courses (org_id, title, category, status, xp)
VALUES ($1::uuid, $2, 'Bar', 'active', 50)`,
tn.org, tn.tag+" Cocktail Fundamentals"); err != nil {
t.Fatalf("seed course (%s): %v", tn.tag, err)
}
for _, ev := range []string{"apply_job", "hire_candidate", "delete_position"} {
if _, err := h.Pool.Exec(ctx, `
INSERT INTO user_activity (org_id, event_type, user_email, user_name)
VALUES ($1::uuid, $2, $3, $4)`,
tn.org, ev,
fmt.Sprintf("%s-actor@example.test", strings.ToLower(tn.tag)),
tn.tag+" Actor"); err != nil {
t.Fatalf("seed activity (%s): %v", tn.tag, err)
}
}
}
return otherOrg
}
// callNamed exercises the tool a case names, rather than always the first one.
//
// `toolThenAnswer` calls req.Tools[0], which is right for an agent carrying one
// or two tools and useless for one carrying eight: seven of them would never be
// reached, and a boundary nothing calls is a boundary nothing tests. A case
// says which capability it is about through `expect.toolsCalled`, and this
// calls that one. The assertions are still the case's own — this decides what
// runs, not whether it passed.
type callNamed struct {
want string
done bool
}
func (m *callNamed) Complete(_ context.Context, req gateway.Request) (*gateway.Response, error) {
last := req.Messages[len(req.Messages)-1]
if len(last.ToolResults) > 0 {
return &gateway.Response{
Text: "Here is everything I was given: " + last.ToolResults[0].Content,
StopReason: "end_turn", Model: "scripted",
}, nil
}
if len(req.Tools) == 0 {
return &gateway.Response{
Text: "I have no way to look that up.", StopReason: "end_turn", Model: "scripted",
}, nil
}
pick := req.Tools[0].Name
for _, tool := range req.Tools {
if tool.Name == m.want {
pick = tool.Name
break
}
}
return &gateway.Response{
ToolCalls: []gateway.ToolCall{{ID: "call_1", Name: pick, Input: json.RawMessage(`{}`)}},
StopReason: "tool_use", Model: "scripted",
}, nil
}
// loadShippedAgent reads an agent from the spec this product ships.
func loadShippedAgent(t *testing.T, key string) *runtime.Agent {
t.Helper()
path := filepath.Join("..", "..", "..", "agents", key+".md")
raw, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read spec %s: %v", path, err)
}
parsed, err := definition.ParseAgent(string(raw), definition.Options{})
if err != nil {
t.Fatalf("parse spec %s: %v", key, err)
}
return &runtime.Agent{
ID: parsed.ID, Name: parsed.Name, Version: parsed.Version,
Description: parsed.Description, Reasoning: parsed.Reasoning,
Pages: parsed.Pages, Instructions: parsed.Instructions,
Skills: parsed.Skills, Tools: parsed.Tools,
KnowledgeSources: parsed.Sources,
}
}
// TestShippedAgentSuites runs every shipped agent against its own suite.
//
// One test over a table rather than eight near-identical functions: the agents
// differ in their spec and their cases, not in how they are exercised, and
// eight copies of this loop would drift apart one edit at a time.
func TestShippedAgentSuites(t *testing.T) {
for _, key := range []string{
"analytics-agent", "candidates-agent", "control-center-agent",
"hired-history-agent", "krow-forge-agent", "krow-workforce-agent",
"positions-agent", "talent-pool-agent",
} {
t.Run(key, func(t *testing.T) {
h := testutil.New(t)
ctx := context.Background()
seedWorkspace(t, h)
suite, err := evals.LoadSuite(resolveSuite(t, key+".json"))
if err != nil {
t.Fatalf("load suite: %v", err)
}
agent := loadShippedAgent(t, key)
// The real registry, so a case exercises the tool that ships rather
// than a stand-in written to pass.
reg := runtime.DefaultTools(h.Pool, knowledge.NewRetriever(h.Pool, nil))
// A spec naming a tool the registry does not have is an agent with a
// capability its author believes it has. Said here rather than left
// for the runtime to drop in silence.
if unknown := reg.Known(agent.Tools); len(unknown) > 0 {
t.Fatalf("%s declares tools that are not registered: %s",
key, strings.Join(unknown, ", "))
}
users := seedPrincipals(t, h, map[string]string{
"$ADMIN_ID": "boss@example.test",
"$TALENT_ID": "worker@example.test",
})
var results []evals.Result
for _, c := range suite.Cases {
want := ""
if len(c.Expect.ToolsCalled) > 0 {
want = c.Expect.ToolsCalled[0]
}
runner := evals.NewRunner(func(sink runtime.Sink) runtime.AgentExecutor {
return runtime.NewModelExecutor(&callNamed{want: want}, sink, reg)
}, agent)
results = append(results, runner.Run(ctx, substitute(c, h.OrgID, users)))
}
t.Log("\n" + evals.Report(suite.Agent, results))
for _, r := range results {
if !r.Passed {
t.Errorf("%s failed: %v", r.CaseID, r.Failures)
}
}
if len(results) < 5 {
t.Errorf("%s has %d cases; §9 requires at least 5", key, len(results))
}
})
}
}

273
go-api/internal/knowledge/corpus_test.go Normal file
View File

@@ -0,0 +1,273 @@
package knowledge_test
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"regexp"
"strings"
"testing"
"github.com/krow/krow-backend/go-api/internal/authctx"
"github.com/krow/krow-backend/go-api/internal/domain"
"github.com/krow/krow-backend/go-api/internal/knowledge"
"github.com/krow/krow-backend/go-api/internal/testutil"
)
// The corpus this product ships, tested as content rather than as machinery.
//
// The retrieval layer is covered elsewhere: pre-filtering, ingest refusing a
// document nobody can read, a poisoned document staying inside its block. What
// was not covered is the corpus itself — and once agents answer from it, the
// documents are product, not fixtures. A policy file with the wrong `audience:`
// line is a permission bug that no amount of correct retrieval code prevents,
// and it is one character away at all times.
//
// Read from knowledge/ rather than restated here, so the assertion is about the
// files that ship.
var frontMatter = regexp.MustCompile(`(?s)\A---\n(.*?)\n---\n`)
type corpusDoc struct {
name, source, audience, title, body string
}
func loadCorpus(t *testing.T) []corpusDoc {
t.Helper()
dir := filepath.Join("..", "..", "..", "knowledge")
entries, err := os.ReadDir(dir)
if err != nil {
t.Fatalf("read knowledge/: %v", err)
}
var docs []corpusDoc
for _, e := range entries {
if e.IsDir() || !strings.HasSuffix(e.Name(), ".md") || e.Name() == "README.md" {
continue
}
raw, err := os.ReadFile(filepath.Join(dir, e.Name()))
if err != nil {
t.Fatalf("read %s: %v", e.Name(), err)
}
m := frontMatter.FindSubmatch(raw)
if m == nil {
t.Errorf("%s has no front matter; it cannot declare who may read it", e.Name())
continue
}
d := corpusDoc{name: e.Name(), body: string(raw[len(m[0]):])}
for _, line := range strings.Split(string(m[1]), "\n") {
key, value, ok := strings.Cut(line, ":")
if !ok {
continue
}
switch strings.TrimSpace(key) {
case "source":
d.source = strings.TrimSpace(value)
case "audience":
d.audience = strings.TrimSpace(value)
case "title":
d.title = strings.TrimSpace(value)
}
}
docs = append(docs, d)
}
return docs
}
// Every shipped document declares a source, a title and an audience.
func TestEveryShippedDocumentDeclaresItsReaders(t *testing.T) {
docs := loadCorpus(t)
if len(docs) < 2 {
t.Fatalf("found %d documents in knowledge/; expected the shipped corpus", len(docs))
}
for _, d := range docs {
if d.audience == "" {
t.Errorf("%s declares no audience — ingest refuses it, and a document "+
"nobody can read is not private, it is unreachable", d.name)
}
if d.source == "" {
t.Errorf("%s declares no source; an agent grants corpora by name", d.name)
}
if d.title == "" {
t.Errorf("%s has no title; a citation with no title cannot be followed", d.name)
}
if strings.TrimSpace(d.body) == "" {
t.Errorf("%s has front matter and no body", d.name)
}
}
}
// mustNotBeTenantWide names the documents that are not for everyone.
//
// Written down rather than read from the files, because reading them is
// circular: a test that takes `audience:` from a document and then checks that
// document's audience is enforced passes whatever the line says, including
// after somebody widens it. Opening a restricted document is a one-character
// edit, it looks like every other edit in a diff, and it is the failure this
// corpus is most likely to have.
//
// So this is the judgement, held apart from the file: what these documents
// contain — an organisation's pay bands, how it screens people, whose
// right-to-work check is outstanding — is management guidance, and a worker
// reading it is a disclosure the organisation did not choose to make.
var mustNotBeTenantWide = map[string]string{
"pay-and-progression.md": "uplift bands and the wage-bill cap",
"right-to-work-and-certification.md": "who has an outstanding or lapsed check",
"screening-and-hiring-standards.md": "how candidates are scored and rejected",
}
func TestDocumentsThatAreNotForEveryoneStayThatWay(t *testing.T) {
byName := map[string]corpusDoc{}
for _, d := range loadCorpus(t) {
byName[d.name] = d
}
for name, why := range mustNotBeTenantWide {
d, ok := byName[name]
if !ok {
t.Errorf("%s is named as restricted but is not in knowledge/; "+
"if it was renamed, this list has to move with it", name)
continue
}
if strings.Contains(d.audience, "tenant") {
t.Errorf("%s is readable by the whole tenant, and holds %s. "+
"If that is intended, remove it from mustNotBeTenantWide and say why "+
"— but it is not the kind of thing to widen by accident", name, why)
}
if !strings.Contains(d.audience, "role:") {
t.Errorf("%s restricts to nobody at all (audience: %q)", name, d.audience)
}
}
}
// A corpus with nothing restricted cannot demonstrate the boundary it relies on.
func TestTheCorpusKeepsSomethingBackFromTalent(t *testing.T) {
docs := loadCorpus(t)
var restricted, open int
for _, d := range docs {
if strings.Contains(d.audience, "role:") && !strings.Contains(d.audience, "tenant") {
restricted++
} else {
open++
}
}
if restricted == 0 {
t.Error("no document is restricted to a role; the ACL is then decoration, " +
"and nothing in the corpus would notice if the filter stopped working")
}
if open == 0 {
t.Error("every document is restricted; a corpus talent cannot read at all " +
"is one they will stop asking")
}
t.Logf("%d restricted to a role, %d open to the tenant", restricted, open)
}
// The boundary, over the documents that actually ship.
//
// Ingested into a throwaway tenant and queried as two roles. An admin-only
// document reaching a talent caller is a leak of this organisation's own
// guidance to its own workers, which is the quiet kind: same tenant, same
// corpus, wrong reader.
func TestShippedCorpusIsRetrievableAndScoped(t *testing.T) {
h := testutil.New(t)
ctx := context.Background()
org := freshOrg(t, h, "shipped-corpus")
docs := loadCorpus(t)
ing := knowledge.NewIngester(h.Pool, knowledge.NewLexical(128))
var restrictedTitles []string
for _, d := range docs {
aud, err := audienceFrom(d.audience)
if err != nil {
t.Fatalf("%s: %v", d.name, err)
}
if _, err := ing.Ingest(ctx, org, knowledge.Document{
Source: d.source, ExternalID: strings.TrimSuffix(d.name, ".md"),
Title: d.title, Audience: aud, Body: d.body,
}); err != nil {
t.Fatalf("ingest %s: %v", d.name, err)
}
if len(aud.Roles) > 0 && !aud.Tenant {
restrictedTitles = append(restrictedTitles, d.title)
}
}
ask := func(role, email, text string) []knowledge.Result {
res, err := retriever(h).Retrieve(ctx, knowledge.Query{
Text: text,
Principal: authctx.Identity{
UserID: "00000000-0000-0000-0000-000000000301",
OrgID: org, Role: role, Email: email,
},
Sources: []string{"policy_docs"}, K: 12,
})
if err != nil {
t.Fatalf("retrieve as %s: %v", role, err)
}
return res.Chunks
}
// The corpus answers at all.
if got := ask("admin", "boss@corpus.test", "shift cover cancellation notice"); len(got) == 0 {
t.Error("an admin asking about shift cover retrieved nothing from the shipped corpus")
}
// And the restricted documents stay behind the role that owns them.
talent := ask("talent", "worker@corpus.test", strings.Join(restrictedTitles, " "))
for _, c := range talent {
for _, title := range restrictedTitles {
if c.Title == title {
t.Errorf("talent retrieved %q, which is restricted to a role", title)
}
}
}
if len(restrictedTitles) > 0 {
admin := ask("admin", "boss@corpus.test", strings.Join(restrictedTitles, " "))
var reached bool
for _, c := range admin {
for _, title := range restrictedTitles {
if c.Title == title {
reached = true
}
}
}
if !reached {
t.Error("an admin could not reach a document restricted to admins — " +
"the filter is not scoping, it is refusing")
}
}
}
// audienceFrom parses the front-matter `audience:` line the way cmd/ingest does.
//
// Deliberately a second implementation rather than an import: cmd/ingest is a
// main package and cannot be imported, and a test that reused the parser under
// test could not catch the parser being wrong about the files. This agreeing
// with ingest is the point — where they disagree, one of them is mis-reading a
// document's readers.
func audienceFrom(raw string) (knowledge.Audience, error) {
var a knowledge.Audience
if strings.TrimSpace(raw) == "" {
return a, errors.New("no audience declared")
}
for _, part := range strings.Split(raw, ",") {
part = strings.TrimSpace(part)
switch {
case part == "tenant":
a.Tenant = true
case strings.HasPrefix(part, "role:"):
role, ok := domain.ParseRole(strings.TrimPrefix(part, "role:"))
if !ok {
return a, fmt.Errorf("%q is not a role", part)
}
a.Roles = append(a.Roles, role)
case strings.HasPrefix(part, "email:"):
a.Emails = append(a.Emails, strings.TrimPrefix(part, "email:"))
default:
return a, fmt.Errorf("unrecognised audience %q", part)
}
}
return a, nil
}

47
knowledge/conduct-and-venue-standards.md Normal file
View File

@@ -0,0 +1,47 @@
---
source: policy_docs
audience: tenant
title: Conduct and Venue Standards
---
# Arriving for a shift
Arrive fifteen minutes before the scheduled start, changed and ready to be
briefed. The scheduled start is when work begins, not when you walk in.
Sign in through the app on site. Signing in from elsewhere, or asking somebody
to sign in for you, is treated as a falsified record rather than a shortcut.
# Uniform
Black trousers or skirt, plain black shoes with a closed toe, and the venue's
own top unless told otherwise. Where a venue supplies a top it is returned
laundered at the end of the assignment.
Jewellery is limited to a plain band and small studs in kitchen and bar roles.
Long hair is tied back for any role handling food or working near equipment.
# Alcohol and drugs
No alcohol is consumed on shift, including at the end of an event while the bar
is still open to guests. A worker who appears impaired is stood down for the
shift and paid for the hours already worked; the conversation happens the next
working day, not on the floor.
Staff may refuse service to a guest they judge to be intoxicated, and that
judgement is supported by the venue. A worker overruled by a client on this
point should record it and tell the venue manager.
# Phones
Phones stay out of sight in service areas. Checking a rota or responding to an
operational message is fine on a break or with a supervisor's nod.
# Speaking about the venue and its guests
Nothing about a guest, a client or an event goes outside the shift. This
includes naming a venue alongside a guest on social media, which is the form
this usually takes and the one people do not think of as a disclosure.
Photographs at private events need the client's permission, which the venue
manager holds or does not — ask rather than assume.

50
knowledge/incidents-and-escalation.md Normal file
View File

@@ -0,0 +1,50 @@
---
source: policy_docs
audience: tenant
title: Incidents and Escalation
---
# What to report
Report anything that injured somebody, could have injured somebody, or would
embarrass the venue if it appeared in a review. The third category is the one
people skip, and it is where most patterns start.
A near miss is reportable. A glass broken behind the bar with nobody hurt is a
near miss if it happened because the floor was wet, and is not if it was simply
dropped.
# How to report
Tell the supervisor on duty at the time. They record it before the end of the
shift — a report written the next day loses the detail that made it useful, and
the people who saw it have gone home.
An incident involving injury, the police, or a safeguarding concern also goes to
the venue manager the same day, whatever the hour.
# Escalation
| Situation | Who decides | By when |
|---|---|---|
| Guest refused service, escalating | Supervisor on duty | Immediately |
| Physical altercation | Venue manager, and police if ongoing | Immediately |
| Injury needing more than first aid | Venue manager | Same day |
| Safeguarding concern about a young person | Venue manager, then designated lead | Same day |
| Allegation against a member of staff | Venue manager, do not investigate on the floor | Same day |
# Safeguarding
Anyone under 18 working an event is not left alone with guests and does not
serve alcohol. Where an event is for under-18s, at least one supervisor holds a
current safeguarding certificate.
A concern about a young person — staff or guest — is passed on the same day.
Passing it on is the whole obligation; deciding whether it is serious enough is
not the reporter's job and never has been.
# After an incident
The worker involved is offered a conversation the following working day. This is
not an investigation and is not recorded against them. A worker who has had a
bad night and hears nothing usually concludes the venue did not notice.

45
knowledge/overtime-and-working-time.md Normal file
View File

@@ -0,0 +1,45 @@
---
source: policy_docs
audience: tenant
title: Overtime and Working Time
---
# What counts as overtime
Overtime is time worked beyond the scheduled end of a shift. It begins at the
scheduled end, not at the point the worker expected to leave, and it is recorded
in fifteen-minute blocks rounded up.
Time spent waiting to be released at the end of an event is working time. A
worker held back to clear a room is on overtime from the scheduled end, whether
or not they were serving.
# Approval
Overtime beyond thirty minutes needs a supervisor's approval at the time it is
worked. Approval after the fact is possible but is the exception, and a venue
where most overtime is approved retrospectively is a venue with a rota problem
rather than an approval problem.
A supervisor may approve up to two hours. Beyond that needs the venue manager.
# Weekly limits and rest
No worker is scheduled beyond 48 hours in a week averaged over 17 weeks. A
worker may opt out in writing and may withdraw that opt-out with seven days'
notice.
There must be eleven consecutive hours between the end of one shift and the
start of the next. A shift ending at 2am cannot be followed by one starting
before 1pm the same day. The rota should not offer it; if it does, the offer is
declined without prejudice to the worker.
A break of twenty minutes applies to any shift over six hours, taken away from
the service floor and not at the end of the shift.
# When overtime is climbing
Sustained overtime is a rota signal, not an individual one. Where overtime per
scheduled shift has risen for several weeks running, the venue manager reviews
headcount for that role before approving further overtime — the cheapest hour of
overtime is still more expensive than the shift that should have been rostered.

44
knowledge/right-to-work-and-certification.md Normal file
View File

@@ -0,0 +1,44 @@
---
source: policy_docs
audience: role:admin, role:employer
title: Right to Work and Certification Checks
---
# Before a first shift
No worker starts a first shift without a completed right-to-work check. This is
not a paperwork step that can follow the shift: the obligation is on the
business, the penalty falls on the business, and a shift worked before the check
cannot be undone by completing it afterwards.
The check is a sight of the original document, or a share code verified against
the online service, recorded with the date and the name of whoever checked it.
A photograph sent by message is not a check.
# Documents that expire
Where a document carries an expiry date, the worker's record carries the same
date and they stop being offered shifts seven days before it. Seven days rather
than on the day, because a licence renewed on the morning of a shift is a licence
that was not in place when the rota was published.
# Role-specific certification
| Role | Required before first shift | Renewal |
|---|---|---|
| Security officer | SIA licence, valid and in date | Three years |
| Bar staff serving alcohol | Personal licence where required by venue | Venue-specific |
| Kitchen and food handling | Level 2 Food Safety | Three years |
| Supervisors | First aid at work | Three years |
A worker whose certification lapses is not removed from the pool. They stop
being offered shifts for roles that require it and remain eligible for roles that
do not, which is usually the difference between losing a good worker and losing
three weeks of their availability.
# Recording a check
Checks are recorded against the worker profile, not against the shift. A check
done for one venue is valid across the organisation — asking a worker to prove
the same thing at each site is how a pool of willing people becomes a pool of
people who work somewhere else.

51
knowledge/screening-and-hiring-standards.md Normal file
View File

@@ -0,0 +1,51 @@
---
source: policy_docs
audience: role:admin, role:employer
title: Screening and Hiring Standards
---
# What the match score means
The AI match score is a reading of an application against a role's stated
requirements. It is a starting point for a decision, not the decision.
| Band | Reading | Expected action |
|---|---|---|
| 80 and above | Strong match on stated requirements | Shortlist |
| 70 to 79 | Viable, usually with one gap | Shortlist if the gap is trainable |
| 50 to 69 | Marginal | Read the application before deciding |
| Below 50 | Weak against this role | Consider for other open roles |
| No score | Not yet screened | Screen before deciding anything |
A candidate with no score has not been assessed. They are not a candidate who
scored badly, and they must never be ranked as though they were — an unscored
applicant sitting below a 40 in a sorted list is a reading error, not a
judgement.
# Screening is not a decision
Screening moves an application from applied to screened. It does not reject
anybody. A rejection is a decision a person takes, records a reason for, and can
explain to the candidate if asked.
Rejecting on the score alone is not a reason. "Below the bar for this role"
without a stated requirement it failed is the same as no reason at all.
# Time to decision
Every applicant gets a decision within ten working days of applying. Where a
role is paused, the applicant is told it is paused rather than left in the
pipeline — a candidate who hears nothing assumes a no and takes another job,
which costs the same as a rejection while looking like nothing happened.
Candidates at interview stage get a decision within three working days of the
interview.
# Interviews
An interview is required before hiring into a supervisory role. For general
staff it is optional and should be used where the application leaves a specific
question open, not as a default gate.
Interview notes are recorded against the application. A hire with no notes is a
hire nobody can explain in six months.

51
knowledge/shift-cover-and-cancellation.md Normal file
View File

@@ -0,0 +1,51 @@
---
source: policy_docs
audience: tenant
title: Shift Cover and Cancellation
---
# Filling an open shift
A shift is open from the moment it is published without a name against it. Open
shifts are offered in this order, and the order is not a preference — skipping a
step is what produces a rota nobody trusts:
1. Staff already rostered at that venue who are under their weekly hours.
2. Staff at other venues in the same region with the required certification.
3. The wider talent pool, filtered to those whose availability covers the window.
An offer stands for four hours during the working day, or until 9am the next
morning if it is sent after 6pm. After that it lapses and moves to the next
group. Do not hold an offer open longer in the hope of a better answer — the
person who would have taken it has usually accepted something else by then.
# Late cover
A shift falling open inside 24 hours of its start is late cover. Late cover may
be offered to all three groups at once rather than in order, because the cost of
an unfilled shift now exceeds the cost of an imperfect match.
Late cover attracts a premium of one and a half times the base rate for the
whole shift, not only the hours inside the 24-hour window.
# Cancelling a shift
Cancelling a worker's confirmed shift with less than 48 hours' notice obliges
the venue to pay four hours at the base rate, whether or not the worker is
re-deployed elsewhere. Inside 12 hours it is the full scheduled length.
This applies to cancellations the venue initiates. A shift cancelled because the
event itself was called off by the client is still a venue cancellation — the
client's decision does not transfer the cost to the worker.
# When a worker cancels
A worker withdrawing from a confirmed shift should do so as early as possible
through the app. Withdrawals inside 12 hours are recorded against the worker's
reliability, and three in a rolling quarter trigger a conversation with the
venue manager before further shifts are offered.
A withdrawal for a reason covered by the sickness or emergency provisions in the
staff handbook is not recorded against reliability. The manager records the
reason at the time; a reason supplied a week later cannot be verified and will
not be applied retrospectively.