doormile_backend/docs/DEV_ONBOARDING.md

# Doormile Backend — Developer Onboarding & Working Memory

Read this before touching the backend. It is the portable version of knowledge
that otherwise lives only in one machine's Claude session memory. It covers two
things a new dev (or a fresh Claude session on another machine) needs:

1. **How we use Claude on this project** — the project-memory file, the skills,
   the working conventions.
2. **Operational knowledge that isn't in the code** — deploy topology, build
   gotchas, production landmines, and the incident history behind current
   design choices.

Related docs already in this repo:
- [`CLAUDE.md`](../CLAUDE.md) — the full project memory (architecture, data
  model, route surface). Start there for *what the system is*.
- [`docs/CHANGELOG.md`](CHANGELOG.md) — summary of recent backend changes and new endpoints.
- [`docs/customer-app-api.md`](customer-app-api.md) — Customer App v1 API design and endpoint specs.
- [`docs/customer-api-testing.md`](customer-api-testing.md) — step-by-step test guide and curl references for customer endpoints.
- [`docs/openapi-customer.yaml`](openapi-customer.yaml) — OpenAPI 3.0 specification for customer API.
- [`docs/doormile-flow.md`](doormile-flow.md) — end-to-end booking/assignment flow.
- [`docs/miler-app-api.md`](miler-app-api.md), [`docs/express-console-api.md`](express-console-api.md) — API contracts.
- [`docs/logistics-base-handover.md`](logistics-base-handover.md) — the pickup-source
  and base-handover flow (requests 25–31), including the state transitions.
- [`docs/jupiter2doormile.md`](jupiter2doormile.md) — the legacy→new migration map.
- [`docs/test-booking-runbook.md`](test-booking-runbook.md) — how to run a test booking.
- [`skills.md`](../skills.md) — note on the installed Claude skill pack.

---

## Part 1 — How Claude is used on this project

### 1.1 Project memory (`CLAUDE.md`)
`CLAUDE.md` at the repo root is the single source of project context, loaded
automatically into every Claude session in this repo. It is checked into git, so
it travels to every machine and every dev. If you change how the system works,
update `CLAUDE.md` in the same change — it is treated as authoritative.

Sections in `CLAUDE.md` are tagged **[verified this session]** (confirmed
against source) vs **[carried forward]** (reported by a prior session, not
re-verified). Respect the distinction — don't treat carried-forward claims as
confirmed.

### 1.2 Session memory (machine-local — this is why this doc exists)
Claude also keeps per-fact memory files under
`~/.claude/projects/<project>/memory/`, indexed by `MEMORY.md`. These are
**not in git** and **do not travel** to another machine or dev. They accumulate
operational facts, incidents, and gotchas across sessions.

Part 2 below is a distillation of those files into a form the whole team can
read. When a memory fact changes, update *both* the memory file (for Claude) and
this doc (for humans).

### 1.3 Claude skills in use
The repo has the `addyosmani/agent-skills` pack installed at `.agents/skills/`
and symlinked into `.claude/skills/`. These are third-party, broad-trigger
skills that run with full agent permissions. Roster (see `skills.md` for the
full table):

`api-and-interface-design`, `browser-testing-with-devtools`,
`ci-cd-and-automation`, `code-review-and-quality`, `code-simplification`,
`context-engineering`, `debugging-and-error-recovery`,
`deprecation-and-migration`, `documentation-and-adrs`,
`doubt-driven-development`, `frontend-ui-engineering`,
`git-workflow-and-versioning`, `idea-refine`, `incremental-implementation`,
`interview-me`, `observability-and-instrumentation`,
`performance-optimization`, `planning-and-task-breakdown`,
`security-and-hardening`, `shipping-and-launch`, `source-driven-development`,
`spec-driven-development`, `test-driven-development`, `using-agent-skills`.

**How we treat them:** available on request, *not* auto-adopted over the
conventions in `CLAUDE.md` §7. The established conventions win: use the `utils`
response helpers, the `constants` enums, reuse `AssignMilerToBooking` and
`haversineKM`, prefer minimal-leverage fixes. Invoke a skill explicitly when its
scope fits; don't let a broad trigger override house style.

### 1.4 Standing working preferences (from `CLAUDE.md`)
- Direct, honest technical assessments. Don't declare victory early.
- Production-grade from the start, minimal-effort highest-leverage fixes.
- **Warn before any consequential server/schema change.**
- Once a decision is made, proceed and report — but re-ask on money/data-correctness.

---

## Part 2 — Operational knowledge (not derivable from the code)

### 2.1 Build & toolchain
- Go **1.25** (`go.mod` says `go 1.25.0`). Module `doormile`.
- On the primary dev Mac, Go is installed at `/Users/tenext/go` but **not on
  PATH**. Prefix: `export PATH="/Users/tenext/go/bin:$PATH"` before any `go`
  command, or a bare `go build` returns "command not found".
- `go build ./...`, `go vet ./...`, `go test ./...` all pass repo-wide.
- `scratch/*.go` are throwaway `func main()` scripts tagged `//go:build ignore`
  so the toolchain skips them. If a `main redeclared` error appears, it's a
  stripped build tag under `scratch/`, **not** a real app problem.
- `go build .` emits a ~65MB `doormile` binary in the repo root; it's
  gitignored — delete it, don't commit it.
- Tests exist only for pure logic (`isHyperlocal`, `calculateVolumetricWeight`,
  `ParsePage`). Everything else is verified by build+vet, review, or live test.

### 2.2 Deploy topology — how a change reaches production
- Live backend runs on **k3s** (not plain docker), server `66.116.225.226` port
  **4422** (SSH, key-based auth from the primary Mac; `KUBECONFIG=/etc/rancher/k3s/k3s.yaml`).
- Workload: **statefulset `doormile`**, namespace `doormile`, **3 replicas**.
  Binary is `/app/server` in the pod; image
  `docker.io/doormile/doormile-backend:latest`, `imagePullPolicy: Always`.
  Pods report `APP_ENV=staging`.
- **There is NO CI.** Nothing watches the repo. A `kubectl rollout restart`
  alone rebuilds nothing — it re-pulls the same image digest.
- **A change reaches prod only by a manual build+push** (needs Docker daemon up
  + Docker Hub creds):
  ```
  docker build -t doormile/doormile-backend:latest .
  docker push  doormile/doormile-backend:latest
  kubectl rollout restart statefulset/doormile -n doormile
  ```
- **The image builds from the working tree** (`ADD . /app/`). Uncommitted local
  edits and any throwaway file under `cmd/`/`scratch/` get baked in. **Always
  `git status` before building**, and **confirm code is committed *after* a
  deploy** — `git log origin/main` tells you nothing about what's running.
- **Verify what's actually live** rather than assuming: `kubectl exec -n
  doormile doormile-0 -- ls -la /app` shows the binary's build date; or probe a
  response field only the new code emits. Group auth (`/admin/*`, `/hub/*`,
  `/miler/*`) returns 401 for unmatched routes, so a 404-vs-401 probe can't tell
  you if a route exists — use an authenticated request.

### 2.3 Env vars & the source-of-truth manifest
- Env is **inline on the statefulset spec** (`.spec.template.spec.containers[0].env`),
  no ConfigMap/envFrom. Set a flag with
  `kubectl -n doormile set env statefulset/doormile KEY=value` (auto-rolls). A
  plain restart does **not** add a var that isn't already in the spec.
- **Source-of-truth manifest**: `/opt/kubernetes/manifests/doormile/miletruth.yaml`
  (git repo at `/opt/kubernetes/.git`; second copy under `/root/kubernetes/...`).
  A `kubectl apply` of it **overwrites** live `set env` changes — so any live
  flag change must also be written into this manifest or the next deploy reverts
  it.
- The manifest supplies DB/Redis/NATS passwords via `secretKeyRef`
  (`doormile-secrets`) while the live pods carry literals — verify that secret
  exists before relying on `kubectl apply`.
- **Known live flag:** `MILER_COLLECTED_STATE_ENABLED=true` (hyperlocal two-step
  pickup). `TRUSTED_PROXIES` must be set (api.doormile.com sits behind a
  reverse proxy) or per-IP rate limits collapse all clients into one bucket.
- **`MILER_HUB_HANDOVER_ENABLED` — default off, and must stay off** until a rider
  build that calls `POST /miler/consignments/:id/inward-at-hub` is live. On, a
  hub-routed parcel stops at `Created` until the rider records the handover; off,
  pickup-complete marks it `Inwarded_at_Hub` immediately, which is what the
  deployed app expects. Flipping it early strands every intercity parcel on
  `Created` with no button in the app to advance it and no row in any base's
  received list. See [`logistics-base-handover.md`](logistics-base-handover.md).

### 2.4 Timezone convention (subtle — read before touching any time field)
The DB and backend time helpers run on **IST (Asia/Kolkata) wall-clock**.
`dbLocation = Asia/Kolkata`; `DBNow()`/`DBToday()` (`utils/helper.go`) return
IST wall-clock digits *tagged as UTC*. Consequence: clients (e.g. the miler app)
that send a timestamp — such as `logdate` on `POST /miler/logs` — should send
**IST wall-clock (phone local time in India), not UTC**. Sending UTC misaligns
Redis zset scores and time-window queries by 5h30m.

### 2.5 Postgres CHECK constraints predate the codebase
Status-column CHECK constraints are **not** created by GORM AutoMigrate — they
predate this codebase. Adding a new status *constant* in Go is not enough; the
DB rejects it with **SQLSTATE 23514**. Before adding any status enum value,
widen the matching `*_status_check` constraint in `migrations/migrate.go`. This
is also why the base-handover reconciliation path raises a `Lost` exception
rather than a more precise `Handover_Not_Received` — the latter would need
`consignmentexceptions` widened first.
Constraints exist on: `consignments`, `pickupbookings`,
`milerprofiles.availabilitystatus`, `bookingassignments`,
`consignmentexceptions`, `tripsheets`. `consignmenthistory` has no status check.
(This bit us live: `consignments_status_check` was missing `Collected_By_Miler`
and `Cancelled`, 500-ing every hyperlocal pickup and admin cancel.)

### 2.6 Miler telemetry pipeline (HTTP → Redis; there is NO MQTT)
MQTT is a jupiter concept; Doormile has none. Miler telemetry is 4 HTTP
endpoints, identity always taken from `c.Locals("userid")` (never the body):
- `PUT /miler/location` → Postgres (`MilerProfile` lat/lng/pincode) + Redis
  (`miler:gps:{userid}` 30min TTL, `milers:locations` GEO set feeding dispatch).
- `POST /miler/logs` → **Redis-only** periodic telemetry point
  (`miler_periodic_log:{userid}:{ts}` + zsets, scored by timestamp).
- `POST /miler/status` → Redis-only (`miler_status:{userid}`).
- `POST /miler/consignments/logs` → Redis (list+zset) + Postgres
  `ConsignmentHistory`; takes an **array** (batch).

Device sensors (GPS/speed/heading/accuracy, battery/is_charging, connection)
come from Flutter plugins regardless of transport — MQTT isn't needed to collect
them. Console reads of the trail use **newest-first** fetch
(`ZRevRangeByScore`), then reverse to chronological, so a `?limit=N` window
keeps the *latest* fixes (fixed in `6e5da09` — previously `?limit=1` returned
the day's first blank early-boot ping).

### 2.7 Route optimizer
`routes.workolik.com` (env `ROUTE_OPTIMIZER_URL`) = the rider-bike FastAPI
service (OR-Tools + Valhalla). Doormile endpoint
`POST /api/v1/optimization/doormile/sequence`, contract matches
`internal/routing/optimizer.go`. As of `f6d339a`,
`routing.SequenceMilerStopsAsync` fires after every assignment path. A `step=0`
in prod is never the service being down — it means the rider had <2 active
stops, a stop had missing coords, or the assignment predated the wiring.

### 2.8 NATS ownership
Doormile has its **own** NATS (`nats://66.116.226.161:4223`, user `doormile`),
separate from jupiter's (`nats.workolik.com:4222`). Streams are declared by the
Go app in `db/streams.go` (`EnsureStreams`, add-only — never deletes/drops).
**Adding a `js.Publish` without adding its subject to `streamSubjects` silently
drops the event.** Publishing is best-effort: `if db.Js != nil { ... }`,
warn-log on failure, never fail the request. Do not run the old Python
`setup_jetstream.py` scripts — they used to clobber the subject list (now
neutered to read-only, but that change lives only on disk, not in git).

### 2.9 Config gotcha: milers need `configid = 1001`
`LoginMiler`/`VerifyMilerPin` look up `WHERE contactno = ? AND configid = 1001`.
`AppUser.configid` column-defaults to `1`, so any miler created without
explicitly setting configid authenticates against nothing and returns a
misleading `404 no miler account found` even though the row exists and is
Active. `CreateMiler` now defaults it to 1001. **When a miler "doesn't exist"
but the row is visibly there, check `configid` first.** Same trap applies to
`AppCustomer`.

---

## Part 3 — Access control status (know before adding client logins)

- **Admin/express console (`/admin/*`) has NO tenant scoping.** `LoginAdmin`
  hardcodes `tenantid = 0` in the JWT and no admin handler filters by tenant —
  every console login sees every tenant's data. **Do not create a client-facing
  `doormile_auth` login** until this is fixed (mirror the `HubStaffAccount.Tenantid
  *int` pattern: nil = Doormile staff/unrestricted, set = client/scoped).
- **Hub console** is only partially scoped: `scopeBookingsToOwnTenant` is applied
  at ~3 of ~20 hub handlers that return booking/consignment data.
- Recurring flaw class in this codebase: **trusting a client-supplied
  identifier** (body `userid`, path `:userid`, request `tenantid`). When
  reviewing any handler, confirm identity comes from the token and ownership is
  proven before read/write. (Several account-takeover/IDOR bugs of this shape
  were fixed 2026-08-05.)

---

## Part 4 — Deliberate decisions (do not re-raise unprompted)

These are conscious calls by the project owner, recorded so they aren't
re-litigated:
- **`.env` and the Firebase key are committed to git.** Flagged as critical,
  deliberately deferred. Config also hardcodes the same values as `getEnv`
  fallbacks. Don't re-raise unless the owner opens the topic. **Do not add new
  secret literals to any committed file.**
- **`/crm/*` stays unauthenticated** — the field-sales Flutter app sends no
  token. Revisit only when that app can send a key.
- **B2C `PickupBooking.Tenantid` stays nil** — whether to attribute
  direct-to-consumer traffic to a Doormile-ops tenant is a business decision.
- **Delivery OTP stays off for DailyGrubs.**
- Hyperlocal is decided per-booking with a 30km coord fallback (Option A); a
  tenant-level service-type flag (Option B) was deferred.
- `PartnerInfo` vs `Tenant` naming, and `Customer` (legacy) vs `AppCustomer`
  (new B2C) duplication — flagged, not acted on.

---

## Part 5 — Credentials & test accounts (where they live, not the values)

Secrets are **not** reproduced here (see Part 4). Pointers:
- **App/DB/Redis/NATS secrets** — `.env` (committed) and the `doormile-secrets`
  k8s secret referenced by `miletruth.yaml`.
- **Live production DB** — `logistics` on `31.97.228.132:5433`. Read-only
  inspection via a throwaway Go program using `.env` creds is the safe path.
- **Live backend Redis** — the manifest and `.env` have historically drifted
  (stale `31.97.228.132:6379` vs live `66.116.226.255:6380`); confirm which the
  running pods actually use before trusting either.
- **Test rosters** live in Claude session memory (machine-local): the Coimbatore
  miler roster (6 riders, PIN `1234`), 3 customer app test accounts, and the
  DailyGrubs onboarding (tenant 13, master admin `developer@doormile.com`). Ask
  the owner for current values rather than assuming — they rotate.

> **Note on production writes:** the auto-mode classifier blocks production
> writes (kubectl set env, DB UPDATE/ALTER) inconsistently. Do **not** route
> around a block — hand the exact command to the owner to run. Read-only
> inspection is fine.