From e2e8b537f85e79d5c5207efda5762efe9f86cb7e Mon Sep 17 00:00:00 2001 From: Suriyakumarvijayanayagam Date: Wed, 2 Sep 2026 10:47:01 +0530 Subject: [PATCH] docs for dharaneesh --- docs/DEV_ONBOARDING.md | 259 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 259 insertions(+) create mode 100644 docs/DEV_ONBOARDING.md diff --git a/docs/DEV_ONBOARDING.md b/docs/DEV_ONBOARDING.md new file mode 100644 index 0000000..8438527 --- /dev/null +++ b/docs/DEV_ONBOARDING.md @@ -0,0 +1,259 @@ +# Doormile Backend — Developer Onboarding & Working Memory + +Read this before touching the backend. It is the portable version of knowledge +that otherwise lives only in one machine's Claude session memory. It covers two +things a new dev (or a fresh Claude session on another machine) needs: + +1. **How we use Claude on this project** — the project-memory file, the skills, + the working conventions. +2. **Operational knowledge that isn't in the code** — deploy topology, build + gotchas, production landmines, and the incident history behind current + design choices. + +Related docs already in this repo: +- [`CLAUDE.md`](../CLAUDE.md) — the full project memory (architecture, data + model, route surface). Start there for *what the system is*. +- [`docs/doormile-flow.md`](doormile-flow.md) — end-to-end booking/assignment flow. +- [`docs/miler-app-api.md`](miler-app-api.md), [`docs/express-console-api.md`](express-console-api.md) — API contracts. +- [`docs/jupiter2doormile.md`](jupiter2doormile.md) — the legacy→new migration map. +- [`docs/test-booking-runbook.md`](test-booking-runbook.md) — how to run a test booking. +- [`skills.md`](../skills.md) — note on the installed Claude skill pack. + +--- + +## Part 1 — How Claude is used on this project + +### 1.1 Project memory (`CLAUDE.md`) +`CLAUDE.md` at the repo root is the single source of project context, loaded +automatically into every Claude session in this repo. It is checked into git, so +it travels to every machine and every dev. If you change how the system works, +update `CLAUDE.md` in the same change — it is treated as authoritative. + +Sections in `CLAUDE.md` are tagged **[verified this session]** (confirmed +against source) vs **[carried forward]** (reported by a prior session, not +re-verified). Respect the distinction — don't treat carried-forward claims as +confirmed. + +### 1.2 Session memory (machine-local — this is why this doc exists) +Claude also keeps per-fact memory files under +`~/.claude/projects//memory/`, indexed by `MEMORY.md`. These are +**not in git** and **do not travel** to another machine or dev. They accumulate +operational facts, incidents, and gotchas across sessions. + +Part 2 below is a distillation of those files into a form the whole team can +read. When a memory fact changes, update *both* the memory file (for Claude) and +this doc (for humans). + +### 1.3 Claude skills in use +The repo has the `addyosmani/agent-skills` pack installed at `.agents/skills/` +and symlinked into `.claude/skills/`. These are third-party, broad-trigger +skills that run with full agent permissions. Roster (see `skills.md` for the +full table): + +`api-and-interface-design`, `browser-testing-with-devtools`, +`ci-cd-and-automation`, `code-review-and-quality`, `code-simplification`, +`context-engineering`, `debugging-and-error-recovery`, +`deprecation-and-migration`, `documentation-and-adrs`, +`doubt-driven-development`, `frontend-ui-engineering`, +`git-workflow-and-versioning`, `idea-refine`, `incremental-implementation`, +`interview-me`, `observability-and-instrumentation`, +`performance-optimization`, `planning-and-task-breakdown`, +`security-and-hardening`, `shipping-and-launch`, `source-driven-development`, +`spec-driven-development`, `test-driven-development`, `using-agent-skills`. + +**How we treat them:** available on request, *not* auto-adopted over the +conventions in `CLAUDE.md` §7. The established conventions win: use the `utils` +response helpers, the `constants` enums, reuse `AssignMilerToBooking` and +`haversineKM`, prefer minimal-leverage fixes. Invoke a skill explicitly when its +scope fits; don't let a broad trigger override house style. + +### 1.4 Standing working preferences (from `CLAUDE.md`) +- Direct, honest technical assessments. Don't declare victory early. +- Production-grade from the start, minimal-effort highest-leverage fixes. +- **Warn before any consequential server/schema change.** +- Once a decision is made, proceed and report — but re-ask on money/data-correctness. + +--- + +## Part 2 — Operational knowledge (not derivable from the code) + +### 2.1 Build & toolchain +- Go **1.25** (`go.mod` says `go 1.25.0`). Module `doormile`. +- On the primary dev Mac, Go is installed at `/Users/tenext/go` but **not on + PATH**. Prefix: `export PATH="/Users/tenext/go/bin:$PATH"` before any `go` + command, or a bare `go build` returns "command not found". +- `go build ./...`, `go vet ./...`, `go test ./...` all pass repo-wide. +- `scratch/*.go` are throwaway `func main()` scripts tagged `//go:build ignore` + so the toolchain skips them. If a `main redeclared` error appears, it's a + stripped build tag under `scratch/`, **not** a real app problem. +- `go build .` emits a ~65MB `doormile` binary in the repo root; it's + gitignored — delete it, don't commit it. +- Tests exist only for pure logic (`isHyperlocal`, `calculateVolumetricWeight`, + `ParsePage`). Everything else is verified by build+vet, review, or live test. + +### 2.2 Deploy topology — how a change reaches production +- Live backend runs on **k3s** (not plain docker), server `66.116.225.226` port + **4422** (SSH, key-based auth from the primary Mac; `KUBECONFIG=/etc/rancher/k3s/k3s.yaml`). +- Workload: **statefulset `doormile`**, namespace `doormile`, **3 replicas**. + Binary is `/app/server` in the pod; image + `docker.io/doormile/doormile-backend:latest`, `imagePullPolicy: Always`. + Pods report `APP_ENV=staging`. +- **There is NO CI.** Nothing watches the repo. A `kubectl rollout restart` + alone rebuilds nothing — it re-pulls the same image digest. +- **A change reaches prod only by a manual build+push** (needs Docker daemon up + + Docker Hub creds): + ``` + docker build -t doormile/doormile-backend:latest . + docker push doormile/doormile-backend:latest + kubectl rollout restart statefulset/doormile -n doormile + ``` +- **The image builds from the working tree** (`ADD . /app/`). Uncommitted local + edits and any throwaway file under `cmd/`/`scratch/` get baked in. **Always + `git status` before building**, and **confirm code is committed *after* a + deploy** — `git log origin/main` tells you nothing about what's running. +- **Verify what's actually live** rather than assuming: `kubectl exec -n + doormile doormile-0 -- ls -la /app` shows the binary's build date; or probe a + response field only the new code emits. Group auth (`/admin/*`, `/hub/*`, + `/miler/*`) returns 401 for unmatched routes, so a 404-vs-401 probe can't tell + you if a route exists — use an authenticated request. + +### 2.3 Env vars & the source-of-truth manifest +- Env is **inline on the statefulset spec** (`.spec.template.spec.containers[0].env`), + no ConfigMap/envFrom. Set a flag with + `kubectl -n doormile set env statefulset/doormile KEY=value` (auto-rolls). A + plain restart does **not** add a var that isn't already in the spec. +- **Source-of-truth manifest**: `/opt/kubernetes/manifests/doormile/miletruth.yaml` + (git repo at `/opt/kubernetes/.git`; second copy under `/root/kubernetes/...`). + A `kubectl apply` of it **overwrites** live `set env` changes — so any live + flag change must also be written into this manifest or the next deploy reverts + it. +- The manifest supplies DB/Redis/NATS passwords via `secretKeyRef` + (`doormile-secrets`) while the live pods carry literals — verify that secret + exists before relying on `kubectl apply`. +- **Known live flag:** `MILER_COLLECTED_STATE_ENABLED=true` (hyperlocal two-step + pickup). `TRUSTED_PROXIES` must be set (api.doormile.com sits behind a + reverse proxy) or per-IP rate limits collapse all clients into one bucket. + +### 2.4 Timezone convention (subtle — read before touching any time field) +The DB and backend time helpers run on **IST (Asia/Kolkata) wall-clock**. +`dbLocation = Asia/Kolkata`; `DBNow()`/`DBToday()` (`utils/helper.go`) return +IST wall-clock digits *tagged as UTC*. Consequence: clients (e.g. the miler app) +that send a timestamp — such as `logdate` on `POST /miler/logs` — should send +**IST wall-clock (phone local time in India), not UTC**. Sending UTC misaligns +Redis zset scores and time-window queries by 5h30m. + +### 2.5 Postgres CHECK constraints predate the codebase +Status-column CHECK constraints are **not** created by GORM AutoMigrate — they +predate this codebase. Adding a new status *constant* in Go is not enough; the +DB rejects it with **SQLSTATE 23514**. Before adding any status enum value, +widen the matching `*_status_check` constraint in `migrations/migrate.go`. +Constraints exist on: `consignments`, `pickupbookings`, +`milerprofiles.availabilitystatus`, `bookingassignments`, +`consignmentexceptions`, `tripsheets`. `consignmenthistory` has no status check. +(This bit us live: `consignments_status_check` was missing `Collected_By_Miler` +and `Cancelled`, 500-ing every hyperlocal pickup and admin cancel.) + +### 2.6 Miler telemetry pipeline (HTTP → Redis; there is NO MQTT) +MQTT is a jupiter concept; Doormile has none. Miler telemetry is 4 HTTP +endpoints, identity always taken from `c.Locals("userid")` (never the body): +- `PUT /miler/location` → Postgres (`MilerProfile` lat/lng/pincode) + Redis + (`miler:gps:{userid}` 30min TTL, `milers:locations` GEO set feeding dispatch). +- `POST /miler/logs` → **Redis-only** periodic telemetry point + (`miler_periodic_log:{userid}:{ts}` + zsets, scored by timestamp). +- `POST /miler/status` → Redis-only (`miler_status:{userid}`). +- `POST /miler/consignments/logs` → Redis (list+zset) + Postgres + `ConsignmentHistory`; takes an **array** (batch). + +Device sensors (GPS/speed/heading/accuracy, battery/is_charging, connection) +come from Flutter plugins regardless of transport — MQTT isn't needed to collect +them. Console reads of the trail use **newest-first** fetch +(`ZRevRangeByScore`), then reverse to chronological, so a `?limit=N` window +keeps the *latest* fixes (fixed in `6e5da09` — previously `?limit=1` returned +the day's first blank early-boot ping). + +### 2.7 Route optimizer +`routes.workolik.com` (env `ROUTE_OPTIMIZER_URL`) = the rider-bike FastAPI +service (OR-Tools + Valhalla). Doormile endpoint +`POST /api/v1/optimization/doormile/sequence`, contract matches +`internal/routing/optimizer.go`. As of `f6d339a`, +`routing.SequenceMilerStopsAsync` fires after every assignment path. A `step=0` +in prod is never the service being down — it means the rider had <2 active +stops, a stop had missing coords, or the assignment predated the wiring. + +### 2.8 NATS ownership +Doormile has its **own** NATS (`nats://66.116.226.161:4223`, user `doormile`), +separate from jupiter's (`nats.workolik.com:4222`). Streams are declared by the +Go app in `db/streams.go` (`EnsureStreams`, add-only — never deletes/drops). +**Adding a `js.Publish` without adding its subject to `streamSubjects` silently +drops the event.** Publishing is best-effort: `if db.Js != nil { ... }`, +warn-log on failure, never fail the request. Do not run the old Python +`setup_jetstream.py` scripts — they used to clobber the subject list (now +neutered to read-only, but that change lives only on disk, not in git). + +### 2.9 Config gotcha: milers need `configid = 1001` +`LoginMiler`/`VerifyMilerPin` look up `WHERE contactno = ? AND configid = 1001`. +`AppUser.configid` column-defaults to `1`, so any miler created without +explicitly setting configid authenticates against nothing and returns a +misleading `404 no miler account found` even though the row exists and is +Active. `CreateMiler` now defaults it to 1001. **When a miler "doesn't exist" +but the row is visibly there, check `configid` first.** Same trap applies to +`AppCustomer`. + +--- + +## Part 3 — Access control status (know before adding client logins) + +- **Admin/express console (`/admin/*`) has NO tenant scoping.** `LoginAdmin` + hardcodes `tenantid = 0` in the JWT and no admin handler filters by tenant — + every console login sees every tenant's data. **Do not create a client-facing + `doormile_auth` login** until this is fixed (mirror the `HubStaffAccount.Tenantid + *int` pattern: nil = Doormile staff/unrestricted, set = client/scoped). +- **Hub console** is only partially scoped: `scopeBookingsToOwnTenant` is applied + at ~3 of ~20 hub handlers that return booking/consignment data. +- Recurring flaw class in this codebase: **trusting a client-supplied + identifier** (body `userid`, path `:userid`, request `tenantid`). When + reviewing any handler, confirm identity comes from the token and ownership is + proven before read/write. (Several account-takeover/IDOR bugs of this shape + were fixed 2026-08-05.) + +--- + +## Part 4 — Deliberate decisions (do not re-raise unprompted) + +These are conscious calls by the project owner, recorded so they aren't +re-litigated: +- **`.env` and the Firebase key are committed to git.** Flagged as critical, + deliberately deferred. Config also hardcodes the same values as `getEnv` + fallbacks. Don't re-raise unless the owner opens the topic. **Do not add new + secret literals to any committed file.** +- **`/crm/*` stays unauthenticated** — the field-sales Flutter app sends no + token. Revisit only when that app can send a key. +- **B2C `PickupBooking.Tenantid` stays nil** — whether to attribute + direct-to-consumer traffic to a Doormile-ops tenant is a business decision. +- **Delivery OTP stays off for DailyGrubs.** +- Hyperlocal is decided per-booking with a 30km coord fallback (Option A); a + tenant-level service-type flag (Option B) was deferred. +- `PartnerInfo` vs `Tenant` naming, and `Customer` (legacy) vs `AppCustomer` + (new B2C) duplication — flagged, not acted on. + +--- + +## Part 5 — Credentials & test accounts (where they live, not the values) + +Secrets are **not** reproduced here (see Part 4). Pointers: +- **App/DB/Redis/NATS secrets** — `.env` (committed) and the `doormile-secrets` + k8s secret referenced by `miletruth.yaml`. +- **Live production DB** — `logistics` on `31.97.228.132:5433`. Read-only + inspection via a throwaway Go program using `.env` creds is the safe path. +- **Live backend Redis** — the manifest and `.env` have historically drifted + (stale `31.97.228.132:6379` vs live `66.116.226.255:6380`); confirm which the + running pods actually use before trusting either. +- **Test rosters** live in Claude session memory (machine-local): the Coimbatore + miler roster (6 riders, PIN `1234`), 3 customer app test accounts, and the + DailyGrubs onboarding (tenant 13, master admin `developer@doormile.com`). Ask + the owner for current values rather than assuming — they rotate. + +> **Note on production writes:** the auto-mode classifier blocks production +> writes (kubectl set env, DB UPDATE/ALTER) inconsistently. Do **not** route +> around a block — hand the exact command to the owner to run. Read-only +> inspection is fine.