docs for dharaneesh
This commit is contained in:
259
docs/DEV_ONBOARDING.md
Normal file
259
docs/DEV_ONBOARDING.md
Normal file
@@ -0,0 +1,259 @@
|
||||
# Doormile Backend — Developer Onboarding & Working Memory
|
||||
|
||||
Read this before touching the backend. It is the portable version of knowledge
|
||||
that otherwise lives only in one machine's Claude session memory. It covers two
|
||||
things a new dev (or a fresh Claude session on another machine) needs:
|
||||
|
||||
1. **How we use Claude on this project** — the project-memory file, the skills,
|
||||
the working conventions.
|
||||
2. **Operational knowledge that isn't in the code** — deploy topology, build
|
||||
gotchas, production landmines, and the incident history behind current
|
||||
design choices.
|
||||
|
||||
Related docs already in this repo:
|
||||
- [`CLAUDE.md`](../CLAUDE.md) — the full project memory (architecture, data
|
||||
model, route surface). Start there for *what the system is*.
|
||||
- [`docs/doormile-flow.md`](doormile-flow.md) — end-to-end booking/assignment flow.
|
||||
- [`docs/miler-app-api.md`](miler-app-api.md), [`docs/express-console-api.md`](express-console-api.md) — API contracts.
|
||||
- [`docs/jupiter2doormile.md`](jupiter2doormile.md) — the legacy→new migration map.
|
||||
- [`docs/test-booking-runbook.md`](test-booking-runbook.md) — how to run a test booking.
|
||||
- [`skills.md`](../skills.md) — note on the installed Claude skill pack.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — How Claude is used on this project
|
||||
|
||||
### 1.1 Project memory (`CLAUDE.md`)
|
||||
`CLAUDE.md` at the repo root is the single source of project context, loaded
|
||||
automatically into every Claude session in this repo. It is checked into git, so
|
||||
it travels to every machine and every dev. If you change how the system works,
|
||||
update `CLAUDE.md` in the same change — it is treated as authoritative.
|
||||
|
||||
Sections in `CLAUDE.md` are tagged **[verified this session]** (confirmed
|
||||
against source) vs **[carried forward]** (reported by a prior session, not
|
||||
re-verified). Respect the distinction — don't treat carried-forward claims as
|
||||
confirmed.
|
||||
|
||||
### 1.2 Session memory (machine-local — this is why this doc exists)
|
||||
Claude also keeps per-fact memory files under
|
||||
`~/.claude/projects/<project>/memory/`, indexed by `MEMORY.md`. These are
|
||||
**not in git** and **do not travel** to another machine or dev. They accumulate
|
||||
operational facts, incidents, and gotchas across sessions.
|
||||
|
||||
Part 2 below is a distillation of those files into a form the whole team can
|
||||
read. When a memory fact changes, update *both* the memory file (for Claude) and
|
||||
this doc (for humans).
|
||||
|
||||
### 1.3 Claude skills in use
|
||||
The repo has the `addyosmani/agent-skills` pack installed at `.agents/skills/`
|
||||
and symlinked into `.claude/skills/`. These are third-party, broad-trigger
|
||||
skills that run with full agent permissions. Roster (see `skills.md` for the
|
||||
full table):
|
||||
|
||||
`api-and-interface-design`, `browser-testing-with-devtools`,
|
||||
`ci-cd-and-automation`, `code-review-and-quality`, `code-simplification`,
|
||||
`context-engineering`, `debugging-and-error-recovery`,
|
||||
`deprecation-and-migration`, `documentation-and-adrs`,
|
||||
`doubt-driven-development`, `frontend-ui-engineering`,
|
||||
`git-workflow-and-versioning`, `idea-refine`, `incremental-implementation`,
|
||||
`interview-me`, `observability-and-instrumentation`,
|
||||
`performance-optimization`, `planning-and-task-breakdown`,
|
||||
`security-and-hardening`, `shipping-and-launch`, `source-driven-development`,
|
||||
`spec-driven-development`, `test-driven-development`, `using-agent-skills`.
|
||||
|
||||
**How we treat them:** available on request, *not* auto-adopted over the
|
||||
conventions in `CLAUDE.md` §7. The established conventions win: use the `utils`
|
||||
response helpers, the `constants` enums, reuse `AssignMilerToBooking` and
|
||||
`haversineKM`, prefer minimal-leverage fixes. Invoke a skill explicitly when its
|
||||
scope fits; don't let a broad trigger override house style.
|
||||
|
||||
### 1.4 Standing working preferences (from `CLAUDE.md`)
|
||||
- Direct, honest technical assessments. Don't declare victory early.
|
||||
- Production-grade from the start, minimal-effort highest-leverage fixes.
|
||||
- **Warn before any consequential server/schema change.**
|
||||
- Once a decision is made, proceed and report — but re-ask on money/data-correctness.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — Operational knowledge (not derivable from the code)
|
||||
|
||||
### 2.1 Build & toolchain
|
||||
- Go **1.25** (`go.mod` says `go 1.25.0`). Module `doormile`.
|
||||
- On the primary dev Mac, Go is installed at `/Users/tenext/go` but **not on
|
||||
PATH**. Prefix: `export PATH="/Users/tenext/go/bin:$PATH"` before any `go`
|
||||
command, or a bare `go build` returns "command not found".
|
||||
- `go build ./...`, `go vet ./...`, `go test ./...` all pass repo-wide.
|
||||
- `scratch/*.go` are throwaway `func main()` scripts tagged `//go:build ignore`
|
||||
so the toolchain skips them. If a `main redeclared` error appears, it's a
|
||||
stripped build tag under `scratch/`, **not** a real app problem.
|
||||
- `go build .` emits a ~65MB `doormile` binary in the repo root; it's
|
||||
gitignored — delete it, don't commit it.
|
||||
- Tests exist only for pure logic (`isHyperlocal`, `calculateVolumetricWeight`,
|
||||
`ParsePage`). Everything else is verified by build+vet, review, or live test.
|
||||
|
||||
### 2.2 Deploy topology — how a change reaches production
|
||||
- Live backend runs on **k3s** (not plain docker), server `66.116.225.226` port
|
||||
**4422** (SSH, key-based auth from the primary Mac; `KUBECONFIG=/etc/rancher/k3s/k3s.yaml`).
|
||||
- Workload: **statefulset `doormile`**, namespace `doormile`, **3 replicas**.
|
||||
Binary is `/app/server` in the pod; image
|
||||
`docker.io/doormile/doormile-backend:latest`, `imagePullPolicy: Always`.
|
||||
Pods report `APP_ENV=staging`.
|
||||
- **There is NO CI.** Nothing watches the repo. A `kubectl rollout restart`
|
||||
alone rebuilds nothing — it re-pulls the same image digest.
|
||||
- **A change reaches prod only by a manual build+push** (needs Docker daemon up
|
||||
+ Docker Hub creds):
|
||||
```
|
||||
docker build -t doormile/doormile-backend:latest .
|
||||
docker push doormile/doormile-backend:latest
|
||||
kubectl rollout restart statefulset/doormile -n doormile
|
||||
```
|
||||
- **The image builds from the working tree** (`ADD . /app/`). Uncommitted local
|
||||
edits and any throwaway file under `cmd/`/`scratch/` get baked in. **Always
|
||||
`git status` before building**, and **confirm code is committed *after* a
|
||||
deploy** — `git log origin/main` tells you nothing about what's running.
|
||||
- **Verify what's actually live** rather than assuming: `kubectl exec -n
|
||||
doormile doormile-0 -- ls -la /app` shows the binary's build date; or probe a
|
||||
response field only the new code emits. Group auth (`/admin/*`, `/hub/*`,
|
||||
`/miler/*`) returns 401 for unmatched routes, so a 404-vs-401 probe can't tell
|
||||
you if a route exists — use an authenticated request.
|
||||
|
||||
### 2.3 Env vars & the source-of-truth manifest
|
||||
- Env is **inline on the statefulset spec** (`.spec.template.spec.containers[0].env`),
|
||||
no ConfigMap/envFrom. Set a flag with
|
||||
`kubectl -n doormile set env statefulset/doormile KEY=value` (auto-rolls). A
|
||||
plain restart does **not** add a var that isn't already in the spec.
|
||||
- **Source-of-truth manifest**: `/opt/kubernetes/manifests/doormile/miletruth.yaml`
|
||||
(git repo at `/opt/kubernetes/.git`; second copy under `/root/kubernetes/...`).
|
||||
A `kubectl apply` of it **overwrites** live `set env` changes — so any live
|
||||
flag change must also be written into this manifest or the next deploy reverts
|
||||
it.
|
||||
- The manifest supplies DB/Redis/NATS passwords via `secretKeyRef`
|
||||
(`doormile-secrets`) while the live pods carry literals — verify that secret
|
||||
exists before relying on `kubectl apply`.
|
||||
- **Known live flag:** `MILER_COLLECTED_STATE_ENABLED=true` (hyperlocal two-step
|
||||
pickup). `TRUSTED_PROXIES` must be set (api.doormile.com sits behind a
|
||||
reverse proxy) or per-IP rate limits collapse all clients into one bucket.
|
||||
|
||||
### 2.4 Timezone convention (subtle — read before touching any time field)
|
||||
The DB and backend time helpers run on **IST (Asia/Kolkata) wall-clock**.
|
||||
`dbLocation = Asia/Kolkata`; `DBNow()`/`DBToday()` (`utils/helper.go`) return
|
||||
IST wall-clock digits *tagged as UTC*. Consequence: clients (e.g. the miler app)
|
||||
that send a timestamp — such as `logdate` on `POST /miler/logs` — should send
|
||||
**IST wall-clock (phone local time in India), not UTC**. Sending UTC misaligns
|
||||
Redis zset scores and time-window queries by 5h30m.
|
||||
|
||||
### 2.5 Postgres CHECK constraints predate the codebase
|
||||
Status-column CHECK constraints are **not** created by GORM AutoMigrate — they
|
||||
predate this codebase. Adding a new status *constant* in Go is not enough; the
|
||||
DB rejects it with **SQLSTATE 23514**. Before adding any status enum value,
|
||||
widen the matching `*_status_check` constraint in `migrations/migrate.go`.
|
||||
Constraints exist on: `consignments`, `pickupbookings`,
|
||||
`milerprofiles.availabilitystatus`, `bookingassignments`,
|
||||
`consignmentexceptions`, `tripsheets`. `consignmenthistory` has no status check.
|
||||
(This bit us live: `consignments_status_check` was missing `Collected_By_Miler`
|
||||
and `Cancelled`, 500-ing every hyperlocal pickup and admin cancel.)
|
||||
|
||||
### 2.6 Miler telemetry pipeline (HTTP → Redis; there is NO MQTT)
|
||||
MQTT is a jupiter concept; Doormile has none. Miler telemetry is 4 HTTP
|
||||
endpoints, identity always taken from `c.Locals("userid")` (never the body):
|
||||
- `PUT /miler/location` → Postgres (`MilerProfile` lat/lng/pincode) + Redis
|
||||
(`miler:gps:{userid}` 30min TTL, `milers:locations` GEO set feeding dispatch).
|
||||
- `POST /miler/logs` → **Redis-only** periodic telemetry point
|
||||
(`miler_periodic_log:{userid}:{ts}` + zsets, scored by timestamp).
|
||||
- `POST /miler/status` → Redis-only (`miler_status:{userid}`).
|
||||
- `POST /miler/consignments/logs` → Redis (list+zset) + Postgres
|
||||
`ConsignmentHistory`; takes an **array** (batch).
|
||||
|
||||
Device sensors (GPS/speed/heading/accuracy, battery/is_charging, connection)
|
||||
come from Flutter plugins regardless of transport — MQTT isn't needed to collect
|
||||
them. Console reads of the trail use **newest-first** fetch
|
||||
(`ZRevRangeByScore`), then reverse to chronological, so a `?limit=N` window
|
||||
keeps the *latest* fixes (fixed in `6e5da09` — previously `?limit=1` returned
|
||||
the day's first blank early-boot ping).
|
||||
|
||||
### 2.7 Route optimizer
|
||||
`routes.workolik.com` (env `ROUTE_OPTIMIZER_URL`) = the rider-bike FastAPI
|
||||
service (OR-Tools + Valhalla). Doormile endpoint
|
||||
`POST /api/v1/optimization/doormile/sequence`, contract matches
|
||||
`internal/routing/optimizer.go`. As of `f6d339a`,
|
||||
`routing.SequenceMilerStopsAsync` fires after every assignment path. A `step=0`
|
||||
in prod is never the service being down — it means the rider had <2 active
|
||||
stops, a stop had missing coords, or the assignment predated the wiring.
|
||||
|
||||
### 2.8 NATS ownership
|
||||
Doormile has its **own** NATS (`nats://66.116.226.161:4223`, user `doormile`),
|
||||
separate from jupiter's (`nats.workolik.com:4222`). Streams are declared by the
|
||||
Go app in `db/streams.go` (`EnsureStreams`, add-only — never deletes/drops).
|
||||
**Adding a `js.Publish` without adding its subject to `streamSubjects` silently
|
||||
drops the event.** Publishing is best-effort: `if db.Js != nil { ... }`,
|
||||
warn-log on failure, never fail the request. Do not run the old Python
|
||||
`setup_jetstream.py` scripts — they used to clobber the subject list (now
|
||||
neutered to read-only, but that change lives only on disk, not in git).
|
||||
|
||||
### 2.9 Config gotcha: milers need `configid = 1001`
|
||||
`LoginMiler`/`VerifyMilerPin` look up `WHERE contactno = ? AND configid = 1001`.
|
||||
`AppUser.configid` column-defaults to `1`, so any miler created without
|
||||
explicitly setting configid authenticates against nothing and returns a
|
||||
misleading `404 no miler account found` even though the row exists and is
|
||||
Active. `CreateMiler` now defaults it to 1001. **When a miler "doesn't exist"
|
||||
but the row is visibly there, check `configid` first.** Same trap applies to
|
||||
`AppCustomer`.
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — Access control status (know before adding client logins)
|
||||
|
||||
- **Admin/express console (`/admin/*`) has NO tenant scoping.** `LoginAdmin`
|
||||
hardcodes `tenantid = 0` in the JWT and no admin handler filters by tenant —
|
||||
every console login sees every tenant's data. **Do not create a client-facing
|
||||
`doormile_auth` login** until this is fixed (mirror the `HubStaffAccount.Tenantid
|
||||
*int` pattern: nil = Doormile staff/unrestricted, set = client/scoped).
|
||||
- **Hub console** is only partially scoped: `scopeBookingsToOwnTenant` is applied
|
||||
at ~3 of ~20 hub handlers that return booking/consignment data.
|
||||
- Recurring flaw class in this codebase: **trusting a client-supplied
|
||||
identifier** (body `userid`, path `:userid`, request `tenantid`). When
|
||||
reviewing any handler, confirm identity comes from the token and ownership is
|
||||
proven before read/write. (Several account-takeover/IDOR bugs of this shape
|
||||
were fixed 2026-08-05.)
|
||||
|
||||
---
|
||||
|
||||
## Part 4 — Deliberate decisions (do not re-raise unprompted)
|
||||
|
||||
These are conscious calls by the project owner, recorded so they aren't
|
||||
re-litigated:
|
||||
- **`.env` and the Firebase key are committed to git.** Flagged as critical,
|
||||
deliberately deferred. Config also hardcodes the same values as `getEnv`
|
||||
fallbacks. Don't re-raise unless the owner opens the topic. **Do not add new
|
||||
secret literals to any committed file.**
|
||||
- **`/crm/*` stays unauthenticated** — the field-sales Flutter app sends no
|
||||
token. Revisit only when that app can send a key.
|
||||
- **B2C `PickupBooking.Tenantid` stays nil** — whether to attribute
|
||||
direct-to-consumer traffic to a Doormile-ops tenant is a business decision.
|
||||
- **Delivery OTP stays off for DailyGrubs.**
|
||||
- Hyperlocal is decided per-booking with a 30km coord fallback (Option A); a
|
||||
tenant-level service-type flag (Option B) was deferred.
|
||||
- `PartnerInfo` vs `Tenant` naming, and `Customer` (legacy) vs `AppCustomer`
|
||||
(new B2C) duplication — flagged, not acted on.
|
||||
|
||||
---
|
||||
|
||||
## Part 5 — Credentials & test accounts (where they live, not the values)
|
||||
|
||||
Secrets are **not** reproduced here (see Part 4). Pointers:
|
||||
- **App/DB/Redis/NATS secrets** — `.env` (committed) and the `doormile-secrets`
|
||||
k8s secret referenced by `miletruth.yaml`.
|
||||
- **Live production DB** — `logistics` on `31.97.228.132:5433`. Read-only
|
||||
inspection via a throwaway Go program using `.env` creds is the safe path.
|
||||
- **Live backend Redis** — the manifest and `.env` have historically drifted
|
||||
(stale `31.97.228.132:6379` vs live `66.116.226.255:6380`); confirm which the
|
||||
running pods actually use before trusting either.
|
||||
- **Test rosters** live in Claude session memory (machine-local): the Coimbatore
|
||||
miler roster (6 riders, PIN `1234`), 3 customer app test accounts, and the
|
||||
DailyGrubs onboarding (tenant 13, master admin `developer@doormile.com`). Ask
|
||||
the owner for current values rather than assuming — they rotate.
|
||||
|
||||
> **Note on production writes:** the auto-mode classifier blocks production
|
||||
> writes (kubectl set env, DB UPDATE/ALTER) inconsistently. Do **not** route
|
||||
> around a block — hand the exact command to the owner to run. Read-only
|
||||
> inspection is fine.
|
||||
Reference in New Issue
Block a user