Files

dharaneesh-r f531b42456 updates on the md files

2026-09-07 16:52:44 +05:30

16 KiB

Raw Blame History

Doormile Backend — Developer Onboarding & Working Memory

Read this before touching the backend. It is the portable version of knowledge that otherwise lives only in one machine's Claude session memory. It covers two things a new dev (or a fresh Claude session on another machine) needs:

How we use Claude on this project — the project-memory file, the skills, the working conventions.
Operational knowledge that isn't in the code — deploy topology, build gotchas, production landmines, and the incident history behind current design choices.

Related docs already in this repo:

CLAUDE.md — the full project memory (architecture, data model, route surface). Start there for what the system is.
docs/CHANGELOG.md — summary of recent backend changes and new endpoints.
docs/customer-app-api.md — Customer App v1 API design and endpoint specs.
docs/customer-api-testing.md — step-by-step test guide and curl references for customer endpoints.
docs/openapi-customer.yaml — OpenAPI 3.0 specification for customer API.
docs/doormile-flow.md — end-to-end booking/assignment flow.
docs/miler-app-api.md, docs/express-console-api.md — API contracts.
docs/logistics-base-handover.md — the pickup-source and base-handover flow (requests 25–31), including the state transitions.
docs/jupiter2doormile.md — the legacy→new migration map.
docs/test-booking-runbook.md — how to run a test booking.
skills.md — note on the installed Claude skill pack.

Part 1 — How Claude is used on this project

1.1 Project memory (`CLAUDE.md`)

CLAUDE.md at the repo root is the single source of project context, loaded automatically into every Claude session in this repo. It is checked into git, so it travels to every machine and every dev. If you change how the system works, update CLAUDE.md in the same change — it is treated as authoritative.

Sections in CLAUDE.md are tagged [verified this session] (confirmed against source) vs [carried forward] (reported by a prior session, not re-verified). Respect the distinction — don't treat carried-forward claims as confirmed.

1.2 Session memory (machine-local — this is why this doc exists)

Claude also keeps per-fact memory files under ~/.claude/projects/<project>/memory/, indexed by MEMORY.md. These are not in git and do not travel to another machine or dev. They accumulate operational facts, incidents, and gotchas across sessions.

Part 2 below is a distillation of those files into a form the whole team can read. When a memory fact changes, update both the memory file (for Claude) and this doc (for humans).

1.3 Claude skills in use

The repo has the addyosmani/agent-skills pack installed at .agents/skills/ and symlinked into .claude/skills/. These are third-party, broad-trigger skills that run with full agent permissions. Roster (see skills.md for the full table):

api-and-interface-design, browser-testing-with-devtools, ci-cd-and-automation, code-review-and-quality, code-simplification, context-engineering, debugging-and-error-recovery, deprecation-and-migration, documentation-and-adrs, doubt-driven-development, frontend-ui-engineering, git-workflow-and-versioning, idea-refine, incremental-implementation, interview-me, observability-and-instrumentation, performance-optimization, planning-and-task-breakdown, security-and-hardening, shipping-and-launch, source-driven-development, spec-driven-development, test-driven-development, using-agent-skills.

How we treat them: available on request, not auto-adopted over the conventions in CLAUDE.md §7. The established conventions win: use the utils response helpers, the constants enums, reuse AssignMilerToBooking and haversineKM, prefer minimal-leverage fixes. Invoke a skill explicitly when its scope fits; don't let a broad trigger override house style.

1.4 Standing working preferences (from `CLAUDE.md`)

Direct, honest technical assessments. Don't declare victory early.
Production-grade from the start, minimal-effort highest-leverage fixes.
Warn before any consequential server/schema change.
Once a decision is made, proceed and report — but re-ask on money/data-correctness.

Part 2 — Operational knowledge (not derivable from the code)

2.1 Build & toolchain

Go 1.25 (go.mod says go 1.25.0). Module doormile.
On the primary dev Mac, Go is installed at /Users/tenext/go but not on PATH. Prefix: export PATH="/Users/tenext/go/bin:$PATH" before any go command, or a bare go build returns "command not found".
go build ./..., go vet ./..., go test ./... all pass repo-wide.
scratch/*.go are throwaway func main() scripts tagged //go:build ignore so the toolchain skips them. If a main redeclared error appears, it's a stripped build tag under scratch/, not a real app problem.
go build . emits a ~65MB doormile binary in the repo root; it's gitignored — delete it, don't commit it.
Tests exist only for pure logic (isHyperlocal, calculateVolumetricWeight, ParsePage). Everything else is verified by build+vet, review, or live test.

2.2 Deploy topology — how a change reaches production

Live backend runs on k3s (not plain docker), server 66.116.225.226 port 4422 (SSH, key-based auth from the primary Mac; KUBECONFIG=/etc/rancher/k3s/k3s.yaml).
Workload: statefulset doormile, namespace doormile, 3 replicas. Binary is /app/server in the pod; image docker.io/doormile/doormile-backend:latest, imagePullPolicy: Always. Pods report APP_ENV=staging.
There is NO CI. Nothing watches the repo. A kubectl rollout restart alone rebuilds nothing — it re-pulls the same image digest.

A change reaches prod only by a manual build+push (needs Docker daemon up

Docker Hub creds):

docker build -t doormile/doormile-backend:latest .
docker push  doormile/doormile-backend:latest
kubectl rollout restart statefulset/doormile -n doormile

The image builds from the working tree (ADD . /app/). Uncommitted local edits and any throwaway file under cmd//scratch/ get baked in. Always git status before building, and confirm code is committed after a deploy — git log origin/main tells you nothing about what's running.
Verify what's actually live rather than assuming: kubectl exec -n doormile doormile-0 -- ls -la /app shows the binary's build date; or probe a response field only the new code emits. Group auth (/admin/*, /hub/*, /miler/*) returns 401 for unmatched routes, so a 404-vs-401 probe can't tell you if a route exists — use an authenticated request.

2.3 Env vars & the source-of-truth manifest

Env is inline on the statefulset spec (.spec.template.spec.containers[0].env), no ConfigMap/envFrom. Set a flag with kubectl -n doormile set env statefulset/doormile KEY=value (auto-rolls). A plain restart does not add a var that isn't already in the spec.
Source-of-truth manifest: /opt/kubernetes/manifests/doormile/miletruth.yaml (git repo at /opt/kubernetes/.git; second copy under /root/kubernetes/...). A kubectl apply of it overwrites live set env changes — so any live flag change must also be written into this manifest or the next deploy reverts it.
The manifest supplies DB/Redis/NATS passwords via secretKeyRef (doormile-secrets) while the live pods carry literals — verify that secret exists before relying on kubectl apply.
Known live flag: MILER_COLLECTED_STATE_ENABLED=true (hyperlocal two-step pickup). TRUSTED_PROXIES must be set (api.doormile.com sits behind a reverse proxy) or per-IP rate limits collapse all clients into one bucket.
MILER_HUB_HANDOVER_ENABLED — default off, and must stay off until a rider build that calls POST /miler/consignments/:id/inward-at-hub is live. On, a hub-routed parcel stops at Created until the rider records the handover; off, pickup-complete marks it Inwarded_at_Hub immediately, which is what the deployed app expects. Flipping it early strands every intercity parcel on Created with no button in the app to advance it and no row in any base's received list. See logistics-base-handover.md.

2.4 Timezone convention (subtle — read before touching any time field)

The DB and backend time helpers run on IST (Asia/Kolkata) wall-clock. dbLocation = Asia/Kolkata; DBNow()/DBToday() (utils/helper.go) return IST wall-clock digits tagged as UTC. Consequence: clients (e.g. the miler app) that send a timestamp — such as logdate on POST /miler/logs — should send IST wall-clock (phone local time in India), not UTC. Sending UTC misaligns Redis zset scores and time-window queries by 5h30m.

2.5 Postgres CHECK constraints predate the codebase

Status-column CHECK constraints are not created by GORM AutoMigrate — they predate this codebase. Adding a new status constant in Go is not enough; the DB rejects it with SQLSTATE 23514. Before adding any status enum value, widen the matching *_status_check constraint in migrations/migrate.go. This is also why the base-handover reconciliation path raises a Lost exception rather than a more precise Handover_Not_Received — the latter would need consignmentexceptions widened first. Constraints exist on: consignments, pickupbookings, milerprofiles.availabilitystatus, bookingassignments, consignmentexceptions, tripsheets. consignmenthistory has no status check. (This bit us live: consignments_status_check was missing Collected_By_Miler and Cancelled, 500-ing every hyperlocal pickup and admin cancel.)

2.6 Miler telemetry pipeline (HTTP → Redis; there is NO MQTT)

MQTT is a jupiter concept; Doormile has none. Miler telemetry is 4 HTTP endpoints, identity always taken from c.Locals("userid") (never the body):

PUT /miler/location → Postgres (MilerProfile lat/lng/pincode) + Redis (miler:gps:{userid} 30min TTL, milers:locations GEO set feeding dispatch).
POST /miler/logs → Redis-only periodic telemetry point (miler_periodic_log:{userid}:{ts} + zsets, scored by timestamp).
POST /miler/status → Redis-only (miler_status:{userid}).
POST /miler/consignments/logs → Redis (list+zset) + Postgres ConsignmentHistory; takes an array (batch).

Device sensors (GPS/speed/heading/accuracy, battery/is_charging, connection) come from Flutter plugins regardless of transport — MQTT isn't needed to collect them. Console reads of the trail use newest-first fetch (ZRevRangeByScore), then reverse to chronological, so a ?limit=N window keeps the latest fixes (fixed in 6e5da09 — previously ?limit=1 returned the day's first blank early-boot ping).

2.7 Route optimizer

routes.workolik.com (env ROUTE_OPTIMIZER_URL) = the rider-bike FastAPI service (OR-Tools + Valhalla). Doormile endpoint POST /api/v1/optimization/doormile/sequence, contract matches internal/routing/optimizer.go. As of f6d339a, routing.SequenceMilerStopsAsync fires after every assignment path. A step=0 in prod is never the service being down — it means the rider had <2 active stops, a stop had missing coords, or the assignment predated the wiring.

2.8 NATS ownership

Doormile has its own NATS (nats://66.116.226.161:4223, user doormile), separate from jupiter's (nats.workolik.com:4222). Streams are declared by the Go app in db/streams.go (EnsureStreams, add-only — never deletes/drops). Adding a js.Publish without adding its subject to streamSubjects silently drops the event. Publishing is best-effort: if db.Js != nil { ... }, warn-log on failure, never fail the request. Do not run the old Python setup_jetstream.py scripts — they used to clobber the subject list (now neutered to read-only, but that change lives only on disk, not in git).

2.9 Config gotcha: milers need `configid = 1001`

LoginMiler/VerifyMilerPin look up WHERE contactno = ? AND configid = 1001. AppUser.configid column-defaults to 1, so any miler created without explicitly setting configid authenticates against nothing and returns a misleading 404 no miler account found even though the row exists and is Active. CreateMiler now defaults it to 1001. When a miler "doesn't exist" but the row is visibly there, check configid first. Same trap applies to AppCustomer.

Part 3 — Access control status (know before adding client logins)

Admin/express console (/admin/*) has NO tenant scoping. LoginAdmin hardcodes tenantid = 0 in the JWT and no admin handler filters by tenant — every console login sees every tenant's data. Do not create a client-facing doormile_auth login until this is fixed (mirror the HubStaffAccount.Tenantid *int pattern: nil = Doormile staff/unrestricted, set = client/scoped).
Hub console is only partially scoped: scopeBookingsToOwnTenant is applied at ~3 of ~20 hub handlers that return booking/consignment data.
Recurring flaw class in this codebase: trusting a client-supplied identifier (body userid, path :userid, request tenantid). When reviewing any handler, confirm identity comes from the token and ownership is proven before read/write. (Several account-takeover/IDOR bugs of this shape were fixed 2026-08-05.)

Part 4 — Deliberate decisions (do not re-raise unprompted)

These are conscious calls by the project owner, recorded so they aren't re-litigated:

.env and the Firebase key are committed to git. Flagged as critical, deliberately deferred. Config also hardcodes the same values as getEnv fallbacks. Don't re-raise unless the owner opens the topic. Do not add new secret literals to any committed file.
/crm/* stays unauthenticated — the field-sales Flutter app sends no token. Revisit only when that app can send a key.
B2C PickupBooking.Tenantid stays nil — whether to attribute direct-to-consumer traffic to a Doormile-ops tenant is a business decision.
Delivery OTP stays off for DailyGrubs.
Hyperlocal is decided per-booking with a 30km coord fallback (Option A); a tenant-level service-type flag (Option B) was deferred.
PartnerInfo vs Tenant naming, and Customer (legacy) vs AppCustomer (new B2C) duplication — flagged, not acted on.

Part 5 — Credentials & test accounts (where they live, not the values)

Secrets are not reproduced here (see Part 4). Pointers:

App/DB/Redis/NATS secrets — .env (committed) and the doormile-secrets k8s secret referenced by miletruth.yaml.
Live production DB — logistics on 31.97.228.132:5433. Read-only inspection via a throwaway Go program using .env creds is the safe path.
Live backend Redis — the manifest and .env have historically drifted (stale 31.97.228.132:6379 vs live 66.116.226.255:6380); confirm which the running pods actually use before trusting either.
Test rosters live in Claude session memory (machine-local): the Coimbatore miler roster (6 riders, PIN 1234), 3 customer app test accounts, and the DailyGrubs onboarding (tenant 13, master admin developer@doormile.com). Ask the owner for current values rather than assuming — they rotate.

Note on production writes: the auto-mode classifier blocks production writes (kubectl set env, DB UPDATE/ALTER) inconsistently. Do not route around a block — hand the exact command to the owner to run. Read-only inspection is fine.

16 KiB Raw Blame History Unescape Escape