# Doormile — Full Project Memory Portable context for a fresh Claude session working on this project on any machine or account. Covers the **whole** Doormile system, not just one slice of it. Read fully before touching the codebase. A note on provenance: sections marked **[verified this session]** were confirmed by directly reading this repo's code. Sections marked **[carried forward]** come from Suriya's own prior-session notes and have NOT been independently re-verified against source — treat them as reported state, not confirmed state, until checked. --- ## 1. What Doormile is Suriya is building **Doormile**, a real commercial logistics platform for the Indian market (Coimbatore, Hyderabad, Bengaluru, Chennai) — hub-based parcel courier/delivery, not a prototype. Treat everything as production-grade from the start. Stakeholders: Doormile's own ops team, partner tenants (client logistics/travel companies), hub staff, milers (delivery riders), and end customers. **Suriya's standing preferences** (apply without being asked): - Direct, honest technical assessments over diplomatic framing. If something isn't done, say so plainly. Don't declare victory early. - Production-grade code from the start, not "refactor later." - **Warn before any consequential server/schema change** before making it. - Concise answers — don't over-explain. - Once a decision is made, don't keep re-asking for confirmation on the same scope — proceed and report back. But *do* ask when a decision is genuinely ambiguous or touches money/data-correctness in a way that can't be safely guessed. - Minimal-effort, highest-leverage fixes over big rewrites, until he explicitly asks for the big rewrite. - Works across multiple Claude sessions/machines simultaneously, treating Claude as a co-architect across the full stack — this file exists so any of those sessions can pick up full context. --- ## 2. System architecture — the whole stack **[carried forward, substantial infra reported as built and verified in prior sessions]** - **Backend**: Go + Fiber, deployed on **Kubernetes**, at `api.doormile.com`. 200 registered routes **[verified this session, exact count]** — see §7. This is the primary booking/assignment API and the primary trigger for miler assignment, calling the AI decision layer with a 5-second timeout fallback so a slow AI response never blocks a booking. - **AI dispatch layer**: a Python autonomous agent swarm (8 agents, **JARVIS** as master orchestrator) on a dedicated server, connected to NATS JetStream. The `DispatchAgent` operates as a NATS *watcher*, not the assignment trigger — the Go backend triggers assignment directly (see §4); the agent swarm reasons about it, doesn't gate it. - **AI decision engine**: `routemate.workolik.com` — runs Claude Sonnet for miler-assignment reasoning, scoring candidates using hub load, on-time rate, and hub capacity. Uses pgvector-backed RAG memory with local embeddings (`all-MiniLM-L6-v2`, 384-dim) specifically to avoid ongoing per-call API cost. **[verified this session]**: the Go side of this is `models.AgentDecision` (`agentdecisions` table) with a `context_embedding vector(1536)` column and an ivfflat index (raw SQL in `migrations/migrate.go`, not GORM AutoMigrate) — note the dimension mismatch worth flagging (1536 in the Go model/migration vs 384 for MiniLM); worth confirming which embedding model is actually populating that column before trusting vector search quality. - **Event bus**: NATS JetStream, 6 persistent streams, dedicated server. **[verified this session]**: `db.Js` is the package-level JetStream handle; convention across the codebase is `if db.Js != nil { ... }` and warn-log on publish failure rather than failing the request — publishing is best-effort, never blocking. Confirmed publish call sites include `booking.assigned` (from `internal/assignment`) and NATS publishes inside `AssignMilerToBooking` (`controllers/booking_assignment_service.go`). - **Data stores**: PostgreSQL + pgvector 0.7.4, dedicated server, 37 tables under GORM AutoMigrate **[verified this session, exact count from `migrations/migrate.go`]**. Redis (`go-redis/v9`) for high-frequency ephemeral state — GPS pings, live miler status/location (GEO-indexed at `milers:locations`, used by `queryNearbyMilers` for assignment radius search), booking cache. Durable business state always lands in Postgres; Redis is never the system of record for anything that needs to survive a flush. - **Ingress/infra**: Kubernetes, Traefik, nginx. --- ## 3. The client-facing surfaces Five distinct front doors into this system. Only the backend API surface for each has been directly inspected this session (via `routes.go`); the actual client codebases (Flutter, React) have not been opened in this session. 1. **Customer app (B2C)** — Flutter. Auth via Firebase OTP (phone). Backend surface: 19 customer routes **[verified this session]** (`customer`/`customerAuth` groups) — booking creation, tracking, `AppCustomer`/`AppCustomerLocation`. **[carried forward]**: reported built and verified in prior sessions; the last live end-to-end test was blocked here — see §9. 2. **Miler app** — Flutter, for delivery riders. Backend surface: 38 routes **[verified this session]** (`miler`/`milerAuth` groups) — duty start/stop, GPS pings, assignment accept/reject/cancel, delivery complete/skip, break logs, support tickets. This session added `ResetMilerPin`, `MilerCancelAssignment`, `MilerSkipDelivery` to close gaps found against the old Nearle rider app (§8). 3. **Admin console** — React, converted from the old `NearlExpress` /`doormile_express_console` codebase. Backend surface: 89 routes **[verified this session]** (`admin`/`adminAuth` groups) — bookings, partner management, reports, pricing, express (formerly "CRM") bookings. This session added partner CRUD, bulk booking create/cancel, reports, password change, miler notify (§8). 4. **Hub console** — React, separate from the admin console. Backend surface: 31 routes **[verified this session]** (`hub`/`hubAuth` groups) — per-hub booking queues, tripsheet building, manual/batch assignment, hub messaging. Auth is separate from admin (`middlewares.HubStaffAuth`, sets `c.Locals("hubid")`). This session added tenant-scoping (closed a real cross-tenant leak) and `HubBatchAssign` (§8). 5. **CRM (field sales)** — a genuinely separate feature from "CRM bookings" (see §8 naming note). `POST/GET /crm/clients`, `GET/PUT/DELETE /crm/clients/:id` — 5 routes, all open/no-auth by design: field reps register leads from the Flutter side, the web console reads without a separate CRM login. Models: `DoormileClient`, `DoormileAuth`. **Not deeply audited** — only the route registration was read this session. Total: 200 routes = miler 38 + admin 89 + hub 31 + customer 19 + crm 5 + internal 5 + redisUsers 4 + bookingCache 3 + 2 top-level pricing routes + websocket routes. **[verified this session]** --- ## 4. AI dispatch & assignment — how a booking actually gets a miler **[verified this session]** — read directly from `internal/assignment/crm_assignment.go` and `booking_assignment_service.go`. - Two entry points, same core: `AssignCustomerMiler` (B2C bookings) and `AssignCRMMiler` (express/console-originated bookings — name predates this session's "CRM"→"express" rename, deliberately left unrenamed since it's a working assignment engine, not just a label). Both are fire-and-forget goroutines called after transaction commit, retrying up to 5 times, 2 minutes apart (~10 minutes total), before giving up and publishing a failure event (`publishAssignmentFailed`, `reasonNoMilerAvailable`) to NATS so the failure isn't silently invisible. - `TryAssignOnce` is the synchronous single-attempt variant — used when a hub staff member manually triggers "auto-assign" from the console and needs an immediate answer rather than the ~10-minute retry loop. Returns a rich `AutoAssignResult` (assigned/escalated, miler name, distance, AI reasoning text, candidates found) for the UI to show. - Core flow (`tryAssign`): Redis `GEOSEARCH` on `milers:locations` (10km radius, top 10, sorted nearest-first) → `selectMilerWithAI` (the actual Claude Sonnet call to `routemate.workolik.com`, logs an `AgentDecision` row with reasoning + optional embedding) → `commitAssignment` (single DB transaction: create `BookingAssignment`, update `PickupBooking.status` to `BookingMilerAssigned`, flip `MilerProfile.availabilitystatus`) → publish `booking.assigned` to NATS (best-effort) → push notifications to miler and customer. - `HubBatchAssign` (**built this session**, `controllers/hubController.go`) is a separate, simpler path: a greedy nearest-available-rider heuristic (haversine distance, no AI call, no retry loop) for clearing a hub's pending-pickup queue in one batch call, capped per rider (default 5), reusing `AssignMilerToBooking` for the actual transactional assignment. **Explicitly not** a replacement for `selectMilerWithAI`'s reasoning, and **not** a multi-stop route/VRP solver — see the routing gap below. **Logistics/routing — what exists vs what doesn't:** - What exists: hub network (`Hub` model, `originhubid`/`currenthubid`/ `destinationhubid` on bookings/consignments for tracking a parcel's hub path), `Tripsheet`/`TripsheetItem` for hub-to-hub batch transport, the nearest-rider assignment logic above, `HubBatchAssign`'s greedy queue clearer. - What does **not** exist in Doormile yet: any multi-stop route sequencing / vehicle-routing-problem solver. Nearle used external paid services (`routes.workolik.com`, and `routemate.workolik.com` was itself originally a Nearle-side concept) for that; nothing in Doormile replaces true stop-sequencing optimization today. `HubBatchAssign` only decides *who* gets which single booking, not *what order* a rider should run multiple stops in. Flag this clearly if asked whether routing is "done" — it isn't, by design, pending a real conversation about whether it's needed yet. --- ## 5. Data model reference (`models/*.go`) **[verified this session]** - `PickupBooking` (`pickupbookings`) — customer's pickup request, pre-hub. Has `Tenantid *int` (**added this session**, see §8). `Bookingsource`: `"Customer_App"` (B2C) or `"CRM_Console"` (console-created — deliberately left as `"CRM_Console"` even though the outward API/route naming was changed to "express", see §8). - `BookingParcel`, `BookingServiceOption`, `BookingPayment`, `BookingAssignment` (status: Assigned/Accepted/Rejected/Reassigned/ Completed/Cancelled, carries `AgentDecisionID *uint64` linking back to the AI reasoning that produced it), `BookingVehicleRequirement`. - `Consignment` (`consignments`) — the shipment once picked up. Has its own `Tenantid int` (not nullable, pre-existing). `Attemptcount int` (used by `MilerSkipDelivery`, added this session). `ConsignmentHistory` (event log), `ConsignmentException` (Lost/Damaged/Misrouted/Receiver_Refused/ Missing_Contents/Undeliverable). - `Hub`, `Vehicle`, `Tripsheet`, `TripsheetItem`, `DeliveryProof`. - `AppUser` (`appusers`) — shared login table for staff/miler/admin roles (`Roleid`: 1 admin, 3 manager, 4 rep/exec, 5 miler, 6 hub staff via a separate `HubStaffAccount` table). `MilerProfile` — actual rider profile (availability status, current lat/lng, rating). `MilerDutyLog`, `MilerBreakLog`, `MilerSupportTicket`. - `AppCustomer`/`AppCustomerLocation` — B2C app customers (separate from legacy `Customer`/`CustomerLocation` kept for backward compat — flagged as a future duplicate-data risk, not resolved). - `HubStaffAccount` — `Tenantid *int`: **nil = Doormile staff (sees everything)**, **set = partner-tenant staff (own tenant's data only)**. Load-bearing distinction — see the tenant-leak fix in §8. - `PartnerInfo` (`partnerinfo`) — fleet/vehicle-supplying partner companies. **Distinct concept from `Tenant`** (`tenants` — client companies Doormile delivers *for*). Confusingly close names, flagged, not acted on. - `DoormileClient`/`DoormileAuth` — the real, separate CRM feature (§3.5), not to be confused with "CRM bookings" (renamed to "express bookings" this session specifically to end that naming collision). - `Pricing`, `DoormilePricing`, `CompetitorBranch`, `CarrierPricing` — pricing engine + competitive intel. - `AgentDecision` — AI dispatch-reasoning log (§4), pgvector `context_embedding` column. - Redis-only ephemeral structs (`models/redis.go`): `MilerLog`, `MilerStatus`, `ConsignmentLog`, `CachedUser` — never durable state. --- ## 6. Current environment / seeded data state **[carried forward, not re-verified this session]** - Database seeded with 16 hubs (4 per city across the 4 target cities), 10+ milers, 10 tenants (4 Doormile-ops-owned + 6 named partner tenants), 5 hub staff accounts, 55+ pricing rules. - Credentials on file from prior sessions (values not repeated here — confirm current values rather than assuming): admin login at `suriya@doormile.com`; hub staff accounts pattern `hub.[city]@doormile.in` plus partner variants; miler test phone numbers + PINs. - Auth: Firebase OTP for customer login — this cannot be scripted/bypassed from the command line, which is why the last live E2E test stalled (§9). --- ## 7. Codebase conventions (read before writing any Go here) **[verified this session]** - **Response helpers** (`utils` package): `utils.OK(c, data)`, `utils.Created(c, data)`, `utils.Message(c, "text")`, `utils.List(c, slice, total)`, `utils.Paginated(c, slice, total, page)`, `utils.BadRequest/NotFound/Internal/Unauthorized/Forbidden/Conflict(c, msg)`. Always use these, never hand-roll `c.JSON`. - **DB**: `db.DB` is the package-level `*gorm.DB`. `db.Rdb` is Redis (`*redis.Client`, `go-redis/v9`). `db.Js` is NATS JetStream (nil-check before publishing, warn-log on failure, never fail the request over it). `db.Ctx` for Redis calls needing a context. - **Auth**: `c.Locals("userid")` (int), `c.Locals("tenantid")` (int, only set via `middlewares.AuthMiddleware` — the *requesting user's own* tenant, not necessarily the resource's tenant; conflating these two was exactly the bug fixed this session, §8). Hub staff auth is separate (`middlewares.HubStaffAuth`), sets `c.Locals("hubid")` and `c.Locals("userid")` = `hubstaffaccountid`. - **DTOs**: `dto/*.go` (`admin.go`, `auth.go`, `booking.go`, `client.go`), one struct per request shape. - **Constants** (`constants/constants.go`): every status enum lives here as typed string constants. Always use these, never inline string literals. - **Soft delete**: only some models have `Deletedat *time.Time` (`Hub`, `Vehicle`, `Consignment`, `Tripsheet`, `TripsheetItem`, `ConsignmentException`, `DoormilePricing`, `CarrierPricing`). Others (`PartnerInfo`, `Tenant`) have no soft-delete column — hard `Delete`. Check the struct before assuming either way. - **Assignment logic reuse**: `AssignMilerToBooking(bookingID, milerUserID int, assignedByUserID *int)` in `controllers/booking_assignment_service.go` is the one transactional path for "assign this miler to this booking." Reuse it, don't reimplement — both `HubAssignMiler`/`AdminAssignMiler` and `HubBatchAssign` call it. The AI-driven `commitAssignment` in `internal/assignment` is a separate transactional writer used by the auto-assignment retry path — don't conflate the two; know which one a given call site needs. - **Distance calc**: `haversineKM(lat1, lon1, lat2, lon2)` defined once in `hubController.go`, used package-wide. Don't redefine it. - **Hub tenant scoping**: `scopeBookingsToOwnTenant(c, query)` in `hubController.go` (**added this session**) — restricts a bookings query to the requesting hub staff's own tenant if partner-scoped, no-ops for Doormile staff. Use on any new hub-console bookings query. - **Go toolchain was unavailable in the sandbox that wrote this session's changes.** Every change was verified by hand (field/column names cross-checked against real structs, brace-balance via `grep -o "{" | wc -l` vs `}`) at write time. **[verified separately, on Suriya's own machine]**: `go build ./...` and `go vet ./...` were both run afterward and passed clean (only pulled two missing indirect modules, `tinylib/msgp` and `philhofer/fwd`). This was the first actual compiler verification of this code — commit `c272a33`, pushed to `origin/main`. --- ## 8. The Jupiter → Doormile migration (this session's scoped work) This section is what one specific session did: closing API/schema gaps between the legacy Nearle ("jupiter") system and the new Doormile backend, for the **rider app and console specifically** (hub console gap-closure was also done incidentally while fixing the tenant leak, but wasn't the primary target). ### 8.1 Why jupiter/Nearle was being replaced Legacy Go+Fiber backend (`backend_jupiter`, module `nearle`), Postgres, Redis, base URL `jupiter.nearle.app` (+ a separate write path `queue.workolik.com` with TLS verification disabled and a hardcoded IP pin). Consumed by a Flutter rider app and a React admin console (`doormile_express_console` — literal ancestor of the current Doormile admin console). Found to be structurally broken on live-DB analysis: - `riderlogs`: 1.17M rows, 643MB, **zero indexes**, 68M lifetime UPDATEs vs 762K inserts, caused by an unindexed `UPDATE ... WHERE userid=?` rewriting ~97K rows per call — the real root cause of read timeouts previously blamed on Redis. - Every table had only its primary key indexed; `orders`/`deliveries` never autovacuumed. - `getdeliveries` returned every row **21×** (unconstrained `LEFT JOIN tenantpricing`, `DISTINCT` over 87 columns that didn't dedupe anything). - `createdeliveries` had a quadratic insert bug (slice declared outside a loop kept accumulating) — confirmed live: 66,446 deliveries → 132,826 `deliveryqueues` rows (~2×) with duplicate `deliveryid`s. - v2 endpoints wrote **only to Redis**, invisible to v1/v3 Postgres reads — genuine split-brain, with a Redis `INCR` ID space independent of the Postgres sequence (collision risk). - `orders` had 75 columns (~20 never populated once across 137K rows). `deliveries` had 92 columns, six lat/lng pairs for 3 real points, status spread across 6 separate text+timestamp columns instead of an events table. - `PUT /deliveries/updatedelivery` was overloaded for **11 different real actions** (8 rider status transitions + 3 unrelated console actions), distinguished only by which JSON fields happened to be non-empty. - The "exhaustive" API docs undercounted real usage by ~8 endpoints (including the login endpoint itself and a whole `/v1/substitutions` CRUD feature), found only by cross-checking against actual console source. Doormile's new schema was already solving most of this structurally before this session (Redis-only telemetry, real event-log tables, 37 normalized tables with real FKs and typed columns, parcel/courier domain model instead of retail/food-delivery shaped). ### 8.2 What was built this session (14 new endpoints) | Method | Path | Handler | |---|---|---| | POST | `/miler/reset-pin` | `ResetMilerPin` | | POST | `/miler/bookings/:bookingid/cancel` | `MilerCancelAssignment` | | POST | `/miler/consignments/:id/skip` | `MilerSkipDelivery` | | GET | `/admin/reports` | `GetAdminReports` | | PUT | `/admin/profile/password` | `AdminChangePassword` | | GET | `/admin/partners` | `GetPartners` | | POST | `/admin/partners` | `CreatePartner` | | GET | `/admin/partners/:id` | `GetPartnerDetails` | | PUT | `/admin/partners/:id` | `UpdatePartner` | | DELETE | `/admin/partners/:id` | `DeletePartner` | | POST | `/admin/milers/:id/notify` | `AdminNotifyMiler` | | POST | `/admin/expressbooking/bulk` | `AdminBulkCreateBookings` | | POST | `/admin/bookings/bulk-cancel` | `AdminBulkCancelBookings` | | POST | `/hub/bookings/batch-assign` | `HubBatchAssign` | Plus two real bugs found and fixed incidentally while doing tenant- separation work (not requested, found along the way): 1. **Data mis-attribution**: `BookingPickupComplete` set the resulting `Consignment.Tenantid` from the *completing miler's own* tenant rather than the booking's actual tenant — silently mis-attributed shipments for any miler carrying parcels across tenants. Fixed to use `booking.Tenantid` when set. 2. **Cross-tenant data leak**: `GetHubUnassignedBookings`/ `GetHubBookingsRange` had no tenant scoping — a partner tenant's hub staff could see every other tenant's bookings at the same hub. Fixed via `scopeBookingsToOwnTenant`. Naming cleanup: `CreateCRMBooking`→`CreateExpressBooking`, `/admin/crmbooking`→`/admin/expressbooking` (+ `/bulk`), because "CRM bookings" collided with the genuinely separate CRM feature (§3.5). Deliberately **not** renamed: the stored `Bookingsource: "CRM_Console"` value and `internal/assignment/crm_assignment.go`'s `AssignCRMMiler` — a working assignment engine and an already-written DB value, not just a label; renaming those is a deeper change than was asked for. Deliberately skipped: rider substitutions (`/v1/substitutions` in Nearle) — Suriya's own call, looked low-traffic in the old system. ### 8.3 Verified vs not, for this migration slice **Verified**: every new handler's field/column names checked by hand against real structs; brace-balance confirmed after every edit; no duplicate symbol definitions. **`go build ./...` and `go vet ./...` both pass clean** (verified on Suriya's machine, not the sandbox that wrote the code) — committed as `c272a33`, pushed to `origin/main`. The hand-verified code compiled correctly the first time it hit a real toolchain. **Not verified**: 1. No integration test has hit any of the 14 new endpoints — a clean compile says the code is well-formed, not that it behaves correctly against a real DB/Redis/NATS. 2. The `Tenantid` migration hasn't executed against a real DB yet — will run automatically via `AutoMigrate` next deploy (additive, nullable, safe). 3. **Nothing on the client side has changed.** The rider Flutter app and `doormile_express_console` still call `jupiter.nearle.app`. API coverage existing on Doormile does not mean traffic is using it. Not a flip-a-URL cutover either — response shapes are completely different (flat 87-column Nearle rows vs nested Doormile JSON) — every screen that parses a response needs rewriting, not just repointing. --- ## 9. Current blockers & open work (whole-project level) **[carried forward]** - Last live end-to-end system test stalled on **customer JWT acquisition** — customer login requires Firebase OTP on a real phone, can't be scripted from the command line. Needs either a real-phone run or a load-test workaround. - A **load test** targeting high concurrent bookings (capacity check before go-live) was in progress at the end of the last relevant session, not finished. - Go-live preparation across the 4 target cities is still ahead. **Migration-slice-specific (§8.3)**: `go build` verification is now done (commit `c272a33`). Still pending: integration testing of the 14 new endpoints, running the `Tenantid` migration, and the client-app rewrites. --- ## 10. Deliberately skipped / open decisions - **Rider substitutions** — skipped, low old-system traffic. Revisit if it turns out to matter. - **B2C tenant attribution** — `PickupBooking.Tenantid` stays nil for B2C bookings; whether direct-to-consumer traffic should be attributed to one of the 4 Doormile-ops tenants (per city) is a business decision, not something to guess at. - **Batch route optimization** — `HubBatchAssign` is a single-booking nearest-rider heuristic, not a multi-stop VRP solver (§4). Untested against real volume vs. whatever the old paid external services provided. - **`PartnerInfo` vs `Tenant` naming confusion** — flagged, not acted on. - **`Customer`/`CustomerLocation` (legacy) vs `AppCustomer` (new B2C)** — two customer-shaped tables coexisting, flagged as a future duplicate-data risk, not resolved. - **`AgentDecision.context_embedding` dimension (1536) vs the reported MiniLM embedding size (384)** — flagged this session (§2), not investigated further; worth resolving before relying on vector search quality from that column. --- ## 11. Suggested next steps, in order 1. Resolve the customer-JWT E2E test blocker (real phone or load-test workaround) and finish the load test. 2. ~~`go build ./...` in `DoormileBackend`~~ — **done**, clean pass, commit `c272a33` pushed to `origin/main`. 3. Stand up a test/staging DB, let `AutoMigrate` run, smoke-test the 14 new migration-session endpoints with real requests. 4. Pick one low-risk console slice (e.g. Reports or Partner management — net-new UI, not replacing something live) and wire it to Doormile instead of jupiter — first real proof the cutover works end to end. 5. Only after that: rewrite the higher-traffic screens (orders/deliveries list, rider status updates) and the rider app's request/response handling. 6. Resolve the B2C tenant-attribution question before it's load-bearing for real revenue reporting. 7. Confirm the `AgentDecision` embedding-dimension question. 8. Decide on substitutions and batch-optimization sophistication once real usage data says whether they're actually needed. 9. Go-live preparation across the 4 target cities.