updates on the reverse logistics

docs/reverse-logistics-plan.md

@@ -0,0 +1,279 @@
+# Reverse Logistics — Analysis & Implementation Plan
+
+Status: **Phases 1–3 built 2026-10-05 (uncommitted, not deployed)** — see §10.
+Drafted 2026-10-05.
+Scope: `doormile_backend` (rules + endpoints), `krow_talent_app` (ops console),
+and the rider app (Flutter, separate team). The customer app is touched only
+where noted.
+
+Defaults below are **proposals**; every one marked ⚑ is an open decision
+listed in §9.
+
+---
+
+## 1. What exists today (verified in code)
+
+| Area | Finding | Where |
+|---|---|---|
+| Return statuses | `RTO_Initiated`, `Returned_to_Sender` are defined and **allowed by the DB constraint** — but **nothing in the code ever sets them** | `doormile_backend/constants/constants.go:91-92`, `migrations/migrate.go:124` |
+| Return fields on the parcel | `Consignment` already has `Returnreason`, `Returninitiatedat`, `Returndeliveredat`, `Parentconsignmentid` — **all unused** | `doormile_backend/models/audit.go:70-73` |
+| Failed delivery | Rider taps *skip* → `MilerSkipDelivery` adds 1 to `Attemptcount`, logs `Delivery_Skipped` history. After **3** attempts it opens an `Undeliverable` exception — and stops there. The parcel stays `Out_for_Delivery` with the rider; no return leg, no owner, client not told | `doormile_backend/controllers/milerAppController.go:990-1070` |
+| Exception types | `Receiver_Refused`, `Undeliverable`, `Damaged`, `Lost`, `Misrouted`, `Missing_Contents` exist | `constants.go:186-195` |
+| Rider "what next" | `nextActionForConsignment` maps status → rider action. `RTO`/`Returned` fall into `none` ("past this rider's leg") — so a rider is never asked to bring a parcel back | `controllers/logisticsHandoverController.go:233-251` |
+| Admin status change | `PUT /admin/consignments/:id/status` accepts **any** status string with no transition rules (a separate bug: it can set `Delivered` on a cancelled parcel) | `controllers/adminController.go:3133-3190` |
+| Console | Only traces: an `rto` badge tone (`components/ds/StatusBadge.jsx:47`), `rto → skipped` mapping (`api/doormile/queries.js:110`), a comment in `lib/orderFlow.js:108`. Status update offers only Out_for_Delivery / Delivered / Cancelled | `krow_talent_app/src/...` |
+| Pricing | `Pricing` has base/per-km/per-kg/handling — **no return charge** | `models` Pricing |
+| Customer returns (reverse pickup) | Not present anywhere | — |
+
+**Conclusion:** the data model is mostly ready; the lifecycle, the rules, the
+endpoints, the console screens and the rider task are all missing.
+
+---
+
+## 2. Scope
+
+| Flow | Description | Phase |
+|---|---|---|
+| **A. RTO — Return to Origin** | Delivery fails → parcel goes back to the sender's pickup point (⚑ or a hub) | **Phase 1–3 (this plan)** |
+| **B. Customer return** | Receiver sends a delivered item back; rider picks up from receiver, returns to sender | Phase 5 (later) |
+| **C. Exchange** | Deliver new + collect old in one visit | Out of scope for now |
+
+---
+
+## 3. RTO lifecycle (proposed)
+
+```
+Collected_By_Miler / Out_for_Delivery
+        │  failed attempt (rider skip)  → Attemptcount++  (as today)
+        │
+        ├─ attempts ≥ N (⚑ default 3)  ─┐
+        ├─ ops "Initiate RTO" (console) ─┼─▶ RTO_Initiated
+        └─ (⚑ later) client request  ───┘      returnreason, returninitiatedat set
+                                                history: RTO_Initiated
+                                                rider next action: return_to_sender
+                         │
+          ┌──────────────┼───────────────────────────┐
+          ▼              ▼                           ▼
+  rider returns    ops "Re-attempt delivery"   ops "Mark returned"
+  to sender        (cancels the RTO)           (manual close, e.g. hub drop)
+          │              │                           │
+          ▼              ▼                           ▼
+ Returned_to_Sender  Out_for_Delivery       Returned_to_Sender
+ returndeliveredat   (attempts kept)        returndeliveredat
+```
+
+Rules:
+- RTO only from `Collected_By_Miler`, `Out_for_Delivery` (or `Inwarded_at_Hub` ⚑).
+- `Returned_to_Sender` is terminal.
+- Every transition writes `ConsignmentHistory` with actor + reason.
+- Opening RTO resolves the linked `Undeliverable` exception (if any) with
+  resolution "RTO initiated".
+- COD: a returned parcel collects nothing; `Codcollected` stays 0.
+
+---
+
+## 4. Backend — `doormile_backend`
+
+| # | Change | Detail |
+|---|---|---|
+| B1 | **Transition guard** | One function `canTransition(from, to)` used by every consignment status write; `PUT /admin/consignments/:id/status` refuses illegal moves (also fixes the any-status bug). |
+| B2 | `POST /admin/consignments/:id/rto` `{reason, note}` | Staff only. Sets `RTO_Initiated`, `returnreason`, `returninitiatedat`; history; resolves the Undeliverable exception; publishes NATS `consignment.rto_initiated`; notifies the rider. Idempotent. |
+| B3 | `POST /admin/consignments/:id/rto/cancel` `{note}` | Back to `Out_for_Delivery` (re-attempt). |
+| B4 | `POST /admin/consignments/:id/rto/complete` `{note}` | Ops closes it manually → `Returned_to_Sender`, `returndeliveredat`. |
+| B5 | `GET /admin/returns?status&from&to&tenantid&hubid&pageno` | List for the Returns page: tracking no, client, sender, reason, attempts, initiated/returned times, rider, age. Tenant-scoped like other admin lists. |
+| B6 | **Auto-RTO** in `MilerSkipDelivery` | When `Attemptcount ≥ RTO_AUTO_AFTER_ATTEMPTS` (env, ⚑ default 3; `0` = off) → call the same B2 logic instead of only opening an exception. |
+| B7 | **Rider task** | New next action `return_to_sender`; `nextActionForConsignment(RTO_Initiated)` returns it; rider queue includes the return stop (sender's pickup coords). New `POST /miler/consignments/:id/return-complete` `{lat, lon, photourl?, receivedby}` → `Returned_to_Sender`. **Behind flag `MILER_RTO_FLOW_ENABLED`** (default off) — the deployed rider app doesn't know `return_to_sender`, same pattern as `MILER_HUB_HANDOVER_ENABLED`. |
+| B8 | Booking/customer stage | For customer-app bookings, record the return in `cxstage`. ⚠ The customer app renders unknown stage keys as `booked`, so a new "returning" stage needs a client release — until then do **not** add a new key. |
+| B9 | Tests | Transition table, each endpoint's gates (staff/tenant/owner), auto-RTO at N attempts, flag off/on rider queue, idempotency. Postgres-gated tests for the SQL. |
+
+Schema: **none for Phase 1–3** (columns exist). ⚑ Return-to-hub needs one
+additive nullable column (`returnhubid`) — Phase 4.
+
+---
+
+## 5. Console — `krow_talent_app`
+
+| # | Change | Where |
+|---|---|---|
+| C1 | API + hooks: `initiateRto`, `cancelRto`, `completeRto`, `getReturns` + `useInitiateRto` … with invalidation of deliveries/returns keys | `src/api/doormile/endpoints.js`, `src/lib/doormileHooks.js` |
+| C2 | **Deliveries page actions**: on failed / out-for-delivery rows — *Initiate RTO* (reason picker: Receiver refused, Address not found, Customer unavailable, Attempts exhausted, Other + note), *Re-attempt*, *Mark returned*. Show attempt count | `src/pages/doormile/deliveries/Deliveries.jsx` |
+| C3 | **Status tabs**: add *RTO* and *Returned* tabs/counts (today `rto` collapses into `skipped`) | `queries.js` status map, `StatusBadge` |
+| C4 | **Returns page** `/doormile/returns`: tabs Initiated · In return · Returned; filters (date, client, hub, reason); age/SLA column; XLSX export; nav entry under Fleet Ops (staff) | new `src/pages/doormile/returns/Returns.jsx`, `App.jsx`, `AdminLayout.jsx` |
+| C5 | **Exceptions page**: an `Undeliverable` / `Receiver_Refused` exception gets a *Start RTO* button | `src/pages/doormile/exceptions/Exceptions.jsx` |
+| C6 | **Order / consignment timeline** shows attempts, RTO start, return | booking detail drawer |
+| C7 | **Reports**: return rate per client and per reason on Orders Summary | `src/pages/doormile/reports/` |
+| C8 | Client (tenant) logins: read-only view of their own returns (no actions) | Returns page + role check |
+| C9 | Tests: actions call the right endpoints, reason required, tabs/counts, Returns filters, client read-only | `tests/integration/` |
+
+---
+
+## 6. Rider app (Flutter — separate team)
+
+- Handle next action `return_to_sender`: show the return stop, navigate to the
+  sender, "Returned" button → `POST /miler/consignments/:id/return-complete`
+  with location (+ optional photo / receiver name).
+- Release before turning on `MILER_RTO_FLOW_ENABLED`. Until then, ops close
+  returns from the console (B4) and riders are told by push.
+
+---
+
+## 7. Phases & order
+
+| Phase | Content | Repos | Depends on |
+|---|---|---|---|
+| **1** | B1 guard, B2–B5 endpoints, B9 tests | backend | — |
+| **2** | C1–C5, C9 | console | Phase 1 deployed |
+| **3** | B6 auto-RTO, B7 rider task (flag off) + rider app release, then flag on | backend + rider app | Phase 1 |
+| **4** | Return-to-hub option (`returnhubid`), C6 timeline, C7 reports, C8 client view, return charges | all | decisions ⚑ |
+| **5** | Flow B — customer returns (reverse pickup booking linked by `Parentconsignmentid`) | all + customer app | product spec |
+
+Rough effort: Phase 1 ≈ 1.5–2 days · Phase 2 ≈ 2 days · Phase 3 ≈ 1 day backend
+(+ rider app team) · Phase 4/5 to estimate after decisions.
+
+---
+
+## 8. Risks
+
+- **Rider app compatibility** — a new next action unknown to the deployed app
+  must stay behind a flag (B7).
+- **Customer app stage keys** — unknown keys render as `booked` (B8).
+- **Ops discipline** — until the rider task ships, returns depend on ops
+  closing them in the console.
+- **Existing any-status endpoint** — B1 changes its behaviour: callers that
+  relied on setting arbitrary statuses will now get 400. The console only
+  sends Out_for_Delivery / Delivered / Cancelled, which stay allowed.
+- **Deploy** — no migration in Phase 1–3; Phase 4 adds one nullable column
+  (needs approval).
+
+---
+
+## 9. Open decisions ⚑
+
+1. **Flows first:** A (RTO) only, then B? *(proposed: yes)*
+2. **Return destination:** sender's pickup point, nearest hub, or per client?
+   *(proposed: sender for Phase 1; hub option in Phase 4)*
+3. **Who starts RTO:** auto after N attempts (N = ?), ops, client, or all?
+   *(proposed: ops + auto after 3; client later)*
+4. **Can RTO start from a hub** (`Inwarded_at_Hub`)? *(proposed: yes)*
+5. **Return charges:** billed to the client? Same rate, fixed fee, or free?
+   *(proposed: decide before Phase 4; Phase 1–3 record only)*
+6. **Rider app:** is the Flutter team available for B7, and when?
+7. **Client visibility:** may tenants see their own returns? *(proposed: read-only)*
+
+---
+
+## 10. Implementation status (2026-10-05)
+
+Built with the proposed defaults: return to the **sender**, started by **ops or
+automatically after 3 failed attempts**, rider task **behind a flag (off)**.
+Nothing is committed or pushed.
+
+### Backend — `doormile_backend`
+| Item | Where |
+|---|---|
+| RTO core: `startRTO`, `completeRTO`, cancel → previous status (recorded as `[from:<status>]` in the history remark), exception auto-resolve, rider push | `controllers/consignmentReturn.go` |
+| `POST /admin/consignments/:id/rto` · `/rto/cancel` · `/rto/complete` (Doormile staff only) | `routes/routes.go` |
+| `GET /admin/returns?status=initiated\|returned\|all&from&to&pageno&pagesize` (tenant-scoped; client logins read their own) | same |
+| `POST /miler/consignments/:id/return-complete` — 403 `RTO_FLOW_DISABLED` unless `MILER_RTO_FLOW_ENABLED=true` | same |
+| Status guard on `PUT /admin/consignments/:id/status` (unknown status, leaving Delivered/Cancelled/Returned, RTO statuses → 400) | `controllers/adminController.go` |
+| Auto-RTO after `RTO_AUTO_AFTER_ATTEMPTS` (default 3, `0` = off) | `MilerSkipDelivery` in `controllers/milerAppController.go` |
+| `MilerSkipDelivery` ownership now multi-destination safe (`milerConsignmentForRider`) — fixes skip on orders 2..N | same |
+| Next action `return_to_sender` (flagged); rider consignment read adds `returning`, `can_return`, `return_reason`, `return_to` | `constants`, `logisticsHandoverController.go`, `milerAppController.go` |
+| Tests: 7 unit + 5 route-gate tests | `controllers/consignmentReturn_test.go`, `routes/routes_rto_test.go` |
+
+New env vars: `RTO_AUTO_AFTER_ATTEMPTS` (default 3), `MILER_RTO_FLOW_ENABLED`
+(default off). **No schema change.** No customer-app stage key added (B8).
+
+### Console — `krow_talent_app`
+| Item | Where |
+|---|---|
+| API + hooks: `initiateRto`, `cancelRto`, `completeRto`, `getReturns`, `RTO_REASONS`; `useReturns`, `useInitiateRto`, `useCancelRto`, `useCompleteRto` | `src/api/doormile/endpoints.js`, `src/lib/doormileHooks.js` |
+| Status mapping: `RTO_Initiated → rto`, `Returned_to_Sender → returned` (legacy `rto`/`returned` keys unchanged) + counts | `src/api/doormile/queries.js` |
+| Shared dialogs: Start return (reason + note, pre-select), Re-attempt / Mark returned | `src/components/doormile/RtoDialogs.jsx` |
+| Deliveries: **In return** + **Returned** tabs; Return to sender / Re-attempt / Mark returned (staff only); status-update and cancel hidden on returning rows | `src/pages/doormile/deliveries/Deliveries.jsx` |
+| **Returns page** `/doormile/returns` (tabs, date range, search, export, pagination; read-only for clients); nav under Fleet Ops | `src/pages/doormile/returns/Returns.jsx`, `App.jsx`, `AdminLayout.jsx` |
+| Exceptions: **Start return** on open Undeliverable / Receiver_Refused | `src/pages/doormile/exceptions/Exceptions.jsx` |
+| Tests: 13 new; deliveries counts test updated | `tests/integration/returns.test.jsx`, `tests/api/deliveries.test.js` |
+
+### Verified / not verified
+- ✅ `go build`, `go vet`, all 17 backend packages; console suites pass except
+  the 13 Agent Studio tests already failing since the 2026-09-30 redesign;
+  console production build OK.
+- ✅ **End-to-end on a real database (2026-10-05).** This ran on a throwaway
+  local stack: PostgreSQL 17.10 on 127.0.0.1, the backend built from this tree
+  with `MILER_RTO_FLOW_ENABLED=true`, and the console on a local port pointed
+  at it. Production was not touched.
+  - Browser, as staff:
+    - Return to sender (reason + note), then Re-attempt (back to
+      Out_for_Delivery), then Start again, then Mark returned. Result:
+      `Returned_to_Sender`, the rider's assignment `Completed`, and the full
+      status history.
+    - Exceptions → Start return pre-selects the reason, and the exception
+      leaves the list.
+    - The Returns page lists rows with correct India times.
+  - API:
+    - 3 skips start RTO automatically and resolve the Undeliverable exception.
+    - The rider read gives `next_action=return_to_sender` with `return_to`
+      set to the sender's coordinates.
+    - Rider return-complete closes the assignment.
+    - Every guard answers 400 / `INVALID_STATE` as designed.
+  - Client login: no return buttons on Deliveries; Returns is read-only;
+    `rto/cancel` and `rto/complete` return 403.
+- Bugs the browser test found, all fixed:
+  1. Notify rider showed on returned and delivered rows. It is now hidden.
+  2. Exceptions didn't refresh after Start return. `RETURN_KEYS` now includes
+     the exceptions key.
+  3. The dialog subtitle from Exceptions read "Order consignment #…". It now
+     uses a `label` prop.
+- Backend review (2026-10-05), fixed:
+  1. **Race conditions.** Start, complete and cancel used to read the parcel
+     and then save the whole row. A rider delivering at the same moment, or
+     a double click, could be overwritten. Each one is now a compare-and-set
+     (`moveConsignment`: update only if the status is still the one read).
+     A lost race either becomes the same no-op as a repeat, or returns
+     "changed, refresh".
+  2. **Re-attempt clears the return.** `returnreason` and
+     `returninitiatedat` are cleared, so a parcel delivered afterwards doesn't
+     carry a stale reason. The history keeps the reason.
+  3. **"OTHER" check.** A reason sent as "OTHER" or "Other " skipped the
+     note-required check. The check now uses the normalised reason
+     (`rtoReasonText`).
+  4. **Note length.** The 500 limit is counted in characters, not bytes.
+     Before, 200 Tamil characters were refused even though the console
+     allows 500.
+  5. **Wrong rider notified.** Starting a return on a parcel already handed
+     over at a base pushed "do not attempt delivery" to its old pickup rider.
+     Now only a rider holding the parcel is notified
+     (`riderHoldsParcel`: Created, Collected_By_Miler, Out_for_Delivery).
+- Impact on existing users (checked 2026-10-05). Decide these before
+  deploying:
+  - **Rider app (deployed build):** it shows a parcel from the *booking*
+    status, which a return doesn't change. So a parcel in return still looks
+    deliverable; tapping deliver or skip returns a 400. The rider also can't go
+    off duty until ops press **Mark returned**, because the assignment stays
+    open. Automatic return is **on by default** (3 attempts), so this starts
+    on deploy. Either set `RTO_AUTO_AFTER_ATTEMPTS=0` until the rider app
+    supports returns, or make sure ops close returns daily.
+  - **Console "Update status":** Delivered and Cancelled parcels can no longer
+    be changed (400). Before, ops could undo a wrong "Delivered" or reopen a
+    cancelled parcel. That is now refused by design.
+  - **Customer app:** a returned parcel keeps showing "Out for delivery"
+    (there is no customer stage for returns).
+  - **Rider pay:** closing a return marks the assignment Completed with 0 km
+    and 0 charges. The return trip isn't paid.
+  - **Not affected:** rider delivered counts and reports (Delivered only), the
+    hub console (a parcel in return simply drops out of its status-based
+    lists), the B2C booking flow, the existing status values, and the
+    database schema (no change).
+- Tests added: `TestRiderHoldsParcel`, `TestRTOReasonText`, plus `controllers/consignmentReturn_pg_test.go`
+  (5 tests on real Postgres: lifecycle, refusals, no overwrite of a
+  concurrent delivery, 10 simultaneous starts → one return, cancel). These
+  skip unless `REGISTRY_TEST_DSN` is set, and passed on PostgreSQL 17.
+- Timestamps stay `time.Now()`, not `utils.DBNow()`. `AutoMigrate` creates
+  `timestamptz` columns, where `DBNow()` would store a time 5 h 30 m ahead.
+  `time.Now()` is correct for both column types because the Dockerfile sets
+  `TZ=Asia/Kolkata`.
+- Seen, not fixed (pre-existing, outside RTO): the Exceptions page shows
+  "Raised" times about 6 h ahead of IST.
+- Not built: Phase 4 (return-to-hub, timeline, reports, charges), Phase 5
+  (customer returns), the rider-app UI (Flutter team).