Drain bills to the back office automatically, over MQTT or HTTP

Turns the orders table into a queue that empties itself. Bills were only
uploaded when a cashier pressed Sync at end of day; a till that was never
pressed held a day's takings indefinitely.

Drain engine (lib/data/sync/sync_engine.dart)
- Triggers on sale committed, network regained, 5-minute poll, head-office
  request, and the manual button.
- Single flight: a busy till firing a trigger per sale would otherwise have
  several passes reading the same pending rows and send every bill twice.
  A trigger arriving mid-drain is queued and replayed, so nothing is dropped.
- Exponential backoff with +/-20% jitter to a 5-minute ceiling. The jitter
  matters: a store's terminals all fail at the same instant when the line
  drops, and would retry in lockstep without it.
- Halts rather than loops on a failure retrying cannot fix (bad credential,
  refused batch). Pressing Sync clears the halt.

Transports (lib/data/remote/)
- OrderTransport interface; MQTT, HTTP and simulated implementations. The
  repository does not know which is in use.
- MQTT: QoS 1 uplink, application-level ACK correlated by batch_id on a return
  topic, retained Last Will for terminal-offline detection, downlink for
  catalogue pushes and remote sync requests.
- A broker PUBACK is never treated as acceptance. It means the broker holds
  the bytes, not that the ledger took the sale. Only ids the back office names
  are marked synced; silence leaves a bill pending.
- HTTP carries a stable idempotency key across retries of the same bills.

Retention
- Accepted bills are kept 7 days instead of deleted, so a batch the back
  office later loses can be re-sent in full. Purged after that; archived
  totals stay forever.
- forBusinessDate now reads pending rows only. A retained bill exists in both
  the orders table and day_archive, and summing both would overstate the day.

Fixes found while building this
- SyncEngine._refreshPending wrote state.copyWith(pending: await ...). Dart
  evaluates the receiver before the awaited argument, so a connectivity drop
  during the wait was silently overwritten by the stale snapshot. Caught by
  the first run of the new engine tests.
- PrinterSettingsController wrote state after four awaits with no mounted
  check, throwing "used after dispose" when Settings was left mid-load. This
  was pre-existing and reached the cashier as a red screen.

Also
- Header pill now reports real sync state: LIVE / n QUEUED / SYNCING /
  SYNC HALTED, with an explanation of where the bills are.
- Settings shows the route, last upload, next retry and retention window.
- docs/sync-contract.md states what the back office must implement, including
  the idempotency requirement that at-least-once delivery makes mandatory.

Tests: 90 -> 129 passing. New coverage for backoff shape and jitter band,
single flight, halting, ACK correlation and partial acceptance, at-least-once
duplicate handling, retention and purge, and no double-counting after a sync.
Suite run six times clean.

Not addressed: bills already synced by an older build went up overstated and
still need server-side reconciliation. Broker credentials have no Settings
editor yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Suriya
2026-08-01 10:57:29 +05:30
parent af3933092f
commit 0a49323858
29 changed files with 2855 additions and 112 deletions

View File

@@ -0,0 +1,154 @@
import 'dart:async';
import 'dart:convert';
import 'package:http/http.dart' as http;
import '../../core/config/sync_config.dart';
import 'order_transport.dart';
/// Ships bills over plain HTTP.
///
/// No downlink and no persistent connection — but a failure is a status code,
/// a request is reproducible with curl, and there is no broker to run. Worth
/// having as the route to bring up first, and as the fallback when a broker is
/// unreachable but the internet is not.
///
/// The endpoint must answer with the ids it committed:
///
/// ```json
/// { "accepted": ["<order-id>", ...], "rejected": { "<order-id>": "reason" } }
/// ```
///
/// A bare `200 OK` is not treated as success for any bill. Silence about which
/// rows landed is not the same as landing them, and guessing here would retire
/// a day's takings on an empty response.
class HttpOrderTransport implements OrderTransport {
HttpOrderTransport({required this.config, http.Client? client})
: _client = client ?? http.Client();
final SyncConfig config;
final http.Client _client;
final _connection = StreamController<bool>.broadcast();
bool _reachable = true;
@override
String get label => 'HTTP ${config.httpBaseUrl}';
@override
bool get isConnected => _reachable;
/// Nothing to receive: HTTP is a route out, not in.
@override
Stream<DownlinkMessage> get downlink => const Stream.empty();
@override
Stream<bool> get connectionState => _connection.stream;
@override
Future<void> connect() async {}
@override
Future<PushReceipt> pushOrders(List<Map<String, Object?>> orders) async {
if (orders.isEmpty) return const PushReceipt(accepted: []);
if (config.httpBaseUrl.isEmpty) {
throw const TransportException(
'No back-office URL configured for this terminal.',
retryable: false,
);
}
final uri = Uri.parse('${config.httpBaseUrl}/orders');
http.Response response;
try {
response = await _client
.post(
uri,
headers: {
'content-type': 'application/json',
if (config.apiKey != null)
'authorization': 'Bearer ${config.apiKey}',
// Lets the endpoint collapse a retried batch server-side rather
// than relying on every order id being checked individually.
'idempotency-key': _batchKey(orders),
},
body: jsonEncode({
'schema': 1,
'store_id': config.storeId,
'terminal_id': config.terminalId,
'sent_at': DateTime.now().toIso8601String(),
'orders': orders,
}),
)
.timeout(config.ackTimeout);
} on Exception catch (e) {
_setReachable(false);
throw TransportException('Upload failed: $e');
}
if (response.statusCode == 401 || response.statusCode == 403) {
_setReachable(false);
throw TransportException(
'The back office rejected this terminal\'s credentials '
'(${response.statusCode}). Bills are safe locally, but sync will keep '
'failing until the terminal is re-authorised.',
retryable: false,
);
}
if (response.statusCode >= 300) {
_setReachable(false);
throw TransportException(
'Back office returned ${response.statusCode}: '
'${_trim(response.body)}',
);
}
_setReachable(true);
Map<String, Object?> body;
try {
body = jsonDecode(response.body) as Map<String, Object?>;
} on Exception {
throw TransportException(
'Back office answered 200 with a body this terminal could not read, '
'so no bill was marked synced: ${_trim(response.body)}',
);
}
final accepted = (body['accepted'] as List<Object?>? ?? const [])
.whereType<String>()
.toList();
final rejected = <String, String>{};
final raw = body['rejected'];
if (raw is Map) {
raw.forEach((k, v) => rejected['$k'] = '$v');
}
return PushReceipt(accepted: accepted, rejected: rejected);
}
/// Stable for a given set of bills, so a retry after a timeout carries the
/// same key as the attempt that may already have landed.
String _batchKey(List<Map<String, Object?>> orders) =>
orders.map((o) => o['id']).join('|').hashCode.toRadixString(16);
void _setReachable(bool value) {
if (_reachable == value) return;
_reachable = value;
_connection.add(value);
}
static String _trim(String body) =>
body.length <= 200 ? body : '${body.substring(0, 200)}';
@override
Future<void> dispose() async {
_client.close();
await _connection.close();
}
}

View File

@@ -0,0 +1,332 @@
import 'dart:async';
import 'dart:convert';
import 'package:flutter/foundation.dart';
import 'package:mqtt_client/mqtt_client.dart';
import 'package:mqtt_client/mqtt_server_client.dart';
import 'package:uuid/uuid.dart';
import '../../core/config/sync_config.dart';
import 'order_transport.dart';
/// Ships bills over MQTT and listens for what head office pushes back.
///
/// ### Why the broker's acknowledgement is not enough
///
/// QoS 1 gives a PUBACK from the *broker*, meaning "I hold these bytes". It
/// says nothing about whether the back office parsed the batch, or whether the
/// ledger accepted it. Marking bills synced on PUBACK would retire a day's
/// takings on the word of a message queue.
///
/// So every publish carries a `batch_id` and this waits for the back office to
/// answer on [SyncConfig.ackTopic] naming the ids it actually committed. No
/// answer means no sync, and the bills are sent again.
///
/// ### Duplicates are expected
///
/// QoS 1 is at-least-once, and a batch whose ack is lost will be re-sent in
/// full. The back office must key on `order.id` and upsert. Every id is a UUID
/// minted at the till, so this costs the server one unique index.
class MqttOrderTransport implements OrderTransport {
MqttOrderTransport({
required this.config,
MqttClient Function(SyncConfig)? clientFactory,
}) : _clientFactory = clientFactory ?? _defaultClient;
final SyncConfig config;
final MqttClient Function(SyncConfig) _clientFactory;
static const _uuid = Uuid();
MqttClient? _client;
/// Batches published but not yet answered, keyed by `batch_id`.
final _awaitingAck = <String, Completer<PushReceipt>>{};
final _downlink = StreamController<DownlinkMessage>.broadcast();
final _connection = StreamController<bool>.broadcast();
/// Guards against two drains racing to open the same connection.
Future<void>? _connecting;
StreamSubscription<List<MqttReceivedMessage<MqttMessage>>>? _updates;
static MqttClient _defaultClient(SyncConfig config) {
final client = MqttServerClient.withPort(
config.brokerHost,
config.clientId,
config.brokerPort,
);
client.secure = config.useTls;
// Well under the shortest NAT timeout a shop router is likely to impose;
// a connection that dies silently is worse than one that pings.
client.keepAlivePeriod = 20;
client.autoReconnect = true;
client.resubscribeOnAutoReconnect = true;
client.logging(on: false);
return client;
}
@override
String get label => 'MQTT ${config.brokerHost}:${config.brokerPort}';
@override
bool get isConnected =>
_client?.connectionStatus?.state == MqttConnectionState.connected;
@override
Stream<DownlinkMessage> get downlink => _downlink.stream;
@override
Stream<bool> get connectionState => _connection.stream;
// ------------------------------------------------------------- Connection
@override
Future<void> connect() {
if (isConnected) return Future.value();
return _connecting ??= _doConnect().whenComplete(() => _connecting = null);
}
Future<void> _doConnect() async {
if (config.brokerHost.isEmpty) {
throw const TransportException(
'No MQTT broker configured for this terminal.',
retryable: false,
);
}
final client = _client ??= _clientFactory(config);
client.onDisconnected = () {
_connection.add(false);
// Nobody is coming to answer these. Failing them now returns the bills
// to pending immediately instead of holding the drain for the full ack
// timeout on a connection that is already gone.
_failAllAwaiting('Connection to the broker dropped.');
};
client.onConnected = () => _connection.add(true);
// Retained, so head office sees this terminal's last known state even if
// its dashboard connects hours later. The broker publishes it on our
// behalf if the till loses power mid-shift — which is the only way to tell
// "closed for the night" from "unplugged".
client.connectionMessage = MqttConnectMessage()
.withClientIdentifier(config.clientId)
.withWillTopic(config.statusTopic)
.withWillMessage(jsonEncode({'state': 'offline'}))
.withWillQos(MqttQos.atLeastOnce)
.withWillRetain();
try {
await client.connect(config.username, config.password);
} on Exception catch (e) {
client.disconnect();
throw TransportException('Could not reach the broker: $e');
}
if (client.connectionStatus?.state != MqttConnectionState.connected) {
final status = client.connectionStatus;
client.disconnect();
throw TransportException(
'Broker refused the connection: '
'${status?.returnCode?.name ?? 'unknown'}',
// Bad credentials or an unauthorised client id will be refused just as
// firmly on the next attempt; backing off forever would only hide it.
retryable: status?.returnCode != MqttConnectReturnCode.notAuthorized,
);
}
client
..subscribe(config.ackTopic, MqttQos.atLeastOnce)
..subscribe(config.catalogueTopic, MqttQos.atLeastOnce)
..subscribe(config.commandTopic, MqttQos.atLeastOnce);
await _updates?.cancel();
_updates = client.updates?.listen(_onUpdates);
_publish(
config.statusTopic,
jsonEncode({
'state': 'online',
'terminal_id': config.terminalId,
'at': DateTime.now().toIso8601String(),
}),
retain: true,
);
_connection.add(true);
}
void _onUpdates(List<MqttReceivedMessage<MqttMessage>> events) {
for (final event in events) {
final message = event.payload;
if (message is! MqttPublishMessage) continue;
handleInbound(
event.topic,
MqttPublishPayload.bytesToStringAsString(message.payload.message),
);
}
}
// ----------------------------------------------------------------- Uplink
@override
Future<PushReceipt> pushOrders(List<Map<String, Object?>> orders) async {
if (orders.isEmpty) return const PushReceipt(accepted: []);
await connect();
final batchId = _uuid.v4();
final completer = Completer<PushReceipt>();
_awaitingAck[batchId] = completer;
try {
_publish(
config.orderTopic,
jsonEncode({
'schema': 1,
'batch_id': batchId,
'store_id': config.storeId,
'terminal_id': config.terminalId,
'sent_at': DateTime.now().toIso8601String(),
'orders': orders,
}),
);
return await completer.future.timeout(
config.ackTimeout,
onTimeout: () => throw TransportException(
'The back office did not confirm the batch within '
'${config.ackTimeout.inSeconds}s. The bills are still on this '
'terminal and will be sent again.',
),
);
} finally {
_awaitingAck.remove(batchId);
}
}
void _publish(String topic, String payload, {bool retain = false}) {
final builder = MqttClientPayloadBuilder()..addString(payload);
try {
_client!.publishMessage(
topic,
MqttQos.atLeastOnce,
builder.payload!,
retain: retain,
);
} on Exception catch (e) {
throw TransportException('Publish to $topic failed: $e');
}
}
// --------------------------------------------------------------- Downlink
/// Registers a batch as awaiting its ack, without publishing one.
///
/// Lets a test drive the correlation rules — which is where the logic that
/// decides whether a bill counts as banked actually lives — without standing
/// up a broker.
@visibleForTesting
Future<PushReceipt> awaitAck(String batchId) {
final completer = Completer<PushReceipt>();
_awaitingAck[batchId] = completer;
return completer.future;
}
/// Routes one inbound message. Separated from the client so the correlation
/// and parsing rules can be tested without a broker.
@visibleForTesting
void handleInbound(String topic, String payload) {
Map<String, Object?> body;
try {
body = jsonDecode(payload) as Map<String, Object?>;
} on FormatException {
// A malformed message must not take down the connection: the next one
// may be a perfectly good ack releasing a day's bills.
debugPrint('Discarded unparseable MQTT message on $topic');
return;
}
if (topic == config.ackTopic) {
_resolveAck(body);
return;
}
if (topic == config.catalogueTopic) {
_downlink.add(
DownlinkMessage(kind: DownlinkKind.catalogueChanged, payload: body),
);
return;
}
if (topic == config.commandTopic) {
final command = body['command'] as String?;
_downlink.add(
DownlinkMessage(
kind: command == 'sync'
? DownlinkKind.syncRequested
: DownlinkKind.unknown,
payload: body,
),
);
}
}
void _resolveAck(Map<String, Object?> body) {
final batchId = body['batch_id'] as String?;
if (batchId == null) return;
// An ack for a batch we are no longer waiting on — we timed out, or the
// terminal restarted. Harmless: those bills are still pending and will go
// up again, and the back office keys on order id.
final completer = _awaitingAck.remove(batchId);
if (completer == null || completer.isCompleted) return;
final accepted = (body['accepted'] as List<Object?>? ?? const [])
.whereType<String>()
.toList();
final rejected = <String, String>{};
final raw = body['rejected'];
if (raw is Map) {
raw.forEach((k, v) => rejected['$k'] = '$v');
} else if (raw is List) {
// Tolerates a back office that sends bare ids with no reason.
for (final id in raw.whereType<String>()) {
rejected[id] = 'Rejected by the back office';
}
}
completer.complete(PushReceipt(accepted: accepted, rejected: rejected));
}
void _failAllAwaiting(String reason) {
final waiting = List.of(_awaitingAck.values);
_awaitingAck.clear();
for (final c in waiting) {
if (!c.isCompleted) c.completeError(TransportException(reason));
}
}
@override
Future<void> dispose() async {
_failAllAwaiting('The terminal is shutting down.');
await _updates?.cancel();
if (isConnected) {
// A clean goodbye, so head office does not see a till it thinks crashed.
try {
_publish(
config.statusTopic,
jsonEncode({'state': 'offline', 'clean': true}),
retain: true,
);
} on TransportException {
// Already going down; nothing useful to do about it.
}
}
_client?.disconnect();
await _downlink.close();
await _connection.close();
}
}

View File

@@ -0,0 +1,99 @@
import 'dart:async';
/// Raised when a batch could not be handed to the back office.
///
/// Distinct from *rejection*: a transport failure means nobody knows whether
/// the bills arrived, so every row stays pending and is tried again. A
/// rejection means the back office looked at a bill and refused it, which
/// retrying will not fix.
class TransportException implements Exception {
const TransportException(this.message, {this.retryable = true});
final String message;
/// Whether trying again could plausibly succeed. A dropped connection is
/// retryable; a rejected certificate or a bad credential is not, and the
/// engine should stop rather than hammer the broker.
final bool retryable;
@override
String toString() => message;
}
/// What the back office said about one batch.
///
/// [accepted] is the contract that matters: only ids named here are marked
/// synced. Anything absent stays pending, whatever the transport reported at
/// its own layer.
class PushReceipt {
const PushReceipt({required this.accepted, this.rejected = const {}});
final List<String> accepted;
/// Order id → why the back office refused it. Retrying these unchanged will
/// fail again, so the engine surfaces them instead of looping.
final Map<String, String> rejected;
bool get isEmpty => accepted.isEmpty && rejected.isEmpty;
}
/// Something the cloud pushed down to this terminal.
///
/// Only a transport with a live connection can deliver these; request/response
/// transports expose an empty stream.
class DownlinkMessage {
const DownlinkMessage({required this.kind, this.payload = const {}});
final DownlinkKind kind;
final Map<String, Object?> payload;
}
enum DownlinkKind {
/// The catalogue changed at head office — re-import rather than wait for
/// tomorrow morning's pull.
catalogueChanged,
/// Head office is asking this terminal to upload now.
syncRequested,
/// Anything this build does not recognise. Kept rather than dropped so a
/// newer server talking to an older terminal is visible in the events log
/// instead of silently ignored.
unknown,
}
/// How completed bills leave the terminal.
///
/// The drain engine owns *when* to send and what to do when sending fails;
/// this owns only the wire. Swapping HTTP for MQTT is a change of
/// implementation here and nothing else.
abstract class OrderTransport {
/// Shown in the events log so a cashier reporting a problem can say which
/// route the terminal was using.
String get label;
/// Opens the connection. Safe to call when already open.
///
/// A request/response transport has nothing to open and returns at once.
Future<void> connect();
/// Hands a batch over and reports what the back office committed.
///
/// Implementations must not report an id as accepted until the *application*
/// has confirmed it. A broker acknowledging receipt of the bytes is not the
/// back office confirming the sale.
///
/// Throws [TransportException] when the outcome is unknown.
Future<PushReceipt> pushOrders(List<Map<String, Object?>> orders);
/// Cloud-initiated messages. Empty for transports that cannot receive.
Stream<DownlinkMessage> get downlink;
/// Whether the route is currently usable. Drives the header's live pill and
/// wakes the drain engine when it flips to true.
Stream<bool> get connectionState;
bool get isConnected;
Future<void> dispose();
}

View File

@@ -0,0 +1,62 @@
import 'dart:async';
import 'order_transport.dart';
/// Stands in for the back office when no broker or endpoint is configured.
///
/// Keeps the terminal demonstrable on a bare laptop: bills queue, drain, and
/// respect the Settings offline switch exactly as they would against a real
/// server, so the drain engine can be exercised without one.
class SimulatedOrderTransport implements OrderTransport {
SimulatedOrderTransport({required this.isOffline});
/// Read on every call rather than copied, so the Settings switch takes effect
/// immediately instead of at the next restart.
final bool Function() isOffline;
final _downlink = StreamController<DownlinkMessage>.broadcast();
final _connection = StreamController<bool>.broadcast();
@override
String get label => 'Simulated';
@override
bool get isConnected => !isOffline();
@override
Stream<DownlinkMessage> get downlink => _downlink.stream;
@override
Stream<bool> get connectionState => _connection.stream;
@override
Future<void> connect() async {}
@override
Future<PushReceipt> pushOrders(List<Map<String, Object?>> orders) async {
await Future<void>.delayed(
Duration(milliseconds: 400 + orders.length * 60),
);
if (isOffline()) {
throw const TransportException(
'Simulate offline is ON in Settings, so the upload was failed on '
'purpose. Every bill is still stored on this terminal.',
);
}
return PushReceipt(
accepted: orders.map((o) => o['id']! as String).toList(),
);
}
/// Lets the events screen prove the downlink path end to end without a
/// broker.
void emit(DownlinkMessage message) => _downlink.add(message);
@override
Future<void> dispose() async {
await _downlink.close();
await _connection.close();
}
}