Research: Flutter Integration Testing Tools Evaluation

Issue: #533 | Focus: Maestro vs. Marionette MCP Status: Completed | Target Apps: KROW Client App & KROW Staff App

1. Executive Summary & Recommendation

After performing a hands-on spike implementing core authentication flows (Login and Signup) for both the KROW Client and Staff applications, we have reached a definitive conclusion regarding the project's testing infrastructure.

🏆 Final Recommendation: Maestro

Maestro is the recommended tool for all production-level integration and E2E testing.

While Marionette MCP provides an impressive AI-driven interaction layer that is highly valuable for local development and exploratory debugging, it is not yet suitable for a stable, deterministic CI/CD pipeline. For KROW Workforce, where reliability and repeatable validation of release builds are paramount, Maestro is the superior architectural choice.

2. Hands-on Spike Findings

Challenge: New signups require dismissing native OS permission dialogs (Location, Notifications) and handling asynchronous OTP (One-Time Password) entry.
Maestro Result: Pass. Successfully dismissed iOS/Android native dialogs and used inputText to simulate OTP entry. The "auto-wait" feature handled the delay between clicking "Verify" and the Dashboard appearing perfectly.
Marionette MCP Result: Fail (Partial). Could not tap the native "Allow" button on OS dialogs, stalling the flow. Required manual intervention to bypass permissions.

Challenge: Reliably targeting TextFields and asserting Successful Login states across different themes/localizations.
Maestro Result: Pass. Used Semantic Identifiers (identifier: 'login_email_field') which remained stable even when UI labels changed. Test execution took ~12 seconds.
Marionette MCP Result: Pass (Inconsistent). The AI successfully identified fields by visible text, but execution time exceeded 60 seconds due to multiple LLM reasoning cycles.

3. Comparative Matrix

Evaluation Criteria	Maestro	Marionette MCP
Deterministic Consistency	10/10 (Tests run the same way every time)	4/10 (AI behavior can vary per run)
Execution Speed	High (Direct binary communication)	Low (Bottlenecked by LLM API latency)
Native Modal Support	Full (Handles OS permissions/dialogs)	None (Limited to the Flutter Widget tree)
CI/CD Readiness	Production Ready (Lightweight CLI)	Experimental (High cost/overhead)
Release Build Testing	Yes (Interacts via Accessibility layer)	No (Requires VM Service / Debug mode)
Learning Curve	Low (YAML is human-readable)	Medium (Requires prompt engineering)

4. Deep Dive: Why Maestro Wins for KROW

1. Handling the "Native Wall"

KROW apps rely heavily on native features (Camera for document uploads, Location for hub check-ins). Maestro communicates with the mobile OS directly, allowing it to "click" outside the Flutter canvas. Marionette lives entirely inside the Dart VM; if a native permission popup appears, the test effectively dies.

2. Maintenance & Non-Mobile Engineering Support

KROW’s growth requires that non-mobile engineers and QA teams contribute to testing.

Maestro uses declarative YAML. A search test looks like: tapOn: "Search". It is readable by anyone.
Marionette requires managing an MCP server and writing precise AI prompts, which is harder to standardize across a large team.

3. CI/CD Pipeline Efficiency

We need our GitHub Actions to run fast. Maestro tests are lightweight and can run in parallel on cloud emulators. Marionette requires an LLM call for every single step, which would balloon our CI costs and increase PR wait times significantly.

5. Implementation & Migration Roadmap

To transition to the recommended Maestro-based testing suite, we will execute the following:

Phase 1: Design System Hardening (Current Sprint)

Update the krow_design_system package to ensure all UiButton, UiTextField, and UiCard components include a Semantics wrapper with an identifier property.
Example: Semantics(identifier: 'primary_action_button', child: child)

Phase 2: Core Flow Implementation

Create a /maestro directory in each app's root.
Implement "Golden Flows": login.yaml, signup.yaml, post_job.yaml, and check_in.yaml.

Phase 3: CI/CD Integration

Configure GitHub Actions to trigger maestro test on every PR merged into dev.
Establish "Release Build Verification" where Maestro runs against the final .apk/.ipa before staging deployment.

Phase 4: Clean Up

Remove marionette_flutter from pubspec.yaml to keep our production binary size optimal and security surface area low.

6. Final Verdict

Maestro is the engine for our automation, while Marionette MCP remains a powerful tool for developers to use locally for code exploration and rapid UI debugging. We will move forward with Maestro for all regression and release-blocking test suites.

5.2 KiB Raw Blame History Unescape Escape