Mobile CI/CD Testing

Current state and future vision for iOS & Android test automation

CURRENT STATE

Test Coverage Today

🍎 iOS

  • ✓ Unit Tests (Fastlane + XCTest)
  • ✓ Snapshot Tests
  • ✓ UI Tests (Manual trigger)
  • ✓ ZHL UI Tests
  • ✓ Draco UI Tests
  • ✓ Listing Search UI Tests
  • ✓ SonarQube Analysis

🤖 Android

  • ✓ Unit Tests (Gradle)
  • ✓ Snapshot Tests
  • ✓ Lint Checks
  • ✓ Emulator.wtf UI (Nightly)
  • ✓ Manual MR UI Tests
  • ✓ Code Coverage (Kover)
APPLES TO APPLES • 30-DAY COMPARISON

iOS vs Android: Line-by-Line

Metric 🍎 iOS 🤖 Android
Total Pipelines (30 days) 1,001 351
Overall Success Rate 48.5% 79.7%
Overall Failure Rate 51.4% 20.2%
Total Failures 491 70
Main Branch Success 26.2% 78.7%
Main Branch Failures 245 / 332 25 / 117
Release Branch Success 76.5% 80.0%
Primary Failure Source UI Tests Emulator.wtf
Top Failing Job test:ui:draco ui-tests:nightly
Coverage Tool llvm-cov Kover
Coverage Integration ✓ SonarQube ✓ SonarQube + Badge
Coverage Job In test:unit Dedicated job
KEY INSIGHTS

What We're Doing Well

⚡

Fast Feedback

Unit tests run on every MR, providing immediate feedback to developers

📊

Coverage Tracking

Comprehensive coverage reports integrated with SonarQube for quality metrics

🔄

Parallel Execution

Tests run independently with no dependencies, maximizing pipeline speed

CHALLENGES

Areas for Improvement

🎯

Manual UI Tests

iOS UI tests are manual-only on MRs, reducing confidence before merge

⏰

Limited Android UI

Emulator.wtf runs nightly only, catching issues after they're merged

🔍

Test Visibility

No unified dashboard for test results across both platforms

📈

Flakiness Tracking

No systematic approach to identifying and fixing flaky tests

DATA ANALYSIS • LAST 30 DAYS • 1,352 PIPELINES

Pipeline Success Rates

🍎 iOS

48.5%
Success Rate
1,001 TOTAL PIPELINES:
✓ 464 Success
✗ 491 Failed
+ 46 other (canceled/manual/running)

🤖 Android

79.7%
Success Rate
351 TOTAL PIPELINES:
✓ 276 Success
✗ 70 Failed
+ 5 other (running/skipped)
🔴 CATASTROPHIC: MAIN BRANCH CRISIS

Main Branch Instability

🍎

iOS Main Branch

73.7%
Failure Rate

245 failures out of 332 main pipelines

3 out of 4 main builds fail

🤖

Android Main Branch

21.3%
Failure Rate

25 failures out of 117 main pipelines

Better but still concerning

Target: <5% main branch failure rate · Currently: 3.5x-14.7x over target
FAILURE PATTERNS • 491 iOS FAILURES • 70 ANDROID FAILURES

Where Pipelines Fail

🍎 iOS Top Failures

  • ✗ Main branch: 245 failures (50% of all)
  • ✗ fix-qa-1: 55 failures
  • ✗ ant/OptimizePipeline: 37 failures
  • ✗ fix-qa: 22 failures
  • ✗ Release branches: 4 failures (23.5%)
Root Cause: UI tests (test:ui:draco, test:ui:zhl)

🤖 Android Top Failures

  • ✗ Main branch: 25 failures (36% of all)
  • ✗ MR !1682: 9 failures (highly flaky)
  • ✗ MR !1662: 4 failures
  • ✗ Various MRs: 2-3 failures each
  • ✗ Release branches: 1 failure (20%)
Root Cause: Emulator.wtf nightly UI tests
VISION

Future Testing Strategy

Today

  • • UI tests manual or nightly
  • • Catch bugs post-merge
  • • Fragmented test results
  • • Manual flake investigation
→

Tomorrow

  • • Automated UI tests on MRs
  • • Catch bugs pre-merge
  • • Unified test dashboard
  • • Automated flake detection
ROADMAP

Next Steps

🚨

1. iOS Main Emergency

P0 IMMEDIATE: 73.7% main failure (245/332). Root cause: UI tests. War room needed. This blocks ALL development.

🔧

2. Stabilize UI Tests

P0 THIS WEEK: Fix test:ui:draco and test:ui:zhl. Add retries, improve assertions, reduce flakiness. 245 failures traced to this.

🎯

3. Android Main Health

P1 THIS SPRINT: Reduce 21.3% to <5%. Fix emulator.wtf nightly tests. Address MR !1682 flakiness (9 failures).

📊

4. Monitoring & Prevention

P1 NEXT SPRINT: Build flakiness dashboard. Track test health. Alert on degradation. Prevent future crises.

Shift Left on Quality

Catch issues earlier, ship with confidence