Pull-request checks

What a check is

A check is a matrix: every flow × every device your visitors use × a few people, run against the pull request’s preview and against the base (production, or whatever you pass as base). The people are the same on both sides, in the same order, so the difference is the build and not the luck of the draw. The verdict is per flow and per device: a flow that works on desktop and fails on a phone is reported as a regression on mobile, not averaged away.

The flows come from your workspace (simulithic map wrote them; simulithic flows edits them), so there is nothing to keep in the repository. One comment is posted, and updated in place on later pushes: the verdict, the per-flow cells, links to the recordings, and what went wrong with a fix prompt where the cause is clear.

Exit codes: 0 when nothing regressed, 2 on a regression, 1 when the check itself could not run.

GitHub Actions

Mint a token for CI with the CLI and store it as the repository secret SIMULITHIC_TOKEN; with the GitHub CLI that is one line, simulithic token --name github-ci | gh secret set SIMULITHIC_TOKEN. It is the only secret a check needs; nothing else goes in the repository. The token is printed once, is separate from your own login, and lasts 90 days. Then add a workflow that runs when the preview deployment is ready. With Vercel that is the deployment_status event; note that GitHub only fires it from the default branch, so the workflow file has to be merged before it runs on PRs.

name: Simulithic check
on:
  deployment_status:
jobs:
  check:
    if: github.event.deployment_status.state == 'success' && github.event.deployment.environment == 'Preview'
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    env:
      SIMULITHIC_TOKEN: ${{ secrets.SIMULITHIC_TOKEN }}
      PREVIEW_URL: ${{ github.event.deployment_status.target_url || github.event.deployment_status.environment_url }}
    steps:
      - run: curl -fsSL https://app.simulithic.com/cli/install.sh | sh
      - run: |
          ~/.simulithic/cli/simulithic ci \
            --url "$PREVIEW_URL" \
            --base "https://app.yourproduct.com" \
            --project ws_xxxxxx --users 3 --fail-on high \
            --repo "$GITHUB_REPOSITORY" --sha "$GITHUB_SHA"

--repo and --sha let the check find the pull request for the deployment and post the comment itself. On other CI systems pass --pr with the number, or use --out comment.mdand --json and post them however you like.

Options

Size and cost

A web check is flows × devices × people × 2 sessions. Six flows on two devices with three people is 72 sessions, about two dollars and a few minutes. --users 2 halves the cost, and --only sends people through the flows you name alone. The check prints its plan and estimate before it starts.

Native apps: Mac, Android and iOS

A native app gets the same check — a Mac app (Swift, Electron, Tauri or Flutter, built for Apple Silicon), an Android app (an .apk with arm64-v8a: native, React Native, Expo, Flutter) or an iOS app (a simulator build, the .app from xcodebuild -sdk iphonesimulator; an .ipa is a device build and cannot run). Nothing runs on your CI box but the build: the PR’s build is uploaded, compared with the newest build uploaded from the base branch, and both run on a Simulithic worker — a Mac, a pristine Android 14 phone per run, or an iPhone simulator whose app data is wiped between people — each person starting from a just-installed app. The journeys come from mapping the app once (simulithic map --app build/MyApp.app, or --app app-release.apk): a worker explores it, typing into its fields and pressing what that enables, and writes the journeys into your workspace, so there is nothing to keep in your repo. Two commands in CI, after whatever builds the app:

      - run: curl -fsSL https://app.simulithic.com/cli/install.sh | sh
      # on every push to main: the base for later checks
      - if: github.event_name == 'push'
        run: ~/.simulithic/cli/simulithic upload build/MyApp.app
      # on every pull request: this build against the newest main build
      - if: github.event_name == 'pull_request'
        run: ~/.simulithic/cli/simulithic ci --app build/MyApp.app --base-branch main --users 2 --out comment.md

      - run: ./gradlew assembleRelease
      - run: curl -fsSL https://app.simulithic.com/cli/install.sh | sh
      - if: github.event_name == 'push'
        run: ~/.simulithic/cli/simulithic upload app/build/outputs/apk/release/app-release.apk
      - if: github.event_name == 'pull_request'
        run: ~/.simulithic/cli/simulithic ci --app app/build/outputs/apk/release/app-release.apk --base-branch main --users 2 --out comment.md

      - run: xcodebuild -workspace App.xcworkspace -scheme App -configuration Release -sdk iphonesimulator -destination 'generic/platform=iOS Simulator' -derivedDataPath build CODE_SIGNING_ALLOWED=NO
      - run: curl -fsSL https://app.simulithic.com/cli/install.sh | sh
      - if: github.event_name == 'push'
        run: ~/.simulithic/cli/simulithic upload build/Build/Products/Release-iphonesimulator/App.app
      - if: github.event_name == 'pull_request'
        run: ~/.simulithic/cli/simulithic ci --app build/Build/Products/Release-iphonesimulator/App.app --base-branch main --users 2 --out comment.md

upload reads the bundle id or package, version and architecture from the app itself (no Xcode, aapt or Java needed on the box) and the commit, branch and PR from the CI it runs on (GitHub Actions, Bitrise, GitLab, CircleCI, Xcode Cloud, EAS); the same bytes twice are not sent again. It refuses what could not run on our phones — an x86-only APK, or a React Native / Expo development build with no JavaScript bundled (upload a release build) — and notes whether the app hard-requires Google Play services (Maps, Billing, Google Sign-In), which the phone tier does not have. On a phone, a crash of the app during a person’s session (Android: the logcat crash buffer and the app’s own error lines; iOS: the simulator’s crash report) is reported as a defect with the trace as the fix prompt, and so is an app that goes blank under the person, which is how React Native fails in release builds. The comment is in comment.md for your workflow to post; --journeys <file> overrides the workspace’s journeys for one run. Not on GitHub Actions? The two commands run anywhere the build exists.

A cross-platform app keeps one journey set under its bundle id, mapped from whichever platform you ran map --app on last; each platform’s map is kept. See Simulate for what runs where today.