QA.pro from CI.

Create an API key in Settings → API keys (Team plan and up) and store it as a secret. The workflow below tests every successful preview deployment. On Cloud, hosted runners take preview URLs under a domain you verified (a verified apex covers preview-123.shop.example); any other host waits for your own runner, and the run's waiting field says so.

# .github/workflows/qapro.yml
name: QA.pro
on: [deployment_status]
jobs:
  check:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - name: Test the preview URL
        env:
          QAPRO_URL: https://qa.pro          # or your own server
          QAPRO_KEY: ${{ secrets.QAPRO_KEY }}
          TARGET: ${{ github.event.deployment_status.target_url }}
        run: |
          set -u
          run=$(curl -sS --fail-with-body -X POST "$QAPRO_URL/api/v1/runs" -H "Authorization: Bearer $QAPRO_KEY" \
                -H "Content-Type: application/json" -d "{\"urls\": [\"$TARGET\"]}") || { echo "::error::could not start the run: $run"; exit 1; }
          # a run may be split in two when some sites need your own runner: poll every id the API returned
          ids=$(echo "$run" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(" ".join(str(i) for i in [d["id"]] + d.get("also_queued", [])))')
          fail=0
          for id in $ids; do
            echo "run $QAPRO_URL/runs/$id"
            status=queued; st=""
            for i in $(seq 1 90); do
              sleep 20
              st=$(curl -sS --fail-with-body "$QAPRO_URL/api/v1/runs/$id" -H "Authorization: Bearer $QAPRO_KEY") || { echo "poll: $st"; sleep 20; continue; }   # a transient 502 must not fail the job
              status=$(echo "$st" | python3 -c 'import json,sys; print(json.load(sys.stdin)["status"])')
              if [ "$status" = done ] || [ "$status" = failed ] || [ "$status" = cancelled ]; then break; fi
            done
            if [ "$status" = queued ] || [ "$status" = running ]; then                 # nothing finished it in 30 minutes: say why
              waiting=$(echo "$st" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("waiting") or "no runner was free")')
              echo "::error::run $id: $waiting"; fail=1; continue
            fi
            echo "$st" | python3 -c '
          import json, sys
          d = json.load(sys.stdin); s = d["summary"]; r = s["red_sites"]
          print("run %d: %d/%d tests passed, %d failing %s, %d of %d results" % (d["id"], s["pass"], s["ended"], r, "site" if r == 1 else "sites", s["results"], s["expected"]))
          ok = d["status"] == "done" and r == 0 and s["ended"] > 0 and s["results"] >= s["expected"]
          sys.exit(0 if ok else 1)' || fail=1
          done
          exit $fail

Self-hosted, the run needs one of your runners online (Runners page). Ad-hoc URLs use the team's default tests; pass "checks": ["add_to_cart", "search_product"] to choose.

Raw file: ci.md.