# Guidepup Step 1 Playbook

Use this when you need to get VoiceOver automation working reliably in this repo.

This repo is a static HTML/CSS test harness, not a running Drupal site.
The `drupal` implementation name is a behavior variant label for comparison only.

## Why Guidepup is in this repo

Guidepup drives a real VoiceOver session and captures what it actually speaks.
That is the only track here that can answer the questions the deterministic tests
cannot: whether the proposal produces a genuinely different, better screen-reader
result rather than just tidier CSS or markup. See "Deck chairs versus real
improvement" in `TESTING.md`.

Getting preflight to pass is a prerequisite, not the goal. The goal is a captured,
comparable spoken log for current Drupal versus proposed, read as a *difference*.
A green preflight only means VoiceOver can be driven; it is not evidence about the
proposal.

### VoiceOver captures run standalone, not under `@playwright/test`

VoiceOver automation needs a browser window it can bring to the front. A direct
`webkit.launch()` from a plain Node script surfaces a `Playwright` GUI process
that AppleScript can front; the `@playwright/test` runner launches WebKit as a
background process with no frontable window, so the VoiceOver cursor never enters
web content and captures come back empty or wrong.

The differential VoiceOver capture is therefore a standalone runner:

```sh
npm run test:boundary-speech
```

It launches a fresh browser per variant and verifies (via `ensureInWebContent` in
`guidepup/at-helpers.js`) that the VoiceOver cursor actually reached the page's web
content before trusting any spoken phrase; it errors rather than record a bad
result if window fronting fails. On top of Accessibility and Automation, the
controlling app also needs Full Disk Access for Guidepup to mount its VoiceOver
preferences (otherwise start fails with an `EPERM` symlink error).

The deterministic checks do not use VoiceOver and run normally under the test
runner:

```sh
npm run test:skip-focus      # SFNSP focus resumption after the skip link
npm run test:focus-within    # :focus-within wrapper reveal, differential
```

See `guidepup/logs/at-findings.md` for the observed results these encode.

## Goal

Get this command passing:

```sh
npm run test:voiceover:preflight
```

If preflight passes, `npm run test:voiceover` can run the full 4-implementation traversal.

Preflight now opens the test page URL (`tests.html?implementation=proposed` by default) before VoiceOver movement checks.

If preflight does not pass, run:

```sh
npm run test:voiceover:diagnose
```

This prints a JSON diagnosis for AppleScript baseline checks and Guidepup move/phrase checks.
The VoiceOver scripts will try to turn VoiceOver off automatically before starting it again. If that fails, they now print a friendly message telling you to turn it off manually.

## Run order

From `hidden/`:

```sh
npm install
npx playwright install
npx @guidepup/setup setup
npx @guidepup/setup install
npm run test:voiceover:preflight
```

Quickly open the manual GUI setup screens from terminal:

```sh
npm run setup:voiceover:open-ui
```

Run per-element hidden content checks (records where expected phrases appear):

```sh
npm run test:elements
npm run test:elements:headed
npm run test:elements:strict
npm run test:elements:strict:headed
npm run test:elements:summary
npm run test:elements:summary:strict
```

Outputs:

- `guidepup/logs/elements-report.json`
- `guidepup/logs/elements-report.md`

Strict mode treats `proposed` as canonical and fails if other implementations diverge in status or phrase presence.
Summary mode prints only failures and baseline divergences from the latest report.

## Manual macOS steps (required)

Complete every step in the official guide:

- <https://www.guidepup.dev/docs/guides/manual-voiceover-setup>

Do not skip this section. `setup` + `install` are not sufficient by themselves.

## Terminal access setup (iTerm2 specific)

Guidepup runs AppleScript from the app that launched the command.
If you run tests in iTerm2, iTerm2 needs the permissions.
If you run tests in VS Code integrated terminal, VS Code needs the permissions.

Pick one terminal app and run all VoiceOver tests from that app consistently.

### 1) Accessibility permission

Open System Settings -> Privacy & Security -> Accessibility.

Ensure your terminal app is enabled:

- iTerm2 (if using iTerm2)
- Visual Studio Code (if using integrated terminal)

If missing, add it with the plus button, then toggle it on.

### 2) Automation permission (Apple Events)

Open System Settings -> Privacy & Security -> Automation.

Expand your terminal app entry and allow automation for:

- VoiceOver
- System Events

If you do not see these toggles yet, run the quick trigger commands below once; macOS should prompt.

### 3) Quick trigger commands (run in your chosen terminal)

```sh
/usr/bin/osascript -e 'tell application "VoiceOver" to activate'
/usr/bin/osascript -e 'tell application "System Events" to UI elements enabled'
npm run test:voiceover:diagnose
```

Accept every macOS permission prompt.

### 4) If permissions look right but preflight still fails

Reset TCC permissions for your terminal app, then retry and re-approve prompts.

For iTerm2:

```sh
tccutil reset Accessibility com.googlecode.iterm2
tccutil reset AppleEvents com.googlecode.iterm2
```

For VS Code:

```sh
tccutil reset Accessibility com.microsoft.VSCode
tccutil reset AppleEvents com.microsoft.VSCode
```

Then run again:

```sh
npx @guidepup/setup setup
npx @guidepup/setup install
npm run test:voiceover:diagnose
npm run test:voiceover:preflight
```

If preflight still fails with `right doesn\'t understand the "move" message (-1708)`, use the fallback section at the end of this file.

If `npm run test:voiceover:diagnose` shows AppleScript checks passing but `guidepupStart` still reports `VoiceOver cannot be started`, treat that as a startup/mount problem rather than a permissions problem:

- Re-run `npx @guidepup/setup setup`.
- Re-run `npx @guidepup/setup install`.
- Make sure you are launching every VoiceOver command from the same terminal app.
- Then run `npm run test:voiceover:diagnose` again before retrying preflight.

If the start failure persists after that, keep using `npm run test:validate` as the repeatable gate and record manual VoiceOver notes in `MANUAL_SCENARIO_SUMMARY.md` until the startup path is restored.

## Current known failure in this workspace

Preflight may fail in two different ways in this workspace:

- AppleScript automation error `-1708`
- `VoiceOver cannot be started`

The `-1708` case usually means VoiceOver is reachable but the move command is not.
The `VoiceOver cannot be started` case means the startup path itself failed, even if AppleScript permission checks passed.

For the `-1708` case, the usual symptom is:

- `VoiceOver unable to move`
- `right doesn’t understand the “move” message`

This indicates VoiceOver is still not fully controllable from this user session.

## Quick recovery checklist

1. Re-run `npx @guidepup/setup setup`.
2. Re-run `npx @guidepup/setup install`.
3. Re-check manual guide settings and permissions (VoiceOver + Automation + Accessibility).
4. Ensure VoiceOver can be controlled in the current login session.
5. Run `npm run test:voiceover:preflight` again.

## Running against the hosted page URL

Use `HIDDEN_TEST_URL` when you want to test the published page instead of local file URLs:

```sh
HIDDEN_TEST_URL="https://mgifford.github.io/accessibility-sandbox/hidden/tests.html" npm run test:voiceover:diagnose
HIDDEN_TEST_URL="https://mgifford.github.io/accessibility-sandbox/hidden/tests.html" npm run test:voiceover:preflight
HIDDEN_TEST_URL="https://mgifford.github.io/accessibility-sandbox/hidden/tests.html" npm run test:voiceover
```

To visibly confirm the browser page is loading, use headed mode:

```sh
HIDDEN_TEST_URL="https://mgifford.github.io/accessibility-sandbox/hidden/tests.html" npm run test:voiceover:preflight:headed
HIDDEN_TEST_URL="https://mgifford.github.io/accessibility-sandbox/hidden/tests.html" npm run test:voiceover:headed
HIDDEN_TEST_URL="https://mgifford.github.io/accessibility-sandbox/hidden/tests.html" npm run test:voiceover:headed:long
HIDDEN_TEST_URL="https://mgifford.github.io/accessibility-sandbox/hidden/tests.html" npm run test:voiceover:single:proposed:capture
HIDDEN_TEST_URL="https://mgifford.github.io/accessibility-sandbox/hidden/tests.html" npm run test:voiceover:single:proposed:scan
HIDDEN_TEST_URL="https://mgifford.github.io/accessibility-sandbox/hidden/tests.html" npm run test:voiceover:single:proposed:scan:noheadings
HIDDEN_TEST_URL="https://mgifford.github.io/accessibility-sandbox/hidden/tests.html" npm run test:voiceover:single:proposed:scan:sequential
HIDDEN_TEST_URL="https://mgifford.github.io/accessibility-sandbox/hidden/tests.html" npm run test:voiceover:single:drupal:headed:long
HIDDEN_TEST_URL="https://mgifford.github.io/accessibility-sandbox/hidden/tests.html" npm run test:voiceover:single:drupal:capture
```

If you run commands from the repository root by mistake, use npm prefix mode:

```sh
npm --prefix /Users/mike.gifford/accessibility-sandbox/hidden run test:voiceover:single:proposed:scan:sequential
```

The full VoiceOver traversal can exceed Playwright's default 30s timeout. Use `test:voiceover:headed:long` for slower machines/sessions.
If you want one stable pass before comparing all implementations, use `test:voiceover:single:proposed:capture` first.
If VoiceOver seems to stop early or repeat the same phrase, use `test:voiceover:single:proposed:scan` for phased heading/link navigation with step-by-step progress logs.
If you hear repeated `heading not found`, use `test:voiceover:single:proposed:scan:noheadings`.
If you want no heading/link probe jumps at all, use `test:voiceover:single:proposed:scan:sequential`.
Use `single:drupal:*` only when you explicitly want to compare that variant.
After that, use `test:voiceover:strict` for full strict comparison assertions.
You can also tune runtime with env vars:

- `VOICEOVER_TEST_TIMEOUT_MS` (default `180000`)
- `VOICEOVER_TRAVERSAL_STEPS` (default `160`)
- `VOICEOVER_STEP_DELAY_MS` (default `220`)
- `VOICEOVER_STAGNATION_LIMIT` (default `28`)
- `VOICEOVER_REPEAT_PHRASE_LIMIT` (default `10`)
- `VOICEOVER_HEADING_PROBE_EVERY` (default `18`; use `99999` to effectively disable heading probes)
- `VOICEOVER_LINK_PROBE_EVERY` (default `11`)
- `VOICEOVER_IMPLEMENTATIONS` (default `drupal,a11yproject,govuk,proposed`; set `proposed` for a single pass)
- `VOICEOVER_MIN_CHECKPOINT_MATCHES` (default `1`)
- `VOICEOVER_REQUIRE_SPOKEN_LOG` (default `0`; set `1` for strict runs)

Important: run these commands from `accessibility-sandbox/hidden`.
If run from `drupal-core`, npm will report `Missing script` for `test:voiceover:*` because those scripts are only defined in this folder's `package.json`.

## Interpreting diagnose output with fallback

If you see this combination:

- `guidepupMoveNext`: failed with `-1708`
- `guidepupMoveNextFallback`: succeeded

then terminal permissions are generally fine, and this environment should use keyboard command fallback for movement.

If `guidepupLastPhrase` is empty in diagnose, treat that as non-blocking unless full traversal tests also fail.
Likewise, preflight may print a warning about empty phrase/item capture; if movement succeeded and no preflight failure is reported, continue to `npm run test:voiceover`.

## Decision tree

1. Run `npm run test:voiceover:diagnose`.
2. If AppleScript baseline checks fail:
   - Focus on macOS permissions and manual setup steps.
3. If AppleScript baseline checks pass, but `guidepupMoveNext` fails with `-1708`:
   - You are likely hitting a Guidepup/VoiceOver command compatibility issue on this macOS version or still have an incomplete VoiceOver manual setup detail.
4. In that state, keep using deterministic validation (`npm run test:validate`) and manual Scenario 4/9 review until VoiceOver automation is unblocked.

## Evidence to keep

- Preflight output (pass/fail)
- `npm run test:voiceover` output
- Any generated logs under `guidepup/logs/`

## Session note file (recommended)

Create a timestamped copy of the manual template before each run:

```sh
ts=$(date +%Y%m%d-%H%M%S)
cp guidepup/logs/manual-scenarios-comparison-template.md guidepup/logs/manual-scenarios-${ts}.md
echo "Created guidepup/logs/manual-scenarios-${ts}.md"
```

Use the new file to capture Scenario 4 and Scenario 9 observations for that specific run.

## After preflight passes

Run:

```sh
npm run test:voiceover
```

Expected outputs:

- `guidepup/logs/voiceover-drupal.json`
- `guidepup/logs/voiceover-a11yproject.json`
- `guidepup/logs/voiceover-govuk.json`
- `guidepup/logs/voiceover-proposed.json`
- `guidepup/logs/voiceover-differences.md`

## Practical fallback when blocked

If preflight remains blocked at `VoiceOver unable to move`:

- Treat `npm run test:validate` as your repeatable gate.
- Use `MANUAL_SCENARIO_SUMMARY.md` for consistent manual VoiceOver and zoom notes.
- Keep the `test:voiceover:diagnose` JSON output as reproducible evidence for future Guidepup/macOS compatibility follow-up.