* ci: add resolution guidance to failing snapshot PR comment
When snapshots differ from the main baseline, the comment now includes
a short blockquote explaining both resolution paths:
- merge the latest main (in case upstream baselines have moved)
- add the update-snapshots label to acknowledge intentional changes
* ci: wait for main baseline workflow before downloading artifact
Before downloading the snapshot-baselines artifact on PR runs, resolve
main's current HEAD SHA and check if the snapshot-tests.yml run for
that exact commit is still in-progress or queued. If so, poll every
10 s (up to 10 min) until it completes, then proceed.
This eliminates the race condition where a PR job starts while main's
baseline upload is still in-flight, causing it to pull the previous
(stale) artifact and produce false snapshot failures.
The wait targets only the run for the current HEAD SHA — not an older
in-progress run from a different commit — so two rapid commits to main
can't trick the check into waiting for the wrong run.
Also bumps job timeout-minutes from 20 → 30 to accommodate the wait.
---------
Co-authored-by: openhands <openhands@all-hands.dev>
* ci: store snapshot baselines as GitHub Actions artifacts, not in git
Move baseline PNG storage from git to a 90-day GitHub Actions artifact
named 'snapshot-baselines', uploaded on every push to main.
- PRs download the latest main-branch artifact and run Playwright
comparison against it; no more checked-in PNGs causing merge conflicts.
- New 'post-snapshot-comment.mjs' script classifies each snapshot as
Changed/New/Unchanged, commits images to .pr/snapshots/<run_id>/ and
posts a PR comment with collapsed <details> sections showing
side-by-side expected/actual/diff images via raw.githubusercontent.com.
- For fork PRs or if push fails, falls back to a workflow run link for
downloading the 'snapshot-test-results' artifact.
- Force-refresh baselines any time via workflow_dispatch force_update=true
(replaces the old update_snapshots=true flow that committed PNGs to git).
- Remove 44 baseline PNGs from git; gitignore tests/e2e/__snapshots__/.
- Update AGENTS.md with the new workflow model.
Co-authored-by: openhands <openhands@all-hands.dev>
* ci: fix bootstrap case — pass CI when no baseline artifact exists yet
When no main-branch 'snapshot-baselines' artifact has been uploaded yet
(e.g. this very first run after merging from an old baseline-in-git flow),
the comparison step fails because Playwright has nothing to compare against.
Gate the 'Fail if differences' step on has_baselines==true so the bootstrap
PR passes with all snapshots shown as new. Once it merges to main the
artifact is created and subsequent PRs compare normally.
Also derive the PR comment status from the classification (changed.length > 0)
rather than from TEST_OUTCOME, which is 'failure' in the bootstrap case
despite zero actual regressions.
Co-authored-by: openhands <openhands@all-hands.dev>
* ci: delete stale comment and re-post on each push; always embed new snapshot images
- Replace PATCH-in-place with DELETE + POST so every push posts a fresh
comment whose image URLs reference the current run's .pr/snapshots/<run_id>/.
Editing in-place would leave raw.githubusercontent.com URLs pointing at
the previous run's images once new images are committed under a new run_id.
- Embed new snapshot images inside the collapsed <details> section when
commitSha is available; add a fallback artifact-download link when the
push fails (e.g. fork PRs).
- Handle 204 No Content returned by DELETE in githubFetch.
Co-authored-by: openhands <openhands@all-hands.dev>
* ci: fix find-run to query artifacts API by name, not workflow runs by status
The previous approach (find latest successful run of snapshot-tests.yml on
main) would match old runs that predate the artifact upload step, causing
actions/download-artifact to hard-fail with 'Artifact not found' before the
comparison or comment steps could run.
Fix: query the artifacts REST API directly for name=snapshot-baselines,
filtering to non-expired artifacts from the main branch. This guarantees
we only match runs that actually uploaded the baseline artifact.
Also add continue-on-error: true to the download step as a safety net
against the artifact expiring between the API lookup and the download.
Co-authored-by: openhands <openhands@all-hands.dev>
* chore: snapshot images for run 25929925639 [skip ci]
* ci: replace .pr/snapshots on each run instead of accumulating per-run dirs
Previously each CI run committed images under .pr/snapshots/<run_id>/, so
reruns would accumulate multiple directories on the PR branch. The PR comment
always pointed to the current run's images (via SHA in the raw.githubusercontent
URL), but old directories silently piled up.
Fix: use a fixed .pr/snapshots/ path and git rm -rf --ignore-unmatch it before
staging new images. Each run completely replaces the previous images rather than
appending alongside them. Raw URLs still use the commit SHA so they remain stable
per push.
Co-authored-by: openhands <openhands@all-hands.dev>
* chore: snapshot images for run 25930435218 [skip ci]
* ci: allow review_requested on draft same-repo PRs to trigger pr-review
GitHub does not fire pull_request events for review_requested on draft PRs —
only pull_request_target fires. The existing if condition rejected
pull_request_target for same-repo PRs via the fork check, so requesting
all-hands-bot or openhands-agent on a draft PR was always silently skipped.
Add a carve-out: pull_request_target is also accepted for same-repo PRs
when draft==true AND action==review_requested. Non-draft same-repo PRs are
unaffected — they continue to be handled by the pull_request event, and the
draft==true guard prevents pull_request_target from also running (no duplicate).
Co-authored-by: openhands <openhands@all-hands.dev>
* ci: add update-snapshots label bypass for intentional snapshot changes
When snapshot diffs are expected (UI redesign, intentional change, etc.) the
author now adds the 'update-snapshots' label to the PR to acknowledge them:
- Fail step gains a !contains(labels, 'update-snapshots') guard so CI passes
even when Playwright reports differences.
- The PR comment status adjusts: ❌ 'N snapshots differ — add label to
acknowledge' when unapproved, ✅ 'N snapshots changed — acknowledged via
label' when approved.
- The snapshot workflow now triggers on labeled/unlabeled events so that adding
or removing the label immediately re-runs CI with the current label state in
scope (no manual re-run or empty commit needed).
- New baselines are uploaded automatically when the PR merges to main, so no
separate 'regenerate on main' step is needed.
Co-authored-by: openhands <openhands@all-hands.dev>
* docs: update snapshot testing section in AGENTS.md
Add details on: artifact lookup by name, delete-then-post comment behavior,
fixed .pr/snapshots/ path (no accumulation), update-snapshots label bypass
for intentional changes, labeled/unlabeled triggers, bootstrap behavior.
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: git rm must run before copyFile, not after
On the second CI run the branch already has .pr/snapshots/ tracked from the
previous run. The old order was: copyFile → git rm → git add. git rm removes
tracked files from disk, which deleted the freshly written images, leaving the
directory empty and causing 'fatal: pathspec did not match any files'.
Fix: run git rm --ignore-unmatch before copyFile so the tracked files are
cleared from disk first; then copyFile writes clean new files with nothing
to conflict; then git add finds them as expected.
Co-authored-by: openhands <openhands@all-hands.dev>
* chore: snapshot images for run 25931876588 [skip ci]
* fix: push snapshot images to orphan branch, not PR branch
Pushing to the PR branch with [skip ci] caused required checks to never
run on the HEAD commit, permanently blocking the PR.
New approach:
- publishImages() creates a fresh git repo in a temp directory, adds the
images, and force-pushes to snapshot-artifacts/pr-<N> — a dedicated
ephemeral branch that no CI workflow watches.
- The PR branch is never touched by CI, so required checks always run on
the actual code commits.
- [skip ci] is removed; no loop prevention is needed because nothing
triggers snapshot CI on the artifacts branch.
- Images in the orphan commit live at changed/<relPath>-{actual,expected,diff}.png
and new/<relPath>.png (no .pr/snapshots/ prefix).
- raw.githubusercontent.com/<owner>/<repo>/<sha>/changed/... URLs are
stable because they pin the orphan commit SHA.
- pr-artifacts.yml gains a closed trigger + cleanup-snapshot-artifacts job
that deletes snapshot-artifacts/pr-<N> when the PR merges or is abandoned.
- Stale .pr/snapshots/ files from previous CI runs removed from this branch.
Co-authored-by: openhands <openhands@all-hands.dev>
* docs: update AGENTS.md — orphan branch image storage, branch cleanup
* docs: tighten AGENTS.md snapshot section and pr-artifacts description
- Remove duplicate gitignore mention (already stated in baseline-storage line)
- Update pr-artifacts.yml description to cover both cleanup responsibilities:
.pr/live-e2e/ (on approval) and snapshot-artifacts/pr-<N> (on close)
Co-authored-by: openhands <openhands@all-hands.dev>
---------
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix: restore data-testid="chat-interface" removed by #340
PR #340 added left padding to the ChatInterface wrapper div but
accidentally dropped the data-testid attribute in the same edit.
This broke both the collapsible-thinking snapshot tests and the live
e2e test, which both use getByTestId('chat-interface') as the load
signal and screenshot target.
Co-authored-by: openhands <openhands@all-hands.dev>
* chore: update baseline snapshots [skip ci]
* ci: trigger re-run after snapshot baseline update
---------
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* feat: add Playwright visual snapshot testing infrastructure
- Add snapshot test file for home and settings pages
- Configure playwright.config.ts with snapshot settings
- Add npm scripts: test:e2e:snapshots and test:e2e:snapshots:update
- Create CI workflow (.github/workflows/snapshot-tests.yml)
- Include baseline snapshots for chromium
Closes#390
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: properly mock analytics consent in snapshot tests
- Add setupMocks helper with showConsentModal parameter
- Set user_consents_to_analytics: false to hide modal by default
- Add dedicated test for analytics consent modal appearance
- Document snapshot testing patterns in AGENTS.md
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: add explicit modal absence assertions in snapshot tests
- All non-modal tests now assert consent modal has count 0
- Use rootLayout consistently for all snapshots
- Tests will fail fast if modal incorrectly appears
Note: Snapshots need regeneration - CI will fail until baselines updated
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: dismiss consent modal before taking snapshots
- Add dismissConsentModal helper to click 'Confirm preferences'
- Call dismissConsentModal after page load in all non-modal tests
- Regenerate all baseline snapshots without modal overlay
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: make snapshot tests work in mock mode CI
- Add file API mock to prevent proxy errors
- Make consent modal test skip if modal doesn't appear in mock mode
- Tests now pass in both local and CI environments
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: generate snapshots in CI environment
- Add workflow_dispatch with update_snapshots option
- Remove local snapshots (will be generated in CI)
- CI can now update and commit snapshots automatically
- Update AGENTS.md with snapshot testing details
To generate snapshots: Run workflow manually with 'Update baseline snapshots' checked
Co-authored-by: openhands <openhands@all-hands.dev>
* chore: update baseline snapshots [skip ci]
* docs: add CI snapshot update instructions to AGENTS.md
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: address review comments
- Use setupMocks(page, true) for consent modal test instead of try-catch
- Align global threshold to 0.01 (1%) matching documented standard
Co-authored-by: openhands <openhands@all-hands.dev>
* chore: update baseline snapshots [skip ci]
* fix: stabilize flaky consent modal test
- Wait for root-layout to be visible before checking modal
- Add networkidle wait for settings query to resolve
- Increase modal visibility timeout to 10s for lazy-load
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: wait for settings API response to stabilize consent modal test
- Use Promise.all to wait for settings response during navigation
- Increase root-layout visibility timeout to 10s
- Ensures settings data is loaded before checking for modal
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: increase timeouts for consent modal test
- Set test timeout to 60s
- Use networkidle for goto
- Increase element visibility timeouts to 15s
Co-authored-by: openhands <openhands@all-hands.dev>
---------
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* fix: also tag prerelease versions as latest on npm
Since we don't have stable releases yet, prerelease versions (alpha, beta, rc)
should also be tagged as 'latest' so users running 'npm install @openhands/agent-canvas'
get the most recent version rather than needing to specify @alpha explicitly.
The workflow now:
1. Publishes with the prerelease tag (e.g., 'alpha')
2. Also adds the 'latest' tag to that version
Co-authored-by: openhands <openhands@all-hands.dev>
* docs: note npm dist-tag cleanup issue
Co-authored-by: openhands <openhands@all-hands.dev>
* docs: warn about prerelease npm latest tag
Co-authored-by: openhands <openhands@all-hands.dev>
---------
Co-authored-by: openhands <openhands@all-hands.dev>
* Add demo flow E2E coverage
* Add live Agent Server E2E
* Stabilize live E2E CI
* Stabilize live Agent Server E2E
* Use default pull request workflow triggers
* Organize Playwright E2E tests
* Comment live E2E results on PR
* Fix live E2E comment permissions
* Stabilize live E2E PR reporting
* Add collapsible live E2E evidence
* Embed live E2E media in PR report
* Use release assets for live E2E media
* Stabilize live E2E media and auth
* Use raw URLs for live E2E media
* Update tests for SDK workspace API
* Use PR artifacts for live E2E media
* Move live E2E scripts under tests
* chore: Update PR QA artifacts
* Clarify live E2E test layout
* Document and simplify live E2E local runs
* Remove unrelated non-test diffs
* Preserve HEAD git ref in workspace client
* Strengthen live Agent Server E2E
* chore: Update PR QA artifacts
* Remove workspace session URL normalization
* Remove obsolete mock E2E regressions
* Disable live E2E trace capture
* Harden live E2E workflow
* Harden live E2E review fixes
* Fix live E2E manual checkout
* chore: Update PR QA artifacts
* Address live E2E re-review feedback
* Address live E2E security review feedback
* Address live E2E approval suggestions
* chore: address PR review feedback (#195)
* chore: address live e2e review followups (#195)
* chore: Remove PR-only artifacts
* chore: address latest live e2e review
* fix(ci): drop --ignore-scripts so typescript-client git dep builds
After merging main (PR #278), source files import directly from
@openhands/typescript-client subpath exports (e.g. /clients,
/workspace/remote-workspace). These resolve to dist/ files that are
generated by the package's prepare script. The --ignore-scripts flag
on npm ci prevented that script from running, so CI's typecheck
failed with TS2307 'Cannot find module' for every subpath import.
Main's CI uses plain 'npm ci' (no --ignore-scripts) and passes.
Align this branch to match.
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: restore avatar-menu and css-isolation regression tests
These were moved from tests/ to tests/e2e/regressions/ in 08b8e12
but then mistakenly deleted in 5c5a39b. The live E2E framework is
additive — it should not remove existing browser regression coverage.
The placeholder.spec.ts is not restored since it was a no-op stub.
Co-authored-by: openhands <openhands@all-hands.dev>
---------
Co-authored-by: allhands-bot <allhands-bot@users.noreply.github.com>
Co-authored-by: openhands <openhands@all-hands.dev>
* Add npm publish workflow and release infrastructure
- Add .github/workflows/npm-publish.yml for automated npm publishing on GitHub releases
- Update CI to verify library build (npm run build:lib) and package contents
- Add CHANGELOG.md for version history tracking
- Update README.md with npm installation and usage documentation
Closes#197
Co-authored-by: openhands <openhands@all-hands.dev>
* correct package version
* chore: update npm-publish workflow for trusted publishing
- Remove NODE_AUTH_TOKEN secret dependency
- Keep id-token: write permission for OIDC
- Add provenance flag for npm attestations
- Add comment explaining trusted publisher setup on npmjs.com
Co-authored-by: openhands <openhands@all-hands.dev>
* feat: add CLI entry point for npx execution
- Add bin/agent-canvas.mjs as executable CLI
- Add bin field to package.json for npm bin linking
- Include bin/ and build/ directories in published files
- CLI serves the built application with SPA routing support
- Supports --port, --host, and --help options
Co-authored-by: openhands <openhands@all-hands.dev>
* refactor: consolidate npm executable to use dev-docker infrastructure
- bin/agent-canvas.mjs now uses dev-with-automation.mjs main() with
dev-docker.mjs's Docker-specific agent-server starter
- Added --static and --static-dir support to dev-with-automation.mjs
so the npm executable serves pre-built static assets instead of Vite
- Added startStaticFrontend() function that uses static-server.mjs
- npm executable runs full stack: Docker agent-server + uvx automation
backend + static frontend + ingress proxy
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: include scripts/ in npm package files
The bin/agent-canvas.mjs executable imports from scripts/dev-with-automation.mjs
and scripts/dev-docker.mjs, so the scripts directory must be included in the
published package.
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: address review comments
- Fix CHANGELOG.md version mismatch: 1.6.0 -> 1.0.0-alpha.1 to match package.json
- Add NODE_AUTH_TOKEN env var to npm-publish workflow for authentication
- Add CLI entry point mention to CHANGELOG
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: use OIDC trusted publishing (no NPM_TOKEN needed)
npm trusted publishing with OIDC doesn't require NODE_AUTH_TOKEN.
Instead it uses short-lived OIDC tokens generated by GitHub Actions.
Requirements:
- id-token: write permission (already set)
- npm CLI 11.5.1+ (added npm install -g npm@latest step)
- Trusted publisher configured on npmjs.com
See: https://docs.npmjs.com/trusted-publishers/
Co-authored-by: openhands <openhands@all-hands.dev>
* chore: bump version to 1.0.0-alpha.2
Co-authored-by: openhands <openhands@all-hands.dev>
* Build app assets before npm publish
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: use Node 24 for npm trusted publishing
Trusted publishing requires Node 22.14.0+ and npm 11.5.1+.
Node 24 ships with npm 11.x which meets the requirement.
Node 22.12.0 (previous) ships with npm 10.x which doesn't support OIDC.
Also removed the manual npm upgrade step since Node 24 includes
a compatible npm version by default.
Co-authored-by: openhands <openhands@all-hands.dev>
* chore: align all workflows to Node 24 and regenerate lockfile
- Update ci.yml to use Node 24
- Update sdk-version-sync.yml to use Node 24
- Regenerate package-lock.json with npm 11.12.1
All workflows now use Node 24 which ships with npm 11.x,
required for OIDC trusted publishing (npm 11.5.1+).
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: remove incorrect LLM env vars from CLI help
LLM_MODEL and LLM_API_KEY were listed in the help text but aren't
actually used by the scripts. LLM settings are configured through
the web UI settings page instead.
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: address PR review feedback
Critical fixes:
- Guard prepare script to only run in dev context (check for ../.git)
- Add missing existsSync import in dev-with-automation.mjs
Workflow improvements:
- Update checkout/setup-node actions to v6 for consistency
- Add npm version validation (must be 11.5.1+ for trusted publishing)
- Add package version validation (must match release tag)
CLI improvements:
- Add try-catch for dynamic imports with helpful error message
- Use console.error directly instead of imported logError/c
Documentation:
- Fix README export names: ChatInterface→ChatPanel, Terminal→TerminalPanel
- Add dist/ to .gitignore
Co-authored-by: openhands <openhands@all-hands.dev>
* ci: trigger npm publish on tag push instead of release
Simpler workflow - just push a tag like v1.0.0-alpha.2 to publish.
Co-authored-by: openhands <openhands@all-hands.dev>
* chore: remove tarball and add *.tgz to gitignore
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: npm publish errors
1. Fix bin path - remove './' prefix (npm pkg fix)
2. Add --tag for prerelease versions (alpha/beta/rc)
Co-authored-by: openhands <openhands@all-hands.dev>
* fix: add repository field for npm provenance verification
npm provenance requires repository.url to match the GitHub Actions
source. Also added description, homepage, and bugs fields.
Co-authored-by: openhands <openhands@all-hands.dev>
---------
Co-authored-by: openhands <openhands@all-hands.dev>
* feat: update SDK to 1.22.0 and add CI version sync check
- Update DEFAULT_AGENT_SERVER_VERSION from 1.21.1 to 1.22.0 in dev-safe.mjs
- Update SDK version references in AGENTS.md
- Add scripts/check-sdk-version-sync.mjs to verify automation project uses
matching SDK versions for openhands-sdk, openhands-tools, openhands-workspace,
and openhands-agent-server
- Add .github/workflows/sdk-version-sync.yml CI workflow with:
- Path-filtered PR/push triggers for version-related file changes
- repository_dispatch triggers (sdk-version-check, sdk-release) for
external repos to notify when SDK deps change
- workflow_dispatch with optional version override
- Scheduled runs every 6 hours to catch upstream changes
- PyPI version checking support (--check-pypi flag)
The check script supports:
- EXPECTED_SDK_VERSION env var override for CI triggers
- --check-pypi flag to also display latest PyPI versions
- --help for usage documentation
To trigger from external repos (e.g., OpenHands/automation or SDK repo):
curl -X POST -H "Authorization: token \$GITHUB_TOKEN" \\
https://api.github.com/repos/OpenHands/agent-canvas/dispatches \\
-d '{"event_type": "sdk-version-check"}'
* fix: check released PyPI version instead of GitHub main branch
The SDK version sync check now fetches dependencies from the released
openhands-automation package on PyPI (version specified by
DEFAULT_AUTOMATION_VERSION in dev-with-automation.mjs) rather than
fetching pyproject.toml from the GitHub main branch.
This ensures we're checking the actual released version that users
would install, not the development version on main.
* fix: address review feedback for SDK version sync check
- Add env var overrides for automation package name and version
- Add retry logic with exponential backoff for PyPI API failures
- Add semantic version normalization for comparing versions
- Fix repository_dispatch to use client_payload.version
- Improve regex to handle parenthesized dependency formats
- Add comprehensive test coverage for helper functions
* fix: add type casts for dynamic module import in tests
* chore: update automation version to 1.0.0a2
- Update DEFAULT_AUTOMATION_VERSION in dev-with-automation.mjs
- Update AGENTS.md documentation
- Update test expectation
---------
Co-authored-by: openhands <openhands@all-hands.dev>
This repo typically has large PRs spanning multiple files, so enabling
sub-agent delegation lets the review bot fan out file-level reviews to
dedicated sub-agents for better coverage.
Co-authored-by: openhands <openhands@all-hands.dev>
Group npm packages by feature area so related libraries land in a
single PR rather than as separate, racing updates. This avoids the
package-lock.json conflicts we saw when sequential PRs all touched
the same lockfile entries (e.g. tailwindcss + @tailwindcss/vite),
and reduces churn for users that watch the dependencies label.
Groups:
- tailwind, tanstack, i18next, react, react-router, testing,
eslint, monaco, xterm, types
High-impact packages that are not assigned to a group (vite,
framer-motion, axios, posthog-js, lucide-react, etc.) still get
their own PR so each can be reviewed and tested independently.
GitHub Actions updates are now also grouped into a single weekly PR.
Co-authored-by: openhands <openhands@all-hands.dev>
