Fix stale Docker image tags & add version consistency CI + update-sdk skill (#13315)

Co-authored-by: openhands <openhands@all-hands.dev>
2026-03-22 05:37:20 +08:00 · 2026-03-10 01:23:30 +08:00
parent fe2e50fc7d
commit 8356170193
6 changed files with 435 additions and 3 deletions
--- a/.agents/skills/update-sdk/SKILL.md
+++ b/.agents/skills/update-sdk/SKILL.md
@@ -0,0 +1,123 @@
+---
+name: update-sdk
+description: This skill should be used when the user asks to "update SDK", "bump SDK version", "pin SDK to a commit", "test unreleased SDK", "update agent-server image", "bump the version", "prepare a release", "what files change for a release", or needs to know how SDK packages are managed in the OpenHands repository. For detailed reference material, see references/docker-image-locations.md and references/sdk-pinning-examples.md in this skill directory.
+---
+
+# Update SDK
+
+Bump SDK packages (`openhands-sdk`, `openhands-agent-server`, `openhands-tools`), pin them to unreleased commits for testing, and cut an OpenHands release.
+
+## Quick Summary — How Many Files Change?
+
+| Activity | Manual edits | Auto-regenerated | Total |
+|----------|:------------:|:----------------:|:-----:|
+| **SDK bump** (released PyPI version) | 2 | 3 | **5** |
+| **SDK pin** (unreleased git commit) | 3 | 3 | **6** |
+| **Release commit** (version bump) | 3 | 0 | **3** |
+
+The 3 auto-regenerated files are always: `poetry.lock`, `uv.lock`, `enterprise/poetry.lock`.
+
+## SDK Package Bump — 2 Files + 3 Lock Files
+
+Land as a separate PR before the release. Examples: `929dcc3` (SDK 1.11.5), `cd235cc` (SDK 1.11.4).
+
+| File | What to change |
+|------|----------------|
+| `pyproject.toml` | `openhands-sdk`, `openhands-agent-server`, `openhands-tools` in **two** sections: the `dependencies` array (PEP 508) **and** `[tool.poetry.dependencies]` |
+| `openhands/app_server/sandbox/sandbox_spec_service.py` | `AGENT_SERVER_IMAGE` constant — set to `ghcr.io/openhands/agent-server:<version>-python` |
+
+Then regenerate lock files:
+```bash
+poetry lock && uv lock && cd enterprise && poetry lock && cd ..
+```
+
+## Docker Image Locations — All Hardcoded References
+
+For the complete inventory of every file containing a hardcoded Docker image tag or repository, see `references/docker-image-locations.md`. Key files that must stay in sync during an SDK bump:
+
+| File | Image reference | Updated during SDK bump? |
+|------|----------------|:------------------------:|
+| `openhands/app_server/sandbox/sandbox_spec_service.py` | `AGENT_SERVER_IMAGE = 'ghcr.io/openhands/agent-server:<tag>-python'` | ✅ Yes |
+| `docker-compose.yml` | `AGENT_SERVER_IMAGE_TAG` default | ✅ Should be |
+| `containers/dev/compose.yml` | `AGENT_SERVER_IMAGE_REPOSITORY` + `_TAG` defaults | ✅ Should be |
+
+> **CI enforcement:** `.github/workflows/check-version-consistency.yml` validates version consistency and compose file image references on every PR and push to main.
+
+### ⚠️ Docker Image Tag Gotcha (merge-commit SHA)
+
+The SDK CI in `software-agent-sdk` repo tags Docker images with the **GitHub Actions merge-commit SHA**, NOT the PR head-commit SHA. When pinning to an SDK PR branch:
+
+1. Check the SDK PR description for the actual image tag (look for the `AGENT_SERVER_IMAGES` section)
+2. Or query the CI logs: the "Consolidate Build Information" job prints `"short_sha": "<tag>"`
+3. The merge-commit SHA differs from the head SHA shown in the PR
+
+For released SDK versions, images use a version tag (e.g., `1.12.0-python`) — no merge-commit ambiguity.
+
+## Cutting a Release — 3 Files
+
+A release commit updates the version string across 3 files. Gold-standard examples: 1.3.0 (`d063c8c`), 1.4.0 (`495f48b`).
+
+| File | What to change |
+|------|----------------|
+| `pyproject.toml` | `version = "X.Y.Z"` under `[tool.poetry]` |
+| `frontend/package.json` | `"version": "X.Y.Z"` |
+| `frontend/package-lock.json` | `"version": "X.Y.Z"` in **two** places (root object and `packages[""]`) |
+
+> **Note:** `openhands/version.py` reads the version from `pyproject.toml` at runtime — no manual edit needed there.
+
+### Compose Files (2 files)
+
+Both compose files should use `ghcr.io/openhands/agent-server` with the current SDK version tag.
+
+| File | What to verify |
+|------|----------------|
+| `docker-compose.yml` | `AGENT_SERVER_IMAGE_REPOSITORY` defaults to agent-server, `AGENT_SERVER_IMAGE_TAG` is current |
+| `containers/dev/compose.yml` | Same — must use agent-server, not runtime |
+
+### Release Workflow
+
+#### Step 1: Verify the SDK bump has landed
+
+```bash
+grep -n "openhands-sdk\|openhands-agent-server\|openhands-tools" pyproject.toml
+grep -n "AGENT_SERVER_IMAGE" openhands/app_server/sandbox/sandbox_spec_service.py
+grep "AGENT_SERVER_IMAGE_TAG" docker-compose.yml containers/dev/compose.yml
+```
+
+#### Step 2: Bump version numbers
+
+```bash
+# Edit pyproject.toml, frontend/package.json, frontend/package-lock.json
+git add pyproject.toml frontend/package.json frontend/package-lock.json
+git commit -m "Release X.Y.Z"
+git tag X.Y.Z
+```
+
+Create a `saas-rel-X.Y.Z` branch from the tagged commit for the SaaS deployment pipeline.
+
+#### Step 3: CI builds Docker images automatically
+
+The `ghcr-build.yml` workflow triggers on tag pushes and produces:
+- `ghcr.io/openhands/openhands:X.Y.Z`, `X.Y`, `X`, `latest`
+- `ghcr.io/openhands/runtime:X.Y.Z-nikolaik`, `X.Y-nikolaik`
+
+The tagging logic lives in `containers/build.sh` — when `GITHUB_REF_NAME` matches a semver pattern (`^[0-9]+\.[0-9]+\.[0-9]+$`), it auto-generates major, major.minor, and `latest` tags.
+
+## Development: Pin SDK to an Unreleased Commit
+
+For detailed examples of all pinning formats (commit, branch, uv-only), see `references/sdk-pinning-examples.md`.
+
+### Files to change (3 manual + 3 lock files)
+
+| File | What to change |
+|------|----------------|
+| `pyproject.toml` | Pin all 3 SDK packages in **both** `dependencies` and `[tool.poetry.dependencies]` |
+| `openhands/app_server/sandbox/sandbox_spec_service.py` | `AGENT_SERVER_IMAGE` — use the merge-commit SHA tag, NOT the head-commit SHA |
+| `docker-compose.yml` | `AGENT_SERVER_IMAGE_TAG` default (for local development) |
+| `poetry.lock` | Auto-regenerated via `poetry lock` |
+| `uv.lock` | Auto-regenerated via `uv lock` |
+| `enterprise/poetry.lock` | Auto-regenerated via `cd enterprise && poetry lock` |
+
+### CI guard
+
+The `check-package-versions.yml` workflow blocks merging to `main` if `[tool.poetry.dependencies]` contains any `rev` fields. This ensures unreleased SDK pins do not accidentally ship in a release.
--- a/.agents/skills/update-sdk/references/docker-image-locations.md
+++ b/.agents/skills/update-sdk/references/docker-image-locations.md
@@ -0,0 +1,84 @@
+# Docker Image Locations — Complete Inventory
+
+Every file in the OpenHands repository containing a hardcoded Docker image tag, repository, or version-pinned image reference. Organized by update cadence.
+
+## Updated During SDK Bump (must change)
+
+These files contain image tags that **must** be updated whenever the SDK version or pinned commit changes.
+
+### `openhands/app_server/sandbox/sandbox_spec_service.py`
+- **Line:** `AGENT_SERVER_IMAGE = 'ghcr.io/openhands/agent-server:<tag>-python'`
+- **Format:** `<sdk-version>-python` for releases (e.g., `1.12.0-python`), `<7-char-commit-hash>-python` for dev pins
+- **Source of truth** for which agent-server image the app server pulls at runtime
+- **⚠️ Gotcha:** When pinning to an SDK PR, the image tag is the **merge-commit SHA** from GitHub Actions, not the PR head-commit SHA. Check the SDK PR description or CI logs for the correct tag.
+
+### `docker-compose.yml`
+- **Lines:**
+  ```yaml
+  - AGENT_SERVER_IMAGE_REPOSITORY=${AGENT_SERVER_IMAGE_REPOSITORY:-ghcr.io/openhands/agent-server}
+  - AGENT_SERVER_IMAGE_TAG=${AGENT_SERVER_IMAGE_TAG:-<tag>-python}
+  ```
+- Used by `docker compose up` for local development
+
+### `containers/dev/compose.yml`
+- **Lines:**
+  ```yaml
+  - AGENT_SERVER_IMAGE_REPOSITORY=${AGENT_SERVER_IMAGE_REPOSITORY:-ghcr.io/openhands/agent-server}
+  - AGENT_SERVER_IMAGE_TAG=${AGENT_SERVER_IMAGE_TAG:-<tag>-python}
+  ```
+- Used by the dev container setup
+- **Known issue:** On main as of 1.4.0, this file still points to `ghcr.io/openhands/runtime` instead of `agent-server`, and the tag is `1.2-nikolaik` (stale from the V0 era). The `check-version-consistency.yml` CI workflow catches this.
+
+## Updated During Release Commit (version string only)
+
+### `pyproject.toml`
+- **Line:** `version = "X.Y.Z"` under `[tool.poetry]`
+- The Python version is derived from this at runtime via `openhands/version.py`
+
+### `frontend/package.json`
+- **Line:** `"version": "X.Y.Z"`
+
+### `frontend/package-lock.json`
+- **Two places:** root `"version": "X.Y.Z"` and `packages[""].version`
+
+## Dynamic References (auto-derived, no manual update)
+
+### `openhands/version.py`
+- Reads version from `pyproject.toml` at runtime → `openhands.__version__`
+
+### `openhands/resolver/issue_resolver.py`
+- Builds `ghcr.io/openhands/runtime:{openhands.__version__}-nikolaik` dynamically
+
+### `openhands/runtime/utils/runtime_build.py`
+- Base repo URL `ghcr.io/openhands/runtime` is a constant; version comes from elsewhere
+
+### `.github/scripts/update_pr_description.sh`
+- Uses `${SHORT_SHA}` variable at CI runtime, not hardcoded
+
+### `enterprise/Dockerfile`
+- `ARG BASE="ghcr.io/openhands/openhands"` — base image, version supplied at build time
+
+## V0 Legacy Files (separate update cadence)
+
+These reference the V0 runtime image (`ghcr.io/openhands/runtime:X.Y-nikolaik`) for local Docker/Kubernetes paths. They are **not** updated as part of a V1 release but may be updated independently.
+
+### `Development.md`
+- `export SANDBOX_RUNTIME_CONTAINER_IMAGE=ghcr.io/openhands/runtime:X.Y-nikolaik`
+
+### `openhands/runtime/impl/kubernetes/README.md`
+- `runtime_container_image = "docker.openhands.dev/openhands/runtime:X.Y-nikolaik"`
+
+### `enterprise/enterprise_local/README.md`
+- Uses `ghcr.io/openhands/runtime:main-nikolaik` (points to `main`, not versioned)
+
+### `third_party/runtime/impl/daytona/README.md`
+- Uses `${OPENHANDS_VERSION}` variable, not hardcoded
+
+## Image Registries
+
+| Registry | Usage |
+|----------|-------|
+| `ghcr.io/openhands/agent-server` | V1 agent-server (sandbox) — built by SDK repo CI |
+| `ghcr.io/openhands/openhands` | Main app image — built by `ghcr-build.yml` |
+| `ghcr.io/openhands/runtime` | V0 runtime sandbox — built by `ghcr-build.yml` |
+| `docker.openhands.dev/openhands/*` | Mirror/CDN for the above images |
--- a/.agents/skills/update-sdk/references/sdk-pinning-examples.md
+++ b/.agents/skills/update-sdk/references/sdk-pinning-examples.md
@@ -0,0 +1,103 @@
+# SDK Pinning Examples
+
+Examples from real commits showing how to pin SDK packages to unreleased commits, branches, or released versions.
+
+## Pin to a Specific Commit
+
+Example from commit `169fb76` (pinning all 3 packages to SDK commit `100e9af`):
+
+### `dependencies` array (PEP 508 format)
+
+```toml
+"openhands-agent-server @ git+https://github.com/OpenHands/software-agent-sdk.git@100e9af#subdirectory=openhands-agent-server",
+"openhands-sdk @ git+https://github.com/OpenHands/software-agent-sdk.git@100e9af#subdirectory=openhands-sdk",
+"openhands-tools @ git+https://github.com/OpenHands/software-agent-sdk.git@100e9af#subdirectory=openhands-tools",
+```
+
+### `[tool.poetry.dependencies]` (Poetry format)
+
+```toml
+openhands-sdk = { git = "https://github.com/OpenHands/software-agent-sdk.git", rev = "100e9af", subdirectory = "openhands-sdk" }
+openhands-agent-server = { git = "https://github.com/OpenHands/software-agent-sdk.git", rev = "100e9af", subdirectory = "openhands-agent-server" }
+openhands-tools = { git = "https://github.com/OpenHands/software-agent-sdk.git", rev = "100e9af", subdirectory = "openhands-tools" }
+```
+
+### `openhands/app_server/sandbox/sandbox_spec_service.py`
+
+```python
+AGENT_SERVER_IMAGE = 'ghcr.io/openhands/agent-server:<merge-commit-sha>-python'
+```
+
+**⚠️ Important:** The image tag is the **merge-commit SHA** from the SDK CI, not the commit hash used in `pyproject.toml`. Look up the correct tag from the SDK PR description or CI logs.
+
+## Pin to a Branch
+
+Example from commit `430ee1c` (pinning to branch `openhands/issue-2228-sdk-settings-schema`):
+
+### `[tool.poetry.dependencies]`
+
+```toml
+openhands-sdk = { git = "https://github.com/OpenHands/software-agent-sdk.git", branch = "openhands/issue-2228-sdk-settings-schema", subdirectory = "openhands-sdk" }
+openhands-agent-server = { git = "https://github.com/OpenHands/software-agent-sdk.git", branch = "openhands/issue-2228-sdk-settings-schema", subdirectory = "openhands-agent-server" }
+openhands-tools = { git = "https://github.com/OpenHands/software-agent-sdk.git", branch = "openhands/issue-2228-sdk-settings-schema", subdirectory = "openhands-tools" }
+```
+
+## Using `[tool.uv.sources]` Override
+
+When only `uv` needs the override (keep PyPI versions in the main arrays), add a `[tool.uv.sources]` section. Example from commit `1daca49`:
+
+```toml
+[tool.uv.sources]
+openhands-sdk = { git = "https://github.com/OpenHands/software-agent-sdk.git", subdirectory = "openhands-sdk", rev = "4170cca" }
+openhands-agent-server = { git = "https://github.com/OpenHands/software-agent-sdk.git", subdirectory = "openhands-agent-server", rev = "4170cca" }
+openhands-tools = { git = "https://github.com/OpenHands/software-agent-sdk.git", subdirectory = "openhands-tools", rev = "4170cca" }
+```
+
+## Released PyPI Version (standard release)
+
+Example from commit `929dcc3` (SDK 1.11.5):
+
+### `dependencies` array
+
+```toml
+"openhands-agent-server==1.11.5",
+"openhands-sdk==1.11.5",
+"openhands-tools==1.11.5",
+```
+
+### `[tool.poetry.dependencies]`
+
+```toml
+openhands-sdk = "1.11.5"
+openhands-agent-server = "1.11.5"
+openhands-tools = "1.11.5"
+```
+
+### `openhands/app_server/sandbox/sandbox_spec_service.py`
+
+For released versions, the image tag uses the version number:
+
+```python
+AGENT_SERVER_IMAGE = 'ghcr.io/openhands/agent-server:1.11.5-python'
+```
+
+However, **some releases use a commit-hash tag** even for the released version. Check which tag format exists on GHCR. Example from `929dcc3`:
+
+```python
+AGENT_SERVER_IMAGE = 'ghcr.io/openhands/agent-server:010e847-python'
+```
+
+## Regenerate Lock Files
+
+After any change to `pyproject.toml`, always regenerate:
+
+```bash
+poetry lock
+uv lock
+cd enterprise && poetry lock && cd ..
+```
+
+## CI Guards
+
+- **`check-package-versions.yml`**: Blocks merge to `main` if `[tool.poetry.dependencies]` contains `rev` fields (prevents shipping unreleased SDK pins)
+- **`check-version-consistency.yml`**: Validates version strings match across `pyproject.toml`, `package.json`, `package-lock.json`, and verifies compose files use `agent-server` images
--- a/.github/workflows/check-version-consistency.yml
+++ b/.github/workflows/check-version-consistency.yml
@@ -0,0 +1,122 @@
+name: Check Version Consistency
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  check-version-consistency:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: "3.12"
+
+      - name: Check version and Docker image tag consistency
+        run: |
+          python - <<'PY'
+          import json
+          import re
+          import sys
+          import tomllib
+
+          errors = []
+          warnings = []
+
+          # ── 1. Extract the canonical version from pyproject.toml ──────────
+          with open("pyproject.toml", "rb") as f:
+              pyproject = tomllib.load(f)
+          version = pyproject["tool"]["poetry"]["version"]
+          major_minor = ".".join(version.split(".")[:2])
+          print(f"📦 pyproject.toml version: {version} (major.minor: {major_minor})")
+
+          # ── 2. Check frontend/package.json ────────────────────────────────
+          with open("frontend/package.json") as f:
+              pkg = json.load(f)
+          if pkg["version"] != version:
+              errors.append(
+                  f"frontend/package.json version is '{pkg['version']}', expected '{version}'"
+              )
+          else:
+              print(f"  ✔ frontend/package.json: {pkg['version']}")
+
+          # ── 3. Check frontend/package-lock.json (2 places) ───────────────
+          with open("frontend/package-lock.json") as f:
+              lock = json.load(f)
+          for key, val in [
+              ("root.version", lock.get("version")),
+              ('packages[""].version', lock.get("packages", {}).get("", {}).get("version")),
+          ]:
+              if val != version:
+                  errors.append(
+                      f"frontend/package-lock.json {key} is '{val}', expected '{version}'"
+                  )
+              else:
+                  print(f"  ✔ frontend/package-lock.json {key}: {val}")
+
+          # ── 4. Check compose files use agent-server images ─────────────────
+          # Both compose files should use ghcr.io/.../agent-server (not runtime).
+          # Agent-server tags use SDK version (e.g. "1.12.0-python") or commit
+          # hashes (e.g. "31536c8-python") — both are acceptable.
+          repo_pattern = re.compile(r"AGENT_SERVER_IMAGE_REPOSITORY[^}]*:-([^}]+)")
+          tag_pattern = re.compile(r"AGENT_SERVER_IMAGE_TAG:-([^}]+)")
+
+          for filepath in ["docker-compose.yml", "containers/dev/compose.yml"]:
+              try:
+                  with open(filepath) as f:
+                      content = f.read()
+              except FileNotFoundError:
+                  warnings.append(f"{filepath}: file not found")
+                  continue
+
+              repos = repo_pattern.findall(content)
+              tags = tag_pattern.findall(content)
+
+              if not repos:
+                  warnings.append(f"{filepath}: no AGENT_SERVER_IMAGE_REPOSITORY default found")
+              else:
+                  repo = repos[0]
+                  if "agent-server" not in repo:
+                      errors.append(
+                          f"{filepath}: AGENT_SERVER_IMAGE_REPOSITORY defaults to '{repo}', "
+                          f"expected an agent-server image (not runtime)"
+                      )
+                  else:
+                      print(f"  ✔ {filepath} image repository: {repo}")
+
+              if not tags:
+                  warnings.append(f"{filepath}: no AGENT_SERVER_IMAGE_TAG default found")
+              else:
+                  tag = tags[0]
+                  if not tag:
+                      errors.append(f"{filepath}: AGENT_SERVER_IMAGE_TAG default is empty")
+                  else:
+                      print(f"  ✔ {filepath} image tag: {tag}")
+
+          # ── 5. Report ─────────────────────────────────────────────────────
+          print()
+          if warnings:
+              print("⚠ Warnings:")
+              for w in warnings:
+                  print(f"  {w}")
+              print()
+
+          if errors:
+              print("❌ FAILED: Version inconsistencies found:\n")
+              for e in errors:
+                  print(f"  ✖ {e}")
+              print(
+                  "\nAll version numbers and Docker image tags must be consistent."
+                  "\nSee .agents/skills/update-sdk/SKILL.md for the full checklist."
+              )
+              sys.exit(1)
+          else:
+              print("✅ All version numbers and Docker image tags are consistent.")
+          PY
--- a/containers/dev/compose.yml
+++ b/containers/dev/compose.yml
@@ -12,8 +12,8 @@ services:
      - SANDBOX_API_HOSTNAME=host.docker.internal
      - DOCKER_HOST_ADDR=host.docker.internal
      #
-      - AGENT_SERVER_IMAGE_REPOSITORY=${AGENT_SERVER_IMAGE_REPOSITORY:-ghcr.io/openhands/runtime}
-      - AGENT_SERVER_IMAGE_TAG=${AGENT_SERVER_IMAGE_TAG:-1.2-nikolaik}
+      - AGENT_SERVER_IMAGE_REPOSITORY=${AGENT_SERVER_IMAGE_REPOSITORY:-ghcr.io/openhands/agent-server}
+      - AGENT_SERVER_IMAGE_TAG=${AGENT_SERVER_IMAGE_TAG:-1.12.0-python}
      - SANDBOX_USER_ID=${SANDBOX_USER_ID:-1234}
      - WORKSPACE_MOUNT_PATH=${WORKSPACE_BASE:-$PWD/workspace}
    ports:
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -8,7 +8,7 @@ services:
    container_name: openhands-app-${DATE:-}
    environment:
      - AGENT_SERVER_IMAGE_REPOSITORY=${AGENT_SERVER_IMAGE_REPOSITORY:-ghcr.io/openhands/agent-server}
-      - AGENT_SERVER_IMAGE_TAG=${AGENT_SERVER_IMAGE_TAG:-31536c8-python}
+      - AGENT_SERVER_IMAGE_TAG=${AGENT_SERVER_IMAGE_TAG:-1.12.0-python}
      #- SANDBOX_USER_ID=${SANDBOX_USER_ID:-1234} # enable this only if you want a specific non-root sandbox user but you will have to manually adjust permissions of ~/.openhands for this user
      - WORKSPACE_MOUNT_PATH=${WORKSPACE_BASE:-$PWD/workspace}
    ports: