github/OpenHands
1
0
Fork 0
mirror of https://github.com/OpenHands/OpenHands.git synced 2025-12-26 05:48:36 +08:00
Code Issues Packages Projects Releases Wiki Activity

Fix issue #5739: [Bug]: Move ./evaluation/swe_bench/scripts/cleanup_remote_runtime.sh to general eval utils (#5740)

Browse Source
This commit is contained in:
OpenHands 2024-12-25 17:17:06 -05:00 committed by GitHub
parent 3dd39c59b6
commit bfb191b5c7
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 3 additions and 36 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
evaluation/benchmarks/commit0_bench/README.md
View File

@ -73,7 +73,7 @@ ALLHANDS_API_KEY="YOUR-API-KEY" RUNTIME=remote SANDBOX_REMOTE_RUNTIME_API_URL="h
To clean-up all existing runtime you've already started, run:
```bash
ALLHANDS_API_KEY="YOUR-API-KEY" ./evaluation/benchmarks/commit0_bench/scripts/cleanup_remote_runtime.sh
ALLHANDS_API_KEY="YOUR-API-KEY" ./evaluation/utils/scripts/cleanup_remote_runtime.sh
```
### Specify a subset of tasks to run infer

4
evaluation/benchmarks/swe_bench/README.md
View File

@ -80,7 +80,7 @@ ALLHANDS_API_KEY="YOUR-API-KEY" RUNTIME=remote SANDBOX_REMOTE_RUNTIME_API_URL="h
To clean-up all existing runtime you've already started, run:
```bash
ALLHANDS_API_KEY="YOUR-API-KEY" ./evaluation/benchmarks/swe_bench/scripts/cleanup_remote_runtime.sh
ALLHANDS_API_KEY="YOUR-API-KEY" ./evaluation/utils/scripts/cleanup_remote_runtime.sh
```
### Specify a subset of tasks to run infer
@ -178,7 +178,7 @@ evaluation/benchmarks/swe_bench/scripts/eval_infer_remote.sh evaluation/evaluati
To clean-up all existing runtimes that you've already started, run:
```bash
ALLHANDS_API_KEY="YOUR-API-KEY" ./evaluation/benchmarks/swe_bench/scripts/cleanup_remote_runtime.sh
ALLHANDS_API_KEY="YOUR-API-KEY" ./evaluation/utils/scripts/cleanup_remote_runtime.sh
```
## Visualize Results

33
evaluation/benchmarks/swe_bench/scripts/cleanup_remote_runtime.sh
View File

@ -1,33 +0,0 @@
#!/bin/bash
# API base URL
BASE_URL="https://runtime.eval.all-hands.dev"
# Get the list of runtimes
response=$(curl --silent --location --request GET "${BASE_URL}/list" \
--header "X-API-Key: ${ALLHANDS_API_KEY}")
n_runtimes=$(echo $response | jq -r '.total')
echo "Found ${n_runtimes} runtimes. Stopping them..."
runtime_ids=$(echo $response | jq -r '.runtimes | .[].runtime_id')
# Function to stop a single runtime
stop_runtime() {
local runtime_id=$1
local counter=$2
echo "Stopping runtime ${counter}/${n_runtimes}: ${runtime_id}"
curl --silent --location --request POST "${BASE_URL}/stop" \
--header "X-API-Key: ${ALLHANDS_API_KEY}" \
--header "Content-Type: application/json" \
--data-raw "{\"runtime_id\": \"${runtime_id}\"}"
echo
}
export -f stop_runtime
export BASE_URL ALLHANDS_API_KEY n_runtimes
# Use GNU Parallel to stop runtimes in parallel
echo "$runtime_ids" | parallel -j 16 --progress stop_runtime {} {#}
echo "All runtimes have been stopped."

0
evaluation/benchmarks/commit0_bench/scripts/cleanup_remote_runtime.sh → evaluation/utils/scripts/cleanup_remote_runtime.sh
View File