mirror of https://github.com/OpenHands/OpenHands.git synced 2025-12-26 13:52:43 +08:00

History

(refactor) Make Runtime class synchronous (#3661 )

* change runtime to be synchronous

* fix test runtime with the new interface

* fix arg

* fix eval

* fix missing config attribute

* fix plugins

* fix on_event by revert it back to async

* update upload_file endpoint

* fix argument to upload file

* remove unncessary async for eval;
fix evaluation run in parallel

* use asyncio to run controller for eval

* revert file upload

* truncate eval test result output

2024-08-30 01:37:03 +00:00

scripts

[Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230 )

2024-08-06 17:21:45 +00:00

__init__.py

Add AgentBench. (#2012 )

2024-06-01 07:58:14 +00:00

helper.py

Rename OpenDevin to OpenHands (#3472 )

2024-08-20 00:44:54 +08:00

README.md

Rename OpenDevin to OpenHands (#3472 )

2024-08-20 00:44:54 +08:00

run_infer.py

(refactor) Make Runtime class synchronous (#3661 )

2024-08-30 01:37:03 +00:00

README.md

AgentBench Evaluation

This folder contains evaluation harness for evaluating agents on the AgentBench: Evaluating LLMs as Agents. We currently only support running on the osbench subset.

Setup Environment and LLM Configuration

Please follow instruction here to setup your local development environment and LLM.

Start the evaluation

./evaluation/agent_bench/scripts/run_infer.sh [model_config] [git-version] [agent] [eval_limit]

model_config, e.g. eval_gpt4_1106_preview, is the config group name for your LLM settings, as defined in your config.toml.
git-version, e.g. HEAD, is the git commit hash of the OpenHands version you would like to evaluate. It could also be a release tag like 0.6.2.
agent, e.g. CodeActAgent, is the name of the agent for benchmarks, defaulting to CodeActAgent.
eval_limit, e.g. 10, limits the evaluation to the first eval_limit instances. By default, the script evaluates the entire SWE-bench_Lite test set (300 issues). Note: in order to use eval_limit, you must also set agent.

Following is the basic command to start the evaluation.

You can update the arguments in the script evaluation/agent_bench/scripts/run_infer.sh, such as --max-iterations, --eval-num-workers and so on.

--agent-cls, the agent to use. For example, CodeActAgent.
--llm-config: the LLM configuration to use. For example, eval_gpt4_1106_preview.
--max-iterations: the number of iterations to run the evaluation. For example, 30.
--eval-num-workers: the number of workers to use for evaluation. For example, 5.
--eval-n-limit: the number of examples to evaluate. For example, 100.

./evaluation/agent_bench/scripts/run_infer.sh eval_gpt35_turbo HEAD CodeActAgent 1