Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
Co-authored-by: Engel Nyst <enyst@users.noreply.github.com>
Co-authored-by: Engel Nyst <engel.nyst@gmail.com>
* added optional START_ID env flag to resume from that instance id
* prepare_dataset: fix comparisons by using instance id's as int
* aider bench complete_runtime: close runtime to close container
* added matrix display of instance id for logging
* fix typo in summarize_results.py saying summarise_results
* changed start_id to skip_num to skip rows from dataset (start_id wasn't supportable)
* doc changes about huggingface spaces to temporarily point back to OD
* move multi-line bash tests to test_runtime;
support multi-line bash for esruntime;
* add testcase to handle PS2 prompt
* use bashlex for bash parsing to handle multi-line commands;
add testcases for multi-line commands
* revert ghcr runtime change
* Apply stash
* fix run as other user;
make test async;
* fix test runtime for run as od
* add run-as-devin to all the runtime tests
* handle the case when username is root
* move all run-as-devin tests from sandbox;
only tests a few cases on different user to save time;
* move over multi-line echo related tests to test_runtime
* fix user-specific jupyter by fixing the pypoetry virtualenv folder
* make plugin's init async;
chdir at initialization of jupyter plugin;
move ipy simple testcase to test runtime;
* support agentskills import in
move tests for jupyter pwd tests;
overload `add_env_vars` for EventStreamRuntime to update env var also in Jupyter;
make agentskills read env var lazily, in case env var is updated;
* fix ServerRuntime agentskills issue
* move agnostic image test to test_runtime
* merge runtime tests in CI
* fix enable auto lint as env var
* update warning message
* update warning message
* test for different container images
* change parsing output as debug
* add exception handling for update_pwd_decorator
* fix unit test indentation
* add plugins as default input to Runtime class;
remove init_sandbox_plugins;
implement add_env_var (include jupyter) in the base class;
* fix server runtime auto lint
* Revert "add exception handling for update_pwd_decorator"
This reverts commit 2b668b1506e02145cb8f87e321aad62febca3d50.
* tries to print debugging info for agentskills
* explictly setting uid (try fix permission issue)
* Revert "tries to print debugging info for agentskills"
This reverts commit 8be4c86756f0e3fc62957b327ba2ac4999c419de.
* set sandbox user id during testing to hopefully fix the permission issue
* add browser tools for server runtime
* try to debug for old pwd
* update debug cmd
* only test agnostic runtime when TEST_RUNTIME is Server
* fix temp dir mkdir
* load TEST_RUNTIME at the beginning
* remove ipython tests
* only log to file when DEBUG
* default logging to project root
* temporarily remove log to file
* fix LLM logger dir
* fix logger
* make set pwd an optional aux action
* fix prev pwd
* fix infinity recursion
* simplify
* do not import the whole od library to avoid logger folder by jupyter
* fix browsing
* increase timeout
* attempt to fix agentskills yet again
* clean up in testcases, since CI maybe run as non-root
* add _cause attribute for event.id
* remove parent
* add a bunch of debugging statement again for CI :(
* fix temp_dir fixture
* change all temp dir to follow pytest's tmp_path_factory
* remove extra bracket
* clean up error printing a bit
* jupyter chdir to self.config.workspace_mount_path_in_sandbox on initialization
* jupyter chdir to self.config.workspace_mount_path_in_sandbox on initialization
* add typing for tmp dir fixture
* clear the directory before running the test to avoid weird CI temp dir
* remove agnostic test case for server runtime
* Revert "remove agnostic test case for server runtime"
This reverts commit 30e2181c3fc1410e69596c2dcd06be01f1d016b3.
* disable agnostic tests in CI
* fix test
* make sure plugin arg is not passed when no plugin is specified;
remove redundant on_event function;
* move mock prompt
* rename runtime
* remove extra logging
* refactor run_controller's interface;
support multiple runtime for integration test;
filter out hostname for prompt
* uncomment other tests
* pass the right runtime to controller
* log runtime when start
* uncomment tests
* improve symbol filters
* add intergration test prompts that seemd ok
* add integration test workflow
* add python3 to default ubuntu image
* symlink python and fix permission to jupyter pip
* add retry for jupyter execute server
* fix jupyter pip install;
add post-process for jupyter pip install;
simplify init by add agent_skills path to PYTHONPATH;
add testcase to tests jupyter pip install;
* fix bug
* use ubuntu:22.04 for eventstream integration tests
* add todo
* update testcase
* remove redundant code
* fix unit test
* reduce dependency for runtime
* try making llama-index an optional dependency that's not installed by default
* remove pip install since it seemd not needed
* log ipython execution;
await write message since it returns a future
* update ipy testcase
* do not install llama-index in CI
* do not install llama-index in the app docker as well
* set sandbox container image in the integration test script
* log plugins & env var for runtime
* update conftest for sha256
* add git
* remove all non-alphanumeric chalracters
* add working ipy module tests!
* default to use host network
* remove is_async from browser to make thing a little more reliable;
retry loading browser when error;
* add sleep to wait a bit for http server
* kill http server before regenerate browsing tests
* fix browsing
* only set sandbox container image if undefined
* skip empty config value
* update evaluation to use the latest run_controller
* revert logger in execute_server to be compatible with server runtime
* revert logging level to fix jupyter
* set logger level
* revert the logging
* chmod for workspace to fix permission
* support getting timeout from action
* update test for server runtime
* try to fix file permission
* fix test_cmd_run_action_serialization_deserialization test (added timeout)
* poetry: pip 24.2, torch 2.2.2
* revert adding pip to pyproject.toml
* add build to dependencies in pyproject.toml
* forgot poetry lock --no-update
* fix a DelegatorAgent prompt_002.log (timeout)
* fix a DelegatorAgent prompt_003.log (timeout)
* couple more timeout attribs in prompt files
* some more prompt files
* prompts galore
* add clarification comment for timeout
* default timeout to config
* add assert
* update integraton tests for eventstream
* update integration tests
* fix timeout for action<->dict
* remove redundant on_event
* default to use instance image
* update run_controller interface
* add logging for copy
* refactor swe_bench for the new design
* fix action execution timeout
* updatelock
* remove build sandbox locally
* fix runtime
* use plain for-loop for single process
* remove extra print
* get swebench inference working
* print whole `test_result` dict
* got swebench patch post-process working
* update swe-bench evaluation readme
* refactor using shared reset_logger function
* move messy swebench prompt to a different file
* support the ability to specify whether to keep prompt
* support the ability to specify whether to keep prompt
* fix dockerfile
* fix import and remove unnecessary strip logic
* fix action serialization
* get agentbench running
* remove extra ls for agent bench
* fix agentbench metric
* factor out common documentation for eval
* update biocoder doc
* remove swe_env_box since it is no longer needed
* get biocoder working
* add func timeout for bird
* fix jupyter pwd with ~ as user name
* fix jupyter pwd with ~ as user name
* get bird working
* get browsing evaluation working
* make eda runnable
* fix id column
* fix eda run_infer
* unify eval output using a structured format;
make swebench coompatible with that format;
update client source code for every swebench run;
do not inject testcmd for swebench
* standardize existing benchs for the new eval output
* set update source code = true
* get gaia standardized
* fix gaia
* gorilla refactored but stuck at language.so to test
* refactor and make gpqa work
* refactor humanevalfix and get it working
* refactor logic reasoning and get it working
* refactor browser env so it works with eventstream runtime for eval
* add initial version of miniwob refactor
* fix browsergym environment
* get miniwob working!!
* allowing injecting additional dependency to OD runtime docker image
* allowing injecting additional dependency to OD runtime docker image
* support logic reasoning with pre-injected dependency
* get mint working
* update runtime build
* fix mint docker
* add test for keep_prompt;
add missing await close for some tests
* update integration tests for eventstream runtime
* fix integration tests for server runtime
* refactor ml bench and toolqa
* refactor webarena
* fix default factory
* Update run_infer.py
* add APIError to retry
* increase timeout for swebench
* make sure to hide api key when dump eval output
* update the behavior of put source code to put files instead of tarball
* add dishash to dependency
* sendintr when timeout
* fix dockerfile copy
* reduce timeout
* use dirhash to avoid repeat building for update source
* fix runtime_build testcase
* add dir_hash to docker build pipeline
* revert api error
* update poetry lock
* add retries for swebench run infer
* fix git patch
* update poetry lock
* adjust config order
* fix mount volumns
* enforce all eval to use "instance_id"
* remove file store from runtime
* make file_store public inside eventstream
* move the runtime logic inside `main` out
* support using async function for process_instance_fn
* refactor run_infer with the create_time
* fix file store
* Update evaluation/toolqa/utils.py
Co-authored-by: Graham Neubig <neubig@gmail.com>
* fix typo
---------
Co-authored-by: tobitege <tobitege@gmx.de>
Co-authored-by: super-dainiu <78588128+super-dainiu@users.noreply.github.com>
Co-authored-by: Graham Neubig <neubig@gmail.com>
* setup boilerplate and README
* setup test script and load dataset
* add temp intg that works
* refactor code
* add solution evaluation through 'fake_user_response_fn'
* finish integrating MATH subset
* Update evaluation/mint/run_infer.py
* Update evaluation/mint/run_infer.sh
* Update opendevin/core/main.py
* remove redudant templates, add eval_note, update README
* use <execute_ipython> tag instead of <execute>
* hardcode AGENT option for run_infer.sh
* Update evaluation/mint/task.py
Co-authored-by: Yufan Song <33971064+yufansong@users.noreply.github.com>
* fix: bug no message returned when task's success
* change message to make the agent exit
* import bash abstractmethod
* install all required packages inside sandbox before the agent runs, adjust prompt
* add subset eval folder separation and test for gsm8k
* fix bug in Reasoning task result check, add requirements.txt
* Fix syntax error in evaluation/mint/run_infer.py
* update README, add default values for `SUBSET` and `EVAL_LIMIT`
---------
Co-authored-by: Yufan Song <33971064+yufansong@users.noreply.github.com>
Co-authored-by: yufansong <yufan@risingwave-labs.com>
Co-authored-by: Boxuan Li <liboxuan@connect.hku.hk>
* adding draft evaluation code for EDA, using chatgpt as the temporal agent for now
* Update README.md
* Delete frontend/package.json
* reverse the irrelevant changes
* reverse package.json
* use chatgpt as the codeactagent
* integrate with opendevin
* Update evaluation/EDA/README.md
* Update evaluation/EDA/README.md
* Use poetry to manage packages
* integrate with opendevin
* minor update
* minor update
* update poetry
* update README
* clean-up infer scripts
* add run_infer script and improve readme
* log final success and final message & ground truth
---------
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xingyao Wang <xingyao6@illinois.edu>
Co-authored-by: yufansong <yufan@risingwave-labs.com>
Co-authored-by: Boxuan Li <liboxuan@connect.hku.hk>
* Preliminary HumanEvalFix integration
* Clean paths
* fix: set workspace path correctly for config
fix: task in that contains /
* add missing run_infer.sh
* update run_infer w/o hard coded agent
* fix typo
* change `instance_id` to `task_id`
* add the warning and env var setting to run_infer.sh
* reset back workspace mount at the end of each instance
* 10 max iter is probably enough for humanevalfix
* Remove unneeded section
Co-authored-by: Xingyao Wang <xingyao6@illinois.edu>
* Fix link
Co-authored-by: Yufan Song <33971064+yufansong@users.noreply.github.com>
* Use logger
Co-authored-by: Yufan Song <33971064+yufansong@users.noreply.github.com>
* Update run_infer.py
fix a bug:
ERROR:concurrent.futures:exception calling callback for <Future at 0x309cbc470 state=finished raised NameError>
concurrent.futures.process._RemoteTraceback:
* Update README.md
* Update README.md
* Update README.md
* Update README.md
added an example
* Update README.md
added: enable_auto_lint = true
* Update pyproject.toml
add: evaluate package
* Delete poetry.lock
update poetry.lock
* update poetry.lock
update poetry.lock
* Update README.md
* Update README.md
---------
Co-authored-by: Xingyao Wang <xingyao6@illinois.edu>
Co-authored-by: Yufan Song <33971064+yufansong@users.noreply.github.com>
Co-authored-by: Robert <871607149@qq.com>
* add draft dockerfile for build all
* add rsync for build
* add all-in-one docker
* update prepare scripts
* Update swe_env_box.py
* Add swe_entry.sh (buggy now)
* Parse the test command in swe_entry.sh
* Update README for instance eval in sandbox
* revert specialized config
* replace run_as_devin as an init arg
* set container & run_as_root via args
* update swe entry script
* update env
* remove mounting
* allow error after swe_entry
* update swe_env_box
* move file
* update gitignore
* get swe_env_box a working demo
* support faking user response & provide sandox ahead of time;
also return state for controller
* tweak main to support adding controller kwargs
* add module
* initialize plugin for provided sandbox
* add pip cache to plugin & fix jupyter kernel waiting
* better print Observation output
* add run infer scripts
* update readme
* add utility for getting diff patch
* use get_diff_patch in infer
* update readme
* support cost tracking for codeact
* add swe agent edit hack
* disable color in git diff
* fix git diff cmd
* fix state return
* support limit eval
* increase t imeout and export pip cache
* add eval limit config
* return state when hit turn limit
* save log to file; allow agent to give up
* run eval with max 50 turns
* add outputs to gitignore
* save swe_instance & instruction
* add uuid to swebench
* add streamlit dep
* fix save series
* fix the issue where session id might be duplicated
* allow setting temperature for llm (use 0 for eval)
* Get report from agent running log
* support evaluating task success right after inference.
* remove extra log
* comment out prompt for baseline
* add visualizer for eval
* use plaintext for instruction
* reduce timeout for all; only increase timeout for init
* reduce timeout for all; only increase timeout for init
* ignore sid for swe env
* close sandbox in each eval loop
* update visualizer instruction
* increase max chars
* add finish action to history too
* show test result in metrics
* add sidebars for visualizer
* also visualize swe_instance
* cleanup browser when agent controller finish runinng
* do not mount workspace for swe-eval to avoid accidentally overwrite files
* Revert "do not mount workspace for swe-eval to avoid accidentally overwrite files"
This reverts commit 8ef77390543e562e6f0a5a9992418014d8b3010c.
* Revert "Revert "do not mount workspace for swe-eval to avoid accidentally overwrite files""
This reverts commit 016cfbb9f0475f32bacbad5822996b4eaff24a5e.
* run jupyter command via copy to, instead of cp to mount
* only print mixin output when failed
* change ssh box logging
* add visualizer for pass rate
* add instance id to sandbox name
* only remove container we created
* use opendevin logger in main
* support multi-processing infer
* add back metadata, support keyboard interrupt
* remove container with startswith
* make pbar behave correctly
* update instruction w/ multi-processing
* show resolved rate by repo
* rename tmp dir name
* attempt to fix racing for copy to ssh_box
* fix script
* bump swe-bench-all version
* fix ipython with self-contained commands
* add jupyter demo to swe_env_box
* make resolved count two column
* increase height
* do not add glob to url params
* analyze obs length
* print instance id prior to removal handler
* add gold patch in visualizer
* fix interactive git by adding a git --no-pager as alias
* increase max_char to 10k to cover 98% of swe-bench obs cases
* allow parsing note
* prompt v2
* add iteration reminder
* adjust user response
* adjust order
* fix return eval
* fix typo
* add reminder before logging
* remove other resolve rate
* re adjust to new folder structure
* support adding eval note
* fix eval note path
* make sure first log of each instance is printed
* add eval note
* fix the display for visualizer
* tweak visualizer for better git patch reading
* exclude empty patch
* add retry mechanism for swe_env_box start
* fix ssh timeout issue
* add stat field for apply test patch success
* add visualization for fine-grained report
* attempt to support monologue agent by constraining it to single thread
* also log error msg when stopeed
* save error as well
* override WORKSPACE_MOUNT_PATH and WORKSPACE_BASE for monologue to work in mp
* add retry mechanism for sshbox
* remove retry for swe env box
* try to handle loop state stopped
* Add get report scripts
* Add script to convert agent output to swe-bench format
* Merge fine grained report for visualizer
* Update eval readme
* Update README.md
* Add CodeAct gpt4-1106 output and eval logs on swe-bench-lite
* Update the script to get model report
* Update get_model_report.sh
* Update get_agent_report.sh
* Update report merge script
* Add agent output conversion script
* Update swe_lite_env_setup.sh
* Add example swe-bench output files
* Update eval readme
* Remove redundant scripts
* set iteration count down to false by default
* fix: Issue where CodeAct agent was trying to log cost on local llm and throwing Undefined Model execption out of litellm (#1666)
* fix: Issue where CodeAct agent was trying to log cost on local llm and throwing Undefined Model execption out of litellm
* Review Feedback
* Missing None Check
* Review feedback and improved error handling
---------
Co-authored-by: Robert Brennan <accounts@rbren.io>
* fix prepare_swe_util scripts
* update builder images
* update setup script
* remove swe-bench build workflow
* update lock
* remove experiments since they are moved to hf
* remove visualizer (since it is moved to hf repo)
* simply jupyter execution via heredoc
* update ssh_box
* add initial docker readme
* add pkg-config as dependency
* add script for swe_bench all-in-one docker
* add rsync to builder
* rename var
* update commit
* update readme
* update lock
* support specify timeout for long running tasks
* fix path
* separate building of all deps and files
* support returning states at the end of controller
* remove return None
* support specify timeout for long running tasks
* add timeout for all existing sandbox impl
* fix swe_env_box for new codebase
* update llm config in config.py
* support pass sandbox in
* remove force set
* update eval script
* fix issue of overriding final state
* change default eval output to hf demo
* change default eval output to hf demo
* fix config
* only close it when it is NOT external sandbox
* add scripts
* tweak config
* only put in hostory when state has history attr
* fix agent controller on the case of run out interaction budget
* always assume state is always not none
* remove print of final state
* catch all exception when cannot compute completion cost
* Update README.md
* save source into json
* fix path
* update docker path
* return the final state on close
* merge AgentState with State
* fix integration test
* merge AgentState with State
* fix integration test
* add ChangeAgentStateAction to history in attempt to fix integration
* add back set agent state
* update tests
* update tests
* move scripts for setup
* update script and readme for infer
* do not reset logger when n processes == 1
* update eval_infer scripts and readme
* simplify readme
* copy over dir after eval
* copy over dir after eval
* directly return get state
* update lock
* fix output saving of infer
* replace print with logger
* update eval_infer script
* add back the missing .close
* increase timeout
* copy all swe_bench_format file
* attempt to fix output parsing
* log git commit id as metadata
* fix eval script
* update lock
* update unit tests
* fix argparser unit test
* fix lock
* the deps are now lightweight enough to be incude in make build
* add spaces for tests
* add eval outputs to gitignore
* remove git submodule
* readme
* tweak git email
* update upload instruction
* bump codeact version for eval
---------
Co-authored-by: Bowen Li <libowen.ne@gmail.com>
Co-authored-by: huybery <huybery@gmail.com>
Co-authored-by: Bart Shappee <bshappee@gmail.com>
Co-authored-by: Robert Brennan <accounts@rbren.io>
* a starting point for SWE-Bench evaluation with docker
* fix the swe-bench uid issue
* typo fixed
* fix conda missing issue
* move files based on new PR
* Update doc and gitignore using devin prediction file from #81
* fix typo
* add a sentence
* fix typo in path
* fix path
---------
Co-authored-by: Binyuan Hui <binyuan.hby@alibaba-inc.com>
* adding code to fetch and convert devin's output for evaluation
* update README.md
* update code for fetching and processing devin's outputs
* update code for fetching and processing devin's outputs
