github/OpenHands
1
0
Fork 0
mirror of https://github.com/OpenHands/OpenHands.git synced 2025-12-26 05:48:36 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs: local serving with ollama documented (#8807)

Browse Source 
Co-authored-by: Engel Nyst <enyst@users.noreply.github.com>
This commit is contained in:
Alexander 2025-06-17 14:18:18 +03:00 committed by GitHub
parent ddaa186971
commit d81d2f62cb
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 12 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
docs/usage/llms/local-llms.mdx
View File

@ -126,6 +126,18 @@ vllm serve all-hands/openhands-lm-32b-v0.1 \
--enable-prefix-caching
```
### Create an OpenAI-Compatible Endpoint with Ollama
- Install Ollama following [the official documentation](https://ollama.com/download).
- For Ollama configuration, use `ollama/<modelname>` as custom model in web. Api key also can be set to `ollama`.
- Example launch command for Devstral LM 24B:
```bash
OLLAMA_CONTEXT_LENGTH=32768 OLLAMA_HOST=0.0.0.0:11434 OLLAMA_KEEP_ALIVE=-1 nohup ollama serve&
#The minimum context size is ~8196, even the system prompt won't fit smaller
ollama pull devstral:latest
```
## Advanced: Run and Configure OpenHands
### Run OpenHands