Muliple Models - Fallback

[tholum]

What problem or use case are you trying to solve?
While running, I would like to use groq/llama3 for my agent, but I run into issues with rate limiting ( It limits me to 6000 tpm, but opendevin will make requests larger then that ), I would love it to do the normal retry 3 or 5 times, but then on failer attempt a secondary model, and even better, have unlimited fallback models, so we could do stuff like groq/llama3 => openai/gpt4o => google/gemini, so we can get the benifit of the cheaper models, but the power of being able to have more context

 
Describe the UX of the solution you'd like
in the settings dialog add an "Add an fallback model" and put in the model and credentials

Do you have thoughts on the technical implementation?

Describe alternatives you've considered
The alternative I am considering, but will be a bit hacky is creating an external service, that proxy's the llm requests, and does the fall through its self, It will work as a work around, but having it in the app could allow for more advanced features in the future, like prioritizing different models for different tasks

Additional context

[imkebe]

^^ I have setup external LiteLLM to do so. OpenDevin uses LiteLLM internally.
Most important is to set for OpenDevin container with variables :

• LLM_BASE_URL of the external proxy
• LLM_API_KEY - create the key for accessing LLM's in external LiteLLM

In OpenDevin UI :

• use openai/your_llm_name_alias as a model - it's crucial that you set openai namespace for all models you access through proxy
• use a API Key - created in the external LiteLLM

For LiteLLM config follow it's documentation.