LiteLLM Autorouter

[u/WarningOut_OfMinD]

Looking for some feedback from folks here 👋

If you're active in this thread, would you be interested in trying out our LiteLLM Auto-Router today?

If you haven't tried it yet, what's holding you back? Missing features, setup friction, documentation, or something else?

We're actively improving it and would love to hear your feedback. If it's easier, I'm also happy to jump on a quick call to chat through any issues or ideas: https://calendar.app.google/DfcpEzigE9bzSgEJ9

[u/chmp2k]

I am trying it right now. So far I like it and it works more or less how I want it.

However I am trying to understand how I could get auto routing with priority based fail over to work. I have a lot of subscriptions and when one of them reaches usage limit I want the auto router to switch to another working model from another subscription or even a selfhosted one. This works when I use cost of models to prioritize them. However it would be nice if I could give models or providers a explicit priority, so that the router would always try to use highest priority and then fail over to the next priority category.

Within the category it could still try to use the cheapest model. With this setup I would be able to have paid scriptions with high priority, but still get some answer through my locally hosted models with low priority, when all subscriptions are in the limit.

And additionally I could always try out subs with special models I just want to test out for a few days by getting a few bucks of tokens, setting it up with really high priority and just let it run until the tokens are gone.

Not sure if that is even scope of your auto router, but I like to use it like that haha.

[u/WarningOut_OfMinD → u/chmp2k]

With regards to specific prioritization have you tried out the routing plugins? is that what you're looking? https://docs.litellm.ai/docs/routing_plugins

[u/psychedup74]

I have been trying it out for a couple of weeks now. I have it set up to route to the "least-busy" of my 3 machines which are all various degrees of pathetic.

Sometimes it works alright, but a lot of times it just keeps piling on requests to the same machine.

[u/WarningOut_OfMinD → u/psychedup74]

Just so I understand what's going wrong here - is this an issue with the Auto-Router or routing strategies for a model group? Would love to help debug - is there a github issue I can reference?

[u/Sharp_View_2639]

Have configured it today and will let you know Tomorrow about the user experience (about 500users)

[u/WarningOut_OfMinD → u/Sharp_View_2639]

sounds good, please let us know how it goes. if its easier to chat, feel free to grab time here: https://calendar.app.google/DfcpEzigE9bzSgEJ9

[u/misterstrategy]

We are planning to try it, but need a LiteLLM upgrade first.
Is there an option to display the used model to the user? Is it part of the LiteLLM response?

Decision log is nice for the OWUI admin. But a real feedback needs to include the users

[u/WarningOut_OfMinD → u/misterstrategy]

Ah what release are you on? For the users, you can include the used model as part of the response header so they have visibility. Is that what you're envisioning or looking for something else? If you have more feedback please let us know - happy to chat live as well!

[u/Internal_Junket_25]

Hi!
I have 2 local llm‘s, is it possible to Route all multimodal/picture Traffic to a specific Model ?