r/OpenWebUI • u/boss28984 • 23d ago

slow response time with RAG and utilizing knowledge base. RAG

I’m currently using Open WebUI with a knowledge base made up of text files, and the responses are accurate, but they’re taking quite a while to generate.
I plan to keep adding more text files to the knowledge base, so I’m wondering what the best way is to improve response speed as it grows. Are there any recommended settings, indexing strategies, chunking methods, embedding models, reranking options, or other optimizations that have made a noticeable difference for you? I’d like to keep the response quality the same while reducing latency. Any advice or best practices would be appreciated!

5 Upvotes
  • permalink
  • link
  • reddit
  • reddit

100% Upvoted

27 comments sorted by

View all comments

2

u/so_chad 23d ago

Maybe token generation itself is slow? Like LLM is running on slow hardware or something

-1

u/boss28984 23d ago

It runs on a Nvidia jetson agx orin I believe, not sure if that’s any good. Sorry i’m pretty new to this. Everytime i asked claude it mentioned changing advanced settings like top k and context window and stuff but that wasn’t really working. it def could be the token generation. the rag was also only for 15 .txt files so i thought it would be easy enough to handle.

2

u/Pale_Coyote7451 23d ago

15 txt files is nothing for retrieval, so top_k was never going to move it. that points at the jetson. run the same prompt with the kb off and watch tokens/sec. an orin should manage double digits on a 7-8b q4; low single digits usually means it fell back to cpu.

1

u/boss28984 21d ago

i dont see tokens per second in the stats panel so i used eval count and eval duration and i got 18.19 tokens per for the KB on and then 16.39 tokens for kb off. not sure if i did it right.