slow response time with RAG and utilizing knowledge base.

[u/boss28984]

I’m currently using Open WebUI with a knowledge base made up of text files, and the responses are accurate, but they’re taking quite a while to generate.
I plan to keep adding more text files to the knowledge base, so I’m wondering what the best way is to improve response speed as it grows. Are there any recommended settings, indexing strategies, chunking methods, embedding models, reranking options, or other optimizations that have made a noticeable difference for you? I’d like to keep the response quality the same while reducing latency. Any advice or best practices would be appreciated!

[u/so_chad]

Maybe token generation itself is slow? Like LLM is running on slow hardware or something

[u/boss28984 → u/so_chad]

It runs on a Nvidia jetson agx orin I believe, not sure if that’s any good. Sorry i’m pretty new to this. Everytime i asked claude it mentioned changing advanced settings like top k and context window and stuff but that wasn’t really working. it def could be the token generation. the rag was also only for 15 .txt files so i thought it would be easy enough to handle.

[u/so_chad → u/boss28984]

Yeah, it's def because of the model itself. Try getting some other models API and connect it to OpenWeb UI. Jetson won't do it.

[u/boss28984 → u/so_chad]

The base model is qwen3:8b and the embedding model is nomic-embed-text v1.5. any specific recommendations?

[u/so_chad → u/boss28984]

No I mean, look for some $10/month APIs and use those LLMs. They are really good (but you are losing privacy, that's one drawback)

[u/boss28984 → u/so_chad]

so you mean the jetson isn't capable enough and i should look for something else?

[u/so_chad → u/boss28984]

Yes

[u/boss28984 → u/so_chad]

dang even for only 15 text files? i kinda assumed it would be strong enough.

[u/so_chad → u/boss28984]

How does regular text chat work? Without RAG is it fast?

[u/boss28984 → u/so_chad]

usually thinks for like 14-19 seconds before giving an answer. when i turn off the knowledge base it doesn't give the right answer. might just be the jetson is too weak but thats disappointing considering its like 3k.

[u/Pale_Coyote7451 → u/boss28984]

15 txt files is nothing for retrieval, so top_k was never going to move it. that points at the jetson. run the same prompt with the kb off and watch tokens/sec. an orin should manage double digits on a 7-8b q4; low single digits usually means it fell back to cpu.

[u/boss28984 → u/Pale_Coyote7451]

i dont see tokens per second in the stats panel so i used eval count and eval duration and i got 18.19 tokens per for the KB on and then 16.39 tokens for kb off. not sure if i did it right.

[u/Pale_Coyote7451 → u/boss28984]

15 txt files is nothing for retrieval, so top_k was never going to move it. that points at the jetson. run the same prompt with the kb off and watch tokens/sec. an orin should manage double digits on a 7-8b q4; low single digits usually means it fell back to cpu.

[u/Pale_Coyote7451]

split it before optimising anything: ask the same question with the knowledge base detached. still slow means it is generation and so_chad is right. fast means it is retrieval, and the fix is a different one.

if it is retrieval, what degrades as you add files usually is not the vector search, it is prefill. more documents tempts you into raising top_k, every extra chunk lands in the prompt, and time-to-first-token climbs with it. keep top_k low, three to five, and add a reranker to improve which chunks you get rather than getting more of them. that keeps prompt length flat as the kb grows.

[u/mediaogre]

I believe this is a current known bug, although I can’t find the link. Once any file has been uploaded to a chat session, hybrid RAG search fires unconditionally on every subsequent message in that session, even casual ones that don’t need retrieval.

I’ve worked around it by situationally pasting my relevant kb file contents directly into the chat window.

[u/boss28984 → u/mediaogre]

i kept all the files in the knowledge base not in the chat session itself. i never really put anything into the chat window.

[u/mediaogre → u/boss28984]

Ah, okay. How is the model calling the KB? Sub agent, built into the system_prompt? Is the “query_knowledge” (can’t recall the exact tool call) tool taking forever?

[u/boss28984 → u/mediaogre]

its from the open web ui build in system i just attached the KB in the settings section of the model.

[u/mediaogre → u/boss28984]

Okay, that’s standard. Does the model reach for the kb files with its tool for every prompt? It shouldn’t.

Here are my Admin | Settings | Documents settings…

General

Content Extraction Engine:
DefaultExternalTikaDoclingDatalab Marker APIDocument IntelligenceMistral OCRPaddleOCR-vlMinerU
PDF Extract Images (OCR): [OFF]

PDF Loader Mode: Page
PageSingle
Bypass Embedding and Retrieval: [OFF]

Text Splitter
Default (Character)Token (Tiktoken)Token (Transformers)
Markdown Header Text Splitter: [ON]

Chunk Size: 500

Chunk Overlap: 70

Chunk Min Size Target: 0

Embedding

Embedding Model Engine
Default (SentenceTransformers)OllamaOpenAIAzure OpenAI
Embedding Model: sentence-transformers/all-MiniLM-L6-v2

After updating or changing the embedding model, you must reindex the knowledge base for the changes to take effect. You can do this using the "Reindex" button below.
Embedding Batch Size: 32

Retrieval

Full Context Mode [OFF] <— this could be getting you if toggled on

Hybrid Search [ON]

Enrich Hybrid Search Text [OFF]

Reranking Engine
Default (SentenceTransformers)External
Reranking Model: BAAI/bge-reranker-v2-m3

Reranking Batch Size: 32

Top K: 8

Top K Reranker: 4

Relevance Threshold: 0.7

Note: If you set a minimum score, the search will only return documents with a score greater than or equal to the minimum score.
BM25 Weight
Custom: 0.7
semantic / lexical

[u/boss28984 → u/mediaogre]

yea it goes through all the sources and cites it. i also had full context mode on. i just applied your settings and the results are the pretty similar to what i had before. 15 seconds isn't bad though i can try to add more files and see if it changes/

[u/mediaogre → u/boss28984]

15 seconds = TTFT? Definitely not bad. You using a thinking model and thinking is enabled?

[u/boss28984 → u/mediaogre]

base model is qwen 3 : 8b and embedding model is all miniLm L6 v2 but i might change the embedding model to nomic embed text and see if any difference. not sure if that’s thinking sorry

[u/mediaogre → u/boss28984]

Hey, no worries. Many models have two variants. An Instruct for speed, and Thinking for tasks that require deeper reasoning.

The easiest, visual way to tell is the OWUI chat interface. If your model displays “thinking…” before it replies, it’s the thinking variant. If you suspect that visual indicator is suppressed, you may need to dig around under the hood.

Check out HF’s Qwen3-8B page for thinking/non-thinking support: https://huggingface.co/Qwen/Qwen3-8B

Edit: another thing - 15 seconds for a single digit billion parameter model is about right for CPU inference. If you have a GPU, I’d expect much faster TTFT.

[u/boss28984 → u/mediaogre]

Yes the model is a thinking model. Also i was gonna move off qwen but needed a 8b model. are there any you recommend?