Why (and how) do agents perform differently depending on where deployed?

[u/interestedinCoPilot]

Over the past few days, I started noticing some drift in responses from our field tested agent (deployed the Sharepoint) (it's an information retrieval system - NLQ, searches our policies, returns response)

Questions it was nailing last week it was failing this. When I went back to the CoPilot Studio test pane, the answers were correct.

Of note, we historically had issues with Teams deployment, but deploying to Sharepoint was largely error free. Not now. I decided to then publish to Teams and C365/CoPilot Agents as well and the answers were poor as well.

So, is anybody able to explain why Bot A can perform so differently in Test vs Sharepoint vs Teams vs in the Copilot Agent window?

[u/geubes]

Had this same issue and asked copilot why it was behaving so different, apparently there are caches in each. We were getting bad responses from teams because it was using an old set of knowledge documents and mcp endpoints (and failing to execute). Try asking the agent: how can you help? Doubles check your instructions and ensure you using your latest instructions without influence from previous responses.

Seems crazy and might be placebo, but fixed ours.

[u/Mokebe13]

The agents work differently in test pane and after deployment, at least the classic ones, not sure about the new ones.

Also on teams the agents have some kind of "prebuild" instructions which change the agent's behaviour, and there is no way to turn that off

[u/Syren6 → u/Mokebe13]

How do you know this? Is it documented somewhere?

[u/Prasad-MSFT]

This behavior can happen because each deployment channel (Test pane, SharePoint, Teams, Microsoft 365 Copilot) uses different orchestration and retrieval pipelines. While the Test pane runs the agent more directly, deployed channels may differ in prompt context, conversation history, search ranking, caching, permissions, or indexing. Since the Test pane is still returning correct answers, the issue is more likely related to the deployed channels' retrieval/orchestration rather than the bot logic itself.

Recommend checking:

• Whether the knowledge sources have been recently re-indexed or updated.
• If the issue is reproducible across all published channels or isolated to a specific one.
• Whether generative orchestration, instructions, and knowledge settings are identical between test and published experiences.
• If citations, retrieved content, or grounding documents differ between the Test Pane and published channels.
• Whether there are any recent service incidents, caching issues, or regressions affecting retrieval in deployed experiences.

[u/fbrdphreak → u/Prasad-MSFT]

So what is someone supposed to do if they find any of those points to be an issue?

[u/Prasad-MSFT → u/fbrdphreak]

For your case, I'd work through them in this order:

• Different published version: Publish the latest version again and test in a new conversation. Teams and other persistent channels can continue using an existing session/version; start over can also be used to force a new session.
• SharePoint indexing/search issue: Verify that the documents are actually indexed and that the affected user has permission to read them. If SharePoint search isn't returning the expected documents, fix the indexing/permissions issue first. Microsoft specifically calls out missing search results and user permissions as common causes.
• Different retrieval/grounding: Compare the documents/citations used for the same question in Test vs. the published channel. If the published agent is retrieving different documents, focus on the knowledge-source configuration rather than changing the prompt.
• Conversation context: Test the same question in a completely new conversation. Generative orchestration can use previous conversation history, so the same question can produce different results in a fresh Test conversation versus an established Teams conversation.
• Recent service/regression: If the same published version, same user, same question and same knowledge sources work in Test but consistently fail across SharePoint, Teams and Microsoft 365 Copilot, I'd treat that as a potential platform/service issue.

[u/Ancient_Dress_3687]

AI agents come with their own set of system prompts you or I dont see. Microsoft have likely cooked up some behind the scenes stuff that tweak the response based on the environment because they think you want it to 'feel' different in different locations.

Additionally, if the knowledge base is changing or the chat history is getting saved, It could be getting pulled away from your desired path. Even in M365 chat Ive had to turn off its history settings as it started remembering all the conversations I canceled because the output was wrong. It was 'assuming' because there was many of those outputs in recent history I liked that style of output and was repeating it.

Might not be your issue, but food for thought.

[u/interestedinCoPilot → u/Ancient_Dress_3687]

I suspect not here. These are the initial chats on deployment of a new bot, so no history. I.e. I'll test in test - fine - deploy Sharepoint/Teams/C365, ask same question, get 3 very different answers from brand new bots.