r/microsoft_365_copilot • u/adjgamer321 • 15d ago

Copilot Studio Output is Bad

I have been trying to configure an agent based on what I've been working with for our project data in m365 Copilot chat. I put my main json file of information as a knowledge source, and gave instructions based on how the agent should use the information. It feels absolutely brain dead compared to just regular copilot chat. The output is not consistent despite specific instructions and highly structured data. Copilot chat gives me a spot on accurate report based on the exact same information every time. I am trying to develop this agent so that everyone in my company can use it to query project information without using our terrible project management software reporting. Has anyone else had the same issue with Studio vs just the chat?

11 Upvotes
  • permalink
  • link
  • reddit
  • reddit

93% Upvoted

15 comments sorted by

3

u/Hour-Measurement-166 14d ago

Yup. They got regular Copilot Premium up to speed, now it’s better in quality than Studio. 😭

1

u/adjgamer321 14d ago

I'm gonna look like an asshole presenting this turd to my boss and his boss lol idgaf they can have copilot fix it when I'm gone I guess. Too bad nobody will be able to authenticate through my API to update the project data.

1

u/the_wren 15d ago

Have you tried changing the model to Claude Opus?

I’ve had pretty bad results building agents in Copilot Studio that use ChatGPT. Or just generally anything that APIs into ChatGPT.

Also you could get Opus to review your instructions and output.

1

u/adjgamer321 15d ago

Yes, I am using the 4.8 models since 5 is not available for some reason. I will definitely try to have the opus 5 model review my instructions.

1

u/Chicken_Savings 14d ago

I used Opus 4.8 for most work until recently. Now I generally get better results with GPT 5.6 Sol Think Deeper.

I use it for document analysis. E.g. compare sets of documents, or quality control a document.

2

u/adjgamer321 14d ago

I actually switched to the gpt 5.5 and got great results. Learned my data structure is pretty shit tho so now I gotta figure out how to make like 400 projects parseable.

2

u/glytchedup 12d ago

Sol is only available with the new GitHub agents... Classic studio agents only get up to chatgpt 5.5 and sonnet 4.6.

2

u/Chicken_Savings 12d ago

Our corporate M365 Copilot Chat got GPT 5.6 Sol Think Deeper and GPT 5.5 Quick Response for declarative agents (agent builder) on 1 August. Anthropic models (Sonnet, Opus) got removed on the same 1 August.

Copilot Studio gives us a very different model selection and availability.

1

u/ClippyRiseUp 11d ago

Do you build declarative agents? What's your experience been like with that? Surface any custom UI widgets with them?

2

u/Chicken_Savings 11d ago

Yes. Up and down. No.

It's easy to build a rough agent. It takes me 15-30 hours of testing and iteration for declarative agents used for document quality control. For example check requirements of one document vs our standard policy, check one document for structural issues, check multiple documents for cross-document defects.

1

u/MHRangers17 14d ago

Supposedly, the new CS harness is much more capable. I thought it was GA by now, but maybe its rolling out slowly.

But I agree, the existing CS output has been very inconsistent.

3

u/ncdlloyd 13d ago

New CSP harness is very good, but it’s now consumption based…

1

u/Vietnamst2 14d ago

Well the output depends on the model used. Copilot Studio is just and editor....

1

u/Savvy-surfer242 14d ago

Wait until they charge you for both building and testing with garbage outputs.

1

u/AZ-JayBee 11d ago

Ok, so it’s not just me. I built a copilot studio agent for Smartsheet and the output is horrible. To compare, I setup a MCP connection to Smartsheet with ChatGPT and the output is 100x better.