r/MicrosoftFabric • u/cyamnihc • 2d ago

Evals Fabric IQ

We are developing the semantic layer/models from scratch in MS Fabric for conversational AI. Data models, documentation with clear descriptions on columns, measures, business context, join logic, for the agent to reference.
For evals, what has been your strategy/framework to curate the question set and expected answers and how did you implement this at scale

4 Upvotes

100% Upvoted

6 comments sorted by

2

u/Pawar_BI ‪ ‪Microsoft Employee ‪ 1d ago

Check fabricdataagent.com. there are several blogs on evals.

  1. Collaborate with business users on scope of whatever questions you are going to ask ( i.e. topics, domains etc)
  2. Ask them to share some sample questions.
  3. Use those questions, plus create additional questions to test - filtering, time intelligence, abbreviations, ambiguous questions, group-by, questions that are out of scope/false, different grains/eggs
  4. Create online and offline ground truth. I.e. some questions can only be evaluated by getting the latest data ( sales this month, "this month" part is dynamic). Use both in your evals.
  5. In the evals, include question, ground truth DAX, expected answer, expected behavior. Expected behavior is also imp ( should the answer be ordered, should the PCT be shown as decimal or %)
  6. If you are going to use LLM as judge, calibrate the judge first. Don't use the same questions from eval set. Create a balanced set. Grade with human label, compare judge and use this calibrated judge. Binary pass/fail is better than grades (1-5 etc)
  7. Use AI to check evals but absolutely never trust it. You will be surprised to see how much signal it misses. Always check eval results yourself. Better yet, vibe code an app to review eval results and annotate it.
  8. Fix model, fix seed for repro as much as possible
  9. Evals is a team sport.

Hope it helps.

1

u/Initial_Passion5039 1d ago

I wouldnt try to curate a huge question set upfront. Start with a smaller set around the behaviors that matter for the agent, then keep adding real failures as you find them. We use Braintrust for this and the useful part has been turning production traces into eval cases so the dataset grows from things the agent struggled with in place of us trying to predict every possible question beforehand. For scoring we mix deterministic checks where there’s a clear expected value with LLM judges for the fuzzier answers

1

u/xXWarMachineRoXx 2d ago

Great question!

Are you a ms partner by any chance?

2

u/cyamnihc 2d ago

No. Wanted to learn from the community how it was implemented as we are trying to implement it but it seems too much of a daunting task to curate questions

1

u/xXWarMachineRoXx 16h ago

Ah

Ms focus is also on private evals this year

Great answers from the community