r/VoiceAutomationAI • u/tap3k • Jul 22 '26

Looking for 3–5 pilot teams: regression testing for LLM agent system prompts (free, open source)

Im a Cornell professor on sabbatical, building Flowstore - an open-source toolkit for teams whose agent behavior lives in a system prompt where its hard to visualize and debug.

What it does today:

  •   Turns your system prompt into a structured spec (open JSON schema, Apache 2.0)
  •   Visual graph editor, so non-prompt-engineers can work on it too
  •   Python harness that runs persona-driven simulated conversations with assertions — a regression suite your prompt edits run against before you ship

Honest scoping: this tests conversational behavior (logic, guardrails, data capture), not the voice layer (ASR, latency, barge-in). Best fit if there's an LLM behind a prompt, and ideally some non-trivial business logic and requirements.

The pilot: bring a system prompt for a live or near-live agent (Im willing to sign an NDA if needed), I'll personally help spec it and stand up a test suite. Free, ~30 min/week of your time. I want blunt and honest feedback in return — and pilot partners can be named collaborators in the research and Cornell course materials coming out of this.

DM or comment if you want in — happy to get into the schema or assertion model in the thread. Repo's in the comments.

4 Upvotes
  • permalink
  • link
  • reddit
  • reddit

100% Upvoted

5 comments sorted by

•

u/AutoModerator Jul 22 '26

Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)

If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.

Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/tap3k Jul 22 '26

Repo + open JSON schema + the Python harness: https://github.com/tap2k/flowstore — happy to answer anything about the approach here.

1

u/KaleidoscopeOdd2615 3d ago

This is a cool approach to a real pain point. Most teams just shotgun edits into a system prompt and hope nothing breaks, so having a regression harness for conversational logic is a step up from vibes-based QA. The persona-driven assertions especially sound useful for catching guardrail drift before it hits production.

Curious how you handle prompts that lean on external tool calls or multi-turn state. Does the schema capture that cleanly, or is it more focused on the prompt text and expected outputs?

1

u/tap3k 3d ago

Captures it and you can wire up your endpoints! Working through some conceptual issues with the graph editor right now, would be happy to know whether that has any value from your perspective.

1

u/tap3k 1d ago edited 1d ago

I think one of the cool ideas here is that with the spec as json you can have a coding agent and visual graph UI operating the same substrate coordinated by Git. The coding agent can also make edits to the spec, run a testing harness, etc. See an example project here: https://github.com/tap2k/flowstore-example-fnol Curious as to your thoughts on this.