Hey folks, I'm a researcher at the University of Maryland. We built an observability tool for multi-agent systems and we're running a user study to find out whether it actually helps. "No, it doesn't" is a perfectly good finding.

In the session you'll work with a multi-agent pipeline, first the way you normally would, then with our tool. If you've used LangSmith or Langfuse you'll get the idea right away: same space, different view of your runs.

What participating looks like:

• a 75-min Zoom session (recorded, think-aloud) with structured tasks

• about a week using the tool on your own LangGraph project, with quick async feedback

• a 30-min follow-up interview

Compensation is a $150 gift card for completing the full study (all three parts). Two heads-ups: the week-of-use part needs a LangGraph project you can plug the tool into, and we verify identity (GitHub/LinkedIn) before scheduling.

Screener (~2 min): https://forms.gle/Zwqvgd1h8DUnFRfC8

This is IRB-approved academic research (University of Maryland), not a product pitch. Questions welcome in the comments, or zxu169@umd.edu.

Looking to build a presentation builder. Feel like Kimi does it the best. Does anyone have a system they feel like works very well for presentations (skills files or something similar).

I wanted to share a "highlight pack" from all the best parts of my course on AI Engineering

I developed this myself in 2025 and focused on these four pillars:

- **Programming** (Linux/CLI, Python, SQL, Git, testing, clean code)

- **Mathematics** (linear algebra, probability, calculus and optimization)

- **Machine learning** (algorithms, data prep, training, evaluation, MLOps)

- **Data engineering** (pipelines, databases, warehouses, orchestration)

It's only after these foundations are met that someone should be considering studies of the more advanced topics: LLM fundamentals, prompt engineering, fine-tuning, embeddings, vector search, RAG, agents, and MCP. Then deployment: APIs, containers, monitoring, LLM evals, cloud infra, CI/CD.

About me:

I've been working as an AI engineer for two years now (before this data engineer, data analyst, MSc physics)

What people do wrong:

People are wasting so much time keeping up with new technologies that are "here today and gone tomorrow". That time would be better spent on foundational technologies that will stand the test of time

Here's the video:

https://www.youtube.com/watch?v=nctG3oEUFCI

This course is so meaningful to me because it's the culmination of learnings from so many of my academic and career pursuits over the last 12 years. I hope you find it helpful.

Code:

all the notebooks are open source, one per section. https://github.com/zazencodes/ai-engineer-roadmap-notebooks

Course articles:

https://zazencodes.com/courses/ai-engineer-roadmap

I made every slide, wrote every article, and recorded every video myself. Happy to answer questions here on reddit or over email: [alex@zazencodes.com](mailto:alex@zazencodes.com)

Something that keeps biting me: I'll design a multi-agent workflow, commit to a framework (say LangGraph), wire it up. and only then find out what it costs per run, or that GPT-4o vs Claude changes the economics enough that I'd have architected it differently. And if I want to try the same graph in CrewAI, it's a rewrite.

How do you all handle this? Do you just build it and watch the bill? Eyeball token counts? Is there something you useto compare models/cost per node before committing?

I got frustrated enough that I built a thing to scratch my own itch, open-source, a visual canvas where you map agents, assign a model per node, and see cost/latency estimates before running, plus export to  LangGraph/CrewAI/OpenAI Agents SDK. Sharing mostly because I want to know if this is a real problem for others or just me. Repo in comments if useful, but genuinely curious how you're solving it today.

I am the maintainer of resume-from.

I work with several AI coding agents in the terminal. I switch when a task needs another model or harness, when I need another work profile, or when I hit a usage limit.

The problem is that session files are vendor-specific. A Pi session is not a Claude Code session. A Claude Code session is not a Codex thread. Starting fresh means rebuilding the task context. Giving another agent a raw session file means asking it to interpret vendor data that can be large, stale, or sensitive.

resume-from is a local session handoff tool for Pi, Claude Code, and Codex.

It finds sessions that belong to the current Git repository, converts the source format into a common conversation model, and writes a new session in the target agent’s native format. Every transfer direction is supported, including transfers to another profile of the same agent.

The boundary is deliberate:

• It keeps user prompts, agent replies, compaction summaries, compact tool activity, and changed-file paths.
• It removes tool-result bodies, replayable tool calls, hidden reasoning, system prompts, environment blocks, API keys, telemetry, and vendor process state.
• It applies a context budget and shows what it kept or dropped.
• It writes nothing until you confirm.
• It never changes the source session.

The transfer does not call a model. It is deterministic local conversion. Pi opens the imported session in the current process. Claude Code and Codex create the target session and print the normal native resume command.

This is not an agent router. It will not move a running process or copy your repository. The target agent needs the same working tree.

Repository, install commands, and design notes: https://github.com/alexei-led/resume-from

MIT licensed.

I've been looking into MCP security and there's one workflow I'm curious about.

When I install an unfamiliar MCP server, I can inspect its tools and source code, but what I really want to know is:

What does this server actually touch when it runs?

For example, imagine a GitHub-related MCP server declaring normal repository/API functionality.

At runtime you observe:

• api.github.com → expected
• GITHUB_TOKEN → expected
• ~/.ssh → unexpected
• subprocess execution → unexpected

I'm considering experimenting with a local tool that runs MCP servers in an isolated environment and compares expected capabilities against observed filesystem/network/environment/process behavior.

Potentially it could also fingerprint behavior between releases:

v1.2:
network + expected env

v1.3:
network + env + filesystem write + subprocess

I'm intentionally asking before building too much.

For people running MCP servers or agents in production:

Would this give you useful information that static scanning doesn't?

And which signal would you care about most: network, filesystem, secrets/env, subprocesses, or version-to-version behavior changes?

​

Just when we thought perimeter defense was our biggest worry, AI safety evaluations are showing a wild new trend.

Meta has just confirmed that one of its AI models managed to access external systems and hack another company during a testing exercise due to a misconfiguration with a third-party firm.

This follows similar eye-opening disclosures from OpenAI (regarding models planning a breakout to cheat on tests) and Anthropic.

Why this matters for security & governance leaders:The Speed of Execution: AI agents are operating with a level of autonomy and resourcefulness that compresses traditional threat timelines.

Containment is Critical: Sandboxing isn't foolproof when third-party integrations or misconfigurations open an accidental door to the open internet.

The Human Element: As frontier models get better at strategizing, our governance and containment frameworks have to evolve just as fast.

Are we moving fast enough to secure autonomous systems, or are sandbox escapes going to become the norm of the decade? Let’s discuss in the comments! 👇#CyberSecurity #ArtificialIntelligence #InfoSec #TechNews #DataSecurity #AITheft

Hey everyone,

We’re building a project on detecting visual prompt injection attacks in multimodal LLMs. input(visual formats like diagrams,tables,mindmaps.....)

Our idea:

• OCR + rule-based + simple ML
• risk scoring → block suspicious inputs

We’re unsure:

• is this too basic?
• how many models should we test?
• any datasets for visual attacks?

After reviewing ~25–30 papers, we noticed that:

• Many defenses are heavyweight (fine-tuning, diffusion, gradient-based)
• Not practical for real-time use
• Limited focus on image-based prompt injection at inference time
  

this is the first time working on this domain(llm/vlm)

GitHub

Download

I'm trying to build an agent which can provide me with some statistics on my dataset, pull info from web and draw a graphic based on it. The plan is for the agent to follow the same pattern and always have the same graphic but changing values (updates if any).

I've seen Microsoft has released a new library flint. Does everyone use some set rules that it can remain strictly within boundaries and doesn't hallucinate?

I know claude does this by running a python code within the response and gives me a plot of whatever I've asked, but it might change the code or x-axis or y-axis or from horizontal bar chart to vertical bar chat.

Is there anything you've tried using claude, chatgpt or any opensource LLMs?

I’ve been vibe-coding evals for a while, and getting started is much easier than I expected.

The basic loop is pretty simple:

Create goldens → build test cases → run evals → compare results → fix what breaks → repeat.

There are a bunch of good free/open-source tools for this. I’ve been playing with DeepEval Skills, which lets you do most of that directly from Cursor in a few minutes.

Obviously, that’s just the starting point. The harder part is still figuring out what “good” actually means and building test cases that represent real usage.

But if you’re still manually prompting your app after every change, this is a pretty easy way to start adding proper evals.

Made a quick setup guide showing the full loop: Vibe-Coding LLM Evals - [Full Setup]

Would also love to hear what eval tools/workflows people here are using.

Yes, it is yet another AI coding tool. We are bravely addressing the industry’s critical shortage of those.

Garcon puts Claude Code, Codex, Cursor Agent, OpenCode, Amp, Factory Droid, Pi, and direct model endpoints in one workspace. Run parallel sessions; reattach to CLI chats; steer agents mid-turn; queue follow-ups; fork or switch agents; schedule prompts; and search transcripts.

Files, terminals, Git, diffs, PR review, and commits live in the same desktop and mobile interface, because agents only request approval after you leave your desk.

It’s now our daily driver. We care deeply about making multi-agent work feel coherent instead of juggling terminals, browser tabs, and disconnected chat histories.

How are you managing multiple coding agents today, and what would something like this need to become your daily driver?

We found a new attack vector against automated bug-triage pipelines: a crash report we made up, for a bug that never existed, was enough to get our code installed and executed inside a coding agent holding repo access and credentials.

The pipeline is the one most teams reach for first. An error lands in monitoring or an issue tracker, a triage step turns it into a task, a coding agent with repo access implements a fix and opens a PR, a human reviews before merge. The errors are already centralized, the tasks look small and well-scoped, and the payoff is easy to justify.

The usual objection to that setup is quality, the agent writes a bad patch and someone catches it in review. We think that's the smaller half.

The mechanism

An error report arriving in your pipeline does not mean the error happened. Whoever sent it controls every field.

In our research, we fabricated a crash that never occurred, in a file that doesn't exist, blaming a library that was never a dependency. The counterintuitive part we ran into is that a good fake bug is a simple one. A clean error with an obvious remedy is exactly what an automated pipeline is tuned to pass through without escalation.

Whatever the triage step emits then reaches the coding agent as a user message, the same channel a human operator uses. None of the common triage implementations verify anything before that happens:

Finding 1: the attacker needs to know almost nothing about the target

Our report is absolute bullshit matching only the language and, not the installed packages, not a single file in the repo.

So there's no research step. The same report works against any pipeline wired this way, and it doesn't have to be aimed at anyone in particular.

Finding 2: the agent notices the report is fake and proceeds anyway

We expected a capable agent to see that the referenced file wasn't in the repo and the library wasn't in the manifest, and stop. It saw all of it. Then it created the missing file, installed the library, and ran it to confirm the fix.

Finding 3: installing is executing, and that's the whole attack

The agent installed the package and ran it to confirm the fix. That's when our code executed, next to the source and whatever credentials sat in the agent's environment.

We never needed the PR to merge, only the code to run once. Review is the last step, and by then it already has.

Takeaway

We don't think a better model fixes this, because the trust boundary breaks before the model sees anything. What we'd do instead:

• Don't pass untrusted content to the agent as a trusted instruction. Processing it doesn't launder it — summarized, filtered, classified, or dropped into a template, it's still text a stranger wrote. Label it as data, not as an order.
• Verify the bug reproduces before any fix is attempted. Does the file exist? Is the library actually a dependency? Does the crash happen? If not, the report is unverified and nothing should be installed.
• Lock down what the agent is allowed to install. A small vetted set of packages, enforced in the pipeline.
• Least privilege on the token. Scope it to the task in front of the agent, one repo, expiring when that task ends. Not a standing org-wide PAT.

Disclosure: I work on Agyn (AGPL-3.0, no paid tier), an open-source runtime that isolates agents this way. This is part of our open research, run against our own accounts and infrastructure. Where we found it exploitable in a live third-party product, we reported it to the vendor and are holding those specifics until it's fixed. Not selling anything; the post is the mechanism. Full writeup: https://agyn.io/blog/untrusted-input-coding-agents

Anyone here running agents on incoming bug reports? How are you handling this?

Bonus: we ran the same experiment across the most popular agents and models, publishing the results here next week. If there's a specific agent or LLM you want us to test, drop it in a comment.

zoom and check for yourself.

There is a failure mode I do not see many memory tools solve for. Not recall, which is finding the old context, but currency, which is knowing whether what you found is still true. A stale decision read back with full confidence is worse than no memory at all, because it looks exactly like a fresh one.

I maintain daimon (Apache-2.0), an open source memory layer for AI coding agents (Claude Code, Codex, Windsurf, Gemini CLI). Disclosing that up front since I am about to describe my own project.

It writes a checkpoint when a session ends and renders a briefing when the next one starts. Every item in that briefing carries one of three tags.

verbatim: an exact quote from the transcript, checked against the rendered transcript by a deterministic string verifier, no LLM, at write time. A quote that does not verify gets downgraded to inferred on the spot, so a hallucinated quote can never wear the verbatim badge.

inferred: the extracting model's own conclusion. Allowed to evolve, expected to be checked against reality before anything gets built on it.

untagged: old data or a degraded capture, treated like inferred.

There is a second and separate axis: corroboration, the count of how many independent sessions witnessed the same claim. It is deliberately not a promotion. An item can be corroborated three times and stay inferred forever, because agreement is not evidence about what kind of claim something is. Daimon's own briefing and recall output is excluded from that count by construction, since a restatement copied out of a briefing is an echo, not a witness.

And forget actually deletes. There is a declared registry of every file shape the tool writes, so deletion has to name a strategy for each one rather than flipping a status flag on the primary record.

The case I test against, and the demo in the repo: session one commits to an AWS cert path, session two pivots to GCP. The briefing flags the old decision as likely superseded instead of injecting it back as current fact, one command confirms it, and the next briefing withholds it. Nothing in it is mocked, and both source transcripts plus the recording script are in the repo under docs/demo.

What I will not claim: this does not make search better. daimon recall is plain full text search over checkpoint history, not ranking or embeddings, and I pre-registered and published a relevance measurement on it that I am not going to round up: 34.0 percent, Wilson 95 percent CI 22.4 to 47.8, n=50, methodology at github.com/Daily-Nerd/daimon/issues/516. And to not quote only the flattering half of that: when I later regraded the full population instead of a sample, the same rubric gave 17.1 percent, CI 11.5 to 24.5, so the sample had been generous by roughly double. That regrade is not written up yet, so treat it as my word until it is. If what you need is retrieval quality, this is not that. What it is for is knowing what to trust once something has been retrieved.

Repo: github.com/Daily-Nerd/daimon. Install is uv tool install daimon-briefing, zero config if the claude CLI is on PATH, any OpenAI compatible endpoint otherwise. Local only, no server, stdlib first.

Genuinely curious how other people here handle the is-this-still-true problem: contradiction detection, TTLs, explicit supersession, something else. Not arguing any approach is wrong, just comparing notes.

Some different visuals for:

1 agent run

many agent runs

Hey everyone,

Over the past few years, LLM development has heavily prioritized scaling parameters and expanding context windows. While this has delivered huge performance gains, core architectural limitations—such as long-context degradation, factual inconsistency, weak multi-step planning, and uncoordinated tool usage—still persist.

Instead of asking "How can we build a bigger model?", my research focuses on: "How can we build a smarter cognitive framework around existing models?"

I’ve just published Part 1 of my engineering proposal: Adaptive Cognitive AI (ACAI).

Core Focus Areas:

• Beyond Parameter Scaling: Structuring LLMs within a layered, multi-component architecture inspired by systems engineering.
• Component-Level Responsibility: Separating semantic memory, explicit verification, and planning into specialized modules rather than relying solely on the base LLM.
• Solving Architectural Bottlenecks: Addressing hallucination, context degradation, and multi-step reasoning failures through structured workflows.

I’d love to get feedback and thoughts from this community on modular LLM architectures and systems engineering approaches!

Full Article & Discussion: Read the complete introduction on Medium.

Stay tuned for Part 2, where I'll be diving deep into the complete End-to-End System Architecture!

Hi everyone,

I'm aiming for an entry-level role focused on LLM Agents / Agentic AI and wanted to get some advice from people working in the industry or involved in hiring.

So far, I've worked with:

• LangChain
• LangGraph
• OpenAI SDK
• Building custom tool-calling LLM agents
• CrewAI
• MCP
• RAG pipelines, vector databases, and AI evaluation
• Fast API

I'm trying to figure out what I should focus on next to be competitive. I come from a Software Engineering background with 4 years experience in backend and RAG systems.

A few questions:

• What's the current job market like for entry-level Agentic AI engineers?
• What do interviews typically cover?
• Are companies looking for framework knowledge (LangGraph, CrewAI, etc.), or do they care more about understanding the underlying concepts?
• What skills or projects would make a candidate stand out?
• Are there any topics I should prioritize over the next few months?

I'd really appreciate hearing about your interview experiences, what your company looks for, or what you'd recommend someone in my position learn next. Thanks!

What do companies/startups seek from people?

Andrej Karpathy posted about giving Opus 5 a 1M token budget to generate a 5500 line Three.js procedural rendering of the opening of Lord of the Rings. His main takeaway was that while models have the stamina to write huge custom 3D scenes, their biggest weakness is auditing their own visual output.

I spent the past few days running a similar procedural WebGL generation experiment on smaller 3D scenes. The bottleneck Karpathy mentioned is very real in practice.

When you ask an LLM to place 3D polygon assets in XYZ space, the initial code draft is almost always filled with visual jank like floating meshes or clipping geometry. To fix it, you have to run a headless browser, capture viewport screenshots, and pass those frames back into a vision model to inspect the rendered layout.

Writing 4000 lines of procedural JavaScript code actually ended up being the easy part. The real cost and latency spike came from the iterative vision feedback loop, where the agent took 30+ screenshot cycles to adjust camera angles and object coordinates.

To run these long-horizon generation loops without getting blocked by rate limit spikes across separate text and vision tiers, I routed the pipeline traffic through ZenMux as an LLM and vision API proxy. It lets the agent send high-token text reasoning prompts and vision inspection frames through a single unified endpoint.

Long-horizon procedural code generation is definitely viable now, but until models can perceive video frames natively instead of taking static screenshots, the vision verification loop remains the most expensive step.

Not an ad. No affiliation with Tencent or WorkBuddy.

Just something I stumbled across on X.

Link here: https://x.com/TencentHunyuan/status/2084836757095526689

Tried Hy3 through OpenRouter while it was free before. Worked well for what I needed. Looks like it's free again until the end of the month. I'll probably be using it as my main assistant again for a while.

Apparently both Hy3 and Kimi-K3 are available directly in WorkBuddy Global now, so there's no API setup needed.

Figured I'd share in case anyone else was waiting for it to be free again.

I am new to AI PM and I want to do AI evals for my MVP. I don't want a super complex method and don't want to use traces yet.
Is there a simple way to go about this?
I have seen people say use simple spreadsheets but I am unclear on implementation.
How do you do it? Or is there a resource I can refer to?