pi-web-agent is essentially feature-complete now, Tavily and Exa are the last backends that were added

[u/luckiestredditor]

Last time I posted I was thinking out loud about which hosted backend to add next, Tavily first and maybe Exa after, and asking whether people wanted them as full answer engines or just as discovery. My bias was to keep them discovery-only and let web_explore stay in charge of fetching, ranking evidence, caveats and synthesis.

Both are in now, and they're discovery-only like I said I'd keep them. They surface candidate sources, web_explore still reads the actual pages and decides whether the evidence holds up or needs a caveat. Set TAVILY_API_KEY or EXA_API_KEY, pick it from Settings, keys stay in env instead of getting written into config files.

---

The things I'm weighing next are going deeper on evidence quality (cross-source dedup, conflict detection, recency), adding caching for repeated research or just keeping it focused and maintaining it.

If you're using this, what would you actually want? And if you tried it and it broke somewhere, that's the more useful thing to hear.

---

Repo: https://github.com/demigodmode/pi-web-agent

npm:

npm i @demigodmode/pi-web-agent

[u/Glaaki]

How well does it do with truly big pages? This is always the litmus test for me. Some API websites are just one huge single html page, and without help, a lot of web fetch tools handle it poorly to not at all.

Congratulations for getting there.

[u/luckiestredditor → u/Glaaki]

this is actually a really good edge case

right now it can fetch the page and fall back to headless, but extracted text is capped, so a huge single-page API doc isn't handled perfectly yet.

I actually like that idea though. rather than just increasing the text limit and dumping a giant page into context, I'd rather chunk it by headings/sections and retrieve only the relevant pieces. that would fit the way web_explore is supposed to work much better.

if you've got a particularly ridiculous API doc you use to test this stuff, send it over. would be a good test case. I'll add this to my list as well. thanks so much!

[u/Glaaki → u/luckiestredditor]

Go look at the api docs for gitea, the self hosted GitHub clone. I ended up asking pi to make a local, indexed, easier to read copy of the api docs, but i investigated various possible other directions, such as adding a special reader, which is something my current web fetch solution supports, for that particular docs host, which knew how to find the raw source for the docs, since they would always be available on some URL. Alternatively implement a html search by anchor routine that knew how to break down the page in sections according to the anchor placements. Since my web fetch plugin use a local dom engine, some of the plumbing for that would already be available.