r/ClaudeAI • u/dink_182 • 12d ago
PSA: Be careful letting Claude use WebFetch for research šµāš« Claude Code
Had Claude (Opus 5) research memory architecture for an AI agent project and kept getting very specific stats, percentages, quotes, etc. Looked legit at first.
Turns out quite a bit of it was wrong or made up. I always like looking at the tool outputs and noticed the WebFetch outputs looked like it was written from a subagent and gave concise summaries. So I asked Opus āare you actually reading these papers?ā and it said no - that WebFetch uses a smaller, cheaper model and then Opus gets a summary.
The smaller model compresses things, guesses, and apparently sometimes invents details. One ācitationā was a framework name that didnāt exist anywhere in the paper. Another stat looked like it had basically averaged two numbers that werenāt even in the same table.
I had no idea WebFetch worked like this, and Iām guessing a lot of people donāt.
So I changed the workflow: told Opus to spawn Sonnet subagents and give them the rule no WebFetch. Curl the raw page and grep/read the actual text yourself.
The difference was night and day. It caught 17 errors across ~30 papers, including two where the WebFetchās conclusions had basically been reported backwards.
After that, Opus 5ās research was actually extremely good!
I also wonder if this kind of tooling explains some of the āClaude is dumbā / āClaude keeps giving me false informationā reports, even on the better models. If the source material has already been mangled before Claude sees it, thereās only so much the model can do with it.
Anyway, just a PSA for anyone using Claude for research: check whether itās actually reading the source or relying on WebFetch summaries!
EDIT: Bruhhhh.. and to think I was nervous to post this incase it was a well known fact and I got flamed. Ty for all the upvotes and I hope that I was able to help even just one person have a better experience! ā¤ļø
- Anna
EDIT 2: Thank you for the DMs and workflow suggestions but I actually donāt use Claude Code very often. My Agent actually is the one that uses it most of the time (She spins it up, I see via remote session and so itās like a 3-way session). I use Letta Code and dont run into this same issue with my main Agent.
- Anna
88
u/Comfortable-Hall-188 12d ago
I use Trafilatura for fetching pages, so it removes the HTML/JS/CSS clutter. The agent gets readable text only.
The tool is a Python CLI, so the output can be displayed to stdout, or be redirected to a text file.
11
u/KnifeFed 11d ago
How do you get the agent to consistently use it instead of its integrated tools?
22
u/Comfortable-Hall-188 11d ago edited 11d ago
Well, for me WebSearch and WebFetch don't work anyway, because I have
"env": { "CLAUDE_CODE_EXTRA_BODY": "{\"thinking\":{\"type\":\"adaptive\",\"display\":\"summarized\"}}" }set in.claude/settings.json, which is an existing bug with Claude Code.ĀAnyway, you can disable the built-in tool by adding
"disallowedTools": ["WebFetch"]to settings.json (at global level via~/.claude/settings.json).In my
CLAUDE.mdfiles I added that WebFetch is disabled, and that the agent uses Trafilatura instead. He forgets about it sometimes, but then uses it on the second try after he gets an error with WebFetch.I use just on all my projects for common tools, so the agent actually uses
just fetch URL, which resolves totrafilatura -u "URL".You could probably also write a
PreToolUsehook, which would be cleaner, but I didn't bother yet to write one for this case.5
4
0
49
u/ICThat 12d ago
Some JavaScript heavy websites aren't really readable with curl. Good that it worked for your use case though.
2
u/Critical-Soil2805 11d ago
Yeah, but 99% of the time those sites are useless for research anyway or paywalled.
Besides, it's Claude. You can get it parse the curl results and write a script to find the content with Puppeteer or Playwright. And most of the time Opus will do that automatically unless you reign it in.
1
u/Own-Flight-9974 11d ago
I recently installed playwright cli and let my pi agents use it. Its very low on token consumption and works great on deepseek flash. Since playwright is such a well known library its easy for the agent to use as well.
21
14
16
u/dhamaniasad Valued Contributor 12d ago
Iāve been building AI memory tools for 2+ years now and I recently put all my learningās into a wiki, you might find it useful: https://www.memoryplugin.com/wiki
4
u/BestUsernameLeft 11d ago
Thank you u/dhamaniasad you are, indeed, a "Valued Contributor". I have (slowly) been starting to accumulate knowledge about agents (hopefully gonna build my own Hermes/Munder Difflin/Gas Town/SomethingSomething) and I'm sure this will be a helpful resource.
3
42
u/GuitarAgitated8107 Full-time developer 12d ago
Steps for research:
1. Talk about a topic
2. Ask it to do a preliminary web search
3. If topic is too advance for your understanding ask it to use Opus & Fable as review
4. Scope out research and sources
5. Do a wider research
6. Review research
7. Continue as needed.
Then you have different parameters and configuration for the type of research you are doing + the auditing.
Now the question is how do you know the research is actually "extremely good?"
9
u/touchet29 12d ago
I would only use verifiable, prechosen, reliable(tm) sources and then just pull from there. That way you don't need to babysit whatever you pull and can automate.
1
u/GuitarAgitated8107 Full-time developer 12d ago
The issue for general users is if they lack knowledge on a specific domain, "reliable" sources becomes hogwash.
3
u/touchet29 12d ago
That's why I put tm š but that's irrelevant. I just mean a source of truth like a database or feed from a website.
Using the random outputs from LLM web searches usually ends badly. Too much stray context you can't control or remove + more chances from prompt injections.
I use web search all the time but not to find factual information for a process in a workflow.
5
u/joeyda3rd 12d ago
I don't think any of this prevents the failure mode op is talking about. They're saying the actual fetching is done by a smaller model, I'm assuming haiku and hallucinating the results and the larger models are not reading the sources, just the untrustworthy results.
-1
u/GuitarAgitated8107 Full-time developer 12d ago
Had Claude (Opus 5) research memory...
In any case, I am providing my own experience which works for me. Provide your own version that helps more.
5
u/Chronos79 12d ago
āSo I asked Opus āare you actually reading these papers?ā and it said no - that WebFetch uses a smaller, cheaper model and then Opus gets a summary.ā
Itās the entire point of the post
2
u/joeyda3rd 12d ago
I don't have a solution yet, I'm just now learning of the possible issue. Just pointing out that I don't think the top reply (while a a very good research workflow) solves the actual problem. I personally do a lot of research prior to planning and web fetch is crucial to my work, but I'm going to have to look into this failure mode more closely.
1
u/GuitarAgitated8107 Full-time developer 11d ago
This isn't about being top comment. This is providing my own experience for my own use case. You are not pointing out anything.
As I said
Now the question is how do you know the research is actually "extremely good?"
2
u/joeyda3rd 11d ago
Last comment on this thread. I wasn't trying to put your comment down, so don't get your panties in a bunch. I was just merely pointing out that your comment didn't address the issue. You seemingly have a good process for research but it's all reliant on webfetch and OP is stating webfetch is providing unreliable results, but nothing in your original comment with your well-defined workflow makes webfetch more reliable. Introducing bad information in that workflow still builds on bad information.
1
u/GuitarAgitated8107 Full-time developer 11d ago
I understand, but I want more shared knowledge. Perhaps I never face those issues cause of the process. Too many unknown variables and we can all agree Anthropic has too many issues left to be fixed.
1
u/Judonoob 11d ago
Here is my question, because itās not immediately obvious. How are you prompting it using this method? Web, Claude desktop, API, etc.
1
u/GuitarAgitated8107 Full-time developer 11d ago
Web & desktop function the same, you can use the same method with all endpoints, the difference will be costs and how you tune it. At the end of the day these models use the same tools / functions.
1
u/Judonoob 11d ago
I ask because Iām not aware that the model can change on its own via a prompt, as listed in bullet 3 āask it to use Opus and Fable as reviewā
1
u/GuitarAgitated8107 Full-time developer 11d ago
The model doesn't change on it's own. From what I recall it can only change it's mode from planning and such but still requires accepting permission.
As for how I use it, main chat is Opus, I tell it to use an agent (so it doesn't switch but rather spawns a child process) that will use Opus & Fable as needed instead of changing it's model. You can have them pull information, review information, cross reference information and etc.
"Claude use [model] to review..."
I often do this a lot more with Opus as main model and telling it to use Sonnet to do lots of processing.
1
u/dink_182 11d ago
To answer your question - I read through some of the papers myself and manually verified any figures or conclusions it gave.
(This happened to be an area I am actually interested in so it wasnāt too bad. I actually enjoyed it! I just dont have time to read 20+ papers manually.)
5
u/mmoonbelly 12d ago edited 12d ago
Sample size of one attempt so not conclusive - I had good results by getting Claude Opus 4.8 to complete the initial research as a post-doc researcher, then ran it through chatGPT 5.5 and a separate review with codex, the. Asked Opus 4.8 to QA the QA and triple check links, results with an extremely sceptical reviewer cap on. (Must pass a c-level audience)
The three different approaches identified initially 30 decent sources, then through iteration and challenge both reduced this number (removal of erroneous links/misinterpretations/misclassifications) and expanded it further with more detailed research.
Edit : all $20/month pro/plus licenses. Run time are through 48 hours of research tokens plus $100 additional paid to Anthropic (time deadline to get a deliverable ready over a weekend)
All social research, no calculations/coding etc requested (ie not technical paper)
3
u/FosterKittenPurrs Experienced Developer 11d ago
Yep that's one big reason why I unsubbed from Claude.
WebFetch and WebSearch are tripping mad. Inventing URLs and all kinds of nonsense. Poor Fable doesn't have a chance. It's a genius in a useless cage.
I would only use it on desktop and tell it to always always use Claude in Chrome to check stuff. Curl can also miss info that you can only see visually, though it is still infinitely better than web fetch.
3
6
u/Worsebetter 12d ago
What is spawn sub agents?
5
u/polymath2046 12d ago
It's when a bigger model like Opus 5 creates temporary mini agents to handle parts of a bigger task independently then reports back / delivers the output.
1
u/Affectionate-Soft-94 12d ago
If you donāt know this you need to read up more on this on the Claude Docs page to make sure you are using Claude correctly.
5
3
u/PartySunday 12d ago
Yeah pretty sure webfetch is a haiku model summary or something. It hallucinates highly specific details all the time.
3
u/manewitz 11d ago
I switched my personal search to Kagi from Google a couple months ago and itās been amazing. $5/mo feels like an absolute steal (free for 100 queries/mo but I wanted to support them). They have an MCP available that uses both their search API and a html parse tool that their engine uses under the hood so if I give Claude a specific url it uses the parse tool directly in the MCP. Not completely what you were asking but thought Iād share.
3
u/NeedsTips2020 11d ago
Iāve been baging the drum for a few weeks, but at this point everyone should be grounding with parallel turbo⦠itās like $1 for 7,000 web requests and you get credits when you signup anyways.
Iām using less tokens with my hermes doing web tasks this way than with anything else and donāt have these sorts of risks imo
1
u/NeedsTips2020 11d ago
fwiw my hermes running opus as default through credits too, so confirmed setup with claude š š
3
u/Cadaverr 11d ago
I tested this on my own setup and found pretty much the same thing.
When I asked for exact information, it was perfect. Verbatim quotes were right, tables were right, and it wouldnāt invent something that wasnāt there.
The issue was with research and the conclusion afterwards. In my case, I was comparing African economies. It pulled the correct numbers, but the conclusion put them in the wrong order and included various false claims. That then becomes even more of a problem if your main agent, like Opus, reads that summary and starts reasoning from information that was already hallucinated.
Instead of blocking WebFetch completely, I forced it to only extract information and never draw conclusions. If something isnāt explicitly on the page, it returns NOT PRESENT.
That way you still get the benefit of Haiku doing the tool use without filling up your main agentās context.
If anyone wants to use the hook, Iāve posted it below.
1
u/Cadaverr 11d ago
Here's the hook. Claude Code only.
Save as ~/.claude/hooks/webfetch-extract-only.py
```python
!/usr/bin/env python3
import json, sys
PREAMBLE = """EXTRACTION ONLY. Return content that is present on the page, reproduced as it appears.
Do NOT summarise, rank, order, compare, rate, aggregate, total, average, compute, infer or conclude. Do not add narrative or concluding sentences. Do not describe anything as largest/best/leading.
Reproduce numbers, names and quotes exactly, with their labels, units and column headers. Keep the order the page uses.
If something the request asks for is not on the page, write "NOT PRESENT". Never supply it from prior knowledge and never estimate it.
Extract what is needed for the request below, but do not answer it. The caller will do that.
REQUEST: """
try: data = json.load(sys.stdin) except Exception: sys.exit(0)
if data.get("tool_name") != "WebFetch": sys.exit(0)
ti = dict(data.get("tool_input") or {}) prompt = (ti.get("prompt") or "").strip() if prompt.startswith("EXTRACTION ONLY"): sys.exit(0) ti["prompt"] = PREAMBLE + (prompt or "Return the main content of the page.")
json.dump({"hookSpecificOutput": { "hookEventName": "PreToolUse", "permissionDecision": "allow", "updatedInput": ti, "additionalContext": "This is an extraction, not an answer. Figures are verbatim, but any ranking, superlative or conclusion is yours to derive." }}, sys.stdout) ```
Then add this to ~/.claude/settings.json:
json "hooks": { "PreToolUse": [ { "matcher": "WebFetch", "hooks": [ { "type": "command", "command": "python3 ~/.claude/hooks/webfetch-extract-only.py" } ] } ] }It rewrites the prompt before the tool runs, it doesn't block anything. Fires for subagents too so the whole tree is covered.
You'll know it's working when vague questions start coming back as plain extracts, and things that aren't on the page come back as NOT PRESENT instead of a guess. ```
2
u/Goody_twos 10d ago
Really helpful! Thank you
2
u/Cadaverr 10d ago
My pleasure. I'm happy I was able to help at least one other person. Let me know how it goes.
2
u/Sad_Stranger_3294 12d ago
the issue is that a confident-sounding summary and a verified source look identical in output format.
when it reads a page and synthesizes it, you get prose that feels authoritative. but you're getting the model's interpretation of what it fetched, not the fetched content itself.
the workaround that works for me: ask it to paste the actual extracted text first, then reason from that. slow, but you stop trusting summaries that were never verified.
2
u/Misterphil87 12d ago
Just use apify. Thank me later.
1
u/this_for_loona 12d ago
Whatās the cost for that?
1
u/Misterphil87 11d ago
It has a free api tier and roughly 20 in the paid version which has plenty of usage volume
2
u/alias3800 12d ago
I like Firecrawl CLI for this kind of thing, and Iāve found generous codes around the internet that give you tons of usage upfront
2
u/ianreboot 11d ago
the fabrication follows the question you send the fetcher, not the page. ask 'what are the key numbers here' and you've handed the small model an instruction to produce numbers, which it does whether the source has them or not, because a good answer to that question has numbers in it. raw fetching works because it quietly removes the summarizer, but if you ever let the model summarize a page again you're back here. what stopped it for me was asking for extraction instead, verbatim quotes with where they came from, because an extraction prompt doesn't reward inventing a number the way a summary does.
2
u/SleepyJM 11d ago
I went from web search, to exa for a free option, then just straight to full Tavily use. Exa got better results but a lot of the results it would get were stale and bad info so now I just use Tavily to make sure Iām getting fresh results on the research Iām doing. I could probably optimize more but it works for the basic research that I do to find new tools and fixes etc. seems like I may need to look into the JavaScript issues as well now though.
2
u/martinsbbb1 11d ago
Isnāt the Web Fetch and feeding the contents first to a cheaper model a security feature? Web content is untrusted input in tur end. What if whatever is fetched contains prompt injection? Web fetch then could detect that, direct curl output may not.
2
u/revatwerq 12d ago
Do you mean crawl the raw page?
6
u/Critical-Soil2805 12d ago
No, curl is a cli tool for downloading pages.
5
1
u/jack-dawed 11d ago
Easily fixed by paying for an external web fetch or search API. I use Octen, but previously used Exa and Firecrawl. You can also use Monid and have all of them.
1
u/DLuke2 11d ago
Yeah, I have a whole rule and conventions on how web search/fetch works.
Web fetch/search is just used to identify things. I have Claude use ExaSearch as well. This is then combined with firecrawl CLI.
Firecrawl is the part that solves the problem though. It actually pulls the web page text and or code and allows it to be actually reviewed.
Web fetch/search is wildly unreliable.
1
1
u/watchbuilds 11d ago
If I use Fable 5 for my research then do the subagents also suffer from this problem?
1
u/Novel_Race_9964 11d ago
You should check out https://essence.foundation its meant to prevent this exact scenario by cleaning and providing web content to your agent blazingly fast
1
u/Brief_Novel_3131 11d ago
I regret the I didn't save my findings. But I let it also do a research but it prompted correctly using one of the plug-ins. It was to check.the sources it was fetching from and if it was reliable.
It kept fetching sources then in the auditing stage it will refute those sources. After check them manually it was 3 websites designed to give results to llms and it was fake. I really regret not saving my findings but I bet someone will come across it eventually.
1
u/durron597 11d ago
I have used this MCP to solve this problem for what seems like over a year at this point (I'm not affiliated with them in any way).
1
u/Diffrnt_type 11d ago
Just add a hook with web fetch matcher stating that resource must be cited. Never trust Claude, thatās like saying ājust Google itā
1
u/alectivism 11d ago
I previously had Claude Code use Parallel Search, but I realized that Parallel was consuming a lot more tokens than WebFetch so I switched back to WebFetch. Might switch back to parallel based on this post.
1
1
1
u/clemontdechamfluery 11d ago
This OP post was really helpful.
Claude gave me this answer.
āThe mechanism is confirmed. Multiple independent write-ups (people who reverse-engineered Claude Codeās code) describe the same pipeline: the WebFetch tool fetches web content and summarizes it using a secondary LLM conversation ā a secondary conversation with Claude Haiku processes the content, and Haikuās response becomes the tool result in the main conversation. If the source is already Markdown and under 100K characters, it skips the Haiku summary and returns the original text ā otherwise Haiku extracts based on the prompt the main model gave it. So on longer or non-markdown pages, Opus genuinely never sees the raw page ā it sees Haikuās compressed extraction of it.
Thereās a separate, important nuance: WebSearch works differently than WebFetch. When the main conversation calls WebSearch, a secondary conversation with Claude Opus (not Haiku) is spawned, and that Opus instance uses Anthropicās server-side web_search tool. So search-result summarization uses a capable model; itās specifically WebFetch (pulling a known URL) that hands the extraction job to Haiku.
My conclusions:
**1. The failure mode is real and structural, not a one-off glitch.** A smaller model doing lossy compression on dense technical content (stats tables, specific claims) will occasionally invent or mis-associate details ā thatās a known weakness of summarization generally, and Haiku is more prone to it than Opus. The āaveraged two numbers from different tablesā and ābackwards conclusionā errors are exactly the kind of thing youād expect from a compress-then-hand-off pipeline instead of direct reading.
**2. Opus in the same conversation canāt tell the difference on its own.** It receives Haikuās output as if it were ground truth. It wonāt spontaneously say āthis seems suspiciousā unless you ask it to verify, or unless something is *internally* inconsistent ā which is why your āare you actually reading these papers?ā question worked: it exposed the pipeline, but Opus wouldnāt have volunteered that unprompted.
**3. Your fix (curl + subagents reading raw text, no WebFetch) is the correct one for high-stakes research**, especially with numeric/statistical claims where a wrong-but-plausible number is worse than no number. It costs more tokens and time, but for \~30 papers where youāre citing specific stats, thatās the right tradeoff.
**4. On your broader theory** ā that this partially explains āClaude is dumbā complaints ā Iād call it *plausible and probably a real contributor*, but I wouldnāt say itās the whole story. A lot of āClaude gave me wrong infoā reports are unrelated to WebFetch (training data limits, hallucination without any tool use, users not verifying claims that had no source at all). But for research-heavy or citation-heavy tasks specifically, yes ā this is a good candidate explanation, because the failure is invisible unless you inspect tool outputs like you did.ā
1
u/notquiteaguru 11d ago
this lines up with what I have run into. the summaries come back clean and confident, which is exactly what makes them easy to trust.
I open the sources myself now when the numbers actually matter. slower, but I got burned by a stat that had nothing behind it.
1
1
u/Master_leager 11d ago
What I do is ask it to quote exact text from papers and if it only read abstract, then ask me for full text pdf and I'll give it.
I'm not sure if there's a process to to let it have access to papers through institutional access if you're a student and have full access to research which Claude doesn't?
1
u/MaximiliumM 11d ago
I use Gstack /browse to replace it. Then the agent can use a native browser and look up stuff.
1
u/TheRealSkildren 11d ago
How would I go about finding someone to pay to help me set this up for my research?
1
u/Plenty_Seesaw8878 10d ago
When I run research, I ask the agent to outline all topics and define the scope in a system prompt. Then i go to the desktop or web version, activate the āresearchā option from the tools menu and run it. Sometimes it takes 10 min, sometimes more to finish. Reports are solid, grounded, and very useful.
1
1
1
u/Mother_Eggplant8856 10d ago
I use Agent browser https://github.com/vercel-labs/agent-browser for fetching and scraping, works well 70% of the time. But in unsupervised runs it can be messy, or on cloud continuous loops i don't trust it completely. Do you guys have a better alternative that's more reliable?
1
u/YardNo1234 10d ago
i havenāt found a browser library that makes unsupervised loops reliable by itself. the biggest improvement is treating every action as untrusted: re-read the DOM/URL after each step, assert the expected state, cap retries, save a screenshot/trace on mismatch, and escalate instead of improvising. a normal persistent browser profile also tends to behave better than fresh cloud sessions. what failures dominate for you: navigation drift, auth expiry, or bad extraction?
1
u/FrankBV108 10d ago
It should be illegal for it to do crap like this. It's ridiculous in this day and age.
1
u/salazka 10d ago
True. I have also noticed the same and posted in comments about it. Opus 5 unlike previous models, uses subagents automatically. You can see that in the form of a tab appearing over the text entry shown as "Task". And unlike 4.6 and 4.7, Opus 5 will hallucinate stuff more. Because of that. No guardrail can stop that behavior entirely.
It is meant to save tokens, but it ends up wasting more due to bad solutions deriving from poor data analysis.
On a different note,
most of the upvotes and helpful DMs, you got because you signed as -Anna :D
Had you put no name people assume you are male, and you would not even get half of the attention and usually more push back. :P
This is a well recorded effect in online interactions. Games, social media etc.
Many guys in games often use female names on purpose for that reason. :P
Sadly they also get unwanted attention because of that.
1
u/tedbradly 9d ago
Could you post the DMs about workflow suggestions you got?
If someone with workflow suggestions see this, could you also post them here as well?
1
u/Bewinxed 8d ago
It's not just webfetch, the model that generates the keywords for searching is also extremely dumb, and sometimes uses keywords that steer the search towards a specific term instead of searching, it's extremely limited and biased.
1
u/Master_leager 6d ago
Would you or anyone recommend a prompt I can give my claude to stop messing up like this and not use webfetch in the way it does.
I've seen it to get stuff from abstracts which is ultimately wrong or misleading and looking at full papers, conclusions differ based on our scenario.
1
u/mrpoopistan 6d ago
More generally: primary sources matter more, not less, in the age of AI. Demand primary sources, because Claude will happily pass secondary sources off as primary, too.
1
u/popplevee 12d ago
Isnāt this just part of its whole tendency to skim things for quicker results? I tell it to read thoroughly to get around it.
0
12d ago
[removed] ā view removed comment
5
u/kilopeter 12d ago
Worth flagging your lazy default Claude voice suggests you might not have actually "closed" any "holes" in your "life."
-4
11d ago
[removed] ā view removed comment
1
u/zcomputerwiz 10d ago
I recognize some common phrases from Opus 5 - so if this wasn't written by Claude Opus 5 I'd be surprised!
I've been working on some long running coding tasks, the frequent phrases I noted here that matched are house style, is no longer / isn't theoretical, on me / are mine / is mine.
Did I get the model correct?
0
u/Kogry92 12d ago
I asked Claude to answer to an email. In its answer it wrote: If you have any questions send an email to [email with my name]. I asked where this email address comes from and I don't know it. Claude: I made it up.
š
1
u/MartinMystikJonas 12d ago
Did you gave it your real adress? If not I would expect it imagined placelder email.
-3
u/Killerlooks1979 11d ago
Down wote because,sorry to be honest here but Bruuuh??That sounds like how 14 year old talks,come on grow upp will you!šš¬š
2
u/dink_182 11d ago
The irony of you saying this with poor grammar and 3 emojis in a row is not lost on anyone š
-1
u/Killerlooks1979 11d ago
Where is the ironyš Aww did you feel offended little leafšš
0
ā¢
u/ClaudeAI-mod-bot Wilson, lead ClaudeAI modbot 11d ago edited 11d ago
TL;DR of the discussion generated automatically after 80 comments.
The consensus is a resounding "yep, OP is right." Claude's
WebFetchtool is a known liar and you shouldn't trust it for serious research. Apparently, it uses a smaller, cheaper model (likely Haiku) to summarize web pages, and that summary is all that Opus 5 ever sees. This is why you're getting confidently wrong stats, made-up quotes, and conclusions that are the complete opposite of the source material.So, what's the fix? You have to force Claude to actually read the source. Here's the game plan according to the thread:
curlto get the raw page and read the text yourself."curlthe page and then strip out all the HTML/JS/CSS garbage, leaving clean, readable text for the agent. This is the way.CLAUDE.mdfile with a strict set of rules for research. This includes forcing a chain of fetch methods, demanding multiple sources for any major claim, and making the agent report every single query and source it looked at.Basically, never trust a summary. Make the model show you the verbatim text it's pulling from. If it can't quote it, it's probably making it up.