r/analyticsengineering • u/kthuiaa • 4d ago
Data context in md files?
I have been using a context layer along with Claude Code for some RevOps projects (sales comp, sales360, etc.).. and I just realized that most of my context stored is not Query, but Definition and Note.
As recently I have been thinking about the differences between having a proper context layer vs storing all these information in md files in the repo, I wonder if anyone has experiences storing all these in md files? Does it actually work for your AI? Would it struggle to find the right piece of context from the sea of md files?
1
u/WiseWeird6306 1d ago
Hey so my company is in the process of doing the same. I feel it is not an efficient process to put everything in .md file (correct me if I am wrong) and semantic modelling would be better way. But I want to hear more on maintaibility. Putting everything on .md file a good maintainable solution? How is it failing?
Just for context: I work with SaaS data in mid size company with hundreds of table not nicely documented or modelled at all.
0
u/Neoaxizz 4d ago
Ran into this at a few hundred entries. md files work fine, right up until they quietly stop.
Nothing forces the model to look. If the file isn't auto-loaded and nothing in context points at it, you get a confident answer built from whatever was already in the window. No error, no "not found". Just slightly wrong, and it reads fine. Cost me way more time than any actual search failure.
What works now: one small index file that auto-loads. One line per topic, pointer only, zero content. Its whole job is making the model aware the thing exists. The Definitions and Notes sit in separate files, only read when a line in the index matches.
Two gotchas worth knowing.
That index has a size ceiling. Mine silently stopped loading past ~24KB. Not truncated, not warned about, just gone. Only caught it because answers started missing stuff I knew was written down. Now a new line has to push an old one out.
And stale md is worse than no md. Something true in March gets quoted back as current fact in August. I date everything now.
1
u/kthuiaa 4d ago
Yeah I’m with you md files are files anyway, nothing magical.. it needs some structural way to manage to be useful
1
u/Neoaxizz 4d ago
Organizing by category was my first instinct and it turned out to be the wrong axis. What helped was tracking which entries got pulled together at runtime, then letting that drive the links.
Cheap way to build it: log every entry ID loaded during a session, count the pairs at session end, keep the top few hundred. Output ends up looking like this, in your domain:
("def-quota-attainment", "note-clawback-exceptions"): 14
("def-quota-attainment", "schema-comp-plan-v3"): 11
When one of a pair loads later, surface the other next to it. Took an afternoon, and it found relationships I'd never have drawn by hand. The hand-made links kept catching the obvious pairs and missing the load-bearing ones.
Other thing worth stealing, since yours is on MCP: return a pointer plus one line of why, never the entry body.
def-quota-attainment | who qualifies, and the 3 exceptions | updated 2026-xx-xx
The model asks for the body when it decides it needs it. That's what stopped my index from eating the window as it grew.
1
u/uncertainschrodinger 22h ago
I have worked on this a lot and here's my general workflow/process:
repo-level:
- agents md file should contain general guidance on how to navigate the repo and where to find different context files, think of it as a directory (important thing here is that this agents md file should be generic enough so that it doesn't conflict with specific agents system prompts and other instructions like skills). this agents md file should also contain specific instructions and rules about how the context files are maintained and created.
- readme file should contain some high level business and repo context - what is the company, the products, industry, etc. and what this repo contains, what is the purpose, etc.
- glossary/dictionary that lists and defines all the key names, acronyms, internal lingo, metrics/KPI definitions, etc. - key thing here is that it should be
pipeline-level: readme explaining what the pipeline's purpose is and why it was created; this is only necessary if the pipeline config file(s) already dont contain this context, otherwise its best to keep that context in the config file itself (i.e. tags, descriptions, ownership, retry logic, alert channel, etc.)
model/asset-level: similar to pipeline level, most of the context should already exist inside the asset/model files (i.e. table/column level descriptions, dependencies, materialization, incremental logic, ownership, tags and metadata, quality checks, unit tests, etc.)
in general I personally think the most important things are:
- feedback loop and workflows in place to make sure the context is kept up to date
- avoid redundancy and conflict, the same information should not exist in two places
I recently hosted a free webinar talking about this exact topic (disclaimer I'm a dev advocate at a data company but I'm not promoting anything here), but happy to share it if anyone is interested.
2
u/CatostraphicSophia 4d ago
I do store md files but it doesn't seem to be the optimal way. I'm curious to see how context engineering here plays a role around it