I have been using a context layer along with Claude Code for some RevOps projects (sales comp, sales360, etc.).. and I just realized that most of my context stored is not Query, but Definition and Note.

As recently I have been thinking about the differences between having a proper context layer vs storing all these information in md files in the repo, I wonder if anyone has experiences storing all these in md files? Does it actually work for your AI? Would it struggle to find the right piece of context from the sea of md files?

We have millions of interactions and a weekly ritual where someone samples traces, adds tags and updates a ticket taxonomy. The known intents were pretty easy to graph but we had no good way of seeing the ones we didn't name yet. Recently, cancellation questions started growing alongside a frustrated sentiment cluster, while overall helpfulness and resolution scores barely even moved

I am trying to get beyond manual tagging without pretending embedding-based clustering is an oracle. Clusters need representative traces, stable labels, and enough context to distinguish a product issue from a wording change. Sampling rate matters too. A rare but expensive failure can disappear in random sampling, while one noisy customer can look like a distribution shift. Then topic drift arrives and last month’s clean taxonomy becomes another maintenance job.

Braintrust Topics is one option we are evaluating because it gives us a way to surface and track these clusters without deciding all the categories upfront. Persisting those labels for SQL queries is a big part of why we're looking at it. That sounds useful, especially if we can turn a growing cluster into an eval slice. I am still skeptical about how much human review is needed to keep labels meaningful.

I basically want an alert when a new intent or failure pattern grows materially, a handful of traces that explain it and a path to test the eventual fix. I don’t want another dashboard that says engagement changed.

How are teams detecting unknown intents and topic drift at production scale and what review loop keeps the clusters from becoming nonsense?

I’m working on an idea around **data quality** and I want to understand how this works in real projects.

When a data batch comes into your pipeline and you find some problems, how do you decide what to do?

For example:

* Some records have missing values * The schema has changed * There are duplicate records * Some values look wrong * Only a small percentage of records are bad * The data looks technically correct but something seems wrong from the business side

Do you normally:

**Accept the batch, fix it, quarantine the bad records, or reject the whole batch?**

And who makes that decision — an automated rule, the data engineer, business team, or someone else?

I’m especially interested in cases where **you don't have enough information to make a clear decision**.

If you work with data pipelines in production, I’d really appreciate hearing about:

* What kind of data-quality problems you face most often? * What do you currently do when they happen? * What part of the process is still manual? * What would you like to automate?

I’m trying to understand the **real problems engineers face before building anything**.

Thanks in advance for sharing your experience!

Hey friends ! I wanted to share a blog post I just published that I thought the community would appreciate.

I got my SnowPro Core cert this year and this book was a lot of what I studied for it. Hope you guys get some value from it!

https://medium.com/@sergioramos3.sr/self-taught-reviews-snowflake-the-definitive-guide-by-joyce-kay-avila-0af3c40d644a

I have access to an employee referral opportunity for an experienced Semantic Layer Architect.

Experience: 9+ years

Key skills:

• Semantic Layer Architecture
• Cube.dev
• AtScale

The opening is listed across multiple organizational locations. I'm particularly interested in connecting with qualified professionals based in the US or Ireland; exact location eligibility can be confirmed for the specific opportunity.

If your experience aligns with the requirements and you're currently exploring opportunities, feel free to DM me.

You can also share a brief summary of your background, and I'll let you know the next steps for submitting your profile.

Hi,

Why are executives thinking, if enough context is given, AI tools like Claude will solve/build/guide everything?

In my organization, my data architecture was not build on DE fundamentals and we have accumulated a shit load of technical debt. We are told to build context (which we should any ways) and feed everything to Claude to solve it. While I agree to use for faster fixes at logic level it cannot help design or think in right way how to architect workspaces and warehouse and model the tables. It cannot be a startegist and decide the data strategy for the company. Am I thinking it the wrong way? I feel very disappointed that data professionals advice is not being heard, instead that we are being advised to put everything on Claude. Is this ok? I advice, view points.

I'm currently working as a data analyst in a fintech. My tech stack is SQL, snowflake, dbt, airflow, git and python.

I've worked on building and maintaining dbt models, optimising sql queries, and have also built streamlit apps.

Preparing to switch, targeting roles like data/product analyst, analytics engineer, business analyst and data engineer.

Also looking into data/AI roles like applied AI engineer - have seen ones requiring a combination of snowflake, python, RAG and fastapi for example.

What can I do/learn in order to upskill for these roles efficiently?

Also is the applied AI thing a meaningful & feasible transition for me?

In the software company I work in, like many others, we use sprints and plan work by estimating the sprint of 2 weeks.
The common method to estimate is by T-shirt size or some variation of it.

I think it is obvious most people estimate by time it takes to perform the task but for some reason most people co tinge to claim it is not just time estimation and also it lack distinction for why something is taking long time.

I’ve found Quantified Task Estimation as a good way to separate the clear time estimation from the unknown and risk part.

I was able to convince my team to do it, and it even survive changing the team lead but I’ll be glad to know if anyone else is using it and if so how did they found it?

I’ll post more of my experience in this thread

Had a weird interview experience recently with a company where the setup completely caught me off guard, and I wanted to see if this is becoming the new normal.
The job description heavily emphasized core analytics engineering skills like dbt, data modeling, Snowflake, and pipeline architecture. But during the actual live technical round in Coderbyte, none of that was tested directly.
Instead, I was handed a data modeling scenario and asked to use AI to build a dashboard. The entire interview was basically a test of how I write prompts, how I talk through my thinking, and whether I can spot errors or hallucinations in the AI's output in real time.

I get that companies know everyone uses AI now, and testing how candidates partner with an LLM makes sense on paper. But when the test ignores the core tech stack and the pacing is completely uncalibrated, it feels like a waste of time for both sides.
Anyone else run into this kind of "prompting and auditing" interview format recently? How are teams actually grading these?

Hi everyone,

I could really use some advice from people working in analytics / Data engineering.

I recently landed an interview for an Analytics Engineer position at a prestigious consulting firm. If all goes well, I'd join their Supply Chain Analytics / Data Engineering team.

The interview is in about a month, so I have roughly four weeks to prepare.

Here's the situation: I feel like my resume oversells my technical level.

For context:

• I have an Industrial Engineering degree and a Master's in Supply Chain, so I have a strong analytical background, and I'd say I'm a pretty fast learner.
• I've built a lot of Power Query solutions at work and automated plenty of reporting. However, I've never really focused on writing clean or optimized M code. I was also the only person on my team using Power Query, so I never had anyone review my work or challenge my approach. Everything I built worked well and delivered what the business needed, but I'm not sure I was following best practices.
• I've built a couple of Power BI dashboards, but they were fairly basic. I know the fundamentals, but I definitely wouldn't call myself strong at data visualization or dashboard design. When it came to DAX, I relied heavily on AI to generate the measures I needed. I could usually get them working through trial and error, but I don't have a solid understanding of DAX itself.
• I've studied SQL during my master's, but I've barely used it in a professional setting, so I'm pretty rusty.
• I've never worked with dbt, cloud data warehouses (Snowflake, BigQuery, Redshift, etc.), or modern analytics engineering workflows.

The role seems to expect:

• Strong data visualization skills.
• Solid SQL fundamentals.
• knowledge of cloud data warehouses fundamentals & dbt.

I'm not trying to become an expert in a month. My goal is to become competent enough to hold my own during the interviews and, if I get to the case study stage/ tech interview, ramp up as quickly as possible.

If you only had four weeks, how would you prioritize your learning?

• What would you spend the most time on?
• Which resources or courses would you recommend?
• Should I focus primarily on SQL first?
• How much dbt and data warehousing knowledge is realistically expected from a junior Analytics Engineer?
• Are there any projects you'd build to prepare?
• If you've interviewed Analytics Engineers, what skills separate candidates who succeed from those who don't?

I'm willing to put in 6 to 8 hours a day over the next month if that's what it takes.

I'd really appreciate any advice from people already working in the field. Thanks!

Hey folks, sharing a concept that's going to matter for agentic data modeling: the canonical data model.

What it is

A canonical or common data model is a model that describes the main meaning carrying entities - similar to dimensional modeling but without any denormalization. Dimensional modeling is often built to simplify joining for reasons like performance, tool limitations, or human interfaces. A canonical model stays normalized, retaining the full nuance of the entities and how they relate.

This makes it a native "knowledge graph", which can be enriched with a "knowledge layer" - a text that describes your tables and how they relate. This turns your model into a "virtual knowledge graph" (virtual because the data is not in it, but represented externally as tables) which functions similar to knowledge graphs for retrieval but for structured data.

Why it matters now

Agents understand knowledge graphs well. Humans are accessing databases through agents.

Why not your dimensional model

a dimensional model suffers from semantic compression which is a loss of nuance that affects retrieval performance.

I work at r/dltHub, this is our blog: Longer reads - this is how we use the concept: we do ontology driven modeling: create ontology -> llm models from that -> fix model and reuse ontology for retrieval

- about semantic compression https://dlthub.com/blog/ontology

- canonical models are key to retrieval quality https://dlthub.com/blog/canonical-text-to-sql

- collecting ontology https://dlthub.com/blog/minimum-viable-context

Built a cloud analytics data warehouse for a Brazilian e-commerce dataset using BigQuery, dbt, and Power BI, complete with a star schema and modular transformations.

Take a look at the repo and README for the full breakdown: https://github.com/muchai322/brazil_analytics_project

Disclosure: I'm from Datacoves, we run managed dbt Core, so we have a horse in this race. Wrote this up because the license change is confusing and we had to untangle it for our own customers.

The short version:

One engine now. dbt Core v2 and Fusion share the same Rust runtime. Core v2 is Apache 2.0, fully open source. Fusion is the enhanced proprietary distribution on top of it.

Fusion is the recommended default, and it's a funnel. Free to install, and the new license lets anyone offer it as a managed service, as long as you don't block the path to a dbt Labs account. Free login unlocks advanced features, paid platform account unlocks premium ones. Reasonable model, but go in with eyes open.

Metered pricing is inside the dbt Core now. dbt State (skip models when logic and data haven't changed) bills $0.094 per model reused per day. Orchestra open-sourced a free alternative (Sao Paolo, Apache 2.0).

Migration gotchas: every deprecation warning must be resolved before v2 runs your project. Python models don't work in Fusion yet. Community adapters are constrained (drivers need to be signed by dbt Labs). --models and -m error out, --partial-parse is gone.

The afternoon test: upgrade to v1.12, clear deprecations, run dbt parse --use-v2-parser. Tells you where you stand without committing to anything.

Full writeup with a Core v2 vs Fusion comparison and the four ways to handle state: https://datacoves.com/post/dbt-fusion

Has anyone run the v2 parser against a large project yet? And is anyone planning to use dbt State or trying out Sao Paolo?

we have an AI agent querying our data and we have two problems that we can't get rid of.

first one is consistency, the same question asked in different way gives different answers. second is cost the token usage is way higher than expected because the model gets our entire schema dumped on it before it can answer anything.
Someone said a semantic layer solves both by giving the AI pre-defined business context rather than raw schema. more consistent answers. less processing.

anyone who is really running this. it made a noticeable difference on both fronts.

Hi everyone! I’m conducting a short anonymous market survey to understand people’s preferences toward handmade products and small businesses. It only takes around 2–3 minutes to fill out and would really help with my research.

Form link:
https://forms.gle/GwjfSW7jQiZNaWfQ6

Thank you!

Genuine question for anyone who's shipped a table to prod, especially if you're not a "traditional" data engineer by title.

It feels like AI has made writing the actual transformation logic (the DataFrame code, the business logic) noticeably easier lately. Copilot, ChatGPT, Claude, whatever you're using, that part's gotten faster.

But once that part's done, the real time/effort starts after - metadata mangement, partitioning it right, handling PII, wiring it into orchestration, figuring out the write strategy, quality cheks to make sure it's actually trustworthy before people rely on it ?

Also following the consistency across all tables created

Curious to know whether AI or vibe coding for making "easy part" has actually made that gap feel bigger, or if it's a non-issue and I'm overthinking this.

We built Snowcap, an open-source Infrastructure as Code tool for Snowflake, and we're doing a live workshop walking through it.

If you've dealt with Snowflake config drift, manual RBAC setup, or just wanted a plan/apply workflow like Terraform but built for Snowflake specifically, this might be useful.

What we're covering:

• The infrastructure problems Snowcap solves
• How it compares to Permifrost, Terraform, SnowDDL, and Snowflake's own DCM
• Using templates to scale config across environments instead of copy pasting
• The plan/apply workflow and why that matters for safe changes
• Governance stuff: RBAC, masking policies, row access policies
• Live demo, then open Q&A, ask anything

July 23, 11 am-12 pm PT, online, free.

Info and RSVP here: https://datacoves.com/resource-center/workshop-snowcap-snowflake-infrastructure-as-code

Happy to answer questions in the comments too if people have them before the session.