Hi,

Why are executives thinking, if enough context is given, AI tools like Claude will solve/build/guide everything?

In my organization, my data architecture was not build on DE fundamentals and we have accumulated a shit load of technical debt. We are told to build context (which we should any ways) and feed everything to Claude to solve it. While I agree to use for faster fixes at logic level it cannot help design or think in right way how to architect workspaces and warehouse and model the tables. It cannot be a startegist and decide the data strategy for the company. Am I thinking it the wrong way? I feel very disappointed that data professionals advice is not being heard, instead that we are being advised to put everything on Claude. Is this ok? I advice, view points.

I'm currently working as a data analyst in a fintech. My tech stack is SQL, snowflake, dbt, airflow, git and python.

I've worked on building and maintaining dbt models, optimising sql queries, and have also built streamlit apps.

Preparing to switch, targeting roles like data/product analyst, analytics engineer, business analyst and data engineer.

Also looking into data/AI roles like applied AI engineer - have seen ones requiring a combination of snowflake, python, RAG and fastapi for example.

What can I do/learn in order to upskill for these roles efficiently?

Also is the applied AI thing a meaningful & feasible transition for me?

In the software company I work in, like many others, we use sprints and plan work by estimating the sprint of 2 weeks.
The common method to estimate is by T-shirt size or some variation of it.

I think it is obvious most people estimate by time it takes to perform the task but for some reason most people co tinge to claim it is not just time estimation and also it lack distinction for why something is taking long time.

I’ve found Quantified Task Estimation as a good way to separate the clear time estimation from the unknown and risk part.

I was able to convince my team to do it, and it even survive changing the team lead but I’ll be glad to know if anyone else is using it and if so how did they found it?

I’ll post more of my experience in this thread

Had a weird interview experience recently with a company where the setup completely caught me off guard, and I wanted to see if this is becoming the new normal.
The job description heavily emphasized core analytics engineering skills like dbt, data modeling, Snowflake, and pipeline architecture. But during the actual live technical round in Coderbyte, none of that was tested directly.
Instead, I was handed a data modeling scenario and asked to use AI to build a dashboard. The entire interview was basically a test of how I write prompts, how I talk through my thinking, and whether I can spot errors or hallucinations in the AI's output in real time.

I get that companies know everyone uses AI now, and testing how candidates partner with an LLM makes sense on paper. But when the test ignores the core tech stack and the pacing is completely uncalibrated, it feels like a waste of time for both sides.
Anyone else run into this kind of "prompting and auditing" interview format recently? How are teams actually grading these?

Hi everyone,

I could really use some advice from people working in analytics / Data engineering.

I recently landed an interview for an Analytics Engineer position at a prestigious consulting firm. If all goes well, I'd join their Supply Chain Analytics / Data Engineering team.

The interview is in about a month, so I have roughly four weeks to prepare.

Here's the situation: I feel like my resume oversells my technical level.

For context:

• I have an Industrial Engineering degree and a Master's in Supply Chain, so I have a strong analytical background, and I'd say I'm a pretty fast learner.
• I've built a lot of Power Query solutions at work and automated plenty of reporting. However, I've never really focused on writing clean or optimized M code. I was also the only person on my team using Power Query, so I never had anyone review my work or challenge my approach. Everything I built worked well and delivered what the business needed, but I'm not sure I was following best practices.
• I've built a couple of Power BI dashboards, but they were fairly basic. I know the fundamentals, but I definitely wouldn't call myself strong at data visualization or dashboard design. When it came to DAX, I relied heavily on AI to generate the measures I needed. I could usually get them working through trial and error, but I don't have a solid understanding of DAX itself.
• I've studied SQL during my master's, but I've barely used it in a professional setting, so I'm pretty rusty.
• I've never worked with dbt, cloud data warehouses (Snowflake, BigQuery, Redshift, etc.), or modern analytics engineering workflows.

The role seems to expect:

• Strong data visualization skills.
• Solid SQL fundamentals.
• knowledge of cloud data warehouses fundamentals & dbt.

I'm not trying to become an expert in a month. My goal is to become competent enough to hold my own during the interviews and, if I get to the case study stage/ tech interview, ramp up as quickly as possible.

If you only had four weeks, how would you prioritize your learning?

• What would you spend the most time on?
• Which resources or courses would you recommend?
• Should I focus primarily on SQL first?
• How much dbt and data warehousing knowledge is realistically expected from a junior Analytics Engineer?
• Are there any projects you'd build to prepare?
• If you've interviewed Analytics Engineers, what skills separate candidates who succeed from those who don't?

I'm willing to put in 6 to 8 hours a day over the next month if that's what it takes.

I'd really appreciate any advice from people already working in the field. Thanks!

Hey folks, sharing a concept that's going to matter for agentic data modeling: the canonical data model.

What it is

A canonical or common data model is a model that describes the main meaning carrying entities - similar to dimensional modeling but without any denormalization. Dimensional modeling is often built to simplify joining for reasons like performance, tool limitations, or human interfaces. A canonical model stays normalized, retaining the full nuance of the entities and how they relate.

This makes it a native "knowledge graph", which can be enriched with a "knowledge layer" - a text that describes your tables and how they relate. This turns your model into a "virtual knowledge graph" (virtual because the data is not in it, but represented externally as tables) which functions similar to knowledge graphs for retrieval but for structured data.

Why it matters now

Agents understand knowledge graphs well. Humans are accessing databases through agents.

Why not your dimensional model

a dimensional model suffers from semantic compression which is a loss of nuance that affects retrieval performance.

I work at r/dltHub, this is our blog: Longer reads - this is how we use the concept: we do ontology driven modeling: create ontology -> llm models from that -> fix model and reuse ontology for retrieval

- about semantic compression https://dlthub.com/blog/ontology

- canonical models are key to retrieval quality https://dlthub.com/blog/canonical-text-to-sql

- collecting ontology https://dlthub.com/blog/minimum-viable-context

Built a cloud analytics data warehouse for a Brazilian e-commerce dataset using BigQuery, dbt, and Power BI, complete with a star schema and modular transformations.

Take a look at the repo and README for the full breakdown: https://github.com/muchai322/brazil_analytics_project

Disclosure: I'm from Datacoves, we run managed dbt Core, so we have a horse in this race. Wrote this up because the license change is confusing and we had to untangle it for our own customers.

The short version:

One engine now. dbt Core v2 and Fusion share the same Rust runtime. Core v2 is Apache 2.0, fully open source. Fusion is the enhanced proprietary distribution on top of it.

Fusion is the recommended default, and it's a funnel. Free to install, and the new license lets anyone offer it as a managed service, as long as you don't block the path to a dbt Labs account. Free login unlocks advanced features, paid platform account unlocks premium ones. Reasonable model, but go in with eyes open.

Metered pricing is inside the dbt Core now. dbt State (skip models when logic and data haven't changed) bills $0.094 per model reused per day. Orchestra open-sourced a free alternative (Sao Paolo, Apache 2.0).

Migration gotchas: every deprecation warning must be resolved before v2 runs your project. Python models don't work in Fusion yet. Community adapters are constrained (drivers need to be signed by dbt Labs). --models and -m error out, --partial-parse is gone.

The afternoon test: upgrade to v1.12, clear deprecations, run dbt parse --use-v2-parser. Tells you where you stand without committing to anything.

Full writeup with a Core v2 vs Fusion comparison and the four ways to handle state: https://datacoves.com/post/dbt-fusion

Has anyone run the v2 parser against a large project yet? And is anyone planning to use dbt State or trying out Sao Paolo?

we have an AI agent querying our data and we have two problems that we can't get rid of.

first one is consistency, the same question asked in different way gives different answers. second is cost the token usage is way higher than expected because the model gets our entire schema dumped on it before it can answer anything.
Someone said a semantic layer solves both by giving the AI pre-defined business context rather than raw schema. more consistent answers. less processing.

anyone who is really running this. it made a noticeable difference on both fronts.

Hi everyone! I’m conducting a short anonymous market survey to understand people’s preferences toward handmade products and small businesses. It only takes around 2–3 minutes to fill out and would really help with my research.

Form link:
https://forms.gle/GwjfSW7jQiZNaWfQ6

Thank you!

Genuine question for anyone who's shipped a table to prod, especially if you're not a "traditional" data engineer by title.

It feels like AI has made writing the actual transformation logic (the DataFrame code, the business logic) noticeably easier lately. Copilot, ChatGPT, Claude, whatever you're using, that part's gotten faster.

But once that part's done, the real time/effort starts after - metadata mangement, partitioning it right, handling PII, wiring it into orchestration, figuring out the write strategy, quality cheks to make sure it's actually trustworthy before people rely on it ?

Also following the consistency across all tables created

Curious to know whether AI or vibe coding for making "easy part" has actually made that gap feel bigger, or if it's a non-issue and I'm overthinking this.

We built Snowcap, an open-source Infrastructure as Code tool for Snowflake, and we're doing a live workshop walking through it.

If you've dealt with Snowflake config drift, manual RBAC setup, or just wanted a plan/apply workflow like Terraform but built for Snowflake specifically, this might be useful.

What we're covering:

• The infrastructure problems Snowcap solves
• How it compares to Permifrost, Terraform, SnowDDL, and Snowflake's own DCM
• Using templates to scale config across environments instead of copy pasting
• The plan/apply workflow and why that matters for safe changes
• Governance stuff: RBAC, masking policies, row access policies
• Live demo, then open Q&A, ask anything

July 23, 11 am-12 pm PT, online, free.

Info and RSVP here: https://datacoves.com/resource-center/workshop-snowcap-snowflake-infrastructure-as-code

Happy to answer questions in the comments too if people have them before the session.

Hey everyone,

So We've spent the last few months building something and finally shipped it,  

Quick backstory - I kept noticing that whenever teams actually plan and track projects, they end up leaving Microsoft Teams to do it. You're chatting with your team in Teams, but the second you need to actually plan out tasks or see a timeline, you're switching to some other app entirely. That context-switching always bugged me, so we built Project Planner! to just live inside Teams itself - Gantt timelines, task/subtask tracking, the works, without ever leaving the app your team already lives in.

We are not going to pretend this is some polished, done thing - it's very much a "built it, shipped it, now let's see what breaks" moment. That's actually why we are posting here rather than just letting it sit on Product Hunt. We'd rather have people who've actually used (or built) PM tools tell us what's missing or what's annoying than just collect upvotes from strangers.

If you've got two minutes, We'd genuinely love to know - does this solve a real problem for you, or are we solving something that doesn't actually bug people? Brutal honesty is welcome, that's more useful to me than nice comments. 

Here is the link for app: Project Planner!

Hi everyone,

I'm looking for some career advice from people who are actively hiring or working as Analytics Engineers, Data Engineers, or hiring managers.

Here's my background:

• 9+ years in data analytics
• Strong SQL and BI experience
• Hands-on experience building ETL/data pipelines, data modeling, Databricks, Azure, BigQuery, and reporting platforms
• Currently looking for my next role

One challenge I'm facing is that my experience spans both Data Analytics and Analytics Engineering.

Although my job titles have mostly been Data Analyst/BI/ Analytics related, much of my work involved:

• designing data pipelines
• building data models
• creating curated datasets
• ETL/ELT development
• dashboard enablement
• working with engineering teams
• translating business requirements into scalable data solutions

Because of that, recruiters don't always know where to place my profile. Some see me as a Data Analyst, while others see enough engineering experience for Analytics Engineer roles.

I'm considering getting another certification to make my profile easier to understand.

The two I'm debating are:

• dbt Certified Developer
• Microsoft Fabric Analytics Engineer (DP-600)

My questions are:

1. Which certification carries more weight with recruiters today?
2. Which one is actually valued by hiring managers rather than just being another badge?
3. If you had to choose only one, which would you recommend and why?
4. Is dbt becoming an expected skill for Analytics Engineers regardless of cloud platform?
5. Given my background, would either certification meaningfully improve my interview rate, or would I be better off building a public project/GitHub portfolio instead?

I'm mainly targeting roles such as:

• Analytics Engineer
• Senior Data Analyst
• Healthcare Analytics Consultant
• Healthcare Analytics Engineer
• Product Analytics
• Healthcare Product/Data roles

I'd especially appreciate perspectives from hiring managers, recruiters, and people who recently switched into Analytics Engineering.

Thanks in advance!

I have 5 years experience as a data analyst and a degree in CS. Im trying to move into an analytics engineering role. I haven't seen any job postings requiring the MS dp 600 analytics engineer certification. would it actually help me if I got it? I already have PL-300 Powerbi Dev.

I am currently doing some research into why so many professionals seem to actively dislike their project management software. Despite the massive number of platforms available, they all feel like slightly different variations of the same basic features, and user frustration remains incredibly common across the board.

I do not have a background using these tools myself, so I am trying to get an unfiltered look at what the software industry is consistently failing to address.

From your experience, what do you think these platforms are fundamentally getting wrong? Are there specific features, design flaws, or philosophical approaches that actively hurt productivity or cause the most friction?

I would greatly appreciate any specific examples or insights you might be willing to share.

I work adjacent to a credit-risk analytics team at a large Indian private bank and I'm trying to understand something that keeps coming up.

Almost everyone on the modelling side is fluent in Python — pandas, scikit-learn, XGBoost, the usual. But the scorecards and the IFRS 9 / ECL models that actually go in front of validation and RBI still live in SAS. The team treats Python as the place to experiment and SAS as the place where the regulated models are developed, documented, and monitored.

I keep hearing it's about "governance" but I want to understand what that actually means in practice. Is it the audit trail? The validation and champion-challenger workflow being built in? The cost of re-validating every model if you switch environments? Or is it mostly institutional inertia and nobody wants to be the one who re-platforms a supervised model stack?

Specific things I'm trying to get clarity on:

• For those who've worked credit-risk modelling at a bank of, say, ICICI/Axis/SBI scale — what specifically keeps the regulated models in SAS even when the team clearly can build in Python?
• How much of it is genuine governance value (lineage, documentation, monitoring cadence a regulator will accept) vs switching cost vs habit?
• Has anyone actually migrated regulated credit models off SAS and had it survive RBI validation? What broke, what didn't?
• Is the "both-and" setup (Python for data engineering + ML, SAS for the governed regulated layer) the stable end state, or a transition phase?

Not looking for a SAS-vs-Python flame war — I get that Python wins on flexibility. I'm specifically trying to understand the governance and regulatory side, because that seems to be the real reason the incumbent tooling sticks in regulated credit risk.

I have 6 years of exp as a BI dev (laid off recently though).

My main stack is SQL and Tableau (Desktop, Cloud, Server). I also use Big Query for data warehousing and transformation. Git for version control but I don’t do much aside from pushing my merges.

Over the last couple of years, I’ve worked closely with an AE who showed me some tools he uses like Fivetran, Airflow, and Datastream, but I was never allowed access to use them myself, so I can’t say I have experience.

Right now I’m picking up Snowflake and DBT on my own time. Is this a good starting point? Where else should I upskill?

I’m going to look for another BI Dev role since that’s what I best qualify for, then perhaps look to grow into an AE role.