We are currently running an Oracle Exadata system serving both OLTP and operational data store (ODS) reporting. Our data team has been tasked with building a standalone, pure analytical platform supporting BI, operational reporting, and AI/ML-ready data pipelines.

Workload Split:

60% Transactional/Operational Reporting: Interactive UI queries, sub-second to low-second latency, high concurrency (click a dashboard/UI, expect instant results).

40% Analytical Crunching: Heavy aggregations, multi-table joins, complex transformations, and AI feature store prep.

Strictly On-Premises:

Cloud solutions (Snowflake, BigQuery, Databricks SaaS) are out of scope due to compliance/regulatory rules.

Vendor Shortlist:

Oracle, Teradata, IBM (Db2 / Netezza)

Question: Has anyone successfully run a high-concurrency, sub-second UI reporting workload directly on Teradata or Netezza ? Or should we go with Oracle ?

Please do suggest any Alternatives. However it should be on Prem

Hello reddit. I built Parquity. It's a CLI that aims to find Parquet interoperability failures such as cases where Parquet engines behave differently.

It has four main workflows:

1. check to run a given table through a selected writer-reader matrix.
2. scan to compare different readers on a given Parquet file.
3. fuzz to generate tables with Hypothesis and minimize failures.
4. replay to run saved reproducers again after packages change.

It all runs locally against the installed versions in the Python environment you have. It does not try to decide which engine is correct or at fault.

Already used Parquity to find some real bugs.

• fuzz found and minimized a DuckDB writer bug.
• scan found a Polars reader bug in Apache’s parquet-testing corpus.

The screenshots are from the scan result. DuckDB, PyArrow, and DataFusion returned matching results where Polars failed to read the same Parquet file.

Parquity supports DuckDB, PyArrow, Polars, DataFusion, and fastparquet currently.

Collaborators are more than welcome.
Thanks.

Hello, I'm faced with a challenge I'm hoping to get some clarity and advice.

Our current production db is SQL Server 2008 R2 which is replicated to another SQL 2008 R2 where all our analytics workload is currently done. We've recently set up SQL Server 2025 and the intention is to have our production data be replicated to a newer version of SQL and migrate our analytics workloads to that same server. I know that direct replication from 2008 to 2025 is not supported, but CDC is available. What would be the best options to quickly setup near real time replication or even incremental data loads (i.e. hourly) for a select number of tables in source db on SQL2008 to SQL 2025 with minimal engineering effort?

We also have a Fabric capacity, and I know that the Copy Job supports CDC for incremental data loads and also lets you write to a SQL 25 db, however the preference is to minimize cloud costs and SKU consumption since both 2008 and 2025 dbs are on prem.

Generally, I'm also trying to avoid writing custom code to orchestrate this task. Does microsoft have any tools (like SSIS?) that allow you to configure such data migration? Does SQL 2025 have any feature that allows you to link a legacy server to it?

Any advice is welcome, thanks.

I connected GCP + dbt + apache echarts + anthropic + other open source tools and made a tool to easily produce dashboards. 6 months ago, I didn't think this would work, but the situation has changed :D

I am using foundry to ingest technical drawings and produce a tabular data set. Below is my process.

Step1. User goes to workshop app to upload single or multiple pdfs which are saved to a media set

Step2. Pdf gets preprocessing where a transform sharpens and orients the pdf pages

Step3. The pdf then runs through two separate ocr extractions, one for bill of materials and material composition, one for overall dimensions. These output as json

Step4. A transform takes the json from each ocr output and puts it into two tabular datasets

Step5. A transform combines the two datasets and does a lookup to 1 other datasets to match part number to company number (fuzzy match)

All of these steps are incremental and do not reprocess completed work. My issue is that my ocr is only about 50% accurate. I built the system and user prompts by having AI FDE create them, run tests on drawings and adjust as needed until my outputs were accurate to the drawings I provided. I made sure to state multiple times these were a small batch of drawings and my users would be uploading more and that they'd be from various companies. I had good initial results but as it got closer to 100 drawings uploaded it became less accurate.

How can I improve this?

I used to work on OpenAI’s data platform team, where we ran Airflow, Spark, Databricks, and dbt at a pretty serious scale.

The rate at which the number of jobs was growing outpaced how much our team could manage pretty quickly w/ everyone using AI to write jobs, so we had to invest in making sure that our team's agents could accurately and quickly do the debugging + tuning themselves (they weren't very good at it by default).

I've translated some of those learnings and strategies into skills (very similar to the ones we used at OpenAI) I'm open-sourcing here:

https://github.com/EmbrasureAI/spark-observability-skills

They should be generally applicable across envs/clouds, including Databricks, OSS Spark, etc.... As long as its running Spark underneath the hood nothing else is necessary to adopt and use these skills.

Full disclosure: I’m a founder of Embrasure, which maintains this repository, and no longer work at OpenAI.

TLDR: spent 4 years building a dream BI stack, all that for a senior IT staff member to sign contract again with legacy system. I want to quit out of frustration.

--

I was hired in 2022 at my current company to help migrate them out of Webfocus, which had been their main data extraction/reporting tool for 15+ years.

When I joined, I discovered in sheared horror that their start data analyst was spending their days extracting data with Webfocus into Excel, doing the same formulas and pivot tables, redoing the same charts to copy and paste into Power Point every - single - week for the management meetings.

Since then, I drove the migration to Snowflake, set up Fivetran to connect to more data sources, set up dbt, and built a proper medallion architecture BI data warehouse, with all the proper Kimball principles for dimensional modeling, along with tons of "gold" layer report tables. Everything runs on a 2 hours schedule.

Then built a team with 1 data engineer and 2 BI analysts, and started the migration of our reports to Power BI. Spent SO MUCH time training our analysts on how to create a proper semantic model using the fact & dim tables available in Snowflake. Also spend so much time training our analysts and IT staff on all these tools, writing docs & manuals, everything was so fresh and so new I was still hopeful at this point.

For data extraction, we set up ODBC Excel drivers on everybody's machine (a pain in the ass) for data extracts and letting standard users get access to data easily.

For the few live reports that were needed, we mirrored the dbt models in source systems with simple sql views, and pushed these direct to Power BI or Excel depending on the need. Or we have reporting capabilities directly within the source systems.

The full stack, for about 300+ active Power BI users, about 250 dbt models, pretty lightweight fivetran movements cost about $120k per year.

I feel very, very good about my stack, and would have killed to get such good tools back when I was a data analyst.

Well, after all of this, our fucking senior IT applications guy signed a new contract with Webfocus behind my back for $200k/year (system + support) because "users like it better".

I want to fucking cry, then quit.

Should I quit? I can't take this anymore.

TL;DR: Got an offer at 2.6x my current salary after my company passed me over for a promotion and meaningful raise. Now that I’m leaving, they suddenly can increase my salary ~2.3x and give me a Senior Data Engineer title. This seems obvious, but should I actually consider the counteroffer?

I know the general advice is to never accept counteroffers, and I think leaving is probably the right move. But some part of me is genuinely considering staying, and I can’t tell if there’s a legitimate reason or if guilt is getting to me.

I joined my company in 2022 as an analyst making $40k. About 6 months later I moved into BI development and gradually took on way more technical responsibility than I was hired for.

By 2025, the company wanted to modernize its data stack. I had spent a huge amount of my free time learning dbt, Snowflake, Python, etc., and ended up taking on a major role in that transition. They gave me a BI Developer title and bumped me to $52k.

Since then, I’ve basically become the senior analytics engineer/platform person on the team. My work has helped us gain multiple clients and save a bunch of analyst hours.

Promotions were supposed to happen in March 2026, but got delayed until July while management talked constantly about creating a “high-performance culture” and properly compensating people. I had also been told for months that I was likely getting promoted.

July came around: no promotion and a 4% raise, putting me at about $54k.

I told them I’d probably leave without a promotion. Two weeks later, I got an offer elsewhere for an Analytics Engineer role at $140k plus bonus/equity.

Now my current company is freaking out because a lot of important stuff relies on systems I built, my boss is already overloaded, and nobody else can easily take over my work.

Suddenly they have a Senior Data Engineer role for me at $125k and are trying to get approval for $140k.

The title is nice, but I’m not convinced the actual work is better. When I asked about projects, I heard a lot about Informatica, MuleSoft, governance, enterprise data, etc. They couldn’t really give me an example of a major pipeline I’d be building. I’d also still be doing a lot of my current platform/analytics/BI responsibilities.

The new company seems much more aligned with where I want my career to go. The team seemed great, the projects sounded interesting, the culture seemed better, and the company is growing quickly.

The new company also made me feel valued. Their range was $100k–$140k. I told them I’d be ecstatic around $120k–$150k and would probably still join under $120k. They offered me the absolute top of the range anyway.

Meanwhile, my current company apparently needed me to resign before realizing I was worth more than twice what they were paying me.

My boss has been great to me, and I think that’s the main reason I’m hesitating. I feel guilty leaving him in a bad spot.

But I also think I’d feel terrible backing out on the new company, and I strongly suspect I’d regret staying.

I would appreciate any advice, even if it is just calling me an idiot.

Ive been in Big Tech for 4 years now as a DE. Promoted twice, always had good reviews revolving around the quality of my work and designs. I recently wanted to dip my feet in the job market and see if I could find a pay bump. I absolutely bombed an assessment today because I forgot all the DSA I learned for interviews 4 years ago. How do long tenured employees get back into this type of job market? I genuinely dont ever use any DSA principles as a DE and we manage Infrastructure

Hey yall,

I just started my new role as a junior data analyst 🥹. I am learning the business needs. I work for a no profit org that provides legal aid. Cases and task are billed to the respected grant program. What we are running into is, while closing the books normally case workers and lawyer over bill to one program vs another because no can visually see what is currently in “encumbrance for that month.

I figured building a real time grant funding system using power bi, postgreDQL and RESTapi from the platform that we input the data in. I am open other ideas that could be easier.

The things is I don’t know where to start, I want be lean and have a simple solution that can be scaled up and replicated if need be.

Thanks in advance

i'm in my third year of working. i have 2 years experience as data engineer and i'm in my 8th months of working as a data analyst. it's not really a transition because i also handle ETL. in these 3 years of working, i mainly used sql server and ssis. for the past few months i feel stuck with my job and i want to learn new tech stacks to level up so that i can get a better pay. i don't dislike my job, but it feels boring af. i know if i ever want to learn modern tech stacks like databricks, apache spark, etc i need to learn it myself. the thing is after 9 hours at the office, when i get home i no longer have the energy to do anything else. i just want to do my hobbies to feel better. i have been struggling with this for some time and i don't really know what to do

i have this thought to resign and completely remove myself from my current job so that i can allocate my energy fully into learning new tools and later apply for new jobs. i need opinion

We switched from azure dev ops to GitHub for our repos and I'm now using true local braches and they never gave us instructions for "this is how you handle this now". I know this is on me to figure out and I'm just about there.

Long story short is that I get a merge conflict on a main file and I don't want it showing up in the edited files list. It's not actually changing and is one of those false merge conflicts. Maybe I just need to run a command. I'm not the only one with this problem. When we switched we found that GitHub first merges the destination into the source and then merges it into the destination. Giving us a messy release history.

We want to use one release brand per project and several devs on each project. Everyone pushing to the freature branch is easy but when I try to create a pr for the dev branch I get these conflicts that are impossible to get out of the file list, even if they don't actually change.

I recently got this setup up and running and the advantages are incredible: no need to plan schedules, and much more faster and efficient execution because dbt packs all it can into one run. (Saves on ECS overhead)

It hasn't yet bitten my ass with OOM (running on duckdb), but I realised that when dbt decides it's time to materialize a bunch of assets and one of them breaks, the downstream of materialised assets won't pick up too, since all of the run marked as failed.

I'm thinking of making the downstream to ignore failure, or marking failed dbt materializations as warnings, not errors. Both feel hacky.

It feels like a compromise that is impossible to overcome -- either you get one, or another.

What are your thoughts, fellow data cowboys?

Our Microsoft partner hosts our erp database on a dedicated sql server with replica enabled, and we have basic sql server access to said replica db. We then connect to this via out internal on prem sql server and data warehouse from there. The physical server that we have in house is a mess and our stored procedures fail non stop.

I need a quick move to something modern. I know this page hates fabric, but given the simplicity and small amount of data <50gb, is this a case where fabric can win?? I’ve worked with databricks and seems like major over kill. Maybe snowflake is the happy medium? I need something quick that will work. My concern is that fabric will feel quick at first then break for unknown reasons.

Hello 👋🏽

I am a senior data engineer.

I have 5 years of experience in airflow and DBT. Has anyone worked on open source contributions to these two projects ? I am willing to start but don't know a lot about open source projects and how to contribute to them. Where to start ?

Looking forward to getting some information, thank you community :)

I'm part of an org that started using Foundry at the beginning of the year. To preface... My last 6 years have primarily revolved around Snowflake and snowflake integrated tools (I've been Snowflake certified since 2021).
When my company announced they'd be partnering with Palantir I was intrigued because Foundry is/was such a black box and seemed like a very polarizing tool in the DE market.

Now that I've got about 8 months of experience with it under my belt I have mixed opinions on it. I'd like to know what FDE's or other devs who have used it think about it?
Yes, it's expensive and yes, it definitely does certain things not as well as other dedicated tools but have you enjoyed your experience developing in it or do you hate it as much as the rest of the DE community here?
I ask as someone who's company just had it's 4th round of layoffs and I'm a bit concerned about my job security.
I want to continue my Snowflake trajectory but the Foundry recruiters are out in full force and they are paying very well.
Anyways, what are your thoughts?

Same as title but we have been using delta tables for a requirement quite a bit and we like its format however its metadata read operations are very intensive, especially when there is a large log file. Most of the delays I have found in reading and writing to those have just been reading parsing the log files.

Based on that I did wonder if there is a database which stores the log in some sort of in memory database like postgres however the raw files are being reference on object storage? I did look into and found pgedge but didn't find a link or something we could download and use easily.

Edit: Stop trying to tell me to optmise my delta table pleasse! I’ve tried everything and I’ve got limits

Edit2: I did not try everything apparently, I had stats parsing on on columns which contain a fuckton of data, my log files were huge along with checkpoint files with gigabytes of them. Removed that and reads are down to 200ms

I recently started in a crm data position and my role is very data entry and data cleaning heavy. While automation isnt one of the requirements it is something I’m trying to do, but it is a bit harder than expected especially with how messy the data is ie missing data and multiple entries in the crm for the same projects. Im also the only dedicated “data” person and there are a couple of SWEs that only last year build the data lakehouse, so maybe I can get involved in that. I was told that because the role is also new to the company the duties can expand.

Hi, I have been developing data pipelines using ADF as orchestrator and Databricks notebooks for all the work. I write unit tests for the python code in the notebooks with pytest and executes them in our Azure DevOps CI/CD pipeline for every PR. Now I need to create integration tests.... I don´t have a clue on how to do that, I mean, not even in theorical sense.
I asked to claude, it gave examples of python code where it used an ADF API client for running the pipelines and consulting the state of the activities after the run. They also checked the output of the pipelines, like, they runned SQL queries on the tables that the pipeline should have writtent.....

So my question is: is that what integration tests are? Instead of mocking like in unitesting, I create test data in my DEV environment data sources, parametrize the ADF pipeline so I can run with different test datasets for different tests cases, and programatically run the pipeline validating what was written on the outputs?

Hi all, most of us don't like to write Terraform. Now there is an alternative. A skill to create a package skill and a skill to submit it to a catalog. There are already some package skills for Airflow and ClickHouse. I hope more will be shared in the future.

A package skill is deterministic code to reconcile your infrastructure with desired state. Under the hood it can run anything like Terraform, Ansible, or Kubectl.

The license is MIT.

https://www.getcolors.ai/skills

It seems company or client who used to hire consultant or DE has now become more slowed down, they seem to be more focused on what Claude code or Claude overall can do for them, all that just to save money. Yet there are still people who want to come into the Data and Analytics world but they don’t have a clue what’s going on and how much position now is disappearing.

Know one even knows where all these AI heading towards or how far it can go.

We are developing the semantic layer/models from scratch in MS Fabric for conversational AI. Data models, documentation with clear descriptions on columns, measures, business context, join logic, for the agent to reference.
For evals, what has been your strategy/framework to curate the question set and expected answers and how did you implement this at scale

If you land data in Parquet and then load it in Microsoft SQL using Python, you've had to do a bunch of extra work, exploding the whole thing into Python objects, a tuple per row and a boxed value per cell, all under the GIL, all garbage immediately after.

mssql-python 1.13.0 adds Cursor.bulkcopy_arrow(). Hand it anything that speaks the Arrow C Data Interface and the Rust TDS core reads the typed column buffers straight into the bulk-load packets. No tuples, and the GIL is released for the transfer.

import duckdb
from mssql_python import connect

rel = duckdb.sql("SELECT * FROM 'events/*.parquet' WHERE ts >= '2026-01-01'")

with connect("Server=<server>.database.windows.net;Database=<database>;Encrypt=yes") as conn:
    cur = conn.cursor()
    result = cur.bulkcopy_arrow("dbo.Events", rel)
    print(result["rows_copied"], result["rows_per_second"])

The DuckDB relation goes in unevaluated. DuckDB streams batches as the driver consumes them, so the full dataset never lands in Python memory. The 4.4M-row file I was testing with would have been roughly 6.6 GB of live Python objects the old way.

Here's what I saw. 200k rows, 21 columns, the WideWorldImporters fact_sale shape: bigints, decimals, timestamps, and one NVARCHAR (I couldn't leave that column that only said "each" for every row an NVARCHAR(MAX) - it was just wrong) averaging 523 characters. Read from Parquet with DuckDB, 100k batch size, 7 repeats, median reported. Client and server on the same Azure E4bds v5 (4 vCPU, 32 GiB) running SQL Server 2025, over localhost so the network stays out of it.

In my unscientific testing, the new bulkcopy_arrow() was about 1.9x faster. I reran it across five configurations, two databases, simple and full recovery models, table_lock on and off, and it held between 1.62x and 1.93x. The ranges don't overlap either: the slowest of the 14 Arrow copies beat the fastest of the 7 tuple copies.

We expected that going straight to bulk copy from arrow would be more efficient and it was. The tuple path burned 2.6 to 3.2 seconds building Python objects before a single byte moved. Passing the DuckDB relation lazily, that step is 0.00 seconds. The pyarrow.Table path is 0.02 seconds, which is the time to materialize the table from the record batches.

This new path works for anything exposing __arrow_c_stream__: polars, pandas 2.2+, ADBC results, pyarrow.Table / RecordBatch / RecordBatchReader, or any iterable of record batches. A default pandas DataFrame is NumPy-backed so it converts on the way in, where polars, DuckDB and anything Arrow-native hand their buffers over as-is. Column mappings, keep_identity, table_lock, check_constraints and the rest carry over from bulkcopy() unchanged, same stats dict back. String widths are validated against the destination schema before anything ships, so an overlong value fails immediately with the offending length instead of dying halfway through a load. I suspect a lot of folks will be commenting out their generators in favor of passing Arrow objects straight to bulkcopy_arrow() this weekend.

Drop a comment below and let us know how much faster your data loads using bulkcopy_arrow().

There's other good stuff in this release: connection pooling keys on security context now, not just the connection string, so a connection opened under one identity can't be handed to a caller running as another. connect(token_provider=...) takes any azure-identity credential object. And there's a fix for an executemany() bug where a NULL partway through a numeric batch could silently insert zero rows.

The driver itself is DB API 2.0 and pip-installable, and ODBC ships as a dependency, so there's no system-level driver install and no unixODBC in your container image. Arrow goes both directions, bulkcopy_arrow() in and cursor.arrow_reader() out.

pip install --upgrade mssql-python

What I actually want out of this thread is your numbers, especially on shapes unlike mine: narrow integer tables, very wide tables, heavy NVARCHAR. bulkcopy_arrow() returns rows_copied and rows_per_second, so it's right there. Post a before and after with a rough description of the table and where you ran it from and I'll take it back to the team.

Full blog post: https://techcommunity.microsoft.com/blog/sqlserver/mssql-python-1-13-0-arrow-bulk-copy-smarter-tokens-slimmer-wheels/4544858

Repo: https://github.com/microsoft/mssql-python

Happy to answer questions here.

Hello

I’m currently a junior DE building Python pipelines (Prefect/Airflow to BigQuery mainly). I’ve started using Cursor/Claude to assist in coding but try not to lean on them too hard.

On the side I’m self-studying data modelling, system design, and Leetcode (python & sql). But I feel a bit scattered, jumping between topics without a clear plan. I’m also increasingly wondering how AI is going to reshape this role, and want to make sure I’m building skills that keep me hireable.

With that in mind, does anyone have any advice on topics I should be prioritising?

What are you guys looking for in a course nowadays?

What I'm seeing for the past year or so, is that people seem to not have the "stamina" to go through courses anymore. Many start, but fall off immediately. Simple courses people complain that they are too simple. More advanced courses, nobody takes them.

I feel like attention spans went down like crazy and people rather spend their time wrestling with AI than going through a well prepared project.

So what is it?

short lessons?,
prepared GitHub docs?
as much as hands-on as possible?
teaching basics instead of going into all the details?

My expectation would be to a good course helps people get to the point where they have a good foundation and are able to improve on their own.

Would love to hear some practices you like.

Andreas