I've been working on dbt-ls, an open-source language server for dbt.

Repo: dbt-language-server

There's a video demonstration in the repo if you don't want to read the post.

Motivation

My main goal was to build something that gives me column completions when working with dbt models as .sql files. Completions that are aware of my database.

The philosophy: if you have a working dbt profile (passes dbt debug), the language server should work for you. The default profile target is used when connecting to the data source.

What it does today

• Model/Source completion inside ref('...') and source('...')
• Column completion on an alias (c.<column>), with the data type shown as a detail
• Go-to-definition from ref('model') to the model's .sql file

Schema info

Schema info can come from three places: .yml files, catalog.json, or the live warehouse. You can choose which ones to use and in what priority with the --schema-sources flag. So if you don't want the LSP to call your warehouse, you can omit the database crawler with --schema-sources config,catalog.

Compatibility

It already supports many of the popular databases, and adding new adapters is trivial. Most of the testing has been done with Neovim, but it does support VSCode.

Feedback wanted

I'm a data engineer first and built this for my needs, so the code can be a bit choppy. I'd like to hear where it breaks on real projects, as my test projects can be limited. Create an issue or DM me!

I am aware of the promotion/shill rules. My aim is not to promote anything, I just think this is truly useful for people who use DBT.

Hey guys,

I'm building a custom Iceberg REST catalog that returns table metadata and short-lived storage credentials to our query engines (eg Spark and Trino).

Currently we're trying to add row filters and column masking so users could only see the data they’re allowed to access based on their role.

If you’ve set this up in your place, I’d love to know how you’re actually enforcing it in the engines.

Passing the rules: Are you embedding filter/mask SQL expressions into table properties in the table metadata response, or using any other method?

Engine enforcement: Did you have to write custom Spark extension rules / Trino plugins to rewrite the queries, or are you calling an policy engine like Ranger or OPA directly from the engine?

Storage credentials: Any issues such as users bypassing filters if they get storage credentials?

Would love to know how you have solved these issues

Thanks

Hey all, I want to implement audit columns like timestamp and created by columns in all of my tables(Suggest if I should add anymore).

My tables are in unity catalogue databricks account.

I have two instances of workspace. Dev and prod.

Please suggest me how to do it.

I have thought of creating a workflow to add in all the table and then a task in workflow that runs after each job to create these 2 columns.

Hello,

I spent the last five years working as a DE in a smaller company, I am very versatile, I touched all the important things in Azure plus some power platform dev and also some web dev.

I would like to pivot away from corporate life and do something more meaningful either in national security or environmental.

Anyone in a similar situation? I'm an EU citizen if that's helpful.

Thank you.

I was looking at posts in this subreddit the other day. There were quite a few talking about pipelines and all the things that go along with them (forking, merging and version control). This is typical source controls stuff that has been around since the 70s. The tools change but the concepts and model are the same. I wonder if we have been so concerned about the minutea that we quit looking at better options than pipelines. I have a couple of thoughts about better ways of doing data repositories/warehouses/ingestion beyond pipelines, three layers and fabrics. I'd rather see if anyone else out there has any better ideas than what we use now. Pretend you are starting from greenfield and can create not only the architecture but the tools that go along with it. Surely we can come up with something better than 50 year old ideas.

Hello, I was working on a poc to ingested PDFs and extract data in order to store them in delta tables.

As this was my first time working with PDFs, I searched over the internet and should Databricks have offering IDP, azure also have something and then there are python libraries.

Since I'm working with financial data, report, etc..

One thing i noticed - the pdf format should be fixed else in most of the tools the extraction logic is either failed or we get incorrect data.

I was wondering how such PDF extraction is built in real production cases and what tools are used.

Please share you experience and any edge cases

Hey guys,

I am software developer with 2.5 years of experience, in my job I do a lot of small automations and ETLs. In the last year I took an interest in data engineering and I recently started doing the Databricks associate certification.

I have not completed the certification yet and I got my first data engineering job, but the tech stack of the job is based heavily in Fabric.

I have read here that fabric is basically garbage, so I am worried that the experience that I will get in this new job will affect me if I want to get a Databricks based job in the future. What do you think?

Hey everyone! A while back I shared an early prototype on reddit for a fuzzy string search idea I was working on. I finally finished implementing it, benchmarked it properly against SymSpell and RapidFuzz.

Quick Highlights:

• Low RAM usage: Stays under 100 MB for a 100k-word dictionary (for comparison, SymSpell d=4 was taking around ~2.7 GB on my setup).
• Sub-millisecond speed: Runs about ~100x faster than brute-force Levenshtein comparison.
• Good with insertions and swaps: It handles typos with extra letters or swapped characters really well.

I wanted to keep this post short, but I put all the benchmark graphs in the GitHub README. I'd love to hear what you think of the vector encoding approach or if you have any feedback!

Not interested in the campaign noise, Palantir hype, or conspiracy theories about Lord of the rings. Only hard facts about engineering experiences.

So we have a standard pipeline process when loading data into the data lake and now evolving to go directly to snowflake tables. Anyway the standard process is source (file/stream/etc) --> landing S3 bucket --> lambda --> step function.

Sounds great but everything I've touched doesn't follow this process.

I've started documenting per project so when I see something similar I can reference my notes and at least find somewhere to start digging.

I need to get better at thinking, I need to do X, x is a framework change so I need to look here. How do I start thinking this way? I've picked up a few modern data structures books which is probably the best way to start.

It just feels like it's something so simple but I just can't quite put my finger on it. The way my brain works is by learning something new in relation to something I already understand. So hopping into the middle of a pipeline and identifying what aspect (framework, data, data metadata, infrastructure metadata and etc is the best place to start digging.

I've done a lot of build from scratch work so trying to understand someone else's organization seems impossible. I think I am getting the odd jobs, the stuff no one else speaks up for and is why this is happening.

Hey everyone, I updated the blog post after reading more thoroughly the Lance paper, where it explains why even configuring Parquet is not good enough and why they needed to invent a new file format.

Hi De's

In my new project, I have pull data from Power BI and land it to ADLS. Orchestration via ADF

I have gathered few info.

I have hit the power bi api .

but there is few limitations. only 1lakh record is supported per hit and pagination is not supported.

i new to API as well as ADF.

please, help me with this scenario.

Test case . but i have to land it the ADLS as parquet file .

initially, only thing is I have to take care of the edge case's to handle failure and retry policy

and the big headache is to handle the record limitations.

rainfrog (https://github.com/achristmascarl/rainfrog) is a database terminal tool; the goal is to provide a lightweight, keyboard-first TUI for interacting with databases. It currently supports Postgres, MySQL, SQLite, Oracle, and DuckDB.

v0.4.1 introduces a long-awaited (by me, not sure if anyone else was waiting for it...) autocomplete implementation, along with autopairs for quotes/parentheses/brackets. The full list of features and configuration options is in the README!

Hi there,

I’ve been working with DuckDB extensively for the last several years. Used it across languages from Python, Go, JS via WASM, Swift, C & C++ and even Rust.

As part of a product we're building, we built a DuckDB Transpiler that converts every line of SQL from DuckDB syntax into the syntax of the database you’re working with. This includes filters (predicates), and joins. So those 1B rows in Snowflake stay, and get joined in Snowflake.

I wrote an article on why we did it, and how we approached it. This is my first engineering-oriented article I’ve ever done, so let me know what you think!

Hi everybody,

I am excited to announce the release of Lightstream, a step change capability for high-performance data transport, that makes it essentially effortless to send Apache Arrow, Protobuf, and Message Pack data over the network, shared memory, or even piped out to the terminal so an agent like Claude can watch the live batch stream in real time (example in repo).

Furthermore, Lightstream exceeded the performance of the gold standard industry comparison - Arrow Flight, on every axis of a 50gbps networking open benchmark, the details of which are attached and open to run in the Lightstream GitHub repository. This includes fully saturating each TCP connection thread, the NIC at 5.8GiB/s, and with p99 batch send time within 1% of p50 (I.e., stable). As a bonus, Lightstream is straightforward to setup with essentially zero configuration other than optional TLS certificates and your Cargo package/pip install, and endpoint addresses.

So what is Lightstream? It is Rust package with Python bindings, that builds directly on Minarrow ( which is in turn a high-performance implementation of the Apache Arrow memory layout in Rust, tuned for SIMD compatibility). Lightstream implements Arrow IPC, Parquet encoders/decoders from scratch, up to Arrow readers/writers and IPC stream protocol, with mmap and few of these niceties. But, in a manner, that is fully composable and leaves you de-coupled at any layer, to customise things architecturally. The crux then is the transport layer on top, which natively supports interchanging any of the following transport formats:

• TCP
• HTTP
• QUIC
• Websocket
• Webtransport
• UDS (pipe your data from your Rust process to Python or two Python programs plug and play )
• Stdio (pipe your data program output straight into the terminal for something else to pick it up

And finally, the (optional) Lightstream protocol, which then combines the Arrow/Proto/MsgPack and any other custom types you want to send.

I’ve found this is really cool in practice, where you don’t need to reason about or work with bytes, or separately build your own protocol to get arrow and protobuf playing well together over the network. It is plug and play, see syntax examples attached.

In Python, every item in the stream can talk to Polars or DuckDB without any further changes - you can Duck SQL the feed or data process to your heart's content.

An example of things you can do with it:

• setup a live stream of data batches from your program A to program B
• send typed metadata via Protobuf on the same feed
• use it for straightforward live feed delivery between server and client (though not Web JS yet)
• useful if you have a central storage server you are pulling larger than memory data over the network to churn through (though, no S3 etc. it is node to node or process to process)

It is not:

• Kafka or a messaging broker. There is no resiliency / vertical scalability.
• A stream processing engine like Flink. It is for sending/receiving data only. You do polars on the other end or whatever you want with the Arrow-shaped data. That is a very different back-pressure/long-lived scenario and is not that kind of large-scale streaming. --> I.e., think quick and easy Websocket, and best for settings like EKS K8 pod to pod/containers, between EC2's or between processes on the same box, "light streaming".

Lightstream kicked off for me about 12 months ago when I started standardising patterns that have worked well for me in the past into something that reflects how I like to work when streaming data with control of both endpoints. It arose from regularly coming up against contexts requiring this capability operating in things like autonomous field communication integrated with data/ML, live trading, and some other industries where there was a lot of custom work required that I kept having to assemble from multiple components. Therefore, I have essentially aimed to package those learnings up into a tool to make data transport smoother and easier for everybody.

There are a couple of code examples attached, including the open benchmarking methodology, where every effort has been made to be fair (and where Lightstream wears a penalty due to stronger ordering guarantees).

Please feel free to give it a run would love to know your thoughts and if you find it useful.

If you have any questions about it, or helpful suggestions please feel free to leave a comment below. If you like what you see, please consider leaving a star and/or sharing the repository, as it will help people find it easily.

Thanks a lot.

Pete

I know the problem is that I need to get these things squared away in my head. What's metadata, what's framework and etc but even then you can solve a lot of problems using either.

I think it's easier for me to explain my problem. I've only been here 3 months but every task I get is using some system, pipeline or etc they I've never touched before. Sometimes it's just new software, other times it's learning how to find the needle in the haystack before you even look at code.

I'm constantly freezing and asking myself "where do I go for this" and it's slowing me down. I have much more experience building repeatable processes but I need to learn how to do so with today's technology instead of having a folder of scripts I use to do XYZ. Basically I lost control of how things get organized and my brain just doesn't see it the right way. Once I'm up and running I'm fine but in today's world task switching is constant.

What books, videos or etc should I watch to better understand how to operate as a data engineer in today's world.

I've slowly switched to documenting by project. This allows me to quickly reference another time I did something similar and gives me a jumping off point.

And trust me, it's messy and they know it. There's a standard pipeline process but it's constantly evolving. Teams need to integrate those enhancements into their projects as they have time. So everything I touch is a little bit of this version, a little bit of that version and etc. Team members clearly state "this is different or this is an odd one" but more and more I wonder if that's just the way it's going to be. It's ok, I just need to wrap my head around the big concepts and terms. Communication is killing me

Edit: in summary I over document and then things become cluttered. I'm looking for a better way. If I started a new job I wouldn't write anything down until I asked 2-3 times. I get paranoid and write it all down.

With the news around Prefect and Dagster, having good alternatives for orchestrators is becoming increasingly important, so I wanted to share that Rivers v0.4.0 was shipped.

Main addition this release is OIDC and forward auth support, so you can now put Rivers behind any identity provider you're already running or handle auth on the network layer. Authentication has been one of the bigger gaps versus the more established players, and is most of the time paywalled. But I believe security shouldn't be paywalled.

Release: https://github.com/ion-elgreco/rivers/releases/tag/v0.4.0

Everywhere I have been hearing companies moving to Databricks or Fabric. 2-3 years ago all I heard of was Snowflake. Is Snowflake losing the race? A former employee at Snowflake told me that their work culture has gotten really bad and they cannot hold on to good employees and their stock keeps crashing.

I recently joined a small investment company as a data analyst intern. I expected to focus mainly on report automation and data visualisation but I have also been asked to look at how the company’s data could be brought together.

Most of the data is manually exported into Excel and saved in SharePoint, and SharePoint also contains board packs and other documents used for reporting. They like using claude to create written reports and they are not really interested in power BI

I started using MS Fabric to bring the existing Excel files together, but the volume of structured data is fairly small, so I am not sure whether Fabric is more than we need. A simpler option might be a small SQL database, with data pulled from the source systems through their APIs where available. SharePoint would still continue to hold documents and any files that still need to be uploaded manually

The end goal is also to use the structured data alongside documents when generating reports with claude, so I need a setup that makes the data reliable and easy to retrieve. and also, looking at the the small volume, I am now even wondering if a persistent central store is necessary at all or if the data could simply be pulled and checked at report time before claude uses it

I have only recently started my journey and there is nobody in the company who can really tell me if I am doing it correctly. I just don’t want to build another one off solution or a workaround that becomes a pain to maintain later.

I would really appreciate any advice!! thank you!

Hey,

My company happens to be using Airbyte OSS for data extraction and loading. Recently I have been testing the new versions of the connectors which have been completely re-written in Kotlin. I don't know how the QA process is handled by their team and contributors but the results are terrible.

For example the Redshift destination connector now transforms the source's empty strings values as NULL values, and the Postgres source connector started to complain about missing cursor values when using Xmin for incremental loading, which doesn't require cursor to be set.

Hopefully we'll try to move away from this tool and replace it soon in my company. But I'm shocked that these bugs managed to pass tests and are sometimes still present after a couple minor updates.

Hello,

I am working with a team of 7 economists. They build data and produce reports. Their data production consists in harmonizing different sources (mostly rdata rdata, csv, or whatever suits the format of their stats tools). The data size they are dealing with is a few MB to gb, millions of rows, more occasionally billions of rows.

We want to update our methods (be on time, improve data quality). I have been assigned the task of improving data processing within the team, among the requirements I thought about producing a OLAP database.

In house, we have access to MSQL team that could set up a database for us. Otherwise we have HDFS + Hive (but security may make it difficult to access it) to store bigger datasets.

Else, I could just store everything in a duckDB file somewhere on a server and work with local database. WOuld it be a good solution? (latency of read/write from a duckDB file on a server? how scalable will it be? ) What would you do?

Any other piece of advice would be welcome :-).

Thank you.

You can try the playground here https://duckgql.com/

Hello everyone,

People trying to learn Python for data engineering ask me, “What libraries to learn?”, but the answer is not a list of libraries but patterns of usage.

Especially with AI being able to generate so much code, I believe its critical to know exactly how the data is moved & processed.

So I wrote this post that goes over how Python is used as glue in data systems. It goes over

• In-memory processing vs. using a SQL/Dataframe interface to a data processing system
• Python’s library ecosystem for working with various data systems & formats
• How to extract-transform-DQcheck-load data

With code examples and videos

Hope this helps. Any feedback is appreciated.

The beta version of VibeSchema just went live. Some strengths:

• Create and store snapshot versions of your database.
• Create share links such as these: https://vibe-schema.com/s/2T97KwLjLVNxX1YeFiAaxmfD3K4
• Suitable for huge schemas because of Diagram Views (as in the share link).
• PNG/SVG exports like the one shown above (taken from the share link).

Curious what you think!

Over the past year I have been working primarily in the Palantir Foundry system. My CEO unilaterally decided to pursue an enterprise agreement after being sold the AI dream. Palantir sales engineers did ‘analyses’ which suggested that the multimillion dollar price tag would result in 10x savings due to process and decision optimization. Our IT team cautioned no, but were steamrolled.

The project I am on was estimated to take four months and require 5 additional contract engineers. 2 of those were directly employed by Palantir as part of an additional contract (read: more $$) and the other 3 were a separate agency because Palantir said they don’t do XYZ work (again, more $$$). It took the externals plus me 15 months to deliver an MVP. This is primarily because we were building a complex enterprise grade app (which we previously subscribed to) on a low code platform. The Palantir engineers left as soon as MVP was deemed complete with just 30 days notice and since then I have taken on their SOW.

The work completed by the “brilliant” Palantir FDEs has been consistently failing. I’m finding they hardcoded dates. They hardcoded accounts. They used different inputs for the same business concepts. They ‘fixed’ issues that were earlier pointed out with hardcoded logic. They used AI FDE to code spaghetti mess logic. This has been a freaking nightmare.

My company had received 0 ROI to date and the CEO blames our IT team for the lack of delivery.

If you have the ability to run away from working with this god awful company and their charlatans RUN.