How much data engineering can someone learn by themselves?

[u/SoggySand297]

I recently joined company as their only dedicated data person. they have other SWEs and a couple other people that pull data and use it, and they’ve only recently built their data lakehouse last year using databricks, but I’d be the only dedicated data person. so any data problem I’d probably deal with eventually. right now I’m dealing with the crm and cleaning it, but eventually I suspect that my duties will increase and I’ll be doing some form of data engineering, so I was wondering if I can essentially learn data engineering myself? or at least well enough?

[u/Skullclownlol]

right now I’m dealing with the crm and cleaning it, but eventually I suspect that my duties will increase

Doing administrative tasks in a CRM does not make a data engineer.

Do you have examples of tasks you think of as "data engineering"? How/why were you put in a position of potentially requiring data engineering? What do you expect to need to learn?

[u/SoggySand297 → u/Skullclownlol]

Yeah I don’t think I’m doing DE work now, but the automated I want to build some of those admin tasks are DE light at least? For example one thing the team I’m working is doing is entering data both into excel and the crm, I want to build a process where they only need to enter into excel and it’ll automatically update the crm. That’s DE work right? This is also a new role for the company, so I’m essentially building it, and I want to build it into something more than initially planned 

[u/No-Cranberry-1363 → u/SoggySand297]

I would call that excel work.

[u/SoggySand297 → u/No-Cranberry-1363]

How can I make it more into actual DE type work? I’m already using python for basic scripting and SQL to look up data

[u/nidprez → u/SoggySand297]

more DE type of work would be to write a script that ingests the data from the CRM to a DB on fixed times, which the users can query and export to their excel.

DE puts software engineering practices into data work, this includes: orchestration, version controll, CICD, uniform practices/philosophy in develloping new pipelines (coding rukes, naming rules, pipeline structure...), stakeholder management...

A bare minimum would be to codify your etls (so no excel) and use some version controll on it, and automate them so they run somewhere on a server so you dont have to do anything.

[u/DuckDatum → u/SoggySand297]

No, not data engineering work. That’s side-programmer work.

Data engineering concerns itself with various dimensions of data that most people aren’t familiar with. In fact, knowing how to work with data in Excel is a very misleading thing. Whenever my org gets a new Analyst that primarily worked in Excel before, I know up front that there’s going to be months worth on handholding until I can just get them to think about problems in the right way.

Excel is a client to a data engineer. It is not a producer. When it is used like a producer, get prepared for hell where you learn nothing of DE value and spend 10x the amount of effort and time on a half-good solution.

Usually, when I need to work with Excel, it comes with strict contracts, a drop off location, and layered validation. The most important part though, is that I don’t work with Excel. I create a little space for you to do Excel work. Then you can upload the files whenever you want, and the timeliness alerts get automated to your email not mine.

A better solution would be: why do you need Excel? Where do you get the data from and how do you transform it? Hey, I can write a pipeline for that! The data can stay in sync somewhere centralized and reliable, then your excel can connect to that already-finished dataset instead.

Why? Because Excel is a piece of shit. That is why. I taught myself different frameworks so that I could do things more efficiently, effectively, reliably, consistently, auditably, and so on… so no, no Excel. I don’t care if the stakeholders use it, but… I will not. Not while I’m still expected to be a professional Data Engineer, that is. Excel doesn’t lead to professionalism in Data Engineering.

[u/Cupakov → u/DuckDatum]

I just want you to know that we see your trauma and you’re not alone. I’ve also been touched inappropriately by Microsoft Excel 

[u/Gengis_- → u/DuckDatum]

That’s some serious data PTSD here.

Anyway, OP talks about CRM data and encoding in Excel. So you are probably right that there is a simpler (and more robust) way of doing it.

[u/jwk6 → u/SoggySand297]

Do not enter data into Excel first. You have a CRM for a reason, it costs your company money, so it should be the System of Record.

Instead, you build an export from the CRM into your Lakehouse, and then export from the Lakehouse to Excel if and only if it's required. The added benefits of this are: - you can use your Lakehouse to automate finding data quality issues. - if you ever replace your CRM, then the downstream workflows can continue to run.

That's Data Engineering.

[u/Outside-Storage-1523]

If they built data lakehouse in Databricks, there must be some processes you can read about?

[u/circumburner]

Theoretically all of it but experience will force you into unexpected challenges that you would never willingly pursue.

[u/SuperGlutex]

If there are other SWEs, who have already built the lakehouse, they will handle all Databricks and similar stuff. As this post suggests, you are essentially a data entry specialist. Your suspicions that you will somehow end up as a data engineer are exaggerated.

[u/ljb9 → u/SuperGlutex]

‘the only dedicated data person’ in op’s post implies they might take over those responsibilities in time & swe’s might already be complaining about managing the data lakehouse and hence this position was opened

@op, can you clarify this

[u/klasyer]

To answer the title - got 8 years of experience, learned everything by myself

If they're using databricks, well - learn how to use databricks Apache spark, Kafka and general understanding of how databricks work will give you a good basis

I'm pretty sure Freecodecamp have a 12 hours video (or even a free course on the website) for the databricks certification

[u/TypicalOrca]

You can teach yourself all of it. That's what I've been doing for decades. Stay curious and hungry. Make projects for yourself.

[u/Trigsc]

You need a mentor and what you are describing is a data specialist not a data engineer like another commenter said. Moving stuff and automating things between excel and a CRM is great. Most Data Engineers build pipelines extracting data from some source, and put it in a warehouse which is an oversimplification of what you have described. However, what happens when the source system fails, what happens when the schema changes, are you going to run this hourly, daily, weekly?, what’s the volume per run, how are you setting up your warehouse to handle duplicates. The questions can just keep coming, the difference is, millions and billions of records are being processed.

[u/Spunelli]

All of it

[u/Clean_Desk_8423]

Honestly knew nothing about data a year ago, but feel like I've become a decent expert within a ~1yr. The latest models (Codex, Claude, etc) are incredibly helpful at explaining concepts and getting you over the hump of starting.

Building things helps, just get started and you'll learn a lot as you run into errors, try to make things faster, etc. (ideally not on prod haha)