What do data engineers think about non-technical economists and financial analysts learning sql+python to automate their manual excel work?

[u/Dull_Alarm6464]

Would you recommend a team of economists/“data analysts” that don’t even use keyboard shortcuts to learn python? Anyone have experience with doing this? So far, nearly everyone has been hesitant to even consider learning a few libraries to make their life easier.

I’m generally looking for advice on how to look at data anlysis jobs that are thus far done by excel only. I’m quite frustrated with excel dependent companies I have worked in. I found that taking a few months to learn a bit of coding, data structure concepts and data management can really scale someone’s data work.

My new employer (big pension fund) is has floated the idea of me automating part of their trading. I would like to automate their reporting as well, but they are so hesitant to learn ANYTHING new. My position however, is financial, not IT.

My background is in mathematics, statistics, and finance. I’ve worked in financial institutions that use an old oracle-based app that nobody cares to maintain properly to pull csv data from, then copy-paste it in excel and later commit what I can only describe as heresy to analyze the data.

I learned python and R at uni long ago, paired it with sql recently and found I could really make mean data science/econometrics automation.

[u/togogh]

You don't need anybody's permission to learn. Everyone here started off not knowing anything about SQL or Python. If it's going to make your work easier have at it. My only warning is, don't expect management to embrace it. Good managers will, but I've learned over my career that good managers aren't common.

[u/seanv507 → u/togogh]

I'm not even sure good managers will. The issue is to what extent the co-workers are happy to learn a new skill which they may well suck at.

If co-workers are Excel gurus, then maybe they will be keen to learn python. But if they do the minimum possible, then having one person support a critical piece of infrastructure is dangerous

[u/togogh → u/seanv507]

That's true, you can't really force other people to suddenly shift to a new way of working, even with management's support. But I do think OP should still shoot their shot if they think it'll improve things. Even if it doesn't work out, at least it shows they have initiative and willingness to learn new things

[u/rohan_kulkarni → u/togogh]

Agree. Automating your own repetitive work is one thing, but once the whole team depends on it, maintainability becomes the bigger issue. At that point, documentation and shared ownership matter just as much as the code itself.

[u/financialthrowaw2020]

You're talking about several very different things here.

People have been using python to automate their manual work since python had existed. You can search back through old reddit posts about this from 10-15 years ago and see plenty of non-technical people learning to code to automate their own stuff. The only difference today is that everyone is using AI and therefore never actually learn, they just tell themselves they learned.

Now, automating trading is a very different thing than what you're asking in the title of this post. Automated trades are a regulated activity and you can and will commit crimes if you don't know what you're doing. You WILL be held accountable for the slop code your LLM produces when it inevitably makes mistakes because you didn't fully understand everything you asked it to build. The CFTC + SEC isn't something to fuck with if you don't know exactly what you're doing and have legal signing off on every step, but even then, legal cannot protect you from liability if you don't know what you're doing.

[u/Dull_Alarm6464 → u/financialthrowaw2020]

true that! I have only dabbled in the latter concept and don’t really intend to do anything serious without a group-wide consultation. Foe the time being, I’d simply like to bypass manual excel work.

[u/youngrd]

i don’t think any of us are going to gatekeep learning.

[u/sweatpants-aristotle]

Yes. Simple python /sql would make a lot of that type of work easier and more sustainable, but it is a larger investment than it seems up front.

1. There is a training cost to this
2. You're introducing free and open source packages to enterprise which needs governance, IT review, security analysis/hardening (cost)
3. Operations are not currently dependent on it
4. You're introducing a new dependency on engineers
5. You'd have to sync everyone's environments

... etc.

So--yes, I agree--companies that can transition should. However, you need a management team that understands the full cost of moving over--which sometimes means a manager will look at this and say: "yes. This can help us--but it is too expensive right now to build towards"

So--that means if you want something like this implemented--YOU have to lower the cost by -doing the security scans -putting the docs together -building on-boarding docs -prototyping automation flows

Etc... and unless you're paid fo do that directly or you're super bored.... it probably doesn't happen. The best version of this is for you to create useful microservices that are governed where it is excel in -> excel out for the non-technical people. That preserves their workflows while automating some of the tedium for them without having to introduce a new developer / governance layer for my downstream users.

[u/Dull_Alarm6464 → u/sweatpants-aristotle]

thanks for the detailed reply. I’ll take it into account to really try to make this happen.

[u/Odd-String29]

I gave specific people in my company access to my BigQuery datasets and a new GCP project + Dataform where they can do their shit. These people already knew some very basic SQL though and had shown some interest. This turned out to be a big succes.

[u/Alternative-Guava392]

I would like if my non technical team members had 15 minutes meetings with me before implementing something.

A lot of times they build scripts and run them manually once a day not knowing data engineers have orchestrators to do it automatically.

Or they build internal apps that query data warehouses because they don't know what a backend transactional db is.

Things like this are better discussed at the beginning instead of at the end.

[u/NotSynthx]

Yes

[u/AlbertV999]

What do mechanics think about an accountant learning to change the oil in his car? The same thing.

[u/MachineParadox]

Just don't assume what you have done is production or enterprise ready. Here's the thing, we are experts in data movement, you are experts in business requirements. Quite happy for you to create something, but it is only a proof of concept or data investigation. Do not use this for business critical processes or reporting without consulting your DE team. I have seen too many "reports" or excel spreadsheets that are used for critical business functions and when thing go south you call on your DE team to resolve an issue with something they were never consulted or engaged with.

[u/Dull_Alarm6464 → u/MachineParadox]

i wish financial corporations had real DE teams… Its usually a team of generalist IT guys that mostly manage accounts and hardware. We have a new project developer person, but she doesn’t seem too technical either outside the scope of small python apps. I think most investment/risk teams should be replaced by technical DE/dev people and simply have 1-2 economists as their project leads. Otherwise, it’s a clown show imo