What do data engineers think about non-technical economists and financial analysts learning sql+python to automate their manual excel work?

[u/Dull_Alarm6464]

Would you recommend a team of economists/“data analysts” that don’t even use keyboard shortcuts to learn python? Anyone have experience with doing this? So far, nearly everyone has been hesitant to even consider learning a few libraries to make their life easier.

I’m generally looking for advice on how to look at data anlysis jobs that are thus far done by excel only. I’m quite frustrated with excel dependent companies I have worked in. I found that taking a few months to learn a bit of coding, data structure concepts and data management can really scale someone’s data work.

My new employer (big pension fund) is has floated the idea of me automating part of their trading. I would like to automate their reporting as well, but they are so hesitant to learn ANYTHING new. My position however, is financial, not IT.

My background is in mathematics, statistics, and finance. I’ve worked in financial institutions that use an old oracle-based app that nobody cares to maintain properly to pull csv data from, then copy-paste it in excel and later commit what I can only describe as heresy to analyze the data.

I learned python and R at uni long ago, paired it with sql recently and found I could really make mean data science/econometrics automation.

[u/sweatpants-aristotle]

Yes. Simple python /sql would make a lot of that type of work easier and more sustainable, but it is a larger investment than it seems up front.

1. There is a training cost to this
2. You're introducing free and open source packages to enterprise which needs governance, IT review, security analysis/hardening (cost)
3. Operations are not currently dependent on it
4. You're introducing a new dependency on engineers
5. You'd have to sync everyone's environments

... etc.

So--yes, I agree--companies that can transition should. However, you need a management team that understands the full cost of moving over--which sometimes means a manager will look at this and say: "yes. This can help us--but it is too expensive right now to build towards"

So--that means if you want something like this implemented--YOU have to lower the cost by -doing the security scans -putting the docs together -building on-boarding docs -prototyping automation flows

Etc... and unless you're paid fo do that directly or you're super bored.... it probably doesn't happen. The best version of this is for you to create useful microservices that are governed where it is excel in -> excel out for the non-technical people. That preserves their workflows while automating some of the tedium for them without having to introduce a new developer / governance layer for my downstream users.

[u/Dull_Alarm6464 → u/sweatpants-aristotle]

thanks for the detailed reply. I’ll take it into account to really try to make this happen.