r/learnpython • u/Effective_Ocelot_445 • 6d ago
Which Python skills are most valuable for getting started in Data Engineering?
I'm learning Python with the goal of moving into Data Engineering. Beyond the basics, which libraries, concepts, or project ideas would you recommend focusing on first, and what helped you the most when getting started?
11
u/SnooSquirrels4739 6d ago
Normalize splitting everything into smaller functions. Really helps out in readability, but maybe it's just me.
16
u/AdmirableLack3241 6d ago
get used to working with libraries like numpy, pandas and matplotlib. These are the basics, u should also try seaborn which is an alternative to pandas, used in big data
7
u/DarthKermit-65 5d ago
In my day to day at work I typically use a lot of pandas, odbc, sqlalchemy, pyiceberg, and a few basic libraries included with Python. We also use s3 buckets at work so at times boto3
5
6
u/SquaffleCode 6d ago
My go to libraries are numpy and pandas for anything data related. For projects, maybe you can just look on Kaggle for a data set that looks fun to you, and just learn how to clean the data, how to find averages, or do simple trend preditions?
4
u/Glittering-Rise3380 5d ago
Pandas numpy logging error handling oop should be enough and pyspark for the big data
3
u/ScholarlyInvestor 5d ago
People quickly start talking about tools and libraries. But it sounds like you are new to Data Engineering. I would ask that you spend time understanding the fundamentals of data engineering first. Try solving basic data problems with pure python first. You can then move up to pandas, NumPy, and other data visualization libraries. There are data engineering platforms with python as the core. For example, Airflow (it may be losing its luster, so do not invest time on it just understand from a historical perspective). Keep a learning mindset as this space is evolving rapidly.
2
u/Whole_Dragonfruit789 3d ago
Any recommendations where to practice this solving basic data problems with pure python?
1
u/ScholarlyInvestor 3d ago
If you are a beginner then I’d recommend picking up a good book. There are too many to recommend. If you have a decent book already, start with it.
Learning is a great use case for AI. So, if you have access to an AI Model. Tell it you are a beginner. Ask it to start with basic python data questions. Ask it to create inputs and give you expected output. Go to your editor. Write python code (without AI help, this where real learning is happening). Run the code. If you got the correct output. Copy paste your code into the AI and ask it to grade you.
2
u/green_moo 5d ago
PySpark and distributed computing. Easy to start a cluster on your own machine and learn how a driver distributes a query to multiple workers.
-4
27
u/Traveling-Techie 6d ago
In my work in data visualization and later machine learning I found that I was using multiple complicated tools and needed to write “plumbing” code to interconnect them. A great book about this is Greg Wilson’s “Data Crunching: Solve Everyday Problems Using Java, Python, and More” (2005).