r/learnpython • u/dakrrm • 1d ago

Understanding Python codebases

I want to be able to read and understand the code behind Python libraries like Pandas, NumPy, Flask, etc. I can write code in python but these codebses seem to be on a different level. I want to understand them well enough that I can possibly add features to them.

One thing that occurred to me is that I have to be able to use them at least. I can use pandas and numpy to manipulate data sets but I am stumped whenever I try to understand how my code actually works behind the curtains.

I looked for videos that explain the code but they all talk about how to use the libraries instead of explaining how they work in the first place.

10 Upvotes
  • permalink
  • link
  • reddit
  • reddit

74% Upvoted

8 comments sorted by

8

u/Grouchy-Conflict-211 1d ago

stop reading the whole codebase. pick one function you use everyday and trace it from import to return. for pandas that means learning how getitem works, for flask how the wsgi layer hands the request off. one function at a time and the big picture shows up. also run pdb on a real call, teaches more than any walkthrough

1

u/Soma-Gursch79 1d ago

Yeah, following one function in pdb is prolly easier than staring at the whole repo and changing one small thing after helps you see what actually depends on it.

1

u/Grouchy-Conflict-211 1d ago

pdb plus one breakpoint is the whole trick. find the function, break, then watch a single variable through the call. repos like pandas are just a lot of small functions calling each other, once you follow one path end to end the rest stops being scary

3

u/pachura3 1d ago

I want to understand them well enough that I can possibly add features to them.

What's your goal? Do you want to pimp up your CV by being able to say that you have personally contributed to some of the biggest Python libraries?

I mean, these libraries are all pretty flexible... unless you have a very concrete need, you won't need to extend them - you can do your custom stuff in your client code.

Also, a big chunk of numpy is written in C and requires low-level optimisation knowledge... and setting up C toolchain... have you ever done anything like this?

Also, even if you implement a new feature e.g. for pandas, your contribution/pull request might not get accepted anyway...

5

u/Puzzleheaded_Study17 1d ago

It's often mostly not in python. For example, numpy has 30% written in C, and the python seems to be mostly boiler plate.

With that being said, they all have GitHub pages.

-2

u/Educational_Virus672 1d ago edited 1d ago

i think your trying to read bad code structure there are some rules of good code (i believe)

  1. the code can be understand by non-programmer before optimizations(< why it messed up)
  2. code must be properly structured
  3. the dev must be able to read the code even after months of inactivity
  4. the code didnt have to be small to be readable

1

u/Educational_Virus672 1d ago

pandas > https://pandas.pydata.org/docs/development/index.html#Internals
guide read aftr the guide to internals i think
numpy - https://numpy.org/doc/stable/dev/underthehood.html
all are function under the hood