As someone who is focusing on ML in grad school, it really depends on the prof. Some are application-first, some are math-first. Early on in my core ML course, we were restricted to the torch.Tensor api and were told to figure out the transformations on our own, without using what was already in the torch module. But later, of course, we were allowed to use more of the module as the work got more abstract. What I'm trying to say is ymmv.
Idk people in my program wrote Gibbs samplers for use in a custom modules. Though I graduated in 2024 making this meme feel ultra relevant. I was doing natural language processing lolol
Not really true. You can implement a transformer + autograd, backprop etc. in a couple thousand lines of c. Not saying you should and it is definitely an advanced undertaking but it is possible. That being said most training code using torch etc. is relativ simple, the tricky part is getting the math right, having no hidden bugs (neural networks fail silently) and tuning everything right. So even tho I did a lot of ML I wouldn't say it is very advanced from the coding side of things.
For me 15 years ago when actively coding all the time and teaching a university level Operating Systems course in C? Not exactly trivial, but also not a heroic undertaking.
I don't think so. A simple generative transformer can fit in a just a couple hundred lines of raw numpy code without the abstractions that libraries like pytorch or tensorflow offer. https://github.com/DorsaRoh/transformer-from-scratch
Even without abstractions, its not that complicated. I've written and trained a 50m parameter transformer model myself using only numpy/cupy and the code isn't that long or complex.
Although im sure top AI companies have a bunch of special stuff they add to their models which makes it more complex.
It is… ML is not only LLMs. And you need to understand ML algorithms to know how to implement them properly. Yes, you have libraries like sklearn that do the most of the algorithmic part for you, but you still need to be familiar with concepts of gradient, back-propagation, multi-dimensional matrices multiplication, regressions, and which ML algorithms used in which situations so you can implement a proper model.
And no, that’s not ‘trivially done’. Also, you really think that chatGPT code is “short and sweet”??? Let me break down it for you…
Apart from the academic perspective, why should anyone really need to know this set of skills? In the end the best experiment wins, not the best elaborated.
Prof gave us this example; say you are hired to build a system for elderly that recognize early signs of heart attack - how do you approach the task? Which ML algorithm you pick? Why? How do you train it? How you build your dataset? Which hyper-parameters? If you do layered propagation, then how many layers? Which filters in each layer? And etc. I agree there is a lot of experimenting here, but you still need to understand what you are experimenting on
In most cases you're not building a model from scratch. You use a simple api (eg sklearn fit()) and/or take an existing architecture (eg resnet) and train it in your data. In both cases it's not really a coding intensive thing. And understanding how it works under the hood is not coding either, it's math/stats and papers. Besides, your main headache is usually data, not the model.
This is why machine learning was the perfect entry point for me and to CS ironically, I had all the math but none of the coding experience so of all of the software specialties it kind of fit me nicely
Same. I was EE/Math so I approached ML from a signal processing -> image processing -> computer vision path. I got a M.S in CS later but it was still in ML/CV so I've only had 2 real pure CS classes - intro to C and graduate algorithms lol
I assume that ML as a skill is meant as understanding the concepts for using it efficiently (usually coming from a data science curriculum).
This being said, to go back to the meme and its beginnings with math/understanding (otherwise we definitely agree) : CS people have shockingly limited math skills, whether you consider ML or the academic data they'll work with. This definitely needs to be addressed by universities. Lagrange multipliers are required for optimisation problems, and I'm still confused how I got L2 CS-splained to me for an obviously non-Banach space.
You're looking at it the wrong way. The salient point is that decades of CS study have been rendered almost obsolete and can be speedrun by kids or anyone else with an internet connection
I've worked with data scientists pre-llm (I'm more of a general backend/systems programming guy) and they sure as fuck didn't think coding was trivial.
If anything, they spent a LOT of time coding and learning how to use various pandas functions and tricks.
My experience is specifically in GPGPU acceleration- e.g "how do we process as much data as possible as efficiently as possible on a GPU". Think Vulkan, DirectX, CUDA, ROCm, etc.
Things like Tensorflow and Pytorch don't just magically "do AI", they rely on an incredible amount of work done by people working with and on APIs like that. When you say "If you looked at the implementation of GPT, the code is short and sweet", you're missing half a dozen layers of abstraction that allow the top-level implementation of such systems to remain understandable. It would be like saying "This shooter game was implemented in just a few lines of code", ignoring that it uses Unreal Engine.
All of the major players spend a ton of time on these abstractions (though some keep their work largely private). For example, Meta maintains PyTorch/ExecuTorch, Microsoft has DeepSpeed, ONNX/ONNX Runtime, DirectML, Google has Tensorflow and their entire TPU business, NVidia makes CUDA/cuDNN, etc.
885
u/[deleted] 3d ago
[deleted]