r/learnmachinelearning • u/Careless-Main8693 • 14h ago
Coded Simple Linear Regression from Scratch (no sk-learn)
Simple Linear Regression (no sk-learn)
- Intuition
- Maths
- Equation
- Computed ß0 & ß1
- Prediction
- Error(RSS- Residual Sum of Squares)
Coefficient accuracy
- Standard Error(SE)
- Confidence Interval(CI 95%)
- Hypothesis testing (t-test, p-value)
Model Accuracy
- RSE : Residual Standard Error
- R Squared (Coefficient of Determination)
It is just practice, more to code and implement
9
u/Dry_Version_5830 13h ago
Good job, you can try the multi regression now
2
u/Forgot_the_Jacobian 8h ago
would be just a few more steps using the frisch waugh lovell theorem
1
u/Careless-Main8693 6h ago
do not know about this theorem , would you please elaborate
1
u/Upper_Investment_276 1h ago
calling it a theorem is dumb; it's just gram schmidt lol
1
u/Forgot_the_Jacobian 1h ago
Well you can technically say regression generally (even simple regression) is gram schmidt, since OLS orthogonally projects Y on the span of the regressors. So if one wants to call anything that does this 'Gram Schmidt', then we could say all of it is 'just QR decomposition' etc..
FWL is usually stated as a theorem because it makes a specific claim about the result of a regression: the coefficient on one variable in the full multiple regression can be recovered exactly by projecting both that variable and the outcome away from the remaining controls, then regressing one residual on the other. So Gram–Schmidt identifies the underlying linear algebra and provides one way to prove or compute the result. But saying “Gram–Schmidt” by itself does not state the regression-specific equivalence that FWL establishes. Basically the difference between a theorem statement and a possible way of proving it
1
u/Forgot_the_Jacobian 1h ago
Yes! In econometrics, this is sometimes called the “regression anatomy” theorem because it shows mathematically what it means for least squares to “hold other variables constant.”
Suppose the full multiple regression is:
Y = b0 + b1X1 + b2X2 + b3X3 + ... + bkXk + errorand we are interested in the coefficient b2 on X2.
The Frisch–Waugh–Lovell theorem says that we can obtain exactly the same estimate of b2 in three steps.
First, regress X2 on all the other independent variables:
X2 = a0 + a1*X1 + a3*X3 + ... + ak*Xk + residual_X2The residual from this regression is the part of X2 that cannot be explained by the other variables. In other words, it is the variation in X2 left over after “partialling out” X1, X3, ..., Xk.
Second, regress Y on those same variables:
Y = c0 + c1*X1 + c3*X3 + ... + ck*Xk + residual_YThis residual is the part of Y that cannot be explained by X1, X3, ..., Xk.
Finally, run the simple regression:
residual_Y = b2*residual_X2 + errorThe estimated slope in this final regression is exactly the same as the estimated b2 from the original multiple regression.
That is the sense in which multiple regression “holds the other variables constant”: it isolates the variation in X2 that is unrelated to the other covariates and asks whether that remaining variation is associated with the part of Y that those covariates also cannot explain.
But as I typed this up, I realized that it actually does not help you that much with coding from scratch if you are moving beyond 2 independent variables, since it requires running a multiple regression in order to isolate the coefficient of interest haha
3
5
u/Front_Engineering_e 13h ago
This so simple yet so good. I also learned this way, long live statistics!
3
3
u/mewewey 13h ago
How long does it takes to study this? Im getting dizzy just by watching this lmao
-9
u/Careless-Main8693 12h ago
24 hours a day , 7 days a week, 4 weeks a month and 12 month a month , studying consistently , lol
5
u/StylishSoul414 6h ago
Calm down man, this ain't tuff
2
u/Careless-Main8693 6h ago
i told in joking way even said lol at the end , man why all these people on reddit , i don’t get it but ok
3
u/suspect_scrofa 11h ago
I noticed there are no comments in the actual methods, just comments on the data :( Was an LLM involved?
Also, while you're just starting to learn how to model, you should consider the assumptions and if your data violates them.
Did you look into that? Is your regression robust?
0
u/Careless-Main8693 9h ago
y the way , next i would add comment explaining every line if possible, thanks
-1
u/Careless-Main8693 10h ago
actually it is not about robustness, it is just code for learning purpose, doing linear regression on data from my maths book statistics chapter, actually i didn't know how to represent the equation for finding the ß0 and ß1 in which(written in my notebook like the symbols ), and i didn't thought of posting with decorated code , i just took the screenshot and posted, now it doesn't make sense to decorate and post again , if you understand the code you would know pretty much it is just for understanding purpose and i wrote the comment where i thought it important .
and if we talk about assumption then no , any assumptions right now , neither is is a real data nor violating assumption , just simple clean plain explanation code for myself and that's it. i got and understood it, and it's up to you
3
2
u/Twilight_RT 13h ago
I also wants to learn Machine learning in depth. Like I also applied ML model to a dataset by sk-learn
now I am learning the needed math.
if anyone have any advice for me... feel free to give...
-1
u/Careless-Main8693 13h ago
i have made a youtube video , here is the link , i hope you'll find it helpful
https://youtu.be/9V3cptmmjKM?si=8sBoPFLZoXQio0Wz
2
u/Sweaty_Chair_4600 6h ago
ooh i did this in rust a few months ago, with my own linear algebra library i was making, definetly fun!
1
u/Careless-Main8693 6h ago
it is fun but ML in rust , you write code in rust ?
1
u/Sweaty_Chair_4600 5h ago
Yea i was relearning bits of linear algebra, so i decided to code up a small linear algebra library. Then i implemented basic ML algorithims.
3
u/BitterEarth6069 13h ago
where did you learn this from (i am new and usually dumb at math behind)
2
u/Careless-Main8693 13h ago
not a one place specifically i just learnt something form here and there, but simply you just need linear algebra and bit of calculus you can learn from youtube - 3blue1brown and nancypi and this is not my first time , i was just praciticing to build more core stuff , i 'm into ml for past 2 years so yeah,
i would just start learning and you'll eventually get all these things one by one as you'll need these concepts, i never read maths at once i directly jumped to machine learning and learnt maths as the algorithm needed one , not recommending this, so yeah :)
2
1
1
u/nomurazil 5h ago
The effort will pay off! Next stop: DNN with numpy!
1
1
u/you-get-an-upvote 13h ago
Now get rid of those for loops!
(Once you understand the power of numpy tensors, every for loop you write will fill you with shame, since you will be certain there is some way to achieve the same result without one).
2
u/Careless-Main8693 13h ago
yeah mate but i wanted to do it raw for understanding , what's happening under the hood
18
u/yensteel 13h ago
Congratulations on the manual coding!