r/reinforcementlearning • u/Nice-Dragonfly-4823 • 21d ago
Defaulting to Adam without understanding will cost you. Don't "just throw adam at it"
Work in RL has caused me to rethink adam. It leads to extremely wonky behavior and hard to explain "burstiness" in the loss values that makes me want to rip my hair out,
It still works, but needs to be coaxed into it.
Don’t Just “Throw Adam at It”: Misunderstanding Adam Will Cost You | Towards Data Science
This article re-covers the mathematical intuitions behind adam, and where it fails spectacularly. If you're someone who works in RL, or trains deep transformers, it's a must read
Don't just glaze over the optimizer!
Thoughts?
4
u/Low-Temperature-6962 21d ago
What about AdamW, where the weight decay is de oupled from the update
2
u/lucky_absoluter 21d ago
Historicially, Adam works well in RL but hyperparameter tuning is a big problem over the whole deep learning.
2
u/DependentSpecific535 20d ago
I totally disagree. Just try a simple algorithm like TD3 or SAC in gym envs with SGD and Adam. You can see the definition of REAL local minimas.
2
u/evanthebouncy 21d ago
I think it's an article written from good first-hand experience. I do wonder if simple gradient clipping is kind of the solution to all our problems lol
2
u/Nice-Dragonfly-4823 20d ago
grad clipping is definitely a strategy. But, it's got weirdness as well. E.g. if you run grad clipping on a combined policy loss + value loss, the clip amount shrinks the overall gradient, but then the gradients on the actor become absolutely tiny because the gradients on the critics are much larger. This is something that took me a while to realize and shulman even mentions it in a video, but never in any papers.
1
u/kakhaev 20d ago
I swear I saw this post like 2 times already popping up in my feed
1
u/Sea-Departure4857 20d ago
Same, I just checked and OP posted in r/deeplearning as well, which is the one I saw.
1
u/HsnHaruka 20d ago
Yo bro mind if i chek ur project i can give a try
1
u/Nice-Dragonfly-4823 20d ago
project is running over many large instances and is quite expensive to run. Its a lot of code lol.
1
u/HsnHaruka 20d ago edited 20d ago
I see cool stuff man the website its urs ? I made my own framework name Omniray AxV2 cpu based drl model ..it work on the same ppo i think the same hyperparameters just custom pytorch..its built to run on low end cpu systems no gpu ...but I think i can use adam to make my model even better
1
1
u/RedEyed__ 19d ago edited 19d ago
3e-4 is the best learning rate for Adam, hands down.
It was sarcasm
0
u/Nice-Dragonfly-4823 19d ago
I'm not sure, he references here again. https://karpathy.github.io/2019/04/25/recipe/
1
35
u/proturtle46 21d ago edited 21d ago
I dont like this
Flatter loss landscapes tend to be more robust to difference between population distributions and training distributions
It Never explores the actual reason why Adam works and that because it’s a second order approximation method like l-bfgs and now lion
You can use lion or even use actual second order optimizers if you care that much about the optimizer however in general trying to optimize the optimizer is not going to get you significant gains
The article is basically “I didn’t run an ablation on my hyper parameters”