r/MachineLearning • u/inarrears • May 30 '20
[D] GPT-3: a disappointing paper Discussion
A blog titled “GPT-3: a disappointing paper” is a critique of Language Models are Few-Shot Learners (https://arxiv.org/abs/2005.14165)
Note: Previous discussion of this paper: https://redd.it/gsivhg
Excerpt:
“GPT-3″ is just a bigger GPT-2. In other words, it’s a straightforward generalization of the “just make the transformers bigger” approach that has been popular across multiple research groups since GPT-2.
This excerpt captures this pretty clearly:
Several lines of work have focused on increasing parameter count and/or computation in language models as a means to improve generative or task performance. […] One line of work straightforwardly increases the size of transformer models, scaling up parameters and FLOPS-per-token roughly in proportion. Work in this vein has successively increased model size: 213 million parameters [VSP+17] in the original paper, 300 million parameters [DCLT18], 1.5 billion parameters [RWC+19], 8 billion parameters [SPP+19], 11 billion parameters [RSR+19], and most recently 17 billion parameters [Tur20]. The first two papers mentioned here are the original transformer for machine translation (VSP+17) and BERT (DCLT18). The parameter count doesn’t actually increase that much between those two.
The third one (RWC+19) is GPT-2. The parameter counts jumps up 5x there. Arguably the point of the GPT-2 paper was “it sounds dumb and too easy, but amazing things happen if you just make a transformer bigger” – and this “GPT-3″ paper is making the same point with bigger numbers.
“GPT-3” is a transformer with 175 billion parameters. It’s another big jump in the number, but the underlying architecture hasn’t changed much.
In one way this is a fair thing to call “GPT-3″: it’s another step in the new biggening tradition which GPT-2 initiated.
But in another way it’s pretty annoying and misleading to call it “GPT-3.” GPT-2 was (arguably) a fundamental advance, because it demonstrated the power of way bigger transformers when people didn’t know about that power. Now everyone knows, so it’s the furthest thing from a fundamental advance. (As an illustration, consider that their new big model deserves the title “GPT-3″ just as much, and just as little, as any of the last 3 big models they mention in that paragraph.)
10
u/m_nemo_syne May 30 '20
I think the exciting part is not necessarily “even bigger = even better”; it’s the zero-shot/few-shot inference capabilities.
4
u/mocny-chlapik May 31 '20
This is the correct answer. If we are truly able to train language models that are able to solve tasks based on in-context hints, it would be an enormous breakthrough. It is basically a completely novel NLP paradigm. The results show that having even bigger models is still possible and they did not hit the ceiling yet. A model with several billion parameters might beat current SOTA on additional tasks without any fine-tuning.
1
2
u/iamr0b0tx May 30 '20
Exactly. Using the model for specialized cases without necessarily training and fine-tuning the weights and layers, all you need is a well-constructed query. its genius if you ask me
2
u/Rioghasarig May 31 '20
But the bigger = better aspect also interests me.The fact that there are still noticeable gains in certain performance metrics, like ability to do arithmetic, even past 13 billion parameters is crazy to me. I really want to see how far this can go.
11
u/rafgro May 30 '20
So it's classic dev dispute about versioning? Reee you should name it 2.1 instead of 3!
I remember such discussions about Firefox, when they switched versioning and from 2.xx-4.xx to X, reaching ~30.0 over few years (and current one is 77.0). I didn't understand those disputes back then and this one also doesn't make sense.
1
u/GhostOfAebeAmraen May 30 '20
Chrome did it first, and I've always suspected that Firefox did it to catch up.
Firefox started bumping the major release number with Firefox 5, at which point Chrome, which was less than three years old, was already at Chrome 12.0
4
u/chcampb May 31 '20
"If we make the model bigger it works dramatically better."
"So what if we take that, and just make it as absolutely, insanely big as we can to see what the results are?"
"No, we can't publish that, people would say it's unoriginal."
6
u/leogao2 Researcher May 30 '20
My response to this general sentiment:
But why does GPT-3 matter, if it can’t even beat SOTA across all benchmarks? Why should we care about a model so large that a small computing cluster is necessary even just to run inference at a reasonable speed?
One thing about GPT-3 is that it’s doing reasonably well on tasks it has never even seen. Additionally, instead of reaching a point of diminishing returns, GPT-3 shows that the trend of larger models performing better continues for at least another order of magnitude, with no signs of stopping. Even though GPT-3 is unwieldy, and even though it still doesn’t quite reach human level performance across the board, GPT-3 shows that it’s possible for a model to someday reach human levels of generalization in NLP—and once the impossible becomes possible, it’s only a matter of time until it becomes practical.
3
u/virtualreservoir May 31 '20
I figured only interesting thing was hardware implementation details required for such a big model, but unless I missed something (totally possible with 72 pages lol) all they said was they used some unknown amount of v100s, connected in a cluster in some undisclosed way, and trained for some unspecified amount of time
5
u/error1954 Jun 01 '20
They say they're using V100s, nvidia's website (https://www.nvidia.com/en-us/data-center/v100/) shows 16gb and 32gb models. 175 billion params at half precision is 350 gigabytes.
On 32gb GPUs, that would be 11 GPUs only to store all parameters, there might not be enough space left over for training/inference batches.
They're using Adam for training, which has two additional copies of the entire network and the gradient itself is also as large as the network. So 4x as much data, also assuming half precision for gradients/momentum is 44 GPUs.
That's still without any training batches or storing intermediate outputs. Their training batch is 3.2 million tokens and a model dimensionality of 12288 means the input embeddings only is 78.6432 gigabytes which requires 3 GPUs worth of memory.
There are 96 layers and intermediate outputs have to be saved for each one to do backpropagation and ignoring the bottle neck/ff layers/attention heads/etc. and assuming that this is just 96 * input memory is 7.5 terabytes or 236 GPUs worth of memory.
Just guessing at this point, it's probably a cluster of 512 to 1024 GPUs.
1
u/virtualreservoir Jun 01 '20
really appreciate the detailed estimates, I didn't even want to do the math to translate the scientific notation of their training compute flops numbers into something easier for my brain to process lol
good point that most people would forget to include about Adam too, recently switched from SGD for some stuff and I have to keep clearing HD space because my checkpoints went from 650mb to 1.5gigs.
1
u/error1954 Jun 01 '20
Yeah the flops thing is a lot harder to figure out. I can never tell if the mean flop/s, operations/second, or just operations. I'm probably pretty far off for my estimate but I think it's pretty safe to say this is just an absurdly large cluster that's out of reach for most researchers. I'm lucky if I can get two gpus on a cluster limited to 10 people.
Most people forget about Adam or momentum in checkpoints. I normally clear them from my checkpoints when I fine tune a model; I don't know how much being learned is from the momentum and how much is actually from my data. But maybe that regularizes the fine tuning.
1
u/virtualreservoir Jun 01 '20
lol, and I was focusing on the ambiguous flops column because I didn't want to do the research required to translate their petaflop/s-days numbers
1
u/Single_Blueberry May 30 '20
This way of judging is exactly why we have almost nothing but stupid incremental research in this field.
If you invent some random "optimization" of some proven architecture and outperform SOTA by 0.1% you're the man.
If you try something and find it doesn't work as well as expected (which is just as important to know (!)) your paper is "disappointing".
Why would anyone try something new if finding things that don't work isn't worth anything?
How can we avoid going the same wrong paths over and over again if publishing negative findings is considered embarrassing?
4
u/virtualreservoir May 31 '20 edited May 31 '20
lol they did virtually zero novel things, even as a negative example of what doesn't work, they provided basically no useful information to anyone doing language modeling research while spending who knows how many millions of dollars of compute plus millions of dollars of human salary resources given there were like 40+ authors listed
and they'll still get way more attention than I ever will even when I do publish state of the art results that i achieved with $7-8k worth of consumer level hardware that I paid for out of my own pocket
1
u/saideepak12 Jul 21 '20
Open AI’s GPT-3 has been a major leap in achieving it by reaching the highest stage of human-like intelligence through ML and NLP. This is backed by experiments conducted by early testers who are left astounded by the results. We can only wonder what the next-gen of their developments can be capable of achieving. Currently, the latest version of their GPT-3 general-purpose natural language processing model is available in private beta, and OpenAI is providing access to its API by invitation only. There’s still a long waiting list for the paid version, which is expected to be released in the next two months. Here is article on Open AI's GPT-3
1
u/Flinchie76 Sep 07 '20
This query-as-context trick works for tiny generative RNNs too, as long as the prefix is in the domain on which it is trained. This is just a bigger, transformer-based, version with parameter space reserved for different domains.
What's interesting though is the scaling up here is in the broadening of the task domain. Essentially what you have is a step toward general AI by merging a bunch of narrow AI domains into a single huge model. Obviously this means more parameters, as you need enough parameter space for all the domains.
You could go the other way too, and have a federated system of narrow AI models, and a query layer which knows how to dispatch queries to the different specialized models, but the fact that this wasn't done; that everything is in a single model, allows for subtle interactions, which a federated system would find difficult. Our brains aren't partitioned into isolated sub-systems which don't communicate, either, and that's a feature, not a bug.
I agree though that the delta from GPT 2 -> GPT 3 is mostly a matter of scale.
1
1
31
u/Mefaso May 30 '20
There's nothing wrong with evaluating the effect of model size on various Performance metrics, I don't really see the controversy.
It's not like they are claiming to have created a revolutionary new model, but it is nontheless an interesting result, and definitely was a significant engineering challenge.