Rob Miles discusses GPT-3 on Computerphile, covering how scaling parameters kept improving results over GPT-2 and how the model nearly learned arithmetic without anyone knowing exactly how.
Why Hacker News points people here: HN commenters cite it as a concise explanation of GPT-3's scaling jump and surprising abilities at its release.
“Computerphile summed it up pretty well on GPT-3's release”HN comment, 2023
Shows GPT-3 poems compared with human-written ones
Full comment
yesterday I watched a youtube video about GPT3 ( this video ), and it showed two poems. One was human made, the other was from AI trained on that human's poems. Both poems were pretty good. But one of them had a metaphor about the moon reflecting in ocean waves, being distorted and taking on monstrous forms. I figured this had to be the human one, it was a novel description (because metaphor) of a very real experience (how the moon appears in reflection on the ocean).
Cites itChrisRRJul 2020
“They discuss a paper that studies the ability of GPT-3 to do arithmetic”
Discusses a paper on GPT-3's ability to do arithmetic
Full comment
Your first point is actually covered in the computerphile video on GPT-3. They discuss a paper that studies the ability of GPT-3 to do arithmetic as it's part of the learned text database Check a look this video
Cites itstcredzeroAug 2021
Illustrates the jump in quality from GPT-2 to GPT-3
Full comment
The industry collectively sank untold billions into the blind belief that neural algorithms will somehow turn into "AI." The huge qualitative differences between GPT-2 and GPT-3 seem to suggest that they will, if you just keep adding orders of magnitude more connections and more data. this video
Cites itCSMastermindJan 2023
“Computerphile summed it up pretty well on GPT-3's release”
Explains GPT-2 and GPT-3 scaling with parameters not leveling off
Full comment
> No one in 2018 suspected that large language models would smoothly scale up by simply increasing the number of parameters. link That was kind of the entire point of GPT-2. Computerphile summed it up pretty well on GPT-3's release: this video Here's some quotes from that video: "The thing about gpt2 is just that it was much bigger than anything that came before. It was more parameters and was kind of the point of that paper." ... "They made gpt2 because the curve wasn't leveling off. We've gone 117 times bigger than gpt2 and they're still not leveling off."
Cites itMichaelMoser123Feb 2023
“they explain that gpt3 (almost) learned how to add and multiply, but they don't know how.”
Explains GPT-3 nearly learned addition and multiplication without anyone knowing how
Full comment
maybe ask your favorite language model for advice. They say it's the answer to all questions, sort of like 42. My point is here: this video - they explain that gpt3 (almost) learned how to add and multiply, but they don't know how. To me that's the thing. (the insight comes at 12:29, 15:23) in a sense that is kind of an admission that we don't know the heck what is going on.
