10.3M views · Feb 5, 2025 · 3:31:24 · 10 HN recommenders · 18 mentions · discussed 2025–2026
Andrej Karpathy's three-hour-plus general-audience deep dive into how ChatGPT-style LLMs are built, covering the full training stack from pretraining to post-training. It also gives mental models for their quirks, like hallucination, and how to use them well.
Why Hacker News points people here: HN's most frequent recommendation for a clear, approachable foundation in how LLMs work without heavy math.
“I highly recommend Andrej Karpathy's Deep Dive into LLMs like ChatGPT”HN comment, 2026
“He goes over a ton of different stuff related to model training, but makes it very easy to understand.”
Long but easy-to-follow overview of LLM training.
I think you'll get a lot of use from this video from Andrej Karpathy: this video It is long, but don't get scared off. He goes over a ton of different stuff related to model training, but makes it very easy to understand.
“the first thing I would recommend is to watch Karpathy's 'deep dive into LLMs' video”
Gives instinctive understanding of LLMs and saves time.
Depending on how much time you have, the first thing I would recommend is to watch Karpathy's 'deep dive into LLMs' video ( this video ). It is 3.5 hours but it will give you so much instinctive understanding of how these things work and will likely save you a lot of time. Purely from a technical standpoint, things aren't that difficult really. I'd recommend get yourself an API account with all major providers (because each frontier LLM evidently has their own strengths and weaknesses) and start experimenting with your data. The feeling you'll get when you first get an API response will keep you going! :) Having said that, a caveat is, LLMs do hallucinate from time to time. If your data needs accuracy, definitely don't do the "throw everything at LLMs and let them do the work" but I'm sure you'll figure it out from trial and error. Good luck!
“Anyway, I really like's Karpathy's video because he's very good at explaining LLMs at every level.”
Approachable deep dive into how LLMs work.
Since you're posting here, you're looking for the shortcut. The shortcut is Karpathy's "Let's Build GPT: from scratch, in code, spelled out" video: link Then there is a good video that dives into LLMs and how they work that is quite approachable: this video From there, flesh out knowledge with his other videos, where he goes both extremely light and extremely deep: link Anyway, I really like's Karpathy's video because he's very good at explaining LLMs at every level.
“Karpathy's excellent "deep dive into LLMs like chatgpt" video[0] explains and demonstrates this aspect pretty well”
Explains and demonstrates LLM stochastic nature.
This is like the LLM era version of the search bubble that prevented people from having the same search results for ostensibly identical searches. Also keep in mind that LLMs are stochastic by design. If you haven't seen it, Karpathy's excellent "deep dive into LLMs like chatgpt" video[0] explains and demonstrates this aspect pretty well: [0] this video
“I highly recommend Andrej Karpathy's Deep Dive into LLMs like ChatGPT”
General intro making sense of the tech without heavy math.
I highly recommend Andrej Karpathy's Deep Dive into LLMs like ChatGPT: this video . It's a 3.5 hour general intro to the topic that I think does a great job of making sense of the tech and how it works under the hood, without needing to build up a foundation of math and programming.
“I highly recommend Karpathy’s intro to LLMs videos on YouTube.”
Worthwhile even with basic LLM understanding.
Even if you have some basic understanding of how LLMs work, I highly recommend Karpathy’s intro to LLMs videos on YouTube. - this video - link
“Karpathy has a good intro video on this.”
Good intro on base training versus post-training.
> The models still _generate_ response token-by-token, but they pick tokens _not_ based on tokens that maximize probabilities at each token. This is also not how base training works. In base training the loss is chosen given a context, which can be gigantic. It's never about just the previous token, it's about a whole response in context. The context could be an entire poem, a play, a worked solution to a programming problem, etc, etc. So you would expect to see the same type of (apparent) higher-level planning from base trained models and indeed you do and can easily verify this by downloading a base model from HF or similar and prompting it to complete a poem. The key differences between base and agentic models are 1) the latter behave like agents, and 2) the latter hallucinate less. But that isn't about planning (you still need planning to hallucinate something). It's more to do with post-base training specifically being about providing positive rewards for things which aren't hallucinations. Changing the way the reward function is computed during RL doesn't produce planning, it simply inclines to model to produce responses that are more like the RL targets. Karpathy has a good intro video on this. this video In general the nitpicking seems weird. Yes, on a mechanical level, using a model is still about "given this context, what is the next token". No, that doesn't mean that they don't plan, or have higher-level views of the overal structure of their response, or whatever.
“There is a very good intro on how LLM work by Kharpaty.”
Very good intro to how LLMs work.
There is a very good intro on how LLM work by Kharpaty. It's this video and it's 3.30 hours, so way below the 100 hours mark :)
“Karpathy's LLM overview is a good high level overview on LLMs”
Good high-level overview of LLMs.
Modal's GPU glossary is a good overview about how GPUs work [0]. Karpathy's LLM overview is a good high level overview on LLMs [1]. 3b1b's video (and subsequent videos) on transformers was excellent at helping me understand the math at a high level [2]. This matrix multiplication optimization worklog helped me understand writing better CUDA (not for beginner intro though) [3]. During this process I also asked ChatGPT a lot of questions. I'm definitely open to suggestions about "how to learn" with all the new tools we have. I felt this has not been straightforward to figure out. [0] link [1] this video [2] link [3] link
“I would suggest you look into another Karpathy's video -- Deep Dive into LLMs like ChatGPT.”
Better fit than training-from-scratch for most users.
This is a good resource, however for about 99.99% of people, you are most likely to just use a foundation model like ChatGPT, Claude, Gemini etc. so this knowledge/training will get you neither here or there. I would suggest you look into another Karpathy's video -- Deep Dive into LLMs like ChatGPT. this video
“he mentioned how hard it is for LLMs to have humor and be funny”
Karpathy explains why humor is hard for LLMs.
Okay, I feel like there might've been a breakthrough here. After watching Karpathy's video [0], he mentioned how hard it is for LLMs to have humor and be funny but it seems like Claude 3.7 really nailed it this time? Like, most of these posts are legit funny. [0] - this video
“He said that hallucination was because in the training data there were no examples where the answer is, "I don't know".”
Karpathy on why models hallucinate.
I watched an Andrej Karpathy video recently. He said that hallucination was because in the training data there were no examples where the answer is, "I don't know". Maybe I'm misinterpreting what he was saying though. this video
Explains how LLMs work, hence why counting lines fails.
Why would you expect them to be able to given how they work? [1] Someone where I work was trying to get an LLM to evaluate responses to an internal multiple-choice quiz (A, B or C), putting people into different buckets based on a combination of the total number of correct responses and having answered specific questions correctly. They spent a week "prompt engineering" it back and forth, with subtle changes to their instructions on how the scoring should work, with no appreciable effect on accuracy or consistency. That's another scenario where I felt someone was asking for something with no mechanical sympathy for how it was supposed to happen. Maybe a "thinking" model (why do "AI" companies always abuse terms like this? (rhetorical)) would have been able to get enough stuff into the context for it to be able to get closer to a better outcome, but I took their prompt asked it to write code instead, and got it translated into some overly-commented but simple-enough code which would do the job perfectly every time, including a comment that the instructions they'd provided had a gap where people answering with a certain combination of answers wouldn't fall into any bucket. [1] this video
Used as an example URL for yt-dlp subtitle fetch.
Can also just fetch the subs already in YouTube rather than retranscribing. eg: yt-dlp --write-auto-subs --skip-download " this video "
“That's actually pretty much what Andrej Karpathy mentions as a mitigation for hallucinations here”
Karpathy mentions a mitigation for hallucinations.
That's actually pretty much what Andrej Karpathy mentions as a mitigation for hallucinations here: this video
“After watching Karpathy's video [1] I've been looking for a good source to do a deeper dive.”
Commenter watched it and seeks deeper material.
Anyone else read the book that the author mentions, Build a Large Language Model (from Scratch) [0]? After watching Karpathy's video [1] I've been looking for a good source to do a deeper dive. [0] link [1] this video
“Karpathy's discussion of native computation: (starts at 1:46:56)”
Karpathy's discussion of native computation in transformers.
That's post-training No, it is not. Read the paper. They are discussing an emergent property of the context itself: "For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model." I'm talking about tasks like multiplying two 4-digit numbers (let's say 8-digit, just to be safe, for reasoning models), which 5th or 6th graders in the US are expected to be able to do with no problem - without using a calculator. So am I. See, for example, Karpathy's discussion of native computation: this video (starts at 1:46:56). The first few tokens in the context actually serve as some sort of substrate for general computation. I don't pretend to understand that, and it may still be something of an open research topic, but it's one more unexpected emergent property of transformers. You'd be crazy to trust that property at this stage -- at the time Karpathy was making the video, he needed to explicitly tell the model to "Use code" if he didn't want it to just make up solutions to more complex problems -- but you'd also be crazy to trust answers from a 5th-grader who just learned long division last week. Feeling quite comfortable being classed in the same camp with the greatest living physicist, honestly. Not a great time for you to rest on your intellectual laurels. Same goes for Penrose.
“that turned out to be almost directly from the video”
Source for the 44 TB training data figure.
I agreed when I read your comment, but that turned out to be almost directly from the video this video From around the 2:20 mark he says: “[…] actually ends up being only about 44 TB of disk space. You can get a USB stick for like a TB very easily, or I think this could fit on a single hard drive almost today” So it’s just slightly altered from what was said in the original video. And the LLM rewritten version of it also says “roughly” where he said “almost”, and I guess 44 TB is pretty roughly or pretty almost 32 TB. Although I’d still personally probably put it as “can fit on a pair of decently sized hard drives today” (for example across two 24 TB drives). Regardless, it’s close enough to what was said in the source video that it’s not something the LLM just made up out of nowhere.
