4.4M views · Aug 16, 2022 · 2:25:52 · 17 HN recommenders · 20 mentions · discussed 2022–2025
Andrej Karpathy's first Neural Networks: Zero to Hero lecture, which builds micrograd, a tiny autograd engine, from scratch in Python. It explains backpropagation and neural network training step by step, assuming only basic Python and some high-school calculus.
Why Hacker News points people here: HN treats it as the best starting point for really understanding backprop and neural nets, because Karpathy explains the why with intuition and builds everything in code.
“It's especially useful because he explains the why and provides intuitive explanations instead of just talking about the what and how.”HN comment, 2023
“I recommend watching Andrej Karpathy's recent videos on backprop and neural nets. Makes it really simple and intuitive.”
Makes backprop and neural nets simple and intuitive.
I recommend watching Andrej Karpathy's recent videos on backprop and neural nets. Makes it really simple and intuitive. The spelled-out intro to neural networks and backpropagation: this video Becoming a backprop ninja: link
“Andrej has a brilliant series of videos on youtube where he explores the basics of neural nets right up to transformer architecture through code.”
Series from neural net basics up to building GPT.
For those who dont know, Andrej has a brilliant series of videos on youtube where he explores the basics of neural nets right up to transformer architecture through code. Starts here [0] and most recent is building GPT from scratch in code. this video
“The one that really hit it home for me was the first video on MicroGrad”
Micrograd video explains basic neural networks very well.
The ChatGPT video by Karpathy is the last in a 7 video series. The one that really hit it home for me was the first video on MicroGrad [1]. The second video builds on MicroGrad and is also great for understanding how a basic NN works. [1] this video
“I highly recommend the video from karpathy”
Spells out expression graphs and manual gradients in code.
I highly recommend the video from karpathy this video - This explains the same idea spelled out in code. It really helped me understand the mathematical expression graph of neural networks. He uses graphviz to show the expressions and initially calculate gradients manually and then implement back propagation.
“It's especially useful because he explains the why and provides intuitive explanations instead of just talking about the what and how.”
Explains the why with intuitive explanations.
Andrej Karpathy's "zero to hero" series [1] was how I learned the fundamentals of this stuff. It's especially useful because he explains the why and provides intuitive explanations instead of just talking about the what and how. Would recommend it if you haven't checked it out already. [1] this video
“Watch Karpathy's recent lectures. They're gold.”
Start of Karpathy's lectures building neural nets from the ground up.
Watch Karpathy's recent lectures. They're gold. Start here[1] with micrograd[2]. It doesn't use linear algebra/matrices to start, but the principles are the same. The matrix multiplication is how the weights of the connections between neurons and the input values are combined (to form an activation value that then may lead to that neuron "firing" or not, depending on whether it passes some threshold function). We use matrices to model the connections between neurons - each row is a connection, and each column is a weight corresponding to an input. [1] this video [2] link
“One of the best introductory series (even if you ignore the neural net part of it)”
Intuitive intro to gradients and differentiation.
In the same vein, Karpathy's video series "Neural Networks from zero to hero"[0] touches upon a lot of this and intuitions as well. One of the best introductory series (even if you ignore the neural net part of it) and brushes on gradients, differentiation and what it means intuitively. [0] this video
“its actually Karpathy who teaches us :) so we can understand them and make our own.”
Core study resource that teaches people to understand and build models.
For studying? Mainly watching and re-watching Karpathy's 'Zero To Hero'[1] and Stanford's 'Introduction to Convolutional Neural Networks for Visual Recognition'[2], also a lot of transformers from scratch videos like Umar Jamali's videos[3], and I also study backwards to McCulloch and Pitts. Reading the 30 papers link and so on. And of course Yannic Kilcher[4], and also listening in on the paper discussions they do on discord. Practicing a lot with just doing backpropagation by hand and making toy models by hand to get intuition for the signal flow, and building all kinds of smallish systems, e.g. how far can you push whisper, small qwen3, and kokoro to control your computer with voice? People think that deepseek/mistral/meta etc are democratizing AI, but its actually Karpathy who teaches us :) so we can understand them and make our own. [1] this video [2] link [3] link [4] link
“Andrej Karpathy's awesome micrograd lecture”
Micrograd lecture explored in a blog post.
This blog post analyzes the code with an emphasis on the small details and ideas explored in Andrej Karpathy's awesome micrograd lecture. Link to the original lecture: this video I'd love to hear any thougts/reviews about the blog post.
“I highly recommend this youtube playlist from Andrej Karpathy”
Backprop fundamentals through building a GPT-2 clone.
If you have 20 hours to spare I highly recommend this youtube playlist from Andrej Karpathy this video It starts with the fundamentals of how backpropagation works then advances to building a few simple models and ends with building a GPT-2 clone. It won't taech you everything about AI models but it gives you a solid foundation for branching out.
“Karpathy has a relatively small youtube channel he started a few weeks ago.”
Karpathy's new YouTube channel on neural networks.
I love your approach and philosophy around programming. If anyone is unaware, Karpathy has a relatively small youtube channel he started a few weeks ago. this video
“first video is literally from scratch (i.e. just Python”
First video of the series builds from scratch in plain Python.
this video The entire series is at link , first video is literally from scratch (i.e. just Python, the only external dependency is for plotting graphs if I recall correctly).
“I made a start on it today and found it quite accessible.”
Accessible playlist explaining neural networks.
I think this is a good take. I hope to do the same. For the understanding part, Andrej Karpathy has a YouTube playlist that explains neural networks. I made a start on it today and found it quite accessible. this video
“if you don't have the background, I'd recommend my YouTube videos, see the Zero To Hero playlist”
Recommended for those lacking background, by the video's author.
I wrote this, which might be a bit helpful: link But if you don't have the background, I'd recommend my YouTube videos, see the Zero To Hero playlist: this video
“Andrej Karpathy has a good series of videos going over a lot of the concepts starting here”
Practical series covering many concepts of neural networks.
imo linear algebra is the main thing to refresh yourself on, as most of NN is based on matrix multiplication. The calculus is mostly in the gradient descent which in practice is abstracted away, so having a few decades old understanding might be good enough. For practical implementation Andrej Karpathy has a good series of videos going over a lot of the concepts starting here this video
“covers the basic ideas of backpropagation and autodiff frameworks like PyTorch”
Covers backprop and autodiff basics, ending in a tiny framework.
Andrej Karpathy's guide to Micrograd - this video - is probably the closest equivalent, in that it covers the basic ideas of backpropagation and autodiff frameworks like PyTorch, and ends with an implementation of a very basic framework ( link ) you can train small neural networks in
“neural networks zero to hero is for understanding how neural networks are built from ground up”
Builds neural networks from the ground up.
From my other comment elsewhere. These resources helped me understand the topics better. If anyone wants to understand fundamentals of machine learning, one of the superb resources I have found is, Stanford's "Probability for computer scientists"[1]. It goes into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. But, this is a primarily a probability course that discusses the fundamentals of machine learning. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familiarity with linear algebra topics) There is a course reader for CS109 [2]. You can download pdf version of this. Caltech's learning from data was really good too, if someone is looking for theoretical understanding of ML topics [3]. There is also book for excellent caltech course[4]. Also, neural networks zero to hero is for understanding how neural networks are built from ground up [5]. [1] link [2] link [3] link [4] link [5] this video
That concept is not the easiest to describe succinctly inside a file like this (or -- while we are completely at it, in a Hacker News post like this!), I think (especially as there are various levels of 'beginner' to take into account here). This is considered a very entry level concept (not as an insult -- simply from an information categorization/tagging perspective here :D :)), and I think there might be others who would consider it to be noise if logged in the code or described in the comments/blogpost. After all, there was a disclaimer that you might have missed up front in the blogpost! "This post assumes familiarity with Python, NumPy, and some basic experience training neural networks." So it is in there! But in all of the firehose of info we get maybe it is not that hard to miss. However, I'm here to help! Thankfully the concept is not too terribly difficult, I believe. Effectively, the loss function compresses the task we've described with our labels from our training dataset into our neural network. This includes (ideally, at least), 'all' the information the neural network needs to perform that task well, according to the data we have, at least. If you'd like to know more about the specifics of this, I'd refer you to the original Shannon-Weaver paper on information theory -- Weaver's introduction to the topic is in plain English and accessible to (I believe) nearly anyone off of the street with enough time and energy to think through and parse some of the concepts
“I think the closest thing for pytorch is the Karpathy video”
Closest equivalent for learning how PyTorch works internally.
I think the closest thing for pytorch is the Karpathy video: this video There's also an old book on the internals of numpy - not sure how out of date it is though.
“I just skimmed through the article and immediately noticed that the code was taken from Andrej Karpathy's lectures”
I just skimmed through the article and immediately noticed that the code was taken from Andrej Karpathy's lectures [0][1]. This is not a bad thing since the author included the bibliography. BTW, the author should replace "Andrej Karaphy". [0] link [1] this video
