391K views · May 1, 2023 · 55:02 · 1 HN recommender · 4 mentions · discussed 2023–2024
Tom 7 (suckerpinch) delivers a long, comedic digression on gradient descent and floating-point imprecision. He shows how rounding behavior can supply the nonlinearity needed to build a neural network.
Why Hacker News points people here: Cited as a fun, unconventional example of neural networks built from floating point error.
“A fun example of that (making a neural network using floating point error as a source of non-linearity)”HN comment, 2024
“A fun example of that (making a neural network using floating point error as a source of non-linearity)”
Neural network using floating point error as source of non-linearity
A fun example of that (making a neural network using floating point error as a source of non-linearity): this video
I don’t get what people are trying to say when they say these kinds of things about AI. That human-level writing is as simple as a linear regression? That we could’ve had computer programs capable of human-level writing decades ago? Have they not used these AIs enough to see how powerful they are? Are they seeing the bad outits and thinking that AIs are always doing that poorly? Like seriously, if you’re telling me that it was obvious that a “linear regression” could pass the LSAT I’ve got a macvlan to sell you. Edit: they’re also literally not linear regression! this video
“or for a sillier example”
Silly example of an unconventional activation function
> squash an output into a range This isn't the primary purpose of the activation function, and in fact it's not even necessary. For example see ReLU (probably the most common activation function), leaky ReLU, or for a sillier example: this video
“as a joke, they should implement GPU fluint8 and get baked in non-linearity for the activation function”
Cool stuff. I looked at link and I noticed that that the fp8 support is only for the tensor cores and not the CUDA side. Does that mean training with H100 GPU in fp8 mode would use some software ecosystem that's not the existing vast existing CUDA one? Or am I just misunderstanding CUDA cores vs tensor cores? PS, as a joke, they should implement GPU fluint8 and get baked in non-linearity for the activation function without even using a non-linear function, this video ("GradIEEEnt half decent: The hidden power of imprecise lines" by suckerpinch)
