17.7M views · Sep 30, 2023 · 20:41 · 1 HN recommender · 4 mentions · discussed 2024–2025
Yosh trains a reinforcement learning agent to drive Trackmania until he can no longer beat it. The video explains how the training works and the design choices involved.
Why Hacker News points people here: Recommended as an approachable explanation of how reinforcement learning works in practice, and cited for the constraints needed to stop exploits.
“be sure to check out the videos by Yosh. They're amazing, and do a great job explaining how reinforcement learning works in practice”HN comment, 2024
“be sure to check out the videos by Yosh. They're amazing, and do a great job explaining how reinforcement learning works in practice”
Amazing explanations of how RL works in practice
For those who only read the comments, be sure to check out the videos by Yosh. They're amazing, and do a great job explaining how reinforcement learning works in practice: this video link
“See for example this training experiment for Trackmania [1].”
Example of encoding extra info in RL training
>The entire point of this is to not encode the board state! I am not sure about this. From the article "The 50M parameter model played at 1300 ELO with 99.8% of its moves being legal within one day of training." I thought that the experiment was how well the model will perform, given that it's reward function is to predict text, rather than checkmate. Leela, Alpha0 their reward function is to win the game, checkmate or capture pieces. Also it goes without saying that Leela, Alpha0 cannot make illegal moves. The experiment does not need to include the whole board position if that's a problem, if that's an important point of interest. It could encode more information about squares covered by each side for example. See for example this training experiment for Trackmania [1]. There are techniques that the ML algorithm will *never* figure out by itself if this information is not encoded in it's training data. The point still stands. PGN notation certainly is not a good format if the goal (or one of the goals) of the experiment is to be a good chess player. [1] this video
“it lets you get a world record and a YouTube video with 14M views.”
Depends what you consider fun, and how far you take it. Some people enjoy programming more than repetitive clicking in a GUI. For a clicker game, writing a bot lets you iterate on strategies easier - is it faster to get to level 2 if I buy the upgrade for A or B first? For Trackmania, it lets you get a world record and a YouTube video with 14M views. this video
“someone recently used RL to solve trackmania, and ends up having to add all sorts of constraints/penalties”
RL on Trackmania needed constraints to prevent glitch exploits
There’s a few very cool examples where someone recently used RL to solve trackmania, and ends up having to add all sorts of constraints/penalties to prevent extremely strange exploits/glitches that are discovered IIRC… been a while since I watched. this video link
