150K views · Dec 16, 2016 · 1:07:41 · 9 HN recommenders · 17 mentions · discussed 2016–2025
Python core developer Raymond Hettinger explains how Python's dictionary implementation evolved into the compact, ordered dict in Python 3.6. He uses pictures and small bits of pure Python, and covers key-sharing, compaction and versioning.
Why Hacker News points people here: HN treats it as the definitive, engaging talk on hash table internals and Python dict ordering, and recommends it in tech-talk lists.
“one of the most insightful videos I've seen in a long time.”HN comment, 2016
“one of the most insightful videos I've seen in a long time.”
Insightful talk on evolution of Python dict implementation.
Raymond Hettinger gave a presentation a few weeks ago on the evolution of the Python dict implementation - one of the most insightful videos I've seen in a long time. this video
Offered as a favourite tech talk.
Python dictionaries: A Confluence of Great Ideas (also known as Modern Dictionaries) by Raymond Hettinger this video
“personally, I'd watch any video with "Raymond Hettinger" in the title.”
State of the art in Python dict implementation.
this video Modern Dictionaries by Raymond Hettinger The official description of this talk is below, but personally, I'd watch any video with "Raymond Hettinger" in the title. > Python's dictionaries are stunningly good. Over the years, many great ideas have combined together to produce the modern implementation in Python 3.6. This fun talk is given by Raymond Hettinger, the Python core developer responsible for the set implementation and who designed the compact-and-ordered dict implemented in CPython for Python 3.6 and in PyPy for Python 2.7. He will use pictures and little bits of pure python code to explain all of the key ideas and how they evolved over time. He will also include newer features such as key-sharing, compaction, and versioning. This talk is important because it is the only public discussion of the state of the art as of Python 3.6. Even experienced Python users are unlikely to know the most recent innovations.
“Raymond Hettinger made an interesting talk about Python's dict (hash table)”
Talk on how Python's dict evolved over time.
Raymond Hettinger made an interesting talk about Python's dict (hash table) and how Python's core development team evolved it over time. He's giving the polished version of talk at PyCon this May but if you can't wait for it, here's a rough version: this video
“is a quite interesting and technical talk about these changes”
Technical talk explaining dict changes and iteration order.
This has been changed in Python 3.6 only, due to a change in dictionary implementation to make them more efficient. "Modern Dictionaries by Raymond Hettinger" [1] is a quite interesting and technical talk about these changes, explaining also the change in iteration order IIRC, worth a watch in my opinion! [1] this video
“Talk by Raimond Hettinger about the new implementation with several other implementations”
Talk on new compact dict implementation and alternatives.
Actually they optimized the hashmap, and it turned out a side effect of the optimization was insertion order. First implemented in pypy, then ported to cpython 3.6. Talk by Raimond Hettinger about the new implementation with several other implementations: this video
“I don't know if it is part of the bests, but very interesting anyway”
Talk on Python 3.6 dictionary improvements.
I don't know if it is part of the bests, but very interesting anyway: Modern Dictionaries by Raymond Hettinger [1] It's about Python 3.6 dictionaries improvements. [1] this video
“This is a very interesting talk about the new tech”
Talk on the new ordered dict implementation.
As far as I know, it was implemented by Raymond Hettinger. This is a very interesting talk about the new tech: this video
“He covers the history of dictionaries in python and how they evolved.”
Covers history and evolution of Python dictionaries.
Raymond Hettinger does or did a few talks on dictionaries too. He covers the history of dictionaries in python and how they evolved. It was mentioned in the article's comment section. this video
“I'd recommend Raymond Hettinger's talk on how Pythons hash map data structure has evolved over time”
How Python's hash map evolved over time.
You're welcome! I deliberately didn't go in to that for a few reasons. 1. It would have made this article very, very long. 2. It's a bit out of scope for an article on hashing. 3. I think I might give hash maps their own article in future. Hash maps are fantastically deep. So many different ways to do it. You'll find a lot of material online but I'd recommend Raymond Hettinger's talk on how Pythons hash map data structure has evolved over time: this video .
Cited for dict ordering starting in Python 3.6.
I think this behavior will only be working starting from Python 3.6, as the dictionary obtained by calling `dict.fromkeys()` before this would not keep the ordering [1] [1] this video
Listed among talks on Python dictionaries.
The talks by Brandon Rhodes are these two: * The Mighty Dictionary (PyCon 2010): link * The Dictionary Even Mightier (PyCon 2017): link There's also one by Raymond Hettinger, which seems to have been given first at a meetup and then, in shorter form, at PyCon 2017: * Modern Dictionaries: this video * Modern Python Dictionaries - A confluence of a dozen great ideas (PyCon 2017): link
“raymond hettinger gave a talk about it.”
Talk about the dict improvement in 3.6.
3.6 had a couple of big improvement - f strings, annotation, and if I'm not mistaking, the devs did something to improve the dictionaries; from 3.6 on dicts are ordered. I don't know too much about it but raymond hettinger gave a talk about it. this video
“Sounds like a fun talk.”
Talk explains deletion holes and ordering origin.
This implementation was used from 3.6, right? It's interesting that the idea mail mentions that nothing changes about the implementation (including order) but the memory layout. Which would imply insertion order was already preserved in older versions (not the case IIRC) or the idea underwent a few more changes that did in fact impact order. EDIT: I couldn't quite find an answer but link mentions the behavior happens since then because the implementation only tracks indices in the hash table itself and relies on maintaining entries separately in a second array that gets expanded in insertion order. This would also seem straightforward but it raises a few questions such as how deletion is implemented (efficiently). EDIT2: Okay, the talk ( this video ) mentions they just leave holes until the next resize (at around 42:00). Raymond also mentions there that his original idea didn't preserve ordering but happened due to an additional compacting optimization? Should probably watch the whole thing some time to get the history. Sounds like a fun talk.
Cited for Python dict ordering history.
I was going to make a tongue-in-cheek suggestion about randomizing undefined behaviour, but I suppose that's common in debugging and even seen in some APIs. At one time, in Python, dictionary (hashmap) ordering was randomized so users wouldn't rely on non-specified ordering behaviour. Eventually though, dictionaries were guaranteed insertion ordering. this video
Cited for Python dict quality and attention.
The python dict() is already high-quality and high-performance. You're probably not going to be able to do much better than the current state. If you want faster you end up moving your data into an environment that lets you make more assumptions and be more restrictive with your data and operate on it there. It's how numpy, polaris, and pandas work. Everything in Python is a dict, there's no data structure that's been given more attention. this video
“he did a well known talk called”
Well-known talk with quip about rediscovering 1960s solutions.
I think Raymond Hettinger is called out specially here because he did a well known talk called [Modern Dictionaries]( this video ) where around 32:00 to 35:00 in he makes the quip about how younger developers think they need new data structures to handle new problems, but eventually just end up recreating / rediscovering solutions from the 1960s. “What has been is what will be, and what has been done is what will be done; there is nothing new under the sun.”
