46K views · Aug 6, 2021 · 1:06:20 · 12 HN recommenders · 20 mentions · discussed 2021–2026
Timothy Roscoe of ETH Zurich delivers the joint USENIX ATC/OSDI 2021 keynote arguing that operating systems must reckon with modern hardware, which is a heterogeneous collection of processors, firmware and coprocessors rather than a simple machine. It shows how much runs below and beside Linux.
Why Hacker News points people here: Repeatedly recommended as a revelatory talk on how little control the main OS has over modern SoCs and PCs.
“There's this great USENIX talk by Timothy Roscoe [1]”HN comment, 2022
“I highly recommend Timothy Roscoe's OSDI keynote[0]”
Roscoe's OSDI keynote on OSes and modern heterogeneous hardware.
I know that I recommended it in the talk, but I highly recommend Timothy Roscoe's OSDI keynote[0] -- and may also be worth catching the Twitter Space we did a few weeks ago discussing it.[1] [0] this video [1] link
“There's this great USENIX talk by Timothy Roscoe [1]”
Critique of unholistic OS design and the many hidden RTOSes.
There's this great USENIX talk by Timothy Roscoe [1], which is part of the Enzian Team at ETH Zürich. It's about the dominant unholistic approach to modern operating system design, which is reflected in the vast number of independent, proprietary, under-documented RTOSes running in tandem on a single system, and eventually leading to uninspiring and lackluster OS research (e.g. Linux monoculture). I'm guessing that hardware and software industries just don't have well-aligned interests, which unfortunately leaks into OS R&D. [1] this video
“I found this talk both riveting and compelling”
Modern hardware is an amalgam of processors, not a Von Neumann machine.
This isn’t my space to opine on, but I found this talk both riveting and compelling: this video In a nutshell, the useful fiction of computer-as-Von-Neumann-meaning doesn’t adequately reflect the reality of modern hardware. Not only does the CPU itself not fit that model (with things like speculative execution, sophisticated power and load management…), but the system as a whole is increasingly an amalgamation of different processors and address spaces.
“After watching the OS talk that Cantrill references [0], I really want to know the answer”
Raises the question of OS design across heterogeneous chips.
Very interesting. This reminds me a ton of L4, with its message passing style. I'm curious if it was inspired by L4. It's a great talk, and my only disappointment was that I was hoping to hear about what comes after. After watching the OS talk that Cantrill references [0], I really want to know the answer to "what does an OS that runs across a bunch of heterogeneous chips look like?" Having Hubris be just a single-node OS was a bit of a let down, so I will be staying tuned for the sequels. [0] this video EDIT: Went to go read the docs and: "The operating system that originally made this point was L4, and Hubris’s IPC mechanism is directly inspired by L4’s"
“First of all, watch this, and learn that there's a whole stack of layers below what Linux can even touch these days.”
Shows the layers of hardware and firmware below Linux.
First of all, watch this, and learn that there's a whole stack of layers below what Linux can even touch these days. this video For a theoretical understanding, with a much simpler model, I suggest the Nand2Tetris course.
“I can recommend the related talk "It's Time for Operating Systems to Rediscover Hardware".”
Modern systems are cooperating devices each with own OS, yet main OS pretends control.
I can recommend the related talk "It's Time for Operating Systems to Rediscover Hardware". [1] It explores how modern systems are a set of cooperating devices (each with their own OS) while our main operating systems still pretend to be fully in charge. [1] this video
“If you find this fact surprising, you'll probably want to take a look at this talk from USENIX ATC '21”
Modern computers are clusters of heterogeneous processors.
> So as someone else commented, a mainframe isn't really a computer, it's a whole cluster of closely-coupled computers, many with dedicated functionality. Quite possibly all implemented with different architectures, instruction sets, programming languages, bit widths, everything. Nearly all computers more powerful than a simple microcontroller look like this in 2022. If you find this fact surprising, you'll probably want to take a look at this talk from USENIX ATC '21: this video
“I suggest taking a look at the talk Bryan mentioned”
Explains the many separate OSes running inside a modern PC.
> I would really like to know if my x86 'linux PC', is actually running another 'hypervisor' OS that runs Linux - it seems like a recipe for security vulnerabilities. There's not so much a "hypervisor" OS, but instead a congealed set of many different OSes, some realtime OSes, some possibly old Linux variants, some possibly closed source proprietary one-off OSes. I suggest taking a look at the talk Bryan mentioned: this video
“A nice talk on this”
Main OS runs sandboxed beneath higher-privilege coprocessors.
The IME gets a lot of hate around here, but let's not get distracted by it: higher-privilege co-processors running code outside the main OS' control is becoming (or already is) the norm everywhere. Intel-based PCs are just one instance of it (and perhaps not even the most egregious one). Most hardware has evolved to effectively run the main OS under a sandbox where it "thinks" it is in control, but isn't. A nice talk on this: this video
“A favorite related to talk is Roscoe's keynote titled "It's Time for Operating Systems to Rediscover Hardware”
Kernels ignore massively heterogeneous SoCs.
A favorite related to talk is Roscoe's keynote titled "It's Time for Operating Systems to Rediscover Hardware" [1] in which he discusses some of the outcomes/consequences of kernels (like Linux) largely sticking their heads in the sand on the issue of these massively heterogeneous SoCs. [1] this video
“Watch from about 10:00 to about 20:00 for a decent super high level overview.”
High-level overview of modern boot, management engines before the OS.
Plenty of good links here already. I'll just add that booting is enormously complex, and starts well before any of the dozen comments so far. I skimmed many of them (not all) and they all tell pretty much the same story whether on real hardware or virtual. But modern real hardware first boots a Management Engine (or BMC, or T2, or similar), which prepares things and starts subsystems like hardware controllers, before releasing the CPU where the "OS" finally boots, blissfully unaware that it is running on an abstraction. Watch this video from about 10:00 to about 20:00 for a decent super high level overview. These two are also good for a deeper understanding. link link They both cover UEFI, even though the focus is on T2. I know that's well beyond what you were asking for, but the boot process goes well beyond GRUB.
“I recently listened to this”
Modern SoCs break the simple computer model.
I'd say the computer architecture can vary _greatly_ and there are a bunch of cores in modern SOC's that kind of break the assumption of how a computer works. I recently listened to this: this video Cache coherence, accelerators, various takes on pipelining, ISA extensions. There are many many things that change all the time. Not even touching the SW glue stuff that changes. I don't know the history of react and what is going on in modern browsers, but I suspect there are OS constraints, evolving web standards, extra capabilities, evolving browsers that shape and force the framework to adapt. I am doubtful people change it just for the sake of it.
“I recently came across Timothy Roscoe's Usenix talk[1] on OS development on HN”
I feel like this generally applies to all software developers. Software and its systems have largely evolved through accretion and that creates a strong network effect and eventually, a taller application stack. We're continually making everything more complicated because it's just easier to use other people work, especially when the barrier to entry for that work is low, i.e: Web/JS. As a slight segue, I recently came across Timothy Roscoe's Usenix talk[1] on OS development on HN and I'm starting to think that a complete rebuild of modern computing might occur when the limits of the old OS architectures become too burdensome, at which point there could be an opportunity to "fix" the application stack and try something simpler. I sincerely hope the OS's of the future do not look like Linux or even attempt POSIX compliance. [1] this video
“Roscoe, who gives the 2021 keynote that Cantrill recommends”
Reminds me of Your computer is already a distributed system. Why isn’t your OS? from HotOS 2009: link We argue that a new OS for a multicore machine should be designed ground-up as a distributed system, using concepts from that field. Modern hardware resembles a networked system even more than past large multiprocessors: in addition to familiar latency effects, it exhibits node heterogeneity and dynamic membership changes And now that I look at the author list, I see Roscoe, who gives the 2021 keynote that Cantrill recommends: this video
“only three were not about Linux!”
OSDI keynote noting only three of 103 OS papers were not about Linux.
Related: The keynote presentation from OSDI (Operating Systems Design and and Implementation) online conference: “It’s Time for Operations Systems to Rediscover Hardware”: this video The presenter says that, of the 103 papers about operating systems published by OSDI in 2020 and 2021, only three were not about Linux!
“At 9:00 the speaker lays out some pretty conventional criteria for an operating system”
This reminded me of a recent talk: USENIX ATC '21/OSDI '21 Joint Keynote Address-It's Time for Operating Systems to Rediscover Hardware: this video At 9:00 the speaker lays out some pretty conventional criteria for an operating system (multiplex hardware resources, abstract the hardware platform , protect software principals from each other), and points out that these things are increasingly outside the scope of what people call "the OS". link
“The operating system is not really in charge of anything”
> Maybe we could do all the "smart" stuff in the OS and application code, and just attach commodity dumb flash chips to our computer. Yeah, this is how things are supposed to be done and the fact it's not happening is a huge problem. Hardware makers isolate our operating systems in the exact same way operating systems isolate our processes. The operating system is not really in charge of anything, the hardware just gives it an illusory sanboxed machine to play around in, a machine that doesn't even reflect what hardware truly looks like. The real computers are all hidden and programmed by proprietery firmware. this video
I'm not a RISC-V expert, primarily relying on open-source firmware community knowledge and the opinion of such figures as Ron Minnich, but AFAIK, most firmware for production RISC-V deployments is closed-source, so we can't say if vendors use OpenSBI implementation or some modified version for their malicious purposes. If I need to be corrected, please point me to products that state how they transparently use SBI. There is evidence of transparent use of SMM in coreboot. SBI is not the only RISC-V TEE. There are also other concepts, like MultiZone [1] and PMP-based Keystone [2], advertised as trusted execution environments The openness of specification only matters a little here - UEFI specification about management mode is also open. But IBVs screwed us many times [3] by implementing low-quality SMIs that could elevate privileges, and the spec does not guarantee that implementers follow it. Lack of tooling makes compliance difficult - it slowly changes with various startups seeing that as an opportunity (e.g., Binarly, Eclypsium?). My point is that SMM as TEE is not a unique x86 feature, and other significant architectures have similar mechanisms. OS has a means of figuring out that it was in SMI, so it is not entirely hidden but still has superpowers. In the CNC world SMI latency is measured from OS [4]. Trusted execution environments are hammers and can be used for a good purpose in a transparent way and for malicious purposes. Further typically depends on the trustworthin
“For more”
Linux on Android doesn't multiplex or protect software principals from each other. It did for a brief time, but that led to massive vulnerablilities. Because Linux view of hardware (nice symmetric system with homogeneous cores running single OS with shared memory address space) was different from what was the actual state. For that reason hardware was abstracted away from Linux and it's only used to run Linux programs. It's not containerized. It's isolated to an enclave where damage it can do is minimal. Android OS is essentially a ball of propietary firmware mud. With Linux programs and Dalvik JVM. For more: this video
“Tim Roscoe gave an interesting Keynote at OSDI '21 titled”
Tim Roscoe gave an interesting Keynote at OSDI '21 titled "It's Time for Operating Systems to Rediscover Hardware" - this video . He was involved with the Barrelfish project.
