ghOSt: Fast & Flexible User-Space Delegation of Linux Scheduling

Where threads run, and for how long, strongly affects the throughput, tail latency and security of data center workloads. Yet kernel schedulers are hard to change: they are difficult to implement, test and roll out across a large fleet.

This SOSP 2021 paper from Google presents ghOSt, an infrastructure that delegates kernel scheduling decisions to code running in userspace on Linux.

ghOSt's contribution is a general-purpose delegation mechanism, not one new scheduling policy.

• Policies live in userspace agents, written in any language and updated without rebooting the host.
• The kernel provides state encapsulation (what the agent needs to know), communication and action mechanisms, and helps with synchronization between the kernel and the agent.
• It supports a wide range of models, from per-CPU to centralized and from run-to-completion to preemptive.
• It is meant to make practical the bespoke scheduling policies that recent research built into custom data plane operating systems, which are hard to deploy per application in a multi-tenant environment.

The design adds a ghOSt scheduling class to the Linux kernel. Threads managed by ghOSt are scheduled according to decisions that userspace agents send back, and the agents receive the state they need from the kernel. The public repository describes decisions being actuated as transactions and includes example policies such as centralized and per-CPU FIFO, a CFS-like policy and a Shinjuku-style policy.

The authors evaluate ghOSt on academic workloads and on real-world ones, including Google Snap and Google Search, by comparing it with the kernel scheduler.

These are the authors' reported results, as stated in the abstract.

• Using ghOSt instead of the kernel scheduler, the authors quickly reach comparable throughput and latency.
• On top of that, it enables policy optimization, non-disruptive upgrades and fault isolation for their data center workloads.
• ghOSt incurs low overheads for scheduling actions across the supported scheduling models.
• The implementation is open source.

This analysis is based on the abstract and the project's public documentation. For numbers on specific workloads, go to the paper.

The idea answers a real deployment problem, and the open source release makes it possible to try.

Points to read carefully:

• "Comparable" performance is the headline: the pitch is flexibility and operability more than speed, so the value depends on how much a custom policy gains for your workload.
• Moving decisions out of the kernel adds a communication step between the kernel and the agent. Check in the paper what the centralized and per-CPU models cost in CPU and latency.
• The abstract promises fault isolation. Check in the paper what happens to the threads when an agent crashes or falls behind.
• It needs a modified Linux kernel, which limits adoption outside operators who run their own kernels.

• Upstreaming the delegation interface into mainline Linux, or an equivalent that does not need a fork.
• Policies that use application knowledge better, such as request types or deadlines, and how to express them safely.
• Verifying userspace policies, since a bad policy can starve threads.
• Combining delegated scheduling with other userspace datapath systems (see Snap and Demikernel).