Enabling High-Performance and Secure Userspace NVM File Systems with the Trio Architecture

Non-volatile memory (NVM), such as Intel Optane persistent memory, is reachable with ordinary load and store instructions, and the MMU can enforce permissions on its pages. That invites userspace library file systems (LibFSes) that link into each application, skip the kernel on the data path and can be customized per workload without privileges. The catch is metadata integrity: with direct access, a malicious or buggy application can corrupt metadata that other applications trust. A trusted entity could validate it, but each customized LibFS lays its data out differently, so the entity cannot understand it.

Earlier designs pick one side. Some mediate every metadata update through a trusted kernel or process (Aerie, Strata, SplitFS, KucoFS), which the authors say costs performance, bakes the file system design into the trusted entity and makes customization need privileges. Others (ZoFS, ctFS, MadFS) update metadata directly and give up integrity in whole or in part, which leaves attacks such as rewriting an index pointer so that a victim copying a file writes its own DRAM contents to the destination, holding a lock forever, or removing a non-empty directory.

This SOSP 2023 paper (Koblenz), from EPFL and Rutgers University, proposes Trio, an architecture meant to keep direct access, unprivileged private customization and metadata integrity at once. Terms to keep apart: core state is what a file system cannot regenerate (file contents, names, permissions, the directory hierarchy), and auxiliary state is whatever can be rebuilt from it (file descriptors, locks, caches, bitmaps). The kernel controller grants NVM pages and inodes through the MMU, and the integrity verifier is a trusted userspace process. A trust group is a set of mutually trusting processes of one user. ArckFS is the POSIX-like file system built on Trio, and KVFS and FPFS are two customized ones.

Worth knowing first: how DAX-style access to persistent memory works, page-table permissions, and file system basics (inodes, directory entries, journaling, fsck-style checks). The paper takes secure direct access plus application-level customization from Exokernel and Arrakis, neither of which is in this library. Three pairings of mine, none cited by the paper: Demikernel applies the library-OS idea to datacenter networking, Oxbow (OSDI 2026) revisits the same user-level versus kernel file system trade-off but keeps the kernel page cache and protection, and cache_ext lets each cgroup load its own page cache policy, another case of per-application customization that does not disturb others.

• State separation. The authors treat core state as hard state and auxiliary state as soft state. Trio fixes the layout of a single core state and shares it among all LibFSes and the trusted entities. A LibFS cannot change that layout, but it can redesign its private auxiliary state freely.
• Components and sharing. Each application has its own LibFS. The in-kernel controller programs the MMU to grant access, and the verifier inspects core state. Sharing is per file, with concurrent readers or one exclusive writer. When a writer releases a file, the controller unmaps it and sends it to the verifier, and only if it passes is the file mapped for the next LibFS, which rebuilds its auxiliary state from the core state. The guarantee is that corruption stays confined to the application that caused it. Trust groups skip these costs by sharing one LibFS, and the default trust boundary is a process.
• ArckFS core state and invariants. The core state is minimal: a superblock, a shadow inode table kept by the controller, and file pages. Files and directories are index pages pointing at data pages, and an inode is co-located with its directory entry, so create, stat and delete need only the parent directory's permission. The shadow inode table holds the true permissions, so chmod and chown go through the controller. The verifier, adapted from e2fsck checks but run online on one file, enforces four invariants: valid inode and directory entry fields (file type, no duplicate names, no / in names), valid and not doubly referenced inodes and pages (checked against the controller's records), a connected directory tree (hard links are unsupported, so cycles are already excluded), and correct permissions.
• Corruption and crashes. Before granting write access, the controller checkpoints the file's metadata. If verification fails, the writer gets a timeout to repair it, otherwise the controller keeps the corrupted file private to that writer to avoid data loss and reverts a copy to the checkpoint. A commit call lets a LibFS replace the checkpoint after a passing check. Core state fixes no crash mechanism, so each LibFS registers a recovery program, and because it is untrusted the verifier runs after recovery on every file that was writable at the crash. ArckFS makes metadata operations synchronous and atomic, mostly with 16-byte atomic updates and undo logs for rename, and data operations synchronous but not atomic. Since operations persist at once, fsync is ignored.
• ArckFS data path. A radix tree per file, with a readers-writer inode lock and a range lock, lets threads write disjoint regions in parallel. Directories use a resizable chained hash table, a logging tail per non-full data page (extending NOVA's per-inode log) and an index tail, with per-CPU allocators and journal. For Optane it adopts OdinFS's opportunistic delegation: kernel threads on each NUMA node perform NVM accesses through per-application ring buffers, except small ones (reads under 32KB, writes under 256 bytes). The kernel controller is a Linux 5.13.13 module of 3,791 lines, the LibFS has 7,586 and the verifier 457.
• Customization. KVFS adds get and set calls without file descriptors, holds files of up to 32KB in a fixed array instead of a radix tree and uses a spinlock. FPFS keeps one global hash table from full path to directory entry, which removes directory traversal. Both sit on ArckFS's core state, so no trusted component changes. Customization is limited to auxiliary state, and a design that changes core state, such as a conventional log-structured file system, needs privileges.

The authors use an eight-socket machine with 224 cores (Xeon Platinum 8276L) and 6,144 GB of Optane PM, running Ubuntu 20.04 and Linux 5.13.13. The baselines are five in-kernel file systems (ext4 with DAX, PMFS, NOVA, OdinFS, WineFS) and two userspace ones, SplitFS (metadata through ext4) and Strata (metadata through a privileged process), which runs only with one thread. ArckFS runs with all eight NUMA nodes and 12 delegation threads each, or as ArckFS-no-dele on one node without delegation, and ext4 also runs on a RAID0 of the nodes.

Microbenchmarks are fio (a private 1GB file per thread) and FxMark. Macrobenchmarks are four Filebench workloads and LevelDB db_bench with one thread, 100-byte values and one million objects. For integrity, the authors handcraft eleven attacks, script a corruption for each verifier check and combine scripts, for 134 scenarios in total. Sharing cost uses two applications writing 4KB to a 2MB or 1GB file, or creating files in a directory of 10 or 100 files. Customization runs eight threads on Webproxy extended with a key-value interface (KVFS) and on Varmail with directories 20 levels deep (FPFS).

These are the authors' reported results.

• Single thread. At 4KB, ArckFS-no-dele and SplitFS are 9% to 31% over NOVA thanks to direct access, while delegation makes ArckFS 21% slower than ArckFS-no-dele but still about 6% ahead of NOVA. At 2MB ArckFS leads by 3.1× to 25× on reads and 2.0× to 15× on writes. For open, create and delete it leads by 1.6× to 5.6×, 3.3× to 5.3× and 7.4× to 9.4×. In create, NOVA and Strata spend at least 42% and 44.5% of the time in VFS and digestion.
• Data scalability. On eight NUMA nodes at 224 threads, ArckFS beats OdinFS by up to 1.3× and other file systems by up to 22× on 4KB I/O. In FxMark data workloads it scales linearly on reads and holds peak throughput on writes, 3.2× over OdinFS and up to 850× over others. On one node the 4KB gain is 10% to 12%, and at 2MB all file systems are similar.
• Metadata scalability. At 224 threads ArckFS leads by 5.4× to 334× on open, 7.4× to 25× on enumerate, 2.3× to 21.2× on create, 8.9× to 32.7× on unlink and 16.2× to 36.4× on rename. The authors say VFS bottlenecks (coarse locks on the directory and inode caches, the global rename lock) cap the others. ArckFS does not scale linearly on MWCL and MWUL, because small accesses are not delegated, and drops on MWCM, MWUM and MWRM from hash table contention.
• Integrity. The verifier detects the corruption in all 134 scenarios, and the controller restores a consistent file.
• Sharing cost. For small files and directories it is negligible (under 5 µs) on writes and modest on creates, close to NOVA. With two writers on a 1GB file ArckFS reaches 0.25 GiB/s against 1.95 GiB/s with a trust group, the authors' 7.8× overhead, and 99% of it is mapping and unmapping at 670 ms per iteration against a 100 ms lease. Creating in a 100-file directory takes 32.7 µs against 1.6 µs (20.4×), dominated by verification (300 µs, 81%) and rebuilding auxiliary state (12%).
• Filebench. On eight NUMA nodes ArckFS leads by 1.1× to 27.3× on Fileserver and 1.2× to 4.1× on Webserver, and with up to 16 threads by 2.2× to 8.0× on Webproxy and 2.4× to 34.2× on Varmail.
• LevelDB. In Table 5 (ops per ms), ArckFS reaches 291 on Fill sync against 211 for WineFS, the second best, and 17 for ext4, and 144 on Read random against 142 and 93. The authors summarize it as up to 3.1× over WineFS and 1.5× to 17× over ext4. ArckFS-no-dele is 21% to 34% faster than ArckFS on small-file workloads and 29% slower on Fill 100K.
• Customization. KVFS beats ArckFS by 1.3× and the others by 2.9× on Webproxy. FPFS beats ArckFS by 1.2× and the others by 21× on Varmail with deep directories.

The paper is candid about what direct access with verification costs. It concedes the following.

• Not fully POSIX. Trio cannot allow concurrent writes to one file from several untrusted applications, the MMU cannot enforce a directory's search permission, and some timestamps, such as the access time of a read-only file, cannot be kept exactly.
• Costs and scope. Sharing a file among untrusted writers pays for mapping, verification and rebuilding, which trust groups avoid. Customization is limited to auxiliary state, KVFS and FPFS assume specific workloads (FPFS cannot rename efficiently), and data integrity is left to file permissions and to designs built on ArckFS.
• Evaluation gaps. Crash consistency was tested with unit tests, not with a framework such as Chipmunk. Strata runs on one thread, ZoFS and KucoFS were unavailable, Aerie and ctFS would not run, Webproxy and Varmail stop at 16 threads because of a Filebench bug, and LevelDB runs only where mmap() works.

The following points are a reading of the evidence, not claims of the paper.

• The LevelDB figures disagree. The abstract says 3.1× to 17×, and the body says up to 3.1× over WineFS and 1.5× to 17× over ext4. Table 5 gives 1.01× (Read random) to about 2.0× (Delete random, 494 against 245) over WineFS, while 3.1× is the ratio over ext4 on Fill 100K and 17× is ext4's Fill sync. The abstract's 1.3× for both customized file systems also differs from the body's 1.3× and 1.2×. I use the body and table values here.
• The largest gaps measure VFS more than Trio. The 334× and 850× figures are against kernel file systems that the authors say VFS holds back. The datapath comes from OdinFS, an earlier paper by two of the same authors, so the fairer measure of direct access is the gain over OdinFS, up to 1.3× at 224 threads.
• The mediation comparison is indirect. No experiment adds mediation to ArckFS itself. The mediating designs that ran are SplitFS and Strata (one thread), and the 44% and 68% overheads of mediation come from the ZoFS paper. The only decomposition is the VFS and digestion share in create.
• Sharing cost rests on narrow cases. There are two file sizes and two directory sizes, and the create test forces an unmap after each operation. A large file written by several mutually untrusting applications is where the numbers are worst and trust groups do not apply, and no application workload exercises it.
• Heterogeneous sharing is argued, not measured. I found no experiment where KVFS or FPFS shares a file with ArckFS, and the customized file systems run alone on workloads built to suit them, for 1.2× to 1.3× over ArckFS.
• Security evidence is author-designed. The 134 scenarios follow the authors' own threat list and invariants, with no independent fuzzing or argument that four invariants suffice. The text also does not say how delegation threads, which touch NVM for applications, check that a request stays inside pages mapped to that application.
• Hardware and noise. Delegation and its 32KB and 256-byte thresholds are tuned for Optane, and the paper itself cites a report of Intel ending it (reference 12). CXL devices are motivation, not measured. I found no repetition counts or error bars, and Table 5 gives single values.

• Cheaper sharing among untrusted writers, since the numbers put the cost in mapping, unmapping and verification that grow with file size. Verifying only changed index pages would be one route, a gap I see, not a stated plan.
• Crash-consistency testing with frameworks such as Chipmunk, including adversarial recovery programs supplied by a LibFS.
• Running on CXL-based NVM, which the paper names as a target, and retuning the Optane-specific delegation.
• Protecting file data itself, which the authors treat as orthogonal, and a safe way to change core state without privileges.