docker/transfused repository overview

⁠osxfs

osxfs is a Linux file system designed to solve Docker's -v "bind mount problem" between Docker for Mac and its encapsulated Linux VM. osxfs provides a FUSE file system that supports inotify events in containers and side-storage of file node ownership metadata. These features enable container developer use cases by supporting Linux containers that use inotify to listen for file system events and allowing the Docker for Mac client to run as a regular user on macOS.

⁠System Overview

The osxfs system has two major components:

1. transfused is a Linux FUSE protocol proxy to virtio vsocks
2. com.docker.osxfs is a macOS program running a Linux FUSE server

Starting from the user application, typical file system messages flow as follows:

1. Linux VFS receives a file system operation via a system call
2. The FUSE driver gets called from the VFS demuxer
3. FUSE sends a FUSE message over a /dev/fuse fd previously negotiated
4. transfused reads the FUSE message in a proxy thread and retransmits it over an AF_VSOCK socket (virtio vsock)
5. virtio vsock puts the message for transmission on a ring buffer
6. virtio vsock triggers a hardware fault to notify the hypervisor
7. HyperKit receives the hardware fault and dequeues the message
8. HyperKit forwards the message to a UNIX domain socket
9. osxfs reads the message from the UNIX domain socket and parses it
10. osxfs performs macOS file system operations to perform the requested file system operation
11. osxfs constructs a packet with the error or result of the request and transmits it over the UNIX domain socket
12. HyperKit reads the response from the UNIX domain socket
13. HyperKit puts the response on a ring buffer
14. HyperKit triggers an interrupt in Linux
15. virtio vsock receives the interrupt and dequeues the response
16. virtio vsock sends the response to transfused over the AF_VSOCK socket
17. transfused reads the response in a different thread, buffering until a complete message is read, and then writes it to the /dev/fuse fd
18. FUSE receives the response and queues it for the blocked userspace thread
19. The userspace thread in FUSE unblocks, dequeues, and decodes the response
20. The response modifies VFS state if necessary and is returned via an unblocked syscall to the calling process

This process takes approximately 100 microseconds.

osxfs also contains a server which listens for new container mount and unmount requests in order to enable/disable FSEvents listeners for triggering inotify events. Finally, there is another server bundled in osxfs which listens for control messages from the Docker for Mac GUI and osxfs debugging tools.

The mount namespace of osxfs is controlled from the Docker for Mac GUI and persisted in a git-based database. osxfs's control socket API offers a means to check that remote mount points are suitable for mounting (e.g. not in use by the system).

⁠Performance

Performance of the osxfs system is determined by the number, parallelism, and latency of file system request-response pairs. Performance can be improved by either reducing the number of file system requests that are made, reducing the number that must be sent through FUSE, reducing the number that must be sent to the host OS, or reducing the latency of the end-to-end data path.

CPU consumption of HyperKit (and Linux inside of it) and osxfs is also an important metric to monitor. We would like to reduce the CPU use of these components significantly as the current levels are both unnecessary and harmful to users' battery life.

⁠Data Path Latency

Every stage in the above message flow increases roundtrip data path latency. We believe that the following issues in particular are major latency contributors:

1. Linux kernel thread/core scheduling between user process, transfused, virtio worker threads, and FUSE worker threads seems to have a significant CPU overhead and be correlated with latency. Both non-tickless kernels and kernels running on a single vCPU appear to perform better in general than tickless multi-core kernels (the default).

2. Linux/HyperKit interaction both due to core scheduling and virtio vsock implementation seems to have significant CPU overhead and be correlated with latency. Tickless kernels often sleep and program vLAPIC interrupts which cause hardware faults. The vsock protocol uses interrupts to wake up blocked threads on the other side that may otherwise stay blocked indefinitely waiting on a free buffer space condition. In many cases these interrupts are unnecessary and the vsock spec appears to include a mechanism that may allow their elision which is not implemented in the HyperKit vsock implementation and may or may not be implemented by Linux (or partially implemented by Linux). vsock also uses "credit updates" to keep the each side informed of the other side's available backing buffer. These credit updates are an additional message load that can be removed with deferred credit updates (see docker/hyperkit#92⁠.

3. HyperKit's internal buffering and use of UNIX domain sockets (instead of in-process, shared memory, or Mach ports with Out-of-Line page swapping) may contribute to latency based on benchmarks from Chromium comparing IPC techniques.

4. osxfs's use of LWT for cooperative multi-threading causes frequent invocation of system calls for asynchronous I/O which appears to double its CPU use and increase latency by several tens of microseconds.

For latency analysis, the perfstat tool is available as a subcommand of osxfs. A rudimentary osxfs to transfused ping functionality is also available via SIGUSR2.

⁠Caching

Reducing the number of messages required to be sent and the length of their roundtrip data paths is also a great strategy for improving performance. The Linux VFS has several caches which file systems can use to get platform-consistent and integrated caching. In the case of osxfs, caching implies a trade-off between macOS host and container as osxfs is not the authority on file system state (macOS is). To this end, consistency flags have recently been added to the Docker engine which allow users to specify one of three consistency modes on a per-container-mount basis. These flags enable users to specify the level of consistency needed for specific mounts into their containerized applications which osxfs can then use as permission to enable various Linux caches or, one day, an osxfs-specific Linux-side cache.

The caches in use are the dcache for directory entries and the inode attribute cache for stat metadata. We should be able to enable the writeback cache on a per-node basis but Linux FUSE does not yet offer that capability instead only providing a mount-global writeback cache setting negotiated at mount-time. osxfs is designed so that subsets of the transfuse mounts are bind-mounted into containers which improves cache sharing and removes a huge class of failure modes due to dynamic mounting and unmounting.

Finally, construction of a Linux-side "cache file system" for early interception of structural file system changes is planned. This component would take advantage of the carefully designed semantics of the cached and delegated consistency modes now available on container bind mounts which allow even directory structure changes to be cached during reading and modifying. This is a markedly more aggressive caching strategy than the Linux VFS offers as, in traditional use cases, persisting directory structure metadata was considered crucial.

⁠Measurement

osxfs has some built-in tools to help with performance measurement. For the most detail with the lowest cost, perfstat is the best tool to use.

⁠perfstat

perfstat is available as a subcommand of the osxfs binary. perfstat has two halves, the perfstat subcommand and the perfstat_analyze subcommand. perfstat records the monotonic time latencies and message types into and out of osxfs and transfused. To measure 100,000 1 kibibyte block writes to a file in your pinata directory with perfstat, execute a command like:

v1/cmd/com.docker.osxfs/_build/src/osxfs_ctl.native perfstat --export /Users perfstat-dd.dat -- docker run --rm -v $(pwd):/host -w /host alpine dd if=/dev/zero of=zeros bs=1024 count=100000

To view a summary of the recorded data, execute a command like:

v1/cmd/com.docker.osxfs/_build/src/osxfs_ctl.native perfstat_analyze perfstat-dd.dat

which will produce a report like:

Elapsed: 19.796321007s
Messages: 100028 (0 trimmed, 2 with open transfused data)
HyperKit CPU: 129.7% (25.67s)
osxfs CPU: 52.7% (10.44s)
Time in osxfs with 0 ops outstanding: 72.4% (14.327s)
Time in osxfs with 1 ops outstanding: 27.6% (5.47s)
Time under transfused with 0 ops outstanding: 29.5% (5.845s)
Time under transfused with 1 ops outstanding: 70.5% (13.951s)
Time under transfused with 2 ops outstanding: 0.0% (7.192us)
Time under transfused with 3 ops outstanding: 0.0% (0us)
Message FUSE_WRITE received 100000 (100.0% txns 99.9% t) (50th/90th/98th/100th):
52.569us 65.289us 102us 6.972ms
Message FUSE_LOOKUP received 17 (0.0% txns 0.0% t) (50th/90th/98th/100th):
69.834us 106us 118us 118us
Message FUSE_READLINK received 3 (0.0% txns 0.0% t) (50th/90th/98th/100th):
51.884us 54.915us 54.915us 54.915us
Message FUSE_GETATTR received 2 (0.0% txns 0.0% t) (50th/90th/98th/100th):
58.909us 58.909us 58.909us 58.909us
Message FUSE_FLUSH received 2 (0.0% txns 0.0% t) (50th/90th/98th/100th):
27.095us 27.095us 27.095us 27.095us
Message FUSE_RELEASE received 1 (0.0% txns 0.1% t) (50th/90th/98th/100th):
4.609ms 4.609ms 4.609ms 4.609ms
Message FUSE_GETXATTR received 1 (0.0% txns 0.0% t) (50th/90th/98th/100th):
27.449us 27.449us 27.449us 27.449us
Message FUSE_ACCESS received 1 (0.0% txns 0.0% t) (50th/90th/98th/100th):
68.991us 68.991us 68.991us 68.991us
Message FUSE_CREATE received 1 (0.0% txns 0.0% t) (50th/90th/98th/100th):
303us 303us 303us 303us

The dd 1KiB-block write workload is not very representative of real world use cases but seems to be a favorite among users who want to "test" the file system.

For more realistic use cases, we often look at the performance of ember build⁠, the performance of go list ./... in a bind-mounted golang checkout (FROM golang:1.7), or the performance of un-tar-ing the Linux kernel into an empty bind-mounted directory. We've started compiling a benchmark suite but that work has stalled.