MooseFS Block Device: A File on the Cluster, a Disk on the Host
There are a few ways for applications to access MooseFS. They can use it as a POSIX file system through a mount, connect through library calls without mounting it, or access it as a block device.
The block device option has been available since July 2018. It is useful when an application needs something that behaves like a Linux disk rather than a directory full of files, while the data itself remains stored on the MooseFS cluster.
This post looks at MooseFS Block Device, the mfsbdev tool used to manage it, and how the feature has evolved since its release.
Why use a block device instead of a mount?
A mounted MooseFS file system gives applications the familiar structure of files and directories, along with standard POSIX operations. But not every application works that way.
For example, a hypervisor may need a disk for a virtual machine. It expects a block device that behaves like a physical disk or partition, rather than a file system to browse. Some database engines also work directly with blocks of data, reading and writing at specific offsets without relying on file system operations.
MooseFS Block Device makes this possible. Applications get a block device to work with, while the data stays in a file on the MooseFS cluster and benefits from the same replication and management as other files stored there.
There is another practical advantage. A mounted MooseFS file system uses FUSE, and each FUSE connection has a limit on the traffic it can handle. mfsbdev uses libmfsio, the client library that allows programs to communicate with MooseFS without going through FUSE. As a result, block device access avoids the per-mount FUSE limit.
How mfsbdev works
mfsbdev runs on Linux only and requires the kernel’s nbd (Network Block Device) module to be loaded.
It runs as a daemon. Start it with mfsbdev start, and it connects to a Master Server and listens on a Unix socket. By default, that socket is /dev/mfs/nbdsock. Other mfsbdev commands communicate with the daemon through this socket instead of connecting to the Master Server directly.
To turn a MooseFS file into a block device, use mfsbdev map and specify the file’s path. The command creates a /dev/nbdX device and a symbolic link to it under /dev/mfs. By default, the link uses the master host, port, and file name, so you do not have to rely on the device number.
Once mapped, the device can be used like any other block device. You can create a file system on it or pass it directly to a hypervisor. The other commands cover the usual management tasks:
mfsbdev listshows the devices currently mapped by the daemon.mfsbdev unmapremoves a mapping when it is no longer needed.mfsbdev resizechanges a mapped device’s size without requiring you to unmap it first. It needs Linux kernel 4.18 or later.
Mappings can be read-write or read-only. MooseFS uses a cluster-wide file lock held by the Master Server to prevent conflicting access. A read-write mapping gets an exclusive lock, while read-only mappings use shared locks. This means the same file can be mapped read-only from multiple devices, but a second read-write mapping of that file will be rejected, whether it is on the same host or another one. That helps prevent two independent writers from corrupting the data.
There is an exception for setups that handle synchronization themselves. The mfsbdev map -i option tells the tool to proceed even if it cannot obtain the lock. It still requests the lock, but ignores the result. Use this only when the software accessing the device already coordinates writes safely.
MooseFS Block Device is available in both Community Edition and PRO. It is part of the core MooseFS feature set, alongside the POSIX file system and other storage functionality.
How the feature has changed since 2018
MooseFS Block Device first appeared in version 4.9.0, released on July 30, 2018. The release notes described it as a new Linux module for creating a block device from a file stored in MooseFS, complete with its own management tools. Version 4.10.0 followed two months later with changes to how the daemon manages block devices.
Read-only mappings arrived in version 4.21.8 in May 2020. Before that release, all mappings were read-write. The update also introduced cluster-wide locking, allowing multiple read-only mappings while keeping read-write access exclusive.
Version 4.26.1, released in October 2020, added support for loading initial mappings from a configuration file. When the daemon is started with that file (mfsbdev start -i), it recreates the listed mappings, so you do not have to run every mfsbdev map command again.
The ignore-locks option came later in version 4.40.1, released in July 2022. It gives applications that already coordinate writes themselves a way to manage locking outside MooseFS.
More recent updates have focused on reliability. Version 4.51.0, released in October 2023, fixed an issue with the unmap protocol. Version 5.1.0 includes a fix for a use-after-free issue and another fix related to the unmap protocol.
Further reading
For practical instructions on starting the daemon and mapping, resizing, or unmapping devices, see Block device operations.
To learn how MooseFS Block Device fits alongside the other ways to access MooseFS, see Clients.
For a breakdown of the features available in each edition, see Editions: Community and PRO.