Master Server Requirements
The Master Server keeps every filename, attribute, and directory entry in RAM — 150 million files alone need ~42 GiB just for metadata. This post covers the exact memory formula, why a single fast core beats many slow ones, the disk space equation for changelogs, and how Followers differ from Metaloggers.
Leader Master Server
The Leader Master Server is the central point of a cluster. It is responsible for storing metadata of the file system (file names, attributes, folder structure etc.) in the in-memory database. Any change in metadata is immediately logged to its local disk (changelog). There is one Master Server but there may be many so-called Metaloggers. The Metalogger server just logs all metadata changes to its local disk and may be turned into Master Server manually.
Metalogger
As the Metalogger simply gathers asynchronous online metadata backups from the Leader Master Server, the hardware requirements are not higher than that for the Master Server itself as it needs about the same disk space. The Metalogger should have at least the same amount of disk space (especially free space in /var/lib/mfs) as the main Master Server. If one would like to use the Metalogger as the Master Server in case of the main Master’s failure, the Metalogger machine should have at least the same amount of RAM as the main Master Server and a CPU operating at the same (or at least similar) frequency.
Follower Master Servers – High Availability
For the High Availability configuration, there is one Master Server but there may be many so-called Follower Master Servers (or Followers). A Follower is an asynchronous, online “backup” Master Server that follows all metadata changes, keeps its own copy of the in-memory database, and may be elected as a new Leader Master Server by a majority of Chunkservers in case the Leader Master Server goes down. Hardware requirements for Leader Master Servers and Follower Master Servers are the same.
The Master Server may save the entire metadata database to local disk every hour (non-HA edition) or once a day (HA edition). Followers save their database every hour.
It is always recommended to use either Follower Master Servers (see HA configuration) or Metaloggers to back up the Leader Master Server’s metadata. Losing a file system’s metadata information usually leads to losing data in a cluster. Although possible, it is extremely difficult to recover files from chunks without metadata information. One can use both Follower Master Servers and Metaloggers, but it is usually unreasonable as a Follower Master Server covers Metalogger tasks and much more.
Memory
There should be sufficient memory in the Leader Master Server (and Master Server Followers and Metaloggers) as it needs to store all metadata in the in-memory database. Each file or folder (inode) metadata takes about 300–350 bytes. The size of the metadata database neither depends on the cluster capacity nor on file sizes — rather it is proportional to the number of files. Therefore, for best results, one should estimate the maximum number of files and folders in a cluster and adjust the Master Server memory accordingly.
For example, 150 million files/folders require approx. 42 GiB of RAM for storing metadata, plus Master Server operating system overhead.
CPU
Since the Master Server is a single-threaded process, we recommend using modern processors with high clock speed and a low number of cores, e.g.:
- Intel® Xeon® Gold 5122, 3.70 GHz
- Intel® Xeon® Platinum 8156, 3.70 GHz
A good starting point for selecting a CPU for the Master Server is the single-thread performance rating published by CPU Benchmark.
It’s recommended to disable the hyper-threading CPU feature for Master Servers. Additionally, disabling CPU power management in BIOS (or enabling a mode like “maximum performance”) may have a positive impact on efficiency.
Disks
The Master Server logs each operation to its local disk and occasionally dumps the whole metadata database to its local disk. It is recommended to have redundant local storage (e.g. RAID 1 or RAID 1+0) in the Leader and Follower Master Servers.
The size of incremental logs depends on the number of operations per hour. The length (in hours) of this incremental log is configurable. For example, 20 GiB of space should be enough for storing information for 25 million files and for changelogs to be kept for up to 50 hours. You can calculate the minimum amount of recommended disk space using the following formula:
DISK_SPACE = RAM * (BACK_META_KEEP_PREVIOUS + 2) + (BACK_LOGS + 1) [GiB]where:
RAM– amount of RAM used by the Master Server process [GiB]BACK_LOGS– number of metadata changelog files, default is 50 (from/etc/mfs/mfsmaster.cfg)BACK_META_KEEP_PREVIOUS– number of older metadata files to be kept (default is 1) (also from/etc/mfs/mfsmaster.cfg)
If default values from /etc/mfs/mfsmaster.cfg are used, the formula becomes RAM * 3 + 51. For 128 GiB of RAM used by the Master Server process, one should reserve for /var/lib/mfs: 128 × 3 + 51 = 435 GiB minimum disk space.
For configurations where Master Servers share the same machine with Chunkservers, it is important to ensure the Master Server has dedicated access to its local disk. Otherwise, heavy Chunkserver I/O operations may slow down Master Server disk operations, further causing Master Server slowdowns and ultimately slowing down the entire storage cluster.
Networking
The Master Server neither sends nor receives actual file data. It just deals with metadata, which is usually much smaller in amount than actual file data. As it is at the central point of the cluster, each client and each Chunkserver exchanges data with the same Master Server. Even for clusters with HA configurations, there is always just one Leader Master Server. This leads to a significant number of small network I/O operations. Note that neither Chunkservers nor clients connect to Followers and Metaloggers.
The Master Server may use two separate network interfaces: the first one for serving clients, and the second one for communicating with other servers (which is not LACP).
Download pdf
Now you know the Master Server requirements. If you want to download the PDF document with even more information, download MooseFS Hardware Guide.