What are Master Server requirements
In this article, you will learn more about the Master Server requirements. What hardware we recommend for Master host, the network, and what is the difference between Leader Master Server, Followers, and Metaloggers.
Leader Master Server
The Leader Master Server is the central point of a cluster. It is responsible for storing metadata of the file system (file names, attributes, folders structure etc.) in the in-memory database. Any change in metadata is immediately logged to its local disk (changelog). There is one Master Server but there may be many so-called Metaloggers. The Metalogger server just logs all metadata changes to its local disk and may be turned into Master Server manually.
Metalogger
As the Metalogger simply gathers asynchronous online metadata backups from Leader Master Server – the hardware requirements are not higher than that for the Master Server itself as it needs about the same disk space. The Metalogger should have at least the same amount of disk space (especially the free space in /var/lib/mfs) as the main Master Server. If one would like to use the Metalogger as the Master Server in case of the main Master’s failure, the Metalogger machine should have at least the same amount of RAM as the main Master Server and CPU operating at the same (or at least similar) frequency.
Follower Master Servers – High availability
Whereas for the High Availability configuration, there is one Master Server but there may be many so-called Follower Master Servers (or Followers). A follower is an asynchronous, online “backup”. Master Server that follows all metadata changes keeps its own copy of the in-memory database and may be elected as a new Leader Master Server by a majority of Chunkservers in case Leader Master Server is down. Hardware requirements for Leader Master Servers and Follower Master Servers are the same.
The Master Server may save the entire metadata database to local disk every hour (no HA edition) or once a day (HA edition). Follower saves its database every hour.
It is always recommended to use either Follower Master Servers (see HA configuration) or Metaloggers to backup Leader Master Server’s metadata. Losing a file system’s metadata information usually leads to losing data in a cluster. Although possible, it is extremely difficult to recover files from chunks without metadata information. So, here one can use both Follower Master Servers and Metaloggers but usually it is unreasonable as Follower Master Server covers Metalogger tasks and much more.
Memory
There should be sufficient memory in the Leader Master Server (Master Server Followers, Metaloggers) as it needs to store all metadata in the in-memory database. Each file or folder (inode) metadata takes about 300-350 bytes. The size of the metadata database neither depends on the cluster capacity nor on files sizes, rather it is proportional to the number of files. Therefore, for best results, one should estimate the maximum number of files and folders in a cluster and adjust the Master Server memory accordingly.
For example, 150 million files/folders require approx. 42 GiB of RAM for storing metadata plus Master Server operating system overhead.
CPU
Since Master Server is a single-threaded process, we would recommend using modern processors with high clock and a low number of cores, e.g.:
- Intel® Xeon® Gold 5122, 3.70 GHz,
- Intel® Xeon® Platinum 8156, 3.70 GHz.
A good starting point to help in the selection of CPU for Master Server would be the single-thread performance rating published by CPU Benchmark
It’s recommended to disable the hyper-threading CPU feature for Master Servers. Additionally, disabling CPU power management in BIOS (or enable mode like “maximum performance”) may have a positive impact on efficiency.
Disks
Master Server logs each operation to its local disk. Occasionally it also dumps the whole metadata database to its local disk. It is recommended to have redundant local storage (e.g. RAID 1 or RAID 1+0) in the Leader and Follower Master Servers.
The size of incremental logs depends on the number of operations per hour. Length (in hours) of this incremental log is configurable. For example, the space of 20 GiB should be enough for storing information for 25 million files and for changelogs to be kept for up to 50 hours. One can calculate the minimum amount of recommended disk space using the following formula:
DISK_SPACE = RAM *(BACK_META_KEEP_PREVIOUS+2) + (BACK_LOGS+1)[GiB]
where:
- RAM – an amount of RAM used by Master Server process [GiB]
- BACK_LOGS – number of metadata changelog files, default is 50 (from /etc/mfs/mfsmaster.cfg)
- BACK_META_KEEP_PREVIOUS – number of older metadata files to be kept (default is 1) (also from /etc/mfs/mfsmaster.cfg).
If default values from /etc/mfs/mfsmaster.cfg are used, it is RAM*3+51. For 128 GiB of RAM used by Master Server process, one should reserve for /var/lib/mfs: 128*3 + 51 = 435 GiB minimum disk space.
For configurations where Master Servers share the same machine with Chunkservers, it is important to ensure the Master Server has dedicated access to its local disk. Otherwise, it is possible that Chunkserver’s heavy I/O disk operations slow down Master Server disk operations further causing Master Server slowdowns and finally leading to the slowdown of the entire storage cluster.
Networking
Master Server neither sends nor receives actual file data. It just deals with metadata which is usually much smaller in amount than actual file data. As it is at the central point of the cluster, each client and each Chunkserver exchanges data with the same Master Server. Even for clusters with HA configurations, there is always just one Leader Master Server. This leads to a significant number of small network I/O operations. Please note, neither Chunkservers nor clients connect to Followers and Metaloggers.
Master Server may use two separate network interfaces the first one for serving clients. The second one for communicating with other servers (which is not LACP).
Download pdf
Now you know Master Server requirements. If you want to download the pdf document with even more info, download MooseFS Hardware Guide.