sfsmaster.cfg(5)
================

== NAME

sfsmaster.cfg - main configuration file for sfsmaster

== DESCRIPTION

The file *sfsmaster.cfg* contains configuration of SaunaFS metadata server process.

== SYNTAX

Syntax is:

'OPTION' *=* 'VALUE'

Lines starting with *#* character are ignored.

== OPTIONS

Configuration options:

*PERSONALITY*::
Current 'personality' of this instance of metadata server. Valid values are
'master', 'shadow' and 'ha-cluster-managed'. If installation is managed by an
HA cluster the only valid value is 'ha-cluster-managed', otherwise the only
valid values are 'master' and 'shadow', in which case only one metadata server
in SaunaFS shall have 'master' personality. +
'master' means that this instance of metadata server acts as main metadata
server govering all file system metadata modifications. +
'shadow' means that this instance of metadata server acts as backup metadata
server ready for immediate deployment as new 'master' in case of current
'master' failure. +
Metadata server personality can be changed at any moment as
long as one changes personality from 'shadow' to 'master', changing personality
the other way around is forbidden. +
'ha-cluster-managed' means that this instance is managed
by HA cluster, server runs in 'shadow' mode as long as its not remotely
promoted to 'master'.

*CLUSTER_ID*:: Uniquely identifies this namespace.
Prevents accidental connections to wrong metadata or chunkservers. (default is 'default')

*DATA_PATH*:: where to store metadata files and lock file

*WORKING_USER*:: user to run daemon as

*WORKING_GROUP*:: group to run daemon as (optional - if empty then default user
group will be used)

*SYSLOG_IDENT*:: name of process to place in syslog messages (default is
sfsmaster)

*LOCK_MEMORY*:: whether to perform mlockall() to avoid swapping out sfsmaster
process (default is 0, i.e. no)

*LIMIT_GLIBC_MALLOC_ARENAS*:: Linux only: limit glibc malloc arenas to given
value - prevents from using huge amount of virtual memory. This can influence
performance by reducing memory fragmentation and improving cache locality, but
it may also lead to contention and reduced parallelism in multi-threaded
applications. Use it in constrained memory environments, recommended values are
4 or 8. (default is 0: disabled or let glibc decide)

*NICE_LEVEL*:: nice level to run daemon with (default is -19 if possible; note:
process must be started as root to increase priority)

*EXPORTS_FILENAME*:: alternative name of *sfsexports.cfg* file

*TOPOLOGY_FILENAME*:: alternative name of *sfstopology.cfg* file

*CUSTOM_GOALS_FILENAME*:: alternative name of *sfsgoals.cfg* file

*PREFER_LOCAL_CHUNKSERVER*:: If a client mountpoint has a local chunkserver,
and a given chunk happens to reside locally, then sfsmaster will list the local
chunkserver first. However, when the local client mount is issuing many
read(s)/write(s), to many local chunks, these requests can overload the local
chunkserver and disk subsystem.  Setting this to 0(the default is 1) means that
remote chunkservers will be considered as equivalent to the local chunkserver. +
This is useful when the network is faster than the disk, and when there is
high-IO load on the client mountpoints.

*BACK_LOGS*:: number of metadata change log files (default is 50)

*BACK_META_KEEP_PREVIOUS*:: number of previous metadata files to be kept
(default is 1)

*AUTO_RECOVERY*:: when this option is set (equals 1) master will try to recover
metadata from changelog when it is being started after a crash; otherwise it
will refuse to start and 'sfsmetarestore' should be used to recover the
metadata (default is 0)

*OPERATIONS_DELAY_INIT*:: initial delay in seconds before starting chunk
operations (default is 300)

*OPERATIONS_DELAY_DISCONNECT*:: chunk operations delay in seconds after
chunkserver disconnection (default is 3600)

*MATOML_LISTEN_HOST*:: IP address to listen on for metalogger connections (***
means any)

*MATOML_LISTEN_PORT*:: port to listen on for metalogger connections (default is
9419)

*MATOML_LOG_PRESERVE_SECONDS*:: how many seconds of change logs have to be
preserved in memory (default is 600; note: logs are stored in blocks of 5k
lines, so sometimes real number of seconds may be little bigger; zero disables
extra logs storage)

*MATOCS_LISTEN_HOST*:: IP address to listen on for chunkserver connections (***
means any)

*MATOCS_LISTEN_PORT*:: port to listen on for chunkserver connections (default
is 9420)

*MATOCL_LISTEN_HOST*:: IP address to listen on for client (mount) connections
(*** means any)

*MATOCL_LISTEN_PORT*:: port to listen on for client (mount) connections
(default is 9421)

*MATOTS_LISTEN_HOST*:: IP address to listen on for tapeserver connections (***
means any)

*MATOTS_LISTEN_PORT*:: Port to listen on for tapeserver connections (default is
9424)

*CHUNKS_LOOP_MAX_CPS*:: Chunks loop shouldn't check more chunks per seconds
than given number (default is 100000)

*CHUNKS_LOOP_MIN_TIME*:: Chunks loop will check all chunks in specified time
(default is 300) unless *CHUNKS_LOOP_MAX_CPS* will force slower execution.

*CHUNKS_LOOP_PERIOD*:: Time in milliseconds between chunks loop execution
(default is 1000).

*CHUNKS_LOOP_MAX_CPU*:: Hard limit on CPU usage by chunks loop (percentage
value, default is 60).

*CHUNKS_SOFT_DEL_LIMIT*:: Soft maximum number of chunks to delete on one
chunkserver (default is 10)

*CHUNKS_HARD_DEL_LIMIT*:: Hard maximum number of chunks to delete on one
chunkserver (default is 25)

*CHUNKS_WRITE_REP_LIMIT*:: Maximum number of chunks to replicate to one
chunkserver (default is 2)

*CHUNKS_READ_REP_LIMIT*:: Maximum number of chunks to replicate from one
chunkserver (default is 10)

*ENDANGERED_CHUNKS_PRIORITY*:: Percentage of endangered chunks that should be
replicated with high priority. Example: when set to 0.2, up to 20% of chunks
served in one turn would be extracted from endangered priority queue. When set
to 1 (max), no other chunks would be processed as long as there are any
endangered chunks in the queue (not advised) (default is 0, i.e. there is no
overhead for prioritizing endangered chunks).

*ENDANGERED_CHUNKS_MAX_CAPACITY*:: Max capacity of endangered chunks queue.
This value can limit memory usage of master server if there are lots of
endangered chunks in the system. This value is ignored if
ENDANGERED_CHUNKS_PRIORITY is set to 0. (default is 1Mi, i.e. no more than 1Mi
chunks will be kept in a queue).

*ACCEPTABLE_DIFFERENCE*:: A maximum difference between disk usage on
chunkservers that doesn't trigger chunk rebalancing (default is 0.1, i.e. 10%).

*CHUNKS_REBALANCING_BETWEEN_LABELS*:: When balancing disk usage, allow moving
chunks between servers with different labels (default is 0, i.e. chunks will be
moved only between servers with the same label).

*GLOBALIOLIMITS_FILENAME*:: Configuration of global I/O limits (default is no
I/O limiting)

*GLOBALIOLIMITS_RENEGOTIATION_PERIOD_SECONDS*:: How often mountpoints will
request bandwidth allocations under constant, predictable load (default is 0.1)

*GLOBALIOLIMITS_ACCUMULATE_MS*:: After inactivity, no waiting is required to
transfer the amount of data equivalent to normal data flow over the period of
that many milliseconds (default is 250)

*METADATA_CHECKSUM_INTERVAL*:: how often metadata checksum shall be sent to
backup servers (default is: every 50 metadata updates)

*METADATA_CHECKSUM_RECALCULATION_SPEED*:: how fast should metadata be
recalculated in background (default : 100 objects per function call)

*DISABLE_METADATA_CHECKSUM_VERIFICATION*:: should checksum verification be
disabled while applying changelog

*NO_ATIME*:: when this option is set to 1 inode access time is not updated on
every access, otherwise (when set to 0) it is updated (default is 0)

*METADATA_SAVE_REQUEST_MIN_PERIOD*:: minimal time in seconds between metadata
dumps caused by requests from shadow masters (default is 1800)

*SESSION_SUSTAIN_TIME*:: Time in seconds for which client session data (e.g.
list of open files) should be sustained in the master server after connection
with the client was lost. Values between 60 and 604800 (one week) are accepted.
(default is 86400)

*USE_BDB_FOR_NAME_STORAGE*:: When this option is set to 1 Berkeley DB is used
for storing file/directory names in file (DATA_PATH/name_storage.db). By
default all strings are kept in system memory. (default is 0)

*BDB_NAME_STORAGE_CACHE_SIZE*:: Size of memory cache (in MB) for file/directory
names used by Berkeley DB storage. (default is 10)

*AVOID_SAME_IP_CHUNKSERVERS*:: When this option is set to 1, process of
selecting chunkservers for chunks will try to avoid using those that share the
same ip. (default is 0)

*REDUNDANCY_LEVEL*:: minimum number of required redundant chunk parts that can
be lost before chunk becomes endangered (default is 0)

*SNAPSHOT_INITIAL_BATCH_SIZE*:: This option can be used to specify initial
number of snapshotted nodes that will be atomically cloned before enqueuing the
task for execution in fixed-sized batches. (default is 1000)

*SNAPSHOT_INITIAL_BATCH_SIZE_LIMIT*:: This option specifies the maximum initial
batch size set for snapshot request. (default is 10000)

*FILE_TEST_LOOP_MIN_TIME* Test files loop will try to check all files in
specified time in seconds (default is 3600). It's possible for the loop to take
more time if the master server is busy or the machine doesn't have enough
processing power to make all the needed calculations.

Options below are mandatory for all Shadow instances:

*MASTER_HOST*:: address of the host running SaunaFS metadata server that
currently acts as 'master'

*MASTER_PORT*:: port number where SaunaFS metadata server currently running as
'master' listens for connections from 'shadow's and metaloggers (default is
9420)

*MASTER_RECONNECTION_DELAY*:: delay in seconds before trying to reconnect to
metadata server after disconnection (default is 1)

*MASTER_TIMEOUT*:: timeout (in seconds) for metadata server connections
(default is 60)

*LOAD_FACTOR_PENALTY*:: When set, percentage of load will be added to
chunkserver disk usage to determine most fitting chunkserver. Heavy loaded
chunkservers will be picked for operations less frequently. (default is 0,
correct values are in range from 0 to 0.5)

*PRIORITIZE_DATA_PARTS*:: When set, master server will prioritize data parts in
EC goals to land in the chunkservers with higher percentage of available space.
Could cause parities landing always in the same chunkservers if the cluster is
not well balanced. (default: 1)

*POLL_TIMEOUT_MS*:: Maximum amount of time in milliseconds that the polling
operation will wait for events. The value is applied for the polling in the
events loop. Smaller values could reduce latency at the cost of CPU usage
(default: 50)

*ENABLE_PROMETHEUS (EXPERIMENTAL)*:: Whether to enable Prometheus support and
metric collection. Note that this requires compiling with Prometheus support.
Set to either 1 to enable, or 0 to disable (default is 0)

*PROMETHEUS_HOST (EXPERIMENTAL)*:: Host address where Prometheus metric data
can be collected, must be in the format of HOST:PORT (default 0.0.0.0:9499)

*CREATE_EMPTY_FOLDERS_WHEN_SPACE_DEPLETED (EXPERIMENTAL)* When enabled,
this option allows the system to create metadata for empty folders when
storage space is depleted. This helps preserving the intended file and
directory structure. When this option is disabled, no metadata for folders
will be created if there is insufficient storage space (default: 1).

*LOG_LEVEL*:: Setup logging. Uses the environment variable SAUNAFS_LOG_LEVEL or
config value LOG_LEVEL to determine logging level. Valid log levels are
- 'trace'
- 'debug'
- 'info'
- 'warn' or 'warning'
- 'err' or 'error'
- 'crit' or 'critical'
- 'off'

*TLS_CERT_FILE (EXPERIMENTAL)*::
Path to the TLS certificate file the master/shadow server will use 
for TLS connections (there is no default value).

*TLS_KEY_FILE (EXPERIMENTAL)*::
Path to the TLS private key file the master/shadow server will use
for TLS connections (there is no default value).

*TLS_CA_CERT_FILE (EXPERIMENTAL)*::
Path to the trusted CA certificate which is used to authenticate
the TLS connection (there is no default value).

*USE_CHUNKSERVER_SIDE_CHUNK_LOCK (EXPERIMENTAL)*:: When set to 1, enables sending
chunk part lock messages to the chunkservers. This can be useful to track down which
chunk parts are currently being written. Reloadable (default: 0).

*EMPTY_RESERVED_FILES_PERIOD_MSECONDS (EXPERIMENTAL)*::
Interval for periodic cleaning of reserved files, in milliseconds.
If set to 0, the reserved files deletion is disabled. (Default: 0)

[WARNING]
====
Administrator-only option. Enabling periodic deletion may remove master-side
references to reserved files that are still held by other applications or
client instances that have not released them yet. This can disrupt ongoing
operations and lead to unexpected errors.

Recommended approach: Do not enable automatic deletion unless you fully
understand the impact on running workloads. Prefer manually freeing reserved
files by locating each reference or process using them and stopping it before
releasing the reservation.
====

== NOTES

Chunks in master are tested in loop. Speed (or frequency) is regulated by two
options *CHUNKS_LOOP_MIN_TIME* and *CHUNKS_LOOP_MAX_CPS*. First defines minimal
time of the loop and second maximal number of chunk tests per second. Typically
at the beginning, when number of chunks is small, time is constant, regulated
by *CHUNK_LOOP_MIN_TIME*, but when number of chunks became bigger then time of
loop can increase according to *CHUNKS_LOOP_MAX_CPS*.

Deletion limits are defined as 'soft' and 'hard' limit. When number of chunks
to delete increases from loop to loop then current limit can be temporary
increased above soft limit, but never above hard limit.

== REPORTING BUGS

Report bugs to the GitHub repository <https://github.com/leil-io/saunafs> as an
issue.

== COPYRIGHT

Copyright 2008-2009 Gemius SA

Copyright 2013-2019 Skytechnology sp. z o.o.

Copyright 2023-2024 Leil Storage OÜ

SaunaFS is free software: you can redistribute it and/or modify it under the terms of the GNU
General Public License as published by the Free Software Foundation, version 3.

SaunaFS is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even
the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
Public License for more details.

You should have received a copy of the GNU General Public License along with SaunaFS. If not, see
<http://www.gnu.org/licenses/>.

== SEE ALSO

sfsmaster(8), sfsexports.cfg(5), sfstopology.cfg(5)
