saunafs-uraft(8)
=================

== NAME

saunafs-uraft - SaunaFS high availability cluster management tool

== SYNOPSIS

[verse]
saunafs-uraft [*--help*]

[verse]
saunafs-uraft [*-c 'CFGFILE'*]

== DESCRIPTION

*saunafs-uraft* is a high availability (HA) management daemon for SaunaFS
metadata servers (*sfsmaster*).

[%hardbreaks]
It implements a Raft-inspired leader election protocol across a configured cluster and orchestrates
promotion/demotion of the local metadata service.

[%hardbreaks]
In a healthy cluster, exactly one node is the active master (Leader).
Other metadata-capable nodes run as shadows (Followers) and elector nodes (Followers)
participate in voting but never become Leader.

The leader holds a floating IP (VIP) so clients and chunkservers have a stable endpoint.

The election logic lives in the daemon, while backend-specific actions are delegated
to *saunafs-uraft-helper*:

- Health checks (`isalive`)
- Metadata version checks (`metadata-version`)
- Promotion and demotion (`promote` / `demote`)
- Floating IP actions (`assign-ip` / `drop-ip`)

See saunafs-uraft-helper(8) for the helper contract.

For more configuration info and examples, refer to *saunafs-uraft.cfg* file and
saunafs-uraft.cfg(5) man entry.

== CONCEPTS

*Node types*::

- *Metadata nodes*: run `sfsmaster` and can become Leader.
- *Elector nodes*: participate in voting but never become Leader.

Elector mode is controlled by `URAFT_ELECTOR_MODE`.

*Roles (Raft terms)*::

- *Follower*: passive node that receives heartbeats.
- *Candidate*: node requesting votes during an election.
- *Leader*: node that won an election.

uRaft also reports a `president` flag. Conceptually, `president=1` indicates the Leader is active
with quorum.

*Quorum*::
[%hardbreaks]
The daemon requires a majority to elect a Leader and to keep leadership.
For N voting nodes, quorum is:

quorum = floor(N/2) + 1

== OPERATIONS

At runtime the daemon loops through these responsibilities:

1. Exchange heartbeats with peers.
2. Start an election when heartbeats are missing long enough.
3. Consult the helper for metadata version and health.
4. When elected, call the helper to promote the node to master.
5. When stepping down, call the helper to demote the node to shadow.

Health gating:

- The daemon can temporarily block elections on a node via `blocked_promote`. +
  This is used to avoid electing a node whose local metadata server is unhealthy.

Floating IP:

- When configured, the Leader assigns the VIP and periodically checks it.
- If the VIP disappears unexpectedly, the daemon may attempt to restore it.

== PRODUCTION NOTES

*Cluster sizing*::

- Use an odd number of *voting* nodes (3, 5, 7).
- Keep voters on stable, low-latency links when possible.

*Network requirements*::

- uRaft uses UDP messaging. High packet loss and jitter can cause leader churn.
- For WAN/VPN deployments, adjust timing parameters (see saunafs-uraft.cfg(5)).

*Metadata freshness*::

- Elections consider metadata version. If a node cannot produce a valid metadata version quickly, +
it may be prevented from becoming Leader.

*Failure behavior*::

- If `isalive` reports dead on the current Leader, the safest operational outcome is: +
drop VIP, demote to shadow, and allow another node to be promoted.

*Cluster Expansion*::
[%hardbreaks]
Adding a new node changes the cluster membership and therefore the quorum size.
All uRaft nodes must have the exact same `URAFT_NODE_ADDRESS` list (same order).
Because the membership list is read at process startup, the change takes effect
after restarting *saunafs-uraft* on the nodes.

To expand the cluster with one additional node:

1. Pick the new node's uRaft UDP address and port (for example `new-node:9427`). +
Ensure UDP reachability between the new node and all existing voting nodes.

2. On *every* node (existing nodes and the new node), append a new +
`URAFT_NODE_ADDRESS = <new-node>:9427` line to `saunafs-uraft.cfg`. +
Keep the ordering identical across the cluster.

3. On the new node, set `URAFT_ID` to the index of the newly added `URAFT_NODE_ADDRESS` entry. +
(Example: if you added the 6th entry, set `URAFT_ID = 5`.)

4. If the new node is intended to be elector-only, set `URAFT_ELECTOR_MODE = 1` on that node. +
Otherwise keep it `0`.

5. Restart *saunafs-uraft* across the cluster. +
A rolling restart is usually acceptable, but expect a short leadership change window during restarts.

6. Verify the new node has joined: +
query the uRaft status endpoint (`URAFT_STATUS_PORT`) on multiple nodes and confirm the node count +
and leadership look correct.

After expansion, quorum becomes:

quorum = floor(N/2) + 1

where N is the number of voting nodes.

== MONITORING

*Status endpoint*::
[%hardbreaks]
The daemon exposes a TCP status endpoint (see `URAFT_STATUS_PORT`).
It reports role, term, leader id, `blocked_promote`, and vote/heartbeat summaries.

In general, the status endpoint of any node can be fetched with the command:
```bash
$ echo | nc <node-hostname> <URAFT_STATUS_PORT>
```

Therefore, the TCP status endpoint of the Leader can be queried with the command:
```bash
$ echo | nc sfsmaster 9428
```

A typical TCP status of the Leader looks like below:
```bash
SERVER ID 2
president=1
state=LEADER
term=171
voted_for=4
leader_id=4
data_version=334
loyalty_agreement=0
local_time=2090338
blocked_promote=0
votes=[       1|       1|       1|       1|       1|       1]
heart=[    0.00|    0.00|    0.00|    0.00|    0.00|    0.00]
recv =[       1|       1|       1|       1|       1|       1]
ver  =[     334|     334|     334|       0|     334|     334]
```

The TCP status endpoint in an elector node looks like below:
```bash
SERVER ID 0
president=0
state=FOLLOWER
term=171
voted_for=4
leader_id=4
data_version=334
loyalty_agreement=1
local_time=2091343
blocked_promote=1
```

*Logs*::
[%hardbreaks]
The daemon logs to syslog under the tag `saunafs-uraft`.
Repeated lines like "Isalive timeout" indicate the helper did not respond within the configured
timeout.

== OPTIONS

*--help*::
print usage information and exit

*--config 'CFGFILE'*::
provide an alternative configuration file path.

== REPORTING BUGS

Report bugs to the GitHub repository <https://github.com/leil-io/saunafs> as an issue.

== COPYRIGHT

Copyright 2008-2009 Gemius SA

Copyright 2013-2019 Skytechnology sp. z o.o.

Copyright 2023 Leil Storage OÜ
[%hardbreaks]
SaunaFS is free software: you can redistribute it and/or modify it under the
terms of the GNU General Public License as published by the Free Software
Foundation, version 3.

[%hardbreaks]
SaunaFS is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU General Public License for more details.

[%hardbreaks]
You should have received a copy of the GNU General Public License along with
SaunaFS. If not, see <http://www.gnu.org/licenses/>.

== SEE ALSO
saunafs-uraft-helper(8), saunafs-uraft.cfg(5)
