3. Set up the necessary environment variables where the compiled direct GekkoFS dependencies will be installed at (we assume the path `/home/foo/gekkofs_deps/install` in the following)
- (Optional) (Optional) If you checked out the sources using `git` without the `--recursive` option, you need to
execute the following command from the root of the source directory: `git submodule update --init`
3. Set up the necessary environment variables where the compiled direct GekkoFS dependencies will be installed at (we
assume the path `/home/foo/gekkofs_deps/install` in the following)
This section describes how to run GekkoFS locally or within a cluster environment.
## General
First of all, the GekkoFS daemon (`gkfs_daemon` binary) has to be started on each node. Other tools can be used to
execute the binary on many nodes, e.g., `srun`,
`mpiexec/mpirun`, `pdsh`, or `pssh`.
Now, you need to decide what Mercury NA plugin you want to use for network communication.
`ofi+sockets` is the default and uses the TCP protocol.
When running the daemon binary, the `-P` argument can be used to control which RPC protocol should be used:
-`ofi+sockets` for using the libfabric plugin with TCP (stable)
-`ofi+tcp` for using the libfabric plugin with TCP (slower than sockets)
-`ofi+verbs` for using the libfabric plugin with Infiniband verbs (reasonably stable) and requires
the [rdma-core (formerly libibverbs)](https://github.com/linux-rdma/rdma-core) library
-`ofi+psm2` for using the libfabric plugin with Intel Omni-Path (unstable) and requires
the [opa-psm2](https://github.com/cornelisnetworks/opa-psm2>) library
## The GekkoFS hostsfile
Each GekkoFS daemon needs to register itself in a shared file (*host file*) which needs to be accessible to **all**
GekkoFS clients and daemons. Therefore, the hostsfile describes a file system and which node is part of that specific
GekkoFS file system instance. Conceptually, the hostsfile represents a single GekkoFS file system instance. That is to
say, all daemons of one GekkoFS use the same hostsfile to identify as a server in that file system instance and
namespace.
```{important}
At this time, we only support strongly consistent parallel file systems, such as Lustre or GPFS, for storing the hostsfile, when one GekkoFS file systems consists of multiple servers.
While we will offer an alternative in the future, this means that eventual consistent file systems, e.g., NFS, cannot be used for the hostsfile.
Note that if only one daemon is part of a file system instance, and all GekkoFS client are run on the same node (e.g., a laptop), the hostsfile can be stored on a local file system.
When running the daemon, it requires two mandatory arguments: which specify where the daemon stores its data and
metadata locally, and at which path clients can access the file system (mount point):
1.`-r/--rootdir <fs_data_path>` specifies where the daemon stores its data and metadata locally. In general, the daemon
can use any device with a file system path that is accessible to the user, e.g., a RAMDisk or a node-local SSD.
2.`-m/--mountdir <pseudo_gkfs_mount_dir_path>` specifies a pseudo mount directory used by clients to access GekkoFS.
This pseudo mount directory differs from usual file system mount points which mounted as a kernel-based file system.
Therefore, GekkoFS will **not** appear when typing `mount` on the command line. Rather, the pseudo mount dir is used
later by the client interposition library which intercepts file system operations and processes those which are
within the GekkoFS namespace.
3. (optional) `-H/--hosts-file <hostsfile_path>` specifies the path where the hostsfile is placed. In a distributed
environment, all daemons should use the same file (see above) and, therefore, this argument should be used. By
default, the daemon creates a hostsfile in the current working directory (see below).
Further options are available
```bash
Allowed options
Usage: src/daemon/gkfs_daemon [OPTIONS]
Options:
-h,--help Print this help message and exit
-m,--mountdir TEXT REQUIRED Virtual mounting directory where GekkoFS is available.
-r,--rootdir TEXT REQUIRED Local data directory where GekkoFS data for this daemon is stored.
-s,--rootdir-suffix TEXT Creates an additional directory within the rootdir, allowing multiple daemons on one node.
-i,--metadir TEXT Metadata directory where GekkoFS RocksDB data directory is located. If not set, rootdir is used.
-l,--listen TEXT Address or interface to bind the daemon to. Default: local hostname.
When used with ofi+verbs the FI_VERBS_IFACE environment variable is set accordingly which associates the verbs device with the network interface. In case FI_VERBS_IFACE is already defined, the argument is ignored. Default 'ib'.
-H,--hosts-file TEXT Shared file used by deamons to register their endpoints. (default './gkfs_hosts.txt')
-P,--rpc-protocol TEXT Used RPC protocol for inter-node communication.
