Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
# Running GekkoFS
This section describes how to run GekkoFS locally or within a cluster environment.
## General
First of all, the GekkoFS daemon (`gkfs_daemon` binary) has to be started on each node. Other tools can be used to
execute the binary on many nodes, e.g., `srun`,
`mpiexec/mpirun`, `pdsh`, or `pssh`.
Now, you need to decide what Mercury NA plugin you want to use for network communication.
`ofi+sockets` is the default and uses the TCP protocol.
When running the daemon binary, the `-P` argument can be used to control which RPC protocol should be used:
- `ofi+sockets` for using the libfabric plugin with TCP (stable)
- `ofi+tcp` for using the libfabric plugin with TCP (slower than sockets)
- `ofi+verbs` for using the libfabric plugin with Infiniband verbs (reasonably stable) and requires
the [rdma-core (formerly libibverbs)](https://github.com/linux-rdma/rdma-core) library
- `ofi+psm2` for using the libfabric plugin with Intel Omni-Path (unstable) and requires
the [opa-psm2](https://github.com/cornelisnetworks/opa-psm2>) library
## The GekkoFS hostsfile
Each GekkoFS daemon needs to register itself in a shared file (*host file*) which needs to be accessible to **all**
GekkoFS clients and daemons. Therefore, the hostsfile describes a file system and which node is part of that specific
GekkoFS file system instance. Conceptually, the hostsfile represents a single GekkoFS file system instance. That is to
say, all daemons of one GekkoFS use the same hostsfile to identify as a server in that file system instance and
namespace.
```{important}
At this time, we only support strongly consistent parallel file systems, such as Lustre or GPFS, for storing the hostsfile, when one GekkoFS file systems consists of multiple servers.
While we will offer an alternative in the future, this means that eventual consistent file systems, e.g., NFS, cannot be used for the hostsfile.
Note that if only one daemon is part of a file system instance, and all GekkoFS client are run on the same node (e.g., a laptop), the hostsfile can be stored on a local file system.
```
## GekkoFS daemon start and shutdown
tl;dr example: `<install_path>/bin/gkfs_daemon -r <fs_data_path> -m <pseudo_gkfs_mount_dir_path> -H <hostsfile_path>`
When running the daemon, it requires two mandatory arguments: which specify where the daemon stores its data and
metadata locally, and at which path clients can access the file system (mount point):
1. `-r/--rootdir <fs_data_path>` specifies where the daemon stores its data and metadata locally. In general, the daemon
can use any device with a file system path that is accessible to the user, e.g., a RAMDisk or a node-local SSD.
2. `-m/--mountdir <pseudo_gkfs_mount_dir_path>` specifies a pseudo mount directory used by clients to access GekkoFS.
This pseudo mount directory differs from usual file system mount points which mounted as a kernel-based file system.
Therefore, GekkoFS will **not** appear when typing `mount` on the command line. Rather, the pseudo mount dir is used
later by the client interposition library which intercepts file system operations and processes those which are
within the GekkoFS namespace.
3. (optional) `-H/--hosts-file <hostsfile_path>` specifies the path where the hostsfile is placed. In a distributed
environment, all daemons should use the same file (see above) and, therefore, this argument should be used. By
default, the daemon creates a hostsfile in the current working directory (see below).
Further options are available
```bash
Allowed options
Usage: src/daemon/gkfs_daemon [OPTIONS]
Options:
-h,--help Print this help message and exit
-m,--mountdir TEXT REQUIRED Virtual mounting directory where GekkoFS is available.
-r,--rootdir TEXT REQUIRED Local data directory where GekkoFS data for this daemon is stored.
-s,--rootdir-suffix TEXT Creates an additional directory within the rootdir, allowing multiple daemons on one node.
-i,--metadir TEXT Metadata directory where GekkoFS RocksDB data directory is located. If not set, rootdir is used.
-l,--listen TEXT Address or interface to bind the daemon to. Default: local hostname.
When used with ofi+verbs the FI_VERBS_IFACE environment variable is set accordingly which associates the verbs device with the network interface. In case FI_VERBS_IFACE is already defined, the argument is ignored. Default 'ib'.
-H,--hosts-file TEXT Shared file used by deamons to register their endpoints. (default './gkfs_hosts.txt')
-P,--rpc-protocol TEXT Used RPC protocol for inter-node communication.
Available: {ofi+sockets, ofi+verbs, ofi+psm2} for TCP, Infiniband, and Omni-Path, respectively. (Default ofi+sockets)
Libfabric must have enabled support verbs or psm2.
--auto-sm Enables intra-node communication (IPCs) via the `na+sm` (shared memory) protocol, instead of using the RPC protocol. (Default off)
-c,--clean-rootdir Cleans Rootdir >before< launching the deamon
--version Print version and exit.
````
Shut it down by gracefully killing the process (SIGTERM).
```{note}
It is possible to run multiple independent GekkoFS instances on the same node. Note, that when these GekkoFS instances
are part of the same file system, use the same `rootdir` with different `rootdir-suffixe`.
```
### Running and shutting down GekkoFS as a Slurm job step
To run GekkoFS as a Slurm job step, it is as easy as executing `srun <path_to_daemon_binary>/gkfs_daemon <arguments> &`
to launch GekkoFS in the background. An easy way to check if all daemons have started, is to count the number of lines
in the hostsfile which corresponds to the number of started daemons.
It is recommended that you explicitly tell Slurm which and how many resources it should use on each node.
Noteworthy `srun` arguments are as follows:
- `--disable-status` allows to immediately forward a signal, e.g., `SIGINT`, to the running job
- `-N`, `--ntasks`, and `--ntasks-per-node` allows defining how many daemons are run on how many nodes
- `--cpus-per-task` allows to set the number of CPU resources the daemon can use
If a node features multiple CPU sockets, it is recommended to pin the daemon to one socket with `numactl`, `taskset`, or
similar.
To gracefully shut down all daemons, send `SIGINT` to the background process: `kill -s SIGINT <srun_gekkofs_pid>`. This
command finishes once all daemons have shut down.
## Use the GekkoFS client library
tl;dr example:
```bash
export LIBGKFS_ HOSTS_FILE=<hostfile_path>
LD_PRELOAD=<install_path>/lib64/libgkfs_intercept.so cp ~/some_input_data <pseudo_gkfs_mount_dir_path>/some_input_data
LD_PRELOAD=<install_path>/lib64/libgkfs_intercept.so md5sum ~/some_input_data <pseudo_gkfs_mount_dir_path>/some_input_data
```
Clients read the hostsfile to determine which daemons are part of the GekkoFS instance. Because the client is an
interposition library that is loaded within the context of the application, this information is passed via the
environment variable `LIBGKFS_HOSTS_FILE` pointing to the hostsfile path. The client library itself is loaded for each
application process via the `LD_PRELOAD` environment variable intercepting file system related calls. If they are
within (or hierarchically under) the GekkoFS mount directory they are processed in the library, otherwise they are
passed to the kernel.
Note, if `LD_PRELOAD` is not pointing to the library and, hence the client is not loaded, the mounting directory appears
to be empty.
For MPI applications, the `LD_PRELOAD` and `LIBGKFS_HOSTS_FILE` variables can be passed with the `-x` argument
for `mpirun/mpiexec`.
### Logging
#### Client logging
The following environment variables can be used to enable logging in the client library: `LIBGKFS_LOG=<module>`
and `LIBGKFS_LOG_OUTPUT=<path/to/file>` to configure the output module and set the path to the log file of the client
library. If not path is specified in `LIBGKFS_LOG_OUTPUT`, the client library will send log messages
to `/tmp/gkfs_client.log`.
The following modules are available:
- `none`: don't print any messages
- `syscalls`: Trace system calls: print the name of each system call, its arguments, and its return value. All system
calls are printed after being executed save for those that may not return, such as `execve()`,
`execve_at()`, `exit()`, and `exit_group()`. This module will only be available if the client library is built
in `Debug` mode.
- `syscalls_at_entry`: Trace system calls: print the name of each system call and its arguments. All system calls are
printed before being executed and therefore their return values are not available in the log. This module will only be
available if the client library is built in `Debug` mode.
- `info`: Print information messages.
- `critical`: Print critical errors.
- `errors`: Print errors.
- `warnings`: Print warnings.
- `mercury`: Print Mercury messages.
- `debug`: Print debug messages. This module will only be available if the client library is built in `Debug` mode.
- `most`: All previous options combined except `syscalls_at_entry`. This module will only be available if the client
library is built in `Debug`
mode.
- `all`: All previous options combined.
- `trace_reads`: Generate log line with extra information in read operations for guided distributor
- `help`: Print a help message and exit.
When tracing sytem calls, specific syscalls can be removed from log messages by setting the `LIBGKFS_LOG_SYSCALL_FILTER`
environment variable. For instance, setting it to `LIBGKFS_LOG_SYSCALL_FILTER=epoll_wait,epoll_create` will filter out
any log entries from the `epoll_wait()` and `epoll_create()` system calls.
Additionally, setting the `LIBGKFS_LOG_OUTPUT_TRUNC` environment variable with a value different from `0` will instruct
the logging subsystem to truncate the file used for logging, rather than append to it.
#### Daemon logging
For the daemon, the `GKFS_DAEMON_LOG_PATH=<path/to/file>` environment variable can be provided to set the path to the
log file, and the log module can be selected with the `GKFS_LOG_LEVEL={off,critical,err,warn,info,debug,trace}`
environment variable whereas `trace` produces the most trace records while `info` is the default value.