From 5fbc9f5046b8eccb66334ac56de060fa78d6d7ae Mon Sep 17 00:00:00 2001
From: Marc Vef <vef@uni-mainz.de>
Date: Mon, 22 Jul 2024 15:12:09 +0200
Subject: [PATCH] Restructuring of Readme

---
 README.md | 258 ++++++++++++++++++++++++++++++++----------------------
 1 file changed, 151 insertions(+), 107 deletions(-)

diff --git a/README.md b/README.md
index e70bceff4..db1c77893 100644
--- a/README.md
+++ b/README.md
@@ -9,9 +9,41 @@ in a HPC cluster to produce a high-performance storage space that can be accesse
 This storage space allows HPC applications and simulations to run in isolation from each other with regards
 to I/O, which reduces interferences and improves performance.
 
+# Table of contents
+
+- [Dependencies](#dependencies)
+  - [Debian/Ubuntu](#debianubuntu)
+  - [CentOS/Red Hat](#centosred-hat)
+- [Step-by-step installation](#step-by-step-installation)
+- [Run GekkoFS](#run-gekkofs)
+  - [The GekkoFS hostsfile](#the-gekkofs-hostsfile)
+  - [The GekkoFS daemon](#the-gekkofs-daemon)
+    - [Manual startup and shut down](#manual-startup-and-shut-down)
+    - [GekkoFS daemon orchestration via the gkfs script (recommended)](#gekkofs-daemon-orchestration-via-the-gkfs-script-recommended)
+  - [The GekkoFS client library](#the-gekkofs-client-library)
+    - [Interposition library via system call interception](#interposition-library-via-system-call-interception)
+    - [User library via linking against the application](#user-library-via-linking-against-the-application)
+  - [Logging](#logging)
+- [Advanced and experimental features](#advanced-and-experimental-features)
+  - [Rename](#rename)
+  - [Replication](#replication)
+  - [Client-side metrics via MessagePack and ZeroMQ](#client-side-metrics-via-messagepack-and-zeromq)
+  - [Server-side statistics via Prometheus](#server-side-statistics-via-prometheus)
+  - [GekkoFS proxy](#gekkofs-proxy)
+  - [File system expansion](#file-system-expansion)
+- [Miscellaneous](#miscellaneous)
+  - [External functions](#external-functions)
+  - [Data placement](#data-placement)
+    - [Simple Hash (Default)](#simple-hash-default)
+    - [Guided Distributor](#guided-distributor)
+  - [Metadata Backends](#metadata-backends)
+  - [CMake options](#cmake-options)
+  - [Environment variables](#environment-variables)
+- [Acknowledgment](#acknowledgment)
+
 # Dependencies
 
-- \>gcc-8 (including g++) for C++11 support
+- \>gcc-12 (including g++) for C++17 support
 - General build tools: Git, Curl, CMake >3.6 (>3.11 for GekkoFS testing), Autoconf, Automake
 - Miscellaneous: Libtool, Libconfig
 
@@ -57,15 +89,13 @@ GekkoFS is now available at:
 - GekkoFS daemon (server): `<install_path>/bin/gkfs_daemon`
 - GekkoFS client interception library: `<install_path>/lib64/libgkfs_intercept.so`
 
-## Use Spack to install GekkoFS (alternative)
+## Spack for installing GekkoFS (alternative)
 
 The Spack tool can be used to easily install GekkoFS and its dependencies. Refer to the
 following [README](scripts/spack/README.md) for details.
 
 # Run GekkoFS
 
-## General
-
 On each node a daemon (`gkfs_daemon` binary) has to be started. Other tools can be used to execute
 the binary on many nodes, e.g., `srun`, `mpiexec/mpirun`, `pdsh`, or `pssh`.
 
@@ -89,7 +119,11 @@ GPFS or Lustre.
 
 *Note: NFS is not strongly consistent and cannot be used for the hosts file!*
 
-## GekkoFS daemon start and shut down
+## The GekkoFS daemon
+
+The GekkoFS daemon is the server component of GekkoFS. It is responsible for managing the file system data and metadata. There are two options to run the daemons on one or several nodes: (1) manually by executing the `gkfs_daemon` binary directly or (2) by using the `gkfs` script (recommended).
+
+### Manual startup and shut down
 
 tl;dr example: `<install_path>/bin/gkfs_daemon -r <fs_data_path> -m <pseudo_gkfs_mount_dir_path> -H <hostsfile_path>`
 
@@ -137,29 +171,7 @@ are part of the same file system, use the same `rootdir` with different `rootdir
 
 Shut it down by gracefully killing the process (SIGTERM).
 
-## Use the GekkoFS client library
-
-tl;dr example:
-
-```bash
-export LIBGKFS_ HOSTS_FILE=<hostfile_path>
-LD_PRELOAD=<install_path>/lib64/libgkfs_intercept.so cp ~/some_input_data <pseudo_gkfs_mount_dir_path>/some_input_data
-LD_PRELOAD=<install_path>/lib64/libgkfs_intercept.so md5sum ~/some_input_data <pseudo_gkfs_mount_dir_path>/some_input_data
-```
-
-Clients read the hostsfile to determine which daemons are part of the GekkoFS instance. Because the client is an
-interposition library that is loaded within the context of the application, this information is passed via the
-environment variable `LIBGKFS_HOSTS_FILE` pointing to the hostsfile path. The client library itself is loaded for each
-application process via the `LD_PRELOAD` environment variable intercepting file system related calls. If they are
-within (or hierarchically under) the GekkoFS mount directory they are processed in the library, otherwise they are
-passed to the kernel.
-
-Note, if `LD_PRELOAD` is not pointing to the library and, hence the client is not loaded, the mounting directory appears
-to be empty.
-
-For MPI application, the `LD_PRELOAD` variable can be passed with the `-x` argument for `mpirun/mpiexec`.
-
-## Run GekkoFS daemons on multiple nodes
+### GekkoFS daemon orchestration via the `gkfs` script (recommended)
 
 The `scripts/run/gkfs` script can be used to simplify starting the GekkoFS daemon on one or multiple nodes. To start
 GekkoFS on multiple nodes, a Slurm environment that can execute `srun` is required. Users can further
@@ -198,7 +210,39 @@ usage: gkfs [-h/--help] [-r/--rootdir <path>] [-m/--mountdir <path>] [-a/--args
             -v, --verbose           Increase verbosity
 ```
 
-### Logging
+## The GekkoFS client library
+
+### Interposition library via system call interception
+
+tl;dr example:
+
+```bash
+export LIBGKFS_ HOSTS_FILE=<hostfile_path>
+LD_PRELOAD=<install_path>/lib64/libgkfs_intercept.so cp ~/some_input_data <pseudo_gkfs_mount_dir_path>/some_input_data
+LD_PRELOAD=<install_path>/lib64/libgkfs_intercept.so md5sum ~/some_input_data <pseudo_gkfs_mount_dir_path>/some_input_data
+```
+
+Clients read the hostsfile to determine which daemons are part of the GekkoFS instance. Because the client is an
+interposition library that is loaded within the context of the application, this information is passed via the
+environment variable `LIBGKFS_HOSTS_FILE` pointing to the hostsfile path. The client library itself is loaded for each
+application process via the `LD_PRELOAD` environment variable intercepting file system related calls. If they are
+within (or hierarchically under) the GekkoFS mount directory they are processed in the library, otherwise they are
+passed to the kernel.
+
+Note, if `LD_PRELOAD` is not pointing to the library and, hence the client is not loaded, the mounting directory appears
+to be empty.
+
+For MPI application, the `LD_PRELOAD` variable can be passed with the `-x` argument for `mpirun/mpiexec`.
+
+### User library via linking against the application
+
+GekkoFS offers a user library that can be linked against the application, which is built by default:
+`libgkfs_user_lib.so` shared library. The corresponding API and developer headers are available in
+`include/client/user_functions.hpp`. Please consult `examples/user_library` for details.
+
+In this case, `LD_PRELOAD` is not necessary. Nevertheless, `LIBGKFS_HOSTS_FILE` is still required.
+
+## Logging
 
 The following environment variables can be used to enable logging in the client library: `LIBGKFS_LOG=<module>`
 and `LIBGKFS_LOG_OUTPUT=<path/to/file>` to configure the output module and set the path to the log file of the client
@@ -244,78 +288,9 @@ For the daemon, the `GKFS_DAEMON_LOG_PATH=<path/to/file>` environment variable c
 log file, and the log module can be selected with the `GKFS_DAEMON_LOG_LEVEL={off,critical,err,warn,info,debug,trace}`
 environment variable.
 
-# Miscellaneous
-
-## External functions
-
-GekkoFS allows to use external functions on your client code, via LD_PRELOAD.
-Source code needs to be compiled with -fPIC. We include a pfind io500 substitution,
-`examples/gfind/gfind.cpp` and a non-mpi version `examples/gfind/sfind.cpp`
-
-## Data distributors
-
-The data distribution can be selected at compilation time, we have 2 distributors available:
-
-### Simple Hash (Default)
-
-Chunks are distributed randomly to the different GekkoFS servers.
-
-### Guided Distributor
-
-The guided distributor allows defining a specific distribution of data on a per directory or file basis.
-The distribution configurations are defined within a shared file (called `guided_config.txt` henceforth) with the
-following format:
-`<path> <chunk_number> <host>`
-
-To enable the distributor, the following CMake compilation flags are required:
-
-* `GKFS_USE_GUIDED_DISTRIBUTION` ON
-* `GKFS_USE_GUIDED_DISTRIBUTION_PATH` `<path_guided_config.txt>`
-
-To use a custom distribution, a path needs to have the prefix `#` (e.g., `#/mdt-hard 0 0`), in which all the data of all
-files in that directory goes to the same place as the metadata.
-Note, that a chunk/host configuration is inherited to all children files automatically even if not using the prefix.
-In this example, `/mdt-hard/file1` is therefore also using the same distribution as the `/mdt-hard` directory.
-If no prefix is used, the Simple Hash distributor is used.
-
-#### Guided configuration file
-
-Creating a guided configuration file is based on an I/O trace file of a previous execution of the application.
-For this the `trace_reads` tracing module is used (see above).
-
-The `trace_reads` module enables a `TRACE_READS` level log at the clients writing the I/O information of the client
-which is used as the input for a script that creates the guided distributor setting.
-Note that capturing the necessary trace records can involve performance degradation.
-To capture the I/O of each client within a SLURM environment, i.e., enabling the `trace_reads` module and print its
-output to a user-defined path, the following example can be used:
-`srun -N 10 -n 320 --export="ALL" /bin/bash -c "export LIBGKFS_LOG=trace_reads;LIBGKFS_LOG_OUTPUT=${HOME}/test/GLOBAL.txt;LD_PRELOAD=${GKFS_PRLD} <app>"`
-
-Then, the `examples/distributors/guided/generate.py` scrpt is used to create the guided distributor configuration file:
-
-* `python examples/distributors/guided/generate.py ~/test/GLOBAL.txt >> guided_config.txt`
-
-Finally, modify `guided_config.txt` to your distribution requirements.
-
-## Metadata Backends
-
-There are two different metadata backends in GekkoFS. The default one uses `rocksdb`, however an alternative based
-on `PARALLAX` from `FORTH`
-is available. To enable it use the `-DGKFS_ENABLE_PARALLAX:BOOL=ON` option, you can also disable `rocksdb`
-with `-DGKFS_ENABLE_ROCKSDB:BOOL=OFF`.
-
-Once it is enabled, `--dbbackend` option will be functional.
-
-## Statistics
-
-GekkoFS daemons are able to output general operations (`--enable-collection`) and data chunk
-statistics (`--enable-chunkstats`) to a specified output file via `--output-stats <FILE>`. Prometheus can also be used
-instead or in addition to the output file. It must be enabled at compile time via the CMake
-argument `-DGKFS_ENABLE_PROMETHEUS` and the daemon argument `--enable-prometheus`. The corresponding statistics are then
-pushed to the Prometheus instance.
-
-## Advanced and experimental features
+# Advanced and experimental features
 
-### Rename
+## Rename
 
 `-DGKFS_RENAME_SUPPORT` allows the application to rename files.
 This is an experimental feature, and some scenarios may not work properly.
@@ -323,14 +298,14 @@ Support for fstat in renamed files is included.
 
 This is disabled by default.
 
-### Replication
+## Replication
 
 The user can enable the data replication feature by setting the replication environment variable:
 `LIBGKFS_NUM_REPL=<num repl>`.
 The number of replicas should go from `0` to the `number of servers - 1`. The replication environment variable can be
 set up for each client independently.
 
-### Client metrics via MessagePack and ZeroMQ
+## Client-side metrics via MessagePack and ZeroMQ
 
 GekkoFS clients support capturing the I/O traces of each individual process and periodically exporting them to a given
 file or ZeroMQ sink via the TCP protocol.
@@ -375,7 +350,15 @@ total_bytes: 1802366
 total_iops: 4
 ```
 
-### GekkoFS proxy
+## Server-side statistics via Prometheus
+
+GekkoFS daemons are able to output general operations (`--enable-collection`) and data chunk
+statistics (`--enable-chunkstats`) to a specified output file via `--output-stats <FILE>`. Prometheus can also be used
+instead or in addition to the output file. It must be enabled at compile time via the CMake
+argument `-DGKFS_ENABLE_PROMETHEUS` and the daemon argument `--enable-prometheus`. The corresponding statistics are then
+pushed to the Prometheus instance.
+
+## GekkoFS proxy
 
 The GekkoFS proxy is an additional (alternative) component that runs on each client and acts as gateway between the
 client and daemons. It can improve network stability, e.g., for opa-psm2, and provides a basis for future asynchronous
@@ -417,7 +400,7 @@ Press 'q' to exit
 Please consult `include/config.hpp` for additional configuration options. Note, GekkoFS proxy does not support
 replication.
 
-### File system expansion
+## File system expansion
 
 GekkoFS supports extending the current daemon configuration to additional compute nodes. This includes redistribution of
 the existing data and metadata and therefore scales file system performance and capacity of existing data. Note,
@@ -469,7 +452,68 @@ srun: sending Ctrl-C to StepId=282378.2
 * [gkfs] Shutdown time: 1.032 seconds
 ```
 
-## All CMake options
+# Miscellaneous
+
+## External functions
+
+GekkoFS allows to use external functions on your client code, via LD_PRELOAD.
+Source code needs to be compiled with -fPIC. We include a pfind io500 substitution,
+`examples/gfind/gfind.cpp` and a non-mpi version `examples/gfind/sfind.cpp`
+
+## Data placement
+
+The data distribution can be selected at compilation time, we have 2 distributors available:
+
+### Simple Hash (Default)
+
+Chunks are distributed randomly to the different GekkoFS servers.
+
+### Guided Distributor
+
+The guided distributor allows defining a specific distribution of data on a per directory or file basis.
+The distribution configurations are defined within a shared file (called `guided_config.txt` henceforth) with the
+following format:
+`<path> <chunk_number> <host>`
+
+To enable the distributor, the following CMake compilation flags are required:
+
+* `GKFS_USE_GUIDED_DISTRIBUTION` ON
+* `GKFS_USE_GUIDED_DISTRIBUTION_PATH` `<path_guided_config.txt>`
+
+To use a custom distribution, a path needs to have the prefix `#` (e.g., `#/mdt-hard 0 0`), in which all the data of all
+files in that directory goes to the same place as the metadata.
+Note, that a chunk/host configuration is inherited to all children files automatically even if not using the prefix.
+In this example, `/mdt-hard/file1` is therefore also using the same distribution as the `/mdt-hard` directory.
+If no prefix is used, the Simple Hash distributor is used.
+
+#### Guided configuration file
+
+Creating a guided configuration file is based on an I/O trace file of a previous execution of the application.
+For this the `trace_reads` tracing module is used (see above).
+
+The `trace_reads` module enables a `TRACE_READS` level log at the clients writing the I/O information of the client
+which is used as the input for a script that creates the guided distributor setting.
+Note that capturing the necessary trace records can involve performance degradation.
+To capture the I/O of each client within a SLURM environment, i.e., enabling the `trace_reads` module and print its
+output to a user-defined path, the following example can be used:
+`srun -N 10 -n 320 --export="ALL" /bin/bash -c "export LIBGKFS_LOG=trace_reads;LIBGKFS_LOG_OUTPUT=${HOME}/test/GLOBAL.txt;LD_PRELOAD=${GKFS_PRLD} <app>"`
+
+Then, the `examples/distributors/guided/generate.py` scrpt is used to create the guided distributor configuration file:
+
+* `python examples/distributors/guided/generate.py ~/test/GLOBAL.txt >> guided_config.txt`
+
+Finally, modify `guided_config.txt` to your distribution requirements.
+
+## Metadata Backends
+
+There are two different metadata backends in GekkoFS. The default one uses `rocksdb`, however an alternative based
+on `PARALLAX` from `FORTH`
+is available. To enable it use the `-DGKFS_ENABLE_PARALLAX:BOOL=ON` option, you can also disable `rocksdb`
+with `-DGKFS_ENABLE_ROCKSDB:BOOL=OFF`.
+
+Once it is enabled, `--dbbackend` option will be functional.
+
+## CMake options
 
 #### Core
 - `GKFS_BUILD_TOOLS` - Build tools (default: OFF)
@@ -495,7 +539,7 @@ srun: sending Ctrl-C to StepId=282378.2
 - `GKFS_ENABLE_ROCKSDB` - Enable RocksDB metadata backend (default: ON)
 - `GKFS_ENABLE_PARALLAX` - Enable Parallax metadata support (default: OFF)
 
-## All environment variables
+## Environment variables
 The GekkoFS daemon, client, and proxy support a number of environment variables to augment its functionality:
 
 ### Client
@@ -545,7 +589,7 @@ until the file is closed. The cache does not impact the consistency of the file
 - `GKFS_PROXY_LOG_PATH` - Path to the log file of the proxy.
 - `GKFS_PROXY_LOG_LEVEL` - Log level of the proxy. Available levels are: `off`, `critical`, `err`, `warn`, `info`, `debug`, `trace`.
 
-## Acknowledgment
+# Acknowledgment
 
 This software was partially supported by the EC H2020 funded NEXTGenIO project (Project ID: 671951, www.nextgenio.eu).
 
-- 
GitLab