Updating README.md

# GekkoFS

This is a file system.

GekkoFS is a file system capable of aggregating the local I/O capacity and performance of each compute node

in a HPC cluster to produce a high-performance storage space that can be accessed in a distributed manner.

This storage space allows HPC applications and simulations to run in isolation from each other with regards 

to I/O, which reduces interferences and improves performance.

# Dependencies

## Rocksdb

- Upgrade your gcc to version at least 4.8 to get C++11 support

- CMake >3.6 (>3.11 for GekkoFS testing)

### Debian/Ubuntu - Dependencies

## Debian/Ubuntu - Dependencies

- Upgrade your gcc to version at least 4.8 to get C++11 support.

- Install snappy. This is usually as easy as: `sudo apt-get install libsnappy-dev`

- Install zlib. Try: `sudo apt-get install zlib1g-dev`

- Install bzip2: `sudo apt-get install libbz2-dev`

- Install zstandard: `sudo apt-get install libzstd-dev`

- Install lz4 `sudo apt-get install liblz4-dev`

- snappy: `sudo apt-get install libsnappy-dev`

- zlib: `sudo apt-get install zlib1g-dev`

- bzip2: `sudo apt-get install libbz2-dev`

- zstandard: `sudo apt-get install libzstd-dev`

- lz4: `sudo apt-get install liblz4-dev`

- uuid: `sudo apt-get install uuid-dev`

- capstone: `sudo apt-get install libcapstone-dev`

### CentOS/Red Hat - Dependencies

- Upgrade your gcc to version at least 4.8 to get C++11 support: yum install gcc48-c++

- Install snappy:

    `sudo yum install snappy snappy-devel`

- Install zlib:

    `sudo yum install zlib zlib-devel`

- Install bzip2:

    `sudo yum install bzip2 bzip2-devel`

- Install ASAN (optional for debugging):

    `sudo yum install libasan`

- Install zstandard:

- snappy: `sudo yum install snappy snappy-devel`

- zlib: `sudo yum install zlib zlib-devel`

- bzip2: `sudo yum install bzip2 bzip2-devel`

- zstandard: 

```bash

   wget https://github.com/facebook/zstd/archive/v1.1.3.tar.gz

   mv v1.1.3.tar.gz zstd-1.1.3.tar.gz

   cd zstd-1.1.3

   make && sudo make install

```

- lz4: `sudo yum install lz4 lz4-devel`

- uuid: `sudo yum install libuuid-devel`

- capstone: `sudo yum install capstone capstone-devel`

# Usage

## Clone and compile direct GekkoFS dependencies

- Go to the `scripts` folder and first clone all dependencies projects. You can choose the according na_plugin

- Go to the `scripts` folder and first clone all dependencies projects. You can choose the according network (na) plugin

(execute the script for help):

```bash

usage: dl_dep.sh [-h] [-l] [-n <NAPLUGIN>] [-c <CLUSTER>] [-d <DEPENDENCY>] 

usage: dl_dep.sh [-h] [-l] [-n <NAPLUGIN>] [-c <CONFIG>] [-d <DEPENDENCY>]

                    source_path

optional arguments:

        -h, --help              shows this help message and exits

        -l, --list-dependencies

                                list dependencies available for download

                                list dependencies available for download with descriptions

        -n <NAPLUGIN>, --na <NAPLUGIN>

                                network layer that is used for communication. Valid: {bmi,ofi,all}

                                defaults to 'all'

        -c <CLUSTER>, --cluster <CLUSTER>

                                additional configurations for specific compute clusters

                                supported clusters: {mogon1,mogon2,fh2}

                                defaults to 'ofi'

        -c <CONFIG>, --config <CONFIG>

                                allows additional configurations, e.g., for specific clusters

                                supported values: {mogon2, mogon1, ngio, direct, all}

                                defaults to 'direct'

        -d <DEPENDENCY>, --dependency <DEPENDENCY>

                                download a specific dependency. If unspecified 

                                all dependencies are built and installed.

                                download a specific dependency and ignore --config setting. If unspecified

                                all dependencies are built and installed based on set --config setting.

                                Multiple dependencies are supported: Pass a space-separated string (e.g., "ofi mercury"

        -v, --verbose           Increase download verbosity

```

- Now use the install script to compile them and install them to the desired directory. You can choose the according

na_plugin (execute the script for help):

(na) network plugin (execute the script for help):

```bash

usage: compile_dep.sh [-h] [-l] [-n <NAPLUGIN>] [-c <CLUSTER>] [-d <DEPENDENCY>] [-j <COMPILE_CORES>]

usage: compile_dep.sh [-h] [-l] [-n <NAPLUGIN>] [-c <CONFIG>] [-d <DEPENDENCY>] [-j <COMPILE_CORES>]

                      source_path install_path

    -n <NAPLUGIN>, --na <NAPLUGIN>

                network layer that is used for communication. Valid: {bmi,ofi,all}

                defaults to 'all'

    -c <CLUSTER>, --cluster <CLUSTER>

                additional configurations for specific compute clusters

                supported clusters: {mogon1,mogon2,fh2}

    -c <CONFIG>, --config <CONFIG>

                allows additional configurations, e.g., for specific clusters

                supported values: {mogon1, mogon2, ngio, direct, all}

                defaults to 'direct'

    -d <DEPENDENCY>, --dependency <DEPENDENCY>

                build and install a specific dependency. If unspecified 

                all dependencies are built and installed.

                download a specific dependency and ignore --config setting. If unspecified

                all dependencies are built and installed based on set --config setting.

                Multiple dependencies are supported: Pass a space-separated string (e.g., "ofi mercury"

    -j <COMPILE_CORES>, --compilecores <COMPILE_CORES>

                number of cores that are used to compile the dependencies

                defaults to number of available cores

```

## Compile GekkoFS

You need to decide what Mercury NA plugin you want to use. The following NA plugins are available, although only BMI is considered stable at the moment.

 - `ofi+sockets` for using the libfabric plugin with TCP

 - `ofi+tcp` for using the libfabric plugin with TCP (new version)

 - `ofi+verbs` for using the libfabric plugin with Infiniband verbs (not threadsafe. Do not use.)

 - `ofi+psm2` for using the libfabric plugin with Intel Omni-Path

 - `bmi+tcp` for using the bmi plugin with the tcp protocol 

If above dependencies have been installed outside of the usual system paths, use CMake's `-DCMAKE_PREFIX_PATH` to

make this path known to CMake.

```bash

mkdir build && cd build

cmake -DCMAKE_BUILD_TYPE=Release -DRPC_PROTOCOL='ofi+sockets' ..

cmake -DCMAKE_BUILD_TYPE=Release ..

make

make install

```

In order to build self-tests, the *optional* GKFS_BUILD_TESTS CMake option needs

```bash

mkdir build && cd build

cmake -DGKFS_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release -DRPC_PROTOCOL='ofi+sockets' ..

cmake -DGKFS_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release ..

make

make test

make install

```

**IMPORTANT:** Please note that the testing framework requires Python 3.6 as an

additional dependency in order to run.

**IMPORTANT:** Please note that the testing framework requires Python 3.6 and >CMake 3.11 as

additional dependencies in order to run.

## Run GekkoFS

First on each node a daemon has to be started. This can be done in two ways using the `gkfs_daemon` binary directly or

the corresponding startup and shutdown scripts. The latter is recommended for cluster usage. It requires pssh (or

parallel-ssh) with python2.

On each node a daemon (`gkfs_daemon` binary) has to be started. Other tools can be used to execute

the binary on many nodes, e.g., `srun`, `mpiexec/mpirun`, `pdsh`, or `pssh`.

### Start and shut down daemon directly

You need to decide what Mercury NA plugin you want to use for network communication. `ofi+sockets` is the default.

The `-P` argument is used for setting another RPC protocol. See below.

 - `ofi+sockets` for using the libfabric plugin with TCP (stable)

 - `ofi+tcp` for using the libfabric plugin with TCP (slower than sockets)

 - `ofi+verbs` for using the libfabric plugin with Infiniband verbs (reasonably stable)

 - `ofi+psm2` for using the libfabric plugin with Intel Omni-Path (unstable)

 - `bmi+tcp` for using the bmi plugin with TCP (alternative to libfabric)

`./build/bin/gkfs_daemon -r <fs_data_path> -m <pseudo_mount_dir_path>`

### Start and shut down daemon directly

Shut it down by gracefully killing the process.

`./build/src/daemon/gkfs_daemon -r <fs_data_path> -m <pseudo_mount_dir_path>`

### Startup and shutdown scripts

Further options:

```bash

Allowed options:

  -h [ --help ]             Help message

  -m [ --mountdir ] arg     pseudo mountdir

  -r [ --rootdir ] arg      data directory

  -i [ --metadir ] arg      metadata directory, if not set rootdir is used for

                            metadata

  -l [ --listen ] arg       Address or interface to bind the daemon on.

                            Default: local hostname.

                            When used with ofi+verbs the FI_VERBS_IFACE

                            environment variable is set accordingly which

                            associates the verbs device with the network

                            interface. In case FI_VERBS_IFACE is already

                            defined, the argument is ignored. Default 'ib'.

  -H [ --hosts-file ] arg   Shared file used by deamons to register their

                            enpoints. (default './gkfs_hosts.txt')

  -P [ --rpc_protocol ] arg Used RPC protocol for inter-node communication.

                            Available: {ofi+sockets, ofi+verbs, ofi+psm2} for

                            (TCP, Infiniband, and Omni-Path, respectively.

                            (Default ofi+sockets)

                            Libfabric must have verbs or psm2 support enabled.

  --auto_sm                 Enables intra-node communication (IPCs) via the

                            `na+sm` (shared memory) protocol, instead of using

                            the RPC protocol. (Default off)

  --version                 print version and exit

```

The scripts are located in `scripts/{startup_gkfs.py, shutdown_gkfs.py}`. Use the -h argument for their usage.

Shut it down by gracefully killing the process (SIGTERM).

## Miscellaneous

Metadata and actual data will be stored at the `<fs_data_path>`. The path where the application works on is set with

`<pseudo_mount_dir_path>`

`<pseudo_mount_dir_path>`.

Run the application with the preload library: `LD_PRELOAD=<path>/build/lib/libgkfs_intercept.so ./application`. In the case of

an MPI application use the `{mpirun, mpiexec} -x` argument.