Merge branch 'release-0.9.0' into 'master'

## [Unreleased]

- Added dirents_extended function and a find substitution proposal.

- Created a Guided Distributor using a mapping file to map chunks to specific

  nodes.

## [0.9.0] - 2022-02-22

### New

- GekkoFS now uses C++17 (!74).

- Added a new `dirents_extended` function which can improve `find` operations.

A corresponding example how to use this function can be found at 

`examples/gfind/gfind.cpp` with a non-mpi version at `examples/gfind/sfind.cpp` 

([!73](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/73)).

- Code coverage reports for the source code are now generated and tracked 

([!77](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/77)).

- Considerable overhaul and new features of the GekkoFS testing facilities 

([!120](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/120), 

[!121](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/121)).

- Namespace have been added to the complete GekkoFS codebase 

([!82](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/82)).

- The system call `socketcall()` is now supported 

([!91](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/91)).

- System call error codes are now checked in `syscall_no_intercept` 

scenarios in non x86 architectures (![!92](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/92)).

- GekkoFS documentation is now automatically generated and published

at [here](https://storage.bsc.es/projects/gekkofs/documentation/index.html) 

([!95](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/95), 

[!109](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/109), 

[!125](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/125)).

- Added a guided distributor mode which allows defining a specific distribution of data 

on a per directory or file basis ([!39](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/39)).

- For developers:

  - A convenience library has been added for unit testing 

  ([!94](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/94)).

  - Code format is now enforced with the `clang-format` tool 

  ([!66](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/66)).

  A new script is available in `scripts/check_format.sh` for easy of use.

  - `GKFS_METADATA_MOD` macro has been added allowing the MetadataModule to be 

  logged, among others ([!98](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/98)).

  - A convenience library has been added for `path_util` 

  ([!102](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/102)).

### Changed

- GekkoFS license has been changed to GNU General Public License version 3 

([!88](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/88))

- Create, stat, and remove operation have been refactored and improved, reducing 

the number of required RPCs per operation ([!60](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/60)).

- Syscall_intercept now supports glibc version 2.3 or newer 

([!72](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/72)).

- All arithmetic operations based on block sizes, and therefore chunk computations,

are now `constexpr` ([!75](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/75)).

- The CI pipeline has been significantly optimized 

([!103](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/103)).

- The GekkoFS dependency download and compile scripts have been severely refactored 

and improved ([!111](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/111)).

- GekkoFS now supports the latest dependency versions 

([!112](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/112)).

### Removed

- Boost is no longer used for the client and daemon ([!90](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/90), 

[!122](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/122), 

[!123](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/123)). 

Note that tests still require `Boost_preprocessor`. 

- Unneeded sources in CMake have been removed ([!101](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/101)).

### Fixed

- Building tests no longer proceeds if virtualenv creation fails ([!68](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/68)).

- An error where unit tests could not be found has been fixed ([!79](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/79)).

- The daemon can now be restarted without losing its namespace ([!85](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/85)).

- An issue has been resolved that required AGIOS even if it wasn't been used ([!104](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/104)).

- Several issues that caused docker images to fail has been resolved ([!105](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/105), 

[!106](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/106), [!107](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/107), 

[!114](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/114)).

- An CMake issue in `path_util` that caused the compilation to fail was fixed ([!115](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/115)). 

- Fixed an issue where `ls` failed because newer kernels use `fstatat()` with `EMPTY_PATH` 

([!116](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/116)).

- Fixed an issue where `LOG_OUTPUT_TRUNC` did not work as expected ([!118](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/118)).

## [0.8.0] - 2020-09-15

## New

### New

- Both client library and daemon have been extended to support the ofi+verbs

  protocol.

- A new Python testing harness has been implemented to support integration

  daemons, the requests are scheduled using the AGIOS scheduling library before

  they are dispatched to the shared backend parallel file system.

- The `fsync()` system call is now fully supported.

## Improved

### Improved

- Argobots tasks in the daemon are now wrapped in a dedicated class,

  effectively removing the dependency. This lays ground work for future

  non-Argobots I/O implementations.

- The `readdir()` implementation has been refactored and improved.

- Improvements on how to the installation scripts manage dependencies.

## Fixed

### Fixed

- The server sometimes crashed due to uncaught system errors in the storage

  backend. This has now been fixed.

- Fixed a bug that broke `ls` on some architectures.

  client applications via `errno` propagation.

## [0.7.0] - 2020-02-05

## Added

 - Adedd support for `eventfd()`and `eventfd2()` system calls.

## Changed

### Added

 - Added support for `eventfd()`and `eventfd2()` system calls.

### Changed

 - Replaced Margo with Mercury in the client library in order to increase

   application compatibility: the Argobots ULTs used by Margo to send and

   process RPCs clashed at times with applications using pthreads.

   can be used to further control messages. Run the client with

   `LIBGKFS_LOG=help` for more details.

 - Improved dependency management in CMake.

## Fixed

### Fixed

 - Relocate internal file descriptors to a private range to avoid interfering

   with client application file descriptors.

 - Handle internal file descriptors created by `fcntl()`.

   `recvmsg()`.

## [0.6.2] - 2019-10-07

## Added

### Added

 - Paths inside kernel pseudo filesystems (`/sys`, `/proc`) are forwarded directly to the kernel and internal path resolution will be skipped. Be aware that also paths  like  `/sys/../tmp/gkfs_mountpoint/asd` will be forwarded to the kernel

 - Added new Cmake flag `CREATE_CHECK_PARENTS` to controls if the existance of the parent node needs to be checked during the creation of a child node.

## Changed

### Changed

 - Daemon logs for RPC handlers have been polished

 - Updated Margo, Mercury and Libfabric dependencies

## Fixed

### Fixed

 - mk_node RPC wasn't propagating errors correctly from daemons

 - README has been improoved and got some minor fixes

 - fix wrong path in log call for mk_symlink function

## [0.6.1] - 2019-09-17

## Added

### Added

 - Added new Cmake flag `LOG_SYSCALLS` to enable/disable syscall logging.

 - Intercept the 64 bit version of `getdents`.

 - Added debian-based docker image.

## Changed

### Changed

 - Disable syscalls logging by default

 - Update Mercury, RocksDB and Libfabric dependencies

## Fixed

### Fixed

 - Fix read at the end of file.

 - Don't create log file when using `--version`/`--help` cli flags.

 - On some systems LD_PRELOAD used on /bin/bash binary was not working.

 - Missing definition of `loff_t` on new version of GCC.

## [0.6.0] - 2019-07-26

## Added

### Added

- Add compile time option to disable shared memory communication `-DUSE_SHM:BOOL=OFF`

## Changed

### Changed

- Deamons does not store anymore information about the others deamons.

- Improoved error handling on deamon initialization

- Decreased RPC timeout 3min -> 3sec

- Update 3rd party dependencies

## Removed

### Removed

- PID file is not used anymore, we use only the new `hosts file` for out of bound communication

- Dropped CCI plugin support

- Dropped hostname-suffix cli option

- Dropped port cli option (use `--listen` instead)

- It is not needed anymore to pass hosts information to deamons, thus the `--hosts` cli have been removed

## Fixed

### Fixed

- Errors on get_dirents RPC are now reported back to clients

- Write errors happenig on deamons are now reported back to clients

- number overflow on lseek didn't allow to use seek on huge files

## [0.5.0] - 2019-04-29

## Changed

### Changed

- Intercept I/O syscalls instead of GlibC function using [syscall intercept library](https://github.com/pmem/syscall_intercept)

## [0.4.0] - 2019-04-18

project(

    GekkoFS

    VERSION 0.8.0

    VERSION 0.9.0

)

enable_testing()

author = 'GekkoFS committers'

# The short X.Y version

version = '0.8.0'

version = '0.9.0'

# The full version, including alpha/beta/rc tags

release = '0.8.0-snapshot+96-g9cafaaa3-dirty'

release = '0.9.0-snapshot+96-g9cafaaa3-dirty'

# Tell sphinx what the primary language being documented is.

primary_domain = 'cpp'

:code:`configuration profiles` which define a set of related software

packages which should be downloaded and installed for a specific GekkoFS

version and/or configuration. To illustrate this, let's take a look at the

contents of the :code:`default` profile for GekkoFS version :code:`0.8.0`:

contents of the :code:`default` profile for GekkoFS version :code:`0.9.0`:

.. code-block:: console

    $ dl_dep.sh -l default:0.8.0

    Configuration profiles for 'default:0.8.0':

    $ dl_dep.sh -l default:0.9.0

    Configuration profiles for '0.9.0':

    * default:0.8.0 (/home/user/gekkofs/source/scripts/profiles/0.8.0/default.specs)

    * default:0.9.0 (/home/user/gekkofs/source/scripts/profiles/0.9.0/default.specs)

      All dependencies

        bzip2: 1.0.6

        zstd: 1.3.2

        lz4: 1.8.0

        snappy: 1.1.7

        capstone: 4.0.1

        bmi: 6ea0b78fce1b964e45102828cdd05df7040a94c8

        libfabric: HEAD@v1.8.1

        libfabric%experimental: HEAD@v1.9.1

        libfabric%verbs: HEAD@v1.7.2

        mercury: 41caa143a07ed179a3149cac4af0dc7aa3f946fd

        argobots: 1.0rc1

        margo: v0.6.3

        rocksdb: 6.2.2

        rocksdb%experimental: 6.11.4

        syscall_intercept: f7cebb7b7e7512a19b78a31ce236ad6ca22636dd

        lz4: 1.9.3

        capstone: 4.0.2

        json-c: 0.15-20200726

        libfabric: HEAD@v1.13.2

        mercury: v2.1.0

        argobots: 1.1

        margo: v0.9.6

        rocksdb: 6.26.1

        syscall_intercept: 2c8765fa292bc9c28a22624c528580d54658813d

        date: e7e1482087f58913b80a20b04d5c58d9d6d90155

        psm2: 11.2.86

        agios: c26a6544200f823ebb8f890dd94e653d148bf226@development

.. code-block:: console

   $ ./dl_dep.sh -p default:0.8.0 /home/user/gfks/deps

    $ ./dl_dep.sh -p default:0.9.0 /home/user/gfks/deps

    Destination path is set to  "/tmp/foo"

    Profile name: default

   Profile version: 0.8.0

    Profile version: 0.9.0

    ------------------------------------

   Cloned 'https://github.com/francielizanon/agios.git' to 'agios' with commit '[c26a6544200f823ebb8f890dd94e653d148bf226]' and flags '--branch=development'

   Downloaded 'https://github.com/pmodels/argobots/archive/v1.0rc1.tar.gz' to 'argobots'

   Downloaded 'https://github.com/google/snappy/archive/1.1.7.tar.gz' to 'snappy'

   Cloned 'https://github.com/radix-io/bmi/' to 'bmi' with commit '[6ea0b78fce1b964e45102828cdd05df7040a94c8]' and flags ''

   Downloaded 'https://github.com/lz4/lz4/archive/v1.8.0.tar.gz' to 'lz4'

   Cloned 'https://github.com/pmem/syscall_intercept.git' to 'syscall_intercept' with commit '[f7cebb7b7e7512a19b78a31ce236ad6ca22636dd]' and flags ''

    Downloaded 'https://github.com/lz4/lz4/archive/v1.9.3.tar.gz' to 'lz4'

    Downloaded 'https://github.com/json-c/json-c/archive/json-c-0.15-20200726.tar.gz' to 'json-c'

    Cloned 'https://github.com/pmem/syscall_intercept.git' to 'syscall_intercept' with commit '[2c8765fa292bc9c28a22624c528580d54658813d]' and flags ''

    Checking patch include/libsyscall_intercept_hook_point.h...

    Checking patch src/intercept.c...

    Hunk #2 succeeded at 700 (offset 31 lines).

    Applied patch include/libsyscall_intercept_hook_point.h cleanly.

    Applied patch src/intercept.c cleanly.

    Applied patch test/test_clone_thread_preload.c cleanly.

   Downloaded 'https://github.com/facebook/zstd/archive/v1.3.2.tar.gz' to 'zstd'

   Downloaded 'https://github.com/intel/opa-psm2/archive/PSM2_11.2.86.tar.gz' to 'psm2'

   Downloaded 'https://github.com/aquynh/capstone/archive/4.0.1.tar.gz' to 'capstone'

    Cloned 'https://github.com/HowardHinnant/date.git' to 'date' with commit '[e7e1482087f58913b80a20b04d5c58d9d6d90155]' and flags ''

   Cloned 'https://xgitlab.cels.anl.gov/sds/margo.git' to 'margo' with commit '[v0.6.3]' and flags ''

   Downloaded 'https://github.com/facebook/rocksdb/archive/v6.2.2.tar.gz' to 'rocksdb'

   Downloaded 'https://github.com/facebook/rocksdb/archive/v6.11.4.tar.gz' to 'rocksdb%experimental'

   Cloned 'https://github.com/mercury-hpc/mercury' to 'mercury' with commit '[41caa143a07ed179a3149cac4af0dc7aa3f946fd]' and flags '--recurse-submodules'

   Cloned 'https://github.com/ofiwg/libfabric.git' to 'libfabric%verbs' with commit '[HEAD]' and flags '--branch=v1.7.2'

   Cloned 'https://github.com/ofiwg/libfabric.git' to 'libfabric' with commit '[HEAD]' and flags '--branch=v1.8.1'

   Cloned 'https://github.com/ofiwg/libfabric.git' to 'libfabric%experimental' with commit '[HEAD]' and flags '--branch=v1.9.1'

   Downloaded 'https://sourceforge.net/projects/bzip2/files/bzip2-1.0.6.tar.gz' to 'bzip2'

    Downloaded 'https://github.com/pmodels/argobots/archive/v1.1.tar.gz' to 'argobots'

    Cloned 'https://github.com/mochi-hpc/mochi-margo' to 'margo' with commit '[v0.9.6]' and flags ''

    Downloaded 'https://github.com/aquynh/capstone/archive/4.0.2.tar.gz' to 'capstone'

    Downloaded 'https://github.com/facebook/rocksdb/archive/v6.26.1.tar.gz' to 'rocksdb'

    Cloned 'https://github.com/ofiwg/libfabric.git' to 'libfabric' with commit '[HEAD]' and flags '--branch=v1.13.2'

    Cloned 'https://github.com/mercury-hpc/mercury' to 'mercury' with commit '[v2.1.0]' and flags '--recurse-submodules'

    Done

It is also possible to download a specific dependency with the :code:`-d`

.. code-block:: console

   $ ./dl_dep.sh -d mercury@default:0.8.0 /home/user/gfks/deps

   $ ./dl_dep.sh -d mercury@default:0.9.0 /home/user/gfks/deps

   Destination path is set to  "/tmp/foo"

   Profile name: default

   Profile version: 0.8.0

   Profile version: 0.9.0

   ------------------------------------

   Cloned 'https://github.com/mercury-hpc/mercury' to 'mercury' with commit '[41caa143a07ed179a3149cac4af0dc7aa3f946fd]' and flags '--recurse-submodules'

   Cloned 'https://github.com/mercury-hpc/mercury' to 'mercury' with commit '[v2.1.0]' and flags '--recurse-submodules'

   Done

.. warning::

.. code-block:: console

   $ ./compile_dep.sh -p default:0.8.0 /home/user/gkfs/deps /home/user/gkfs/install -j8

    $ ./compile_dep.sh -p default:0.9.0 /home/user/gkfs/deps /home/user/gkfs/install -j8

    CORES = 8 (default)

    Sources download path = /tmp/foo

    Installation path = /tmp/bar

    Profile name: default

   Profile version: 0.8.0

    Profile version: 0.9.0

    ------------------------------------

   ######## Installing:  bzip2 ###############################

   ...

   ######## Installing:  zstd ###############################

   ...

    ######## Installing:  lz4 ###############################

    ...

   ######## Installing:  snappy ###############################

   ...

    ######## Installing:  capstone ###############################

    ...

   ######## Installing:  bmi ###############################

    ######## Installing:  json-c ###############################

    ...

    ######## Installing:  libfabric ###############################

    ######## Installing:  date ###############################

    ...

   ######## Installing:  psm2 ###############################

   ...

   ######## Installing:  agios ###############################

   ...

    Done

                deploy specific library versions and/or configurations,

                using a recognizable name. Optionally, PROFILE_NAME may include

                a specific version for the profile, e.g. 'mogon2:latest' or

                'ngio:0.8.0', which will download the dependencies defined for

                'ngio:0.9.0', which will download the dependencies defined for

                that specific version. If unspecified, the 'default:latest' profile

                will be used, which should include all the possible dependencies.

    -d, --dependency DEPENDENCY_NAME[[@PROFILE_NAME][:PROFILE_VERSION]]