diff --git a/CHANGELOG.md b/CHANGELOG.md index 14267ee3c36bcebebe6b54cc38a1faf5377eaa89..ead8b557ee8af8969af4b3fc14388125c920be79 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,13 +6,80 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -- Added dirents_extended function and a find substitution proposal. -- Created a Guided Distributor using a mapping file to map chunks to specific - nodes. +## [0.9.0] - 2022-02-22 +### New +- GekkoFS now uses C++17 (!74). +- Added a new `dirents_extended` function which can improve `find` operations. +A corresponding example how to use this function can be found at +`examples/gfind/gfind.cpp` with a non-mpi version at `examples/gfind/sfind.cpp` +([!73](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/73)). +- Code coverage reports for the source code are now generated and tracked +([!77](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/77)). +- Considerable overhaul and new features of the GekkoFS testing facilities +([!120](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/120), +[!121](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/121)). +- Namespace have been added to the complete GekkoFS codebase +([!82](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/82)). +- The system call `socketcall()` is now supported +([!91](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/91)). +- System call error codes are now checked in `syscall_no_intercept` +scenarios in non x86 architectures (![!92](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/92)). +- GekkoFS documentation is now automatically generated and published +at [here](https://storage.bsc.es/projects/gekkofs/documentation/index.html) +([!95](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/95), +[!109](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/109), +[!125](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/125)). +- Added a guided distributor mode which allows defining a specific distribution of data +on a per directory or file basis ([!39](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/39)). +- For developers: + - A convenience library has been added for unit testing + ([!94](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/94)). + - Code format is now enforced with the `clang-format` tool + ([!66](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/66)). + A new script is available in `scripts/check_format.sh` for easy of use. + - `GKFS_METADATA_MOD` macro has been added allowing the MetadataModule to be + logged, among others ([!98](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/98)). + - A convenience library has been added for `path_util` + ([!102](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/102)). +### Changed +- GekkoFS license has been changed to GNU General Public License version 3 +([!88](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/88)) +- Create, stat, and remove operation have been refactored and improved, reducing +the number of required RPCs per operation ([!60](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/60)). +- Syscall_intercept now supports glibc version 2.3 or newer +([!72](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/72)). +- All arithmetic operations based on block sizes, and therefore chunk computations, +are now `constexpr` ([!75](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/75)). +- The CI pipeline has been significantly optimized +([!103](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/103)). +- The GekkoFS dependency download and compile scripts have been severely refactored +and improved ([!111](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/111)). +- GekkoFS now supports the latest dependency versions +([!112](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/112)). + +### Removed +- Boost is no longer used for the client and daemon ([!90](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/90), +[!122](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/122), +[!123](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/123)). +Note that tests still require `Boost_preprocessor`. +- Unneeded sources in CMake have been removed ([!101](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/101)). + +### Fixed +- Building tests no longer proceeds if virtualenv creation fails ([!68](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/68)). +- An error where unit tests could not be found has been fixed ([!79](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/79)). +- The daemon can now be restarted without losing its namespace ([!85](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/85)). +- An issue has been resolved that required AGIOS even if it wasn't been used ([!104](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/104)). +- Several issues that caused docker images to fail has been resolved ([!105](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/105), +[!106](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/106), [!107](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/107), +[!114](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/114)). +- An CMake issue in `path_util` that caused the compilation to fail was fixed ([!115](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/115)). +- Fixed an issue where `ls` failed because newer kernels use `fstatat()` with `EMPTY_PATH` +([!116](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/116)). +- Fixed an issue where `LOG_OUTPUT_TRUNC` did not work as expected ([!118](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/118)). ## [0.8.0] - 2020-09-15 -## New +### New - Both client library and daemon have been extended to support the ofi+verbs protocol. - A new Python testing harness has been implemented to support integration @@ -38,13 +105,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 daemons, the requests are scheduled using the AGIOS scheduling library before they are dispatched to the shared backend parallel file system. - The `fsync()` system call is now fully supported. -## Improved +### Improved - Argobots tasks in the daemon are now wrapped in a dedicated class, effectively removing the dependency. This lays ground work for future non-Argobots I/O implementations. - The `readdir()` implementation has been refactored and improved. - Improvements on how to the installation scripts manage dependencies. -## Fixed +### Fixed - The server sometimes crashed due to uncaught system errors in the storage backend. This has now been fixed. - Fixed a bug that broke `ls` on some architectures. @@ -52,9 +119,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 client applications via `errno` propagation. ## [0.7.0] - 2020-02-05 -## Added - - Adedd support for `eventfd()`and `eventfd2()` system calls. -## Changed +### Added + - Added support for `eventfd()`and `eventfd2()` system calls. +### Changed - Replaced Margo with Mercury in the client library in order to increase application compatibility: the Argobots ULTs used by Margo to send and process RPCs clashed at times with applications using pthreads. @@ -74,7 +141,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 can be used to further control messages. Run the client with `LIBGKFS_LOG=help` for more details. - Improved dependency management in CMake. -## Fixed +### Fixed - Relocate internal file descriptors to a private range to avoid interfering with client application file descriptors. - Handle internal file descriptors created by `fcntl()`. @@ -82,52 +149,52 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `recvmsg()`. ## [0.6.2] - 2019-10-07 -## Added +### Added - Paths inside kernel pseudo filesystems (`/sys`, `/proc`) are forwarded directly to the kernel and internal path resolution will be skipped. Be aware that also paths like `/sys/../tmp/gkfs_mountpoint/asd` will be forwarded to the kernel - Added new Cmake flag `CREATE_CHECK_PARENTS` to controls if the existance of the parent node needs to be checked during the creation of a child node. -## Changed +### Changed - Daemon logs for RPC handlers have been polished - Updated Margo, Mercury and Libfabric dependencies -## Fixed +### Fixed - mk_node RPC wasn't propagating errors correctly from daemons - README has been improoved and got some minor fixes - fix wrong path in log call for mk_symlink function ## [0.6.1] - 2019-09-17 -## Added +### Added - Added new Cmake flag `LOG_SYSCALLS` to enable/disable syscall logging. - Intercept the 64 bit version of `getdents`. - Added debian-based docker image. -## Changed +### Changed - Disable syscalls logging by default - Update Mercury, RocksDB and Libfabric dependencies -## Fixed +### Fixed - Fix read at the end of file. - Don't create log file when using `--version`/`--help` cli flags. - On some systems LD_PRELOAD used on /bin/bash binary was not working. - Missing definition of `loff_t` on new version of GCC. ## [0.6.0] - 2019-07-26 -## Added +### Added - Add compile time option to disable shared memory communication `-DUSE_SHM:BOOL=OFF` -## Changed +### Changed - Deamons does not store anymore information about the others deamons. - Improoved error handling on deamon initialization - Decreased RPC timeout 3min -> 3sec - Update 3rd party dependencies -## Removed +### Removed - PID file is not used anymore, we use only the new `hosts file` for out of bound communication - Dropped CCI plugin support - Dropped hostname-suffix cli option - Dropped port cli option (use `--listen` instead) - It is not needed anymore to pass hosts information to deamons, thus the `--hosts` cli have been removed -## Fixed +### Fixed - Errors on get_dirents RPC are now reported back to clients - Write errors happenig on deamons are now reported back to clients - number overflow on lseek didn't allow to use seek on huge files ## [0.5.0] - 2019-04-29 -## Changed +### Changed - Intercept I/O syscalls instead of GlibC function using [syscall intercept library](https://github.com/pmem/syscall_intercept) ## [0.4.0] - 2019-04-18 diff --git a/CMakeLists.txt b/CMakeLists.txt index 4efd6a898f8ced8fc48351a8eed2e9e34c6606dd..2f7b9ca6df7c3a7db783fd1ce183b0e1e0fb7840 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -30,7 +30,7 @@ cmake_minimum_required(VERSION 3.6) project( GekkoFS - VERSION 0.8.0 + VERSION 0.9.0 ) enable_testing() diff --git a/docs/sphinx/conf.py.in b/docs/sphinx/conf.py.in index 520dc80eddf6768b4147c47e10c6ce50eec2a201..ab881851b385cef75fd71877a715e96da03df70d 100644 --- a/docs/sphinx/conf.py.in +++ b/docs/sphinx/conf.py.in @@ -22,10 +22,10 @@ copyright = ['2018-2022, Barcelona Supercomputing Center, Spain', '2015-2022, Jo author = 'GekkoFS committers' # The short X.Y version -version = '0.8.0' +version = '0.9.0' # The full version, including alpha/beta/rc tags -release = '0.8.0-snapshot+96-g9cafaaa3-dirty' +release = '0.9.0-snapshot+96-g9cafaaa3-dirty' # Tell sphinx what the primary language being documented is. primary_domain = 'cpp' diff --git a/docs/sphinx/users/scripts.rst b/docs/sphinx/users/scripts.rst index 5b196d3d5a5300a81d2106a9f575f57b70b9af95..e607bc3bc83403a4cd10fdfa9d87e1caa21e015b 100644 --- a/docs/sphinx/users/scripts.rst +++ b/docs/sphinx/users/scripts.rst @@ -24,35 +24,27 @@ on the specifics of the particular GekkoFS build, both scripts rely on :code:`configuration profiles` which define a set of related software packages which should be downloaded and installed for a specific GekkoFS version and/or configuration. To illustrate this, let's take a look at the -contents of the :code:`default` profile for GekkoFS version :code:`0.8.0`: +contents of the :code:`default` profile for GekkoFS version :code:`0.9.0`: .. code-block:: console - $ dl_dep.sh -l default:0.8.0 - Configuration profiles for 'default:0.8.0': + $ dl_dep.sh -l default:0.9.0 + Configuration profiles for '0.9.0': - * default:0.8.0 (/home/user/gekkofs/source/scripts/profiles/0.8.0/default.specs) + * default:0.9.0 (/home/user/gekkofs/source/scripts/profiles/0.9.0/default.specs) All dependencies - bzip2: 1.0.6 - zstd: 1.3.2 - lz4: 1.8.0 - snappy: 1.1.7 - capstone: 4.0.1 - bmi: 6ea0b78fce1b964e45102828cdd05df7040a94c8 - libfabric: HEAD@v1.8.1 - libfabric%experimental: HEAD@v1.9.1 - libfabric%verbs: HEAD@v1.7.2 - mercury: 41caa143a07ed179a3149cac4af0dc7aa3f946fd - argobots: 1.0rc1 - margo: v0.6.3 - rocksdb: 6.2.2 - rocksdb%experimental: 6.11.4 - syscall_intercept: f7cebb7b7e7512a19b78a31ce236ad6ca22636dd + lz4: 1.9.3 + capstone: 4.0.2 + json-c: 0.15-20200726 + libfabric: HEAD@v1.13.2 + mercury: v2.1.0 + argobots: 1.1 + margo: v0.9.6 + rocksdb: 6.26.1 + syscall_intercept: 2c8765fa292bc9c28a22624c528580d54658813d date: e7e1482087f58913b80a20b04d5c58d9d6d90155 - psm2: 11.2.86 - agios: c26a6544200f823ebb8f890dd94e653d148bf226@development @@ -67,38 +59,30 @@ supercomputer) followed by an optional :code:`VERSION_TAG`. .. code-block:: console - $ ./dl_dep.sh -p default:0.8.0 /home/user/gfks/deps - Destination path is set to "/tmp/foo" - Profile name: default - Profile version: 0.8.0 - ------------------------------------ - Cloned 'https://github.com/francielizanon/agios.git' to 'agios' with commit '[c26a6544200f823ebb8f890dd94e653d148bf226]' and flags '--branch=development' - Downloaded 'https://github.com/pmodels/argobots/archive/v1.0rc1.tar.gz' to 'argobots' - Downloaded 'https://github.com/google/snappy/archive/1.1.7.tar.gz' to 'snappy' - Cloned 'https://github.com/radix-io/bmi/' to 'bmi' with commit '[6ea0b78fce1b964e45102828cdd05df7040a94c8]' and flags '' - Downloaded 'https://github.com/lz4/lz4/archive/v1.8.0.tar.gz' to 'lz4' - Cloned 'https://github.com/pmem/syscall_intercept.git' to 'syscall_intercept' with commit '[f7cebb7b7e7512a19b78a31ce236ad6ca22636dd]' and flags '' - Checking patch include/libsyscall_intercept_hook_point.h... - Checking patch src/intercept.c... - Hunk #2 succeeded at 700 (offset 31 lines). - Hunk #3 succeeded at 730 (offset 31 lines). - Checking patch test/test_clone_thread_preload.c... - Applied patch include/libsyscall_intercept_hook_point.h cleanly. - Applied patch src/intercept.c cleanly. - Applied patch test/test_clone_thread_preload.c cleanly. - Downloaded 'https://github.com/facebook/zstd/archive/v1.3.2.tar.gz' to 'zstd' - Downloaded 'https://github.com/intel/opa-psm2/archive/PSM2_11.2.86.tar.gz' to 'psm2' - Downloaded 'https://github.com/aquynh/capstone/archive/4.0.1.tar.gz' to 'capstone' - Cloned 'https://github.com/HowardHinnant/date.git' to 'date' with commit '[e7e1482087f58913b80a20b04d5c58d9d6d90155]' and flags '' - Cloned 'https://xgitlab.cels.anl.gov/sds/margo.git' to 'margo' with commit '[v0.6.3]' and flags '' - Downloaded 'https://github.com/facebook/rocksdb/archive/v6.2.2.tar.gz' to 'rocksdb' - Downloaded 'https://github.com/facebook/rocksdb/archive/v6.11.4.tar.gz' to 'rocksdb%experimental' - Cloned 'https://github.com/mercury-hpc/mercury' to 'mercury' with commit '[41caa143a07ed179a3149cac4af0dc7aa3f946fd]' and flags '--recurse-submodules' - Cloned 'https://github.com/ofiwg/libfabric.git' to 'libfabric%verbs' with commit '[HEAD]' and flags '--branch=v1.7.2' - Cloned 'https://github.com/ofiwg/libfabric.git' to 'libfabric' with commit '[HEAD]' and flags '--branch=v1.8.1' - Cloned 'https://github.com/ofiwg/libfabric.git' to 'libfabric%experimental' with commit '[HEAD]' and flags '--branch=v1.9.1' - Downloaded 'https://sourceforge.net/projects/bzip2/files/bzip2-1.0.6.tar.gz' to 'bzip2' - Done + $ ./dl_dep.sh -p default:0.9.0 /home/user/gfks/deps + Destination path is set to "/tmp/foo" + Profile name: default + Profile version: 0.9.0 + ------------------------------------ + Downloaded 'https://github.com/lz4/lz4/archive/v1.9.3.tar.gz' to 'lz4' + Downloaded 'https://github.com/json-c/json-c/archive/json-c-0.15-20200726.tar.gz' to 'json-c' + Cloned 'https://github.com/pmem/syscall_intercept.git' to 'syscall_intercept' with commit '[2c8765fa292bc9c28a22624c528580d54658813d]' and flags '' + Checking patch include/libsyscall_intercept_hook_point.h... + Checking patch src/intercept.c... + Hunk #2 succeeded at 700 (offset 31 lines). + Hunk #3 succeeded at 730 (offset 31 lines). + Checking patch test/test_clone_thread_preload.c... + Applied patch include/libsyscall_intercept_hook_point.h cleanly. + Applied patch src/intercept.c cleanly. + Applied patch test/test_clone_thread_preload.c cleanly. + Cloned 'https://github.com/HowardHinnant/date.git' to 'date' with commit '[e7e1482087f58913b80a20b04d5c58d9d6d90155]' and flags '' + Downloaded 'https://github.com/pmodels/argobots/archive/v1.1.tar.gz' to 'argobots' + Cloned 'https://github.com/mochi-hpc/mochi-margo' to 'margo' with commit '[v0.9.6]' and flags '' + Downloaded 'https://github.com/aquynh/capstone/archive/4.0.2.tar.gz' to 'capstone' + Downloaded 'https://github.com/facebook/rocksdb/archive/v6.26.1.tar.gz' to 'rocksdb' + Cloned 'https://github.com/ofiwg/libfabric.git' to 'libfabric' with commit '[HEAD]' and flags '--branch=v1.13.2' + Cloned 'https://github.com/mercury-hpc/mercury' to 'mercury' with commit '[v2.1.0]' and flags '--recurse-submodules' + Done It is also possible to download a specific dependency with the :code:`-d` option. In this case, dependency names follow the @@ -106,12 +90,12 @@ option. In this case, dependency names follow the .. code-block:: console - $ ./dl_dep.sh -d mercury@default:0.8.0 /home/user/gfks/deps + $ ./dl_dep.sh -d mercury@default:0.9.0 /home/user/gfks/deps Destination path is set to "/tmp/foo" Profile name: default - Profile version: 0.8.0 + Profile version: 0.9.0 ------------------------------------ - Cloned 'https://github.com/mercury-hpc/mercury' to 'mercury' with commit '[41caa143a07ed179a3149cac4af0dc7aa3f946fd]' and flags '--recurse-submodules' + Cloned 'https://github.com/mercury-hpc/mercury' to 'mercury' with commit '[v2.1.0]' and flags '--recurse-submodules' Done .. warning:: @@ -131,57 +115,41 @@ certain directory (e.g. :code:`/home/user/gkfs/deps`), the .. code-block:: console - $ ./compile_dep.sh -p default:0.8.0 /home/user/gkfs/deps /home/user/gkfs/install -j8 - CORES = 8 (default) - Sources download path = /tmp/foo - Installation path = /tmp/bar - Profile name: default - Profile version: 0.8.0 - ------------------------------------ - + $ ./compile_dep.sh -p default:0.9.0 /home/user/gkfs/deps /home/user/gkfs/install -j8 + CORES = 8 (default) + Sources download path = /tmp/foo + Installation path = /tmp/bar + Profile name: default + Profile version: 0.9.0 + ------------------------------------ + ######## Installing: lz4 ############################### + ... - ######## Installing: bzip2 ############################### - ... + ######## Installing: capstone ############################### + ... - ######## Installing: zstd ############################### - ... + ######## Installing: json-c ############################### + ... - ######## Installing: lz4 ############################### - ... + ######## Installing: libfabric ############################### + ... - ######## Installing: snappy ############################### - ... + ######## Installing: mercury ############################### + ... - ######## Installing: capstone ############################### - ... + ######## Installing: argobots ############################### + ... - ######## Installing: bmi ############################### - ... + ######## Installing: margo ############################### + ... - ######## Installing: libfabric ############################### - ... + ######## Installing: rocksdb ############################### + ... - ######## Installing: mercury ############################### - ... + ######## Installing: syscall_intercept ############################### + ... - ######## Installing: argobots ############################### - ... + ######## Installing: date ############################### + ... - ######## Installing: margo ############################### - ... - - ######## Installing: rocksdb ############################### - ... - - ######## Installing: syscall_intercept ############################### - ... - - ######## Installing: date ############################### - ... - - ######## Installing: psm2 ############################### - ... - - ######## Installing: agios ############################### - ... - Done + Done diff --git a/scripts/compile_dep.sh b/scripts/compile_dep.sh index d430d2d755253a8ddc24eba5e703322ba0c8bc69..fe76c7063710eb3fdfc3c601d551065180cb2854 100755 --- a/scripts/compile_dep.sh +++ b/scripts/compile_dep.sh @@ -80,7 +80,7 @@ optional arguments: deploy specific library versions and/or configurations, using a recognizable name. Optionally, PROFILE_NAME may include a specific version for the profile, e.g. 'mogon2:latest' or - 'ngio:0.8.0', which will download the dependencies defined for + 'ngio:0.9.0', which will download the dependencies defined for that specific version. If unspecified, the 'default:latest' profile will be used, which should include all the possible dependencies. -d, --dependency DEPENDENCY_NAME[[@PROFILE_NAME][:PROFILE_VERSION]] diff --git a/scripts/dl_dep.sh b/scripts/dl_dep.sh index 348ebce632680084c0ac372253afb53124f92435..5cba6d0ca108d2fa335736b789a618ea0f4db3c6 100755 --- a/scripts/dl_dep.sh +++ b/scripts/dl_dep.sh @@ -333,7 +333,7 @@ optional arguments: deploy specific library versions and/or configurations, using a recognizable name. Optionally, PROFILE_NAME may include a specific version for the profile, e.g. 'mogon2:latest' or - 'ngio:0.8.0', which will download the dependencies defined for + 'ngio:0.9.0', which will download the dependencies defined for that specific version. If unspecified, the 'default:latest' profile will be used, which should include all the possible dependencies. -d, --dependency DEPENDENCY_NAME[[@PROFILE_NAME][:PROFILE_VERSION]] diff --git a/scripts/gkfs_dep.sh b/scripts/gkfs_dep.sh index 0ecb639fc618ef6887efa3f20cc38ac85ca458e3..0d0d910e450d27676488777b178929eec45ee750 100755 --- a/scripts/gkfs_dep.sh +++ b/scripts/gkfs_dep.sh @@ -62,7 +62,7 @@ optional arguments: deploy specific library versions and/or configurations, using a recognizable name. Optionally, PROFILE_NAME may include a specific version for the profile, e.g. 'mogon2:latest' or - 'ngio:0.8.0', which will download the dependencies defined for + 'ngio:0.9.0', which will download the dependencies defined for that specific version. If unspecified, the 'default:latest' profile will be used, which should include all the possible dependencies. -d, --dependency DEPENDENCY_NAME[[@PROFILE_NAME][:PROFILE_VERSION]]