Commit ef16432e authored by Marc Vef's avatar Marc Vef
Browse files

Merge branch 'release-0.9.0' into 'master'

Release 0.9.0

See merge request !126
parents 4557e8e5 732bf2bf
Pipeline #2328 passed with stages
in 19 minutes and 35 seconds
......@@ -6,13 +6,80 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
- Added dirents_extended function and a find substitution proposal.
- Created a Guided Distributor using a mapping file to map chunks to specific
nodes.
## [0.9.0] - 2022-02-22
### New
- GekkoFS now uses C++17 (!74).
- Added a new `dirents_extended` function which can improve `find` operations.
A corresponding example how to use this function can be found at
`examples/gfind/gfind.cpp` with a non-mpi version at `examples/gfind/sfind.cpp`
([!73](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/73)).
- Code coverage reports for the source code are now generated and tracked
([!77](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/77)).
- Considerable overhaul and new features of the GekkoFS testing facilities
([!120](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/120),
[!121](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/121)).
- Namespace have been added to the complete GekkoFS codebase
([!82](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/82)).
- The system call `socketcall()` is now supported
([!91](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/91)).
- System call error codes are now checked in `syscall_no_intercept`
scenarios in non x86 architectures (![!92](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/92)).
- GekkoFS documentation is now automatically generated and published
at [here](https://storage.bsc.es/projects/gekkofs/documentation/index.html)
([!95](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/95),
[!109](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/109),
[!125](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/125)).
- Added a guided distributor mode which allows defining a specific distribution of data
on a per directory or file basis ([!39](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/39)).
- For developers:
- A convenience library has been added for unit testing
([!94](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/94)).
- Code format is now enforced with the `clang-format` tool
([!66](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/66)).
A new script is available in `scripts/check_format.sh` for easy of use.
- `GKFS_METADATA_MOD` macro has been added allowing the MetadataModule to be
logged, among others ([!98](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/98)).
- A convenience library has been added for `path_util`
([!102](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/102)).
### Changed
- GekkoFS license has been changed to GNU General Public License version 3
([!88](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/88))
- Create, stat, and remove operation have been refactored and improved, reducing
the number of required RPCs per operation ([!60](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/60)).
- Syscall_intercept now supports glibc version 2.3 or newer
([!72](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/72)).
- All arithmetic operations based on block sizes, and therefore chunk computations,
are now `constexpr` ([!75](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/75)).
- The CI pipeline has been significantly optimized
([!103](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/103)).
- The GekkoFS dependency download and compile scripts have been severely refactored
and improved ([!111](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/111)).
- GekkoFS now supports the latest dependency versions
([!112](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/112)).
### Removed
- Boost is no longer used for the client and daemon ([!90](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/90),
[!122](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/122),
[!123](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/123)).
Note that tests still require `Boost_preprocessor`.
- Unneeded sources in CMake have been removed ([!101](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/101)).
### Fixed
- Building tests no longer proceeds if virtualenv creation fails ([!68](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/68)).
- An error where unit tests could not be found has been fixed ([!79](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/79)).
- The daemon can now be restarted without losing its namespace ([!85](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/85)).
- An issue has been resolved that required AGIOS even if it wasn't been used ([!104](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/104)).
- Several issues that caused docker images to fail has been resolved ([!105](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/105),
[!106](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/106), [!107](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/107),
[!114](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/114)).
- An CMake issue in `path_util` that caused the compilation to fail was fixed ([!115](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/115)).
- Fixed an issue where `ls` failed because newer kernels use `fstatat()` with `EMPTY_PATH`
([!116](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/116)).
- Fixed an issue where `LOG_OUTPUT_TRUNC` did not work as expected ([!118](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/118)).
## [0.8.0] - 2020-09-15
## New
### New
- Both client library and daemon have been extended to support the ofi+verbs
protocol.
- A new Python testing harness has been implemented to support integration
......@@ -38,13 +105,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
daemons, the requests are scheduled using the AGIOS scheduling library before
they are dispatched to the shared backend parallel file system.
- The `fsync()` system call is now fully supported.
## Improved
### Improved
- Argobots tasks in the daemon are now wrapped in a dedicated class,
effectively removing the dependency. This lays ground work for future
non-Argobots I/O implementations.
- The `readdir()` implementation has been refactored and improved.
- Improvements on how to the installation scripts manage dependencies.
## Fixed
### Fixed
- The server sometimes crashed due to uncaught system errors in the storage
backend. This has now been fixed.
- Fixed a bug that broke `ls` on some architectures.
......@@ -52,9 +119,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
client applications via `errno` propagation.
## [0.7.0] - 2020-02-05
## Added
- Adedd support for `eventfd()`and `eventfd2()` system calls.
## Changed
### Added
- Added support for `eventfd()`and `eventfd2()` system calls.
### Changed
- Replaced Margo with Mercury in the client library in order to increase
application compatibility: the Argobots ULTs used by Margo to send and
process RPCs clashed at times with applications using pthreads.
......@@ -74,7 +141,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
can be used to further control messages. Run the client with
`LIBGKFS_LOG=help` for more details.
- Improved dependency management in CMake.
## Fixed
### Fixed
- Relocate internal file descriptors to a private range to avoid interfering
with client application file descriptors.
- Handle internal file descriptors created by `fcntl()`.
......@@ -82,52 +149,52 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`recvmsg()`.
## [0.6.2] - 2019-10-07
## Added
### Added
- Paths inside kernel pseudo filesystems (`/sys`, `/proc`) are forwarded directly to the kernel and internal path resolution will be skipped. Be aware that also paths like `/sys/../tmp/gkfs_mountpoint/asd` will be forwarded to the kernel
- Added new Cmake flag `CREATE_CHECK_PARENTS` to controls if the existance of the parent node needs to be checked during the creation of a child node.
## Changed
### Changed
- Daemon logs for RPC handlers have been polished
- Updated Margo, Mercury and Libfabric dependencies
## Fixed
### Fixed
- mk_node RPC wasn't propagating errors correctly from daemons
- README has been improoved and got some minor fixes
- fix wrong path in log call for mk_symlink function
## [0.6.1] - 2019-09-17
## Added
### Added
- Added new Cmake flag `LOG_SYSCALLS` to enable/disable syscall logging.
- Intercept the 64 bit version of `getdents`.
- Added debian-based docker image.
## Changed
### Changed
- Disable syscalls logging by default
- Update Mercury, RocksDB and Libfabric dependencies
## Fixed
### Fixed
- Fix read at the end of file.
- Don't create log file when using `--version`/`--help` cli flags.
- On some systems LD_PRELOAD used on /bin/bash binary was not working.
- Missing definition of `loff_t` on new version of GCC.
## [0.6.0] - 2019-07-26
## Added
### Added
- Add compile time option to disable shared memory communication `-DUSE_SHM:BOOL=OFF`
## Changed
### Changed
- Deamons does not store anymore information about the others deamons.
- Improoved error handling on deamon initialization
- Decreased RPC timeout 3min -> 3sec
- Update 3rd party dependencies
## Removed
### Removed
- PID file is not used anymore, we use only the new `hosts file` for out of bound communication
- Dropped CCI plugin support
- Dropped hostname-suffix cli option
- Dropped port cli option (use `--listen` instead)
- It is not needed anymore to pass hosts information to deamons, thus the `--hosts` cli have been removed
## Fixed
### Fixed
- Errors on get_dirents RPC are now reported back to clients
- Write errors happenig on deamons are now reported back to clients
- number overflow on lseek didn't allow to use seek on huge files
## [0.5.0] - 2019-04-29
## Changed
### Changed
- Intercept I/O syscalls instead of GlibC function using [syscall intercept library](https://github.com/pmem/syscall_intercept)
## [0.4.0] - 2019-04-18
......
......@@ -30,7 +30,7 @@ cmake_minimum_required(VERSION 3.6)
project(
GekkoFS
VERSION 0.8.0
VERSION 0.9.0
)
enable_testing()
......
......@@ -22,10 +22,10 @@ copyright = ['2018-2022, Barcelona Supercomputing Center, Spain', '2015-2022, Jo
author = 'GekkoFS committers'
# The short X.Y version
version = '0.8.0'
version = '0.9.0'
# The full version, including alpha/beta/rc tags
release = '0.8.0-snapshot+96-g9cafaaa3-dirty'
release = '0.9.0-snapshot+96-g9cafaaa3-dirty'
# Tell sphinx what the primary language being documented is.
primary_domain = 'cpp'
......
......@@ -24,35 +24,27 @@ on the specifics of the particular GekkoFS build, both scripts rely on
:code:`configuration profiles` which define a set of related software
packages which should be downloaded and installed for a specific GekkoFS
version and/or configuration. To illustrate this, let's take a look at the
contents of the :code:`default` profile for GekkoFS version :code:`0.8.0`:
contents of the :code:`default` profile for GekkoFS version :code:`0.9.0`:
.. code-block:: console
$ dl_dep.sh -l default:0.8.0
Configuration profiles for 'default:0.8.0':
$ dl_dep.sh -l default:0.9.0
Configuration profiles for '0.9.0':
* default:0.8.0 (/home/user/gekkofs/source/scripts/profiles/0.8.0/default.specs)
* default:0.9.0 (/home/user/gekkofs/source/scripts/profiles/0.9.0/default.specs)
All dependencies
bzip2: 1.0.6
zstd: 1.3.2
lz4: 1.8.0
snappy: 1.1.7
capstone: 4.0.1
bmi: 6ea0b78fce1b964e45102828cdd05df7040a94c8
libfabric: HEAD@v1.8.1
libfabric%experimental: HEAD@v1.9.1
libfabric%verbs: HEAD@v1.7.2
mercury: 41caa143a07ed179a3149cac4af0dc7aa3f946fd
argobots: 1.0rc1
margo: v0.6.3
rocksdb: 6.2.2
rocksdb%experimental: 6.11.4
syscall_intercept: f7cebb7b7e7512a19b78a31ce236ad6ca22636dd
lz4: 1.9.3
capstone: 4.0.2
json-c: 0.15-20200726
libfabric: HEAD@v1.13.2
mercury: v2.1.0
argobots: 1.1
margo: v0.9.6
rocksdb: 6.26.1
syscall_intercept: 2c8765fa292bc9c28a22624c528580d54658813d
date: e7e1482087f58913b80a20b04d5c58d9d6d90155
psm2: 11.2.86
agios: c26a6544200f823ebb8f890dd94e653d148bf226@development
......@@ -67,38 +59,30 @@ supercomputer) followed by an optional :code:`VERSION_TAG`.
.. code-block:: console
$ ./dl_dep.sh -p default:0.8.0 /home/user/gfks/deps
Destination path is set to "/tmp/foo"
Profile name: default
Profile version: 0.8.0
------------------------------------
Cloned 'https://github.com/francielizanon/agios.git' to 'agios' with commit '[c26a6544200f823ebb8f890dd94e653d148bf226]' and flags '--branch=development'
Downloaded 'https://github.com/pmodels/argobots/archive/v1.0rc1.tar.gz' to 'argobots'
Downloaded 'https://github.com/google/snappy/archive/1.1.7.tar.gz' to 'snappy'
Cloned 'https://github.com/radix-io/bmi/' to 'bmi' with commit '[6ea0b78fce1b964e45102828cdd05df7040a94c8]' and flags ''
Downloaded 'https://github.com/lz4/lz4/archive/v1.8.0.tar.gz' to 'lz4'
Cloned 'https://github.com/pmem/syscall_intercept.git' to 'syscall_intercept' with commit '[f7cebb7b7e7512a19b78a31ce236ad6ca22636dd]' and flags ''
Checking patch include/libsyscall_intercept_hook_point.h...
Checking patch src/intercept.c...
Hunk #2 succeeded at 700 (offset 31 lines).
Hunk #3 succeeded at 730 (offset 31 lines).
Checking patch test/test_clone_thread_preload.c...
Applied patch include/libsyscall_intercept_hook_point.h cleanly.
Applied patch src/intercept.c cleanly.
Applied patch test/test_clone_thread_preload.c cleanly.
Downloaded 'https://github.com/facebook/zstd/archive/v1.3.2.tar.gz' to 'zstd'
Downloaded 'https://github.com/intel/opa-psm2/archive/PSM2_11.2.86.tar.gz' to 'psm2'
Downloaded 'https://github.com/aquynh/capstone/archive/4.0.1.tar.gz' to 'capstone'
Cloned 'https://github.com/HowardHinnant/date.git' to 'date' with commit '[e7e1482087f58913b80a20b04d5c58d9d6d90155]' and flags ''
Cloned 'https://xgitlab.cels.anl.gov/sds/margo.git' to 'margo' with commit '[v0.6.3]' and flags ''
Downloaded 'https://github.com/facebook/rocksdb/archive/v6.2.2.tar.gz' to 'rocksdb'
Downloaded 'https://github.com/facebook/rocksdb/archive/v6.11.4.tar.gz' to 'rocksdb%experimental'
Cloned 'https://github.com/mercury-hpc/mercury' to 'mercury' with commit '[41caa143a07ed179a3149cac4af0dc7aa3f946fd]' and flags '--recurse-submodules'
Cloned 'https://github.com/ofiwg/libfabric.git' to 'libfabric%verbs' with commit '[HEAD]' and flags '--branch=v1.7.2'
Cloned 'https://github.com/ofiwg/libfabric.git' to 'libfabric' with commit '[HEAD]' and flags '--branch=v1.8.1'
Cloned 'https://github.com/ofiwg/libfabric.git' to 'libfabric%experimental' with commit '[HEAD]' and flags '--branch=v1.9.1'
Downloaded 'https://sourceforge.net/projects/bzip2/files/bzip2-1.0.6.tar.gz' to 'bzip2'
Done
$ ./dl_dep.sh -p default:0.9.0 /home/user/gfks/deps
Destination path is set to "/tmp/foo"
Profile name: default
Profile version: 0.9.0
------------------------------------
Downloaded 'https://github.com/lz4/lz4/archive/v1.9.3.tar.gz' to 'lz4'
Downloaded 'https://github.com/json-c/json-c/archive/json-c-0.15-20200726.tar.gz' to 'json-c'
Cloned 'https://github.com/pmem/syscall_intercept.git' to 'syscall_intercept' with commit '[2c8765fa292bc9c28a22624c528580d54658813d]' and flags ''
Checking patch include/libsyscall_intercept_hook_point.h...
Checking patch src/intercept.c...
Hunk #2 succeeded at 700 (offset 31 lines).
Hunk #3 succeeded at 730 (offset 31 lines).
Checking patch test/test_clone_thread_preload.c...
Applied patch include/libsyscall_intercept_hook_point.h cleanly.
Applied patch src/intercept.c cleanly.
Applied patch test/test_clone_thread_preload.c cleanly.
Cloned 'https://github.com/HowardHinnant/date.git' to 'date' with commit '[e7e1482087f58913b80a20b04d5c58d9d6d90155]' and flags ''
Downloaded 'https://github.com/pmodels/argobots/archive/v1.1.tar.gz' to 'argobots'
Cloned 'https://github.com/mochi-hpc/mochi-margo' to 'margo' with commit '[v0.9.6]' and flags ''
Downloaded 'https://github.com/aquynh/capstone/archive/4.0.2.tar.gz' to 'capstone'
Downloaded 'https://github.com/facebook/rocksdb/archive/v6.26.1.tar.gz' to 'rocksdb'
Cloned 'https://github.com/ofiwg/libfabric.git' to 'libfabric' with commit '[HEAD]' and flags '--branch=v1.13.2'
Cloned 'https://github.com/mercury-hpc/mercury' to 'mercury' with commit '[v2.1.0]' and flags '--recurse-submodules'
Done
It is also possible to download a specific dependency with the :code:`-d`
option. In this case, dependency names follow the
......@@ -106,12 +90,12 @@ option. In this case, dependency names follow the
.. code-block:: console
$ ./dl_dep.sh -d mercury@default:0.8.0 /home/user/gfks/deps
$ ./dl_dep.sh -d mercury@default:0.9.0 /home/user/gfks/deps
Destination path is set to "/tmp/foo"
Profile name: default
Profile version: 0.8.0
Profile version: 0.9.0
------------------------------------
Cloned 'https://github.com/mercury-hpc/mercury' to 'mercury' with commit '[41caa143a07ed179a3149cac4af0dc7aa3f946fd]' and flags '--recurse-submodules'
Cloned 'https://github.com/mercury-hpc/mercury' to 'mercury' with commit '[v2.1.0]' and flags '--recurse-submodules'
Done
.. warning::
......@@ -131,57 +115,41 @@ certain directory (e.g. :code:`/home/user/gkfs/deps`), the
.. code-block:: console
$ ./compile_dep.sh -p default:0.8.0 /home/user/gkfs/deps /home/user/gkfs/install -j8
CORES = 8 (default)
Sources download path = /tmp/foo
Installation path = /tmp/bar
Profile name: default
Profile version: 0.8.0
------------------------------------
$ ./compile_dep.sh -p default:0.9.0 /home/user/gkfs/deps /home/user/gkfs/install -j8
CORES = 8 (default)
Sources download path = /tmp/foo
Installation path = /tmp/bar
Profile name: default
Profile version: 0.9.0
------------------------------------
######## Installing: lz4 ###############################
...
######## Installing: bzip2 ###############################
...
######## Installing: capstone ###############################
...
######## Installing: zstd ###############################
...
######## Installing: json-c ###############################
...
######## Installing: lz4 ###############################
...
######## Installing: libfabric ###############################
...
######## Installing: snappy ###############################
...
######## Installing: mercury ###############################
...
######## Installing: capstone ###############################
...
######## Installing: argobots ###############################
...
######## Installing: bmi ###############################
...
######## Installing: margo ###############################
...
######## Installing: libfabric ###############################
...
######## Installing: rocksdb ###############################
...
######## Installing: mercury ###############################
...
######## Installing: syscall_intercept ###############################
...
######## Installing: argobots ###############################
...
######## Installing: date ###############################
...
######## Installing: margo ###############################
...
######## Installing: rocksdb ###############################
...
######## Installing: syscall_intercept ###############################
...
######## Installing: date ###############################
...
######## Installing: psm2 ###############################
...
######## Installing: agios ###############################
...
Done
Done
......@@ -80,7 +80,7 @@ optional arguments:
deploy specific library versions and/or configurations,
using a recognizable name. Optionally, PROFILE_NAME may include
a specific version for the profile, e.g. 'mogon2:latest' or
'ngio:0.8.0', which will download the dependencies defined for
'ngio:0.9.0', which will download the dependencies defined for
that specific version. If unspecified, the 'default:latest' profile
will be used, which should include all the possible dependencies.
-d, --dependency DEPENDENCY_NAME[[@PROFILE_NAME][:PROFILE_VERSION]]
......
......@@ -333,7 +333,7 @@ optional arguments:
deploy specific library versions and/or configurations,
using a recognizable name. Optionally, PROFILE_NAME may include
a specific version for the profile, e.g. 'mogon2:latest' or
'ngio:0.8.0', which will download the dependencies defined for
'ngio:0.9.0', which will download the dependencies defined for
that specific version. If unspecified, the 'default:latest' profile
will be used, which should include all the possible dependencies.
-d, --dependency DEPENDENCY_NAME[[@PROFILE_NAME][:PROFILE_VERSION]]
......
......@@ -62,7 +62,7 @@ optional arguments:
deploy specific library versions and/or configurations,
using a recognizable name. Optionally, PROFILE_NAME may include
a specific version for the profile, e.g. 'mogon2:latest' or
'ngio:0.8.0', which will download the dependencies defined for
'ngio:0.9.0', which will download the dependencies defined for
that specific version. If unspecified, the 'default:latest' profile
will be used, which should include all the possible dependencies.
-d, --dependency DEPENDENCY_NAME[[@PROFILE_NAME][:PROFILE_VERSION]]
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment