Loading CHANGELOG.md +16 −1 Original line number Diff line number Diff line Loading @@ -5,7 +5,20 @@ All notable changes to GekkoFS project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Unreleased ## [Unreleased] ### New - directory optimization with compression and reattemp ([!270](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/270)) - Refactor sfind so it can use SLURM_ environment variables to ask to different servers. - Create a sample bash script to gather all the info (map->reduce) - Compress directory data with zstd. - Make a new config.hpp option for controlling the compression - If directory buffer is not enough it will reattempt with the exact size ### Changed ### Fixed ## [0.9.5] - 2025-08 ### New - Added cppcheck code checking capabilities ([!214](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/214)) - Tests to cover proxy and (malleability) ([!222](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/222)) Loading Loading @@ -52,6 +65,8 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - execve set reentrant_guard as true, so once it returns (vfork) the parent had syscall disabled. - debug should be still set to debug level on syscall for vfork. - In MN5 gkfs_libc.hpp needs to have the libc signatures. - Thread local storage produces some seg fault in Mogon-2 with syscall ([!266](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/266)) - using pthread functions ## [0.9.4] - 2025-03 Loading CMake/FindZStd.cmake +38 −21 Original line number Diff line number Diff line Loading @@ -26,32 +26,49 @@ # SPDX-License-Identifier: GPL-3.0-or-later # ################################################################################ # # - Try to find Facebook zstd library # This will define # ZStd_FOUND # ZStd_INCLUDE_DIR # ZStd_LIBRARIES # # Standard names to search for set(ZStd_NAMES zstd zstd_static) find_path(ZStd_INCLUDE_DIR NAMES zstd.h ) PATH_SUFFIXES include) find_library(ZStd_LIBRARY NAMES zstd ) # Allow ZStd_LIBRARY to be set manually, as the location of the zstd library if(NOT ZStd_LIBRARY) find_library(ZStd_LIBRARY_RELEASE NAMES ${ZStd_NAMES} PATH_SUFFIXES lib) set(ZStd_LIBRARIES ${ZStd_LIBRARY}) set(ZStd_INCLUDE_DIRS ${ZStd_INCLUDE_DIR}) include(SelectLibraryConfigurations) select_library_configurations(ZStd) endif() unset(ZStd_NAMES) mark_as_advanced(ZStd_INCLUDE_DIR) include(FindPackageHandleStandardArgs) FIND_PACKAGE_HANDLE_STANDARD_ARGS(ZStd REQUIRED_VARS ZStd_LIBRARY ZStd_INCLUDE_DIR VERSION_VAR ZStd_VERSION_STRING) if(ZStd_FOUND) set(ZStd_INCLUDE_DIRS ${ZStd_INCLUDE_DIR}) if(NOT ZStd_LIBRARIES) set(ZStd_LIBRARIES ${ZStd_LIBRARY}) endif() if(NOT TARGET ZStd::ZStd) add_library(ZStd::ZStd UNKNOWN IMPORTED) set_target_properties(ZStd::ZStd PROPERTIES INTERFACE_INCLUDE_DIRECTORIES "${ZStd_INCLUDE_DIRS}") find_package_handle_standard_args(ZStd DEFAULT_MSG ZStd_LIBRARY ZStd_INCLUDE_DIR ) set_target_properties(ZStd::ZStd PROPERTIES IMPORTED_LOCATION "${ZStd_LIBRARY}") mark_as_advanced( ZStd_LIBRARY ZStd_INCLUDE_DIR ) No newline at end of file endif() endif() CMakeLists.txt +3 −0 Original line number Diff line number Diff line Loading @@ -161,6 +161,9 @@ find_package(Margo 0.14.0 REQUIRED) message(STATUS "[${PROJECT_NAME}] Checking for syscall_intercept") find_package(Syscall_intercept REQUIRED) message(STATUS "[${PROJECT_NAME}] Checking for Zstd") find_package(ZStd REQUIRED) ### AGIOS: required for scheduling I/O requests if (GKFS_ENABLE_AGIOS) message(STATUS "[${PROJECT_NAME}] Checking for Agios") Loading README.md +3 −0 Original line number Diff line number Diff line Loading @@ -587,6 +587,9 @@ Client-metrics require the CMake argument `-DGKFS_ENABLE_CLIENT_METRICS=ON` (see - `LIBGKFS_METRICS_IP_PORT` - Enable flushing to a set ZeroMQ server (replaces `LIBGKFS_METRICS_PATH`). - `LIBGKFS_PROXY_PID_FILE` - Path to the proxy pid file (when using the GekkoFS proxy). - `LIBGKFS_NUM_REPL` - Number of replicas for data. #### Directory optimizations Set `true` the variable `use_dirents_compression` available at `include/config.hpp` to transfer directories compressed with zstd. #### Caching ##### Dentry cache Improves performance for `ls -l` type operations by caching file metadata for subsequent `stat()` operations during Loading docs/sphinx/users/building.rst +9 −9 Original line number Diff line number Diff line Loading @@ -75,24 +75,24 @@ order to function properly. We list them here for informational purposes. the :ref:`step-by-step installation<step_by_step_installation>` section before attempting a manual install. - `RocksDB <https://github.com/facebook/rocksdb/>`_ version 6.2.2 or newer and its used compression library: - `RocksDB <https://github.com/facebook/rocksdb/>`_ version 10.4.2 or newer and its used compression library: - `lz4 <https://github.com/lz4/lz4>`_ version 1.8.0 or newer. - `lz4 <https://github.com/lz4/lz4>`_ version 1.9.4 or newer. - `Margo <https://github.com/mochi-hpc/mochi-margo/releases>`_ version 0.6.3 and its dependencies: - `Margo <https://github.com/mochi-hpc/mochi-margo/releases>`_ version 0.18.3 and its dependencies: - `Argobots <https://github.com/pmodels/argobots/releases/tag/v1.0.1>`_ version 1.0rc1. - `Mercury <https://github.com/mercury-hpc/mercury/releases/tag/v2.0.0>`_ version 2.0.0. - `Argobots <https://github.com/pmodels/argobots/releases/tag/v1.0.1>`_ version 1.2 - `Mercury <https://github.com/mercury-hpc/mercury/releases/tag/v2.0.0>`_ version 2.4.1rc1 - `libfabric <https://github.com/ofiwg/libfabric>`_ version 1.9 or newer. - `libfabric <https://github.com/ofiwg/libfabric>`_ version 2.2 or newer. - `syscall_intercept <https://github.com/pmem/syscall_intercept>`_ (commit f7cebb7 or newer) and its dependencies: - `syscall_intercept <https://github.com/pmem/syscall_intercept>`_ (commit d8b2a69 or newer) and its dependencies: - `capstone <https://www.capstone-engine.org/>`_ version 6.0.0 or newer. - `capstone <https://www.capstone-engine.org/>`_ version 4.0.1 or newer. - `Date <https://github.com/HowardHinnant/date>`_ (commit e7e1482 or newer). .. important:: Loading Loading
CHANGELOG.md +16 −1 Original line number Diff line number Diff line Loading @@ -5,7 +5,20 @@ All notable changes to GekkoFS project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Unreleased ## [Unreleased] ### New - directory optimization with compression and reattemp ([!270](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/270)) - Refactor sfind so it can use SLURM_ environment variables to ask to different servers. - Create a sample bash script to gather all the info (map->reduce) - Compress directory data with zstd. - Make a new config.hpp option for controlling the compression - If directory buffer is not enough it will reattempt with the exact size ### Changed ### Fixed ## [0.9.5] - 2025-08 ### New - Added cppcheck code checking capabilities ([!214](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/214)) - Tests to cover proxy and (malleability) ([!222](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/222)) Loading Loading @@ -52,6 +65,8 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - execve set reentrant_guard as true, so once it returns (vfork) the parent had syscall disabled. - debug should be still set to debug level on syscall for vfork. - In MN5 gkfs_libc.hpp needs to have the libc signatures. - Thread local storage produces some seg fault in Mogon-2 with syscall ([!266](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/266)) - using pthread functions ## [0.9.4] - 2025-03 Loading
CMake/FindZStd.cmake +38 −21 Original line number Diff line number Diff line Loading @@ -26,32 +26,49 @@ # SPDX-License-Identifier: GPL-3.0-or-later # ################################################################################ # # - Try to find Facebook zstd library # This will define # ZStd_FOUND # ZStd_INCLUDE_DIR # ZStd_LIBRARIES # # Standard names to search for set(ZStd_NAMES zstd zstd_static) find_path(ZStd_INCLUDE_DIR NAMES zstd.h ) PATH_SUFFIXES include) find_library(ZStd_LIBRARY NAMES zstd ) # Allow ZStd_LIBRARY to be set manually, as the location of the zstd library if(NOT ZStd_LIBRARY) find_library(ZStd_LIBRARY_RELEASE NAMES ${ZStd_NAMES} PATH_SUFFIXES lib) set(ZStd_LIBRARIES ${ZStd_LIBRARY}) set(ZStd_INCLUDE_DIRS ${ZStd_INCLUDE_DIR}) include(SelectLibraryConfigurations) select_library_configurations(ZStd) endif() unset(ZStd_NAMES) mark_as_advanced(ZStd_INCLUDE_DIR) include(FindPackageHandleStandardArgs) FIND_PACKAGE_HANDLE_STANDARD_ARGS(ZStd REQUIRED_VARS ZStd_LIBRARY ZStd_INCLUDE_DIR VERSION_VAR ZStd_VERSION_STRING) if(ZStd_FOUND) set(ZStd_INCLUDE_DIRS ${ZStd_INCLUDE_DIR}) if(NOT ZStd_LIBRARIES) set(ZStd_LIBRARIES ${ZStd_LIBRARY}) endif() if(NOT TARGET ZStd::ZStd) add_library(ZStd::ZStd UNKNOWN IMPORTED) set_target_properties(ZStd::ZStd PROPERTIES INTERFACE_INCLUDE_DIRECTORIES "${ZStd_INCLUDE_DIRS}") find_package_handle_standard_args(ZStd DEFAULT_MSG ZStd_LIBRARY ZStd_INCLUDE_DIR ) set_target_properties(ZStd::ZStd PROPERTIES IMPORTED_LOCATION "${ZStd_LIBRARY}") mark_as_advanced( ZStd_LIBRARY ZStd_INCLUDE_DIR ) No newline at end of file endif() endif()
CMakeLists.txt +3 −0 Original line number Diff line number Diff line Loading @@ -161,6 +161,9 @@ find_package(Margo 0.14.0 REQUIRED) message(STATUS "[${PROJECT_NAME}] Checking for syscall_intercept") find_package(Syscall_intercept REQUIRED) message(STATUS "[${PROJECT_NAME}] Checking for Zstd") find_package(ZStd REQUIRED) ### AGIOS: required for scheduling I/O requests if (GKFS_ENABLE_AGIOS) message(STATUS "[${PROJECT_NAME}] Checking for Agios") Loading
README.md +3 −0 Original line number Diff line number Diff line Loading @@ -587,6 +587,9 @@ Client-metrics require the CMake argument `-DGKFS_ENABLE_CLIENT_METRICS=ON` (see - `LIBGKFS_METRICS_IP_PORT` - Enable flushing to a set ZeroMQ server (replaces `LIBGKFS_METRICS_PATH`). - `LIBGKFS_PROXY_PID_FILE` - Path to the proxy pid file (when using the GekkoFS proxy). - `LIBGKFS_NUM_REPL` - Number of replicas for data. #### Directory optimizations Set `true` the variable `use_dirents_compression` available at `include/config.hpp` to transfer directories compressed with zstd. #### Caching ##### Dentry cache Improves performance for `ls -l` type operations by caching file metadata for subsequent `stat()` operations during Loading
docs/sphinx/users/building.rst +9 −9 Original line number Diff line number Diff line Loading @@ -75,24 +75,24 @@ order to function properly. We list them here for informational purposes. the :ref:`step-by-step installation<step_by_step_installation>` section before attempting a manual install. - `RocksDB <https://github.com/facebook/rocksdb/>`_ version 6.2.2 or newer and its used compression library: - `RocksDB <https://github.com/facebook/rocksdb/>`_ version 10.4.2 or newer and its used compression library: - `lz4 <https://github.com/lz4/lz4>`_ version 1.8.0 or newer. - `lz4 <https://github.com/lz4/lz4>`_ version 1.9.4 or newer. - `Margo <https://github.com/mochi-hpc/mochi-margo/releases>`_ version 0.6.3 and its dependencies: - `Margo <https://github.com/mochi-hpc/mochi-margo/releases>`_ version 0.18.3 and its dependencies: - `Argobots <https://github.com/pmodels/argobots/releases/tag/v1.0.1>`_ version 1.0rc1. - `Mercury <https://github.com/mercury-hpc/mercury/releases/tag/v2.0.0>`_ version 2.0.0. - `Argobots <https://github.com/pmodels/argobots/releases/tag/v1.0.1>`_ version 1.2 - `Mercury <https://github.com/mercury-hpc/mercury/releases/tag/v2.0.0>`_ version 2.4.1rc1 - `libfabric <https://github.com/ofiwg/libfabric>`_ version 1.9 or newer. - `libfabric <https://github.com/ofiwg/libfabric>`_ version 2.2 or newer. - `syscall_intercept <https://github.com/pmem/syscall_intercept>`_ (commit f7cebb7 or newer) and its dependencies: - `syscall_intercept <https://github.com/pmem/syscall_intercept>`_ (commit d8b2a69 or newer) and its dependencies: - `capstone <https://www.capstone-engine.org/>`_ version 6.0.0 or newer. - `capstone <https://www.capstone-engine.org/>`_ version 4.0.1 or newer. - `Date <https://github.com/HowardHinnant/date>`_ (commit e7e1482 or newer). .. important:: Loading