Commit 33da5fa5 authored by Ramon Nou's avatar Ramon Nou
Browse files

Infrastructure: Update build system, dependencies and CI 

- Updated CMakeLists.txt and requirements\n- Updated CI configurations\n- Updated common definitions
parent 09befe84

CHANGELOG.md

+3 −0
Original line number Diff line number Diff line
@@ -13,6 +13,9 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 
    - Compress directory data with zstd.

    - Make a new config.hpp option for controlling the compression

    - If directory buffer is not enough it will reattempt with the exact size

  - Metadata server can store small data ([!271](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/271))

    - Using config.hpp use_inline_data = true; and inline_data_size = 4096;

    - Data is stored in base64, as we use string to send the small data content (not bulk transfer)



### Changed

CMakeLists.txt

+4 −2
Original line number Diff line number Diff line
@@ -154,8 +154,10 @@ message(STATUS "[${PROJECT_NAME}] Checking for Argobots") 
find_package(Argobots 1.1 REQUIRED)



### Margo

message(STATUS "[${PROJECT_NAME}] Checking for Margo")

find_package(Margo 0.14.0 REQUIRED)

# message(STATUS "[${PROJECT_NAME}] Checking for Margo")

# find_package(Margo 0.14.0 REQUIRED)

message(STATUS "[${PROJECT_NAME}] Checking for Thallium")

find_package(Thallium REQUIRED)



### syscall-intercept

message(STATUS "[${PROJECT_NAME}] Checking for syscall_intercept")

README.md

+14 −2
Original line number Diff line number Diff line
@@ -510,6 +510,10 @@ Note, that a chunk/host configuration is inherited to all children files automat 
In this example, `/mdt-hard/file1` is therefore also using the same distribution as the `/mdt-hard` directory.

If no prefix is used, the Simple Hash distributor is used.



## Small Data Store

Small files can be stored using the metadata server, this is controlled with the `config.hpp` options:

`use_inline_data = true` and `inline_data_size`



#### Guided configuration file



Creating a guided configuration file is based on an I/O trace file of a previous execution of the application.
@@ -587,8 +591,11 @@ Client-metrics require the CMake argument `-DGKFS_ENABLE_CLIENT_METRICS=ON` (see 
- `LIBGKFS_METRICS_IP_PORT` - Enable flushing to a set ZeroMQ server (replaces `LIBGKFS_METRICS_PATH`).

- `LIBGKFS_PROXY_PID_FILE` - Path to the proxy pid file (when using the GekkoFS proxy).

- `LIBGKFS_NUM_REPL` - Number of replicas for data.

#### Directory optimizations

Set `true` the variable `use_dirents_compression` available at `include/config.hpp` to transfer directories compressed with zstd.

#### Optimization

- `LIBGKFS_USE_INLINE_DATA` - Enable inline data storage for small files (default: ON).

- `LIBGKFS_CREATE_WRITE_OPTIMIZATION` - Optimization for write operations (default: OFF).

- `LIBGKFS_READ_INLINE_PREFETCH` - Prefetch inline data when opening files (default: OFF).

- `LIBGKFS_USE_DIRENTS_COMPRESSION` - Enable compression for directory entries (default: OFF).



#### Caching

##### Dentry cache
@@ -624,10 +631,15 @@ Using two environment variables 
#### Logging

- `GKFS_DAEMON_LOG_PATH` - Path to the log file of the daemon.

- `GKFS_DAEMON_LOG_LEVEL` - Log level of the daemon. Available levels are: `off`, `critical`, `err`, `warn`, `info`, `debug`, `trace`.

#### Optimization

- `GKFS_DAEMON_USE_INLINE_DATA` - Enable inline data storage (default: ON).

- `GKFS_DAEMON_USE_DIRENTS_COMPRESSION` - Enable compression for directory entries (default: OFF).

### Proxy

#### Logging

- `GKFS_PROXY_LOG_PATH` - Path to the log file of the proxy.

- `GKFS_PROXY_LOG_LEVEL` - Log level of the proxy. Available levels are: `off`, `critical`, `err`, `warn`, `info`, `debug`, `trace`.

#### Optimization

- `GKFS_PROXY_USE_DIRENTS_COMPRESSION` - Enable compression for directory entries (default: OFF).



# Acknowledgment

include/client/CMakeLists.txt

+0 −2
Original line number Diff line number Diff line
@@ -44,7 +44,6 @@ target_sources( 
    preload_util.hpp

    cache.hpp

    rpc/rpc_types.hpp

    rpc/forward_management.hpp

    rpc/forward_metadata.hpp

    rpc/forward_data.hpp

    syscalls/args.hpp
@@ -72,7 +71,6 @@ target_sources( 
    preload_util.hpp

    cache.hpp

    rpc/rpc_types.hpp

    rpc/forward_management.hpp

    rpc/forward_metadata.hpp

    rpc/forward_data.hpp

    rpc/forward_malleability.hpp

include/common/common_defs.hpp

+5 −0
Original line number Diff line number Diff line
@@ -59,6 +59,7 @@ namespace tag { 


constexpr auto fs_config = "rpc_srv_fs_config";

constexpr auto create = "rpc_srv_mk_node";

constexpr auto create_write_inline = "rpc_srv_create_write_inline";

constexpr auto stat = "rpc_srv_stat";

constexpr auto remove_metadata = "rpc_srv_rm_metadata";

constexpr auto remove_data = "rpc_srv_rm_data";
@@ -97,6 +98,10 @@ constexpr auto client_proxy_get_dirents_extended = 
constexpr auto proxy_daemon_write = "proxy_daemon_rpc_srv_write_data";

constexpr auto proxy_daemon_read = "proxy_daemon_rpc_srv_read_data";



// inline data operations

constexpr auto write_data_inline = "rpc_srv_write_data_inline";

constexpr auto read_data_inline = "rpc_srv_read_data_inline";



} // namespace tag



namespace protocol {