Add opt-in dependency tuning and MN5 0.9.7 module

# Opt-in daemon bulk staging and deadlines

Defaults unchanged: request-local, zero-initialized buffers; synchronous bulk

transfers without a deadline. The controls below are read by daemon data RPCs,

not by clients or metadata/proxy RPCs. Migration and bulk replica repair use

the same daemon read/write handlers and therefore inherit these settings.

| Variable | Default | Meaning |

|---|---:|---|

| `GKFS_BULK_POOL` | unset | `ON`, `true`, or `1` enables reuse; `OFF`, `false`, or `0` disables it (case-insensitive). |

| `GKFS_BULK_POOL_BYTES` | 67108864 | Total registered-byte budget per daemon server engine. |

| `GKFS_BULK_POOL_SLOTS` | 8 | Fixed slot count; each slot holds `bytes / slots` bytes. |

| `GKFS_BULK_TIMEOUT_MS` | 0 | Positive integer timeout for each daemon data pull/push. |

Set variables before starting the daemon; do not change them during runtime.

Numeric settings accept decimal digits only. Malformed, empty, or overflowing

values are rejected explicitly. An enabled pool requires nonzero slots and at

least one byte per slot; zero/too-small budgets are rejected. Slots never grow.

Exhausted pools and oversized requests fall back to the original allocation /

registration path without waiting for a slot. The pool budget bounds retained

registered pool memory, **not** total request staging memory. Pool construction

allocation/registration failures propagate just as staging failures do.

The pool uses Thallium `bulk_buffer_pool::try_get()` and request-owned leases.

Bulk handles, chunk operations, and synchronous transfers finish before the

lease returns its slot. Reused memory is zeroed to retain sparse-read behavior.

Shutdown stops pool admission and drains leases before I/O stream teardown;

registered pool storage is released before the server engine. A finalize

callback removes the stopped registry entry. Do not call pool shutdown after

Argobots has been torn down, or from a request holding a lease.

Compile-time header guards keep older Thallium builds working: absent pool or

timed-bulk headers reject the corresponding requested opt-in explicitly, while

default settings still compile and run. Verified against

the testing image's Thallium **0.17.1** and Margo **0.24.2**. The timed path uses

only synchronous operators. In upstream Margo v0.24.2 `margo-core.c`, the timer

calls `HG_Bulk_cancel`; the completion callback maps cancellation to

`HG_TIMEOUT`; `margo_bulk_transfer_timed` waits for that completion before

returning. Thus a timeout cannot leave a transfer accessing a returned slot.

Timeout errors map to `ETIMEDOUT`; other transfer exceptions retain `EBUSY`.

This deadline is per transfer, not a whole-RPC or disk-I/O deadline. Partial

completed writes remain possible, as with existing transfer failures.

Standalone source-integration check (install/build dependencies first):

```sh

c++ -std=c++17 -Iinclude tests/standalone/bulk_staging.cpp \

    $(pkg-config --cflags --libs thallium) -o /tmp/bulk-staging-test

/tmp/bulk-staging-test

```

This checks strict numeric/boolean parsing and unsupported-header rejection, distinct concurrent buffers, pool exhaustion and

oversize fallback, reuse with zeroing, successful synchronous pull/push with a

deadline, shutdown, and timeout-to-errno mapping. It does **not** inject a stalled

transport or claim an end-to-end filesystem timeout test.

## Migration and repair scope

Daemon malleability `forward_data` and `forward_data_from_source` in

`src/daemon/malleability/rpc/forward_redistribution.cpp` send ordinary read/write

RPCs. Their actual pull/push is performed by the timed handlers here, including

writes marked `migration=true`. Client bulk `repair_data_copy` in

`src/client/rpc/forward_data.cpp` likewise uses ordinary read/write RPCs. Their

existing response/error checks propagate failure; no outer RPC deadline is

introduced, since abandoning a request before remote transfer completion would

endanger caller-owned registered buffers. Inline repairs carry data in RPC

arguments and have no bulk transfer to time. Source-side migration registration

and local disk I/O are not pooled or timed by this feature.

   users/running

   users/forwarding

   users/scripts

   users/rpc-tracing

   users/daemon-bulk

.. toctree::

   :maxdepth: 2

=================

GekkoFS relies on CMake for building and testing the daemon and client

binaries, which are written in C++17. As such, GekkoFS has the following

base dependencies for its build system:

binaries. The client uses C++17; the daemon and RocksDB-dependent tests require

C++20 when using RocksDB 10.7 or newer, including RocksDB 11.8.1 selected by

the 0.9.7 profiles. GekkoFS has the following base dependencies for its build

system:

- A compiler with C++17 support:

  - `GCC <https://gcc.gnu.org>`_ version 8.0 and newer

  - `Clang <https://clang.llvm.org/>`_ 5.0 and newer.

- `CMake <https://cmake.org>`_ version 3.6 or newer to build the main code.

  With the 0.9.7 RocksDB profiles, use GCC 11 or newer, or Clang 10 or newer.

  - Version 3.11 or newer to build the tests.

- `CMake <https://cmake.org>`_ version 3.16 or newer to build the main code

  and tests.

GekkoFS also needs :code:`curl` and :code:`git` to be installed in

your system if you plan on using the dependency management scripts provided

        spack unload gekkofs

MareNostrum 5 compiler selection

===============================

The 0.9.7 ``mn5`` profile keeps RocksDB 11.8.1: MN5's GPP compiler modules

meet its upstream C++20 requirement (GCC 11 or newer, or Clang 10 or newer).

Module-file inspection found the following choices:

- GCC: ``gcc/12.3.0``, ``gcc/13.2.0``, ``gcc/14.1.0_binutils241``.

- Clang: ``clang/18.1.7``, ``clang/20.1.5``.

- LLVM: ``llvm/18.1.1``, ``llvm/19.1.3``, ``llvm/20.1.8``.

The module files are under ``/apps/GPP/modulefiles/compilers``. GCC 12.3.0

was loaded and its ``g++`` successfully compiled a C++20 syntax check including

the RocksDB 11.8.1 public headers. The other listed modules meet the version

requirement but were not individually build-tested. Intel compiler modules

were not validated for this dependency profile.

For a source build, explicitly select the verified GCC toolchain before

building both dependencies and GekkoFS:

.. code-block:: console

    module purge

    module load gcc/12.3.0

    export CC=gcc

    export CXX=g++

    command -v gcc g++

    g++ --version

Use ``-p mn5:0.9.7`` with the dependency download and compilation scripts.

Use a fresh GekkoFS build directory when changing compilers, and keep the same

compiler runtime for the dependency build, daemon, and clients. The

``GekkoFS/master-unstable`` module sets installed binary and library paths;

its module file does not select a compiler or prove those installed libraries

match the 0.9.7 profile. The header check establishes compiler compatibility,

not a complete MN5 dependency build or runtime test.

Isolated 0.9.7 module

---------------------

The repository provides ``scripts/run/master-0.9.7`` for later installation as

``GekkoFS/master-0.9.7`` on MN5. It loads ``intel/2023.2.0`` for the runtime

libraries required by ``ucx/1.17.0``, then selects ``gcc/14.1.0_binutils241``

for compilation and loads ``libfabric/1.21.0-gcc``. It explicitly resets C/C++

and MPI-wrapper compiler variables to GCC to avoid Intel compiler selection.

The GCC 14 toolchain has not yet been used for a full dependency build.

Use separate installation prefixes:

- Dependencies: ``/apps/GPP/GEKKOFS/deps-0.9.7`` (``GKFS_DEPS``).

- GekkoFS: ``/apps/GPP/GEKKOFS/master-0.9.7`` (``GKFS_BASE``).

The module adds dependency paths for pkg-config, CMake, linking, and runtime

loading. It does not create directories, download dependencies, or compile

anything. Build and install into these prefixes before running applications.

Load it from a clean environment (``module purge`` followed by

``module load GekkoFS/master-0.9.7``) to avoid older compiler/library paths.

The ``mn5:0.9.7`` profile still builds libfabric 2.7.0 into ``GKFS_DEPS``;

that installation takes precedence over the site's libfabric 1.21.0 module.

The ``mn5:0.9.7`` profile enables Mercury's UCX transport alongside OFI,

using UCX from the loaded module rather than building another copy. This makes

both transports available; selecting UCX at runtime still requires a suitable

GekkoFS protocol configuration. UCX module loading and shared-library resolution

were checked, but the combined dependency build and communication tests remain

pending.

Miscellaneous

=========================

# Opt-in daemon bulk staging and deadlines

Defaults unchanged: request-local, zero-initialized buffers; synchronous bulk

transfers without a deadline. The controls below are read by daemon data RPCs,

not by clients or metadata/proxy RPCs. Migration and bulk replica repair use

the same daemon read/write handlers and therefore inherit these settings.

| Variable | Default | Meaning |

|---|---:|---|

| `GKFS_BULK_POOL` | unset | `ON`, `true`, or `1` enables reuse; `OFF`, `false`, or `0` disables it (case-insensitive). |

| `GKFS_BULK_POOL_BYTES` | 67108864 | Total registered-byte budget per daemon server engine. |

| `GKFS_BULK_POOL_SLOTS` | 8 | Fixed slot count; each slot holds `bytes / slots` bytes. |

| `GKFS_BULK_TIMEOUT_MS` | 0 | Positive integer timeout for each daemon data pull/push. |

Set variables before starting the daemon; do not change them during runtime.

Numeric settings accept decimal digits only. Malformed, empty, or overflowing

values are rejected explicitly. An enabled pool requires nonzero slots and at

least one byte per slot; zero/too-small budgets are rejected. Slots never grow.

Exhausted pools and oversized requests fall back to the original allocation /

registration path without waiting for a slot. The pool budget bounds retained

registered pool memory, **not** total request staging memory. Pool construction

allocation/registration failures propagate just as staging failures do.

The pool uses Thallium `bulk_buffer_pool::try_get()` and request-owned leases.

Bulk handles, chunk operations, and synchronous transfers finish before the

lease returns its slot. Reused memory is zeroed to retain sparse-read behavior.

Shutdown stops pool admission and drains leases before I/O stream teardown;

registered pool storage is released before the server engine. A finalize

callback removes the stopped registry entry. Do not call pool shutdown after

Argobots has been torn down, or from a request holding a lease.

Compile-time header guards keep older Thallium builds working: absent pool or

timed-bulk headers reject the corresponding requested opt-in explicitly, while

default settings still compile and run. Verified against

the testing image's Thallium **0.17.1** and Margo **0.24.2**. The timed path uses

only synchronous operators. In upstream Margo v0.24.2 `margo-core.c`, the timer

calls `HG_Bulk_cancel`; the completion callback maps cancellation to

`HG_TIMEOUT`; `margo_bulk_transfer_timed` waits for that completion before

returning. Thus a timeout cannot leave a transfer accessing a returned slot.

Timeout errors map to `ETIMEDOUT`; other transfer exceptions retain `EBUSY`.

This deadline is per transfer, not a whole-RPC or disk-I/O deadline. Partial

completed writes remain possible, as with existing transfer failures.

Standalone source-integration check (install/build dependencies first):

```sh

c++ -std=c++17 -Iinclude tests/standalone/bulk_staging.cpp \

    $(pkg-config --cflags --libs thallium) -o /tmp/bulk-staging-test

/tmp/bulk-staging-test

```

This checks strict numeric/boolean parsing and unsupported-header rejection, distinct concurrent buffers, pool exhaustion and

oversize fallback, reuse with zeroing, successful synchronous pull/push with a

deadline, shutdown, and timeout-to-errno mapping. It does **not** inject a stalled

transport or claim an end-to-end filesystem timeout test.

## Migration and repair scope

Daemon malleability `forward_data` and `forward_data_from_source` in

`src/daemon/malleability/rpc/forward_redistribution.cpp` send ordinary read/write

RPCs. Their actual pull/push is performed by the timed handlers here, including

writes marked `migration=true`. Client bulk `repair_data_copy` in

`src/client/rpc/forward_data.cpp` likewise uses ordinary read/write RPCs. Their

existing response/error checks propagate failure; no outer RPC deadline is

introduced, since abandoning a request before remote transfer completion would

endanger caller-owned registered buffers. Inline repairs carry data in RPC

arguments and have no bulk transfer to time. Source-side migration registration

and local disk I/O are not pooled or timed by this feature.

  The 0.9.7 profiles select RocksDB 11.8.1, which requires C++20:

  GCC 11 or newer, or Clang 10 or newer. This applies to the daemon and

  RocksDB-dependent tests; the client remains on C++17.

  On MareNostrum 5, ``gcc/12.3.0`` meets this requirement and passed a C++20

  compilation check of RocksDB 11.8.1 public headers. The ``mn5:0.9.7`` profile

  therefore retains RocksDB 11.8.1. See :ref:`building_gekkofs` for compiler

  module selection and validation limits.

- `CMake <https://cmake.org>`_ version 3.16 or newer to build the main code and tests.