Merge branch 'tarraf/metrics-aggregator' into 'master'

CMakeUserPresets.json

gkfs/

.gitlab-ci-local/

install-*-test/

In addition, GekkoFS must be compiled with client metrics enabled (disabled by default) via the CMake argument

`-DGKFS_ENABLE_CLIENT_METRICS=ON`.

Client metrics are individually enabled per GekkoFS client process via the following environment variables:

Client metrics/messages are enabled with common `GKFS_` environment variables. Exporting the same variables to both

clients and daemons enables client-side message emission and, when requested, the daemon-side relay:

- `LIBGKFS_ENABLE_METRICS=ON` enables capturing client-side metrics.

- `GKFS_ENABLE_METRICS=ON` enables capturing client-side metrics/messages. Daemons also honor this same variable when

  deciding whether to start the optional relay, so no daemon-specific enable variable is needed.

- `LIBGKFS_METRICS_FLUSH_INTERVAL=10` sets the flush interval to 10 seconds (defaults to 5). All outstanding client

  metrics are flushed when the process ends.

- `LIBGKFS_METRICS_PATH=<path>` sets the path to flush client-metrics (defaults to `/tmp/gkfs_client_metrics`).

- `LIBGKFS_METRICS_IP_PORT=127.0.0.1:5555` enables flushing to a set ZeroMQ server. This option disables flushing to a

- `GKFS_METRICS_IP_PORT=127.0.0.1:5555` enables flushing to a set ZeroMQ server. This option disables flushing to a

  file.

- `LIBGKFS_METRICS_IO_TYPE=w` restricts which of the two `ClientMetrics` instances (write, read) get created at

  all. Defaults to `wr` (both, unchanged behavior). A write-only workload that never reads back what it wrote can

  set this to `w` to skip the read-side thread and socket entirely, since a receiver that only consumes

  `io_type=="w"` messages (e.g. FTIO's default prediction mode) would otherwise unpack and discard every read

  message anyway.

- `GKFS_METRICS_AGGREGATOR=on` routes a client's metrics through the local per-node daemon instead of dialing the

  ZeroMQ sink directly. At high rank density, every rank normally opens its own connection to one remote sink node;

  this collapses that fan-in to one connection per node, and the daemon batches whatever it receives within a short

  window into a single message before forwarding -- so the sink also receives O(nodes) messages instead of

  O(ranks). The daemon needs the same `GKFS_ENABLE_METRICS`, `GKFS_METRICS_AGGREGATOR`, and

  `GKFS_METRICS_IP_PORT` set in its own environment to actually start the aggregator thread. The receiving end needs

  no configuration: a batched message

  is a single top-level msgpack array, a direct message is a flat sequence of scalar fields, and the two are told

  apart by inspecting the first unpacked object's type.

The ZeroMQ export can be tested via the `gkfs_clientmetrics2json` application which is built when enabling the CMake

option `-DGKFS_BUILD_TOOLS=ON`:

- Starting the ZeroMQ server: `gkfs_clientmetrics2json tcp://127.0.0.1:5555`

- `gkfs_clientmetrics2json <path>` can also be used to unpack the Messagepack export from a file.

  Examplarily output with the ZeroMQ sink enabled when running:

  `LD_PRELOAD=libgkfs_intercept.so LIBGKFS_ENABLE_METRICS=ON LIBGKFS_METRICS_IP_PORT=127.0.0.1:5555 gkfs cp testfile /tmp/gkfs_mountdir/testfile`:

  `LD_PRELOAD=libgkfs_intercept.so GKFS_ENABLE_METRICS=ON GKFS_METRICS_IP_PORT=127.0.0.1:5555 gkfs cp testfile /tmp/gkfs_mountdir/testfile`:

```bash

~ $ gkfs_clientmetrics2json tcp://127.0.0.1:5555

- `LIBGKFS_METRICS_FLUSH_INTERVAL` - Set the flush interval for client metrics.

- `LIBGKFS_METRICS_PATH` - Path to flush client metrics.

- `LIBGKFS_METRICS_IP_PORT` - Enable flushing to a set ZeroMQ server (replaces `LIBGKFS_METRICS_PATH`).

- `LIBGKFS_METRICS_IO_TYPE` - Restrict metrics collection to `w`, `r`, or `wr` (default). Skips creating the

  unused side entirely rather than collecting and discarding it.

- `LIBGKFS_METRICS_AGGREGATOR` - Route metrics through the local daemon (one connection per node) instead of

  connecting to the ZeroMQ sink directly from every rank. Requires the same variable set in the daemon's

  environment.

- `LIBGKFS_PROXY_PID_FILE` - Path to the proxy pid file (when using the GekkoFS proxy).

- `LIBGKFS_NUM_REPL` - Number of replicas for data.

#### Optimization

#define GKFS_CLIENT_ENV

#include <config.hpp>

#include <common/env.hpp>

#define ADD_PREFIX(str) CLIENT_ENV_PREFIX str

static constexpr auto HOSTS_FILE = ADD_PREFIX("HOSTS_FILE");

static constexpr auto FORWARDING_MAP_FILE = ADD_PREFIX("FORWARDING_MAP_FILE");

#ifdef GKFS_ENABLE_CLIENT_METRICS

static constexpr auto ENABLE_METRICS = ADD_PREFIX("ENABLE_METRICS");

static constexpr auto METRICS_FLUSH_INTERVAL =

        ADD_PREFIX("METRICS_FLUSH_INTERVAL");

static constexpr auto METRICS_PATH = ADD_PREFIX("METRICS_PATH");

static constexpr auto METRICS_IP_PORT = ADD_PREFIX("METRICS_IP_PORT");

// "w", "r", or "wr"/"rw" (default) -- which ClientMetrics instances to

// create at all. A consumer that only ever reads io_type=="w" messages

// (e.g. FTIO's default prediction mode) shouldn't pay for a read-metrics

// thread+socket per rank that gets discarded downstream every time.

static constexpr auto METRICS_IO_TYPE = ADD_PREFIX("METRICS_IO_TYPE");

// Opt-in, test-only: when set, connect to the local daemon's relay socket

// instead of dialing FTIO directly -- collapses per-rank connections down

// to one per node. The daemon must have GKFS_ENABLE_METRICS and the same

// routing vars set to actually start the relay (see daemon/env.hpp).

#endif

static constexpr auto PROTECT_FD = ADD_PREFIX("PROTECT_FD");

/*

  Copyright 2018-2025, Barcelona Supercomputing Center (BSC), Spain

  Copyright 2015-2025, Johannes Gutenberg Universitaet Mainz, Germany

  This software was partially supported by the

  EC H2020 funded project NEXTGenIO (Project ID: 671951, www.nextgenio.eu).

  This software was partially supported by the

  ADA-FS project under the SPPEXA project funded by the DFG.

  This software was partially supported by the

  the European Union’s Horizon 2020 JTI-EuroHPC research and

  innovation programme, by the project ADMIRE (Project ID: 956748,

  admire-eurohpc.eu)

  This project was partially promoted by the Ministry for Digital Transformation

  and the Civil Service, within the framework of the Recovery,

  Transformation and Resilience Plan - Funded by the European Union

  -NextGenerationEU.

  This file is part of GekkoFS.

  GekkoFS is free software: you can redistribute it and/or modify

  it under the terms of the GNU General Public License as published by

  the Free Software Foundation, either version 3 of the License, or

  (at your option) any later version.

  GekkoFS is distributed in the hope that it will be useful,

  but WITHOUT ANY WARRANTY; without even the implied warranty of

  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

  GNU General Public License for more details.

  You should have received a copy of the GNU General Public License

  along with GekkoFS.  If not, see <https://www.gnu.org/licenses/>.

  SPDX-License-Identifier: GPL-3.0-or-later

*/

#ifndef GKFS_COMMON_ENV_HPP

#define GKFS_COMMON_ENV_HPP

#include <config.hpp>

#define ADD_PREFIX(str) COMMON_ENV_PREFIX str

/* Environment variables shared by GekkoFS clients and daemons */

namespace gkfs::env {

#ifdef GKFS_ENABLE_CLIENT_METRICS

static constexpr auto ENABLE_METRICS = ADD_PREFIX("ENABLE_METRICS");

static constexpr auto METRICS_IP_PORT = ADD_PREFIX("METRICS_IP_PORT");

static constexpr auto METRICS_AGGREGATOR = ADD_PREFIX("METRICS_AGGREGATOR");

#endif

} // namespace gkfs::env

#undef ADD_PREFIX

#endif // GKFS_COMMON_ENV_HPP

 No newline at end of file

    void

    zmq_connect(const std::string& ip_port);

    // Like zmq_connect(), but takes a full zmq endpoint URI as-is (no

    // "tcp://" prefix added) -- used for the LIBGKFS_METRICS_DAEMON_RELAY

    // local ipc:// endpoint.

    void

    zmq_connect_raw(const std::string& endpoint);

    bool

    zmq_is_connected();