feat(metrics): non-blocking flush, opt-in read/write filter, opt-in per-node aggregator

Ahmad Tarrafrequested to merge
tarraf/metrics-aggregator into master
  • flush_msgpack()'s send used zmq::send_flags::none (blocking) despite a comment claiming otherwise; socket also had no LINGER override (default -1, indefinite). Both now non-blocking (dontwait + LINGER=0) -- a rank's final flush at teardown could otherwise stall on a congested FTIO sink
  • Add LIBGKFS_METRICS_IO_TYPE (w/r/wr, default wr = unchanged behavior) -- skips creating read_metrics_ for write-only workloads instead of collecting and discarding it downstream
  • Null-guard every write_metrics()/read_metrics() call site (preload.cpp, fuse_client.cpp, gkfs_data.cpp) since either can now be unset
  • Add LIBGKFS_METRICS_AGGREGATOR (opt-in, off by default) -- ranks push metrics to the local daemon over ipc:///tmp/gkfs_metrics_aggregator.sock instead of dialing the ZeroMQ sink directly; daemon batches a 500ms window into one msgpack array-of-bytes message per node before forwarding
  • Aggregator runs on its own std::thread in the daemon, separate from the Argobots I/O execution streams
  • Receiving end (FTIO) needs no flag: a batch is one top-level msgpack array, a direct message is a flat sequence of 8-9 top-level scalars -- told apart by the first unpacked object's type
  • Fix: daemon target was missing target_compile_definitions(gkfs_daemon PUBLIC GKFS_ENABLE_CLIENT_METRICS) -- the whole aggregator was silently compiled out behind #ifdef with no build error until this was added
  • Document both new env vars in README.md (client-metrics section + env var reference table)

Motivated by a measured 13.5% glass-mode app-phase slowdown at 121 nodes (~960 ranks) with metrics on vs. off, growing with rank density (2.6% at 17N). Root-caused to per-rank connection/socket count, not the metrics payload itself.

Merge request reports

include/client/env.hpp

+10 −2
Changes for include/client/env.hpp: 10 added lines, 2 removed lines.
Original line number Diff line number Diff line
@@ -41,6 +41,7 @@ 
#define GKFS_CLIENT_ENV



#include <config.hpp>

#include <common/env.hpp>



#define ADD_PREFIX(str) CLIENT_ENV_PREFIX str
@@ -61,11 +62,18 @@ static constexpr auto CWD = ADD_PREFIX("CWD"); 
static constexpr auto HOSTS_FILE = ADD_PREFIX("HOSTS_FILE");

static constexpr auto FORWARDING_MAP_FILE = ADD_PREFIX("FORWARDING_MAP_FILE");

#ifdef GKFS_ENABLE_CLIENT_METRICS

static constexpr auto ENABLE_METRICS = ADD_PREFIX("ENABLE_METRICS");

static constexpr auto METRICS_FLUSH_INTERVAL =

        ADD_PREFIX("METRICS_FLUSH_INTERVAL");

static constexpr auto METRICS_PATH = ADD_PREFIX("METRICS_PATH");

static constexpr auto METRICS_IP_PORT = ADD_PREFIX("METRICS_IP_PORT");

// "w", "r", or "wr"/"rw" (default) -- which ClientMetrics instances to

// create at all. A consumer that only ever reads io_type=="w" messages

// (e.g. FTIO's default prediction mode) shouldn't pay for a read-metrics

// thread+socket per rank that gets discarded downstream every time.

static constexpr auto METRICS_IO_TYPE = ADD_PREFIX("METRICS_IO_TYPE");

// Opt-in, test-only: when set, connect to the local daemon's relay socket

// instead of dialing FTIO directly -- collapses per-rank connections down

// to one per node. The daemon must have GKFS_ENABLE_METRICS and the same

// routing vars set to actually start the relay (see daemon/env.hpp).

#endif



static constexpr auto PROTECT_FD = ADD_PREFIX("PROTECT_FD");

include/common/env.hpp

0 → 100644
+59 −0
Changes for include/common/env.hpp: 59 added lines, 0 removed lines.
Original line number Diff line number Diff line 
/*

  Copyright 2018-2025, Barcelona Supercomputing Center (BSC), Spain

  Copyright 2015-2025, Johannes Gutenberg Universitaet Mainz, Germany



  This software was partially supported by the

  EC H2020 funded project NEXTGenIO (Project ID: 671951, www.nextgenio.eu).



  This software was partially supported by the

  ADA-FS project under the SPPEXA project funded by the DFG.



  This software was partially supported by the

  the European Union’s Horizon 2020 JTI-EuroHPC research and

  innovation programme, by the project ADMIRE (Project ID: 956748,

  admire-eurohpc.eu)



  This project was partially promoted by the Ministry for Digital Transformation

  and the Civil Service, within the framework of the Recovery,

  Transformation and Resilience Plan - Funded by the European Union

  -NextGenerationEU.



  This file is part of GekkoFS.



  GekkoFS is free software: you can redistribute it and/or modify

  it under the terms of the GNU General Public License as published by

  the Free Software Foundation, either version 3 of the License, or

  (at your option) any later version.



  GekkoFS is distributed in the hope that it will be useful,

  but WITHOUT ANY WARRANTY; without even the implied warranty of

  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

  GNU General Public License for more details.



  You should have received a copy of the GNU General Public License

  along with GekkoFS.  If not, see <https://www.gnu.org/licenses/>.



  SPDX-License-Identifier: GPL-3.0-or-later

*/



#ifndef GKFS_COMMON_ENV_HPP

#define GKFS_COMMON_ENV_HPP



#include <config.hpp>



#define ADD_PREFIX(str) COMMON_ENV_PREFIX str



/* Environment variables shared by GekkoFS clients and daemons */

namespace gkfs::env {



#ifdef GKFS_ENABLE_CLIENT_METRICS

static constexpr auto ENABLE_METRICS = ADD_PREFIX("ENABLE_METRICS");

static constexpr auto METRICS_IP_PORT = ADD_PREFIX("METRICS_IP_PORT");

static constexpr auto METRICS_AGGREGATOR = ADD_PREFIX("METRICS_AGGREGATOR");

#endif



} // namespace gkfs::env



#undef ADD_PREFIX



#endif // GKFS_COMMON_ENV_HPP
 
 No newline at end of file

include/common/msgpack_util.hpp

+6 −0
Changes for include/common/msgpack_util.hpp: 6 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -142,6 +142,12 @@ public: 
    void

    zmq_connect(const std::string& ip_port);



    // Like zmq_connect(), but takes a full zmq endpoint URI as-is (no

    // "tcp://" prefix added) -- used for the LIBGKFS_METRICS_DAEMON_RELAY

    // local ipc:// endpoint.

    void

    zmq_connect_raw(const std::string& endpoint);



    bool

    zmq_is_connected();

include/daemon/env.hpp

+1 −0
Changes for include/daemon/env.hpp: 1 added line, 0 removed lines.
Original line number Diff line number Diff line
@@ -44,6 +44,7 @@ 
#define GKFS_DAEMON_ENV



#include <config.hpp>

#include <common/env.hpp>



#define ADD_PREFIX(str) COMMON_ENV_PREFIX str

include/config.hpp

+1 −1
Changes for include/config.hpp: 1 added line, 1 removed line.
Original line number Diff line number Diff line
@@ -115,7 +115,7 @@ constexpr auto dir = "metadata"; 
// Blocks are used to store the rename status (-1 is a renamed file)

constexpr auto use_atime = false;

constexpr auto use_ctime = true;

constexpr auto use_mtime = false;

constexpr auto use_mtime = true;

constexpr auto use_link_cnt = false;

constexpr auto use_blocks = true;

/*