Commit d7bad519 authored by Ahmad Tarraf's avatar Ahmad Tarraf
Browse files

feat(metrics): non-blocking flush, opt-in read/write filter, opt-in per-node aggregator 

- flush_msgpack()'s send used zmq::send_flags::none (blocking) despite a comment claiming otherwise; socket also had no LINGER override (default -1, indefinite). Both now non-blocking (dontwait + LINGER=0) -- a rank's final flush at teardown could otherwise stall on a congested FTIO sink
- Add LIBGKFS_METRICS_IO_TYPE (w/r/wr, default wr = unchanged behavior) -- skips creating read_metrics_ for write-only workloads instead of collecting and discarding it downstream
- Null-guard every write_metrics()/read_metrics() call site (preload.cpp, fuse_client.cpp, gkfs_data.cpp) since either can now be unset
- Add LIBGKFS_METRICS_AGGREGATOR (opt-in, off by default) -- ranks push metrics to the local daemon over ipc:///tmp/gkfs_metrics_aggregator.sock instead of dialing the ZeroMQ sink directly; daemon batches a 500ms window into one msgpack array-of-bytes message per node before forwarding
- Aggregator runs on its own std::thread in the daemon, separate from the Argobots I/O execution streams
- Receiving end (FTIO) needs no flag: a batch is one top-level msgpack array, a direct message is a flat sequence of 8-9 top-level scalars -- told apart by the first unpacked object's type
- Fix: daemon target was missing target_compile_definitions(gkfs_daemon PUBLIC GKFS_ENABLE_CLIENT_METRICS) -- the whole aggregator was silently compiled out behind #ifdef with no build error until this was added
- Document both new env vars in README.md (client-metrics section + env var reference table)

Motivated by a measured 13.5% glass-mode app-phase slowdown at 121 nodes (~960 ranks) with metrics on vs. off, growing with rank density (2.6% at 17N). Root-caused to per-rank connection/socket count, not the metrics payload itself.
parent 629d0206

.gitignore

+1 −0
Original line number Diff line number Diff line
@@ -97,3 +97,4 @@ builds/ 
CMakeUserPresets.json

gkfs/

.gitlab-ci-local/

install-*-test/

README.md

+18 −0
Original line number Diff line number Diff line
@@ -400,6 +400,19 @@ Client metrics are individually enabled per GekkoFS client process via the follo 
- `LIBGKFS_METRICS_PATH=<path>` sets the path to flush client-metrics (defaults to `/tmp/gkfs_client_metrics`).

- `LIBGKFS_METRICS_IP_PORT=127.0.0.1:5555` enables flushing to a set ZeroMQ server. This option disables flushing to a

  file.

- `LIBGKFS_METRICS_IO_TYPE=w` restricts which of the two `ClientMetrics` instances (write, read) get created at

  all. Defaults to `wr` (both, unchanged behavior). A write-only workload that never reads back what it wrote can

  set this to `w` to skip the read-side thread and socket entirely, since a receiver that only consumes

  `io_type=="w"` messages (e.g. FTIO's default prediction mode) would otherwise unpack and discard every read

  message anyway.

- `LIBGKFS_METRICS_AGGREGATOR=on` routes a client's metrics through the local per-node daemon instead of dialing the

  ZeroMQ sink directly. At high rank density, every rank normally opens its own connection to one remote sink node;

  this collapses that fan-in to one connection per node, and the daemon batches whatever it receives within a short

  window into a single message before forwarding -- so the sink also receives O(nodes) messages instead of

  O(ranks). The daemon needs the same `LIBGKFS_METRICS_AGGREGATOR` and `LIBGKFS_METRICS_IP_PORT` set in its own

  environment to actually start the aggregator thread. The receiving end needs no configuration: a batched message

  is a single top-level msgpack array, a direct message is a flat sequence of scalar fields, and the two are told

  apart by inspecting the first unpacked object's type.



The ZeroMQ export can be tested via the `gkfs_clientmetrics2json` application which is built when enabling the CMake

option `-DGKFS_BUILD_TOOLS=ON`:
@@ -720,6 +733,11 @@ Client-metrics require the CMake argument `-DGKFS_ENABLE_CLIENT_METRICS=ON` (see 
- `LIBGKFS_METRICS_FLUSH_INTERVAL` - Set the flush interval for client metrics.

- `LIBGKFS_METRICS_PATH` - Path to flush client metrics.

- `LIBGKFS_METRICS_IP_PORT` - Enable flushing to a set ZeroMQ server (replaces `LIBGKFS_METRICS_PATH`).

- `LIBGKFS_METRICS_IO_TYPE` - Restrict metrics collection to `w`, `r`, or `wr` (default). Skips creating the

  unused side entirely rather than collecting and discarding it.

- `LIBGKFS_METRICS_AGGREGATOR` - Route metrics through the local daemon (one connection per node) instead of

  connecting to the ZeroMQ sink directly from every rank. Requires the same variable set in the daemon's

  environment.

- `LIBGKFS_PROXY_PID_FILE` - Path to the proxy pid file (when using the GekkoFS proxy).

- `LIBGKFS_NUM_REPL` - Number of replicas for data.

#### Optimization

include/client/env.hpp

+10 −0
Original line number Diff line number Diff line
@@ -66,6 +66,16 @@ static constexpr auto METRICS_FLUSH_INTERVAL = 
        ADD_PREFIX("METRICS_FLUSH_INTERVAL");

static constexpr auto METRICS_PATH = ADD_PREFIX("METRICS_PATH");

static constexpr auto METRICS_IP_PORT = ADD_PREFIX("METRICS_IP_PORT");

// "w", "r", or "wr"/"rw" (default) -- which ClientMetrics instances to

// create at all. A consumer that only ever reads io_type=="w" messages

// (e.g. FTIO's default prediction mode) shouldn't pay for a read-metrics

// thread+socket per rank that gets discarded downstream every time.

static constexpr auto METRICS_IO_TYPE = ADD_PREFIX("METRICS_IO_TYPE");

// Opt-in, test-only: when set, connect to the local daemon's relay socket

// instead of dialing FTIO directly -- collapses per-rank connections down

// to one per node. The daemon must have the same var set to actually start

// the relay (see daemon/env.hpp).

static constexpr auto METRICS_AGGREGATOR = ADD_PREFIX("METRICS_AGGREGATOR");

#endif



static constexpr auto PROTECT_FD = ADD_PREFIX("PROTECT_FD");

include/common/msgpack_util.hpp

+6 −0
Original line number Diff line number Diff line
@@ -142,6 +142,12 @@ public: 
    void

    zmq_connect(const std::string& ip_port);



    // Like zmq_connect(), but takes a full zmq endpoint URI as-is (no

    // "tcp://" prefix added) -- used for the LIBGKFS_METRICS_DAEMON_RELAY

    // local ipc:// endpoint.

    void

    zmq_connect_raw(const std::string& endpoint);



    bool

    zmq_is_connected();

include/config.hpp

+1 −1
Original line number Diff line number Diff line
@@ -115,7 +115,7 @@ constexpr auto dir = "metadata"; 
// Blocks are used to store the rename status (-1 is a renamed file)

constexpr auto use_atime = false;

constexpr auto use_ctime = true;

constexpr auto use_mtime = false;

constexpr auto use_mtime = true;

constexpr auto use_link_cnt = false;

constexpr auto use_blocks = true;

/*