improve endpoint retries and runtime snapshots

Periodic snapshots include a wall-clock `timestamp_ms`, daemon `pid`,

`uptime_seconds`, sampled Argobots I/O queue depth (`io_queue_depth`) and total

queued tasks (`io_queue_total`), plus migration job and byte counters. The same

current runtime values are exposed in the `GKFS_RUNTIME` gauge family with a

low-cardinality `metric` label. Cumulative RPC/backend events use

queued tasks (`io_queue_total`), migration job/byte counters, daemon operation

totals (`operations_create`, `operations_write`, `operations_read`,

`operations_stat`, `operations_dirent`, `operations_remove`), and read/write

byte totals. The same current runtime values are exposed in the `GKFS_RUNTIME`

gauge family with a low-cardinality `metric` label. Cumulative RPC/backend events use

`GKFS_RUNTIME_TOTAL`; RPC and backend latencies use

`GKFS_RUNTIME_LATENCY_US` summaries.

`GKFS_RUNTIME_LATENCY_US` summaries. Daemon operation events and byte

observations remain available through the existing `IOPS` and `SIZE` Prometheus

families.

File and daemon-log snapshots contain one `METRICS_JSON { ... }` record per

sample, followed by the readable statistics report. The JSON record includes

RPC request/error totals and latency maxima, backend

read/write/error/retry totals and latency maxima, migration progress, and queue

depth. Legacy client MessagePack/libzmq metrics remain unchanged.

depth, daemon operation totals, and read/write byte totals. Legacy client

MessagePack/libzmq metrics remain unchanged.

### Client

#### Core

- `LIBGKFS_HOSTS_FILE` - Path to the hostsfile (created by the daemon and mandatory for the client).

- `LIBGKFS_ENABLE_FORK` - Enable fork support in the client library, used for example in DLIO.

- `LIBGKFS_OFI_INTERFACE` - Force the client-side libfabric interface to use (equivalent to `FI_SOCKETS_IFACE`).

- `LIBGKFS_OFI_INTERFACE` - Force the client-side libfabric interface to use (equivalent to `FI_SOCKETS_IFACE`). For `ofi+sockets`, GekkoFS validates that the named local interface exists before initializing Margo.

- `LIBGKFS_RPC_LOOKUP_RETRIES` - Endpoint lookup attempts during client startup and malleability control RPC setup, in `[1,20]` (default: `3`).

- `LIBGKFS_RPC_LOOKUP_BACKOFF_MS` - Initial endpoint-lookup retry backoff, in milliseconds, in `[1,10000]`; retries use bounded exponential backoff with jitter (default: `50`).

  This prevents clients from binding to loopback (`127.0.0.1`) when daemons are on real NICs (e.g., `ib0`).

  Required in malleable/HPC environments where the client may resolve to a loopback address.

  Example: `export LIBGKFS_OFI_INTERFACE=ib0`

### 6. Improve endpoint and network-interface configuration

**Status: Partial.** Interface precedence, validation, and bounded retries are

implemented. Full timeout/address configuration remains.

**Status: Partial.** Interface precedence, protocol-aware sockets-interface

validation, and configurable bounded lookup retries/backoff are implemented.

Per-RPC timeout and explicit address override configuration remain.

**Problem:** Client endpoint selection can bind to an unintended interface,

causing connection failures and cascading job failures on multi-network hosts.

**Work:**

- Add explicit client interface/address configuration through CLI and

  environment variables.

- Validate protocol, interface, and address combinations at startup.

- Provide explicit client interface configuration through environment variables

  and validate sockets-provider interface names at startup.

- Log selected local and peer addresses at debug level.

- Add configurable connect, request, and retry timeouts.

- Keep lookup retries/backoff configurable. Per-RPC timeout support exists in

  Thallium but requires each forwarder to opt in; do not expose an inert global

  timeout setting.

- Distinguish transient lookup failures from invalid configuration.

- Bound retries with exponential backoff and jitter.

Timestamped snapshots now include sampled I/O queue depth and migration

counters in file/log output and Prometheus gauges. RPC and backend error,

retry, and latency aggregates are included. Legacy client MessagePack/libzmq

metrics remain unchanged. Broader per-operation metrics and distributed tracing

metrics remain unchanged. Daemon operation/byte totals are included in periodic

snapshots. Distributed tracing and broader RPC timeout/materialization metrics

remain.

Define stable, low-cardinality metrics for operation count/bytes/latency/errors;

checkpoint files; this is startup-only and disabled by default.

Snapshots include `timestamp_ms`, `pid`, `uptime_seconds`, sampled I/O queue

depth/total, and migration job/byte counters. Prometheus exposes current

depth/total, migration job/byte counters, daemon operation totals, and read/write

byte totals. Prometheus exposes current

runtime values as `GKFS_RUNTIME{metric="..."}` gauges, cumulative RPC/backend

events as `GKFS_RUNTIME_TOTAL`, and latency observations as

`GKFS_RUNTIME_LATENCY_US` summaries.

`GKFS_RUNTIME_LATENCY_US` summaries. Daemon operation events and byte

observations remain available through the existing `IOPS` and `SIZE` families.

File and daemon-log snapshots also contain a `METRICS_JSON { ... }` record with

RPC latency/errors, backend latency/errors/retries, and migration fields before

the human-readable report. Legacy client MessagePack/libzmq metrics remain

unchanged.

RPC latency/errors, backend latency/errors/retries, migration fields, daemon

operation totals, and read/write byte totals before the human-readable report.

Legacy client MessagePack/libzmq metrics remain unchanged.

### Advanced experimental features

// OFI_INTERFACE is used with the GKFS_ prefix (e.g., LIBGKFS_OFI_INTERFACE)

// LIBGKFS_OFI_INTERFACE is the literal env var name for client-side pinning

static constexpr auto OFI_INTERFACE = ADD_PREFIX("OFI_INTERFACE");

static constexpr auto RPC_LOOKUP_RETRIES = ADD_PREFIX("RPC_LOOKUP_RETRIES");

static constexpr auto RPC_LOOKUP_BACKOFF_MS =

        ADD_PREFIX("RPC_LOOKUP_BACKOFF_MS");

} // namespace gkfs::env

#include <map>

#include <type_traits>

#include <optional>

#include <chrono>

namespace gkfs::metadata {

// function definitions

namespace gkfs::utils {

struct LookupRetryPolicy {

    std::size_t retries{3};

    std::chrono::milliseconds backoff{50};

};

LookupRetryPolicy

lookup_retry_policy();

template <typename E>

constexpr typename std::underlying_type<E>::type

to_underlying(E e) {