Commit 02a62065 authored by Ramon Nou's avatar Ramon Nou
Browse files

harden metrics contracts and disconnect tests

parent 17622406

README.md

+7 −6
Original line number Diff line number Diff line
@@ -971,15 +971,16 @@ collection overhead when unset. Existing command-line options remain supported. 
Periodic snapshots include a wall-clock `timestamp_ms`, daemon `pid`,

`uptime_seconds`, sampled Argobots I/O queue depth (`io_queue_depth`) and total

queued tasks (`io_queue_total`), plus migration job and byte counters. The same

runtime values are exposed to Prometheus in the `GKFS_RUNTIME` gauge family

with a low-cardinality `metric` label.

current runtime values are exposed in the `GKFS_RUNTIME` gauge family with a

low-cardinality `metric` label. Cumulative RPC/backend events use

`GKFS_RUNTIME_TOTAL`; RPC and backend latencies use

`GKFS_RUNTIME_LATENCY_US` summaries.



File and daemon-log snapshots contain one `METRICS_JSON { ... }` record per

sample, followed by the readable statistics report. The JSON record includes

RPC request/error totals and latency maxima, backend read/write/error/retry

totals and latency maxima, migration progress, queue depth, and cache fields.

Client MessagePack metrics additionally report `cache_hits` and

`cache_misses` for client-side dentry-cache activity.

RPC request/error totals and latency maxima, backend

read/write/error/retry totals and latency maxima, migration progress, and queue

depth. Legacy client MessagePack/libzmq metrics remain unchanged.



### Client

#### Core

WIP.md

+5 −5
Original line number Diff line number Diff line
@@ -37,8 +37,8 @@ documentation, or experimental work. 


### 1. Prevent daemon termination from one failed client RPC



**Status: Partial.** Core response containment is implemented; structured error

metadata, rate limiting, and full disconnect coverage remain.

**Status: Partial.** Core response containment and basic rate limiting are

implemented; peer/request-ID metadata and full disconnect coverage remain.



**Problem:** A client disappearing during an RPC can cause an exception while

responding and terminate the daemon. One failed client or resize must not take
@@ -313,9 +313,9 @@ readiness and richer lifecycle/health reporting remain future work. 
explicitly startup-opt-in through `GKFS_DAEMON_*` environment variables.

Timestamped snapshots now include sampled I/O queue depth and migration

counters in file/log output and Prometheus gauges. RPC and backend error,

retry, and latency aggregates are included, and client MessagePack metrics

include dentry-cache hits/misses. Broader per-operation metrics and distributed

tracing remain.

retry, and latency aggregates are included. Legacy client MessagePack/libzmq

metrics remain unchanged. Broader per-operation metrics and distributed tracing

remain.



Define stable, low-cardinality metrics for operation count/bytes/latency/errors;

RPC queues, retries, and timeouts; backend latency and cache hits; migration and

docs/sphinx/users/running.md

+6 −4
Original line number Diff line number Diff line
@@ -300,12 +300,14 @@ Set `GKFS_DAEMON_MIGRATION_CHECKPOINT=ON` to enable atomic per-daemon migration 
checkpoint files; this is startup-only and disabled by default.



Snapshots include `timestamp_ms`, `pid`, `uptime_seconds`, sampled I/O queue

depth/total, and migration job/byte counters. Prometheus exposes the runtime

values as `GKFS_RUNTIME{metric="..."}` gauges.

depth/total, and migration job/byte counters. Prometheus exposes current

runtime values as `GKFS_RUNTIME{metric="..."}` gauges, cumulative RPC/backend

events as `GKFS_RUNTIME_TOTAL`, and latency observations as

`GKFS_RUNTIME_LATENCY_US` summaries.

File and daemon-log snapshots also contain a `METRICS_JSON { ... }` record with

RPC latency/errors, backend latency/errors/retries, and migration fields before

the human-readable report. Client MessagePack output includes cache hit/miss

counters.

the human-readable report. Legacy client MessagePack/libzmq metrics remain

unchanged.



### Advanced experimental features

include/common/rpc/handler_util.hpp

+12 −3
Original line number Diff line number Diff line
@@ -26,7 +26,10 @@ safe_respond(RequestType& req, const ResponseType& resp, 
        const auto failure = transport_failures.fetch_add(1) + 1;

        if(failure == 1 || failure % 100 == 0) {

            try {

                auto logger = spdlog::get("daemon");

                auto logger = spdlog::get("main");

                if(!logger) {

                    logger = spdlog::default_logger();

                }

                if(logger) {

                    logger->warn(

                            "rpc_response_failed=1 rpc_context={} "
@@ -40,7 +43,10 @@ safe_respond(RequestType& req, const ResponseType& resp, 
        return false;

    } catch(const std::exception& e) {

        try {

            auto logger = spdlog::get("daemon");

            auto logger = spdlog::get("main");

            if(!logger) {

                logger = spdlog::default_logger();

            }

            if(logger) {

                logger->error("rpc_response_failed=1 rpc_context={} "

                              "failure_kind=unexpected cause='{}'",
@@ -51,7 +57,10 @@ safe_respond(RequestType& req, const ResponseType& resp, 
        return false;

    } catch(...) {

        try {

            auto logger = spdlog::get("daemon");

            auto logger = spdlog::get("main");

            if(!logger) {

                logger = spdlog::default_logger();

            }

            if(logger) {

                logger->error("rpc_response_failed=1 rpc_context={} "

                              "failure_kind=unknown",

include/common/statistics/stats.hpp

+2 −9
Original line number Diff line number Diff line
@@ -101,8 +101,6 @@ struct RuntimeMetrics { 
    uint64_t backend_writes{0};

    uint64_t backend_errors{0};

    uint64_t backend_retries{0};

    uint64_t cache_hits{0};

    uint64_t cache_misses{0};

};



/**
@@ -194,12 +192,10 @@ private: 
    std::atomic<uint64_t> backend_writes_{0};

    std::atomic<uint64_t> backend_errors_{0};

    std::atomic<uint64_t> backend_retries_{0};

    std::atomic<uint64_t> cache_hits_{0};

    std::atomic<uint64_t> cache_misses_{0};





    bool running =

            true; ///< Controls the destruction of the class/stops the thread

    std::atomic<bool> running{

            true}; ///< Controls the destruction of the class/stops the thread

    /**

     * @brief Sends all the stats to the screen

     * Debug Function
@@ -246,9 +242,6 @@ public: 
    record_backend_write(bool error, uint64_t retries = 0,

                         uint64_t latency_us = 0);



    void

    record_cache(bool hit);



private:

// Prometheus Push structs

#ifdef GKFS_ENABLE_PROMETHEUS