1. Jun 20, 2024
    • Marc Vef's avatar
      Merge branch 'marc/271-client-support-messagepack-for-process-bandwidth-monitoring' into 'master' · 3a7c00de
      Marc Vef authored
      Resolve "Client: Support MessagePack for process bandwidth monitoring"
      GekkoFS clients support capturing the I/O traces of each individual process and periodically exporting them to a given
      file or ZeroMQ sink via the TCP protocol.
      To use this feature, the corresponding ZeroMQ (`libzmq` and `cppzmq`) dependencies are required which can be found in
      the `default_zmq` dependency profile.
      In addition, GekkoFS must be compiled with client metrics enabled (disabled by default) via the CMake argument
      Client metrics are individually enabled per GekkoFS client process via the following environment variables:
      - `LIBGKFS_ENABLE_METRICS=ON` enables capturing client-side metrics.
      - `LIBGKFS_METRICS_FLUSH_INTERVAL=10` sets the flush interval to 10 seconds (defaults to 5). All outstanding client
        metrics are flushed when the process ends.
      - `LIBGKFS_METRICS_PATH=<path>` sets the path to flush client-metrics (defaults to `/tmp/gkfs_client_metrics`).
      - `LIBGKFS_METRICS_IP_PORT=` enables flushing to a set ZeroMQ server. This option disables flushing to a
      The ZeroMQ export can be tested via the `gkfs_clientmetrics2json` application which is built when enabling the CMake
      option `-DGKFS_BUILD_TOOLS=ON`:
      - Starting the ZeroMQ server: `gkfs_clientmetrics2json tcp://`
      - `gkfs_clientmetrics2json <path>` can also be used to unpack the Messagepack export from a file.
        Examplarily output with the ZeroMQ sink enabled when running:
        `LD_PRELOAD=libgkfs_intercept.so LIBGKFS_ENABLE_METRICS=ON LIBGKFS_METRICS_IP_PORT= gkfs cp testfile /tmp/gkfs_mountdir/testfile`:
      ~ $ gkfs_clientmetrics2json tcp://
      Binding to: tcp://
      Waiting for message...
      Received message with size 68
      Generated JSON:
      [extra]avg_thruput_mib: [221.93,175.87,266.81,135.69]
      end_t_micro: [8008,12396,16006,18454]
      flush_t: 18564
      hostname: "evie"
      io_type: "w"
      pid: 1259304
      req_size: [524288,524288,524288,229502]
      start_t_micro: [5755,9553,14132,16841]
      total_bytes: 1802366
      total_iops: 4
      Closes #271
      Closes #271
      See merge request !176
  2. Jun 14, 2024
  3. Jun 11, 2024
  4. Jun 09, 2024