1. Jun 20, 2024
    • Marc Vef's avatar
      Merge branch 'marc/271-client-support-messagepack-for-process-bandwidth-monitoring' into 'master' · 3a7c00de
      Marc Vef authored
      Resolve "Client: Support MessagePack for process bandwidth monitoring"
      
      GekkoFS clients support capturing the I/O traces of each individual process and periodically exporting them to a given
      file or ZeroMQ sink via the TCP protocol.
      To use this feature, the corresponding ZeroMQ (`libzmq` and `cppzmq`) dependencies are required which can be found in
      the `default_zmq` dependency profile.
      In addition, GekkoFS must be compiled with client metrics enabled (disabled by default) via the CMake argument
      `-DGKFS_ENABLE_CLIENT_METRICS=ON`.
      
      Client metrics are individually enabled per GekkoFS client process via the following environment variables:
      
      - `LIBGKFS_ENABLE_METRICS=ON` enables capturing client-side metrics.
      - `LIBGKFS_METRICS_FLUSH_INTERVAL=10` sets the flush interval to 10 seconds (defaults to 5). All outstanding client
        metrics are flushed when the process ends.
      - `LIBGKFS_METRICS_PATH=<path>` sets the path to flush client-metrics (defaults to `/tmp/gkfs_client_metrics`).
      - `LIBGKFS_METRICS_IP_PORT=127.0.0.1:5555` enables flushing to a set ZeroMQ server. This option disables flushing to a
        file.
      
      The ZeroMQ export can be tested via the `gkfs_clientmetrics2json` application which is built when enabling the CMake
      option `-DGKFS_BUILD_TOOLS=ON`:
      
      - Starting the ZeroMQ server: `gkfs_clientmetrics2json tcp://127.0.0.1:5555`
      - `gkfs_clientmetrics2json <path>` can also be used to unpack the Messagepack export from a file.
        Examplarily output with the ZeroMQ sink enabled when running:
        `LD_PRELOAD=libgkfs_intercept.so LIBGKFS_ENABLE_METRICS=ON LIBGKFS_METRICS_IP_PORT=127.0.0.1:5555 gkfs cp testfile /tmp/gkfs_mountdir/testfile`:
      
      ```bash
      ~ $ gkfs_clientmetrics2json tcp://127.0.0.1:5555
      Binding to: tcp://127.0.0.1:5555
      Waiting for message...
      
      Received message with size 68
      Generated JSON:
      [extra]avg_thruput_mib: [221.93,175.87,266.81,135.69]
      end_t_micro: [8008,12396,16006,18454]
      flush_t: 18564
      hostname: "evie"
      io_type: "w"
      pid: 1259304
      req_size: [524288,524288,524288,229502]
      start_t_micro: [5755,9553,14132,16841]
      total_bytes: 1802366
      total_iops: 4
      ```
      
      Closes #271
      
      Closes #271
      
      See merge request !176
      3a7c00de
  2. Jun 14, 2024
  3. Jun 11, 2024
  4. Jun 09, 2024