Commit f077366c authored by Ramon Nou's avatar Ramon Nou
Browse files

Defines to Options

parent 07bb4d4a

CHANGELOG.md

+4 −2
Original line number Diff line number Diff line
@@ -11,15 +11,17 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 
### New



- Added Stats ([!132](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/132)) gathering in servers

  - GKFS_CHUNK_STATS enables chunk usage output

  - Stats output can be enabled with --output-stats <filename>

  - --enable-collection collects normal stats

  - --enable-chunkstats collects extended chunk stats

- Added new experimental metadata backend:

  Parallax ([!110](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/110)).

    - Added support to use multiple metadata backends.

    - Added `--clean-rootdir-finish` argument to remove rootdir/metadir at the end when the daemon finishes.

- Added Prometheus Ouput ([!132](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/132))

- Added Prometheus Output ([!132](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/132))

  - New option to define gateway --prometheus-gateway <gateway:port>

  - Prometheus output is optional with "GKFS_ENABLE_PROMETHEUS"

  - --enable-prometheus creates a thread to push the metrics.



### Changed

CMakeLists.txt

+0 −6
Original line number Diff line number Diff line
@@ -195,12 +195,6 @@ if(GKFS_USE_GUIDED_DISTRIBUTION) 
    message(STATUS "[gekkofs] Guided data distributor input file path: ${GKFS_USE_GUIDED_DISTRIBUTION_PATH}")

endif()



option(GKFS_CHUNK_STATS "Gather Chunk Stats " OFF)

if (GKFS_CHUNK_STATS)

    add_definitions(-DGKFS_CHUNK_STATS)

endif ()

message(STATUS "[gekkofs] Gather Chunk Stats: ${GKFS_CHUNK_STATS}")



option(GKFS_ENABLE_PROMETHEUS "Enable Prometheus Push " OFF)

if(GKFS_ENABLE_PROMETHEUS)

    add_definitions(-DGKFS_ENABLE_PROMETHEUS)

README.md

+2 −4
Original line number Diff line number Diff line
@@ -240,10 +240,8 @@ To enable it use the `-DGKFS_ENABLE_PARALLAX:BOOL=ON` option, you can also disab 
Once it is enabled, `--dbbackend` option will be functional.



### Stats

Pushing stats to Prometheus is enabled with the `-DGKFS_ENABLE_PROMETHEUS` and the setup of the `--output-stats <FILE>`. 

Without the last one, the push to the gateway is disabled.

Stats for each chunk (read-write access) can be enabled with `-DGKFS_CHUNK_STATS`. The server will store file/chunk number stats.



Stats from each server are written to the file specified with `--output-stats <FILE>`. Collection is done with two separate flags `--enable-collection` and `--enable-chunkstats`. For normal and extended chunk stats. The extended chunk stats stores each chunk acccess. 

Pushing stats to Prometheus is enabled with the `-DGKFS_ENABLE_PROMETHEUS` and the flag `--enable-prometheus`. We are using a push model.



### Acknowledgment

include/common/statistics/stats.hpp

+10 −5
Original line number Diff line number Diff line
@@ -37,6 +37,7 @@ 
#include <vector>

#include <deque>

#include <chrono>

#include <optional>

#include <initializer_list>

#include <thread>

#include <iostream>
@@ -135,6 +136,9 @@ private: 


    std::thread t_output;    ///< Thread that outputs stats info

    bool output_thread_;     ///< Enables or disables the output thread

    bool enable_prometheus_; ///< Enables or disables the prometheus output

    bool enable_chunkstats_; ///< Enables or disables the chunk stats output





    bool running =

            true; ///< Controls the destruction of the class/stops the thread
@@ -185,12 +189,13 @@ private: 
public:

    /**

     * @brief Starts the Stats module and initializes structures

     * @param output_thread creates an aditional thread that outputs the stats

     * @param enable_chunkstats Enables or disables the chunk stats

     * @param enable_prometheus Enables or disables the prometheus output

     * @param filename file where to write the output

     * @param prometheus_gateway ip:port to expose the metrics

     */

    Stats(bool output_thread, const std::string& filename,

          const std::string& prometheus_gateway);

    Stats(bool enable_chunkstats, bool enable_prometheus,

          const std::string& filename, const std::string& prometheus_gateway);



    /**

     * @brief Destroys the class, and any associated thread

include/daemon/classes/fs_data.hpp

+17 −3
Original line number Diff line number Diff line
@@ -92,7 +92,9 @@ private: 


    // Statistics

    std::shared_ptr<gkfs::utils::Stats> stats_;

    bool output_stats_ = false;

    bool enable_stats_ = false;

    bool enable_chunkstats_ = false;

    bool enable_prometheus_ = false;

    std::string stats_file_;



    // Prometheus
@@ -233,10 +235,22 @@ public: 
    close_stats();



    bool

    output_stats() const;

    enable_stats() const;



    void

    output_stats(bool output_stats);

    enable_stats(bool enable_stats);



    bool

    enable_chunkstats() const;



    void

    enable_chunkstats(bool enable_chunkstats);



    bool

    enable_prometheus() const;



    void

    enable_prometheus(bool enable_prometheus);



    const std::string&

    stats_file() const;