Commit f077366c authored by Ramon Nou's avatar Ramon Nou
Browse files

Defines to Options

parent 07bb4d4a

CHANGELOG.md

+4 −2
Original line number Original line Diff line number Diff line
@@ -11,15 +11,17 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 
### New

### New





- Added Stats ([!132](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/132)) gathering in servers

- Added Stats ([!132](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/132)) gathering in servers

  - GKFS_CHUNK_STATS enables chunk usage output

  - Stats output can be enabled with --output-stats <filename>

  - Stats output can be enabled with --output-stats <filename>

  - --enable-collection collects normal stats

  - --enable-chunkstats collects extended chunk stats

- Added new experimental metadata backend:

- Added new experimental metadata backend:

  Parallax ([!110](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/110)).

  Parallax ([!110](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/110)).

    - Added support to use multiple metadata backends.

    - Added support to use multiple metadata backends.

    - Added `--clean-rootdir-finish` argument to remove rootdir/metadir at the end when the daemon finishes.

    - Added `--clean-rootdir-finish` argument to remove rootdir/metadir at the end when the daemon finishes.

- Added Prometheus Ouput ([!132](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/132))

- Added Prometheus Output ([!132](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/132))

  - New option to define gateway --prometheus-gateway <gateway:port>

  - New option to define gateway --prometheus-gateway <gateway:port>

  - Prometheus output is optional with "GKFS_ENABLE_PROMETHEUS"

  - Prometheus output is optional with "GKFS_ENABLE_PROMETHEUS"

  - --enable-prometheus creates a thread to push the metrics.





### Changed

### Changed

CMakeLists.txt

+0 −6
Original line number Original line Diff line number Diff line
@@ -195,12 +195,6 @@ if(GKFS_USE_GUIDED_DISTRIBUTION) 
    message(STATUS "[gekkofs] Guided data distributor input file path: ${GKFS_USE_GUIDED_DISTRIBUTION_PATH}")

    message(STATUS "[gekkofs] Guided data distributor input file path: ${GKFS_USE_GUIDED_DISTRIBUTION_PATH}")

endif()

endif()





option(GKFS_CHUNK_STATS "Gather Chunk Stats " OFF)

if (GKFS_CHUNK_STATS)

    add_definitions(-DGKFS_CHUNK_STATS)

endif ()

message(STATUS "[gekkofs] Gather Chunk Stats: ${GKFS_CHUNK_STATS}")



option(GKFS_ENABLE_PROMETHEUS "Enable Prometheus Push " OFF)

option(GKFS_ENABLE_PROMETHEUS "Enable Prometheus Push " OFF)

if(GKFS_ENABLE_PROMETHEUS)

if(GKFS_ENABLE_PROMETHEUS)

    add_definitions(-DGKFS_ENABLE_PROMETHEUS)

    add_definitions(-DGKFS_ENABLE_PROMETHEUS)

README.md

+2 −4
Original line number Original line Diff line number Diff line
@@ -240,10 +240,8 @@ To enable it use the `-DGKFS_ENABLE_PARALLAX:BOOL=ON` option, you can also disab 
Once it is enabled, `--dbbackend` option will be functional.

Once it is enabled, `--dbbackend` option will be functional.





### Stats

### Stats

Pushing stats to Prometheus is enabled with the `-DGKFS_ENABLE_PROMETHEUS` and the setup of the `--output-stats <FILE>`. 

Stats from each server are written to the file specified with `--output-stats <FILE>`. Collection is done with two separate flags `--enable-collection` and `--enable-chunkstats`. For normal and extended chunk stats. The extended chunk stats stores each chunk acccess. 

Without the last one, the push to the gateway is disabled.

Pushing stats to Prometheus is enabled with the `-DGKFS_ENABLE_PROMETHEUS` and the flag `--enable-prometheus`. We are using a push model.

Stats for each chunk (read-write access) can be enabled with `-DGKFS_CHUNK_STATS`. The server will store file/chunk number stats.







### Acknowledgment

### Acknowledgment

include/common/statistics/stats.hpp

+10 −5
Original line number Original line Diff line number Diff line
@@ -37,6 +37,7 @@ 
#include <vector>

#include <vector>

#include <deque>

#include <deque>

#include <chrono>

#include <chrono>

#include <optional>

#include <initializer_list>

#include <initializer_list>

#include <thread>

#include <thread>

#include <iostream>

#include <iostream>
@@ -135,6 +136,9 @@ private: 




    std::thread t_output;    ///< Thread that outputs stats info

    std::thread t_output;    ///< Thread that outputs stats info

    bool output_thread_;     ///< Enables or disables the output thread

    bool output_thread_;     ///< Enables or disables the output thread

    bool enable_prometheus_; ///< Enables or disables the prometheus output

    bool enable_chunkstats_; ///< Enables or disables the chunk stats output







    bool running =

    bool running =

            true; ///< Controls the destruction of the class/stops the thread

            true; ///< Controls the destruction of the class/stops the thread
@@ -185,12 +189,13 @@ private: 
public:

public:

    /**

    /**

     * @brief Starts the Stats module and initializes structures

     * @brief Starts the Stats module and initializes structures

     * @param output_thread creates an aditional thread that outputs the stats

     * @param enable_chunkstats Enables or disables the chunk stats

     * @param enable_prometheus Enables or disables the prometheus output

     * @param filename file where to write the output

     * @param filename file where to write the output

     * @param prometheus_gateway ip:port to expose the metrics

     * @param prometheus_gateway ip:port to expose the metrics

     */

     */

    Stats(bool output_thread, const std::string& filename,

    Stats(bool enable_chunkstats, bool enable_prometheus,

          const std::string& prometheus_gateway);

          const std::string& filename, const std::string& prometheus_gateway);





    /**

    /**

     * @brief Destroys the class, and any associated thread

     * @brief Destroys the class, and any associated thread

include/daemon/classes/fs_data.hpp

+17 −3
Original line number Original line Diff line number Diff line
@@ -92,7 +92,9 @@ private: 




    // Statistics

    // Statistics

    std::shared_ptr<gkfs::utils::Stats> stats_;

    std::shared_ptr<gkfs::utils::Stats> stats_;

    bool output_stats_ = false;

    bool enable_stats_ = false;

    bool enable_chunkstats_ = false;

    bool enable_prometheus_ = false;

    std::string stats_file_;

    std::string stats_file_;





    // Prometheus

    // Prometheus
@@ -233,10 +235,22 @@ public: 
    close_stats();

    close_stats();





    bool

    bool

    output_stats() const;

    enable_stats() const;





    void

    void

    output_stats(bool output_stats);

    enable_stats(bool enable_stats);



    bool

    enable_chunkstats() const;



    void

    enable_chunkstats(bool enable_chunkstats);



    bool

    enable_prometheus() const;



    void

    enable_prometheus(bool enable_prometheus);





    const std::string&

    const std::string&

    stats_file() const;

    stats_file() const;