Push stats to prometheus

Ramon Nourequested to merge
rnou/stats_prometheus into master

This MR pushes the stats to Prometheus, a pull model can easily be implemented.

It is a follow-up of the !128 (closed) MR :

This MR will provide the mechanism to store and check stats inside GekkoFS

There are two types of Stats: Number of operations (i.e., Create) and Operations with Size (i.e. write / read )

The stats are stored to allow the calculation of averages (and other stats) total, 1 min, 5 min and 10 minutes.

Other stats included: File-Chunk more accessed (write-read), using GKFS_CHUNK_STATS in the CMAKE. The structures can show the next info: accesses -- <file> // Chunk_id.

READ CHUNK MAP
1 -- /top/file_a // 4
2 -- /top/file_a // 3
3 -- /top/file_a // 1
/top/file_a // 2
5 -- /top/file_a // 0
WRITE CHUNK MAP
1 -- /top/file_a // 2
3 -- /top/file_a // 1
8 -- /top/file_a // 0

There is a thread for output the data each 10s to the console. The thread can be activated using a command line option, but the normal behaviour should be to allow an external RPC to gather the stats (i.e., a monitoring system).

Using --output-stats:

Stats IOPS_CREATE IOPS/s (avg, 1 min, 5 min, 10 min)                91.66 -         0 -         0 -         0 - 
Stats IOPS_WRITE IOPS/s (avg, 1 min, 5 min, 10 min)                 45.83 -         0 -         0 -         0 - 
Stats IOPS_READ IOPS/s (avg, 1 min, 5 min, 10 min)                  45.83 -         0 -         0 -         0 - 
Stats IOPS_STATS IOPS/s (avg, 1 min, 5 min, 10 min)                 320.8 -         0 -         0 -         0 - 
Stats IOPS_DIRENTS IOPS/s (avg, 1 min, 5 min, 10 min)               45.83 -         0 -         0 -         0 - 
Stats IOPS_REMOVE IOPS/s (avg, 1 min, 5 min, 10 min)                91.66 -         0 -         0 -         0 - 
Stats WRITE_SIZE MB/s (avg, 1 min, 5 min, 10 min)                  0.1353 -         0 -         0 -         0 - 
Stats READ_SIZE MB/s (avg, 1 min, 5 min, 10 min)                   0.1353 -         0 -         0 -         0 -
Edited by Ramon Nou

Merge request reports

docker/0.9.1/deps/Dockerfile

+1 −1
Changes for docker/0.9.1/deps/Dockerfile: 1 added line, 1 removed line.
Original line number Diff line number Diff line
@@ -21,7 +21,7 @@ RUN apt-get update && \ 
		python3-dev \

		python3-venv \

		python3-setuptools \

    libnuma-dev libyaml-dev \

    libnuma-dev libyaml-dev libcurl4-openssl-dev \

    procps && \

    python3 -m pip install --upgrade pip && \

    rm -rf /var/lib/apt/lists/* && \

docs/sphinx/users/running.md

+5 −0
Changes for docs/sphinx/users/running.md: 5 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -79,6 +79,11 @@ Options: 
                              RocksDB is default if not set. Parallax support is experimental.

                              Note, parallaxdb creates a file called rocksdbx with 8GB created in metadir.

  --parallaxsize TEXT         parallaxdb - metadata file size in GB (default 8GB), used only with new files

  --enable-collection         Enables collection of general statistics. Output requires either the --output-stats or --enable-prometheus argument.

  --enable-chunkstats         Enables collection of data chunk statistics in I/O operations.Output requires either the --output-stats or --enable-prometheus argument.

  --output-stats TEXT         Creates a thread that outputs the server stats each 10s to the specified file.

  --enable-prometheus         Enables prometheus output and a corresponding thread.

  --prometheus-gateway TEXT   Defines the prometheus gateway <ip:port> (Default 127.0.0.1:9091).

  --version                   Print version and exit.

````

include/common/statistics/stats.hpp

0 → 100644
+301 −0
Changes for include/common/statistics/stats.hpp: 301 added lines, 0 removed lines.
Original line number Diff line number Diff line 
/*

  Copyright 2018-2022, Barcelona Supercomputing Center (BSC), Spain

  Copyright 2015-2022, Johannes Gutenberg Universitaet Mainz, Germany



  This software was partially supported by the

  EC H2020 funded project NEXTGenIO (Project ID: 671951, www.nextgenio.eu).



  This software was partially supported by the

  ADA-FS project under the SPPEXA project funded by the DFG.



  This file is part of GekkoFS.



  GekkoFS is free software: you can redistribute it and/or modify

  it under the terms of the GNU General Public License as published by

  the Free Software Foundation, either version 3 of the License, or

  (at your option) any later version.



  GekkoFS is distributed in the hope that it will be useful,

  but WITHOUT ANY WARRANTY; without even the implied warranty of

  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

  GNU General Public License for more details.



  You should have received a copy of the GNU General Public License

  along with GekkoFS.  If not, see <https://www.gnu.org/licenses/>.



  SPDX-License-Identifier: GPL-3.0-or-later

*/



#ifndef GKFS_COMMON_STATS_HPP

#define GKFS_COMMON_STATS_HPP



#include <cstdint>

#include <unistd.h>

#include <cassert>

#include <map>

#include <set>

#include <vector>

#include <deque>

#include <chrono>

#include <optional>

#include <initializer_list>

#include <thread>

#include <iostream>

#include <iomanip>

#include <fstream>

#include <atomic>

#include <mutex>

#include <config.hpp>





// PROMETHEUS includes

#ifdef GKFS_ENABLE_PROMETHEUS

#include <prometheus/counter.h>

#include <prometheus/summary.h>

#include <prometheus/exposer.h>

#include <prometheus/registry.h>

#include <prometheus/gateway.h>



using namespace prometheus;

#endif





/**

 * Provides storage capabilities to provide stats about GekkoFS

 * The information is per server.

 * We do not provide accurate stats for 1-5-10 minute stats

 *

 */

namespace gkfs::utils {



/**

 *

 * Number of operations (Create, write/ read, remove, mkdir...)

 * Size of database (metadata keys, should be not needed, any)

 * Size of data (+write - delete)

 * Server Bandwidth (write / read operations)

 *

 * mean, (lifetime of the server)

 * 1 minute mean

 * 5 minute mean

 * 10 minute mean

 *

 * To provide the stats that we need,

 * we need to store the info and the timestamp to calculate it

 * A vector should work, with a maximum of elements,

 */



class Stats {

public:

    enum class IopsOp {

        iops_create,

        iops_write,

        iops_read,

        iops_stats,

        iops_dirent,

        iops_remove,

    }; ///< enum storing IOPS Stats



    enum class SizeOp { write_size, read_size }; ///< enum storing Size Stats



private:

    constexpr static const std::initializer_list<Stats::IopsOp> all_IopsOp = {

            IopsOp::iops_create, IopsOp::iops_write,

            IopsOp::iops_read,   IopsOp::iops_stats,

            IopsOp::iops_dirent, IopsOp::iops_remove}; ///< Enum IOPS iterator



    constexpr static const std::initializer_list<Stats::SizeOp> all_SizeOp = {

            SizeOp::write_size, SizeOp::read_size}; ///< Enum SIZE iterator



    const std::vector<std::string> IopsOp_s = {

            "IOPS_CREATE", "IOPS_WRITE",   "IOPS_READ",

            "IOPS_STATS",  "IOPS_DIRENTS", "IOPS_REMOVE"}; ///< Stats Labels

    const std::vector<std::string> SizeOp_s = {"WRITE_SIZE",

                                               "READ_SIZE"}; ///< Stats Labels



    std::chrono::time_point<std::chrono::steady_clock>

            start; ///< When we started the server





    std::map<IopsOp, std::atomic<unsigned long>>

            iops_mean; ///< Stores total value for global mean

    std::map<SizeOp, std::atomic<unsigned long>>

            size_mean; ///< Stores total value for global mean



    std::mutex time_iops_mutex;

    std::mutex size_iops_mutex;



    std::map<IopsOp,

             std::deque<std::chrono::time_point<std::chrono::steady_clock>>>

            time_iops; ///< Stores timestamp when an operation comes removes if

                       ///< first operation if > 10 minutes Different means will

                       ///< be stored and cached 1 minuted





    std::map<SizeOp, std::deque<std::pair<

                             std::chrono::time_point<std::chrono::steady_clock>,

                             unsigned long long>>>

            time_size; ///< For size operations we need to store the timestamp

                       ///< and the size





    std::thread t_output;    ///< Thread that outputs stats info

    bool output_thread_;     ///< Enables or disables the output thread

    bool enable_prometheus_; ///< Enables or disables the prometheus output

    bool enable_chunkstats_; ///< Enables or disables the chunk stats output





    bool running =

            true; ///< Controls the destruction of the class/stops the thread

    /**

     * @brief Sends all the stats to the screen

     * Debug Function

     *

     * @param d is the time between output

     * @param file_output is the output file

     */

    void

    output(std::chrono::seconds d, std::string file_output);



    std::map<std::pair<std::string, unsigned long long>,

             std::atomic<unsigned int>>

            chunk_reads; ///< Stores the number of times a chunk/file is read

    std::map<std::pair<std::string, unsigned long long>,

             std::atomic<unsigned int>>

            chunk_writes; ///< Stores the number of times a chunk/file is write



    /**

     * @brief Called by output to generate CHUNK map

     *

     * @param output is the output stream

     */

    void

    output_map(std::ofstream& output);





    /**

     * @brief Dumps all the means from the stats

     * @param of Output stream

     */

    void

    dump(std::ofstream& of);





// Prometheus Push structs

#ifdef GKFS_ENABLE_PROMETHEUS

    std::shared_ptr<Gateway> gateway;   ///< Prometheus Gateway

    std::shared_ptr<Registry> registry; ///< Prometheus Counters Registry

    Family<Counter>* family_counter;    ///< Prometheus IOPS counter (managed by

                                        ///< Prometheus cpp)

    Family<Summary>* family_summary;    ///< Prometheus SIZE counter (managed by

                                        ///< Prometheus cpp)

    std::map<IopsOp, Counter*> iops_prometheus; ///< Prometheus IOPS metrics

    std::map<SizeOp, Summary*> size_prometheus; ///< Prometheus SIZE metrics

#endif



public:

    /**

     * @brief Starts the Stats module and initializes structures

     * @param enable_chunkstats Enables or disables the chunk stats

     * @param enable_prometheus Enables or disables the prometheus output

     * @param filename file where to write the output

     * @param prometheus_gateway ip:port to expose the metrics

     */

    Stats(bool enable_chunkstats, bool enable_prometheus,

          const std::string& filename, const std::string& prometheus_gateway);



    /**

     * @brief Destroys the class, and any associated thread

     *

     */

    ~Stats();





    /**

     * @brief Set the up Prometheus gateway and structures

     *

     * @param gateway_ip ip of the prometheus gateway

     * @param gateway_port port of the prometheus gateway

     */

    void

    setup_Prometheus(const std::string& gateway_ip,

                     const std::string& gateway_port);



    /**

     * @brief Adds a new read access to the chunk/path specified

     *

     * @param path path of the chunk

     * @param chunk chunk number

     */

    void

    add_read(const std::string& path, unsigned long long chunk);

    /**

     * @brief Adds a new write access to the chunk/path specified

     *

     * @param path path of the chunk

     * @param chunk chunk number

     */

    void

    add_write(const std::string& path, unsigned long long chunk);





    /**

     * Add a new value for a IOPS, that does not involve any size

     * No value needed as they are simple (1 create, 1 read...)

     * Size operations internally call this operation (read,write)

     *

     * @param IopsOp Which operation to add

     */



    void add_value_iops(enum IopsOp);



    /**

     * @brief Store a new stat point, with a size value.

     * If it involves a IO operations it will call the corresponding

     * operation

     *

     * @param SizeOp Which operation we refer

     * @param value to store (SizeOp)

     */

    void

    add_value_size(enum SizeOp, unsigned long long value);



    /**

     * @brief Get the total mean value of the asked stat

     * This can be provided inmediately without cost

     * @param IopsOp Which operation to get

     * @return mean value

     */

    double get_mean(enum IopsOp);





    /**

     * @brief Get the total mean value of the asked stat

     * This can be provided inmediately without cost

     * @param SizeOp Which operation to get

     * @return mean value

     */

    double get_mean(enum SizeOp);



    /**

     * @brief Get all the means (total, 1,5 and 10 minutes) for a SIZE_OP

     * Returns precalculated values if we just calculated them 1 minute ago

     * @param SizeOp Which operation to get

     *

     * @return std::vector< double > with 4 means

     */

    std::vector<double> get_four_means(enum SizeOp);



    /**

     * @brief Get all the means (total, 1,5 and 10 minutes) for a IOPS_OP

     * Returns precalculated values if we just calculated them 1 minute ago

     * @param IopsOp Which operation to get

     *

     * @return std::vector< double > with 4 means

     */

    std::vector<double> get_four_means(enum IopsOp);

};



} // namespace gkfs::utils



#endif // GKFS_COMMON_STATS_HPP
 
 No newline at end of file

include/daemon/classes/fs_data.hpp

+55 −0
Changes for include/daemon/classes/fs_data.hpp: 55 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -46,6 +46,11 @@ namespace data { 
class ChunkStorage;

}



/* Forward declarations */

namespace utils {

class Stats;

}



namespace daemon {



class FsData {
@@ -85,6 +90,16 @@ private: 
    bool link_cnt_state_;

    bool blocks_state_;



    // Statistics

    std::shared_ptr<gkfs::utils::Stats> stats_;

    bool enable_stats_ = false;

    bool enable_chunkstats_ = false;

    bool enable_prometheus_ = false;

    std::string stats_file_;



    // Prometheus

    std::string prometheus_gateway_ = gkfs::config::stats::prometheus_gateway;



public:

    static FsData*

    getInstance() {
@@ -209,8 +224,48 @@ public: 


    void

    parallax_size_md(unsigned int size_md);



    const std::shared_ptr<gkfs::utils::Stats>&

    stats() const;



    void

    stats(const std::shared_ptr<gkfs::utils::Stats>& stats);



    void

    close_stats();



    bool

    enable_stats() const;



    void

    enable_stats(bool enable_stats);



    bool

    enable_chunkstats() const;



    void

    enable_chunkstats(bool enable_chunkstats);



    bool

    enable_prometheus() const;



    void

    enable_prometheus(bool enable_prometheus);



    const std::string&

    stats_file() const;



    void

    stats_file(const std::string& stats_file);



    const std::string&

    prometheus_gateway() const;



    void

    prometheus_gateway(const std::string& prometheus_gateway_);

};





} // namespace daemon

} // namespace gkfs

include/config.hpp

+5 −0
Changes for include/config.hpp: 5 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -103,6 +103,11 @@ namespace rocksdb { 
constexpr auto use_write_ahead_log = false;

} // namespace rocksdb



namespace stats {

constexpr auto max_stats = 1000000; ///< How many stats will be stored

constexpr auto prometheus_gateway = "127.0.0.1:9091";

} // namespace stats



} // namespace gkfs::config



#endif // GEKKOFS_CONFIG_HPP