Commit 0ade4da6 authored by Ramon Nou's avatar Ramon Nou
Browse files

Added ondemand calculation

parent 371d9d29

CHANGELOG.md

+1 −0
Original line number Diff line number Diff line
@@ -56,6 +56,7 @@ replicas ([!166](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/141) 
- Modified write and reads to use a bitset instead of the traditional hash per chunk in the server.

- Added reattemp support in get_fs_config to other servers, when the initial server fails.

- Added support for Erasure codes ([!168]) using Jerasure lib and adding support for Read error injection.

- Added support for on demand erasure codes calculation with `gkfs_ec_ondemand(fd)`

### New



- Support for client-side per process logging, activated

README.md

+4 −0
Original line number Diff line number Diff line
@@ -384,6 +384,10 @@ The total servers available for data will be -> total servervs - num_repl 
Additionaly the user can enable the `-DGKFS_ENABLE_READ_ERRORS` to inject a  

50% of read errors into the workflow.



We also included a function to call the EC calculation on demand over a open file:

`gkfs_ec_ondemand(int fd)`, to use it requires the environment variable `LIBGKFS_EC_ONDEMAND=1`



The function is useful to calculate the ec only when the file is completed, like checkpoints.

## Acknowledgment



This software was partially supported by the EC H2020 funded NEXTGenIO project (Project ID: 671951, www.nextgenio.eu).

include/client/env.hpp

+3 −0
Original line number Diff line number Diff line
@@ -59,6 +59,9 @@ static constexpr auto METRICS_IP_PORT = ADD_PREFIX("METRICS_IP_PORT"); 
#endif



static constexpr auto NUM_REPL = ADD_PREFIX("NUM_REPL");

static constexpr auto EC_ONDEMAND = ADD_PREFIX("EC_ONDEMAND");





} // namespace gkfs::env



#undef ADD_PREFIX

include/client/gkfs_functions.hpp

+6 −0
Original line number Diff line number Diff line
@@ -175,4 +175,10 @@ extern "C" int 
gkfs_getsingleserverdir(const char* path, struct dirent_extended* dirp,

                        unsigned int count, int server);





#ifdef GKFS_ENABLE_EC

extern "C" int

gkfs_ec_ondemand(const unsigned int fd);

#endif



#endif // GEKKOFS_GKFS_FUNCTIONS_HPP

include/client/preload_context.hpp

+6 −0
Original line number Diff line number Diff line
@@ -111,6 +111,7 @@ private: 
    int replicas_;

    std::shared_ptr<gkfs::messagepack::ClientMetrics> write_metrics_;

    std::shared_ptr<gkfs::messagepack::ClientMetrics> read_metrics_;

    bool ec_ondemand_;



public:

    static PreloadContext*
@@ -240,6 +241,11 @@ public: 
    const std::shared_ptr<gkfs::messagepack::ClientMetrics>

    read_metrics();



    bool

    get_ec_ondemand();



    void

    set_ec_ondemand(const bool ec_ondemand);

};



} // namespace preload