Commit ac1f2de3 authored by Ramon Nou's avatar Ramon Nou
Browse files

Added GKFS_ENABLE_EC 

First Write - ECC

cout -> LOG

WIP Decode EC to restore chunk

read recovery WIP
parent 894a14c8

CHANGELOG.md

+1 −1
Original line number Diff line number Diff line
@@ -55,7 +55,7 @@ replicas ([!166](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/141) 
replicas ([!166](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/141)).

- Modified write and reads to use a bitset instead of the traditional hash per chunk in the server.

- Added reattemp support in get_fs_config to other servers, when the initial server fails.



- Added support for Erasure codes ([!1xx])

### New



- Support for client-side per process logging, activated

CMake/gkfs-options.cmake

+11 −1
Original line number Diff line number Diff line
@@ -364,9 +364,19 @@ gkfs_define_option( 
)



################################################################################

# MSGPack client metrics

# Erasure codes

################################################################################



gkfs_define_option(

  GKFS_ENABLE_EC

  HELP_TEXT "Enable Erasure Code"

  DEFAULT_VALUE OFF

  DESCRIPTION "Use Jerasure for erasure codes reliability"

)



################################################################################

# MSGPack client metrics

################################################################################

gkfs_define_option(

    GKFS_ENABLE_CLIENT_METRICS

    HELP_TEXT "Enable client metrics via MSGPack"

CMakeLists.txt

+8 −2
Original line number Diff line number Diff line
@@ -206,8 +206,14 @@ if(GKFS_ENABLE_PROMETHEUS) 
    find_package(prometheus-cpp REQUIRED) # >= 1.0.0

endif()



find_package(GF_complete)

find_package(Jerasure)

### Jerasure: needed for the calculation of Erasure codes

if(GKFS_ENABLE_EC)

    message(STATUS "[${PROJECT_NAME}] Checking for Jerasure")

    add_compile_definitions(GKFS_ENABLE_EC)

    find_package(GF_complete REQUIRED)

    find_package(Jerasure REQUIRED)

endif()



### Other stuff that can be found out using find_package:



# determine the thread library of the system

README.md

+5 −0
Original line number Diff line number Diff line
@@ -376,6 +376,11 @@ The user can enable the data replication feature by setting the replication envi 
The number of replicas should go from 0 to the number of servers-1. 

The replication environment variable can be set up for each client, independently.



### Erasure codes

The user can enable resilience with erasure codes with -DGKFS_ENABLE_EC

Using `LIBGKFS_NUM_REPL=<num repl>`, the user can define the number of EC servers.

The total servers available for data will be -> total servervs - num_repl



## Acknowledgment



This software was partially supported by the EC H2020 funded NEXTGenIO project (Project ID: 671951, www.nextgenio.eu).

include/client/rpc/forward_data.hpp

+10 −7
Original line number Diff line number Diff line
@@ -43,6 +43,9 @@ struct ChunkStat { 


// TODO once we have LEAF, remove all the error code returns and throw them as

// an exception.

std::pair<int, ssize_t>

ecc_forward_write(const std::string& path, const void* buf, const size_t write_size,

               const int8_t server);

               

std::pair<int, ssize_t>

forward_write(const std::string& path, const void* buf, off64_t offset,
@@ -59,12 +62,12 @@ forward_truncate(const std::string& path, size_t current_size, size_t new_size, 


std::pair<int, ChunkStat>

forward_get_chunk_stat();

#define GKFS_USE_ECC_DISTRIBUTION 1

#ifdef GKFS_USE_ECC_DISTRIBUTION

std::pair <uint64_t, uint64_t > calc_op_chunks(const std::string& path, const bool append_flag,



std::pair<uint64_t, uint64_t>

calc_op_chunks(const std::string& path, const bool append_flag,

               const off64_t in_offset, const size_t write_size,

               const int64_t updated_metadentry_size);

#endif





} // namespace gkfs::rpc