Resolve "File system expansion during runtime"

Marc Vefrequested to merge
marc/294-file-system-expansion-during-runtime into master

Description

GekkoFS supports extending the current daemon configuration to additional compute nodes. This includes redistribution of the existing data and metadata and therefore scales file system performance and capacity of existing data. Note, that it is the user's responsibility to not access the GekkoFS file system during redistribution. A corresponding feature that is transparent to the user is planned. Note also, if the GekkoFS proxy is used, they need to be manually restarted, after expansion.

To enable this feature, the following CMake compilation flags are required to build the gkfs_malleability tool: -DGKFS_BUILD_TOOLS=ON. The gkfs_malleability tool is then available in the build/tools directory. Please consult -h for its arguments. While the tool can be used manually to expand the file system, the scripts/run/gkfs script should be used instead which invokes the gkfs_malleability tool.

The only requirement for extending the file system is a hostfile containing the hostnames/IPs of the new nodes (one line per host). Example starting the file system. The DAEMON_NODELIST in the gkfs.conf is set to a hostfile containing the initial set of file system nodes.:

~/gekkofs/scripts/run/gkfs -c ~/run/gkfs_verbs_expandtest.conf start
* [gkfs] Starting GekkoFS daemons (4 nodes) ...
* [gkfs] GekkoFS daemons running
* [gkfs] Startup time: 10.853 seconds

... Some computation ...

Expanding the file system. Using -e <hostfile> to specify the new nodes. Redistribution is done automatically with a progress bar. When finished, the file system is ready to use in the new configuration:

~/gekkofs/scripts/run/gkfs -c ~/run/gkfs_verbs_expandtest.conf -e ~/hostfile_expand expand
* [gkfs] Starting GekkoFS daemons (8 nodes) ...
* [gkfs] GekkoFS daemons running
* [gkfs] Startup time: 1.058 seconds
Expansion process from 4 nodes to 12 nodes launched...
* [gkfs] Expansion progress:
[####################] 0/4 left
* [gkfs] Redistribution process done. Finalizing ...
* [gkfs] Expansion done.

Stop the file system:

~/gekkofs/scripts/run/gkfs -c ~/run/gkfs_verbs_expandtest.conf stop
* [gkfs] Stopping daemon with pid 16462
srun: sending Ctrl-C to StepId=282378.1
* [gkfs] Stopping daemon with pid 16761
srun: sending Ctrl-C to StepId=282378.2
* [gkfs] Shutdown time: 1.032 seconds

Results

IOR results for writing/reading 768 GiB sequentially (192 procs) before and after expansion

image

MDTest results for creating, stating, removing, 19200000 (192 procs) before and after expansion

image

Closes #294 (closed)

Edited by Marc Vef

Merge request reports

include/client/rpc/forward_malleability.hpp

0 → 100644
+45 −0
Changes for include/client/rpc/forward_malleability.hpp: 45 added lines, 0 removed lines.
Original line number Diff line number Diff line 
/*

  Copyright 2018-2024, Barcelona Supercomputing Center (BSC), Spain

  Copyright 2015-2024, Johannes Gutenberg Universitaet Mainz, Germany



  This software was partially supported by the

  EC H2020 funded project NEXTGenIO (Project ID: 671951, www.nextgenio.eu).



  This software was partially supported by the

  ADA-FS project under the SPPEXA project funded by the DFG.



  This file is part of GekkoFS' POSIX interface.



  GekkoFS' POSIX interface is free software: you can redistribute it and/or

  modify it under the terms of the GNU Lesser General Public License as

  published by the Free Software Foundation, either version 3 of the License,

  or (at your option) any later version.



  GekkoFS' POSIX interface is distributed in the hope that it will be useful,

  but WITHOUT ANY WARRANTY; without even the implied warranty of

  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

  GNU Lesser General Public License for more details.



  You should have received a copy of the GNU Lesser General Public License

  along with GekkoFS' POSIX interface.  If not, see

  <https://www.gnu.org/licenses/>.



  SPDX-License-Identifier: LGPL-3.0-or-later

*/



#ifndef GEKKOFS_CLIENT_FORWARD_MALLEABILITY_HPP

#define GEKKOFS_CLIENT_FORWARD_MALLEABILITY_HPP



namespace gkfs::malleable::rpc {



int

forward_expand_start(int old_server_conf, int new_server_conf);



int

forward_expand_status();



int

forward_expand_finalize();

} // namespace gkfs::malleable::rpc



#endif // GEKKOFS_CLIENT_FORWARD_MALLEABILITY_HPP

include/client/rpc/rpc_types.hpp

+326 −2
Changes for include/client/rpc/rpc_types.hpp: 326 added lines, 2 removed lines.
Original line number Diff line number Diff line
@@ -63,7 +63,9 @@ hg_proc_void_t(hg_proc_t proc, void* data) { 


} // namespace hermes::detail



namespace gkfs::rpc {

namespace gkfs {



namespace rpc {



//==============================================================================

// definitions for fs_config
@@ -3693,8 +3695,330 @@ struct get_dirents_extended_proxy { 
        size_t m_dirents_size;

    };

};

} // namespace rpc

namespace malleable::rpc {



//==============================================================================

// definitions for expand_start

struct expand_start {



    // forward declarations of public input/output types for this RPC

    class input;



    class output;



    // traits used so that the engine knows what to do with the RPC

    using self_type = expand_start;

    using handle_type = hermes::rpc_handle<self_type>;

    using input_type = input;

    using output_type = output;

    using mercury_input_type = rpc_expand_start_in_t;

    using mercury_output_type = rpc_err_out_t;



    // RPC public identifier

    // (N.B: we reuse the same IDs assigned by Margo so that the daemon

    // understands Hermes RPCs)

    constexpr static const uint64_t public_id = 50;



    // RPC internal Mercury identifier

    constexpr static const hg_id_t mercury_id = 0;



    // RPC name

    constexpr static const auto name = gkfs::malleable::rpc::tag::expand_start;



    // requires response?

    constexpr static const auto requires_response = true;



    // Mercury callback to serialize input arguments

    constexpr static const auto mercury_in_proc_cb =

            HG_GEN_PROC_NAME(rpc_expand_start_in_t);



    // Mercury callback to serialize output arguments

    constexpr static const auto mercury_out_proc_cb =

            HG_GEN_PROC_NAME(rpc_err_out_t);



    class input {



        template <typename ExecutionContext>

        friend hg_return_t

        hermes::detail::post_to_mercury(ExecutionContext*);



    public:

        input(const uint32_t old_server_conf, uint32_t new_server_conf)

            : m_old_server_conf(old_server_conf),

              m_new_server_conf(new_server_conf) {}



        input(input&& rhs) = default;



        input(const input& other) = default;



        input&

        operator=(input&& rhs) = default;



        input&

        operator=(const input& other) = default;



        uint32_t

        old_server_conf() const {

            return m_old_server_conf;

        }



        uint32_t

        new_server_conf() const {

            return m_new_server_conf;

        }



        explicit input(const rpc_expand_start_in_t& other)

            : m_old_server_conf(other.old_server_conf),

              m_new_server_conf(other.new_server_conf) {}



        explicit operator rpc_expand_start_in_t() {

            return {m_old_server_conf, m_new_server_conf};

        }



    private:

        uint32_t m_old_server_conf;

        uint32_t m_new_server_conf;

    };



    class output {



        template <typename ExecutionContext>

        friend hg_return_t

        hermes::detail::post_to_mercury(ExecutionContext*);



    public:

        output() : m_err() {}



        output(int32_t err) : m_err(err) {}



        output(output&& rhs) = default;



        output(const output& other) = default;



        output&

        operator=(output&& rhs) = default;



        output&

        operator=(const output& other) = default;



        explicit output(const rpc_err_out_t& out) {

            m_err = out.err;

        }



        int32_t

        err() const {

            return m_err;

        }



    private:

        int32_t m_err;

    };

};



//==============================================================================

// definitions for expand_status

struct expand_status {



    // forward declarations of public input/output types for this RPC

    class input;



    class output;



    // traits used so that the engine knows what to do with the RPC

    using self_type = expand_status;

    using handle_type = hermes::rpc_handle<self_type>;

    using input_type = input;

    using output_type = output;

    using mercury_input_type = hermes::detail::hg_void_t;

    using mercury_output_type = rpc_err_out_t;



    // RPC public identifier

    // (N.B: we reuse the same IDs assigned by Margo so that the daemon

    // understands Hermes RPCs)

    constexpr static const uint64_t public_id = 51;



    // RPC internal Mercury identifier

    constexpr static const hg_id_t mercury_id = 0;



    // RPC name

    constexpr static const auto name = gkfs::malleable::rpc::tag::expand_status;



    // requires response?

    constexpr static const auto requires_response = true;



    // Mercury callback to serialize input arguments

    constexpr static const auto mercury_in_proc_cb =

            hermes::detail::hg_proc_void_t;



    // Mercury callback to serialize output arguments

    constexpr static const auto mercury_out_proc_cb =

            HG_GEN_PROC_NAME(rpc_err_out_t);



    class input {



        template <typename ExecutionContext>

        friend hg_return_t

        hermes::detail::post_to_mercury(ExecutionContext*);



    public:

        input() {}



        input(input&& rhs) = default;



        input(const input& other) = default;



        input&

        operator=(input&& rhs) = default;



        input&

        operator=(const input& other) = default;



        explicit input(const hermes::detail::hg_void_t& other) {}



        explicit operator hermes::detail::hg_void_t() {

            return {};

        }

    };



    class output {



        template <typename ExecutionContext>

        friend hg_return_t

        hermes::detail::post_to_mercury(ExecutionContext*);



    public:

        output() : m_err() {}



        output(int32_t err) : m_err(err) {}



        output(output&& rhs) = default;



        output(const output& other) = default;



        output&

        operator=(output&& rhs) = default;



        output&

        operator=(const output& other) = default;



        explicit output(const rpc_err_out_t& out) {

            m_err = out.err;

        }



        int32_t

        err() const {

            return m_err;

        }



    private:

        int32_t m_err;

    };

};



//==============================================================================

// definitions for expand_finalize

struct expand_finalize {



    // forward declarations of public input/output types for this RPC

    class input;



    class output;



    // traits used so that the engine knows what to do with the RPC

    using self_type = expand_finalize;

    using handle_type = hermes::rpc_handle<self_type>;

    using input_type = input;

    using output_type = output;

    using mercury_input_type = hermes::detail::hg_void_t;

    using mercury_output_type = rpc_err_out_t;



    // RPC public identifier

    // (N.B: we reuse the same IDs assigned by Margo so that the daemon

    // understands Hermes RPCs)

    constexpr static const uint64_t public_id = 52;



    // RPC internal Mercury identifier

    constexpr static const hg_id_t mercury_id = 0;



    // RPC name

    constexpr static const auto name =

            gkfs::malleable::rpc::tag::expand_finalize;



    // requires response?

    constexpr static const auto requires_response = true;



    // Mercury callback to serialize input arguments

    constexpr static const auto mercury_in_proc_cb =

            hermes::detail::hg_proc_void_t;



    // Mercury callback to serialize output arguments

    constexpr static const auto mercury_out_proc_cb =

            HG_GEN_PROC_NAME(rpc_err_out_t);



    class input {



        template <typename ExecutionContext>

        friend hg_return_t

        hermes::detail::post_to_mercury(ExecutionContext*);



    public:

        input() {}



        input(input&& rhs) = default;



        input(const input& other) = default;



        input&

        operator=(input&& rhs) = default;



        input&

        operator=(const input& other) = default;



        explicit input(const hermes::detail::hg_void_t& other) {}



        explicit operator hermes::detail::hg_void_t() {

            return {};

        }

    };



    class output {



        template <typename ExecutionContext>

        friend hg_return_t

        hermes::detail::post_to_mercury(ExecutionContext*);



    public:

        output() : m_err() {}



        output(int32_t err) : m_err(err) {}



        output(output&& rhs) = default;



        output(const output& other) = default;



        output&

        operator=(output&& rhs) = default;



        output&

        operator=(const output& other) = default;



        explicit output(const rpc_err_out_t& out) {

            m_err = out.err;

        }



        int32_t

        err() const {

            return m_err;

        }



    private:

        int32_t m_err;

    };

};



} // namespace gkfs::rpc

} // namespace malleable::rpc

} // namespace gkfs





#endif // GKFS_RPCS_TYPES_HPP

include/client/CMakeLists.txt

+1 −0
Changes for include/client/CMakeLists.txt: 1 added line, 0 removed lines.
Original line number Diff line number Diff line
@@ -72,6 +72,7 @@ target_sources( 
         rpc/forward_management.hpp

         rpc/forward_metadata.hpp

         rpc/forward_data.hpp

    rpc/forward_malleability.hpp

         syscalls/args.hpp

         syscalls/decoder.hpp

         syscalls/errno.hpp

include/client/user_functions.hpp

+30 −2
Changes for include/client/user_functions.hpp: 30 added lines, 2 removed lines.
Original line number Diff line number Diff line
@@ -40,7 +40,8 @@ extern "C" { 


struct linux_dirent64;



namespace gkfs::syscall {

namespace gkfs {

namespace syscall {



int

gkfs_open(const std::string& path, mode_t mode, int flags);
@@ -77,7 +78,34 @@ gkfs_remove(const std::string& path); 


std::vector<std::string>

gkfs_get_file_list(const std::string& path);

} // namespace gkfs::syscall

} // namespace syscall

namespace malleable {



/**

 * @brief Start an expansion of the file system

 * @param old_server_conf old number of nodes

 * @param new_server_conf new number of nodes

 * @return error code

 */

int

expand_start(int old_server_conf, int new_server_conf);



/**

 * @brief Check for the current status of the expansion process

 * @return 0 when finished, positive numbers indicate how many daemons

 * are still redistributing data

 */

int

expand_status();



/**

 * @brief Finalize the expansion process

 * @return error code

 */

int

expand_finalize();

} // namespace malleable

} // namespace gkfs





extern "C" int

include/common/rpc/distributor.hpp

+20 −5
Changes for include/common/rpc/distributor.hpp: 20 added lines, 5 removed lines.
Original line number Diff line number Diff line
@@ -56,6 +56,9 @@ public: 
    virtual unsigned int

    hosts_size() const = 0;



    virtual void

    hosts_size(unsigned int size) = 0;



    virtual host_t

    locate_data(const std::string& path, const chunkid_t& chnk_id,

                unsigned int hosts_size, const int num_copy) = 0;
@@ -64,7 +67,7 @@ public: 
    locate_file_metadata(const std::string& path, const int num_copy) const = 0;



    virtual std::vector<host_t>

    locate_directory_metadata(const std::string& path) const = 0;

    locate_directory_metadata() const = 0;

};
@@ -83,6 +86,9 @@ public: 
    unsigned int

    hosts_size() const override;



    void

    hosts_size(unsigned int size) override;



    host_t

    localhost() const override;
@@ -99,7 +105,7 @@ public: 
                         const int num_copy) const override;



    std::vector<host_t>

    locate_directory_metadata(const std::string& path) const override;

    locate_directory_metadata() const override;

};



class LocalOnlyDistributor : public Distributor {
@@ -116,6 +122,9 @@ public: 
    unsigned int

    hosts_size() const override;



    void

    hosts_size(unsigned int size) override;



    host_t

    locate_data(const std::string& path, const chunkid_t& chnk_id,

                const int num_copy) const override;
@@ -125,7 +134,7 @@ public: 
                         const int num_copy) const override;



    std::vector<host_t>

    locate_directory_metadata(const std::string& path) const override;

    locate_directory_metadata() const override;

};



class ForwarderDistributor : public Distributor {
@@ -144,6 +153,9 @@ public: 
    unsigned int

    hosts_size() const override;



    void

    hosts_size(unsigned int size) override;



    host_t

    locate_data(const std::string& path, const chunkid_t& chnk_id,

                const int num_copy) const override final;
@@ -157,7 +169,7 @@ public: 
                         const int num_copy) const override;



    std::vector<host_t>

    locate_directory_metadata(const std::string& path) const override;

    locate_directory_metadata() const override;

};



/*
@@ -197,6 +209,9 @@ public: 
    unsigned int

    hosts_size() const override;



    void

    hosts_size(unsigned int size) override;



    host_t

    locate_data(const std::string& path, const chunkid_t& chnk_id,

                const int num_copy) const override;
@@ -210,7 +225,7 @@ public: 
                         const int num_copy) const override;



    std::vector<host_t>

    locate_directory_metadata(const std::string& path) const override;

    locate_directory_metadata() const override;

};



} // namespace gkfs::rpc