Resolve "Shared file metadata congestion"

Marc Vefrequested to merge
marc/62-shared-file-metadata-congestion-2 into master

During write operations, the client must update the file size on the responsible metadata daemon. The write size cache can reduce the metadata load on the daemon and reduce the number of RPCs during write operations, especially for many small I/O operations. In the past, we have observed that a daemon can become network-congested, especially for single shared files, many processes, and small I/O operations, which bottlenecks the overall I/O throughput. Nevertheless, the cache can have a broad impact on small I/O operations as 1 RPC for updating the size is removed which already improves small file I/O on a single node.

Note that this cache may impact file size consistency in which stat operations may not reflect the actual file size until the file is closed. The cache does not impact the consistency of the file data itself. We did not observe any issues with the cache for HPC applications and benchmarks, but it technically breaks POSIX. So, for now, I suggest it to be experimental and opt-in.

  • LIBGKFS_WRITE_SIZE_CACHE - Enable caching the write size of files (default: OFF).
  • LIBGKFS_WRITE_SIZE_CACHE_THRESHOLD - Set the number of write operations after which the file size is synchronized with the corresponding daemon (default: 1000). The file size is further synchronized when the file is close()d or when fsync() is called.

Depends on !194 (merged)

Closes #62 (closed)

Edited by Marc Vef

Merge request reports

include/client/cache.hpp

+55 −0
Changes for include/client/cache.hpp: 55 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -39,6 +39,7 @@ 
#include <mutex>

#include <optional>

#include <cstdint>

#include <utility>



namespace gkfs::cache {
@@ -132,6 +133,60 @@ public: 
};

} // namespace dir



namespace file {

class WriteSizeCache {

private:

    // <path<cnt, size>>

    std::unordered_map<std::string, std::pair<size_t, size_t>> size_cache;

    std::mutex mtx_;



    // Flush threshold in number of write ops per file

    size_t flush_threshold_{0};



public:

    WriteSizeCache() = default;



    virtual ~WriteSizeCache() = default;



    /**

     * @brief Record the size of a file and add it to the cache

     * @param path gekkofs path

     * @param size current size to set for given path

     * @return [size_update counter, current cached size]

     */

    std::pair<size_t, size_t>

    record(std::string path, size_t size);



    /**

     * @brief reset entry from the cache

     * @param path

     * @param evict if true, entry is removed from cache, reseted to cnt 0

     * otherwise

     * @return [size_update counter, current cached size]

     */

    std::pair<size_t, size_t>

    reset(const std::string& path, bool evict);



    /**

     * @brief Flush the cache for a given path contacting the corresponding

     * daemon

     * @param path

     * @param evict during flush: if true, entry is removed from cache, reseted

     * to cnt 0 otherwise

     * @return error code and flushed size

     */

    std::pair<int, off64_t>

    flush(const std::string& path, bool evict = true);





    // GETTER/SETTER

    size_t

    flush_threshold() const;



    void

    flush_threshold(size_t flush_threshold);

};

} // namespace file

} // namespace gkfs::cache



#endif // GKFS_CLIENT_CACHE

include/client/env.hpp

+6 −1
Changes for include/client/env.hpp: 6 added lines, 1 removed line.
Original line number Diff line number Diff line
@@ -60,7 +60,12 @@ static constexpr auto METRICS_IP_PORT = ADD_PREFIX("METRICS_IP_PORT"); 


static constexpr auto NUM_REPL = ADD_PREFIX("NUM_REPL");

static constexpr auto PROXY_PID_FILE = ADD_PREFIX("PROXY_PID_FILE");

static constexpr auto DENTRY_CACHE = ADD_PREFIX("DENTRY_CACHE");

namespace cache {

static constexpr auto DENTRY = ADD_PREFIX("DENTRY_CACHE");

static constexpr auto WRITE_SIZE = ADD_PREFIX("WRITE_SIZE_CACHE");

static constexpr auto WRITE_SIZE_THRESHOLD =

        ADD_PREFIX("WRITE_SIZE_CACHE_THRESHOLD");

} // namespace cache



} // namespace gkfs::env

include/client/gkfs_functions.hpp

+3 −0
Changes for include/client/gkfs_functions.hpp: 3 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -157,6 +157,9 @@ gkfs_getdents64(unsigned int fd, struct linux_dirent64* dirp, 
int

gkfs_rmdir(const std::string& path);



int

gkfs_fsync(unsigned int fd);



int

gkfs_close(unsigned int fd);

include/client/preload_context.hpp

+18 −0
Changes for include/client/preload_context.hpp: 18 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -59,6 +59,9 @@ namespace cache { 
namespace dir {

class DentryCache;

}

namespace file {

class WriteSizeCache;

}

} // namespace cache



namespace preload {
@@ -98,6 +101,9 @@ private: 
    std::shared_ptr<FsConfig> fs_conf_;

    std::shared_ptr<gkfs::cache::dir::DentryCache> dentry_cache_;

    bool use_dentry_cache_{false};

    std::shared_ptr<gkfs::cache::file::WriteSizeCache> write_size_cache_;

    bool use_write_size_cache_{false};





    std::string cwd_;

    std::vector<std::string> mountdir_components_;
@@ -249,6 +255,18 @@ public: 
    void

    use_dentry_cache(bool use_dentry_cache);



    std::shared_ptr<gkfs::cache::file::WriteSizeCache>

    write_size_cache() const;



    void

    write_size_cache(std::shared_ptr<gkfs::cache::file::WriteSizeCache>

                             write_size_cache);



    bool

    use_write_size_cache() const;



    void

    use_write_size_cache(bool use_dentry_cache);



    void

    enable_interception();

include/client/preload_util.hpp

+12 −0
Changes for include/client/preload_util.hpp: 12 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -78,6 +78,18 @@ int 
metadata_to_stat(const std::string& path, const gkfs::metadata::Metadata& md,

                 struct stat& attr);



/**

 * @brief Updates write size on the metadata daemon

 * @param path

 * @param count

 * @param offset

 * @param is_append

 * @return <err, return_offset>

 */

std::pair<int, off64_t>

update_file_size(const std::string& path, size_t count, off64_t offset,

                 bool is_append);



void

load_hosts();