Experimental rename approach

Ramon Nourequested to merge
rnou/rename_test into master

This MR proposes a rename approach using the existing SYMLINK architecture.

  1. HAS_RENAME directive enables the needed code and checks.
  2. We use the block count in the metadata (disabled by default) to store a -1 if the file is renamed.
  3. We use the target_path of the symlink architecture (without the IS_LINK flag) to store the original file path.
  4. All the renamed files can't be used (metadata exists but they are not accessible again)

The approach needs a redirection, so a decrease in the performance of the renamed files operations are expected. A new RPC, adding create and update metadata can increase the rename speed.

MDTest benchmark that includes a rename operation works.

Tests are included in the MR.

The rename works in a chain of renames (filea->b->c->d).

  • Enable the filea->fileb to fileb->filea rename. We should check the target_path.
  • Truncate operation works.
  • Lseek operation works (All ops should work now, if they use open)

Conditional testing when rename is enabled. When HAS_RENAME is enabled we check for blocks and target_path in some operations, there are some operations that are not implemented or tested yet.

Introducing the rename check in the get_metadata function reduces modifications along the code, but needs extra parameters similar to the follow_links feature. For now, we consider rename operation as an extra operation that should not be used as default (due to all performance implications). All operations that go thru the open call should work.

It solves #212 (closed)

Edited by Ramon Nou

Merge request reports

include/client/rpc/forward_metadata.hpp

+6 −0
Changes for include/client/rpc/forward_metadata.hpp: 6 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -55,6 +55,12 @@ forward_create(const std::string& path, mode_t mode); 
int

forward_stat(const std::string& path, std::string& attr);



#ifdef HAS_RENAME

int

forward_rename(const std::string& oldpath, const std::string& newpath,

               const gkfs::metadata::Metadata& md);

#endif // HAS_RENAME



int

forward_remove(const std::string& path);

include/client/gkfs_functions.hpp

+4 −0
Changes for include/client/gkfs_functions.hpp: 4 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -149,6 +149,10 @@ gkfs_getdents64(unsigned int fd, struct linux_dirent64* dirp, 
int

gkfs_rmdir(const std::string& path);



#ifdef HAS_RENAME

int

gkfs_rename(const std::string& old_path, const std::string& new_path);

#endif // HAS_RENAME

} // namespace gkfs::syscall



// gkfs_getsingleserverdir is using extern "C" to demangle it for C usage

include/client/hooks.hpp

+3 −0
Changes for include/client/hooks.hpp: 3 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -120,6 +120,9 @@ hook_unlinkat(int dirfd, const char* cpath, int flags); 
int

hook_symlinkat(const char* oldname, int newdfd, const char* newname);



int

hook_flock(unsigned long fd, int flags);



int

hook_access(const char* path, int mask);

include/common/metadata.hpp

+13 −1
Changes for include/common/metadata.hpp: 13 added lines, 1 removed line.
Original line number Diff line number Diff line
@@ -53,6 +53,10 @@ private: 
    blkcnt_t blocks_{}; // allocated file system blocks_

#ifdef HAS_SYMLINKS

    std::string target_path_; // For links this is the path of the target file

#ifdef HAS_RENAME

    std::string rename_path_; // In some cases fd is maintained so we need the

                              // renamed path

#endif

#endif
@@ -133,7 +137,15 @@ public: 
    bool

    is_link() const;



#endif

#ifdef HAS_RENAME

    std::string

    rename_path() const;



    void

    rename_path(const std::string& rename_path);

#endif // HAS_RENAME



#endif // HAS_SYMLINKS

};



} // namespace gkfs::metadata

include/config.hpp

+5 −0
Changes for include/config.hpp: 5 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -62,11 +62,16 @@ namespace metadata { 
constexpr auto dir = "metadata";



// which metadata should be considered apart from size and mode

// Blocks are used to store the rename status (-1 is a renamed file)

constexpr auto use_atime = false;

constexpr auto use_ctime = false;

constexpr auto use_mtime = false;

constexpr auto use_link_cnt = false;

#ifdef HAS_RENAME

constexpr auto use_blocks = true;

#else

constexpr auto use_blocks = false;

#endif // HAS_RENAME

/*

 * If true, all chunks on the same host are removed during a metadata remove

 * rpc. This is a technical optimization that reduces the number of RPCs for