1. Jul 16, 2024
    • Marc Vef's avatar
      Merge branch 'marc/292-add-dentry-cache' into 'master' · 8812ccdf
      Marc Vef authored
      Resolve "Add dentry cache"
      
      This MR adds a directory entry cache for the client to avoid a huge number of stat calls after readdir, e.g., for `ls -l` type operations. It is experimental and thus disabled by default. Can be enabled via `include/config.hpp` or with the env variable `LIBGKFS_DENTRY_CACHE=ON/OFF`.
      
      It works by using the `extended_dir_entry` RPC to receive some metadata along the the dentries from the daemons. This metadata is then placed into the cache and retrieved in a stat operation (for a cache miss, an RPC is sent with vanilla functionality). The cache is discarded upon close but can be changed via `include/config.hpp`. Note, this may cause semantical issues (removed files will remain in the cache forever).
      
      The performance improvements are already noticeable locally for a couple 1000 files.
      
      Depends on !195
      
      Closes #292
      
      Closes #292
      
      See merge request !194
      8812ccdf
  2. Jul 15, 2024
  3. Jul 12, 2024
    • Julius Athenstaedt's avatar
      active following symlinks for integration test · e2644038
      Julius Athenstaedt authored and Marc Vef's avatar Marc Vef committed
      e2644038
    • Julius Athenstaedt's avatar
      catch edgecase of relative paths, Changelog · 9997db6a
      Julius Athenstaedt authored and Marc Vef's avatar Marc Vef committed
      9997db6a
    • Julius Athenstaedt's avatar
      path unit test · 8113718a
      Julius Athenstaedt authored and Marc Vef's avatar Marc Vef committed
      8113718a
    • Julius Athenstaedt's avatar
      userlib for tests · 6a960853
      Julius Athenstaedt authored and Marc Vef's avatar Marc Vef committed
      6a960853
    • Julius Athenstaedt's avatar
      add BUILD flags and register new resolve fn · 34efceff
      Julius Athenstaedt authored and Marc Vef's avatar Marc Vef committed
      34efceff
    • Julius Athenstaedt's avatar
      refactor resolve path · c009d4b0
      Julius Athenstaedt authored and Marc Vef's avatar Marc Vef committed
      c009d4b0
    • Marc Vef's avatar
      Merge branch 'marc/294-file-system-expansion-during-runtime' into 'master' · cddedd6f
      Marc Vef authored
      Resolve "File system expansion during runtime"
      
      # Description
      
      GekkoFS supports extending the current daemon configuration to additional compute nodes. This includes redistribution
      of the existing data and metadata and therefore scales file system performance and capacity of existing data. Note,
      that it is the user's responsibility to not access the GekkoFS file system during redistribution. A corresponding feature
      that is transparent to the user is planned. Note also, if the GekkoFS proxy is used, they need to be manually restarted, after expansion.
      
      To enable this feature, the following CMake compilation flags are required to build the `gkfs_malleability` tool: `-DGKFS_BUILD_TOOLS=ON`.
      The `gkfs_malleability` tool is then available in the `build/tools` directory. Please consult `-h` for its arguments.
      While the tool can be used manually to expand the file system, the `scripts/run/gkfs` script should be used instead which invokes the `gkfs_malleability` tool.
      
      The only requirement for extending the file system is a hostfile containing the hostnames/IPs of the new nodes (one line per host).
      Example starting the file system. The `DAEMON_NODELIST` in the `gkfs.conf` is set to a hostfile containing the initial set of file system nodes.:
      ```bash
      ~/gekkofs/scripts/run/gkfs -c ~/run/gkfs_verbs_expandtest.conf start
      * [gkfs] Starting GekkoFS daemons (4 nodes) ...
      * [gkfs] GekkoFS daemons running
      * [gkfs] Startup time: 10.853 seconds
      ```
      ... Some computation ...
      
      Expanding the file system. Using `-e <hostfile>` to specify the new nodes. Redistribution is done automatically with a progress bar. 
      When finished, the file system is ready to use in the new configuration:
      ```bash
      ~/gekkofs/scripts/run/gkfs -c ~/run/gkfs_verbs_expandtest.conf -e ~/hostfile_expand expand
      * [gkfs] Starting GekkoFS daemons (8 nodes) ...
      * [gkfs] GekkoFS daemons running
      * [gkfs] Startup time: 1.058 seconds
      Expansion process from 4 nodes to 12 nodes launched...
      * [gkfs] Expansion progress:
      [####################] 0/4 left
      * [gkfs] Redistribution process done. Finalizing ...
      * [gkfs] Expansion done.
      ```
      Stop the file system:
      ```bash
      ~/gekkofs/scripts/run/gkfs -c ~/run/gkfs_verbs_expandtest.conf stop
      * [gkfs] Stopping daemon with pid 16462
      srun: sending Ctrl-C to StepId=282378.1
      * [gkfs] Stopping daemon with pid 16761
      srun: sending Ctrl-C to StepId=282378.2
      * [gkfs] Shutdown time: 1.032 seconds
      ```
      
      # Results
      IOR results for writing/reading 768 GiB sequentially (192 procs) before and after expansion
      
      ![image](/uploads/57bd8f3a07a56c496b1ae0b096da24ef/image.png)
      
      MDTest results for creating, stating, removing, 19200000 (192 procs) before and after expansion
      
      ![image](/uploads/7e2f58d864789e657140ced3e9e9716e/image.png)
      
      Closes #294
      
      Closes #294
      
      See merge request !196
      cddedd6f
    • Marc Vef's avatar
      Cleanup, Readme, changelog. · 49263be8
      Marc Vef authored
      49263be8
    • Marc Vef's avatar
      Fix protocol for daemon RPC client · 0f42da53
      Marc Vef authored
      0f42da53
    • Marc Vef's avatar
      Rudimentary Proxy support for extended file systems. · 963b9e4f
      Marc Vef authored
      Proxy must be restarted to know about the file system extension.
      963b9e4f
  4. Jul 11, 2024
  5. Jul 04, 2024
  6. Jul 03, 2024