- May 03, 2024
-
-
Marc Vef authored
To be included in v0.9.3
-
- Mar 13, 2024
-
-
Marc Vef authored
Data replication (client side, synchronous) This MR adds support for data replication using one environment variable: `LIBGKFS_NUM_REPL=<num repl>` The number of replicas should go from 0 to the number of servers-1. The replicas are guided by the client, so it reduces write performance but we mantain the same level of consistency. On the other hand, it may increase read performance on some corner scenearios. Metadata replication is also implemented The replication environment variable can be set up for each client, independently. If a server is down, the data will be read from another replica. The metadata management is also done from another replica. The replication is done in a synchronous way. A new function forward_write is used to sent to the different replicas. The reads are distributed, but this shouldn't produce an performance improvement as the distribution is similar to the original. In the case of the write, the original is sent to the target servers, and then the replicas are processed. This is done to avoid issues if a server, that should host a replica, is not available. In order to process the replicas a new method to check that a chunk needs to be processed inside a server is included, a bitset of 1024 is sent (coded in base-64 in a string). This represents 1024-chunks per write-read operation. If that is exceeded the normal hash check per chunk is done in the server. Exceeding this value, will disable the replica capabilities and produce unknown behaviours. This can be potentially increased. Finally, most of the operations are replica-aware, but some of them are missing yet. i.e., dirent. See merge request !166
-
Marc Vef authored
-
- Mar 10, 2024
-
-
Marc Vef authored
-
Marc Vef authored
-
Added NUM_REPL env variable. (0 no replicas) NUM_REPL num replicas (Replicas < servers) Remove and truncate Metadata replication Metadata replication - Reattempt on stat minimal compilation issues (c++20), srand for repl Bitset Bit set proposal (WIP) Read - Write with bitset (<1024 chunks) Changed bitset to vector Added get_fs_config reattempt Some more resilience on create Added Replica_Check on write (disabled) Added helper vector-bitset functions
-
- Feb 19, 2024
- Feb 15, 2024
-
- Feb 14, 2024
-
-
Marc Vef authored
-
- Feb 06, 2024
-
-
Julius Athenstaedt authored
-
Julius Athenstaedt authored
create logs in separate files for each process of the client.
-
- Jan 31, 2024
-
-
Marc Vef authored
- Dec 20, 2023
-
-
Julius Athenstaedt authored
-
- Nov 28, 2023
-
-
Marc Vef authored
- Nov 23, 2023
-
-
Julius Athenstaedt authored
calculates blocksize by dividing size / 512 as it is legacy in the linux kernel, only if renaming is deactivated
-
- Oct 17, 2023
-
- Oct 16, 2023
-
-
Julius Athenstaedt authored
-
Julius Athenstaedt authored
-
Julius Athenstaedt authored
-
Julius Athenstaedt authored
-
Julius Athenstaedt authored
-
- Oct 02, 2023
-
-
Alberto Miranda authored
Resolve "Remove Parallax Testing" This MR updates testing for the new docker CI environment: - Removes Parallax Testing to make faster tests (we will activate again if needed, or changes are applied) - Adds cstdint for uint types (gcc 10..) - Updates python sh library/package to a newer version. This solves and produces some changes. - Solved a bug in sfind Closes #268 Closes #268 See merge request !169
-
- Sep 29, 2023
- Sep 28, 2023
-
-
Ramon Nou authored
-
- Sep 27, 2023
- Sep 22, 2023
-
-
Ramon Nou authored
-
- Jun 20, 2023
-
-
Marc Vef authored
Resolve "Support Spack and others" ### Usage information Download Spack and setup environment: ```bash git clone -c feature.manyFiles=true https://github.com/spack/spack.git . spack/share/spack/setup-env.sh ``` Add GekkoFS Spack repository to Spack: ```bash spack repo add gekkofs/scripts/spack ``` Check that Spack can find GekkoFS: ``` spack info gekkofs ``` Install GekkoFS (and run optional tests). Check `spack info gekkofs` for available option and versions: ```bash spack install gekkofs # for installing tests dependencies and running tests spack install -v --test=root gekkofs +tests ``` Load GekkoFS into environment: ``` spack load gekkofs ``` If you want to use the latest developer branch of GekkoFS: ``` spack install gekkofs@latest ``` The default is using version 0.9.1 the last stable release. ### TODO - [x] Base Spack functionality, versions, and configuration support - [x] Documentation - [x] Advanced functionality, more detailed configuration support, e.g., Parallax and Prometheus - [x] More easy way to get path to client library - [x] Add GekkoFS client wrapper for `LD_PRELOAD` - [ ] Add final version to main Spack repository if possible. (Not possible right now as it not clear how 3rd party libraries should be treated. Closes #58 Closes #58 See merge request !137
-
Marc Vef authored
-
Marc Vef authored
-
Marc Vef authored
-