gekkofs:master commitshttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commits/master2024-03-18T11:39:18+01:00https://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/f2092c3b54839f0a0e3a27315cfcac5ccd6d4c54Merge branch 'marc/hotfix' into 'master'2024-03-18T11:39:18+01:00Marc Vefvef@uni-mainz.de
Remove debugging output from srv_data.cpp
Leftovers from replication branch that was missed during review.
Currently, prints one error message per chunk on the daemon.
See merge request <a href="/gitlab/hpc/gekkofs/-/merge_requests/184" data-reference-type="merge_request" data-original="hpc/gekkofs!184" data-link="false" data-link-reference="false" data-project="8" data-merge-request="372" data-project-path="hpc/gekkofs" data-iid="184" data-container="body" data-placement="top" title="Remove debugging output from srv_data.cpp" class="gfm gfm-merge_request">!184</a>https://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/a3605d52b0247595b75b2c3a8d54aca7f42643bbRemove debugging output from srv_data.cpp2024-03-18T11:14:49+01:00Marc Vefvef@uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/137dd33fb2f8bc4d655fb03920d1f65c65c67479Merge branch 'rnou/replication' into 'master'2024-03-13T15:37:34+01:00Marc Vefvef@uni-mainz.de
Data replication (client side, synchronous)
This MR adds support for data replication using one environment variable:
`LIBGKFS_NUM_REPL=<num repl>`
The number of replicas should go from 0 to the number of servers-1. The replicas are guided by the
client, so it reduces write performance but we mantain the same level of consistency.
On the other hand, it may increase read performance on some corner scenearios.
Metadata replication is also implemented
The replication environment variable can be set up for each client, independently.
If a server is down, the data will be read from another replica. The metadata management is also done from another replica.
The replication is done in a synchronous way. A new function forward_write is used to sent to the different replicas. The reads are distributed, but this shouldn't produce an performance improvement as the distribution is similar to the original.
In the case of the write, the original is sent to the target servers, and then the replicas are processed. This is done to avoid issues if a server, that should host a replica, is not available.
In order to process the replicas a new method to check that a chunk needs to be processed inside a server is included, a bitset of 1024 is sent (coded in base-64 in a string). This represents 1024-chunks per write-read operation. If that is exceeded the normal hash check per chunk is done in the server. Exceeding this value, will disable the replica capabilities and produce unknown behaviours.
This can be potentially increased.
Finally, most of the operations are replica-aware, but some of them are missing yet. i.e., dirent.
See merge request <a href="/gitlab/hpc/gekkofs/-/merge_requests/166" data-reference-type="merge_request" data-original="hpc/gekkofs!166" data-link="false" data-link-reference="false" data-project="8" data-merge-request="293" data-project-path="hpc/gekkofs" data-iid="166" data-container="body" data-placement="top" title="Data replication (client side, synchronous)" class="gfm gfm-merge_request">!166</a>https://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/2835322a3e84905063d91fb183af9aaf418755f5Review2024-03-13T15:18:36+01:00Marc Vefvef@uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/d11575d6fe1bfd66500ac826d91245d960eec4b2Cleanup RPC code2024-03-10T19:29:34+01:00Marc Vefvef@uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/a66ae9eb31c6b001027995a4317edcc6f6280814Review changes2024-03-10T19:28:04+01:00Marc Vefvef@uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/938b436dbbc73df539d98f24ea430662310f74fdWrite parameters change2024-03-10T09:39:25+01:00Ramon Nouramon.nou@bsc.eshttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/c0ee81e7366ca4752afb8f653bd99f680d1f07eeChangelog change and branch2024-03-10T09:39:25+01:00rnouramon.nou@bsc.es
Added NUM_REPL env variable. (0 no replicas)
NUM_REPL num replicas (Replicas < servers)
Remove and truncate
Metadata replication
Metadata replication - Reattempt on stat
minimal compilation issues (c++20), srand for repl
Bitset
Bit set proposal (WIP)
Read - Write with bitset (<1024 chunks)
Changed bitset to vector
Added get_fs_config reattempt
Some more resilience on create
Added Replica_Check on write (disabled)
Added helper vector-bitset functionshttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/0916c55578dc76a119c3125424cb1d176f4943c3Merge branch 'release-0.9.2' into 'master'2024-02-19T00:37:46+01:00Marc Vefvef@uni-mainz.de
Release 0.9.2
See merge request <a href="/gitlab/hpc/gekkofs/-/merge_requests/181" data-reference-type="merge_request" data-original="hpc/gekkofs!181" data-link="false" data-link-reference="false" data-project="8" data-merge-request="368" data-project-path="hpc/gekkofs" data-iid="181" data-container="body" data-placement="top" title="Release 0.9.2" class="gfm gfm-merge_request">!181</a>https://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/25f8c0e1fdb39b2a05cfc247b6e206e0a3597144Update changelog2024-02-19T00:13:38+01:00Marc Vefvef@uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/8b4fe854e5550328f839f254ac68bcd2fbb49053Version bump 0.9.22024-02-19T00:10:20+01:00Marc Vefvef@uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/800f3af35c856b9703f68b310e8a47e9b8ac9205Updating acks and copyright headers2024-02-19T00:06:02+01:00Marc Vefvef@uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/f41ebb4b29d00c02543656b1900b1b9d24628cd6Merge branch 'jathenst/278-client-logging-add-per-process-log-option' into 'm...2024-02-15T14:41:29+01:00Marc Vefvef@uni-mainz.de
Resolve "Client logging: Add per process log option"
Closes <a href="/gitlab/hpc/gekkofs/-/issues/278" data-reference-type="issue" data-original="#278" data-link="false" data-link-reference="false" data-project="8" data-issue="719" data-project-path="hpc/gekkofs" data-iid="278" data-issue-type="issue" data-container="body" data-placement="top" title="Client logging: Add per process log option" class="gfm gfm-issue">#278</a>
Closes <a href="/gitlab/hpc/gekkofs/-/issues/278" data-reference-type="issue" data-original="#278" data-link="false" data-link-reference="false" data-project="8" data-issue="719" data-project-path="hpc/gekkofs" data-iid="278" data-issue-type="issue" data-container="body" data-placement="top" title="Client logging: Add per process log option" class="gfm gfm-issue">#278</a>
See merge request <a href="/gitlab/hpc/gekkofs/-/merge_requests/179" data-reference-type="merge_request" data-original="hpc/gekkofs!179" data-link="false" data-link-reference="false" data-project="8" data-merge-request="364" data-project-path="hpc/gekkofs" data-iid="179" data-container="body" data-placement="top" title='Resolve "Client logging: Add per process log option"' class="gfm gfm-merge_request">!179</a>https://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/4f16eb71a4abf71a1056ad6e2f45ea837345b77dReview before merge. Minor changes.2024-02-14T14:53:43+01:00Marc Vefvef@uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/b45e7f0a482ce27f41e1b42943496811e08cf1adupdate CHANGELOG, README and docs. use fmt2024-02-06T14:41:58+01:00Julius Athenstaedtjathenst@students.uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/cfb872951c94dd869f4531929599e1de830660d2LIBGKFS_LOG_PER_PROCESS environment variable can be used to2024-02-06T14:41:58+01:00Julius Athenstaedtjathenst@students.uni-mainz.decreate logs in separate files for each process of the client.https://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/c4d2a5a0729498790adc1f4aa57a16227cceb77bMerge branch 'jathenst/225-support-mtime' into 'master'2024-01-31T17:39:33+01:00Marc Vefvef@uni-mainz.de
Resolve "Support mtime"
Closes <a href="/gitlab/hpc/gekkofs/-/issues/225" data-reference-type="issue" data-original="#225" data-link="false" data-link-reference="false" data-project="8" data-issue="473" data-project-path="hpc/gekkofs" data-iid="225" data-issue-type="issue" data-container="body" data-placement="top" title="Support mtime" class="gfm gfm-issue">#225</a>
Closes <a href="/gitlab/hpc/gekkofs/-/issues/225" data-reference-type="issue" data-original="#225" data-link="false" data-link-reference="false" data-project="8" data-issue="473" data-project-path="hpc/gekkofs" data-iid="225" data-issue-type="issue" data-container="body" data-placement="top" title="Support mtime" class="gfm gfm-issue">#225</a>
See merge request <a href="/gitlab/hpc/gekkofs/-/merge_requests/178" data-reference-type="merge_request" data-original="hpc/gekkofs!178" data-link="false" data-link-reference="false" data-project="8" data-merge-request="359" data-project-path="hpc/gekkofs" data-iid="178" data-container="body" data-placement="top" title='Resolve "Support mtime"' class="gfm gfm-merge_request">!178</a>https://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/6c68de44d90172f73eeb6f6e44dfea913195b3f3Review and remove dead code2024-01-31T17:17:16+01:00Marc Vefvef@uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/779f54cce9130ac41dcebed72936a1c5cc593704mtime and ctime update on rocksdb merge2023-12-20T18:09:05+01:00Julius Athenstaedtjathenst@students.uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/f812805062d8840bde736d5659f42ca913895f78Merge branch 'jathenst/261-du-doesn-t-show-size' into 'master'2023-11-28T20:24:04+01:00Marc Vefvef@uni-mainz.de
Resolve "du doesn't show size"
Closes <a href="/gitlab/hpc/gekkofs/-/issues/261" data-reference-type="issue" data-original="#261" data-link="false" data-link-reference="false" data-project="8" data-issue="633" data-project-path="hpc/gekkofs" data-iid="261" data-issue-type="issue" data-container="body" data-placement="top" title="du doesn't show size" class="gfm gfm-issue">#261</a>
Closes <a href="/gitlab/hpc/gekkofs/-/issues/261" data-reference-type="issue" data-original="#261" data-link="false" data-link-reference="false" data-project="8" data-issue="633" data-project-path="hpc/gekkofs" data-iid="261" data-issue-type="issue" data-container="body" data-placement="top" title="du doesn't show size" class="gfm gfm-issue">#261</a>
See merge request <a href="/gitlab/hpc/gekkofs/-/merge_requests/177" data-reference-type="merge_request" data-original="hpc/gekkofs!177" data-link="false" data-link-reference="false" data-project="8" data-merge-request="350" data-project-path="hpc/gekkofs" data-iid="177" data-container="body" data-placement="top" title="Resolve "du doesn't show size"" class="gfm gfm-merge_request">!177</a>https://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/e227163b78b5057e6b5bbb3f1f04215a5b141130Review: config name change2023-11-28T19:53:53+01:00Marc Vefvef@uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/ad3f49688625a85a174a45a0d0f0f3cfdbcff965calculates blocksize by dividing size / 512 as it is legacy in the linux...2023-11-23T12:30:45+01:00Julius Athenstaedtjathenst@students.uni-mainz.decalculates blocksize by dividing size / 512 as it is legacy in the linux kernel, only if renaming is deactivated
https://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/0ba03e5ffdeff3be438369322aeead22d86ef732Merge branch 'jathenst/129-add-gkfs_-prefix-to-cmake-options' into 'master'2023-10-17T14:52:09+02:00Marc Vefvef@uni-mainz.de
Resolve "Add GKFS_ prefix to CMake options"
Closes <a href="/gitlab/hpc/gekkofs/-/issues/129" data-reference-type="issue" data-original="#129" data-link="false" data-link-reference="false" data-project="8" data-issue="353" data-project-path="hpc/gekkofs" data-iid="129" data-issue-type="issue" data-container="body" data-placement="top" title="Add GKFS_ prefix to CMake options" class="gfm gfm-issue">#129</a>
Closes <a href="/gitlab/hpc/gekkofs/-/issues/129" data-reference-type="issue" data-original="#129" data-link="false" data-link-reference="false" data-project="8" data-issue="353" data-project-path="hpc/gekkofs" data-iid="129" data-issue-type="issue" data-container="body" data-placement="top" title="Add GKFS_ prefix to CMake options" class="gfm gfm-issue">#129</a>
See merge request <a href="/gitlab/hpc/gekkofs/-/merge_requests/173" data-reference-type="merge_request" data-original="hpc/gekkofs!173" data-link="false" data-link-reference="false" data-project="8" data-merge-request="333" data-project-path="hpc/gekkofs" data-iid="173" data-container="body" data-placement="top" title='Resolve "Add GKFS_ prefix to CMake options"' class="gfm gfm-merge_request">!173</a>https://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/e9a691e3e61dfcac9efe58c2f144b26398c6947aGKFS_ prefix for CLIENT_LOG_MESSAGE_SIZE2023-10-16T17:12:02+02:00Julius Athenstaedtjathenst@students.uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/39840ca1af5f9bcb2ece5638e2a066e6a62d365bGKFS_ prefix for CREATE_CHECK_PARENTS2023-10-16T16:24:28+02:00Julius Athenstaedtjathenst@students.uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/9efd7f449a06e2a1d5a7acd6fe648ad43ebca8aeGKFS_ prefix for ENABLE_CLIENT_LOG2023-10-16T16:24:21+02:00Julius Athenstaedtjathenst@students.uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/a4b09aba05b8f3404744a711eadf6fad29781fdbGKFS_ Prefix for MAX_OPEN_FDS2023-10-16T16:24:12+02:00Julius Athenstaedtjathenst@students.uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/d441ffa26603bb56c3a1bc095c2aca7aef913226GKFS_ Prefix for MAX_INTERNAL_FDS2023-10-16T16:23:26+02:00Julius Athenstaedtjathenst@students.uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/91d693386128c8ee2bac98a1b4261b3fbbc3ca20Merge branch 'rnou/268-remove-parallax-testing' into 'master'2023-10-02T15:57:16+02:00Alberto Mirandaalberto.miranda@bsc.es
Resolve "Remove Parallax Testing"
This MR updates testing for the new docker CI environment:
- Removes Parallax Testing to make faster tests (we will activate again if needed, or changes are applied)
- Adds cstdint for uint types (gcc 10..)
- Updates python sh library/package to a newer version. This solves and produces some changes.
- Solved a bug in sfind
Closes <a href="/gitlab/hpc/gekkofs/-/issues/268" data-reference-type="issue" data-original="#268" data-link="false" data-link-reference="false" data-project="8" data-issue="671" data-project-path="hpc/gekkofs" data-iid="268" data-issue-type="issue" data-container="body" data-placement="top" title="Solving CI issues" class="gfm gfm-issue">#268</a>
Closes <a href="/gitlab/hpc/gekkofs/-/issues/268" data-reference-type="issue" data-original="#268" data-link="false" data-link-reference="false" data-project="8" data-issue="671" data-project-path="hpc/gekkofs" data-iid="268" data-issue-type="issue" data-container="body" data-placement="top" title="Solving CI issues" class="gfm gfm-issue">#268</a>
See merge request <a href="/gitlab/hpc/gekkofs/-/merge_requests/169" data-reference-type="merge_request" data-original="hpc/gekkofs!169" data-link="false" data-link-reference="false" data-project="8" data-merge-request="321" data-project-path="hpc/gekkofs" data-iid="169" data-container="body" data-placement="top" title='Resolve "Remove Parallax Testing"' class="gfm gfm-merge_request">!169</a>https://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/1ecefdf48ee94aadcbe4a3da8161d6ff0d863a7bshutil which2023-09-29T10:39:41+02:00Ramon Nouramon.nou@bsc.eshttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/274ca10de5ec1e8d69a1afcf4680b9abfbcf52b3PosixPath cannot be used in env (Python sh)2023-09-29T10:39:38+02:00Ramon Nouramon.nou@bsc.es
Change all env PosixPath to strhttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/6ad676de570e5b23fad8d496e64950d34435d606Updated sh library2023-09-28T13:53:24+02:00Ramon Nouramon.nou@bsc.eshttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/2a3d2b6c0b0516e9a4dc9a3b29c52ded05482ffe_process_exists using kill 0 instead of ps (ci)2023-09-27T12:28:25+02:00Ramon Nouramon.nou@bsc.es
restore rocksdb
Removed extra debug linehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/9c3a1b09d12395ab9c8d58155df5dad915b03fdfUpdate some gcc version issues2023-09-27T12:28:25+02:00Ramon Nouramon.nou@bsc.eshttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/33026d64f37acb4d04b2e94b532de729308b0f94FDS to 60002023-09-27T12:28:23+02:00Ramon Nouramon.nou@bsc.es
Added max internal fdshttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/b9c8b400184f8a3ad1b1286e99dbca5eb81d4966Reducing nprocs2023-09-27T12:28:19+02:00Ramon Nouramon.nou@bsc.es
Reduce OPEN_FDS
Missing optionhttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/26975f740f7d2ef2fccae742ab805b1429909869Removed Parallax in the CI2023-09-22T11:28:33+02:00Ramon Nouramon.nou@bsc.eshttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/2730a9ce4b44247078a0b043e31a5a9dce483c39Merge branch 'marc/58-support-spack-and-others' into 'master'2023-06-20T15:05:02+00:00Marc Vefvef@uni-mainz.de
Resolve "Support Spack and others"
### Usage information
Download Spack and setup environment:
```bash
git clone -c feature.manyFiles=true <a href="https://github.com/spack/spack.git" rel="nofollow noreferrer noopener" target="_blank">https://github.com/spack/spack.git</a>
. spack/share/spack/setup-env.sh
```
Add GekkoFS Spack repository to Spack:
```bash
spack repo add gekkofs/scripts/spack
```
Check that Spack can find GekkoFS:
```
spack info gekkofs
```
Install GekkoFS (and run optional tests). Check `spack info gekkofs` for available option and versions:
```bash
spack install gekkofs
# for installing tests dependencies and running tests
spack install -v --test=root gekkofs +tests
```
Load GekkoFS into environment:
```
spack load gekkofs
```
If you want to use the latest developer branch of GekkoFS:
```
spack install gekkofs@latest
```
The default is using version 0.9.1 the last stable release.
### TODO
- [x] Base Spack functionality, versions, and configuration support
- [x] Documentation
- [x] Advanced functionality, more detailed configuration support, e.g., Parallax and Prometheus
- [x] More easy way to get path to client library
- [x] Add GekkoFS client wrapper for `LD_PRELOAD`
- [ ] Add final version to main Spack repository if possible. (Not possible right now as it not clear how 3rd party libraries should be treated.
Closes <a href="/gitlab/hpc/gekkofs/-/issues/58" data-reference-type="issue" data-original="#58" data-link="false" data-link-reference="false" data-project="8" data-issue="228" data-project-path="hpc/gekkofs" data-iid="58" data-issue-type="issue" data-container="body" data-placement="top" title="Support Spack and others" class="gfm gfm-issue">#58</a>
Closes <a href="/gitlab/hpc/gekkofs/-/issues/58" data-reference-type="issue" data-original="#58" data-link="false" data-link-reference="false" data-project="8" data-issue="228" data-project-path="hpc/gekkofs" data-iid="58" data-issue-type="issue" data-container="body" data-placement="top" title="Support Spack and others" class="gfm gfm-issue">#58</a>
See merge request <a href="/gitlab/hpc/gekkofs/-/merge_requests/137" data-reference-type="merge_request" data-original="hpc/gekkofs!137" data-link="false" data-link-reference="false" data-project="8" data-merge-request="167" data-project-path="hpc/gekkofs" data-iid="137" data-container="body" data-placement="top" title='Resolve "Support Spack and others"' class="gfm gfm-merge_request">!137</a>https://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/3f59f7ad255305728cef20afb23f4caae20ae8b7Fix Sphinx syntax2023-06-20T16:27:08+02:00Marc Vefvef@uni-mainz.dehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/commit/8321783bc58923369dac261da1a6f1c169f41403Added README FAQ2023-06-20T16:27:08+02:00Marc Vefvef@uni-mainz.de