Working recovery EC

Polishing touches for Erasure codes

Ramon Nou authored Jun 13, 2023 and Ramon Nou committed Oct 03, 2023

WIP

Missing write with char**

First Write - ECC

cout -> LOG

WIP Decode EC to restore chunk

read recovery WIP

Ramon Nou authored May 23, 2023 and Ramon Nou committed Oct 03, 2023

Parity calculation (WIP)

Reed Solomon  coding (WIP)

Ramon Nou authored Apr 13, 2023 and Ramon Nou committed Oct 03, 2023

Added NUM_REPL env variable. (0 no replicas)

NUM_REPL  num replicas  (Replicas < servers)

Remove and truncate

Metadata replication - WIP

Metadata replication - Reattempt on stat

minimal compilation issues (c++20), srand for repl

Bitset

Bit set proposal (WIP)

Read - Write with bitset (<1024 chunks)

Changed bitset to vector

Added get_fs_config reattempt

Some more resilience on create

Added Replica_Check on write (disabled)

Added helper vector-bitset functions

Ramon Nou authored Apr 13, 2023 and Ramon Nou committed Oct 03, 2023

WIP - REPL

Added NUM_REPL env variable. (0 no replicas)

NUM_REPL  num replicas  (Replicas < servers)

Remove and truncate

Metadata replication - WIP

Metadata replication - Reattempt on stat

minimal compilation issues (c++20), srand for repl

Bitset

Bit set proposal (WIP)

Read - Write with bitset (<1024 chunks)

Changed bitset to vector

Added get_fs_config reattempt

Some more resilience on create

Added Replica_Check on write (disabled)

Added helper vector-bitset functions

Resolve "Remove Parallax Testing"

This MR updates testing for the new docker CI environment:
- Removes Parallax Testing to make faster tests (we will activate again if needed, or changes are applied)
- Adds cstdint for uint types (gcc 10..)
- Updates python sh library/package to a newer version. This solves and produces some changes.
- Solved a bug in sfind

Closes #268

Closes #268

See merge request !169

Change all env PosixPath to str

restore rocksdb

Removed extra debug line

Added max internal fds

Reduce OPEN_FDS

Missing option

Resolve "Support Spack and others"

### Usage information

Download Spack and setup environment:
```bash
git clone -c feature.manyFiles=true https://github.com/spack/spack.git
. spack/share/spack/setup-env.sh
```
Add GekkoFS Spack repository to Spack:
```bash
spack repo add gekkofs/scripts/spack
```
Check that Spack can find GekkoFS:
```
spack info gekkofs
```
Install GekkoFS (and run optional tests). Check `spack info gekkofs` for available option and versions:
```bash
spack install gekkofs
# for installing tests dependencies and running tests
spack install -v --test=root gekkofs +tests
```
Load GekkoFS into environment:
```
spack load gekkofs
```
If you want to use the latest developer branch of GekkoFS:
```
spack install gekkofs@latest
```
The default is using version 0.9.1 the last stable release.

### TODO

- [x] Base Spack functionality, versions, and configuration support
- [x] Documentation
- [x] Advanced functionality, more detailed configuration support, e.g., Parallax and Prometheus
- [x] More easy way to get path to client library
- [x] Add GekkoFS client wrapper for `LD_PRELOAD`
- [ ] Add final version to main Spack repository if possible. (Not possible right now as it not clear how 3rd party libraries should be treated.





















Closes #58

Closes #58

See merge request !137

Arm Support

This adds ARM support.

It adds a new profile to download the syscall_intercept that supports ARM (fork).
The modifications doesn't use ARM define, as it is not needed, we only need to check if a syscall exists or not.

Closes #244 
We can close !127 also, I think that this merge updates and solves some issues.

Closes #244

See merge request !160