Tommaso Tocci authored Apr 17, 2018 and Marc Vef committed Apr 23, 2018

In order to make the file-size-update operation atomic it is necessary
to make the following action atomic:

 - READ old size value from rocksDB
 - COMPUTE new size based on the write operation
 - WRITE back the new value rocksDB

We use rocksDB merge operation [1] in order to implement the atomic update.

[1]: https://github.com/facebook/rocksdb/wiki/Merge-Operator

Tommaso Tocci authored Apr 17, 2018 and Marc Vef committed Apr 23, 2018

with the current storage back-end, chunks of a file are stored in
several files in a folder on a real kernel-space filesystem.

Removing a file means that we need to remove the entire directory and
all its sub-files (rm -r).
This operation is not atomic and it can raise errors if performed
concurrently..

This commit handles properly this kind of errors.

Intercept remove call

See merge request zdvresearch_bsc/adafs!48

Use special file name "nonexisting" for testing non existing file

Various fix for dependencies compile script

See merge request zdvresearch_bsc/adafs!47

Use INSTALL_PATH env variable to set rocksDB installtion path instead of using sed hack

Correctly set compilation flags in release mode

See merge request zdvresearch_bsc/adafs!46

Signed-off-by: Marc Vef <vef@uni-mainz.de>

CMAKE_CXX_FLAGS_RELEASE already defines "-DNDEBUG" and "-O3".
We set them just to be sure that even in older/newer version of Cmake
they are set correctly

dl script: exit on donwload task errors

See merge request zdvresearch_bsc/adafs!43

Signed-off-by: Marc Vef <vef@uni-mainz.de>

Log RocksDB open error details

See merge request zdvresearch_bsc/adafs!41

Signed-off-by: Marc Vef <vef@uni-mainz.de>

check dlopen errors

See merge request zdvresearch_bsc/adafs!40

Signed-off-by: Marc Vef <vef@uni-mainz.de>

Problem: If one of the donwload task fails the script do not stop and
the other task continue to donwload sources.

Solution: If one of the task fails, print an informative error message
and send a signal to the parent process that kills all the spawned tasks
and exits with error code.

Version bump dependencies. ADA-FS code modifications for Margo 0.3

See merge request zdvresearch_bsc/adafs!35

In case the glibc could not be loaded through dlopen print the relative
error and exit from the client

Fix linking of MPI libraries in ifs_test

See merge request zdvresearch_bsc/adafs!37

Signed-off-by: Marc Vef <vef@uni-mainz.de>

Tommaso Tocci authored Apr 11, 2018 and Marc Vef committed Apr 11, 2018

the MPI::MPI_CXX imported target only works for CMake > 3.9

In the case we are using a prior version we need to set include paths
and link libraries manually.

Also, startup and shutdown scripts now have a new argument
to decide on where the created pssh hostfile is stored.

It is only relevant for sparse files but the current implementation
has a too high overhead on a single node (~10%). We need to find
a better solution at a later point.

Random IO

See merge request zdvresearch_bsc/adafs!36

Signed-off-by: Marc Vef <vef@uni-mainz.de>

This is an optimization for mdtest which shows very slow performance
in removing files, getting worse with an increasing number of nodes.

cli: look for --help flag first

See merge request zdvresearch_bsc/adafs!34

Signed-off-by: Marc Vef <vef@uni-mainz.de>

Since the introduction of required parameters wasn't possible to show
the help message.
In fact the --help flag was checked after all other parameters.

mdtest summary calculation of multiple iterations might be flawed.
Also Daemon is restarted for each iteration.

Correctly set errno on open() and other open-related improovements

See merge request zdvresearch_bsc/adafs!33

Signed-off-by: Marc Vef <vef@uni-mainz.de>