Tommaso Tocci authored Apr 19, 2018 and Marc Vef committed Apr 23, 2018

The creation was performed through a simple rocksDB PUT operation.

Multiple creation attempts on the same file would override the
previously inserted metadata, in particular the file size.

The creation is now handled with a new MergeOperand.
Now a creation attampt on an already existing file will be converted in
a NOP.

Tommaso Tocci authored Apr 19, 2018 and Marc Vef committed Apr 23, 2018

The new MergeOperand base class can be subclassed in order to implement
new merge-operands with their specific parameters and semantics.

Tommaso Tocci authored Apr 19, 2018 and Marc Vef committed Apr 23, 2018

The UpdateSizeOperand has been renamed IncreaseSizeOperand.

The operand was using three parameters size,offset and append_flag.

Since we only need the (size + offset) sum in order to apply the
operand, the offset field has been removed.

Tommaso Tocci authored Apr 17, 2018 and Marc Vef committed Apr 23, 2018

In order to make the file-size-update operation atomic it is necessary
to make the following action atomic:

 - READ old size value from rocksDB
 - COMPUTE new size based on the write operation
 - WRITE back the new value rocksDB

We use rocksDB merge operation [1] in order to implement the atomic update.

[1]: https://github.com/facebook/rocksdb/wiki/Merge-Operator

Tommaso Tocci authored Apr 17, 2018 and Marc Vef committed Apr 23, 2018

with the current storage back-end, chunks of a file are stored in
several files in a folder on a real kernel-space filesystem.

Removing a file means that we need to remove the entire directory and
all its sub-files (rm -r).
This operation is not atomic and it can raise errors if performed
concurrently..

This commit handles properly this kind of errors.

Intercept remove call

See merge request zdvresearch_bsc/adafs!48

Use special file name "nonexisting" for testing non existing file

Various fix for dependencies compile script

See merge request zdvresearch_bsc/adafs!47

Use INSTALL_PATH env variable to set rocksDB installtion path instead of using sed hack

Correctly set compilation flags in release mode

See merge request zdvresearch_bsc/adafs!46

Signed-off-by: Marc Vef <vef@uni-mainz.de>

CMAKE_CXX_FLAGS_RELEASE already defines "-DNDEBUG" and "-O3".
We set them just to be sure that even in older/newer version of Cmake
they are set correctly

dl script: exit on donwload task errors

See merge request zdvresearch_bsc/adafs!43

Signed-off-by: Marc Vef <vef@uni-mainz.de>

Log RocksDB open error details

See merge request zdvresearch_bsc/adafs!41

Signed-off-by: Marc Vef <vef@uni-mainz.de>

check dlopen errors

See merge request zdvresearch_bsc/adafs!40

Signed-off-by: Marc Vef <vef@uni-mainz.de>

Problem: If one of the donwload task fails the script do not stop and
the other task continue to donwload sources.

Solution: If one of the task fails, print an informative error message
and send a signal to the parent process that kills all the spawned tasks
and exits with error code.

Version bump dependencies. ADA-FS code modifications for Margo 0.3

See merge request zdvresearch_bsc/adafs!35

In case the glibc could not be loaded through dlopen print the relative
error and exit from the client

Fix linking of MPI libraries in ifs_test

See merge request zdvresearch_bsc/adafs!37

Signed-off-by: Marc Vef <vef@uni-mainz.de>

Tommaso Tocci authored Apr 11, 2018 and Marc Vef committed Apr 11, 2018

the MPI::MPI_CXX imported target only works for CMake > 3.9

In the case we are using a prior version we need to set include paths
and link libraries manually.

Also, startup and shutdown scripts now have a new argument
to decide on where the created pssh hostfile is stored.

It is only relevant for sparse files but the current implementation
has a too high overhead on a single node (~10%). We need to find
a better solution at a later point.

Random IO

See merge request zdvresearch_bsc/adafs!36

Signed-off-by: Marc Vef <vef@uni-mainz.de>

This is an optimization for mdtest which shows very slow performance
in removing files, getting worse with an increasing number of nodes.