Setting the properly the JMALLOC_LIBRARIES variable into the find module
made possible to avoid the conditonal inclusion of the library

Tommaso Tocci authored Apr 19, 2018 and Marc Vef committed Apr 27, 2018

The creation was performed through a simple rocksDB PUT operation.

Multiple creation attempts on the same file would override the
previously inserted metadata, in particular the file size.

The creation is now handled with a new MergeOperand.
Now a creation attampt on an already existing file will be converted in
a NOP.

Tommaso Tocci authored Apr 19, 2018 and Marc Vef committed Apr 27, 2018

The new MergeOperand base class can be subclassed in order to implement
new merge-operands with their specific parameters and semantics.

Tommaso Tocci authored Apr 19, 2018 and Marc Vef committed Apr 27, 2018

The UpdateSizeOperand has been renamed IncreaseSizeOperand.

The operand was using three parameters size,offset and append_flag.

Since we only need the (size + offset) sum in order to apply the
operand, the offset field has been removed.

Tommaso Tocci authored Apr 17, 2018 and Marc Vef committed Apr 27, 2018

In order to make the file-size-update operation atomic it is necessary
to make the following action atomic:

 - READ old size value from rocksDB
 - COMPUTE new size based on the write operation
 - WRITE back the new value rocksDB

We use rocksDB merge operation [1] in order to implement the atomic update.

[1]: https://github.com/facebook/rocksdb/wiki/Merge-Operator

Tommaso Tocci authored Apr 17, 2018 and Marc Vef committed Apr 27, 2018

with the current storage back-end, chunks of a file are stored in
several files in a folder on a real kernel-space filesystem.

Removing a file means that we need to remove the entire directory and
all its sub-files (rm -r).
This operation is not atomic and it can raise errors if performed
concurrently..

This commit handles properly this kind of errors.

Use INSTALL_PATH env variable to set rocksDB installtion path instead of using sed hack

CMAKE_CXX_FLAGS_RELEASE already defines "-DNDEBUG" and "-O3".
We set them just to be sure that even in older/newer version of Cmake
they are set correctly

Problem: If one of the donwload task fails the script do not stop and
the other task continue to donwload sources.

Solution: If one of the task fails, print an informative error message
and send a signal to the parent process that kills all the spawned tasks
and exits with error code.

In case the glibc could not be loaded through dlopen print the relative
error and exit from the client

Also, startup and shutdown scripts now have a new argument
to decide on where the created pssh hostfile is stored.

It is only relevant for sparse files but the current implementation
has a too high overhead on a single node (~10%). We need to find
a better solution at a later point.

This is an optimization for mdtest which shows very slow performance
in removing files, getting worse with an increasing number of nodes.

Since the introduction of required parameters wasn't possible to show
the help message.
In fact the --help flag was checked after all other parameters.

mdtest summary calculation of multiple iterations might be flawed.
Also Daemon is restarted for each iteration.

Postpone the creation of the OpenFile object until we are sure that the
file actually exists and we have enough access rigths for it.

Problem:
In the case the path for which the check access is performed doesn't
exists the server will set `out.err` with the correct error number (ENOENT).
On client side the value of out.err is used as return value but the
errno variable is not set.

Assert is put after a more detailed error message is written to the log.

Previously, host rpc addresses have been looked up lazily when a request
was done. This is not necessary as all hosts are looked up anyways
while the file system is running.

Now, all RPC addresses are looked up when the environment is initialized.
After that, the rpc_address map is only accessed read-only and does not
require a mutex anymore.