Tommaso Tocci authored Apr 19, 2018 and Marc Vef committed Apr 23, 2018

The creation was performed through a simple rocksDB PUT operation.

Multiple creation attempts on the same file would override the
previously inserted metadata, in particular the file size.

The creation is now handled with a new MergeOperand.
Now a creation attampt on an already existing file will be converted in
a NOP.

Tommaso Tocci authored Apr 19, 2018 and Marc Vef committed Apr 23, 2018

The new MergeOperand base class can be subclassed in order to implement
new merge-operands with their specific parameters and semantics.

Tommaso Tocci authored Apr 19, 2018 and Marc Vef committed Apr 23, 2018

The UpdateSizeOperand has been renamed IncreaseSizeOperand.

The operand was using three parameters size,offset and append_flag.

Since we only need the (size + offset) sum in order to apply the
operand, the offset field has been removed.

Tommaso Tocci authored Apr 17, 2018 and Marc Vef committed Apr 23, 2018

In order to make the file-size-update operation atomic it is necessary
to make the following action atomic:

 - READ old size value from rocksDB
 - COMPUTE new size based on the write operation
 - WRITE back the new value rocksDB

We use rocksDB merge operation [1] in order to implement the atomic update.

[1]: https://github.com/facebook/rocksdb/wiki/Merge-Operator

Tommaso Tocci authored Apr 17, 2018 and Marc Vef committed Apr 23, 2018

with the current storage back-end, chunks of a file are stored in
several files in a folder on a real kernel-space filesystem.

Removing a file means that we need to remove the entire directory and
all its sub-files (rm -r).
This operation is not atomic and it can raise errors if performed
concurrently..

This commit handles properly this kind of errors.

In case the glibc could not be loaded through dlopen print the relative
error and exit from the client

It is only relevant for sparse files but the current implementation
has a too high overhead on a single node (~10%). We need to find
a better solution at a later point.

This is an optimization for mdtest which shows very slow performance
in removing files, getting worse with an increasing number of nodes.

Since the introduction of required parameters wasn't possible to show
the help message.
In fact the --help flag was checked after all other parameters.

Postpone the creation of the OpenFile object until we are sure that the
file actually exists and we have enough access rigths for it.

Problem:
In the case the path for which the check access is performed doesn't
exists the server will set `out.err` with the correct error number (ENOENT).
On client side the value of out.err is used as return value but the
errno variable is not set.

Assert is put after a more detailed error message is written to the log.

Previously, host rpc addresses have been looked up lazily when a request
was done. This is not necessary as all hosts are looked up anyways
while the file system is running.

Now, all RPC addresses are looked up when the environment is initialized.
After that, the rpc_address map is only accessed read-only and does not
require a mutex anymore.

Since POSIX.1-2001 the S_IFMT (0170000) bitmask of the file mode must
contain the file type.

At the moment we correctly support
 - [S_IFREG] regular file
 - [S_IFDIR] directory

Previously, clients would look for the daemon by name which was not robust.
If the daemon was started with valgrind for example, clients would not find
the daemon.

Similar to other applications a daemon now creates a pid file with
information about its pid and its started mountdir.

Proper errorhandling on server and client side was added.

Multiple daemons per node are explicitly not allowed for now!

Review and fixes that died when rebasing all commits

Nafi3 authored Mar 12, 2018 and Marc Vef committed Mar 23, 2018

This commit gives new options regarding separation of metadir and rootdir location. If only rootdir destination is given both metadata and data is stored at the same path

target was the index of the cycle and target[target] was the actual
identifier of the target. In some part of the code target was considered
as the identifier directly.

Now the index of the cycle is i and target is the actual identifier of
the target.

The calculus (on clients) about the last chunk involved into a
write/read operation was bugged.

A new blocks_calc_util library has been introduced in order to make
easier to perform this kind of calculus.

When rocksdb is build with jemalloc linking with adafs was failing

Instead use margo_iforward. Also only two bulk_handles are now used:
1 for margo_ipc and one for margo_rpc RPC requests.

The LRU map caused multiple issues with memory mappings and caused
severe errors and hanging situations. It has been replaced with a
std::map.

An Argobots mutex was added for rpc address lookups as many threads
might insert and lookup this map in parallel. This caused bulk_transfer
seqfauls as well as other file system crashes.

Cleanup of file system logging.