Setting the properly the JMALLOC_LIBRARIES variable into the find module
made possible to avoid the conditonal inclusion of the library

Tommaso Tocci authored Apr 19, 2018 and Marc Vef committed Apr 27, 2018

The creation was performed through a simple rocksDB PUT operation.

Multiple creation attempts on the same file would override the
previously inserted metadata, in particular the file size.

The creation is now handled with a new MergeOperand.
Now a creation attampt on an already existing file will be converted in
a NOP.

Tommaso Tocci authored Apr 19, 2018 and Marc Vef committed Apr 27, 2018

The new MergeOperand base class can be subclassed in order to implement
new merge-operands with their specific parameters and semantics.

Tommaso Tocci authored Apr 19, 2018 and Marc Vef committed Apr 27, 2018

The UpdateSizeOperand has been renamed IncreaseSizeOperand.

The operand was using three parameters size,offset and append_flag.

Since we only need the (size + offset) sum in order to apply the
operand, the offset field has been removed.

Tommaso Tocci authored Apr 17, 2018 and Marc Vef committed Apr 27, 2018

In order to make the file-size-update operation atomic it is necessary
to make the following action atomic:

 - READ old size value from rocksDB
 - COMPUTE new size based on the write operation
 - WRITE back the new value rocksDB

We use rocksDB merge operation [1] in order to implement the atomic update.

[1]: https://github.com/facebook/rocksdb/wiki/Merge-Operator

Tommaso Tocci authored Apr 17, 2018 and Marc Vef committed Apr 27, 2018

with the current storage back-end, chunks of a file are stored in
several files in a folder on a real kernel-space filesystem.

Removing a file means that we need to remove the entire directory and
all its sub-files (rm -r).
This operation is not atomic and it can raise errors if performed
concurrently..

This commit handles properly this kind of errors.

Since the introduction of required parameters wasn't possible to show
the help message.
In fact the --help flag was checked after all other parameters.

Previously, clients would look for the daemon by name which was not robust.
If the daemon was started with valgrind for example, clients would not find
the daemon.

Similar to other applications a daemon now creates a pid file with
information about its pid and its started mountdir.

Proper errorhandling on server and client side was added.

Multiple daemons per node are explicitly not allowed for now!

Review and fixes that died when rebasing all commits

Nafi3 authored Mar 12, 2018 and Marc Vef committed Mar 23, 2018

This commit gives new options regarding separation of metadir and rootdir location. If only rootdir destination is given both metadata and data is stored at the same path

When rocksdb is build with jemalloc linking with adafs was failing

The LRU map caused multiple issues with memory mappings and caused
severe errors and hanging situations. It has been replaced with a
std::map.

An Argobots mutex was added for rpc address lookups as many threads
might insert and lookup this map in parallel. This caused bulk_transfer
seqfauls as well as other file system crashes.

Cleanup of file system logging.

Before data was pulled first then written or read first and then pushed back

Only use 1 segment now but with multiple transfers

Use cmake in order to properly check and use both the Boost library and
the pthread one.

daemon related cmake directives have been pushed into a standalon cmake
file into the src/daemon folder.

- Previously the directory hierarchy was not clear regarding to which
  file belong to which part of the project (client or daemon).
  Further, we will have other clients in the future (such as Fuse).
- CMake files now differentiate between include dirs for all targets
  and target specific ones.
- Removed duplicate -pg flag.
- Not listing header files when adding executables or libraries
  is considered bad practice. Note that include_directories() is adding
  include paths to the code while adding all files used for executables
  and libraries provide the context which files belong to each binary.
  When only include_directories() is set, CMake assumes that all files
  belong to a binary (which is not necessarily true). As a result,
  some IDEs may break as the do not support this assumption.

  In general we should almost always favor explicitness over implicitness.

ADA-FS's statfs will fill the struct mostly with rough estimations.
It is not supposed to return accurate data but "some" data for applications to work

Preload lib: RPC/IPC Mercury/Margo environments are only initialized if the mountdir path is used by the application
For that, the daemon writes the mountdir (it was started with) to a configurable path to disk.
The libraries read this value and only the process that uses part of this path will actually trigger the environment initialization.
This forbids prior behavior where applications initialize the whole environment numerous times, although unused.
For example, any MPI program will spawn multiple processes, all with the LD_PRELOAD environment variable set.

Added: register daemon to system for storing auxiliary files + better daemon startup and shutdown handling

- Add: adafs_access, open with check if object exists
- Add: root metadentry to database of each node
- Modified: Open, unlink, mkdir, rmdir and combined their functionality to "create node" and "remove node"