Nafi3 authored Mar 12, 2018 and Marc Vef committed Mar 23, 2018

This commit gives new options regarding separation of metadir and rootdir location. If only rootdir destination is given both metadata and data is stored at the same path

target was the index of the cycle and target[target] was the actual
identifier of the target. In some part of the code target was considered
as the identifier directly.

Now the index of the cycle is i and target is the actual identifier of
the target.

The calculus (on clients) about the last chunk involved into a
write/read operation was bugged.

A new blocks_calc_util library has been introduced in order to make
easier to perform this kind of calculus.

In the case the host running the startup script is not one of the node
that will run adafs it does not make sense to check for the binary of
the daemon locally.

When rocksdb is build with jemalloc linking with adafs was failing

Instead use margo_iforward. Also only two bulk_handles are now used:
1 for margo_ipc and one for margo_rpc RPC requests.

The download script do not work if the specified output path
contains more then 1 non-existing folder.
This is due to the -f flag used in the readlink command.
We now use the less strict -m one.

The LRU map caused multiple issues with memory mappings and caused
severe errors and hanging situations. It has been replaced with a
std::map.

An Argobots mutex was added for rpc address lookups as many threads
might insert and lookup this map in parallel. This caused bulk_transfer
seqfauls as well as other file system crashes.

Cleanup of file system logging.

Before data was pulled first then written or read first and then pushed back

Only use 1 segment now but with multiple transfers

Source folder was removed if existed before but never recreated

Since the download of sources are performed in parallel, we cannot use
anymore a single log file.

The output of the download commands has been reduced to errrors only and
it will be outputted directly on stderr. Thus the log file stuff have been
completely removed.

Moreover the clonedeps function now takes the REPO as second parameter
and additional (non mandatory) git flags as fourth parameter.

Additional fixes have been introduced to run commands in script mode
(-q, -f, ...)

For dependencies in which a specific tags is specified we can download
the code snapshot instead of cloning the whole repo.

Perform all the download and git clone action in parallel using several
processes.

Due to the new parallel use of the clonedeps and wgetdeps functions,
some adjustment to printed message has been made. Now the function write
on stdout only once when they complete.

Use cmake in order to properly check and use both the Boost library and
the pthread one.

We need to be explicit about I/O as it cannot be simply repeated. A response
from the daemon will come eventually, either success or failure.

Previously, an ES was created for each destination in each read or write.
Creating and freeing these ES is costly. Also, these ESs had separate pools,
which is bad practice. Multiple ESs work on a single pool. The ABT_sched
decides which ULT or Tasklet is run by which ES at any time.

This change does also reduce CPU load as library IO RPC calls used CPU time
while waiting for the response.