Commit 9cafaaa3 authored by Ramon Nou's avatar Ramon Nou
Browse files

Resolve "[Alya] Create a new data distributor - Moves instantiation outside read/write"

Closes #88 
Closes #125 

The distributor will read a shared file including [filename] [host] [size] [offset] and distribute the files accordingly.
We assume, as it should happen in SLURM that the nodes are in alphabetical order.

Distributors are instantiated per read/write operation. They should be moved to FsData to avoid the overhead. It is also necessary to reduce the creation cost when the distributor reads the data mappings from a file or similar.

See merge request !39
parents 2eeb84a8 32016967
Loading
Loading
Loading
Loading
Loading
+5 −0
Original line number Diff line number Diff line
@@ -5,7 +5,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

- Added dirents_extended function and a find substitution proposal.

- Created a Guided Distributor using a mapping file to map chunks to specific
  nodes.

## [0.8.0] - 2020-09-15
## New
- Both client library and daemon have been extended to support the ofi+verbs
+7 −0
Original line number Diff line number Diff line
@@ -178,6 +178,13 @@ add_definitions(-DLIBGKFS_LOG_MESSAGE_SIZE=${CLIENT_LOG_MESSAGE_SIZE})
message(STATUS "[gekkofs] Maximum log message size in the client library: ${CLIENT_LOG_MESSAGE_SIZE}")
mark_as_advanced(CLIENT_LOG_MESSAGE_SIZE)

option(GKFS_USE_GUIDED_DISTRIBUTION "Use guided data distributor " OFF)
message(STATUS "[gekkofs] Guided data distributor: ${GKFS_USE_GUIDED_DISTRIBUTION}")

set(GKFS_USE_GUIDED_DISTRIBUTION_PATH "/tmp/guided.txt" CACHE STRING "File Path for guided distributor")
set_property(CACHE GKFS_USE_GUIDED_DISTRIBUTION_PATH PROPERTY STRINGS)
message(STATUS "[gekkofs] Guided data distributor input file path: ${GKFS_USE_GUIDED_DISTRIBUTION_PATH}")

configure_file(include/common/cmake_configure.hpp.in include/common/cmake_configure.hpp)

# Imported target
+57 −20
Original line number Diff line number Diff line
@@ -228,6 +228,7 @@ The following modules are available:
   module will only be available if the client library is built in `Debug`
   mode.
 - `all`: All previous options combined.
 - `trace_reads`: Generate log line with extra information in read operations for guided distributor
 - `help`: Print a help message and exit.

When tracing sytem calls, specific syscalls can be removed from log messages by
@@ -244,12 +245,48 @@ can be provided to set the path to the log file, and the log module can be
selected with the `GKFS_LOG_LEVEL={off,critical,err,warn,info,debug,trace}`
environment variable.


### External functions

GekkoFS allows to use external functions on your client code, via LD_PRELOAD. 
Source code needs to be compiled with -fPIC. We include a pfind io500 substitution,
 `examples/gfind/gfind.cpp` and a non-mpi version `examples/gfind/sfind.cpp`

### Data distributors
The data distribution can be selected at compilation time, we have 2 distributors available:

## Simple Hash (Default)
Chunks are distributed randomly to the different GekkoFS servers.

## Guided Distributor
Guided distributor distributes chunks using a shared file with the next format:
`<path> <chunk_number> <host>`

Moreover if you prepend a path with #, all the data from that path will go to the same place as the metadata. 
Specifically defined paths (without#) will be prioritary.

i.e.,
#/mdt-hard 0 0 

GekkoFS will store data and metadata to the same server. The server will still be random (0 0 has no meaning, yet).
 
Chunks not specified, are distributed using the Simple Hash distributor.

To generate such file we need to follow a first execution, using the trace_reads log option

This will enable a `TRACE_READS` level log at the clients offering several lines that can be used to generate the input file.
In this stage, each node should generate a separated file this can be done in SLURM using the next line :
`srun -N 10 -n 320 --export="ALL" /bin/bash -c "export LIBGKFS_LOG_OUTPUT=${HOME}/test/GLOBAL.txt;LD_PRELOAD=${GKFS_PRLD} <app>"`

Then, use the `examples/distributors/guided/generate.py` to create the output file.
* `python examples/distributors/guided/generate.py ~/test/GLOBAL.txt >> guided.txt`

This should work if the nodes are sorted in alphabetical order, which is the usual scenario. Users should take care of multi-server configurations.

Finally, enable the distributor using the next compilation flags:
* `GKFS_USE_GUIDED_DISTRIBUTION` ON
* `GKFS_USE_GUIDED_DISTRIBUTION_PATH` `<full path to guided.txt>`

### Acknowledgment

This software was partially supported by the EC H2020 funded project NEXTGenIO (Project ID: 671951, www.nextgenio.eu).
+40 −0
Original line number Diff line number Diff line
###
#  Copyright 2018-2020, Barcelona Supercomputing Center (BSC), Spain
#  Copyright 2015-2020, Johannes Gutenberg Universitaet Mainz, Germany

#  This software was partially supported by the
#  EC H2020 funded project NEXTGenIO (Project ID: 671951, www.nextgenio.eu).

#  This software was partially supported by the
#  ADA-FS project under the SPPEXA project funded by the DFG.

#  SPDX-License-Identifier: MIT
###
#!/usr/bin/python3
import re
import sys
import collections

file = sys.argv[1]

pattern = re.compile(r".+(read\ )(.*)( host: )(\d+).+(path: )(.+),.+(chunk_start: )(\d+).+(chunk_end: )(\d+)")

d = collections.OrderedDict()

with open(file) as f:
    for line in f:
        result = pattern.match(line)
        if result:
            d[result.group(2)] = 1
keys = sorted(d.keys())
i = 0
for key in keys:
    d[key] = i
    i = i + 1

with open(file) as f:
    for line in f:
        result = pattern.match(line)
        if result:
            for i in range(int(result.group(8)), int(result.group(10))+1):
                print (result.group(6), i, d[result.group(2)])
+13 −2
Original line number Diff line number Diff line
@@ -47,7 +47,7 @@

namespace gkfs::log {

enum class log_level : short {
enum class log_level : unsigned int {
    print_syscalls = 1 << 0,
    print_syscalls_entry = 1 << 1,
    print_info = 1 << 2,
@@ -57,6 +57,7 @@ enum class log_level : short {
    print_hermes = 1 << 6,
    print_mercury = 1 << 7,
    print_debug = 1 << 8,
    print_trace_reads = 1 << 9,

    // for internal use
    print_none = 0,
@@ -117,6 +118,7 @@ static const auto constexpr warning = log_level::print_warnings;
static const auto constexpr hermes = log_level::print_hermes;
static const auto constexpr mercury = log_level::print_mercury;
static const auto constexpr debug = log_level::print_debug;
static const auto constexpr trace_reads = log_level::print_trace_reads;
static const auto constexpr none = log_level::print_none;
static const auto constexpr most = log_level::print_most;
static const auto constexpr all = log_level::print_all;
@@ -125,7 +127,8 @@ static const auto constexpr help = log_level::print_help;
static const auto constexpr level_names = utils::make_array(
        "syscall",
        "syscall", // sycall_entry uses the same name as syscall
        "info", "critical", "error", "warning", "hermes", "mercury", "debug");
        "info", "critical", "error", "warning", "hermes", "mercury", "debug",
        "trace_reads");

inline constexpr auto
lookup_level_name(log_level l) {
@@ -533,6 +536,14 @@ static_buffer::grow(std::size_t size) {
        }                                                                      \
    } while(0);

#define LOG_TRACE_READS(...)                                                   \
    do {                                                                       \
        if(gkfs::log::get_global_logger()) {                                   \
            gkfs::log::get_global_logger()->log(                               \
                    gkfs::log::trace_reads, __func__, __LINE__, __VA_ARGS__);  \
        }                                                                      \
    } while(0);

#ifdef GKFS_DEBUG_BUILD

#define LOG_SYSCALL(...)                                                       \
Loading