Commit e28fc75e authored by Ramon Nou's avatar Ramon Nou
Browse files

Merge branch 'enable_fork' into 'master' 

Enable fork

Interception is not enabled normally on fork workloads as we need to do several low performing steps. With this merge we enable a LIBGKFS_ENABLE_FORK=1 that can be used on for example, DLIO

See merge request !299
parents ceceaa4a 1d49e4de

CHANGELOG.md

+3 −0
Original line number Diff line number Diff line
@@ -34,6 +34,9 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 
  - Optimized and fixed fuse ([!293](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/293))

    - New optimization options

    - Less contention with threading using sharding

  - Fork support in the client library ([!299](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/299))

    - Use LIBGKFS_ENABLE_FORK=1 to enable fork support in the client library.

    - This is used for example in DLIO.

README.md

+1 −0
Original line number Diff line number Diff line
@@ -608,6 +608,7 @@ The GekkoFS daemon, client, and proxy support a number of environment variables 
- `LIBGKFS_CREATE_CHECK_PARENTS` - Enable checking parent directory for existence before creating children.

- `LIBGKFS_SYMLINK_SUPPORT` - Enable support for symbolic links.

- `LIBGKFS_RENAME_SUPPORT` - Enable support for rename.

- `LIBGKFS_ENABLE_FORK` - Enable fork support in the client library, used for example in DLIO.

#### Logging

- `LIBGKFS_LOG` - Log module of the client. 

Available modules are: `none`, `syscalls`, `syscalls_at_entry`, `info`, `critical`, `errors`, `warnings`, `mercury`, `debug`, `most`, `all`, `trace_reads`, `help`.

include/client/env.hpp

+1 −0
Original line number Diff line number Diff line
@@ -94,6 +94,7 @@ static constexpr auto USE_INLINE_DATA = ADD_PREFIX("USE_INLINE_DATA"); 
static constexpr auto CREATE_WRITE_OPTIMIZATION =

        ADD_PREFIX("CREATE_WRITE_OPTIMIZATION");

static constexpr auto READ_INLINE_PREFETCH = ADD_PREFIX("READ_INLINE_PREFETCH");

static constexpr auto ENABLE_FORK = ADD_PREFIX("ENABLE_FORK");



} // namespace gkfs::env

src/client/preload.cpp

+38 −7
Original line number Diff line number Diff line
@@ -609,11 +609,42 @@ at_parent() { 


void

at_child() {

    // In the child, we should not touch the rpc_engine because it is inherited

    // and points to shared resources (Ref count > 0).

    // If we destroy it (by overwriting it), we finalize the network stack,

    // which might close sockets shared with the parent.

    // If we use it, we race with the parent.

    // Best is to disable interception so we don't use GekkoFS until exec().

    if(gkfs::env::var_is_set(gkfs::env::ENABLE_FORK)) {

        LOG(WARNING,

            "Fork interception enabled. Re-initializing GekkoFS client...");



        // We must intentionally leak the old RPC/IPC engine to prevent

        // finalizing the parent's network connections

        auto* leaked_rpc_engine =

                new std::shared_ptr<thallium::engine>(CTX->rpc_engine());

        auto* leaked_ipc_engine =

                new std::shared_ptr<thallium::engine>(CTX->ipc_engine());

        (void) leaked_rpc_engine;

        (void) leaked_ipc_engine;



        CTX->clear_hosts();

        CTX->rpc_engine(nullptr);

        if(CTX->use_proxy()) {

            CTX->clear_proxy_host();

            CTX->ipc_engine(nullptr);

        }



        // Disable interception temporarily

        CTX->disable_interception();



        // Re-initialize the environment and Margo engines

        gkfs::preload::init_environment();



        // Re-enable interception

        CTX->enable_interception();

        LOG(INFO, "GekkoFS client successfully re-initialized in child.");

    } else {

        // In the child, we should not touch the rpc_engine because it is

        // inherited and points to shared resources (Ref count > 0). If we

        // destroy it (by overwriting it), we finalize the network stack, which

        // might close sockets shared with the parent. If we use it, we race

        // with the parent. Best is to disable interception so we don't use

        // GekkoFS until exec().

        CTX->disable_interception();

    }

}