client: reload routing state after mutate without restarting

    - Hardened RPC and proxy error propagation, startup readiness, daemon disconnect handling, and malleability/migration cleanup.

    - Fixed client syscall and symlink edge cases, descriptor lifecycle issues, and compatibility-corpus snapshot/report generation.

    - Hardened `sfind` directory-record parsing and filtered-directory handling.

  - Client-side in-process reload of routing state after `gkfs_malleability mutate` (#393)

    - A long-lived client (`fuse_client` or LD_PRELOAD'd process) no longer retries removed

      daemons forever after a mutate. A dedicated watcher thread reparses the hosts file and

      atomically swaps in a new `RoutingSnapshot` on `LIBGKFS_RELOAD_SIGNAL` (default

      `SIGUSR1`), without restarting the client.

## [0.9.5] - 2025-08

To build the control tool, configure with `-DGKFS_BUILD_TOOLS=ON`. This builds `gkfs_malleability`. The control tool reads the marker hostfile directly; it does not need separate expand or shrink hostfiles and it does not start daemons. The `scripts/run/gkfs` wrapper uses the same model: prepare one workspace hostfile with `+` and `-` markers, make sure added daemons are already running, then run `scripts/run/gkfs mutate` to call `gkfs_malleability mutate start/status/finalize` and let finalize clean the workspace hostfile.

The clients can be restarted, or the user can sent to them a signal (SIGUSR1) to force a reload of the hostfile.

### Hostfile marker model

Malleability uses one workspace hostfile. Entries in this file are the source of truth:

| `GKFS_DISTRIBUTION_STRATEGY` | Common placement strategy. Supported values include `simple_hash` and `random_slicing` |

| `GKFS_RANDOM_SLICING_CUTSHIFT` | Set to `ON` to use CutShift for Random Slicing expand-only operations |

| `GKFS_EXPAND_ON_DEMAND` | Set to `ON` to enable v1 expand-only lazy data movement |

| `LIBGKFS_RELOAD_SIGNAL` | set by default to `SIGUSR1`, enables reloading of the hostfile|

## Expand-on-demand malleability

- `LIBGKFS_ASYNC_WRITE=ON` - Enable client-side asynchronous write caching for data writes (default: OFF).

##### Malleability

- `LIBGKFS_RELOAD_SIGNAL` - default SIGUSR1, Signal to reload hostfile at the end of a malleability operation.

### Daemon

#### Core

- `GKFS_DAEMON_CREATE_EXIST_CHECK` - Check for existence of file metadata before create in RocksDB.

static constexpr auto CWD = ADD_PREFIX("CWD");

static constexpr auto HOSTS_FILE = ADD_PREFIX("HOSTS_FILE");

static constexpr auto FORWARDING_MAP_FILE = ADD_PREFIX("FORWARDING_MAP_FILE");

// Signal a live client watches for to reload its routing state (hosts +

// distributor) after a `gkfs_malleability mutate` completes, without

// restarting the process. Configurable in case the host application already

// uses the default (SIGUSR1) for its own purposes.

static constexpr auto RELOAD_SIGNAL = ADD_PREFIX("RELOAD_SIGNAL");

#ifdef GKFS_ENABLE_CLIENT_METRICS

static constexpr auto METRICS_FLUSH_INTERVAL =

        ADD_PREFIX("METRICS_FLUSH_INTERVAL");

#include <vector>

#include <string>

#include <mutex>

#include <atomic>

#include <unordered_set>

#include <config.hpp>

#include <client/rpc/repair_queue.hpp>

enum class RelativizeStatus { internal, external, fd_unknown, fd_not_a_dir };

/**

 * Immutable snapshot of a client's daemon routing state: which daemons it

 * talks to, how it maps a path to one of them, and which index in `hosts` is

 * this process itself. Rebuilt whole and swapped atomically by

 * commit_routing() -- readers on any thread get a consistent view (never a

 * distributor built against a hosts list from before/after it) without

 * blocking on RPC I/O for the swap itself.

 */

struct RoutingSnapshot {

    std::vector<thallium::endpoint> hosts;

    std::shared_ptr<gkfs::rpc::Distributor> distributor;

    uint64_t local_host_id{0};

};

/**

 * Singleton class of the client context with all relevant global data

 */

    PreloadContext();

    std::shared_ptr<gkfs::filemap::OpenFileMap> ofm_;

    std::shared_ptr<gkfs::rpc::Distributor> distributor_;

    std::shared_ptr<FsConfig> fs_conf_;

    std::shared_ptr<gkfs::cache::dir::DentryCache> dentry_cache_;

    bool use_dentry_cache_{false};

    std::vector<std::string> mountdir_components_;

    std::string mountdir_;

    std::vector<thallium::endpoint> hosts_;

    // Daemon routing state. Never touch these three fields directly -- go

    // through commit_routing()/the read accessors, which are the only things

    // that keep them consistent with each other under concurrent RPC traffic

    // (see RoutingSnapshot).

    std::shared_ptr<const RoutingSnapshot> routing_{

            std::make_shared<const RoutingSnapshot>()};

    mutable std::mutex routing_mutex_;

    thallium::endpoint proxy_host_;

    uint64_t local_host_id_;

    uint64_t fwd_host_id_;

    std::string rpc_protocol_;

    bool auto_sm_{false};

    std::thread async_write_thread_;

    bool async_write_stop_{false};

    // Watches for LIBGKFS_RELOAD_SIGNAL (default SIGUSR1, blocked

    // process-wide so it only ever arrives here via sigwait()) and reloads

    // routing_ from the current hosts file when it fires -- see

    // gkfs::utils::reload_client_topology().

    std::thread reload_watcher_thread_;

    std::atomic<bool> reload_watcher_stop_{false};

    std::atomic<bool> reload_in_progress_{false};

    gkfs::rpc::repair_queue repair_queue_;

    mutable std::mutex repair_queue_mutex_;

    std::condition_variable repair_queue_cv_;

    const std::string&

    cwd() const;

    const std::vector<thallium::endpoint>&

    std::shared_ptr<const std::vector<thallium::endpoint>>

    hosts() const;

    /**

     * Replaces the daemon routing state (hosts, distributor, and this

     * process's index into hosts) as one atomic unit. This is the ONLY way

     * routing_ is mutated -- init_environment() and

     * gkfs::utils::reload_client_topology() both funnel through it, so a

     * reader can never observe a distributor built against a hosts list

     * from a different generation.

     */

    void

    hosts(const std::vector<thallium::endpoint>& addrs);

    commit_routing(std::vector<thallium::endpoint> hosts,

                   std::shared_ptr<gkfs::rpc::Distributor> distributor,

                   uint64_t local_host_id);

    void

    clear_hosts();

    uint64_t

    local_host_id() const;

    void

    local_host_id(uint64_t id);

    uint64_t

    fwd_host_id() const;

    const std::shared_ptr<gkfs::filemap::OpenFileMap>&

    file_map() const;

    void

    distributor(std::shared_ptr<gkfs::rpc::Distributor> distributor);

    std::shared_ptr<gkfs::rpc::Distributor>

    distributor() const;

    void

    async_write_worker();

    /**

     * Blocks LIBGKFS_RELOAD_SIGNAL (default SIGUSR1) on the calling thread --

     * call once, early, before any other thread (Margo/Argobots/FUSE) is

     * spawned, so every thread this process ever creates inherits the

     * blocked mask and the signal can only ever be consumed by

     * reload_watcher_worker()'s sigwait().

     */

    void

    block_reload_signal();

    void

    start_reload_watcher_thread();

    void

    stop_reload_watcher_thread();

    void

    reload_watcher_worker();

    void

    enqueue_async_write(const std::string& path, const void* buf,

                        off64_t offset, size_t count, int8_t num_copies);

std::vector<std::pair<std::string, std::string>>

read_hosts_file();

void

/**

 * @brief Resolves Mercury/Thallium addresses for a set of daemons and finds

 * this process's own position among them.

 * @param hosts vector<pair<hostname, Mercury URI address>>

 * @return {resolved endpoints (same order/index as `hosts`), this host's

 * index into that vector}

 * @throws std::runtime_error through lookup_endpoint()

 *

 * Pure computation -- does not touch PreloadContext. Callers

 * (init_environment() and reload_client_topology()) are responsible for

 * committing the result via PreloadContext::commit_routing(), together with a

 * distributor built from the same (hosts, local_host_id) pair, so the three

 * never drift out of sync.

 */

std::pair<std::vector<thallium::endpoint>, uint64_t>

connect_to_hosts(const std::vector<std::pair<std::string, std::string>>& hosts);

void

void

lookup_proxy_addr();

/**

 * @brief Reparses the current hosts file and rebuilds this client's routing

 * state (hosts + distributor) in place, then commits it atomically via

 * PreloadContext::commit_routing() -- without restarting the process or

 * tearing down the Margo/Mercury engine.

 *

 * Mirrors MalleableManager::reload_hosts_file() on the daemon side. Called

 * from PreloadContext::reload_watcher_worker() after LIBGKFS_RELOAD_SIGNAL

 * fires (default SIGUSR1), which our own launcher scripts send to every live

 * client right after `gkfs_malleability mutate finalize` completes.

 *

 * Refuses to run (logs and returns, leaving the current routing state

 * untouched) if the hosts file still carries '+'/'-' markers -- i.e. if a

 * mutate is still in progress rather than finalized.

 */

void

reload_client_topology();

} // namespace gkfs::utils