Resolve "loopback on client - need a Gekko ENV variable or option as the server" (bb73fd34) · Commits · hpc / gekkofs

CHANGELOG.md

+4 −0

Original line number	Diff line number	Diff line
		@@ -44,6 +44,9 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
		- Metadata batching ([!305](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/305))
		- Added client-side metadata batching for file/node creation to reduce metadata RPC bottlenecks.
		- Introduced new environment variables: `LIBGKFS_METADATA_BATCH` and `LIBGKFS_METADATA_BATCH_THRESHOLD`.
		- Client interface and hostfile removel ([!313](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/313))
		- Demon option to avoid removing hostfile on exit : ENV variable , GKFS_KEEP_HOSTS_FILE or option --keep-hosts
		- Client Interface selection avoiding loopback setup : LIBGKFS_OFI_INTERFACE=ib0



		@@ -67,6 +70,7 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
		- mmap and dangling fd issues
		- Fix remove chunk bug ([!294](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/294))
		- Fix decompress_and_parse_entries_standard() Unexpected end of buffer while parsing name bug ([!312](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/312))
		- Fix client dissapearing on malleability ends with an error. ([!313](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/313))


		## [0.9.5] - 2025-08

README.md

+26 −0

Original line number	Diff line number	Diff line
		@@ -20,6 +20,7 @@ to I/O, which reduces interferences and improves performance.
		- [The GekkoFS daemon](#the-gekkofs-daemon)
		- [Manual startup and shut down](#manual-startup-and-shut-down)
		- [GekkoFS daemon orchestration via the gkfs script (recommended)](#gekkofs-daemon-orchestration-via-the-gkfs-script-recommended)
		- [Preserving the hosts file on daemon shutdown](#preserving-the-hosts-file-on-daemon-shutdown)
		- [The GekkoFS client library](#the-gekkofs-client-library)
		- [Interposition library via system call interception](#interposition-library-via-system-call-interception)
		- [Interposition library via libc call interception](#interposition-library-via-libc-call-interception)
		@@ -168,6 +169,7 @@ Options:
		--output-stats TEXT Creates a thread that outputs the server stats each 10s to the specified file.
		--enable-prometheus Enables prometheus output and a corresponding thread.
		--prometheus-gateway TEXT Defines the prometheus gateway <ip:port> (Default 127.0.0.1:9091).
		--keep-hosts Preserves the hosts file on daemon shutdown instead of deleting it.
		--version Print version and exit.
		-t,--time TEXT Set a limit on the total run time of the slurm job allocation. Default is 15min.
		-A,--account TEXT Account for the slurm job (only required for job allocation)
		@@ -179,6 +181,25 @@ are part of the same file system, use the same `rootdir` with different `rootdir

		Shut it down by gracefully killing the process (SIGTERM).

		### Preserving the hosts file on daemon shutdown

		By default, the hosts file is destroyed when the daemon shuts down. To preserve the hosts file during daemon shutdown
		(useful in malleable workloads or when daemons shut down simultaneously), use either:

		Command line option:
		```bash
		gkfs_daemon --keep-hosts -r <fs_data_path> -m <mount_dir_path> -H <hostsfile_path>
		```

		Environment variable:
		```bash
		export GKFS_KEEP_HOSTS_FILE=ON
		gkfs_daemon -r <fs_data_path> -m <mount_dir_path> -H <hostsfile_path>
		```

		Important: When the hosts file is preserved, it will naturally become stale as daemons deregister. This is expected
		behavior in malleable workloads where explicit shrink/expand operations manage the hosts file lifecycle.

		### GekkoFS daemon orchestration via the `gkfs` script (recommended)

		The `scripts/run/gkfs` script can be used to simplify starting the GekkoFS daemon on one or multiple nodes. To start
		@@ -682,6 +703,10 @@ The GekkoFS daemon, client, and proxy support a number of environment variables
		- `LIBGKFS_SYMLINK_SUPPORT` - Enable support for symbolic links.
		- `LIBGKFS_RENAME_SUPPORT` - Enable support for rename.
		- `LIBGKFS_ENABLE_FORK` - Enable fork support in the client library, used for example in DLIO.
		- `LIBGKFS_OFI_INTERFACE` - Force the client-side libfabric interface to use (equivalent to `FI_SOCKETS_IFACE`).
		This prevents clients from binding to loopback (`127.0.0.1`) when daemons are on real NICs (e.g., `ib0`).
		Required in malleable/HPC environments where the client may resolve to a loopback address.
		Example: `export LIBGKFS_OFI_INTERFACE=ib0`
		#### Logging
		- `LIBGKFS_LOG` - Log module of the client.
		Available modules are: `none`, `syscalls`, `syscalls_at_entry`, `info`, `critical`, `errors`, `warnings`, `mercury`, `debug`, `most`, `all`, `trace_reads`, `help`.
		@@ -758,6 +783,7 @@ During write/pwrite operations, when the asynchronous write cache is enabled, th
		- `GKFS_DAEMON_CREATE_EXIST_CHECK` - Check for existence of file metadata before create in RocksDB.
		- `GKFS_DAEMON_SYMLINK_SUPPORT` - Enable support for symbolic links.
		- `GKFS_DAEMON_RENAME_SUPPORT` - Enable support for rename.
		- `GKFS_KEEP_HOSTS_FILE` - Preserve the hosts file on daemon shutdown instead of destroying it (default: OFF, use with `--keep-hosts` CLI flag).
		#### Logging
		- `GKFS_DAEMON_LOG_PATH` - Path to the log file of the daemon.
		- `GKFS_DAEMON_LOG_LEVEL` - Log level of the daemon. Available levels are: `off`, `critical`, `err`, `warn`, `info`, `debug`, `trace`.

include/client/env.hpp

+5 −0

Original line number	Diff line number	Diff line
		@@ -100,6 +100,11 @@ static constexpr auto METADATA_BATCH_THRESHOLD =
		ADD_PREFIX("METADATA_BATCH_THRESHOLD");
		static constexpr auto ASYNC_WRITE = ADD_PREFIX("ASYNC_WRITE");

		// Libfabric interface pinning (consumed by libfabric at HG_init() time)
		// OFI_INTERFACE is used with the GKFS_ prefix (e.g., LIBGKFS_OFI_INTERFACE)
		// LIBGKFS_OFI_INTERFACE is the literal env var name for client-side pinning
		static constexpr auto OFI_INTERFACE = ADD_PREFIX("OFI_INTERFACE");

		} // namespace gkfs::env

		#undef ADD_PREFIX

include/client/preload_context.hpp

+8 −0

Original line number	Diff line number	Diff line
		@@ -186,6 +186,8 @@ private:
		std::thread async_write_thread_;
		bool async_write_stop_{false};

		std::string ofi_interface_;


		public:
		static PreloadContext*
		@@ -434,6 +436,12 @@ public:
		void
		use_async_write(bool use_async_write);

		std::string
		ofi_interface() const;

		void
		ofi_interface(const std::string& ofi_interface);

		void
		start_async_write_thread();

include/common/rpc/handler_util.hpp

0 → 100644

+38 −0

Original line number	Diff line number	Diff line
		#pragma once
		#include <thallium.hpp>
		#include <spdlog/spdlog.h>

		namespace gkfs::utils {

		/**
		* @internal
		* Safe wrapper around thallium::request::respond() that contains
		* any margo_exception throws and logs them instead of aborting.
		* This is needed because respond() can throw when the client has
		* vanished mid-RPC (common in malleable workloads).
		* @endinternal
		*/
		template <typename RequestType, typename ResponseType>
		void
		safe_respond(RequestType& req, const ResponseType& resp) {
		try {
		req.respond(resp);
		} catch(const thallium::margo_exception& e) {
		// Client vanished — log and silently discard.
		// This is a normal part of malleable workloads, not an error.
		auto logger = spdlog::get("daemon");
		if(logger) {
		logger->debug(
		"handler: client vanished mid-RPC, respond failed: {}",
		e.what());
		}
		} catch(const std::exception& e) {
		// Unknown error — log but do not abort
		auto logger = spdlog::get("daemon");
		if(logger) {
		logger->error("handler: unexpected respond error: {}", e.what());
		}
		}
		}

		} // namespace gkfs::utils
		No newline at end of file