version random slicing layout metadata (f84a85f1) · Commits · hpc / gekkofs

README.md

+3 −2

Original line number	Diff line number	Diff line
		@@ -516,6 +516,7 @@ Malleability uses one workspace hostfile. Entries in this file are the source of
		\| `+node uri ...` \| daemon being added; new daemons write this automatically when started with `GKFS_DAEMON_EXPAND=ON` \|
		\| `-node uri ...` \| daemon being removed \|
		\| `# GKFS_RS_INTERVAL host=<id> start=<x> end=<y>` \| Random Slicing interval-table comment; ignored by normal hostfile parsing but loaded by Random Slicing clients/daemons \|
		\| `# GKFS_RS_LAYOUT version=1 hash=<hash>` \| Versioned canonical hash for the adjacent Random Slicing interval table; a present record must match before clients/daemons install the table \|

		Important: `+` and `-` daemons are still reachable during `mutate start`. The marker describes the future topology after finalize, not process liveness.

		@@ -757,9 +758,9 @@ export GKFS_RANDOM_SLICING_CUTSHIFT=ON

		These are normal GekkoFS environment variables, not `scripts/run/gkfs` options. The wrapper does not interpret them; it only inherits and forwards its process environment when starting daemons or running `gkfs_malleability`.

		For `random_slicing`, `mutate start` writes the chosen interval table into the workspace hostfile as `# GKFS_RS_INTERVAL ...` comments. `mutate finalize` preserves those comments while removing `-` entries and promoting `+` entries. Later clients and restarted daemons load the comments and reconstruct the same Random Slicing layout instead of rebuilding a fresh equal layout.
		For `random_slicing`, `mutate start` writes the chosen interval table into the workspace hostfile as `# GKFS_RS_INTERVAL ...` comments and a matching `# GKFS_RS_LAYOUT version=1 hash=...` record. `mutate finalize` preserves this layout metadata while removing `-` entries and promoting `+` entries. Later clients and restarted daemons validate the hash, then reconstruct the same Random Slicing layout instead of rebuilding a fresh equal layout.

		CutShift is applied for expand-only Random Slicing operations (`+` entries present, no `-` entries). Other Random Slicing mutate cases currently fall back to the final equal RS layout.
		With `GKFS_RANDOM_SLICING_CUTSHIFT=ON`, CutShift uses the persisted old layout to minimize movement for expand, shrink, and mixed marker transitions. If a legacy hostfile lacks a valid old interval table, the mutation falls back to a rebuilt equal old layout before producing and persisting the final table.

		Using `GKFS_DISTRIBUTION_STRATEGY=random_slicing` together with `GKFS_RANDOM_SLICING_CUTSHIFT=ON` is recommended for
		expand-heavy workflows because it avoids rebuilding an entirely new balanced placement from scratch. Instead, CutShift

WIP.md

+12 −9

Original line number	Diff line number	Diff line
		@@ -115,9 +115,11 @@ size, metadata, sparse regions, and rename/truncate semantics.

		### 4. Complete the Random Slicing + CutShift state model

		Status: Partial. Interval persistence, validation, reconstruction, and
		layout hashes are implemented. Versioned replay and movement-bound validation
		remain.
		Status: Done for persisted interval-table reconstruction. The hostfile
		stores a versioned Random Slicing table and canonical hash; clients and daemons
		validate and reconstruct it at startup/reload. CutShift transition and movement
		bound coverage is maintained in unit tests. Deterministic replay history is not
		needed while the final table remains authoritative.

		Problem: Rebuilding a distributor from only the final host count can lose
		the stateful CutShift transition and cause excessive movement. Restarted
		@@ -125,12 +127,13 @@ clients and daemons must reconstruct the same layout.

		Work:

		- Persist the final interval table or deterministic replay history in the
		hostfile or a versioned sidecar.
		- Define canonical host sorting and stable host IDs.
		- Reconstruct old and new layouts from marker sets during mutation.
		- Validate interval coverage, non-overlap, ordering, and numeric boundaries.
		- Add a layout hash to logs and status output.
		- Persist the final interval table and version/hash metadata in the hostfile.
		- Use canonical `(hostname, URI)` sorting and compact stable IDs for each final
		topology.
		- Reconstruct old and new layouts from marker sets during mutation, validate
		coverage/non-overlap/order/numeric bounds, and reject a mismatched versioned
		layout hash at startup or reload.
		- Keep the layout hash in logs and test transition movement bounds.

		Acceptance: All participants compute the same layout hash after repeated
		expand/shrink cycles. Movement is measured against the expected CutShift bound.

include/common/malleability_markers.hpp

+9 −0

Original line number	Diff line number	Diff line
		@@ -32,6 +32,7 @@
		#include <string>
		#include <vector>
		#include <cstdint>
		#include <optional>

		namespace gkfs {
		namespace malleable {
		@@ -82,6 +83,11 @@ struct RandomSlicingIntervalComment {
		double end{0.0};
		};

		struct RandomSlicingLayoutComment {
		uint64_t version{0};
		std::string hash;
		};

		/**
		* @brief Check if a line starts with a malleability marker
		* @param line The line to check
		@@ -140,6 +146,9 @@ write_clean_hostfile(const std::string& path, const HostfileMarkers& markers);
		std::vector<RandomSlicingIntervalComment>
		parse_rs_interval_comments(const std::string& path);

		std::optional<RandomSlicingLayoutComment>
		parse_rs_layout_comment(const std::string& path);

		void
		write_clean_hostfile(
		const std::string& path, const HostfileMarkers& markers,

src/client/preload.cpp

+10 −3

Original line number	Diff line number	Diff line
		@@ -330,7 +330,7 @@ init_environment() {
		}
		if(config.is_random_slicing()) {
		if(const auto* hostfile = std::getenv(gkfs::env::HOSTS_FILE)) {
		auto rs = dynamic_cast<gkfs::rpc::RandomSlicingDistributor*>(
		auto* rs = static_cast<gkfs::rpc::RandomSlicingDistributor*>(
		distributor.get());
		auto comments =
		gkfs::malleable::parse_rs_interval_comments(hostfile);
		@@ -343,14 +343,21 @@ init_environment() {
		static_cast<gkfs::rpc::host_t>(comment.host_id)});
		}
		if(rs && !intervals.empty()) {
		const auto layout =
		gkfs::malleable::parse_rs_layout_comment(hostfile);
		const auto hash = gkfs::rpc::interval_table_hash(intervals);
		if(layout &&
		(layout->version != 1 \|\| layout->hash != hash)) {
		throw std::runtime_error(
		"Random-slicing layout metadata does not match hostfile intervals");
		}
		if(!rs->set_intervals(intervals)) {
		throw std::runtime_error(
		"Invalid random-slicing interval table in hostfile");
		}
		LOG(INFO,
		"{}() Loaded {} random-slicing intervals from hostfile (layout_hash={})",
		__func__, intervals.size(),
		gkfs::rpc::interval_table_hash(intervals));
		__func__, intervals.size(), hash);
		}
		}
		}

src/common/malleability_markers.cpp

+103 −4

Original line number	Diff line number	Diff line
		@@ -46,6 +46,41 @@ namespace malleable {

		namespace {

		constexpr const char* rs_interval_prefix = "# GKFS_RS_INTERVAL ";
		constexpr const char* rs_layout_prefix = "# GKFS_RS_LAYOUT ";

		std::string
		rs_layout_hash(const std::vector<RandomSlicingIntervalComment>& intervals) {
		auto sorted = intervals;
		std::sort(sorted.begin(), sorted.end(),
		[](const auto& lhs, const auto& rhs) {
		if(lhs.start != rhs.start) {
		return lhs.start < rhs.start;
		}
		if(lhs.end != rhs.end) {
		return lhs.end < rhs.end;
		}
		return lhs.host_id < rhs.host_id;
		});

		std::ostringstream canonical;
		canonical << std::fixed << std::setprecision(9);
		for(const auto& interval : sorted) {
		canonical << interval.host_id << ':'
		<< static_cast<float>(interval.start) << ':'
		<< static_cast<float>(interval.end) << ';';
		}

		uint64_t hash = 14695981039346656037ULL;
		for(const auto byte : canonical.str()) {
		hash ^= static_cast<uint64_t>(static_cast<unsigned char>(byte));
		hash *= 1099511628211ULL;
		}
		std::ostringstream formatted;
		formatted << std::hex << std::setw(16) << std::setfill('0') << hash;
		return formatted.str();
		}

		void
		write_atomic(const std::string& path, const std::vector<std::string>& lines) {
		const auto temporary_path =
		@@ -132,7 +167,8 @@ parse_hostfile_markers(const std::string& path) {
		// Preserve non-random-slicing comments. Random-slicing comments are
		// regenerated from the validated interval table.
		if(line[0] == '#') {
		if(line.rfind("# GKFS_RS_INTERVAL ", 0) != 0) {
		if(line.rfind(rs_interval_prefix, 0) != 0 &&
		line.rfind(rs_layout_prefix, 0) != 0) {
		result.comments.push_back(line);
		}
		continue;
		@@ -245,13 +281,13 @@ parse_rs_interval_comments(const std::string& path) {

		std::string line;
		while(std::getline(file, line)) {
		if(line.rfind("# GKFS_RS_INTERVAL ", 0) != 0) {
		if(line.rfind(rs_interval_prefix, 0) != 0) {
		continue;
		}

		RandomSlicingIntervalComment interval;
		std::istringstream iss(
		line.substr(std::string("# GKFS_RS_INTERVAL ").size()));
		line.substr(std::string(rs_interval_prefix).size()));
		std::string token;
		bool have_host = false;
		bool have_start = false;
		@@ -283,6 +319,58 @@ parse_rs_interval_comments(const std::string& path) {
		return intervals;
		}

		std::optional<RandomSlicingLayoutComment>
		parse_rs_layout_comment(const std::string& path) {
		std::ifstream file(path);
		if(!file.is_open()) {
		throw std::runtime_error(fmt::format(
		"Failed to open hostfile for RS layout metadata: '{}': {}",
		path, strerror(errno)));
		}

		std::optional<RandomSlicingLayoutComment> layout;
		std::string line;
		while(std::getline(file, line)) {
		if(line.rfind(rs_layout_prefix, 0) != 0) {
		continue;
		}
		if(layout) {
		throw std::runtime_error(
		"Multiple random-slicing layout metadata comments found");
		}

		RandomSlicingLayoutComment parsed;
		bool have_version = false;
		bool have_hash = false;
		std::istringstream iss(
		line.substr(std::string(rs_layout_prefix).size()));
		std::string token;
		while(iss >> token) {
		const auto pos = token.find('=');
		if(pos == std::string::npos) {
		throw std::runtime_error(fmt::format(
		"Malformed random-slicing layout metadata: '{}'",
		line));
		}
		const auto key = token.substr(0, pos);
		const auto value = token.substr(pos + 1);
		if(key == "version") {
		parsed.version = static_cast<uint64_t>(std::stoull(value));
		have_version = true;
		} else if(key == "hash") {
		parsed.hash = value;
		have_hash = !parsed.hash.empty();
		}
		}
		if(!have_version \|\| !have_hash) {
		throw std::runtime_error(fmt::format(
		"Incomplete random-slicing layout metadata: '{}'", line));
		}
		layout = std::move(parsed);
		}
		return layout;
		}

		void
		write_clean_hostfile(
		const std::string& path, const HostfileMarkers& markers,
		@@ -315,6 +403,11 @@ write_clean_hostfile(
		lines.push_back(line.str());
		}

		if(!rs_intervals.empty()) {
		lines.push_back(fmt::format("{}version=1 hash={}", rs_layout_prefix,
		rs_layout_hash(rs_intervals)));
		}

		write_atomic(path, lines);
		}

		@@ -332,7 +425,8 @@ write_rs_interval_comments(
		std::vector<std::string> lines;
		std::string line;
		while(std::getline(in, line)) {
		if(line.rfind("# GKFS_RS_INTERVAL ", 0) == 0) {
		if(line.rfind(rs_interval_prefix, 0) == 0 \|\|
		line.rfind(rs_layout_prefix, 0) == 0) {
		continue;
		}
		lines.push_back(line);
		@@ -354,6 +448,11 @@ write_rs_interval_comments(
		<< interval.start << " end=" << interval.end;
		updated_lines.push_back(updated_line.str());
		}
		if(!rs_intervals.empty()) {
		updated_lines.push_back(fmt::format("{}version=1 hash={}",
		rs_layout_prefix,
		rs_layout_hash(rs_intervals)));
		}
		write_atomic(path, updated_lines);
		}