fixes and refactor: preparing for 0.9.6 release

Ramon Nourequested to merge
rnou/p0 into master

Summary

This merge request improves GekkoFS reliability, test coverage, malleability support, client error handling, and CI tooling.

The changes also harden sfind against malformed directory records and make ThreadSanitizer builds work reliably in the CI environment.

Main changes

Reliability and RPC handling

  • Contain exceptions from daemon and proxy RPC handlers.
  • Make RPC responses safe when clients disconnect.
  • Improve error propagation from migration and mutation operations.
  • Add stronger validation around migration state transitions.
  • Improve endpoint lookup retries and client interface selection.
  • Improve daemon readiness detection and hostfile lifecycle handling.
  • Make hostfile cleanup atomic.

Malleability and random slicing

  • Add validation and fingerprinting for random-slicing layouts.
  • Improve migration status reporting.
  • Handle migration failures explicitly.
  • Reject invalid mutation transitions.
  • Improve migration write verification and retry handling.
  • Add coverage for expansion, shrinking, mutation failures, and migration behavior.
  • Improve runtime metrics and Prometheus type handling.

Client and syscall behavior

  • Add fchownat interception support and syscall coverage.
  • Return explicit errors for unsupported metadata operations.
  • Align flock handling between syscall and libc paths.
  • Improve internal symlink traversal safety.
  • Improve client logging and reduce shared log contention.
  • Add RAII handling for daemon and migration file descriptors.

Testing and CI

  • Add compatibility corpus workflows and tests.
  • Add RPC consistency checker tooling and CI integration.
  • Improve deterministic integration-test seed handling.
  • Remove the obsolete BATS test system.
  • Improve daemon crash, startup, disconnect, and resilience tests.
  • Install clang-tidy in the testing image.
  • Improve static-analysis and ABI-checking workflows.
  • Stabilize statistics tests under TSAN.

Sanitizer support

  • Avoid executing sanitizer-instrumented filesystem probes during CMake configuration.
  • Build TSAN executables without PIE.
  • Run TSAN Catch2 discovery and tests with ASLR disabled using: setarch x86_64 -R.
  • Prevent TSAN shadow-memory mapping failures during Catch2 test discovery.

sfind hardening

  • Validate directory record sizes before reading fields.
  • Reject truncated records.
  • Handle missing NUL terminators safely.
  • Use bounded directory-entry names for regex and path operations.
  • Protect empty working paths during path construction.

Summary:

  • Added runtime, operation, I/O, backend, migration, and latency metrics for Prometheus. - Added GKFS_DAEMON_PROMETHEUS_AUTH=user:pass for optional Pushgateway basic authentication. - Added unit coverage for metrics output and invalid authentication values.
  • Distributed RPC tracing and client RPC deadlines.
    • Added sampled trace correlation IDs across client, proxy, and daemon RPCs with GKFS_ENABLE_TRACE and GKFS_TRACE_SAMPLE_RATE.
    • Added optional client RPC deadlines through LIBGKFS_RPC_TIMEOUT_MS without retrying potentially non-idempotent operations.
  • Hostfile and malleability lifecycle metadata.
    • Added atomic hostfile updates, marker preservation, Random Slicing layout records, and layout hash validation across restart and mutate operations.
    • Added RAII file-descriptor handling and shared hostfile-management utilities.
  • Expanded diagnostics and engineering tooling.
    • Added RPC consistency checking, static-analysis helpers, compatibility-corpus workflows, logging-scale measurements, CI report indexing, and metadata reporting tools.
    • - Reduced client logging prefixes and routed client startup summaries to the configured client log destination.
    • Standardized client and daemon log levels to full uppercase names such as INFO, DEBUG, and ERROR.
    • Periodic daemon statistics can be written to the configured daemon log file with GKFS_DAEMON_STATS_LOG=ON.
    • Updated external dependencies, including fmt, spdlog, CLI11, GSL, and MessagePack, and raised the minimum CMake version for current test tooling.
    • Added CMake presets and configuration reports for debug, release, coverage, and sanitizer builds.
    • Reworked integration and CI execution with deterministic workflows, pytest timeouts/load distribution, sanitizer coverage, and RPC consistency checks.
    • Removed the legacy Bats-based dependency-script test setup in favor of the Python testing workflows.
    • Improved client operation diagnostics, readiness handling, retry/disconnect behavior, descriptor management, and per-process/sample-based logging controls.
    • - Fixed buffer-size validation issues in filtered directory listing paths.
    • Prometheus operation counters now update when Prometheus is enabled, including IOPS_WRITE and IOPS_READ.
    • Reduced noisy daemon errors for expected missing sparse chunk files; ENOENT reads are logged at debug level.
    • Hardened RPC and proxy error propagation, startup readiness, daemon disconnect handling, and malleability/migration cleanup.
    • Fixed client syscall and symlink edge cases, descriptor lifecycle issues, and compatibility-corpus snapshot/report generation.
    • Hardened sfind directory-record parsing and filtered-directory handling.
Edited by Ramon Nou

Merge request reports