Commit d68c1f60 authored by Ramon Nou's avatar Ramon Nou
Browse files

Merge branch 'rnou/p0' into 'master' 

fixes and refactor: preparing for 0.9.6 release

## Summary

This merge request improves GekkoFS reliability, test coverage, malleability support, client error handling, and CI tooling.

The changes also harden `sfind` against malformed directory records and make ThreadSanitizer builds work reliably in the CI environment.

## Main changes

### Reliability and RPC handling

- Contain exceptions from daemon and proxy RPC handlers.
- Make RPC responses safe when clients disconnect.
- Improve error propagation from migration and mutation operations.
- Add stronger validation around migration state transitions.
- Improve endpoint lookup retries and client interface selection.
- Improve daemon readiness detection and hostfile lifecycle handling.
- Make hostfile cleanup atomic.

### Malleability and random slicing

- Add validation and fingerprinting for random-slicing layouts.
- Improve migration status reporting.
- Handle migration failures explicitly.
- Reject invalid mutation transitions.
- Improve migration write verification and retry handling.
- Add coverage for expansion, shrinking, mutation failures, and migration behavior.
- Improve runtime metrics and Prometheus type handling.

### Client and syscall behavior

- Add `fchownat` interception support and syscall coverage.
- Return explicit errors for unsupported metadata operations.
- Align `flock` handling between syscall and libc paths.
- Improve internal symlink traversal safety.
- Improve client logging and reduce shared log contention.
- Add RAII handling for daemon and migration file descriptors.

### Testing and CI

- Add compatibility corpus workflows and tests.
- Add RPC consistency checker tooling and CI integration.
- Improve deterministic integration-test seed handling.
- Remove the obsolete BATS test system.
- Improve daemon crash, startup, disconnect, and resilience tests.
- Install clang-tidy in the testing image.
- Improve static-analysis and ABI-checking workflows.
- Stabilize statistics tests under TSAN.

### Sanitizer support

- Avoid executing sanitizer-instrumented filesystem probes during CMake configuration.
- Build TSAN executables without PIE.
- Run TSAN Catch2 discovery and tests with ASLR disabled using:
  `setarch x86_64 -R`.
- Prevent TSAN shadow-memory mapping failures during Catch2 test discovery.

### `sfind` hardening

- Validate directory record sizes before reading fields.
- Reject truncated records.
- Handle missing NUL terminators safely.
- Use bounded directory-entry names for regex and path operations.
- Protect empty working paths during path construction.


Summary:
   - Added runtime, operation, I/O, backend, migration, and latency metrics for Prometheus.
    - Added `GKFS_DAEMON_PROMETHEUS_AUTH=user:pass` for optional Pushgateway basic authentication.
    - Added unit coverage for metrics output and invalid authentication values.
  - Distributed RPC tracing and client RPC deadlines.
    - Added sampled trace correlation IDs across client, proxy, and daemon RPCs with `GKFS_ENABLE_TRACE` and `GKFS_TRACE_SAMPLE_RATE`.
    - Added optional client RPC deadlines through `LIBGKFS_RPC_TIMEOUT_MS` without retrying potentially non-idempotent operations.
  - Hostfile and malleability lifecycle metadata.
    - Added atomic hostfile updates, marker preservation, Random Slicing layout records, and layout hash validation across restart and mutate operations.
    - Added RAII file-descriptor handling and shared hostfile-management utilities.
  - Expanded diagnostics and engineering tooling.
    - Added RPC consistency checking, static-analysis helpers, compatibility-corpus workflows, logging-scale measurements, CI report indexing, and metadata reporting tools.
    -     - Reduced client logging prefixes and routed client startup summaries to the configured client log destination.
    - Standardized client and daemon log levels to full uppercase names such as `INFO`, `DEBUG`, and `ERROR`.
    - Periodic daemon statistics can be written to the configured daemon log file with `GKFS_DAEMON_STATS_LOG=ON`.
    - Updated external dependencies, including fmt, spdlog, CLI11, GSL, and MessagePack, and raised the minimum CMake version for current test tooling.
    - Added CMake presets and configuration reports for debug, release, coverage, and sanitizer builds.
    - Reworked integration and CI execution with deterministic workflows, pytest timeouts/load distribution, sanitizer coverage, and RPC consistency checks.
    - Removed the legacy Bats-based dependency-script test setup in favor of the Python testing workflows.
    - Improved client operation diagnostics, readiness handling, retry/disconnect behavior, descriptor management, and per-process/sample-based logging controls.
    -     - Fixed buffer-size validation issues in filtered directory listing paths.
    - Prometheus operation counters now update when Prometheus is enabled, including `IOPS_WRITE` and `IOPS_READ`.
    - Reduced noisy daemon errors for expected missing sparse chunk files; `ENOENT` reads are logged at debug level.
    - Hardened RPC and proxy error propagation, startup readiness, daemon disconnect handling, and malleability/migration cleanup.
    - Fixed client syscall and symlink edge cases, descriptor lifecycle issues, and compatibility-corpus snapshot/report generation.
    - Hardened `sfind` directory-record parsing and filtered-directory handling.

See merge request !317
parents 59724f6e 2e0d6906