Update README

  - Client interface and hostfile removel ([!313](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/313))

    - Demon option to avoid removing hostfile on exit : ENV variable , GKFS_KEEP_HOSTS_FILE or option --keep-hosts

    - Client Interface selection avoiding loopback setup : LIBGKFS_OFI_INTERFACE=ib0

  - Random Slicing + cutshift ([!316](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/316))

    - Added marker-based expand, shrink, and mixed mutate workflow using one shared hostfile.

    - Added Random Slicing placement with CutShift for expand-only resizes to reduce inter-node data movement.

    - Added optional expand-on-demand data movement for pure expand operations.

    - Added malleability integration coverage and user documentation.

### Changed 

    - LIBGKFS/ GKFS _SYMLINK_SUPPORT, _RENAME_SUPPORT and _CREATE_CHECK_PARENTS.

    - Now all the performance options are in config.hpp and env variables.

  - Added DIRECT_IO on server side to increase performance ([!312](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/312))

  - Shared metrics/message routing variables now use the common `GKFS_` prefix.

### Fixed

  - SYS_lstat does not exists on some architectures, change to newfstatat ([!269](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/269))

  - Fix remove chunk bug ([!294](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/294))

  - Fix decompress_and_parse_entries_standard() Unexpected end of buffer while parsing name bug ([!312](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/312))

  - Fix client dissapearing on malleability ends with an error. ([!313](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/313))

  - Fixed buffer-size validation issues in filtered directory listing paths.

## [0.9.5] - 2025-08

For shrink-only operations, no new daemons are needed; just mark leaving entries with `-`. For expand-only operations, no `-` entries are needed; start the added daemons with `GKFS_DAEMON_EXPAND=ON` so they append `+` entries.

### User-level resize examples

The examples below use plain GekkoFS commands: start daemons with `gkfs_daemon` and control topology changes with

`gkfs_malleability`. They assume GekkoFS was configured with tools enabled, for example `-DGKFS_BUILD_TOOLS=ON`, and that

`gkfs_daemon`, `gkfs_malleability`, and `libgkfs_intercept.so` come from the same installation.

Common shell setup:

```bash

# installation and workspace paths

export GKFS_INSTALL=/path/to/install

export GKFS_WORKDIR=/scratch/$USER/gkfs

export GKFS_HOSTFILE=$GKFS_WORKDIR/gkfs_hosts.txt

export GKFS_ROOTDIR=/local/ssd/$USER/gkfs-root

export GKFS_MOUNTDIR=$GKFS_WORKDIR/mount

mkdir -p "$GKFS_WORKDIR" "$GKFS_MOUNTDIR"

# client and tool input: one shared hostfile for the current GekkoFS instance

export LIBGKFS_HOSTS_FILE=$GKFS_HOSTFILE

# daemon-side hostfile path; the same path is also passed with -H below

export GKFS_HOSTS_FILE=$GKFS_HOSTFILE

# keep the hosts file while daemons are stopped or resized; this is important for marker-based mutations

export GKFS_DAEMON_KEEP_HOSTS_FILE=ON

# choose one placement strategy and export it everywhere: daemons, clients, proxies, and tools

export GKFS_DISTRIBUTION_STRATEGY=simple_hash

# CutShift substantially reduces inter-node data movement during expand-only resizes

# alternative:

# export GKFS_DISTRIBUTION_STRATEGY=random_slicing

# export GKFS_RANDOM_SLICING_CUTSHIFT=ON

# recommended on multi-node systems to avoid clients choosing loopback

export LIBGKFS_OFI_INTERFACE=ib0

```

Client environment for applications:

```bash

export LIBGKFS_HOSTS_FILE=$GKFS_HOSTFILE

export LD_PRELOAD=$GKFS_INSTALL/lib/libgkfs_intercept.so

export GKFS_DISTRIBUTION_STRATEGY=simple_hash  # or random_slicing, matching daemon/tool env

export LIBGKFS_OFI_INTERFACE=ib0

```

Before every resize operation, stop or pause applications so that no process accesses the mounted GekkoFS namespace while

redistribution is running. After `mutate finalize`, restart applications or force them to reload the updated hostfile. If

the GekkoFS proxy is used, restart proxies manually after the topology change.

Use the following helper loop to wait for a topology change to finish when running `gkfs_malleability` manually:

```bash

wait_for_mutate() {

    while ! "$GKFS_INSTALL/bin/gkfs_malleability" mutate status 2>&1 | grep -q "No mutate"; do

        sleep 2

    done

}

```

#### Expand: add daemon nodes

Start the initial instance, for example on two nodes:

```bash

rm -f "$GKFS_HOSTFILE"

printf "node01\nnode02\n" > "$GKFS_WORKDIR/nodes-current.txt"

srun --nodelist="$GKFS_WORKDIR/nodes-current.txt" \

     --ntasks=2 --ntasks-per-node=1 \

     "$GKFS_INSTALL/bin/gkfs_daemon" \

     -r "$GKFS_ROOTDIR" \

     -m "$GKFS_MOUNTDIR" \

     -H "$GKFS_HOSTFILE" \

     -l ib0 -P ofi+verbs &

```

To expand from two to four daemons, start only the new daemons with `GKFS_DAEMON_EXPAND=ON`. The added daemons append

`+` entries to the existing hostfile; do not delete the existing hostfile.

```bash

# applications must be stopped or paused here

printf "node03\nnode04\n" > "$GKFS_WORKDIR/nodes-add.txt"

GKFS_DAEMON_EXPAND=ON \

srun --nodelist="$GKFS_WORKDIR/nodes-add.txt" \

     --ntasks=2 --ntasks-per-node=1 \

     "$GKFS_INSTALL/bin/gkfs_daemon" \

     -r "$GKFS_ROOTDIR" \

     -m "$GKFS_MOUNTDIR" \

     -H "$GKFS_HOSTFILE" \

     -l ib0 -P ofi+verbs &

# run redistribution, poll status, and finalize the marked hostfile

"$GKFS_INSTALL/bin/gkfs_malleability" mutate start

wait_for_mutate

"$GKFS_INSTALL/bin/gkfs_malleability" mutate finalize

```

After a successful finalize, `gkfs_hosts.txt` contains four unmarked active entries. Keep an up-to-date list of all active

daemon nodes for later administration:

```bash

printf "node01\nnode02\nnode03\nnode04\n" > "$GKFS_WORKDIR/nodes-current.txt"

```

Optional lazy data movement for pure expand:

```bash

export GKFS_EXPAND_ON_DEMAND=ON

"$GKFS_INSTALL/bin/gkfs_malleability" mutate start

wait_for_mutate

"$GKFS_INSTALL/bin/gkfs_malleability" mutate finalize

```

With `GKFS_EXPAND_ON_DEMAND=ON`, metadata is still redistributed during `mutate start`, but data chunks are fetched and

materialized lazily on first access. Use this only for pure expand operations; shrink and mixed mutate use eager data

migration.

#### Shrink: remove daemon nodes

For shrink-only operations, do not start new daemons. Mark each daemon that should leave by prefixing its hostfile line

with `-`, then run mutate. The leaving daemons must remain running and reachable until finalize completes; `mutate

finalize` stops the daemons marked with `-` automatically.

Example shrink from four to two daemons:

```bash

# applications must be stopped or paused here

cp "$GKFS_HOSTFILE" "$GKFS_HOSTFILE.bak"

# mark node03 and node04 for removal. Match the real host names as they appear in the hostfile.

sed -i -E 's/^(node03[[:space:]])/-\1/' "$GKFS_HOSTFILE"

sed -i -E 's/^(node04[[:space:]])/-\1/' "$GKFS_HOSTFILE"

"$GKFS_INSTALL/bin/gkfs_malleability" mutate start

wait_for_mutate

"$GKFS_INSTALL/bin/gkfs_malleability" mutate finalize

```

After finalize, the hostfile contains only the remaining active daemon entries and the daemons that left the topology have

been stopped automatically. Update your active-node list:

```bash

printf "node01\nnode02\n" > "$GKFS_WORKDIR/nodes-current.txt"

```

#### Mutate: replace nodes in one operation

A mixed mutate, or node swap, combines shrink and expand. Mark the leaving daemon lines with `-`, start replacement

daemons with `GKFS_DAEMON_EXPAND=ON` so they append `+` entries, then run the same mutate workflow.

Example replacement of `node02` by `node05` while keeping the daemon count constant:

```bash

# applications must be stopped or paused here

cp "$GKFS_HOSTFILE" "$GKFS_HOSTFILE.bak"

# mark the old daemon that should leave

sed -i -E 's/^(node02[[:space:]])/-\1/' "$GKFS_HOSTFILE"

# start the replacement daemon; it appends a '+node05 ...' entry

printf "node05\n" > "$GKFS_WORKDIR/nodes-add.txt"

GKFS_DAEMON_EXPAND=ON \

srun --nodelist="$GKFS_WORKDIR/nodes-add.txt" \

     --ntasks=1 --ntasks-per-node=1 \

     "$GKFS_INSTALL/bin/gkfs_daemon" \

     -r "$GKFS_ROOTDIR" \

     -m "$GKFS_MOUNTDIR" \

     -H "$GKFS_HOSTFILE" \

     -l ib0 -P ofi+verbs &

# redistribute from the before topology to the after topology, stop '-' daemons, and clean markers

"$GKFS_INSTALL/bin/gkfs_malleability" mutate start

wait_for_mutate

"$GKFS_INSTALL/bin/gkfs_malleability" mutate finalize

# update the active-node list for future administration

printf "node01\nnode03\nnode04\nnode05\n" > "$GKFS_WORKDIR/nodes-current.txt"

```

Useful daemon and tool options for these examples:

| Option or command | Meaning |

|---|---|

| `gkfs_daemon -r, --rootdir <path>` | Local data directory for each daemon |

| `gkfs_daemon -m, --mountdir <path>` | GekkoFS mount directory visible to clients |

| `gkfs_daemon -H, --hosts-file <path>` | Shared hostfile written by daemons and read by clients/tools |

| `gkfs_daemon -l <iface>` | Network interface or address to bind, for example `ib0` |

| `gkfs_daemon -P <protocol>` | RPC protocol, for example `ofi+sockets` or `ofi+verbs` |

| `gkfs_malleability mutate start` | Start redistribution from the marked hostfile topology |

| `gkfs_malleability mutate status` | Poll redistribution status |

| `gkfs_malleability mutate finalize` | Stop daemons marked with `-`, clean the hostfile markers, and complete the topology change |

### Random Slicing and CutShift

Set the same distribution variables for daemons, clients, proxies, and tools:

CutShift is applied for expand-only Random Slicing operations (`+` entries present, no `-` entries). Other Random Slicing mutate cases currently fall back to the final equal RS layout.

Using `GKFS_DISTRIBUTION_STRATEGY=random_slicing` together with `GKFS_RANDOM_SLICING_CUTSHIFT=ON` is recommended for

expand-heavy workflows because it avoids rebuilding an entirely new balanced placement from scratch. Instead, CutShift

adjusts the existing Random Slicing interval layout and assigns ranges to the added daemons, so most data remains on its

current owner and only a smaller subset has to move between nodes.

Reference: Random Slicing is based on "Random slicing: Efficient and scalable data placement for large-scale storage systems" by Alberto Miranda, Sascha Effert, Yangwook Kang, Ethan L. Miller, Ivan Popov, Andre Brinkmann, Tom Friedetzky, and Toni Cortes, published in ACM Transactions on Storage 10(3), 2014. See the [Google Scholar entry](https://scholar.google.com/citations?view_op=view_citation&hl=en&user=hK-ogNAAAAAJ&cstart=20&pagesize=80&sortby=pubdate&citation_for_view=hK-ogNAAAAAJ:RGFaLdJalmkC).

### Manual `gkfs_malleability` workflow

gkfs_malleability mutate start

while ! gkfs_malleability mutate status 2>&1 | grep -q "No mutate"; do sleep 2; done

# finalize: stop '-' daemons, promote '+', preserve RS interval comments

# finalize: stop '-' daemons automatically, promote '+', preserve RS interval comments

gkfs_malleability mutate finalize

```

export LIBGKFS_HOSTS_FILE=/path/to/gkfs_workspace.txt

gkfs_malleability mutate start

while ! gkfs_malleability mutate status 2>&1 | grep -q "No mutate"; do sleep 2; done

# finalize stops the daemon marked with '-' automatically

gkfs_malleability mutate finalize

```