scord
[](https://img.shields.io/gitlab/pipeline-status/eu/admire/io-scheduler?gitlab_url=https%3A%2F%2Fstorage.bsc.es%2Fgitlab%2F&logo=gitlab)
[](https://storage.bsc.es/gitlab/eu/admire/io-scheduler/-/commits/main)
[](https://storage.bsc.es/gitlab/eu/admire/io-scheduler/-/releases)
[](https://img.shields.io/gitlab/license/eu/admire/io-scheduler?gitlab_url=https%3A%2F%2Fstorage.bsc.es%2Fgitlab)
[](https://en.wikipedia.org/wiki/C%2B%2B20)
The storage coordinator for HPC clusters
## Overview
Scord is an open-source storage coordination service designed for Linux HPC
clusters. As a storage coordination service, Scord provides facilities for
system administrators to easily model their storage hierarchies and expose them
to users in a consistent manner. Once the storage architecture is modelled,
Scord can efficiently orchestrate data transfers between the different
storage tiers in order to support complex HPC workloads using node-local
storage, burst buffers and ad-hoc storage systems.
> **ℹ️** **Important**
> This software was partially supported by the EuroHPC-funded project ADMIRE
(Project ID: 956748, https://www.admire-eurohpc.eu).
## Building and installing
This section describes how to build and install Scord. Two main options
are basically supported out of the box:
- Automatically building both Scord and its dependencies using
[Spack](https://github.com/spack/spack).
- Build and install Scord manually.
### Building scord and its dependencies with Spack
Scord and all its dependencies can be built using
[Spack](https://github.com/spack/spack). If you already have Spack, make sure
you have the latest release. If you use a clone of the Spack `develop`
branch, be sure to pull the latest changes.
#### Installing Spack
If you haven't already, install Spack with the following commands:
```shell
$ git clone -c feature.manyFiles=true https://github.com/spack/spack
```
This will create a directory called `spack` in your machine. Once you have
cloned Spack, we recommend sourcing the appropriate script for your shell.
This will add Spack to your `PATH` and enable the use of the `spack` command:
```shell
# For bash/zsh/sh
$ . spack/share/spack/setup-env.sh
# For tcsh/csh
$ source spack/share/spack/setup-env.csh
# For fish
$ . spack/share/spack/setup-env.fish
```
Since Scord is not yet available in the official Spack repositories, you need
to add the Scord Spack repository to the local Spack namespace in your
machine. To do so, download the
[`spack/` directory](https://storage.bsc.es/gitlab/eu/admire/io-scheduler/-/tree/main/spack)
located in the root of Scord's repository root to your machine (e.g. under
`~/projects/scord/spack`) and execute the following:
```shell
spack repo add ~/projects/scord/spack/
```
You should now be able to fetch information from the Scord package using
Spack:
```shell
spack info scord
```
If that worked, you are now ready to install Scord:
```shell
spack install scord
```
You can include or remove variants with Spack when a custom Scord
build is desired. The available variants are listed below:
| Variant | Command | Default | Description |
|---------|-------------|---------|--------------------------------------------------------------------------|
| OFI | `scord+ofi` | True | Use [libfabric](https://github.com/ofiwg/libfabric) as transport library |
| UCX | `scord+ucx` | False | Use [ucx](https://github.com/openucx/ucx.git) as transport library |
> ⚠️ **Attention**
> The initial install could take a while as Spack will install build
> dependencies (autoconf, automake, m4, libtool, pkg-config, etc.) as well as
> any dependencies of dependencies (cmake, perl, etc.) if you don’t already
> have these dependencies installed through Spack.
After the installation completes, remember that you first need to load
Scord in order to use it:
```shell
spack load scord
```
### Building scord manually
If you prefer to build and install Scord from sources, you can also do so
but bear in mind that compiling and running Scord requires up-to-date
versions of various software packages that will need to be available in your
system. Also beware that using excessively old versions of these packages can
cause indirect errors that are very difficult to track down.
With that in mind, the following software packages are required to build Scord
and need to be available in the system:
- A C++ compiler that supports the C++20 standard, for example
[GCC](https://gcc.gnu.org)/[Clang](https://clang.llvm.org/) versions
8.0/6.0 or later.
- [CMake](https://cmake.org) 3.19 or later.
- pkg-config 0.29.1 (earlier versions will probably work but haven't been tested).
- [Margo](https://github.com/mochi-hpc/mochi-margo) version 0.9.8 and later, and its dependencies:
- [Argobots](https://github.com/pmodels/argobots) version 1.1 or later.
- [Mercury](https://github.com/mercury-hpc/mercury) version 2.0.1 or later.
- [libfabric](https://github.com/ofiwg/libfabric) version 1.14.0rc3 and/or
[ucx](https://github.com/openucx/ucx.git) version 1.13.
- [JSON-C](https://github.com/json-c/json-c) version 0.13.1.
- [Thallium](https://github.com/mochi-hpc/mochi-thallium) version 0.10.1 or
later.
- [libconfig-dev] version 1.4.9 or later.
- [agios](https://github.com/francielizanon/agios) (development branch)
- [redis-plus-plus](https://github.com/sewenew/redis-plus-plus) version 1.3.
3 or later, and its dependencies:
- [hiredis](https://github.com/redis/hiredis) version 0.14.1 or later.
The following libraries are also required by Scord, but will be
automatically downloaded and compiled by the project as part of the standard
build process.
- [{fmt}](https://fmt.dev/latest/index.html) version 8.0.1 or later.
- [spdlog](https://github.com/gabime/spdlog) version 1.9.2 or later.
- [Catch2](https://github.com/catchorg/Catch2) version 3.0.1 or later.
- [tl/expected](https://github.com/TartanLlama/expected).
- [Rapid YAML](https://github.com/biojppm/rapidyaml) version 0.5.0 and later.
> **ℹ️** **Important**
Margo and Argobots use `pkg-config` to ensure they compile and link correctly
with all of their dependencies' libraries. When building Scord manually,
you'll need to appropriately set either the ``PKG_CONFIG_PATH`` environment
variable or the ``CMAKE_PREFIX_PATH`` CMake variable to the appropriate
installation paths where the ``.pc`` files for Argobots and Margo reside.
#### Building
You may either install the full sources or clone the repository directly. The
package relies on CMake to build the service, which requires you to do an
out-of-source build. Thus, once the Scord sources are located in an
appropriate `SCORD_SOURCES` directory (for example, `$HOME/scord`), you can
build the service by running the following commands:
```bash
# replace