Resolve "Release Cargo v0.3.0"

Closes #27

See merge request !19

Directory support

This MR adds two features:

- Creation of directories on the output. As cargo is normally running in the prolog the user has zero capabilities to create directories on the adhoc filesystem (as it is ephemeral). This MR creates the needed directories.

- If the input contains a directory, we get all the files recursively. Before this MR only files can be used as input. The behaviour is that the input is used as prefix, i.e., relative directory, and will be put inside the output.

- Prepares for 0.3.0 release 

 The Merge request goes over !16

See merge request !18

Ramon Nou authored Oct 30, 2023 and Ramon Nou committed Nov 07, 2023

updated thread

Moving info from master to worker progress

loop update

rebase task

added chrono fmt

Progress loop extracted

Reduce simultaneous transfers

wrong Error code in write

error control

Missing size for write

Added BW, solved sleep bug (intialization)

BW workflow finished

Support for creating directories

Directory Support

Reduce CPU usage with dynamic wait

Only call create directories, if the path is not 0

Updated Spack for 0.4.0

bw shaping workflow

This MR :
- Breaks pread / pwrite processes in progressive steps so we can control dataflow
- Adds BW_shaping RPC and MPI messages 
- Adds BW information inside the transfer_update message.
-

See merge request !16

Draft: Resolve "Cargo headers are not properly installed."

Closes #23

See merge request !17

Resolve "error: '/usr/bin/ld: cannot find -lconfig.'"

Closes #22

See merge request !15

Fixes #22

Resolve "Add `systemd` service files for Cargo"

Closes #20

See merge request !14

Directories `CMAKE_INSTALL_BINDIR` and `CMAKE_INSTALL_DATADIR` are now
exported as package variables `CARGO_BIN_INSTALL_DIR` and
`CARGO_DATA_INSTALL_DIR`. This allows consuming projects to retrieve
these values by using `find_package(cargo)`.

Resolve "Add a CLI tool for requesting data transfers."

This MR adds the `ccp` tool to Cargo so that data transfers can be started
from the command line. The tool supports the following arguments:

```console
$ ccp -h
Cargo parallel copy tool
Usage: ccp [OPTIONS]

Options:
  -h,--help                   Print this help message and exit
  -s,--server ADDRESS         Address of the Cargo server (can also be
                              provided via the CCP_SERVER environment
                              variable).
  -i,--input SRC...           Input dataset(s)
  -o,--output DST...          Output dataset(s)
  --if FLAGS                  Flags for input datasets. Accepted values
                                - posix: read data using POSIX (default)
                                - mpio: read data using MPI-IO
  --of FLAGS                  Flags for output datasets. Accepted values
                                - posix: write data using POSIX (default)
                                - mpio: write data using MPI-IO
```                                

Once a Cargo server is up and running, `ccp` can be used as follows:

```console
$ srun -N4 cargo -l ofi+tcp://192.18.0.10:62000 -o cargo.log
[...]
$ ccp -s ofi+tcp://192.18.0.10:62000 -i input0.dat input1.dat -o output0.dat output1.dat
```

Modes of operation that control how transfers should be executed can be selected
through the the `--if` and `--of` options. For now, `posix` and `mpio` modes are
supported. Note however that not all combinations are supported yet, and that 
performance will depend on whether the underlying file system supports parallel
I/O through MPI-IO.

Closes #19

See merge request !13

Resolve "Race condition when shutting down"

Closes #17

See merge request !12

Resolve "Update Spack recipe to v0.2.0"

Closes #16

See merge request !11

Resolve "Release Cargo 0.2.0"

Closes #15

See merge request !10

Resolve "`std::filesystem::filesystem_error`s not reported in log"

Closes #13

See merge request !9

1. Catch `std::system_error`s, log them, and report them as Cargo
   system errors.
2. Catch and report `std:::exception`s, log them, and report them as
   Cargo `error_code::other`

Resolve "Automatically run tests in  pipelines"

This MR enables automatic testing using `ctest`, and also 
updates the CI pipelines so that they run `build` and `test`
jobs exercising the code. We also add a `Dockerfile` that builds
an appropriate CI environment to build, test, and run Cargo.

Closes #6

Closes #6

See merge request !8

Add job for integration tests