Skip to content

Resolve "Add a CLI tool for requesting data transfers."

Alberto Miranda requested to merge amiranda/19-add-a-cli-tool-for-requesting-data-transfers into main

This MR adds the ccp tool to Cargo so that data transfers can be started from the command line. The tool supports the following arguments:

$ ccp -h
Cargo parallel copy tool
Usage: ccp [OPTIONS]

Options:
  -h,--help                   Print this help message and exit
  -s,--server ADDRESS         Address of the Cargo server (can also be
                              provided via the CCP_SERVER environment
                              variable).
  -i,--input SRC...           Input dataset(s)
  -o,--output DST...          Output dataset(s)
  --if FLAGS                  Flags for input datasets. Accepted values
                                - posix: read data using POSIX (default)
                                - mpio: read data using MPI-IO
  --of FLAGS                  Flags for output datasets. Accepted values
                                - posix: write data using POSIX (default)
                                - mpio: write data using MPI-IO

Once a Cargo server is up and running, ccp can be used as follows:

$ srun -N4 cargo -l ofi+tcp://192.18.0.10:62000 -o cargo.log
[...]
$ ccp -s ofi+tcp://192.18.0.10:62000 -i input0.dat input1.dat -o output0.dat output1.dat

Modes of operation that control how transfers should be executed can be selected through the the --if and --of options. For now, posix and mpio modes are supported. Note however that not all combinations are supported yet, and that performance will depend on whether the underlying file system supports parallel I/O through MPI-IO.

Closes #19 (closed)

Edited by Alberto Miranda

Merge request reports