Directory support

Ramon Nourequested to merge
rnou/directory_support into main

This MR adds two features:

  • Creation of directories on the output. As cargo is normally running in the prolog the user has zero capabilities to create directories on the adhoc filesystem (as it is ephemeral). This MR creates the needed directories.

  • If the input contains a directory, we get all the files recursively. Before this MR only files can be used as input. The behaviour is that the input is used as prefix, i.e., relative directory, and will be put inside the output.

  • Prepares for 0.3.0 release

The Merge request goes over !16 (merged)

Edited by Ramon Nou

Merge request reports

lib/cargo.hpp

+3 −0
Changes for lib/cargo.hpp: 3 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -74,6 +74,9 @@ public: 
    [[nodiscard]] bool

    supports_parallel_transfer() const noexcept;



    void

    path(std::string path);



    template <typename Archive>

    void

    serialize(Archive& ar) {

lib/libcargo.cpp

+5 −0
Changes for lib/libcargo.cpp: 5 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -72,6 +72,11 @@ dataset::path() const noexcept { 
    return m_path;

};



void

dataset::path(std::string path) {

    m_path = std::move(path);

};



bool

dataset::supports_parallel_transfer() const noexcept {

    return m_type == dataset::type::parallel;

spack/packages/cargo/package.py

+1 −0
Changes for spack/packages/cargo/package.py: 1 added line, 0 removed lines.
Original line number Diff line number Diff line
@@ -36,6 +36,7 @@ class Cargo(CMakePackage): 
    version("latest", branch="main")

    version("0.1.0", sha256="981d00adefbc2ea530f57f8428bd7980e4aab2993a86d8ae4274334c8f055bdb", deprecated=True)

    version("0.2.0", sha256="fd7fa31891b3961dcb376556ec5fa028bf512d96a7c688a160f9dade58dae36f")

    version("0.3.0", branch="rnou/directory_support")



    # build variants

    variant('build_type',

src/proto/mpi/message.hpp

+1 −0
Changes for src/proto/mpi/message.hpp: 1 added line, 0 removed lines.
Original line number Diff line number Diff line
@@ -250,6 +250,7 @@ struct fmt::formatter<cargo::shaper_message> : formatter<std::string_view> { 
    }

};





template <>

struct fmt::formatter<cargo::shutdown_message> : formatter<std::string_view> {

    // parse is inherited from formatter<string_view>.

src/worker/mpio_read.cpp

+2 −1
Changes for src/worker/mpio_read.cpp: 2 added lines, 1 removed line.
Original line number Diff line number Diff line
@@ -187,7 +187,8 @@ mpio_read::progress(int ongoing_index) { 
                            .count();

            if((elapsed_seconds) > 0) {

                bw((m_block_size / (1024.0 * 1024.0)) / (elapsed_seconds));

                LOGGER_INFO("BW (write) Update: {} / {} = {} mb/s [ Sleep {} ]",

                LOGGER_DEBUG(

                        "BW (write) Update: {} / {} = {} mb/s [ Sleep {} ]",

                        m_block_size / 1024.0, elapsed_seconds, bw(),

                        sleep_value());

            }