Commit 3996ad84 authored by Ramon Nou's avatar Ramon Nou
Browse files

Merge branch 'jathenst/188-dep_scripts-unify-d-and-p-flags' into 'master' 

Resolve "dep_scripts: Unify `-d` and `-p` flags"

This MR removes the separate `-d` and `-p` and combines them in one. `-d` was removed, specific dependencies are now set with `-p`. Therefore, the following options are available:

- `-p default:latest` for accessing the latest dependencies from the default profile
- `-p mercury@default:latest` for accessing the latest mercury dependencies only from the default profile

Closes #188

Closes #188

See merge request !174
parents c81a9591 6a33df54

CHANGELOG.md

+1 −0
Original line number Diff line number Diff line
@@ -8,6 +8,7 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 
## [Unreleased]

### New

### Changed

- Unify dependency scripts (dl and compile): Unify `-d` and `-p` flags ([!174](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/174)).

### Removed

### Fixed

README.md

+2 −0
Original line number Diff line number Diff line
@@ -74,6 +74,8 @@ GekkoFS testing support: `python38-devel` (**>Python-3.6 required**) 
4. Download and compile the direct dependencies, e.g.,

    - Download example: `gekkofs/scripts/dl_dep.sh /home/foo/gekkofs_deps/git`

    - Compilation example: `gekkofs/scripts/compile_dep.sh /home/foo/gekkofs_deps/git /home/foo/gekkofs_deps/install`

    - Specific profiles or dependencies: use `-p` option to specify the profile, e.g., `mogon2:latest` which is equivalent

      to `mogon2:` or for a dependency in the mogon2:0.8.0 profile: e.g. `libfabric@mogon2:0.8.0`, the profile can be omitted

    - Consult `-h` for additional arguments for each script

5. Compile GekkoFS and run optional tests

    - Create build directory: `mkdir gekkofs/build && cd gekkofs/build`

scripts/compile_dep.sh

+15 −40
Original line number Diff line number Diff line
@@ -50,8 +50,7 @@ declare -A PROFILE_CLONEDEPS_ARGS PROFILE_CLONEDEPS_PATCHES 
usage_short() {

	echo "

usage: compile_dep.sh -h |

                      -p PROFILE_NAME[:PROFILE_VERSION] |

                      -d DEPENDENCY_NAME[[@PROFILE_NAME][:PROFILE_VERSION]] |

                      -p DEPENDENCY_NAME[[@PROFILE_NAME:][PROFILE_VERSION]] |

                      -l [PROFILE_NAME:[PROFILE_VERSION]] |

                      -h

                      [ -P PROFILES_DIR ] [ -j COMPILE_CORES] [ -n ]
@@ -74,16 +73,7 @@ optional arguments: 
    -h, --help  Shows this help message and exits

    -l, --list-dependencies [[PROFILE_NAME:]PROFILE_VERSION]

                List dependencies available for building and installation

    -p, --profile PROFILE_NAME[:PROFILE_VERSION]

                Allows installing a pre-defined set of dependencies as defined

                in \${PROFILES_DIR}/PROFILE_NAME.specs. This is useful to

                deploy specific library versions and/or configurations,

                using a recognizable name. Optionally, PROFILE_NAME may include

                a specific version for the profile, e.g. 'mogon2:latest' or

                'ngio:0.9.3', which will download the dependencies defined for

                that specific version. If unspecified, the 'default:latest' profile

                will be used, which should include all the possible dependencies.

    -d, --dependency DEPENDENCY_NAME[[@PROFILE_NAME][:PROFILE_VERSION]]

    -p, --profile DEPENDENCY_NAME[[@PROFILE_NAME:][PROFILE_VERSION]]

                Build and install a specific dependency, ignoring any --profile

                option provided. If PROFILE_NAME is unspecified, the 'default'

                profile will be used. Similarly, if PROFILE_VERSION is
@@ -373,30 +363,10 @@ while [[ $# -gt 0 ]]; do 
    case ${key} in

    -p | --profile)



        [ -n "${EXECUTION_MODE}" ] && exec_mode_error || EXECUTION_MODE='profile'



        if [[ -z "$2" ]]; then

            echo "ERROR: Missing argument for -p/--profile option"

            exit 1

        fi



        if [[ "$2" =~ ^(.*):(.*)$ ]]; then

            PROFILE_NAME="${BASH_REMATCH[1]}"

            PROFILE_VERSION="${BASH_REMATCH[2]}"

        else

            PROFILE_NAME="$2"

            PROFILE_VERSION="${DEFAULT_VERSION}"

        fi



        shift # past argument

        shift # past value

        ;;

    -d | --dependency)



        [ -n "${EXECUTION_MODE}" ] && exec_mode_error || EXECUTION_MODE='dependency'



        if [[ -z "$2" ]]; then

            echo "ERROR: Missing argument for -d/--dependency option"

            echo "ERROR: Missing argument for -p/--profile option"

            exit

        fi
@@ -417,6 +387,16 @@ while [[ $# -gt 0 ]]; do 
                PROFILE_VERSION="${BASH_REMATCH[3]}"

            fi



        # e.g. default:latest

        elif [[ "$2" =~ ^(.*):(.*)$ ]]; then

            if [[ -n "${BASH_REMATCH[1]}" ]]; then

                PROFILE_NAME="${BASH_REMATCH[1]}"

            fi



            if [[ -n "${BASH_REMATCH[2]}" ]]; then

                PROFILE_VERSION="${BASH_REMATCH[2]}"

            fi



        # e.g. mercury@mogon1

        elif [[ "$2" =~ ^(.*)@(.*)$ ]]; then

            if [[ -n "${BASH_REMATCH[1]}"  ]]; then
@@ -431,11 +411,6 @@ while [[ $# -gt 0 ]]; do 
            DEPENDENCY="$2"

        fi



        if [[ ! -n "${DEPENDENCY}" ]]; then

            echo "ERROR: Missing dependency name."

            exit 1

        fi



        shift # past argument

        shift # past value

        ;;

scripts/dl_dep.sh

+14 −40
Original line number Diff line number Diff line
@@ -307,8 +307,7 @@ wgetdeps() { 


usage_short() {

    echo "

usage: dl_dep.sh -p PROFILE_NAME[:PROFILE_VERSION] |

                 -d DEPENDENCY_NAME[[@PROFILE_NAME][:PROFILE_VERSION]] |

usage: dl_dep.sh -p DEPENDENCY_NAME[[@PROFILE_NAME:][PROFILE_VERSION]] |

                 -l [[PROFILE_NAME:]PROFILE_VERSION] |

                 -h

                 [ -P PROFILES_DIR ] [ -n ]
@@ -330,16 +329,7 @@ optional arguments: 
        -h, --help              Shows this help message and exits

        -l, --list-dependencies [[PROFILE_NAME:]PROFILE_VERSION]

                                List dependency configuration profiles available for download

        -p, --profile PROFILE_NAME[:PROFILE_VERSION]

                                Allows downloading a pre-defined set of dependencies as defined

                                in \${PROFILES_DIR}/PROFILE_NAME.specs. This is useful to 

                                deploy specific library versions and/or configurations,

                                using a recognizable name. Optionally, PROFILE_NAME may include

                                a specific version for the profile, e.g. 'mogon2:latest' or

                                'ngio:0.9.3', which will download the dependencies defined for

                                that specific version. If unspecified, the 'default:latest' profile

                                will be used, which should include all the possible dependencies.

        -d, --dependency DEPENDENCY_NAME[[@PROFILE_NAME][:PROFILE_VERSION]]

        -p, --profile DEPENDENCY_NAME[[@PROFILE_NAME:][PROFILE_VERSION]]

                                Build and install a specific dependency, ignoring any --profile

                                option provided. If PROFILE_NAME is unspecified, the 'default'

                                profile will be used. Similarly, if PROFILE_VERSION is unspecified,
@@ -368,31 +358,10 @@ while [[ $# -gt 0 ]]; do 
    case ${key} in

    -p | --profile)



        [ -n "${EXECUTION_MODE}" ] && exec_mode_error || EXECUTION_MODE='profile'



        if [[ -z "$2" ]]; then

            echo "ERROR: Missing argument for -p/--profile option"

            exit 1

        fi



        if [[ "$2" =~ ^(.*):(.*)$ ]]; then

            PROFILE_NAME="${BASH_REMATCH[1]}"

            PROFILE_VERSION="${BASH_REMATCH[2]}"

        else

            PROFILE_NAME="$2"

            PROFILE_VERSION="${DEFAULT_VERSION}"

        fi



        shift # past argument

        shift # past value

        ;;



    -d | --dependency)



        [ -n "${EXECUTION_MODE}" ] && exec_mode_error || EXECUTION_MODE='dependency'



        if [[ -z "$2" ]]; then

            echo "ERROR: Missing argument for -d/--dependency option"

            echo "ERROR: Missing argument for -p/--profile option"

            exit 1

        fi
@@ -410,6 +379,16 @@ while [[ $# -gt 0 ]]; do 
                PROFILE_VERSION="${BASH_REMATCH[3]}"

            fi



        # e.g. default:latest

        elif [[ "$2" =~ ^(.*):(.*)$ ]]; then

            if [[ -n "${BASH_REMATCH[1]}" ]]; then

                PROFILE_NAME="${BASH_REMATCH[1]}"

            fi



            if [[ -n "${BASH_REMATCH[2]}" ]]; then

                PROFILE_VERSION="${BASH_REMATCH[2]}"

            fi



        # e.g. mercury@mogon1

        elif [[ "$2" =~ ^(.*)@(.*)$ ]]; then

            if [[ -n "${BASH_REMATCH[1]}"  ]]; then
@@ -424,11 +403,6 @@ while [[ $# -gt 0 ]]; do 
            DEPENDENCY="$2"

        fi



        if [[ ! -n "${DEPENDENCY}" ]]; then

            echo "ERROR: Missing dependency name."

            exit 1

        fi



        shift # past argument

        shift # past value

        ;;

scripts/gkfs_dep.sh

+4 −16
Original line number Diff line number Diff line
@@ -32,8 +32,7 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" 
usage_short() {

	echo "

usage: compile_dep.sh -h |

                      -p PROFILE_NAME[:PROFILE_VERSION] |

                      -d DEPENDENCY_NAME[[@PROFILE_NAME][:PROFILE_VERSION]] |

                      -p DEPENDENCY_NAME[[@PROFILE_NAME:][PROFILE_VERSION]] |

                      -l [PROFILE_NAME:[PROFILE_VERSION]] |

                      -h

                      [ -P PROFILES_DIR ] [ -j COMPILE_CORES] [ -n ] [ -v ]
@@ -56,7 +55,7 @@ optional arguments: 
    -h, --help  Shows this help message and exits

    -l, --list-dependencies [[PROFILE_NAME:]PROFILE_VERSION]

                        List dependencies available for building and installation

    -p, --profile PROFILE_NAME[:PROFILE_VERSION]

    -p, --dependency DEPENDENCY_NAME[[@PROFILE_NAME:][PROFILE_VERSION]]

                        Allows installing a pre-defined set of dependencies as defined

                        in \${PROFILES_DIR}/PROFILE_NAME.specs. This is useful to

                        deploy specific library versions and/or configurations,
@@ -65,7 +64,6 @@ optional arguments: 
                        'ngio:0.9.3', which will download the dependencies defined for

                        that specific version. If unspecified, the 'default:latest' profile

                        will be used, which should include all the possible dependencies.

    -d, --dependency DEPENDENCY_NAME[[@PROFILE_NAME][:PROFILE_VERSION]]

                        Build and install a specific dependency, ignoring any --profile

                        option provided. If PROFILE_NAME is unspecified, the 'default'

                        profile will be used. Similarly, if PROFILE_VERSION is
@@ -100,11 +98,6 @@ while [[ $# -gt 0 ]]; do 
        shift # past argument

        shift # past value

        ;;

    -d | --dependency)

        DEPENDENCY="$2"

        shift # past argument

        shift # past value

        ;;

    -j | --compilecores)

        COMPILE_ARGS="$COMPILE_ARGS -j $2"

        shift # past argument
@@ -156,12 +149,7 @@ fi 
SOURCE_DIR="$(readlink -mn "${1}")"

INSTALL_DIR="$(readlink -mn "${2}")"

# build arguments

if [[ -n $DEPENDENCY ]]; then

    PROFILE_STR="-d $DEPENDENCY"

if [[ -n $PROFILE ]]; then

        PROFILE_STR="$PROFILE_STR@$PROFILE"

    fi

elif [[ -n $PROFILE ]]; then

    PROFILE_STR="-p $PROFILE"

fi

if [[ -n $PROFILE_STR ]]; then