Libslurmadmcli: installing and testing

Hi,

I'm creating an issue to track progress and problems with the Slurm plugin that handles scord options.

I've pushed a new version that works with the latest API (with adhoc_resources and job_resources).

Looks like you're using an older version of scord that doesn't have the job_resources stuff?

Regarding reallocarray, are you using glibc, and if so which version? There might be a feature test macro to enable.

Hi @clement , yes it was an older version. All is compiling fine (some issues but with the correct .pc works), now.

icc_server / icc_client are working.

About glibc, it is 2.17-326, it doesn't have reallocarray on stdlib.h

How I should test the features we want to show?

Ah yes reallocarray was added to glibc 2.28 I think.

So once you have scord running you can use srun:

srun -N3 --adm-adhoc-nodes=2 --adm-adhoc-context=in_job:shared --adm-adhoc-walltime=3600 hostname

And you should see the request going to the server.

Ah and you have to enable the plugin in /etc/slurm/plugstack.conf.d/admire.conf:

optional libslurmadmcli.so scord_addr=ofi+tcp://127.0.0.1:52000 scord_proto=ofi+tcp://127.0.0.1

I should write this down somewhere.

added in progress integration labels

changed milestone to %ADMIRE mid-term review

Hi, in #78 (closed) we noticed that there was no way to choose the type of the adhoc storage instance to register, so we have implemented it in !53 (merged). Please @clement, @rnou update your working copies and any code using ADM_register_adhoc_storage() to add the extra field.

What's the difference between ADM_register_adhoc_storage and ADM_storage_create? I've been using the latter.

ADM_storage_create() is an internal function just to create the type itself. We should have made it so that it could not be called by client code but didn't get to it yet. It just allocates space and fills it with the values provided but doesn't contact scord to report the creation of the storage tier.

ADM_register_adhoc_storage() is the official way of registering an adhoc_storage instance into scord. As such, it contacts the daemon and returns a valid ADM_storage_t with a valid internal id that scord can use to track subsequent calls on the same storage tier.

We are missing JOB_ID in register_job / register_adhoc (Issue #83 (closed)). gkfs script doesn't use the list of nodes, yet, it relays on SLURM_JOB_ID

I will add it to job::resources since it is simpler than updating the API itself. Also, it makes sense if in the future we add something like a slurm_context to the job::resources.

Actually, it makes more sense to add it directly to ADM_register_job, otherwise it gives the impression that ADM_update_job could be used to change the slurm id of an already registered job :/

Added in !56 (merged). @clement if @rnou approves and integrates the MR into main you'll need to update your calls to ADM_register_job() with an extra argument for the SLURM_JOB_ID.

Sure, I'll add that.

Pushed.

I have a partial issue (but I understand that is due to that the scenario is not covered yet).

When we launch the gkfs script, it issues some srun to launch the daemon instances (in the same nodes, or not using the job id). This instances are going through the plugin, so in scord we get register_job and register_adhoc_storage / deploy_adhoc_storage again.

I managed to avoid it using adhoc_nodes as a shortcut to avoid entering again the loop, as a quick fix

However, I still don't know if we should register that job or not in scord.

You mean a simple srun without the scord options but run in the context of a job where these options have been passed will behaves as if the same options were passed?

If so I think this is expected behaviour for Slurm. There is some environment variables you can unset to avoid that. Alternatively, I can add an option to override arguments processing.

Right, probably is the expected behaviour, now what is happening

1. user srun --adhoc....
2. slurm plugin intercepts and register job/register storage/deploy storage
3. scord Register job / register storage / deploy storage
4. scord "gkfs script (deploy)" uses srun to launch the gkfs_daemon instances
5. slurm plugin intercepts the gkfs srun, and again register job/storage/deploy (without adhoc_nodes, it seems)
6. scord Register job / register storage / deploy storage (again)

Yes, in the absence of the option on the command line, Slurm uses the environment variables SLURM_SPANK__SLURM_SPANK_OPTION_admire_cli_*. This is "useful" because it means you can do an salloc that register all options, and a subsequent naked srun will use them.

To handle the problem:

1. you could unset all the the environment variables that start with SLURM_SPANK__SLURM_SPANK_OPTION_admire_cli_ before calling srun for gkfs deployment
2. I can add an option (something like --adm-adhoc-ignore) to disable processing

Not sure which way is cleaner.

By the way, this made me realise I go through the plugin even in the absence of option, which is not ok. I pushed a fix.

Closing this in favor of newer issues in milestone %Integration with Admire components

closed