The host kernel must be configured to allow user land containers:
-$ sudo sysctl -w kernel.unprivileged_userns_clone=1+
+sudo sysctl -w kernel.unprivileged_userns_clone=1 +sudo sysctl -w kernel.apparmor_restrict_unprivileged_unconfined=0 +sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 +
Docker also provides a tool to verify the kernel configuration:
$ dockerd-rootless-setuptool.sh check --force @@ -353,6 +357,62 @@ exit $rc+
If you wish to accommodate multiple runtimes in your environment, +it is possible to do so with a bit of extra setup. This section outlines one +possible way to do so:
+ ++IgnoreFileConfigJson=true +RunTimeRun="/opt/slurm-oci/run %b %m %u %U %n %j %s %t %@" +RunTimeKill="kill -s SIGTERM %p" +RunTimeDelete="kill -s SIGKILL %p" ++
+#!/bin/bash +if [[ -e ~/.slurm-oci-run ]]; then + ~/.slurm-oci-run "$@" +else + /opt/slurm-oci/slurm-oci-run-default "$@" +fi ++
+#!/bin/bash --login +# Parse +CONTAINER="$1" +SPOOL_DIR="$2" +USER_NAME="$3" +USER_ID="$4" +NODE_NAME="$5" +JOB_ID="$6" +STEP_ID="$7" +TASK_ID="$8" +shift 8 # subsequent arguments are the command to run in the container +# Run +apptainer run --bind /var/spool --containall "$CONTAINER" "$@" ++
chmod +x /opt/slurm-oci/run /opt/slurm-oci/slurm-oci-run-default+
Once this is done, users may create a script at '~/.slurm-oci-run' if +they wish to customize the container run process, such as using a different +container runtime. Users should model this file after the default +'/opt/slurm-oci/slurm-oci-run-default'
+docker execcommand is not supported.
docker composecommand is not supported.
docker pausecommand is not supported.
docker unpausecommand is not supported.
docker swarmcommand is not supported.
docker exec command is not supported.docker swarm command is not supported.docker compose/docker-compose command is not
+ supported.docker pause command is not supported.docker unpause command is not supported.docker swarm command is not supported.docker commands are not supported inside of containers.podman execcommand is not supported.
podman kubecommand is not supported.
podman podcommand is not supported.
podman exec command is not supported.podman-compose command is not supported, due to only being
+ partially implemented. Some compositions may work but each container
+ may be run on different nodes. The network for all containers must be
+ the network_mode: host device.podman kube command is not supported.podman pod command is not supported.podman farm command is not supported.podman commands are not supported inside of containers.Last modified 08 October 2024
+Last modified 27 November 2024
diff --git a/doc/html/faq.shtml b/doc/html/faq.shtml index 9f7d09449e2..4d53e124d4f 100644 --- a/doc/html/faq.shtml +++ b/doc/html/faq.shtml @@ -1231,9 +1231,9 @@ that node may be rendered unusable, but no other harm will result.Do I need to maintain synchronized
clocks on the cluster?
-In general, yes. Having inconsistent clocks may cause nodes to
-be unusable. Slurm log files should contain references to
-expired credentials. For example:
error: Munge decode failed: Expired credential
ENCODED: Wed May 12 12:34:56 2008
@@ -2438,6 +2438,6 @@ dset TV::parallel_configs {
}
!-->
-Last modified 07 November 2024
+Last modified 19 November 2024
diff --git a/doc/html/rest_api.shtml b/doc/html/rest_api.shtml
index 3e29da52719..f540d0c64dd 100644
--- a/doc/html/rest_api.shtml
+++ b/doc/html/rest_api.shtml
@@ -3,7 +3,7 @@
API to access and control Slurm
More information: https://www.schedmd.com/
Contact Info: sales@schedmd.com
- Version: Slurm-24.05.4&openapi/slurmdbd&openapi/slurmctld
+ Version: Slurm-24.05.5&openapi/slurmdbd&openapi/slurmctld
BasePath:
Apache 2.0
https://www.apache.org/licenses/LICENSE-2.0.html
@@ -6169,7 +6169,7 @@
Query Parameter — CSV QOS list default: null format (optional)
- Query Parameter — CSV format list default: null id (optional)
+ Query Parameter — Ignored; process JSON manually to control output format default: null id (optional)
Query Parameter — CSV id list default: null only_defaults (optional)
@@ -6297,7 +6297,7 @@
Query Parameter — CSV QOS list default: null format (optional)
- Query Parameter — CSV format list default: null id (optional)
+ Query Parameter — Ignored; process JSON manually to control output format default: null id (optional)
Query Parameter — CSV id list default: null only_defaults (optional)
@@ -6433,7 +6433,7 @@
Query Parameter — Query flags default: null format (optional)
- Query Parameter — CSV format list default: null rpc_version (optional)
+ Query Parameter — Ignored; process JSON manually to control output format default: null rpc_version (optional)
Query Parameter — CSV RPC version list default: null usage_end (optional)
@@ -7132,7 +7132,7 @@
Query Parameter — CSV QOS list default: null format (optional)
- Query Parameter — CSV format list default: null id (optional)
+ Query Parameter — Ignored; process JSON manually to control output format default: null id (optional)
Query Parameter — CSV id list default: null only_defaults (optional)
@@ -7640,7 +7640,7 @@
Query Parameter — CSV QOS list default: null format (optional)
- Query Parameter — CSV format list default: null id (optional)
+ Query Parameter — Ignored; process JSON manually to control output format default: null id (optional)
Query Parameter — CSV id list default: null only_defaults (optional)
@@ -8156,7 +8156,7 @@
Query Parameter — Query flags default: null format (optional)
- Query Parameter — CSV format list default: null rpc_version (optional)
+ Query Parameter — Ignored; process JSON manually to control output format default: null rpc_version (optional)
Query Parameter — CSV RPC version list default: null usage_end (optional)
@@ -9960,7 +9960,7 @@
Query Parameter — CSV extra list default: null format (optional)
- Query Parameter — CSV format list default: null instance_id (optional)
+ Query Parameter — Ignored; process JSON manually to control output format default: null instance_id (optional)
Query Parameter — CSV instance_id list default: null instance_type (optional)
@@ -10088,7 +10088,7 @@
Query Parameter — CSV extra list default: null format (optional)
- Query Parameter — CSV format list default: null instance_id (optional)
+ Query Parameter — Ignored; process JSON manually to control output format default: null instance_id (optional)
Query Parameter — CSV instance_id list default: null instance_type (optional)
@@ -11488,7 +11488,7 @@
Query Parameter — Include job environment default: null format (optional)
- Query Parameter — CSV format list default: null groups (optional)
+ Query Parameter — Ignored; process JSON manually to control output format default: null groups (optional)
Query Parameter — CSV group list default: null job_name (optional)
@@ -12758,7 +12758,7 @@
Query Parameter — CSV QOS id list default: null format (optional)
- Query Parameter — CSV format list default: null name (optional)
+ Query Parameter — Ignored; process JSON manually to control output format default: null name (optional)
Query Parameter — CSV QOS name list default: null preempt_mode (optional)
@@ -14740,7 +14740,7 @@
Query Parameter — CSV cluster name list default: null format (optional)
- Query Parameter — CSV format name list default: null id (optional)
+ Query Parameter — Ignored; process JSON manually to control output format default: null id (optional)
Query Parameter — CSV id list default: null name (optional)
@@ -15436,7 +15436,7 @@
Query Parameter — CSV QOS id list default: null format (optional)
- Query Parameter — CSV format list default: null name (optional)
+ Query Parameter — Ignored; process JSON manually to control output format default: null name (optional)
Query Parameter — CSV QOS name list default: null preempt_mode (optional)
@@ -15859,7 +15859,7 @@
Query Parameter — CSV cluster name list default: null format (optional)
- Query Parameter — CSV format name list default: null id (optional)
+ Query Parameter — Ignored; process JSON manually to control output format default: null id (optional)
Query Parameter — CSV id list default: null name (optional)
diff --git a/doc/man/man1/srun.1 b/doc/man/man1/srun.1
index 3f6b128bc9f..fc2ff9d03f3 100644
--- a/doc/man/man1/srun.1
+++ b/doc/man/man1/srun.1
@@ -1,4 +1,4 @@
-.TH srun "1" "Slurm Commands" "October 2024" "Slurm Commands"
+.TH srun "1" "Slurm Commands" "November 2024" "Slurm Commands"
.SH "NAME"
srun \- Run parallel jobs
@@ -4647,6 +4647,35 @@ This behavior can be changed by adding \fBSelectTypeParameters=CR_Pack_Nodes\fR
to your slurm.conf. The logic to pack nodes will allow job steps to start on
a single node without having to explicitly request a single node.
+.TP
+\fBExample 11:\fR
+This example demonstrates that adding the \fB\-\-exclusive\fR flag to job
+allocation requests can give different results based on whether you also
+request a certain number of tasks.
+
+Requesting exclusive access with no additional requirements will allow the
+process to access all the CPUs on the allocated node.
+.nf
+$ srun \-l \-\-exclusive bash \-c 'grep Cpus_allowed_list /proc/self/status'
+0: Cpus_allowed_list: 0\-23
+.fi
+
+Adding a request for a certain number of tasks will cause each task to only
+have access to a single CPU.
+.nf
+$ srun \-l \-\-exclusive \-n2 bash \-c 'grep Cpus_allowed_list /proc/self/status'
+0: Cpus_allowed_list: 0
+1: Cpus_allowed_list: 12
+.fi
+
+You can define the number of CPUs per task if you want to give them access to
+more than one CPU.
+.nf
+$ srun \-l \-\-exclusive \-n2 \-\-cpus\-per\-task=12 bash \-c 'grep Cpus_allowed_list /proc/self/status'
+0: Cpus_allowed_list: 0\-5,12\-17
+1: Cpus_allowed_list: 6\-11,18\-23
+.fi
+
.SH "COPYING"
Copyright (C) 2006\-2007 The Regents of the University of California.
Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
diff --git a/doc/man/man5/slurm.conf.5 b/doc/man/man5/slurm.conf.5
index 165c7e2a6e4..98ba99e476f 100644
--- a/doc/man/man5/slurm.conf.5
+++ b/doc/man/man5/slurm.conf.5
@@ -2719,8 +2719,10 @@ plugin.
.TP
\fBreorder_count=#\fR
Specify how many attempts should be made in reordering preemptable jobs to
-minimize the count of jobs preempted.
+minimize the total number of jobs that will be preempted.
The default value is 1. High values may adversely impact performance.
+Changes to the order of jobs on these attempts can be enabled with
+\fBstrict_order\fR.
The logic to support this option is only available in the select/cons_tres
plugin.
.IP
@@ -2735,10 +2737,11 @@ otherwise a SIGTERM will be sent to the tasks.
.TP
\fBstrict_order\fR
-If set, then execute extra logic in an attempt to preempt only the lowest
-priority jobs.
-It may be desirable to set this configuration parameter when there are multiple
-priorities of preemptable jobs.
+When reordering preemptable jobs, place the most recently tested job at the
+front of the list since we are certain that it actually added resources needed
+by the new job. This ensures that with enough reorder attempts, the minimum
+possible number of jobs will be preempted.
+See also \fBreorder_count=#\fR.
The logic to support this option is only available in the select/cons_tres
plugin.
.IP
@@ -4152,6 +4155,13 @@ Enable job steps that span heterogeneous job allocations.
The default value.
.IP
+.TP
+\fBenable_job_state_cache\fR
+Enables an independent cache of job state details within slurmctld. This allows
+processing of `\fBsqueue\fR \-\-only\-job\-state` and replaced RPCs with minimal
+impact on other slurmctld operations.
+.IP
+
.TP
\fBenable_user_top\fR
Enable use of the "scontrol top" command by non\-privileged users.
@@ -4843,13 +4853,6 @@ filenames have no path separators and are located adjacent to slurm.conf.
Glob patterns (See \fBglob\fR (7)) are not supported.
.IP
-.TP
-\fBenable_job_state_cache\fR
-Enables an independent cache of job state details within slurmctld. This allows
-processing of `\fBsqueue\fR \-\-only\-job\-state` and replaced RPCs with minimal
-impact on other slurmctld operations.
-.IP
-
.TP
\fBidle_on_node_suspend\fR
Mark nodes as idle, regardless of current state, when suspending nodes with
diff --git a/doc/man/man8/slurmd.8 b/doc/man/man8/slurmd.8
index e35ff45dad3..7f4a4a3f605 100644
--- a/doc/man/man8/slurmd.8
+++ b/doc/man/man8/slurmd.8
@@ -77,10 +77,13 @@ NodeName=node1 CPUs=16 RealMemory=30000 Gres=gpu:2"
.IP
.TP
-\fB\-\-conf\-server [:]\fR
+\fB\-\-conf\-server [:]\fR
Comma\-separated list of controllers, the first being the primary slurmctld. A
port can (optionally) be specified for each controller. These hosts are where
the slurmd will fetch the configuration from when running in "configless" mode.
+\fBNOTE\fR: If specifying an IPv6 address, wrap the in [] to
+distinguish the address from the port. This is required even if no port is
+specified.
.IP
.TP
diff --git a/slurm.spec b/slurm.spec
index aab4cdd9c1e..37b02792e27 100644
--- a/slurm.spec
+++ b/slurm.spec
@@ -1,7 +1,7 @@
Name: slurm
-Version: 24.05.4
+Version: 24.05.5
%define rel 1
-Release: %{rel}%{?dist}
+Release: %{rel}.%{gittag}%{?dist}%{?gpu}.ug
Summary: Slurm Workload Manager
Group: System Environment/Base
@@ -15,7 +15,10 @@ URL: https://slurm.schedmd.com/
%global slurm_source_dir %{name}-%{version}-%{rel}
%endif
-Source: %{slurm_source_dir}.tar.bz2
+# We do not need this, but then duplicate file warning will pop up
+# %define _build_id_links none
+
+Source: %{slurm_source_dir}.tar.gz
# build options .rpmmacros options change to default action
# ==================== ==================== ========================
@@ -301,7 +304,9 @@ database changes to slurmctld daemons on each cluster
Summary: Slurm\'s implementation of the pmi libraries
Group: System Environment/Base
Requires: %{name}%{?_isa} = %{version}-%{release}
+%if ! %{with pmix}
Conflicts: pmix-libpmi
+%endif
%description libpmi
Slurm\'s version of libpmi. For systems using Slurm, this version
is preferred over the compatibility libraries shipped by the PMIx project.
@@ -362,7 +367,13 @@ BuildRequires: http-parser-devel
%if %{defined suse_version}
BuildRequires: libjson-c-devel
%else
+%if 0%{?fedora} || 0%{?rhel} > 7
BuildRequires: json-c-devel
+Requires: json-c
+%else
+BuildRequires: json-c12-devel
+Requires: json-c12
+%endif
%endif
%description slurmrestd
Provides a REST interface to Slurm.
@@ -407,7 +418,7 @@ export QA_RPATHS=0x5
# Strip out some dependencies
cat > find-requires.sh <<'EOF'
-exec %{__find_requires} "$@" | egrep -v '^libpmix.so|libevent|libnvidia-ml'
+exec %{__find_requires} "$@" | egrep -v '^libpmix.so|libevent'
EOF
chmod +x find-requires.sh
%global _use_internal_dependency_generator 0
@@ -594,16 +605,16 @@ rm -rf %{buildroot}
%files torque
%defattr(-,root,root)
-%{_bindir}/pbsnodes
-%{_bindir}/qalter
-%{_bindir}/qdel
-%{_bindir}/qhold
-%{_bindir}/qrerun
-%{_bindir}/qrls
-%{_bindir}/qstat
-%{_bindir}/qsub
-%{_bindir}/mpiexec
-%{_bindir}/generate_pbs_nodefile
+#%{_bindir}/pbsnodes
+#%{_bindir}/qalter
+#%{_bindir}/qdel
+#%{_bindir}/qhold
+#%{_bindir}/qrerun
+#%{_bindir}/qrls
+#%{_bindir}/qstat
+#%{_bindir}/qsub
+#%{_bindir}/mpiexec
+#%{_bindir}/generate_pbs_nodefile
%{_libdir}/slurm/job_submit_pbs.so
%{_libdir}/slurm/spank_pbs.so
#############################################################################
diff --git a/src/api/job_step_info.c b/src/api/job_step_info.c
index 373f389145c..d1dd90a501d 100644
--- a/src/api/job_step_info.c
+++ b/src/api/job_step_info.c
@@ -315,8 +315,22 @@ static int _get_stepmgr_steps(void *x, void *arg)
slurm_msg_t req_msg;
slurm_msg_t_init(&req_msg);
slurm_msg_set_r_uid(&req_msg, slurm_conf.slurmd_user_id);
- slurm_conf_get_addr(sji->stepmgr, &req_msg.address,
- req_msg.flags);
+
+ if (slurm_conf_get_addr(sji->stepmgr, &req_msg.address, req_msg.flags))
+ {
+ /*
+ * The node isn't in the conf, see if the
+ * controller has an address for it.
+ */
+ slurm_node_alias_addrs_t *alias_addrs = NULL;
+ if (!slurm_get_node_alias_addrs(sji->stepmgr, &alias_addrs)) {
+ add_remote_nodes_to_conf_tbls(alias_addrs->node_list,
+ alias_addrs->node_addrs);
+ slurm_free_node_alias_addrs(alias_addrs);
+ slurm_conf_get_addr(sji->stepmgr, &req_msg.address,
+ req_msg.flags);
+ }
+ }
job_step_info_request_msg_t req_data = {0};
req_data.step_id.job_id = sji->job_id;
@@ -738,7 +752,8 @@ extern int slurm_job_step_stat(slurm_step_id_t *step_id,
memcpy(&req, step_id, sizeof(req));
memcpy(&resp_out->step_id, step_id, sizeof(resp_out->step_id));
- req_msg.protocol_version = use_protocol_ver;
+ req_msg.protocol_version = MIN(SLURM_PROTOCOL_VERSION,
+ use_protocol_ver);
req_msg.msg_type = REQUEST_JOB_STEP_STAT;
req_msg.data = &req;
diff --git a/src/common/fetch_config.c b/src/common/fetch_config.c
index fceccde29a8..0428f3e51bb 100644
--- a/src/common/fetch_config.c
+++ b/src/common/fetch_config.c
@@ -47,6 +47,7 @@
#include "src/common/slurm_protocol_pack.h"
#include "src/common/slurm_resolv.h"
#include "src/common/strlcpy.h"
+#include "src/common/util-net.h"
#include "src/common/xstring.h"
#include "src/common/xmalloc.h"
@@ -63,7 +64,8 @@ static char *client_config_files[] = {
};
-static void _init_minimal_conf_server_config(List controllers);
+static void _init_minimal_conf_server_config(List controllers, bool use_v6,
+ bool reinit);
static int to_parent[2] = {-1, -1};
@@ -114,6 +116,7 @@ static config_response_msg_t *_fetch_parent(pid_t pid)
static void _fetch_child(List controllers, uint32_t flags)
{
config_response_msg_t *config;
+ ctl_entry_t *ctl = NULL;
buf_t *buffer = init_buf(1024 * 1024);
int len = 0;
@@ -128,9 +131,22 @@ static void _fetch_child(List controllers, uint32_t flags)
*/
slurm_conf_unlock();
- _init_minimal_conf_server_config(controllers);
+ ctl = list_peek(controllers);
+
+ if (ctl->has_ipv6 && !ctl->has_ipv4)
+ _init_minimal_conf_server_config(controllers, true, false);
+ else
+ _init_minimal_conf_server_config(controllers, false, false);
+
config = fetch_config_from_controller(flags);
+ if (!config && ctl->has_ipv6 && ctl->has_ipv4) {
+ warning("%s: failed to fetch remote configs via IPv4, retrying with IPv6: %m",
+ __func__);
+ _init_minimal_conf_server_config(controllers, true, true);
+ config = fetch_config_from_controller(flags);
+ }
+
if (!config) {
error("%s: failed to fetch remote configs: %m", __func__);
safe_write(to_parent[1], &len, sizeof(int));
@@ -150,6 +166,16 @@ static void _fetch_child(List controllers, uint32_t flags)
_exit(1);
}
+static int _get_controller_addr_type(void *x, void *arg)
+{
+ ctl_entry_t *ctl = (ctl_entry_t *) x;
+
+ host_has_addr_family(ctl->hostname, NULL, &ctl->has_ipv4,
+ &ctl->has_ipv6);
+
+ return SLURM_SUCCESS;
+}
+
extern config_response_msg_t *fetch_config(char *conf_server, uint32_t flags)
{
char *env_conf_server = getenv("SLURM_CONF_SERVER");
@@ -178,9 +204,21 @@ extern config_response_msg_t *fetch_config(char *conf_server, uint32_t flags)
server = strtok_r(tmp, ",", &save_ptr);
while (server) {
ctl_entry_t *ctl = xmalloc(sizeof(*ctl));
+ char *tmp_ptr = NULL;
+
+ if (server[0] == '[')
+ server++;
+
strlcpy(ctl->hostname, server, sizeof(ctl->hostname));
- if ((port = xstrchr(ctl->hostname, ':'))) {
+ if ((tmp_ptr = strchr(ctl->hostname, ']'))) {
+ *tmp_ptr = '\0';
+ tmp_ptr++;
+ } else {
+ tmp_ptr = ctl->hostname;
+ }
+
+ if ((port = xstrchr(tmp_ptr, ':'))) {
*port = '\0';
port++;
ctl->port = atoi(port);
@@ -198,6 +236,8 @@ extern config_response_msg_t *fetch_config(char *conf_server, uint32_t flags)
}
}
+ list_for_each(controllers, _get_controller_addr_type, NULL);
+
/* If the slurm.key file exists, assume we're using auth/slurm */
sack_jwks = get_extra_conf_path("slurm.jwks");
sack_key = get_extra_conf_path("slurm.key");
@@ -331,7 +371,8 @@ static int _print_controllers(void *x, void *arg)
return SLURM_SUCCESS;
}
-static void _init_minimal_conf_server_config(List controllers)
+static void _init_minimal_conf_server_config(List controllers, bool use_v6,
+ bool reinit)
{
char *conf = NULL, *filename = NULL;
int fd;
@@ -343,11 +384,17 @@ static void _init_minimal_conf_server_config(List controllers)
if (slurm_conf.authinfo)
xstrfmtcat(conf, "AuthInfo=%s\n", slurm_conf.authinfo);
+ if (use_v6)
+ xstrcat(conf, "CommunicationParameters=EnableIPv6");
+
if ((fd = dump_to_memfd("slurm.conf", conf, &filename)) < 0)
fatal("%s: could not write temporary config", __func__);
xfree(conf);
- slurm_init(filename);
+ if (reinit)
+ slurm_conf_reinit(filename);
+ else
+ slurm_init(filename);
close(fd);
xfree(filename);
diff --git a/src/common/port_mgr.c b/src/common/port_mgr.c
index c78abc3956b..dc89241a7c9 100644
--- a/src/common/port_mgr.c
+++ b/src/common/port_mgr.c
@@ -342,6 +342,9 @@ static int _resv_port_alloc(uint16_t resv_port_cnt,
static int last_port_alloc = 0;
static int dims = -1;
+ xassert(!*resv_ports);
+ xassert(!*resv_port_array);
+
if (dims == -1)
dims = slurmdb_setup_cluster_dims();
@@ -389,6 +392,28 @@ extern int resv_port_step_alloc(step_record_t *step_ptr)
int rc;
int port_inx;
+ if (step_ptr->resv_port_array || step_ptr->resv_ports) {
+ /*
+ * Both resv_ports and resv_port_array need to be NULL.
+ * If they are not that could lead to resv_ports never being
+ * freed on nodes, eventually making those nodes unable to
+ * schedule jobs since their ports could have been allocated
+ * without being freed. By setting resv_ports and
+ * resv_port_array to NULL in job_array_split() guarantees that,
+ * but try to catch this issue if it happens in future.
+ */
+ error("%pS allocated reserved ports while it already had reserved ports %s",
+ step_ptr, step_ptr->resv_ports);
+
+ /*
+ * We can't just call _resv_port_free() because it is not
+ * guaranteed that the node_bitmap or resv_port_cnt is the same
+ * from when resv_port_array was allocated.
+ */
+ xfree(step_ptr->resv_port_array);
+ xfree(step_ptr->resv_ports);
+ }
+
rc = _resv_port_alloc(step_ptr->resv_port_cnt,
step_ptr->step_node_bitmap, &step_ptr->resv_ports,
&step_ptr->resv_port_array, &port_inx);
@@ -408,6 +433,29 @@ extern int resv_port_job_alloc(job_record_t *job_ptr)
int rc;
int port_inx;
+ if (job_ptr->resv_port_array || job_ptr->resv_ports) {
+ /*
+ * Both resv_ports and resv_port_array need to be NULL.
+ * If they are not that could lead to resv_ports never being
+ * freed on nodes, eventually making those nodes unable to
+ * schedule jobs since their ports could have been allocated
+ * without being freed. By setting resv_ports and
+ * resv_port_array to NULL in job_array_split() guarantees that,
+ * but try to catch this issue if it happens in future.
+ */
+ error("%pJ allocated reserved ports while it already had reserved ports %s. Ports may be lost, which will require a restart of the slurmctld daemon to resolve.",
+ job_ptr, job_ptr->resv_ports);
+
+ /*
+ * We can't just call _resv_port_free() because it is not
+ * guaranteed that the node_bitmap or resv_port_cnt is the same
+ * from when resv_port_array was allocated. A restart of the
+ * controller will restore any lost ports.
+ */
+ xfree(job_ptr->resv_port_array);
+ xfree(job_ptr->resv_ports);
+ }
+
rc = _resv_port_alloc(job_ptr->resv_port_cnt,
job_ptr->node_bitmap, &job_ptr->resv_ports,
&job_ptr->resv_port_array, &port_inx);
diff --git a/src/common/slurm_protocol_defs.h b/src/common/slurm_protocol_defs.h
index 53f5328a461..ecf4b7f5786 100644
--- a/src/common/slurm_protocol_defs.h
+++ b/src/common/slurm_protocol_defs.h
@@ -798,6 +798,8 @@ typedef struct kill_job_msg {
slurm_step_id_t step_id;
time_t time; /* slurmctld's time of request */
char *work_dir;
+ uint32_t nnodes; /* Number of nodes allocated to the job */
+ uint16_t *job_node_cpus; /* Number of CPUs required on the nodes for the job */
} kill_job_msg_t;
typedef struct reattach_tasks_request_msg {
@@ -831,6 +833,7 @@ typedef struct prolog_launch_msg {
char **spank_job_env; /* SPANK job environment variables */
uint32_t spank_job_env_size; /* size of spank_job_env */
uint32_t uid;
+ uint16_t *job_node_cpus; /* Number of CPUs required on the nodes for the job */
char *user_name_deprecated; /* remove two versions after 23.11 */
char *work_dir; /* full pathname of working directory */
uint16_t x11; /* X11 forwarding setup flags */
diff --git a/src/common/slurm_protocol_pack.c b/src/common/slurm_protocol_pack.c
index 9146c6f29d6..7e02604ea29 100644
--- a/src/common/slurm_protocol_pack.c
+++ b/src/common/slurm_protocol_pack.c
@@ -2531,6 +2531,8 @@ _pack_kill_job_msg(kill_job_msg_t * msg, buf_t *buffer, uint16_t protocol_versio
packstr(msg->nodes, buffer);
packstr_array(msg->spank_job_env, msg->spank_job_env_size,
buffer);
+ pack32(msg->nnodes, buffer);
+ pack16_array(msg->job_node_cpus, msg->nnodes, buffer);
pack_time(msg->start_time, buffer);
pack_time(msg->time, buffer);
packstr(msg->work_dir, buffer);
@@ -2573,6 +2575,10 @@ _unpack_kill_job_msg(kill_job_msg_t ** msg, buf_t *buffer,
safe_unpackstr(&tmp_ptr->nodes, buffer);
safe_unpackstr_array(&tmp_ptr->spank_job_env,
&tmp_ptr->spank_job_env_size, buffer);
+ safe_unpack32(&tmp_ptr->nnodes, buffer);
+ safe_unpack16_array(&tmp_ptr->job_node_cpus,
+ &tmp_ptr->nnodes,
+ buffer);
safe_unpack_time(&tmp_ptr->start_time, buffer);
safe_unpack_time(&tmp_ptr->time, buffer);
safe_unpackstr(&tmp_ptr->work_dir, buffer);
@@ -8245,6 +8251,8 @@ static void _pack_prolog_launch_msg(const slurm_msg_t *smsg, buf_t *buffer)
packstr_array(msg->spank_job_env, msg->spank_job_env_size,
buffer);
+ xassert(msg->nnodes > 0);
+ pack16_array(msg->job_node_cpus, msg->nnodes, buffer);
slurm_cred_pack(msg->cred, buffer, smsg->protocol_version);
if (msg->job_ptr_buf) {
@@ -8278,6 +8286,8 @@ static void _pack_prolog_launch_msg(const slurm_msg_t *smsg, buf_t *buffer)
packstr_array(msg->spank_job_env, msg->spank_job_env_size,
buffer);
+ xassert(msg->nnodes > 0);
+ pack16_array(msg->job_node_cpus, msg->nnodes, buffer);
slurm_cred_pack(msg->cred, buffer, smsg->protocol_version);
} else if (smsg->protocol_version >= SLURM_MIN_PROTOCOL_VERSION) {
gres_prep_pack(msg->job_gres_prep, buffer,
@@ -8302,6 +8312,8 @@ static void _pack_prolog_launch_msg(const slurm_msg_t *smsg, buf_t *buffer)
packstr_array(msg->spank_job_env, msg->spank_job_env_size,
buffer);
+ xassert(msg->nnodes > 0);
+ pack16_array(msg->job_node_cpus, msg->nnodes, buffer);
slurm_cred_pack(msg->cred, buffer, smsg->protocol_version);
packstr(msg->user_name_deprecated, buffer);
}
@@ -8336,6 +8348,9 @@ static int _unpack_prolog_launch_msg(slurm_msg_t *smsg, buf_t *buffer)
safe_unpackstr_array(&msg->spank_job_env,
&msg->spank_job_env_size,
buffer);
+ safe_unpack16_array(&msg->job_node_cpus,
+ &msg->nnodes,
+ buffer);
if (!(msg->cred = slurm_cred_unpack(buffer,
smsg->protocol_version)))
goto unpack_error;
@@ -8377,6 +8392,9 @@ static int _unpack_prolog_launch_msg(slurm_msg_t *smsg, buf_t *buffer)
safe_unpackstr_array(&msg->spank_job_env,
&msg->spank_job_env_size,
buffer);
+ safe_unpack16_array(&msg->job_node_cpus,
+ &msg->nnodes,
+ buffer);
if (!(msg->cred = slurm_cred_unpack(buffer,
smsg->protocol_version)))
goto unpack_error;
@@ -8405,6 +8423,9 @@ static int _unpack_prolog_launch_msg(slurm_msg_t *smsg, buf_t *buffer)
safe_unpackstr_array(&msg->spank_job_env,
&msg->spank_job_env_size,
buffer);
+ safe_unpack16_array(&msg->job_node_cpus,
+ &msg->nnodes,
+ buffer);
if (!(msg->cred = slurm_cred_unpack(buffer,
smsg->protocol_version)))
goto unpack_error;
diff --git a/src/common/slurm_resolv.h b/src/common/slurm_resolv.h
index e0fe650daf6..63de8c74ba7 100644
--- a/src/common/slurm_resolv.h
+++ b/src/common/slurm_resolv.h
@@ -40,6 +40,8 @@ typedef struct {
uint16_t priority;
uint16_t port;
char hostname[1024];
+ bool has_ipv4;
+ bool has_ipv6;
} ctl_entry_t;
/*
diff --git a/src/common/util-net.c b/src/common/util-net.c
index a0fb281e1cb..5720e4b0d1f 100644
--- a/src/common/util-net.c
+++ b/src/common/util-net.c
@@ -253,6 +253,26 @@ extern char *make_full_path(const char *rpath)
return cwd2;
}
+static struct addrinfo *_xgetaddrinfo(const char *hostname, const char *serv,
+ const struct addrinfo *hints)
+{
+ struct addrinfo *result = NULL;
+ int err;
+
+ err = getaddrinfo(hostname, serv, hints, &result);
+ if (err == EAI_SYSTEM) {
+ error_in_daemon("%s: getaddrinfo(%s:%s) failed: %s: %m",
+ __func__, hostname, serv, gai_strerror(err));
+ return NULL;
+ } else if (err != 0) {
+ error_in_daemon("%s: getaddrinfo(%s:%s) failed: %s",
+ __func__, hostname, serv, gai_strerror(err));
+ return NULL;
+ }
+
+ return result;
+}
+
extern struct addrinfo *xgetaddrinfo_port(const char *hostname, uint16_t port)
{
char serv[6];
@@ -262,9 +282,7 @@ extern struct addrinfo *xgetaddrinfo_port(const char *hostname, uint16_t port)
extern struct addrinfo *xgetaddrinfo(const char *hostname, const char *serv)
{
- struct addrinfo *result = NULL;
struct addrinfo hints;
- int err;
bool v4_enabled = slurm_conf.conf_flags & CONF_FLAG_IPV4_ENABLED;
bool v6_enabled = slurm_conf.conf_flags & CONF_FLAG_IPV6_ENABLED;
@@ -300,18 +318,39 @@ extern struct addrinfo *xgetaddrinfo(const char *hostname, const char *serv)
hints.ai_flags |= AI_CANONNAME;
hints.ai_socktype = SOCK_STREAM;
- err = getaddrinfo(hostname, serv, &hints, &result);
- if (err == EAI_SYSTEM) {
- error_in_daemon("%s: getaddrinfo(%s:%s) failed: %s: %m",
- __func__, hostname, serv, gai_strerror(err));
- return NULL;
- } else if (err != 0) {
- error_in_daemon("%s: getaddrinfo(%s:%s) failed: %s",
- __func__, hostname, serv, gai_strerror(err));
- return NULL;
+ return _xgetaddrinfo(hostname, serv, &hints);
+}
+
+extern int host_has_addr_family(const char *hostname, const char *srv,
+ bool *ipv4, bool *ipv6)
+{
+ struct addrinfo hints;
+ struct addrinfo *ai_ptr, *ai_start;
+
+ memset(&hints, 0, sizeof(hints));
+
+ hints.ai_family = AF_UNSPEC;
+ hints.ai_flags = AI_ADDRCONFIG | AI_NUMERICSERV | AI_PASSIVE;
+ if (hostname)
+ hints.ai_flags |= AI_CANONNAME;
+ hints.ai_socktype = SOCK_STREAM;
+
+ ai_start = _xgetaddrinfo(hostname, srv, &hints);
+
+ if (!ai_start)
+ return SLURM_ERROR;
+
+ *ipv4 = *ipv6 = false;
+ for (ai_ptr = ai_start; ai_ptr; ai_ptr = ai_ptr->ai_next) {
+ if (ai_ptr->ai_family == AF_INET6)
+ *ipv6 = true;
+ else if (ai_ptr->ai_family == AF_INET)
+ *ipv4 = true;
}
- return result;
+ freeaddrinfo(ai_start);
+
+ return SLURM_SUCCESS;
}
static int _name_cache_find(void *x, void *y)
diff --git a/src/common/util-net.h b/src/common/util-net.h
index 7d8c7f9ccfb..c85b9b562d1 100644
--- a/src/common/util-net.h
+++ b/src/common/util-net.h
@@ -87,6 +87,8 @@ extern struct addrinfo *xgetaddrinfo_port(const char *hostname,
uint16_t port);
extern char *xgetnameinfo(struct sockaddr *addr, socklen_t addrlen);
+extern int host_has_addr_family(const char *hostname, const char *srv,
+ bool *ipv4, bool *ipv6);
/* Functions responsible for cleanup of getnameinfo cache */
extern void getnameinfo_cache_destroy(void *obj);
extern void getnameinfo_cache_purge(void);
diff --git a/src/plugins/data_parser/v0.0.40/parsers.c b/src/plugins/data_parser/v0.0.40/parsers.c
index 54f1ef33636..fe0437de58b 100644
--- a/src/plugins/data_parser/v0.0.40/parsers.c
+++ b/src/plugins/data_parser/v0.0.40/parsers.c
@@ -8230,7 +8230,7 @@ static const parser_t PARSER_ARRAY(OPENAPI_WARNING)[] = {
static const parser_t PARSER_ARRAY(INSTANCE_CONDITION)[] = {
add_parse(CSV_STRING_LIST, cluster_list, "cluster", "CSV clusters list"),
add_parse(CSV_STRING_LIST, extra_list, "extra", "CSV extra list"),
- add_parse(CSV_STRING_LIST, format_list, "format", "CSV format list"),
+ add_parse(CSV_STRING_LIST, format_list, "format", "Ignored; process JSON manually to control output format"),
add_parse(CSV_STRING_LIST, instance_id_list, "instance_id", "CSV instance_id list"),
add_parse(CSV_STRING_LIST, instance_type_list, "instance_type", "CSV instance_type list"),
add_parse(STRING, node_list, "node_list", "Ranged node string"),
@@ -8305,7 +8305,7 @@ static const parser_t PARSER_ARRAY(JOB_CONDITION)[] = {
add_flags(JOB_CONDITION_DB_FLAGS, db_flags),
add_parse(INT32, exitcode, "exit_code", "Job exit code (numeric)"),
add_flags(JOB_CONDITION_FLAGS, flags),
- add_parse(CSV_STRING_LIST, format_list, "format", "CSV format list"),
+ add_parse(CSV_STRING_LIST, format_list, "format", "Ignored; process JSON manually to control output format"),
add_parse(GROUP_ID_STRING_LIST, groupid_list, "groups", "CSV group list"),
add_parse(CSV_STRING_LIST, jobname_list, "job_name", "CSV job name list"),
add_parse(UINT32_NO_VAL, nodes_max, "nodes_max", "Maximum number of nodes"),
@@ -8335,7 +8335,7 @@ static const parser_t PARSER_ARRAY(JOB_CONDITION)[] = {
static const parser_t PARSER_ARRAY(QOS_CONDITION)[] = {
add_parse(CSV_STRING_LIST, description_list, "description", "CSV description list"),
add_parse(QOS_ID_STRING_CSV_LIST, id_list, "id", "CSV QOS id list"),
- add_parse(CSV_STRING_LIST, format_list, "format", "CSV format list"),
+ add_parse(CSV_STRING_LIST, format_list, "format", "Ignored; process JSON manually to control output format"),
add_parse(QOS_NAME_CSV_LIST, name_list, "name", "CSV QOS name list"),
add_parse_bit_flag_array(slurmdb_qos_cond_t, QOS_PREEMPT_MODES, false, preempt_mode, "preempt_mode", "PreemptMode used when jobs in this QOS are preempted"),
add_parse(BOOL16, with_deleted, "with_deleted", "Include deleted QOS"),
@@ -8394,7 +8394,7 @@ static const parser_t PARSER_ARRAY(ASSOC_CONDITION)[] = {
add_parse(CSV_STRING_LIST, acct_list, "account", "CSV accounts list"),
add_parse(CSV_STRING_LIST, cluster_list, "cluster", "CSV clusters list"),
add_parse(QOS_ID_STRING_CSV_LIST, def_qos_id_list, "default_qos", "CSV QOS list"),
- add_parse(CSV_STRING_LIST, format_list, "format", "CSV format list"),
+ add_parse(CSV_STRING_LIST, format_list, "format", "Ignored; process JSON manually to control output format"),
add_parse(ASSOC_ID_STRING_CSV_LIST, id_list, "id", "CSV id list"),
add_parse(BOOL16, only_defs, "only_defaults", "Filter to only defaults"),
add_parse(CSV_STRING_LIST, parent_acct_list, "parent_account", "CSV names of parent account"),
@@ -8462,7 +8462,7 @@ static const parser_t PARSER_ARRAY(OPENAPI_WCKEY_PARAM)[] = {
add_parser(slurmdb_wckey_cond_t, mtype, false, field, 0, path, desc)
static const parser_t PARSER_ARRAY(WCKEY_CONDITION)[] = {
add_parse(CSV_STRING_LIST, cluster_list, "cluster", "CSV cluster name list"),
- add_parse(CSV_STRING_LIST, format_list, "format", "CSV format name list"),
+ add_parse(CSV_STRING_LIST, format_list, "format", "Ignored; process JSON manually to control output format"),
add_parse(CSV_STRING_LIST, id_list, "id", "CSV id list"),
add_parse(CSV_STRING_LIST, name_list, "name", "CSV name list"),
add_parse(BOOL16, only_defs, "only_defaults", "Only query defaults"),
@@ -8525,7 +8525,7 @@ static const parser_t PARSER_ARRAY(CLUSTER_CONDITION)[] = {
add_parse(STRING_LIST, cluster_list, "cluster", "CSV cluster list"),
add_parse(STRING_LIST, federation_list, "federation", "CSV federation list"),
add_parse_bit_flag_array(slurmdb_cluster_cond_t, CLUSTER_REC_FLAGS, false, flags, "flags", "Query flags"),
- add_parse(STRING_LIST, format_list, "format", "CSV format list"),
+ add_parse(STRING_LIST, format_list, "format", "Ignored; process JSON manually to control output format"),
add_parse(STRING_LIST, rpc_version_list, "rpc_version", "CSV RPC version list"),
add_parse(TIMESTAMP, usage_end, "usage_end", "Usage end (UNIX timestamp)"),
add_parse(TIMESTAMP, usage_start, "usage_start", "Usage start (UNIX timestamp)"),
diff --git a/src/plugins/data_parser/v0.0.41/api.c b/src/plugins/data_parser/v0.0.41/api.c
index 5b213dcd49f..7e665ecb43a 100644
--- a/src/plugins/data_parser/v0.0.41/api.c
+++ b/src/plugins/data_parser/v0.0.41/api.c
@@ -95,7 +95,7 @@ extern int data_parser_p_dump(args_t *args, data_parser_type_t type, void *src,
return ESLURM_NOT_SUPPORTED;
}
- return dump(src, src_bytes, parser, dst, args);
+ return dump(src, src_bytes, NULL, parser, dst, args);
}
extern int data_parser_p_parse(args_t *args, data_parser_type_t type, void *dst,
diff --git a/src/plugins/data_parser/v0.0.41/parsers.c b/src/plugins/data_parser/v0.0.41/parsers.c
index 9d37e8d7dbe..ee8edb9ba21 100644
--- a/src/plugins/data_parser/v0.0.41/parsers.c
+++ b/src/plugins/data_parser/v0.0.41/parsers.c
@@ -8423,7 +8423,7 @@ static const parser_t PARSER_ARRAY(OPENAPI_WARNING)[] = {
static const parser_t PARSER_ARRAY(INSTANCE_CONDITION)[] = {
add_parse(CSV_STRING_LIST, cluster_list, "cluster", "CSV clusters list"),
add_parse(CSV_STRING_LIST, extra_list, "extra", "CSV extra list"),
- add_parse(CSV_STRING_LIST, format_list, "format", "CSV format list"),
+ add_parse(CSV_STRING_LIST, format_list, "format", "Ignored; process JSON manually to control output format"),
add_parse(CSV_STRING_LIST, instance_id_list, "instance_id", "CSV instance_id list"),
add_parse(CSV_STRING_LIST, instance_type_list, "instance_type", "CSV instance_type list"),
add_parse(STRING, node_list, "node_list", "Ranged node string"),
@@ -8501,7 +8501,7 @@ static const parser_t PARSER_ARRAY(JOB_CONDITION)[] = {
add_flags(JOB_CONDITION_DB_FLAGS, db_flags),
add_parse(INT32, exitcode, "exit_code", "Job exit code (numeric)"),
add_flags(JOB_CONDITION_FLAGS, flags),
- add_parse(CSV_STRING_LIST, format_list, "format", "CSV format list"),
+ add_parse(CSV_STRING_LIST, format_list, "format", "Ignored; process JSON manually to control output format"),
add_parse(GROUP_ID_STRING_LIST, groupid_list, "groups", "CSV group list"),
add_parse(CSV_STRING_LIST, jobname_list, "job_name", "CSV job name list"),
add_parse(UINT32_NO_VAL, nodes_max, "nodes_max", "Maximum number of nodes"),
@@ -8531,7 +8531,7 @@ static const parser_t PARSER_ARRAY(JOB_CONDITION)[] = {
static const parser_t PARSER_ARRAY(QOS_CONDITION)[] = {
add_parse(CSV_STRING_LIST, description_list, "description", "CSV description list"),
add_parse(QOS_ID_STRING_CSV_LIST, id_list, "id", "CSV QOS id list"),
- add_parse(CSV_STRING_LIST, format_list, "format", "CSV format list"),
+ add_parse(CSV_STRING_LIST, format_list, "format", "Ignored; process JSON manually to control output format"),
add_parse(QOS_NAME_CSV_LIST, name_list, "name", "CSV QOS name list"),
add_parse_bit_flag_array(slurmdb_qos_cond_t, QOS_PREEMPT_MODES, false, preempt_mode, "preempt_mode", "PreemptMode used when jobs in this QOS are preempted"),
add_parse(BOOL16, with_deleted, "with_deleted", "Include deleted QOS"),
@@ -8590,7 +8590,7 @@ static const parser_t PARSER_ARRAY(ASSOC_CONDITION)[] = {
add_parse(CSV_STRING_LIST, acct_list, "account", "CSV accounts list"),
add_parse(CSV_STRING_LIST, cluster_list, "cluster", "CSV clusters list"),
add_parse(QOS_ID_STRING_CSV_LIST, def_qos_id_list, "default_qos", "CSV QOS list"),
- add_parse(CSV_STRING_LIST, format_list, "format", "CSV format list"),
+ add_parse(CSV_STRING_LIST, format_list, "format", "Ignored; process JSON manually to control output format"),
add_parse(ASSOC_ID_STRING_CSV_LIST, id_list, "id", "CSV id list"),
add_parse(BOOL16, only_defs, "only_defaults", "Filter to only defaults"),
add_parse(CSV_STRING_LIST, parent_acct_list, "parent_account", "CSV names of parent account"),
@@ -8658,7 +8658,7 @@ static const parser_t PARSER_ARRAY(OPENAPI_WCKEY_PARAM)[] = {
add_parser(slurmdb_wckey_cond_t, mtype, false, field, 0, path, desc)
static const parser_t PARSER_ARRAY(WCKEY_CONDITION)[] = {
add_parse(CSV_STRING_LIST, cluster_list, "cluster", "CSV cluster name list"),
- add_parse(CSV_STRING_LIST, format_list, "format", "CSV format name list"),
+ add_parse(CSV_STRING_LIST, format_list, "format", "Ignored; process JSON manually to control output format"),
add_parse(CSV_STRING_LIST, id_list, "id", "CSV id list"),
add_parse(CSV_STRING_LIST, name_list, "name", "CSV name list"),
add_parse(BOOL16, only_defs, "only_defaults", "Only query defaults"),
@@ -8716,7 +8716,7 @@ static const parser_t PARSER_ARRAY(CLUSTER_CONDITION)[] = {
add_parse(STRING_LIST, cluster_list, "cluster", "CSV cluster list"),
add_parse(STRING_LIST, federation_list, "federation", "CSV federation list"),
add_parse_bit_flag_array(slurmdb_cluster_cond_t, CLUSTER_REC_FLAGS, false, flags, "flags", "Query flags"),
- add_parse(STRING_LIST, format_list, "format", "CSV format list"),
+ add_parse(STRING_LIST, format_list, "format", "Ignored; process JSON manually to control output format"),
add_parse(STRING_LIST, rpc_version_list, "rpc_version", "CSV RPC version list"),
add_parse(TIMESTAMP, usage_end, "usage_end", "Usage end (UNIX timestamp)"),
add_parse(TIMESTAMP, usage_start, "usage_start", "Usage start (UNIX timestamp)"),
diff --git a/src/plugins/data_parser/v0.0.41/parsing.c b/src/plugins/data_parser/v0.0.41/parsing.c
index c6db4704e18..ff5bae5b924 100644
--- a/src/plugins/data_parser/v0.0.41/parsing.c
+++ b/src/plugins/data_parser/v0.0.41/parsing.c
@@ -1244,8 +1244,9 @@ static int _foreach_dump_list(void *obj, void *arg)
xassert(args->parser->ptr_offset == NO_VAL);
/* we don't know the size of the items in the list */
- if (dump(&obj, NO_VAL, find_parser_by_type(args->parser->list_type),
- item, args->args))
+ if (dump(&obj, NO_VAL, NULL,
+ find_parser_by_type(args->parser->list_type), item,
+ args->args))
return -1;
return 0;
@@ -1286,7 +1287,8 @@ static int _dump_list(const parser_t *const parser, void *src, data_t *dst,
return SLURM_SUCCESS;
}
-static int _dump_pointer(const parser_t *const parser, void *src, data_t *dst,
+static int _dump_pointer(const parser_t *const field_parser,
+ const parser_t *const parser, void *src, data_t *dst,
args_t *args)
{
const parser_t *pt = find_parser_by_type(parser->pointer_type);
@@ -1302,10 +1304,11 @@ static int _dump_pointer(const parser_t *const parser, void *src, data_t *dst,
while (pt->pointer_type)
pt = find_parser_by_type(pt->pointer_type);
- if (parser->allow_null_pointer) {
+ if (parser->allow_null_pointer ||
+ (field_parser && !field_parser->required)) {
xassert(data_get_type(dst) == DATA_TYPE_NULL);
} else if ((pt->model == PARSER_MODEL_ARRAY) ||
- (pt->obj_openapi == OPENAPI_FORMAT_OBJECT)) {
+ (pt->obj_openapi == OPENAPI_FORMAT_OBJECT)) {
/*
* OpenAPI clients can't handle a null instead of an
* object. Work around by placing an empty dictionary
@@ -1327,7 +1330,7 @@ static int _dump_pointer(const parser_t *const parser, void *src, data_t *dst,
return SLURM_SUCCESS;
}
- return dump(*ptr, NO_VAL, pt, dst, args);
+ return dump(*ptr, NO_VAL, NULL, pt, dst, args);
}
static int _dump_nt_array(const parser_t *const parser, void *src, data_t *dst,
@@ -1345,7 +1348,7 @@ static int _dump_nt_array(const parser_t *const parser, void *src, data_t *dst,
return SLURM_SUCCESS;
for (int i = 0; !rc && array[i]; i++) {
- rc = dump(array[i], NO_VAL,
+ rc = dump(array[i], NO_VAL, NULL,
find_parser_by_type(parser->array_type),
data_list_append(dst), args);
}
@@ -1369,7 +1372,7 @@ static int _dump_nt_array(const parser_t *const parser, void *src, data_t *dst,
if (done)
break;
- rc = dump(ptr, NO_VAL,
+ rc = dump(ptr, NO_VAL, NULL,
find_parser_by_type(parser->array_type),
data_list_append(dst), args);
}
@@ -1518,7 +1521,8 @@ static int _dump_linked(args_t *args, const parser_t *const array,
array->ptr_offset, (uintptr_t) dst, array->key,
(uintptr_t) dst);
- rc = dump(src, NO_VAL, find_parser_by_type(parser->type), dst, args);
+ rc = dump(src, NO_VAL, parser, find_parser_by_type(parser->type), dst,
+ args);
log_flag(DATA, "END: dumping %s parser %s->%s(0x%" PRIxPTR ") for %s(0x%" PRIxPTR ")->%s(+%zd) for data(0x%" PRIxPTR ")/%s(0x%" PRIxPTR ")",
parser->obj_type_string, array->type_string,
@@ -1545,8 +1549,9 @@ static void _check_dump(const parser_t *const parser, data_t *dst, args_t *args)
}
}
-extern int dump(void *src, ssize_t src_bytes, const parser_t *const parser,
- data_t *dst, args_t *args)
+extern int dump(void *src, ssize_t src_bytes,
+ const parser_t *const field_parser,
+ const parser_t *const parser, data_t *dst, args_t *args)
{
int rc;
@@ -1617,7 +1622,7 @@ extern int dump(void *src, ssize_t src_bytes, const parser_t *const parser,
verify_parser_not_sliced(parser);
xassert(data_get_type(dst) == DATA_TYPE_NULL);
- rc = _dump_pointer(parser, src, dst, args);
+ rc = _dump_pointer(field_parser, parser, src, dst, args);
break;
case PARSER_MODEL_NT_PTR_ARRAY:
case PARSER_MODEL_NT_ARRAY:
diff --git a/src/plugins/data_parser/v0.0.41/parsing.h b/src/plugins/data_parser/v0.0.41/parsing.h
index 94e325dde1a..f1b34a1554e 100644
--- a/src/plugins/data_parser/v0.0.41/parsing.h
+++ b/src/plugins/data_parser/v0.0.41/parsing.h
@@ -59,11 +59,12 @@
#undef DATA_DUMP
#undef DATA_PARSE
-extern int dump(void *src, ssize_t src_bytes, const parser_t *const parser,
- data_t *dst, args_t *args);
-#define DUMP(type, src, dst, args) \
- dump(&src, sizeof(src), find_parser_by_type(DATA_PARSER_##type), dst, \
- args)
+extern int dump(void *src, ssize_t src_bytes,
+ const parser_t *const field_parser,
+ const parser_t *const parser, data_t *dst, args_t *args);
+#define DUMP(type, src, dst, args) \
+ dump(&src, sizeof(src), NULL, find_parser_by_type(DATA_PARSER_##type), \
+ dst, args)
extern int parse(void *dst, ssize_t dst_bytes, const parser_t *const parser,
data_t *src, args_t *args, data_t *parent_path);
diff --git a/src/plugins/job_submit/pbs/spank_pbs.c b/src/plugins/job_submit/pbs/spank_pbs.c
index 3559c90230e..943d253ad9c 100644
--- a/src/plugins/job_submit/pbs/spank_pbs.c
+++ b/src/plugins/job_submit/pbs/spank_pbs.c
@@ -65,14 +65,28 @@ int slurm_spank_task_init(spank_t sp, int ac, char **av)
spank_setenv(sp, "PBS_ARRAY_ID", val, 1);
if (spank_getenv(sp, "SLURM_ARRAY_TASK_ID", val, sizeof(val)) ==
ESPANK_SUCCESS)
- spank_setenv(sp, "PBS_ARRAY_INDEX", val, 1);
+ spank_setenv(sp, "PBS_ARRAYID", val, 1);
if (getcwd(val, sizeof(val)))
spank_setenv(sp, "PBS_JOBDIR", val, 1);
- if (spank_getenv(sp, "SLURM_JOB_ID", val, sizeof(val)) ==
- ESPANK_SUCCESS)
- spank_setenv(sp, "PBS_JOBID", val, 1);
+ if (spank_getenv(sp, "SLURM_ARRAY_JOB_ID", val, sizeof(val)) ==
+ ESPANK_SUCCESS) {
+ /* 20 is enough for unit32 (max index is 4M) */
+ char aid[20];
+ if (spank_getenv(sp, "SLURM_ARRAY_TASK_ID", aid, sizeof(aid)) ==
+ ESPANK_SUCCESS) {
+ /* 30k val is large enough for job id and array index */
+ strncat(val, "[", 1);
+ strncat(val, aid, sizeof(aid));
+ strncat(val, "]", 1);
+ }
+ spank_setenv(sp, "PBS_JOBID", val, 1);
+ } else {
+ if (spank_getenv(sp, "SLURM_JOB_ID", val, sizeof(val)) ==
+ ESPANK_SUCCESS)
+ spank_setenv(sp, "PBS_JOBID", val, 1);
+ }
if (spank_getenv(sp, "SLURM_JOB_NAME", val, sizeof(val)) ==
ESPANK_SUCCESS)
@@ -112,6 +126,15 @@ int slurm_spank_task_init(spank_t sp, int ac, char **av)
if (spank_getenv(sp, "SYSTEM", val, sizeof(val)) == ESPANK_SUCCESS)
spank_setenv(sp, "PBS_O_SYSTEM", val, 1);
+ if (spank_getenv(sp, "SLURM_JOB_NUM_NODES", val, sizeof(val)) == ESPANK_SUCCESS)
+ spank_setenv(sp, "PBS_NUM_NODES", val, 1);
+
+ if (spank_getenv(sp, "SLURM_NTASKS", val, sizeof(val)) == ESPANK_SUCCESS)
+ spank_setenv(sp, "PBS_NP", val, 1);
+
+ if (spank_getenv(sp, "SLURM_NTASKS_PER_NODE", val, sizeof(val)) == ESPANK_SUCCESS)
+ spank_setenv(sp, "PBS_NUM_PPN", val, 1);
+
if (spank_getenv(sp, "SLURM_SUBMIT_DIR", val, sizeof(val)) ==
ESPANK_SUCCESS)
spank_setenv(sp, "PBS_O_WORKDIR", val, 1);
diff --git a/src/plugins/prep/script/prep_script_slurmd.c b/src/plugins/prep/script/prep_script_slurmd.c
index 21ea69c4c32..559892426d6 100644
--- a/src/plugins/prep/script/prep_script_slurmd.c
+++ b/src/plugins/prep/script/prep_script_slurmd.c
@@ -272,6 +272,9 @@ static char **_build_env(job_env_t *job_env, slurm_cred_t *cred,
else
setenvf(&env, "SLURM_SCRIPT_CONTEXT", "prolog_slurmd");
+ setenvf(&env, "SLURM_JOB_NODE_CPUS", "%d", job_env->job_node_cpus);
+ setenvf(&env, "SLURM_ACTUAL_NODE_CPUS", "%d", conf->actual_cpus);
+
if (is_epilog && (job_env->exit_code != INFINITE)) {
int exit_code = 0, signal = 0;
if (WIFEXITED(job_env->exit_code))
diff --git a/src/sattach/sattach.c b/src/sattach/sattach.c
index 3c8f7b10e5d..921b6c7bcdd 100644
--- a/src/sattach/sattach.c
+++ b/src/sattach/sattach.c
@@ -408,7 +408,8 @@ static int _attach_to_tasks(slurm_step_id_t stepid,
slurm_msg_set_r_uid(&msg, SLURM_AUTH_UID_ANY);
msg.msg_type = REQUEST_REATTACH_TASKS;
msg.data = &reattach_msg;
- msg.protocol_version = layout->start_protocol_ver;
+ msg.protocol_version = MIN(SLURM_PROTOCOL_VERSION,
+ layout->start_protocol_ver);
if (layout->front_end)
hosts = layout->front_end;
diff --git a/src/slurmctld/controller.c b/src/slurmctld/controller.c
index 42185e4d99a..ec4f4e8272c 100644
--- a/src/slurmctld/controller.c
+++ b/src/slurmctld/controller.c
@@ -817,6 +817,10 @@ int main(int argc, char **argv)
if (slurmctld_config.resume_backup && slurmctld_primary)
break;
+ /* The backup is now meant to relinquish control */
+ if (slurmctld_config.resume_backup && !slurmctld_primary)
+ backup_has_control = false;
+
recover = 2;
}
diff --git a/src/slurmctld/job_mgr.c b/src/slurmctld/job_mgr.c
index d0c02a94447..e99e2d58f21 100644
--- a/src/slurmctld/job_mgr.c
+++ b/src/slurmctld/job_mgr.c
@@ -3477,6 +3477,8 @@ extern job_record_t *job_array_split(job_record_t *job_ptr)
job_ptr_pend->resv_name = xstrdup(job_ptr->resv_name);
if (job_ptr->resv_list)
job_ptr_pend->resv_list = list_shallow_copy(job_ptr->resv_list);
+ job_ptr_pend->resv_ports = NULL;
+ job_ptr_pend->resv_port_array = NULL;
job_ptr_pend->resp_host = xstrdup(job_ptr->resp_host);
if (job_ptr->select_jobinfo) {
job_ptr_pend->select_jobinfo =
@@ -15674,6 +15676,10 @@ void batch_requeue_fini(job_record_t *job_ptr)
}
}
+ /* Reset the priority (begin and accrue times were reset) */
+ if (job_ptr->priority != 0)
+ set_job_prio(job_ptr);
+
/*
* If a reservation ended and was a repeated (e.g., daily, weekly)
* reservation, its ID will be different; make sure
diff --git a/src/slurmctld/node_scheduler.c b/src/slurmctld/node_scheduler.c
index 1f1794f1c1f..672b9cf590d 100644
--- a/src/slurmctld/node_scheduler.c
+++ b/src/slurmctld/node_scheduler.c
@@ -323,6 +323,8 @@ extern void deallocate_nodes(job_record_t *job_ptr, bool timeout,
hostlist_t *hostlist = NULL;
uint16_t use_protocol_version = 0;
uint16_t msg_flags = 0;
+ job_resources_t *job_resrcs_ptr = NULL;
+
#ifdef HAVE_FRONT_END
front_end_record_t *front_end_ptr;
#endif
@@ -347,6 +349,7 @@ extern void deallocate_nodes(job_record_t *job_ptr, bool timeout,
if (!job_ptr->details->prolog_running)
hostlist = hostlist_create(NULL);
+
#ifdef HAVE_FRONT_END
if (job_ptr->batch_host &&
(front_end_ptr = job_ptr->front_end_ptr)) {
@@ -495,9 +498,14 @@ extern void deallocate_nodes(job_record_t *job_ptr, bool timeout,
agent_args->msg_flags = msg_flags;
last_node_update = time(NULL);
- kill_job = create_kill_job_msg(job_ptr, use_protocol_version);
+ kill_job = create_kill_job_msg(job_ptr, use_protocol_version);
kill_job->nodes = xstrdup(job_ptr->nodes);
+ // XXX: verify
+ job_resrcs_ptr = job_ptr->job_resrcs;
+ kill_job->nnodes = job_resrcs_ptr->nhosts;
+ kill_job->job_node_cpus = job_resrcs_ptr->cpus;
+
agent_args->msg_args = kill_job;
set_agent_arg_r_uid(agent_args, SLURM_AUTH_UID_ANY);
agent_queue_request(agent_args);
@@ -2538,6 +2546,7 @@ extern int select_nodes(job_record_t *job_ptr, bool test_only,
{ .assoc = READ_LOCK, .qos = WRITE_LOCK, .tres = READ_LOCK };
List gres_list_pre = NULL;
bool gres_list_pre_set = false;
+ job_record_t *tmp_job;
xassert(job_ptr);
xassert(job_ptr->magic == JOB_MAGIC);
@@ -2942,7 +2951,10 @@ extern int select_nodes(job_record_t *job_ptr, bool test_only,
job_end_time_reset(job_ptr);
- (void) job_array_post_sched(job_ptr);
+ tmp_job = job_array_post_sched(job_ptr);
+ if (tmp_job && (tmp_job != job_ptr) && (orig_resv_port_cnt == NO_VAL16))
+ tmp_job->resv_port_cnt = orig_resv_port_cnt;
+
if (bb_g_job_begin(job_ptr) != SLURM_SUCCESS) {
/* Leave job queued, something is hosed */
error_code = ESLURM_INVALID_BURST_BUFFER_REQUEST;
@@ -3101,7 +3113,6 @@ extern int select_nodes(job_record_t *job_ptr, bool test_only,
}
if (error_code != SLURM_SUCCESS) {
- FREE_NULL_BITMAP(job_ptr->node_bitmap);
if (gres_list_pre_set &&
(job_ptr->gres_list_req != gres_list_pre)) {
FREE_NULL_LIST(job_ptr->gres_list_req);
@@ -3114,6 +3125,7 @@ extern int select_nodes(job_record_t *job_ptr, bool test_only,
resv_port_job_free(job_ptr);
xfree(job_ptr->resv_ports);
}
+ FREE_NULL_BITMAP(job_ptr->node_bitmap);
} else
FREE_NULL_LIST(gres_list_pre);
@@ -3345,6 +3357,10 @@ extern void launch_prolog(job_record_t *job_ptr)
xassert(job_ptr->job_resrcs);
job_resrcs_ptr = job_ptr->job_resrcs;
+
+ prolog_msg_ptr->nnodes = job_resrcs_ptr->nhosts;
+ prolog_msg_ptr->job_node_cpus = job_resrcs_ptr->cpus;
+
setup_cred_arg(&cred_arg, job_ptr);
cred_arg.step_id.job_id = job_ptr->job_id;
cred_arg.step_id.step_id = SLURM_EXTERN_CONT;
diff --git a/src/slurmd/slurmd/cred_context.c b/src/slurmd/slurmd/cred_context.c
index 998c24cf838..8df077ffc65 100644
--- a/src/slurmd/slurmd/cred_context.c
+++ b/src/slurmd/slurmd/cred_context.c
@@ -124,7 +124,14 @@ static job_state_t *_find_job_state(uint32_t jobid)
static void _clear_expired_job_states(void)
{
- time_t now = time(NULL);
+ time_t now;
+
+ if (!cred_job_list) {
+ warning("No cred_job_list, unable to clear expired job states");
+ return;
+ }
+
+ now = time(NULL);
list_delete_all(cred_job_list, _list_find_expired_job_state, &now);
}
@@ -140,7 +147,14 @@ static int _list_find_expired_cred_state(void *x, void *key)
static void _clear_expired_credential_states(void)
{
- time_t now = time(NULL);
+ time_t now;
+
+ if (!cred_state_list) {
+ warning("No cred_state_list, unable to clear expired credential states");
+ return;
+ }
+
+ now = time(NULL);
list_delete_all(cred_state_list, _list_find_expired_cred_state, &now);
}
diff --git a/src/slurmd/slurmd/req.c b/src/slurmd/slurmd/req.c
index a06abed1a30..9b01205998d 100644
--- a/src/slurmd/slurmd/req.c
+++ b/src/slurmd/slurmd/req.c
@@ -40,8 +40,10 @@
\*****************************************************************************/
#include "config.h"
+#include
#define _GNU_SOURCE /* for setresuid() */
+#define __USE_XOPEN_EXTENDED
#include
#include
@@ -2467,6 +2469,31 @@ static void _notify_result_rpc_prolog(prolog_launch_msg_t *req, int rc)
}
}
+static int _get_node_inx(char *hostlist)
+{
+ char *host;
+ int node_inx = -1;
+ hostset_t *hset;
+
+ if (!conf->node_name)
+ return node_inx;
+
+ if ((hset = hostset_create(hostlist))) {
+ int inx = 0;
+ while ((host = hostset_shift(hset))) {
+ if (!strcmp(host, conf->node_name)) {
+ node_inx = inx;
+ free(host);
+ break;
+ }
+ inx++;
+ free(host);
+ }
+ hostset_destroy(hset);
+ }
+ return node_inx;
+}
+
static void _rpc_prolog(slurm_msg_t *msg)
{
int rc = SLURM_SUCCESS;
@@ -2519,6 +2546,7 @@ static void _rpc_prolog(slurm_msg_t *msg)
if (!(slurm_conf.prolog_flags & PROLOG_FLAG_RUN_IN_JOB)) {
int node_id = 0;
job_env_t job_env;
+ int node_inx = 0;
#ifndef HAVE_FRONT_END
/* It is always 0 for front end systems */
@@ -2540,7 +2568,11 @@ static void _rpc_prolog(slurm_msg_t *msg)
job_env.uid = req->uid;
job_env.gid = req->gid;
- rc = run_prolog(&job_env, req->cred);
+ node_inx = _get_node_inx(req->nodes);
+ debug("_rpc_prolog: _get_node_inx returned %d", node_inx);
+ job_env.job_node_cpus = (node_inx >= 0 ? req->job_node_cpus[node_inx] : 0);
+
+ rc = run_prolog(&job_env, req->cred);
_free_job_env(&job_env);
if (rc) {
int term_sig = 0, exit_status = 0;
@@ -5538,6 +5570,7 @@ _rpc_terminate_job(slurm_msg_t *msg)
if (!(slurm_conf.prolog_flags & PROLOG_FLAG_RUN_IN_JOB)) {
int node_id = 0;
job_env_t job_env;
+ int node_inx = -1;
#ifndef HAVE_FRONT_END
/* It is always 0 for front end systems */
node_id = nodelist_find(req->nodes, conf->node_name);
@@ -5557,6 +5590,9 @@ _rpc_terminate_job(slurm_msg_t *msg)
job_env.uid = req->job_uid;
job_env.gid = req->job_gid;
+ node_inx = _get_node_inx(req->nodes);
+ job_env.job_node_cpus = (node_inx >= 0 ? req->job_node_cpus[node_inx] : 0);
+ debug2("Setting job_env.job_cpu_nodes to %d", job_env.job_node_cpus);
_wait_for_job_running_prolog(job_env.jobid);
rc = run_epilog(&job_env, req->cred);
_free_job_env(&job_env);
diff --git a/src/slurmd/slurmd/slurmd.h b/src/slurmd/slurmd/slurmd.h
index 93d20ab9a66..bcb06c06a1d 100644
--- a/src/slurmd/slurmd/slurmd.h
+++ b/src/slurmd/slurmd/slurmd.h
@@ -69,6 +69,7 @@ typedef struct {
uid_t uid;
gid_t gid;
char *work_dir;
+ uint16_t job_node_cpus; /* Number of CPUs used by the job on this node */
} job_env_t;
/*
diff --git a/src/slurmd/slurmstepd/req.c b/src/slurmd/slurmstepd/req.c
index e7da47c75dd..d558c6aa4fb 100644
--- a/src/slurmd/slurmstepd/req.c
+++ b/src/slurmd/slurmstepd/req.c
@@ -607,8 +607,6 @@ static int _handle_job_step_get_info(int fd, stepd_step_rec_t *step, uid_t uid)
buffer);
slurm_send_node_msg(msg.conn_fd, &response_msg);
FREE_NULL_BUFFER(buffer);
-
- slurm_send_rc_msg(&msg, SLURM_SUCCESS);
slurm_free_msg_members(&msg);
done:
@@ -752,9 +750,10 @@ static int _handle_step_layout(int fd, stepd_step_rec_t *step, uid_t uid)
step_layout);
slurm_send_node_msg(msg.conn_fd, &response_msg);
slurm_step_layout_destroy(step_layout);
+ } else {
+ slurm_send_rc_msg(&msg, rc);
}
- slurm_send_rc_msg(&msg, rc);
slurm_free_msg_members(&msg);
done:
@@ -789,6 +788,8 @@ static int _handle_job_sbcast_cred(int fd, stepd_step_rec_t *step, uid_t uid)
slurm_send_node_msg(msg.conn_fd, &response_msg);
slurm_free_sbcast_cred_msg(job_info_resp_msg);
+ slurm_free_msg_members(&msg);
+ return rc;
resp:
slurm_send_rc_msg(&msg, rc);
@@ -839,6 +840,8 @@ static int _handle_het_job_alloc_info(int fd, stepd_step_rec_t *step, uid_t uid)
resp_list);
slurm_send_node_msg(msg.conn_fd, &response_msg);
FREE_NULL_LIST(resp_list);
+ slurm_free_msg_members(&msg);
+ return rc;
resp:
slurm_send_rc_msg(&msg, rc);
@@ -2163,31 +2166,39 @@ _handle_completion(int fd, stepd_step_rec_t *step, uid_t uid)
goto rwfail;
FREE_NULL_BUFFER(buffer);
- if (job_step_ptr && do_stepmgr) {
- int rem = 0;
- uint32_t max_rc;
- slurm_step_id_t temp_id = {
- .job_id = job_step_ptr->job_id,
- .step_het_comp = NO_VAL,
- .step_id = step_id
- };
-
- step_complete_msg_t req = {
- .range_first = first,
- .range_last = last,
- .step_id = temp_id,
- .step_rc = step_rc,
- .jobacct = jobacct
- };
-
- step_partial_comp(&req, uid, true, &rem, &max_rc);
-
- safe_write(fd, &rc, sizeof(int));
- safe_write(fd, &errnum, sizeof(int));
-
- jobacctinfo_destroy(jobacct);
-
- return SLURM_SUCCESS;
+ if (do_stepmgr) {
+ slurm_mutex_lock(&stepmgr_mutex);
+ if (job_step_ptr) {
+ int rem = 0;
+ uint32_t max_rc;
+ slurm_step_id_t temp_id = {
+ .job_id = job_step_ptr->job_id,
+ .step_het_comp = NO_VAL,
+ .step_id = step_id
+ };
+
+ step_complete_msg_t req = {
+ .range_first = first,
+ .range_last = last,
+ .step_id = temp_id,
+ .step_rc = step_rc,
+ .jobacct = jobacct
+ };
+
+ step_partial_comp(&req, uid, true, &rem, &max_rc);
+
+ safe_write(fd, &rc, sizeof(int));
+ safe_write(fd, &errnum, sizeof(int));
+
+ jobacctinfo_destroy(jobacct);
+
+ rc = SLURM_SUCCESS;
+ } else {
+ error("Asked to complete a stepmgr step but we don't have a job_step_ptr. This should never happen.");
+ rc = SLURM_ERROR;
+ }
+ slurm_mutex_unlock(&stepmgr_mutex);
+ return rc;
}
/*
diff --git a/src/slurmd/slurmstepd/slurmstepd.c b/src/slurmd/slurmstepd/slurmstepd.c
index 494fb35dc09..09f20b2eb99 100644
--- a/src/slurmd/slurmstepd/slurmstepd.c
+++ b/src/slurmd/slurmstepd/slurmstepd.c
@@ -881,10 +881,31 @@ _init_from_slurmd(int sock, char **argv, slurm_addr_t **_cli,
if (task_msg->job_ptr &&
!xstrcmp(conf->node_name, task_msg->job_ptr->batch_host)) {
+ slurm_addr_t *node_addrs;
+
/* only allow one stepd to be stepmgr. */
job_step_ptr = task_msg->job_ptr;
job_step_ptr->part_ptr = task_msg->part_ptr;
job_node_array = task_msg->job_node_array;
+
+ /*
+ * job_record doesn't pack its node_addrs array, so get
+ * it from the cred.
+ */
+ if (task_msg->cred &&
+ (node_addrs = slurm_cred_get(
+ task_msg->cred,
+ CRED_DATA_JOB_NODE_ADDRS))) {
+ add_remote_nodes_to_conf_tbls(
+ job_step_ptr->nodes, node_addrs);
+
+ job_step_ptr->node_addrs =
+ xcalloc(job_step_ptr->node_cnt,
+ sizeof(slurm_addr_t));
+ memcpy(job_step_ptr->node_addrs, node_addrs,
+ job_step_ptr->node_cnt *
+ sizeof(slurm_addr_t));
+ }
}
break;
diff --git a/testsuite/README b/testsuite/README
index cd2f8b0ef8a..fe8aeb6d6cc 100644
--- a/testsuite/README
+++ b/testsuite/README
@@ -592,7 +592,6 @@ test21.35 Validate DenyOnLimit QoS flag is enforced on QoS and Associations.
test21.36 Validate that sacctmgr lost jobs fixes lost jobs.
test21.37 sacctmgr show and clear stats
test21.38 sacctmgr modify limits for nested accounts with multiple users
-test21.39 sacctmgr create qos/account job and then delete account/qos
test21.40 Test association plus partition/job QoS unique node limits enforced
test21.41 sacctmgr update job set newwckey=
test21.42 Test if headers returned by sacctmgr show can be used as format= specifiers
@@ -793,6 +792,7 @@ test_102_# Testing of sacctmgr options.
test_102_1 /commands/sacctmgr/test_federation.py
test_102_2 /commands/sacctmgr/test_--usage.py
test_102_3 /commands/sacctmgr/test_--json.py
+test_102_5 sacctmgr create qos/account job and then delete account/qos
test_103_# Testing of salloc options.
=======================================
diff --git a/testsuite/expect/test15.4 b/testsuite/expect/test15.4
index 400e8af1bb9..f0862350f5b 100755
--- a/testsuite/expect/test15.4
+++ b/testsuite/expect/test15.4
@@ -53,7 +53,6 @@ set login_grp_info [get_my_id]
#
# Submit a slurm job that will execute 'id'
#
-set timeout $max_job_delay
spawn $salloc -N1 -t1 $srun $bin_id
expect {
-re "Granted job allocation ($number)" {
@@ -73,15 +72,12 @@ expect {
}
}
-if {$got_job_grps == 0} {
- fail "Did not get user info from slurm job"
-}
+subtest {$got_job_grps != 0} "Verify we were able to get user info from slurm job"
+
#
# Confirm the user id and group id in the slurm job matches that
# of the local 'id' execution.
#
-if {$login_grp_info ne $job_grp_info} {
- fail "Login and slurm user info mismatch"
-}
+subtest {$login_grp_info eq $job_grp_info} "Verify user info from login and job match" "$login_grp_info != $job_grp_info"
diff --git a/testsuite/expect/test21.39 b/testsuite/expect/test21.39
deleted file mode 100755
index 4131a478fc4..00000000000
--- a/testsuite/expect/test21.39
+++ /dev/null
@@ -1,442 +0,0 @@
-#!/usr/bin/env expect
-############################################################################
-# Purpose: Test of Slurm functionality
-# sacctmgr create qos/account job and then delete account/qos
-############################################################################
-# Copyright (C) SchedMD LLC.
-# All rights reserved.
-#
-# This file is part of Slurm, a resource management program.
-# For details, see