easy_slurm.jobs#

For submitting jobs.

easy_slurm.jobs.create_job_dir(job_dir: str, src: Sequence[str])[source]#

Creates job directory and freezes all necessary files.

easy_slurm.jobs.create_job_interactive_script_source(sbatch_options: dict[str, Any], job_dir: str, job_path: str, cleanup_seconds: int) → str[source]#

Returns source for interactive job script.

easy_slurm.jobs.create_job_script_source(sbatch_options: dict[str, Any], on_run: str, on_run_resume: str, setup: str, setup_resume: str, teardown: str, job_dir: str, cleanup_seconds: int, resubmit_limit: int) → str[source]#

Returns source for job script.

easy_slurm.jobs.submit_job(job_dir: str, *, src: Sequence[str] = (), on_run: str = '', on_run_resume: str = '', setup: str = '', setup_resume: str = '', teardown: str = '', sbatch_options: dict[str, Any] = {}, cleanup_seconds: int = 120, submit: bool = True, interactive: bool = False, resubmit_limit: int = 64, config: dict[str, Any] = {}) → str[source]#

Submits job.

Creates job directory with frozen src and submits job to slurm.

Parameters

  • job_dir (str) – Path to directory to keep all job files including src.tar and auto-generated job.sh.

  • src (list[str]) – Path to directories containing only source code. These will be archived in $JOB_DIR/src.tar and extracted during job run into $SLURM_TMPDIR.

  • on_run (str) – Bash code executed in “on_run” stage, but only for new jobs that are running for the first time. Must be a single command only. Optionally, the command may gracefully handle interrupts.

  • on_run_resume (str) – Bash code executed in “on_run” stage, but only for jobs that are resuming from previous incomplete runs. Must be a single command only. Optionally, the command may gracefully handle interrupts.

  • setup (str) – Bash code executed in “setup” stage, but only for new jobs that are running for the first time.

  • setup_resume (str) – Bash code executed in “setup” stage, but only for jobs that are resuming from previous incomplete runs. To reuse the code from setup, simply set this to "setup", which calls the code inside the setup function.

  • teardown (str) – Bash code executed in “teardown” stage.

  • sbatch_options (dict[str, Any]) – Dictionary of options to pass to sbatch.

  • cleanup_seconds (int) – Interrupts a job n seconds before timeout to run cleanup tasks (teardown, auto-schedule new job). Default is 120 seconds.

  • submit (bool) – Submit created job to scheduler. Set this to False if you are manually submitting the created $JOB_DIR later. Default is True.

  • interactive (bool) – Run as a blocking interactive job. Default is False.

  • resubmit_limit (int) – Maximum number of times to auto-submit a job for “resume”. (Not entirely unlike submitting a resume for a job.) Default is 64 resubmissions.

  • config (dict[str, Any]) – A dictionary of configuration values to use for formatting.

Returns

Path to the newly created job directory.

easy_slurm.jobs.submit_job_dir(job_dir: str, interactive: bool)[source]#

Submits a $JOB_DIR created by easy_slurm to slurm.

Note that submit_job already does this for the user, except when it is called with submit=False.