Job Data submodules

slurmise.job_data module

class slurmise.job_data.JobData(job_name: str, slurm_id: str | None = None, categories: dict = <factory>, numerics: dict = <factory>, memory: int | None = None, runtime: int | None = None, cmd: str | None = None)[source]

Bases: object

Jobdata class holds the information of a unique slurm job.

Arguments:

job_name:: The unique command name to execute under slurm.
slurm_id:: The slurm job id assigned by the sceduler for a job run.
categories:: the CLI parameters of this job. This has parameter that affect the performance of the job and are fit seperately.
numerics:: These are parameters that are used as the free variables for fits, such input size, number of iterations etc.
memory:: The maximum amount of memory in MBs this job used.
runtime:: The time this job needed to complete in minutes.

categories: dict

cmd: str | None = None

static from_dataset(job_name: str, slurm_id: str, dataset: Dataset, categories: dict) → JobData[source]

This method creates a JobData object from a HDF5 dataset that describes a job. :arguments:

job_name:

The unique command name to execute under slurm.

slurm_id:

The slurm job id assigned by the sceduler for a job run.

dataset:

The HDF5 dataset used to populate numerics, memory and runtime information of the job.

job_name: str

memory: int | None = None

numerics: dict

runtime: int | None = None

slurm_id: str | None = None

slurmise.job_data.array_safe_eq(a, b) → bool[source]: Check if a and b are equal, even if they are numpy arrays. When a and be are dictionaries call recursively for all key, value pairs.

slurmise.job_data.dc_eq(dc1, dc2) → bool[source]: Checks if two dataclasses which hold numpy arrays are equal

slurmise.job_database module

class slurmise.job_database.JobDatabase(db_file: str, max_retries: int = 10)[source]

Bases: object

This class creates the database to store job information. It saves the database in HDF5 file.

clear()[source]

property db_file

delete(job_data: JobData, delete_all_children: bool = False) → None[source]

Delete jobs with matching job name and categories.

Arguments:

job_data:: JobData object with name and categories which should be removed.
delete_all_children:: When true, will delete recursively any matching jobs

static get_database(db_file: str, max_retries: int = 10) → JobDatabase[source]

Use in context manager to automatically open and close db file.

Arguments:

db_file:: HDF5 file to use as database

Yields:

JobDatabase with opened db file

Finally:

Closes h5py database

static get_group_name(job_data: JobData) → str[source]

static get_table_name(job_data: JobData) → str[source]

static is_dataset(f: Any) → bool[source]: Test if object is an h5py Dataset

static is_slurm_job(f: Any) → bool[source]: Test if object is non-empty or its first element is a Dataset. This is consistent with a slurm job

iterate_database(update_missing: bool = False, job_name: str | None = None) → Generator[tuple[JobData, list[JobData]]][source]: Yield key (query job) value (list of jobs) pairs of the entire database, or of a single job_name when given.

static iterate_jobs(h5py_obj, categories=None) → Generator[tuple[tuple[str, ...], dict[str, Group]]][source]

Helper function to recursively iterate through the database and yield job groups with their categories as tuples. Note, jobs are NOT yielded ordred by slurm-id.

Arguments:

h5py_obj:: the current h5py object to check for jobs
categories:: the categories found on the way to the current h5py object as a tuple of strings

Yields:

Tuple of categories and dict of slurm_id to h5py dataset for each job in the database For example two jobs that have the same name and categories: ((“test_job”, “option1=value1”, “option2=value2”), {“123”: <h5py dataset>, “456”: <h5py dataset>}).

job_exists(job_data: JobData) → bool[source]

print()[source]

static print_hdf5(h5py_obj, level=-1, print_full_name: bool = False, print_attrs: bool = True) → None[source]

Prints the name and shape of datasets in a H5py HDF5 file.

Parameters:

h5py_obj ([h5py.File, h5py.Group]) – the h5py.File or h5py.Group object
level (int) – What level of the file tree you are in
print_full_name – If True, the full tree will be printed as the name, e.g. /group0/group1/group2/dataset: … If False, only the current node will be printed, e.g. dataset:
print_attrs – If True: print all attributes in the file

Return type:

None

query(job_data: JobData, update_missing: bool = False) → list[JobData][source]

Query returns a list of JobData objects based on the requested JobData. The returned jobs match the query JobData’s job name and categories. update_missing will try to get maxRSS and elapsed from sacct if not found in the DB.

Note: It does not descend into all child categories, only the highest matching leaves

query_fit(fit)[source]

record(job_data: JobData, ignore_existing_job: bool = False) → None[source]: It records JobData information in the database. A tree is created based on the job name, categories and slurm id. The leaves of the tree are the memory, runtime and numerics of the JobData.

record_fit(fit)[source]

update(**kargs)[source]

update_missing_data(jobs: list[JobData]) → list[JobData][source]

Update missing mem and runtime for jobs with incomplete data in the db. Takes a list of JobData which was queried from the db, updates the db, and returns the updated job list.

TODO: gather slurm_ids of jobs that need updating and do it in one call

Config submodule

slurmise.config module

class slurmise.config.SlurmiseConfiguration(toml_file: Path)[source]

Bases: object

SlurmiseConfiguration class parses and stores TOML configuration files for slurmise.

add_defaults(job_data: JobData) → JobData[source]: Add default values to a job data object.

correct_minimum(job_data: JobData) → JobData[source]: Ensure predicted values are larger than set minimum.

dry_parse(cmd: str, job_name: str | None = None)[source]

get_model_class(job_name: str)[source]: Returns the model class a job is using.

get_sources(job_name: str) → dict[source]

parse_job_cmd(cmd: str, job_name: str | None = None, slurm_id: str | None = None, step_id: str | None = None) → JobData[source]: Parse a job data dataset into a JobData object.

parse_job_from_dict(variables: dict, job_name: str, slurm_id: str | None = None, step_id: str | None = None) → JobData[source]: Parse a job data dataset into a JobData object.

slurmise.config.find_config_file() → Path[source]: Search for a config file if it is not provided. Search first in the current working directory, and if the file does not exist search in the default slurmise base directory.

Slurm submodule

slurmise.slurm module

slurmise.slurm.get_current_job_id() → str | None[source]: Return the job ID of the current SLURM job, or None when not inside a SLURM job.

slurmise.slurm.get_slurm_job_sacct(slurm_id: str) → dict[source]: Return the JSON output of the sacct command for the given SLURM job.

slurmise.slurm.parse_slurm_job_metadata(slurm_id: str | None = None, step_id: str | None = None) → dict[source]

Return a dictionary of metadata for the current SLURM job. :param slurm_id: The SLURM job ID. If None, the function will attempt to retrieve

the job ID from the SLURM_JOB_ID environment variable.

Parameters:: step_id (str | None) – The SLURM step ID. If None, the function defaults to the last step of the job. If provided, it specifies which step’s metadata to return.
Returns:: A dictionary containing metadata for the specified SLURM job and step.
Return type:: dict

slurmise.slurm.resolve_job_id(slurm_id: str | int | None = None, step_id: str | None = None) → str[source]

Resolve a job ID, falling back to the current SLURM job’s environment. :param slurm_id: The SLURM job ID. If None, the ID is read from the

SLURM_JOB_ID environment variable.

Parameters:: step_id (str | None) – The SLURM step ID. If provided, it is appended to the job ID as “<slurm_id>.<step_id>”.
Returns:: The resolved job ID, with the step ID appended if provided.
Return type:: str

slurmise.slurm.split_job_id(slurm_id: str) → tuple[str, str | None][source]: Split a combined “<slurm_id>.<step_id>” string into its parts.

Fit submodules

slurmise.fit.resource_fit module

class slurmise.fit.resource_fit.ResourceFit(*, query: 'JobData', last_fit_dsize: 'int' = 0, fit_timestamp: 'datetime.datetime' = <factory>, model_metrics: 'dict' = <factory>, path: 'pathlib.Path | None' = None)[source]

Bases: object

fit(jobs: list[JobData], random_state: RandomState | None, **kwargs)[source]

fit_timestamp: datetime

last_fit_dsize: int = 0

classmethod load(query: JobData | None = None, path: str | None = None, **kwargs) → ResourceFit[source]

This method loads a model from a file. The model is loaded from the path provided, or from the path generated from the query.

Parameters:

query (JobData) – The query used to generate the model
path (str) – The path to the model
kwargs – Additional keyword arguments to pass to the model

Returns:

The model

Return type:

ResourceFit

classmethod mean_percent_error(y_true, y_pred) → float[source]: Mean percent error, skipping records whose true value is zero.

model_metrics: dict

path: Path | None = None

predict(job: JobData) → tuple[JobData, list[str]][source]

query: JobData

save(model_params: dict | None = None)[source]: This method saves the basic information of the model, such as its query, when it was last fit, the dataset size of the latest fit, and the type of the model.

slurmise.fit.poly_fit module

class slurmise.fit.poly_fit.PolynomialFit(*, query: 'JobData', last_fit_dsize: 'int' = 0, fit_timestamp: 'datetime.datetime' = <factory>, model_metrics: 'dict' = <factory>, path: 'pathlib.Path | None' = None, degree: 'int' = 2, runtime_model: 'InitVar[Pipeline | None]' = None, memory_model: 'InitVar[Pipeline | None]' = None)[source]

Bases: ResourceFit

degree: int = 2

classmethod load(query: JobData | None = None, path: str | None = None) → PolynomialFit[source]

This method loads a model from a file. The model is loaded from the path provided, or from the path generated from the query.

Parameters:

query (JobData) – The query used to generate the model
path (str) – The path to the model
kwargs – Additional keyword arguments to pass to the model

Returns:

The model

Return type:

ResourceFit

memory_model: dataclasses.InitVar[sklearn.pipeline.Pipeline | None] = None

runtime_model: dataclasses.InitVar[sklearn.pipeline.Pipeline | None] = None

save()[source]: This method saves the basic information of the model, such as its query, when it was last fit, the dataset size of the latest fit, and the type of the model.

slurmise.fit.kneighbors_fit module

class slurmise.fit.kneighbors_fit.KNNFit(*, query: 'JobData', last_fit_dsize: 'int' = 0, fit_timestamp: 'datetime.datetime' = <factory>, model_metrics: 'dict' = <factory>, path: 'pathlib.Path | None' = None, nneighbors: 'int' = 5, runtime_model: 'InitVar[Pipeline | None]' = None, memory_model: 'InitVar[Pipeline | None]' = None)[source]

Bases: ResourceFit

classmethod load(query: JobData | None = None, path: str | None = None) → KNNFit[source]

This method loads a model from a file. The model is loaded from the path provided, or from the path generated from the query.

Parameters:

query (JobData) – The query used to generate the model
path (str) – The path to the model
kwargs – Additional keyword arguments to pass to the model

Returns:

The model

Return type:

ResourceFit

memory_model: dataclasses.InitVar[sklearn.pipeline.Pipeline | None] = None

nneighbors: int = 5

runtime_model: dataclasses.InitVar[sklearn.pipeline.Pipeline | None] = None