asimtools.asimmodules.workflows package

asimtools.asimmodules.workflows.calc_array module

Apply the same asimmodule using multiple calculators

Author: mkphuthi@github.com

asimtools.asimmodules.workflows.calc_array.calc_array(subsim_input: Dict, calc_ids: Sequence[str] | None = None, template_calc_id: str | None = None, calculators: Sequence[Dict] | None = None, template_calculator: Dict | None = None, key_sequence: Sequence[str] | None = None, array_values: Sequence | None = None, file_pattern: str | None = None, linspace_args: Sequence | None = None, arange_args: Sequence | None = None, label_prefix: str | None = None, env_ids: Sequence[str] | str | None = None, calc_input: Dict | None = None, env_input: Dict | None = None, labels: Sequence | str | None = 'values', str_btn_args: Dict | None = None, secondary_key_sequences: Sequence | None = None, secondary_array_values: Sequence | None = None, array_max: int | None = None, skip_failed: bool | None = False, group_size: int = 1) → Dict[source]

Apply the same asimmodule using different calculators and if necessary different environments

Parameters:

  • calc_ids (Sequence, optional) – Deprecated. Use calculators instead. Iterable with calc_ids, defaults to None

  • template_calc_id (str, optional) – Deprecated. Use template_calculator instead. calc_id of the template calculator, defaults to None

  • calculators (Sequence[Dict], optional) – Sequence of calculator dicts, each with a ‘calc_id’ or ‘calc_params’ key, see asimtools.calculators.load_calc(), defaults to None

  • template_calculator (Optional[Dict], optional) – Calculator dict with ‘calc_id’ or ‘calc_params’ key used as template when iterating over key_sequence values, see asimtools.calculators.load_calc(), defaults to None

  • calc_input (Dictionary, optional) – Dictionary of calculator inputs

  • labels (Sequence, optional :param array_values: values to be iterated over in each simulation, defaults to None) – Iterable with custom labels for each calc, defaults to None

  • file_pattern (Optional[str], optional) – pattern of files to be iterated over in each simulation, defaults to None

  • linspace_args (Optional[Sequence], optional) – arguments to pass to numpy.linspace() to be iterated over in each simulation, defaults to None

  • arange_args (Optional[Sequence], optional) – arguments to pass to numpy.arange() to be iterated over in each simulation, defaults to None

  • label_prefix (str, optional) – Prefix to add before labels which can make extracting data from file paths easier, defaults to None

  • env_ids (Optional[Union[Sequence[str],str]], optional) – Environment(s) to be used for each simulation, must either be a list with as many env_ids as array values or a string with the env_id to be used by all simulations, defaults to None

  • secondary_key_sequences (Sequence, optional) – list of other keys to iterate over in tandem with key_sequence to allow changing multiple key-value pairs, defaults to None

  • secondary_array_values (Sequence, optional) – list of other other array_values to iterate over in tandem with array_values to allow changing multiple key-value pairs, defaults to None

  • array_max (Optional[int], optional) – Maximum number of jobs to run simultanteously in job array, defaults to None

  • skip_failed (Optional[bool], optional) – Skip failed jobs and move to next, defaults to False

  • group_size (int, optional) – Number of jobs to group together, defaults to 1

Returns:

Dictionary of results

Return type:

Dict

asimtools.asimmodules.workflows.chained module

Submits a chain of asimmodules in the order specified by steps in input

Author: mkphuthi@github.com

asimtools.asimmodules.workflows.chained.chained(steps: Dict, env_input: Dict | None = None, calc_input: Dict | None = None) → Dict[source]

Run a workflow using the provided sim_input.yaml one after the other, The jobs will run one after the other, if slurm is used, job dependencies will be used. Note that if one of the subsim_inputs internally launches another asimmodule e.g. image_array, calc_array etc, slurm job dependencies will not work as expected. In those cases, we recommend running the steps one by one or interactively

Parameters:

  • steps (Dict) – Dictionary where each key is an ID and the value is a standard sim_input

  • env_input (Optional[Dict], optional) – env_input that overrides global file, defaults to None

  • calc_input (Optional[Dict], optional) – calc_input that overrides global file, defaults to None

Returns:

Dictionary of results

Return type:

Dict

asimtools.asimmodules.workflows.distributed module

Distributes a number of simulations at the same time if possible

Author: mkphuthi@github.com

asimtools.asimmodules.workflows.distributed.distributed(subsim_inputs: Dict, env_input: Dict | None = None, calc_input: Dict | None = None, array_max: int | None = None, group_size: int = 1, skip_failed: bool | None = False) → Dict[source]

Distributes a set of jobs based on inpout parameter. The scnarios are as follows: #. use_slurm: False, interactive: True -> Runs one after the other. #. use_slurm: True, interactive: False, same env_id -> Launches a job # array. #. use_slurm: False, interactive: False, different env_ids -> Launches # multiple jobs.

Parameters:

  • subsim_inputs (Dict) – Dictionary of asimmodules, each key is an ID and each value is a sim_input file

  • env_input (Optional[Dict], optional) – env_input that overrides global file, defaults to None

  • calc_input (Optional[Dict], optional) – calc_input that overrides global file, defaults to None

  • array_max (Optional[int], optional) – Maximum number of jobs to run in array, defaults to None

  • group_size (int, optional) – Number of jobs to group together, defaults to 1

  • skip_failed (Optional[bool], optional) – Skip failed jobs and move to next, releavnt when running one job after the other, defaults to False

Returns:

Dictionary of results

Return type:

Dict

asimtools.asimmodules.workflows.image_array module

Apply the same asimmodule on multiple images

Author: mkphuthi@github.com

asimtools.asimmodules.workflows.image_array.image_array(images: Dict, subsim_input: Dict | None = None, template_sim_input: Dict | None = None, calc_input: Dict | None = None, env_input: Dict | None = None, array_max: int | None = None, skip_failed: bool | None = False, key_sequence: Sequence[str] | None = None, env_ids: Sequence[str] | str | None = None, labels: Sequence | str | None = None, label_prefix: str | None = None, str_btn_args: Dict | None = None, secondary_key_sequences: Sequence | None = None, secondary_array_values: Sequence | None = None, group_size: int = 1) → Dict[source]

Submit the same asimmodule on multiple images and if specified, use different env_ids

Parameters:

  • images (Dict) – Images specification, see asimtools.utils.get_images()

  • subsim_input (Optional[Dict], optional) – sim_input of asimmodule to be run, included for backward compatibility, please use template_sim_input instead, defaults to None

  • template_sim_input (Optional[Dict], optional) – sim_input of asimmodule to be run, defaults to None

  • str_btn_args (Optional[Sequence], optional) – args to pass to asimtools.utils.get_str_btn()

  • calc_input (Optional[Dict], optional) – calc_input to override global file, defaults to None

  • env_input (Optional[Dict], optional) – env_input to override global file, defaults to None

  • array_max (Optional[int], optional) – Maximum jobs to run simultanteously in job array, defaults to None

  • skip_failed (Optional[bool], optional) – Skip failed jobs and move to next, defaults to False

  • labels (Sequence[str], optional) – Custom labels for each image, defaults to None

  • env_ids (Sequence[str], optional) – Sequence of envs for each image, must be the same length as the total number of images, defaults to None

  • labels – Custom labels to use for each simulation, defaults to None

  • label_prefix (str, optional) – Prefix to add before labels which can make extracting data from file paths easier, defaults to None

  • secondary_key_sequences (Sequence, optional) – list of other keys to iterate over in tandem with key_sequence to allow changing multiple key-value pairs, defaults to None

  • secondary_array_values (Sequence, optional) – list of other other array_values to iterate over in tandem with images to allow changing multiple key-value pairs, defaults to None

  • group_size (int, optional) – Number of jobs to group together, defaults to 1

Returns:

Dictionary of results

Return type:

Dict

asimtools.asimmodules.workflows.iterative module

Runs the same asimmodule, iterating over multiple values of a specified argument one after the other based on a sim_input template provided by the user

Author: mkphuthi@github.com

asimtools.asimmodules.workflows.iterative.iterative(template_sim_input: Dict, dependent_file: PathLike | None = None, dependent_file_key_sequence: Sequence[str] | None = None, key_sequence: Sequence[str] | None = None, array_values: Sequence | None = None, file_pattern: str | None = None, linspace_args: Sequence | None = None, arange_args: Sequence | None = None, env_ids: Sequence[str] | str | None = None, calc_input: Dict | None = None, env_input: Dict | None = None, str_btn_args: Dict | None = None, secondary_key_sequences: Sequence | None = None, secondary_array_values: Sequence | None = None) → Dict[source]

Runs the same asimmodule, iterating over multiple values of a specified argument based on a sim_input template provided by the user

Parameters:

  • template_sim_input (Dict) – sim_input containing all the default parameters

  • key_sequence (Optional[Sequence[str]], optional) – Sequence of keys to access value to be iterated over, defaults to None

  • array_values (Optional[Sequence], optional) – values to be iterated over in each simulation, defaults to None

  • file_pattern (Optional[str], optional) – pattern of files to be iterated over in each simulation, defaults to None

  • linspace_args (Optional[Sequence], optional) – arguments to pass to numpy.linspace() to be iterated over in each simulation, defaults to None

  • arange_args (Optional[Sequence], optional) – arguments to pass to numpy.arange() to be iterated over in each simulation, defaults to None

  • labels (Sequence, optional) – Custom labels to use for each simulation, defaults to None

  • label_prefix (str, optional) – Prefix to add before labels which can make extracting data from file paths easier, defaults to None

  • env_ids (Optional[Union[Sequence[str],str]], optional) – Environment(s) to be used for each simulation, must either be a list with as many env_ids as array values or a string with the env_id to be used by all simulations, defaults to None

  • secondary_key_sequences (Sequence, optional) – list of other keys to iterate over in tandem with key_sequence to allow changing multiple key-value pairs, defaults to None

  • secondary_array_values (Sequence, optional) – list of other other array_values to iterate over in tandem with array_values to allow changing multiple key-value pairs, defaults to None

  • array_max (int, optional) – Number of jobs to run at once in scheduler

  • calc_input (Optional[Dict], optional) – calc_input file to use, defaults to None

  • env_input (Optional[Dict], optional) – env_input to use, defaults to None

Returns:

Results

Return type:

Dict

asimtools.asimmodules.workflows.sim_array module

Runs the same asimmodule, iterating over multiple values of a specified argument based on a sim_input template provided by the user. The tasks are run simultaneously if resources are available

Author: mkphuthi@github.com

asimtools.asimmodules.workflows.sim_array.sim_array(template_sim_input: Dict, key_sequence: Sequence[str] | None = None, array_values: Sequence | None = None, file_pattern: str | None = None, linspace_args: Sequence | None = None, arange_args: Sequence | None = None, placeholder: str | None = None, env_ids: Sequence[str] | str | None = None, calc_input: Dict | None = None, env_input: Dict | None = None, labels: Sequence | str | None = 'values', label_prefix: str | None = None, str_btn_args: Dict | None = None, secondary_key_sequences: Sequence | None = None, secondary_array_values: Sequence | None = None, secondary_placeholders: Sequence | None = None, array_max: int | None = None, skip_failed: bool | None = False, as_integers: bool | None = False, group_size: int = 1) → Dict[source]

Runs the same asimmodule, iterating over multiple values of a specified argument based on a sim_input template provided by the user

Parameters:

  • template_sim_input (Dict) – sim_input containing all the default parameters

  • key_sequence (Optional[Sequence[str]], optional) – Sequence of keys to access value to be iterated over, defaults to None

  • array_values (Optional[Sequence], optional) – values to be iterated over in each simulation, defaults to None

  • file_pattern (Optional[str], optional) – pattern of files to be iterated over in each simulation, defaults to None

  • linspace_args (Optional[Sequence], optional) – arguments to pass to numpy.linspace() to be iterated over in each simulation, defaults to None

  • arange_args (Optional[Sequence], optional) – arguments to pass to numpy.arange() to be iterated over in each simulation, defaults to None

  • placeholder (Optional[str], optional) – placeholder is string dict value to replace with the array value

  • labels (Sequence, optional) – Custom labels to use for each simulation, defaults to None

  • label_prefix (str, optional) – Prefix to add before labels which can make extracting data from file paths easier, defaults to None

  • env_ids (Optional[Union[Sequence[str],str]], optional) – Environment(s) to be used for each simulation, must either be a list with as many env_ids as array values or a string with the env_id to be used by all simulations, defaults to None

  • secondary_key_sequences (Sequence, optional) – list of other keys to iterate over in tandem with key_sequence to allow changing multiple key-value pairs, defaults to None

  • secondary_array_values (Sequence, optional) – list of other other array_values to iterate over in tandem with array_values to allow changing multiple key-value pairs, defaults to None

  • secondary_placeholders (Sequence, optional) – list of other other placeholders to iterate over in tandem with array_values to allow changing multiple key-value pairs, defaults to None

  • array_max (int, optional) – Number of jobs to run at once in scheduler

  • calc_input (Optional[Dict], optional) – calc_input file to use, defaults to None

  • env_input (Optional[Dict], optional) – env_input to use, defaults to None

  • skip_failed (Optional[bool], optional) – Skip failed jobs and move to next, defaults to False

  • group_size (int, optional) – Number of jobs to group together, defaults to 1

  • as_integers (bool, False) – Whether to return the values as integers, useful for indexing

Returns:

Results

Return type:

Dict

asimtools.asimmodules.workflows.update_dependencies module

Mostly internal tool used to chain together batch jobs that have nested dependencies

Author: mkphuthi@github.com

asimtools.asimmodules.workflows.update_dependencies.update_dependencies(prev_step_dir: PathLike | None = None, next_step_dir: PathLike | None = None, skip_failed: bool = False) → Dict[source]

Asimmodule for connecting jobs where an asimmodule launches slurm jobs internally, examples are image_array, sim_array and calc_array

Parameters:

  • prev_step_dir (Optional[os.PathLike], optional) – workdir of previous step, usually step-[N-1] where N is current step, defaults to None

  • next_step_dir (Optional[os.PathLike], optional) – workdir of next step, usually step-[N+1] where N is current step, defaults to None

  • skip_failed (bool, optional) – whether to submit if any jobs failed or not, defaults to False

Returns:

Nothing

Return type:

Dict

asimtools.asimmodules.workflows.utils module

asimtools.asimmodules.workflows.utils.prepare_array_vals(key_sequence: Sequence[str] | None = None, array_values: Sequence | None = None, file_pattern: str | None = None, linspace_args: Sequence | None = None, arange_args: Sequence | None = None, env_ids: Sequence[str] | str | None = None, labels: Sequence | str | None = 'values', label_prefix: str | None = None, str_btn_args: Dict | None = None, secondary_key_sequences: Sequence | None = None, secondary_array_values: Sequence | None = None, as_integers: bool | None = False)[source]

Helper function for preparing things needed for the different arrays

Parameters:

  • key_sequence (Optional[Sequence[str]], optional) – Sequence of keys to access value to be iterated over, defaults to None

  • array_values (Optional[Sequence], optional) – values to be iterated over in each simulation, defaults to None

  • file_pattern (Optional[str], optional) – pattern of files to be iterated over in each simulation, defaults to None

  • linspace_args (Optional[Sequence], optional) – arguments to pass to numpy.linspace() to be iterated over in each simulation, defaults to None

  • arange_args (Optional[Sequence], optional) – arguments to pass to numpy.arange() to be iterated over in each simulation, defaults to None

  • labels (Sequence, optional) – Custom labels to use for each simulation. If “str_btn” provide arguments to asimtools.utils.get_str_btn() as additional str_btn_args keyword. If labels is an integer N, the Nth label in the file_pattern or array_values is used, defaults to None

  • label_prefix (str, optional) – Prefix to add before labels which can make extracting data from file paths easier, defaults to None

  • env_ids (Optional[Union[Sequence[str],str]], optional) – Environment(s) to be used for each simulation, must either be a list with as many env_ids as array values or a string with the env_id to be used by all simulations, defaults to None

  • secondary_key_sequences (Sequence, optional) – list of other keys to iterate over in tandem with key_sequence to allow changing multiple key-value pairs, defaults to None

  • secondary_array_values (Sequence, optional) – list of other other array_values to iterate over in tandem with array_values to allow changing multiple key-value pairs, defaults to None

  • as_integers (bool, False) – Whether to return the values as integers, useful for indexing

Returns:

Results

Return type:

Dict

Module contents