Table of Contents

OptimizationControlMechanism

Contents

Overview

An OptimizationControlMechanism is a ControlMechanism that uses an OptimizationFunction to optimize the performance of the Composition for which it is a controller. It does so by using the OptimizationFunction (assigned as its function) to execute its agent_rep – a representation of the Composition to be optimized – under different control_allocations, and selecting the one that optimizes its net_outcome. An OptimizationControlMechanism can be configured to implement various forms of optimization, ranging from fully model-based optimization that uses the Composition itself as the agent_rep to simulate the outcome for a given state (i.e., a combination of the current input and a particular control_allocation), to fully model-free optimization by using a CompositionFunctionApproximator as the agent_rep that learns to predict the outcomes for a state. Intermediate forms of optimization can also be implemented, that use simpler Compositions to approximate the dynamics of the full Composition. The outcome of executing the agent_rep is used to compute a net_outcome for a given state, that takes into account the costs associated with the control_allocation, and is used to determine the optimal control_allocations.

Expected Value of Control

The net_outcome of an OptimizationControlMechanism’s agent_rep is computed – for a given state (i.e., set of state_feature_values and a control_allocation) – as the difference between the outcome computed by its objective_mechanism and the aggregated costs of its control_signals computed by its combine_costs function. If the outcome computed by the objective_mechanism is configured to measure the value of processing (e.g., reward received, time taken to respond, or a combination of these, etc.), and the OptimizationFunction assigned as the OptimizationControlMechanism’s function is configured to find the control_allocation that maximizes its net_outcome (that is, the outcome discounted by the result of the combine_costs function), then the OptimizationControlMechanism is said to be maximizing the Expected Value of Control (EVC). That is, it implements a cost-benefit analysis that weighs the costs of the ControlSignal values associated with a control_allocation against the outcome expected to result from it. The costs are computed based on the cost_options specified for each of the OptimizationControlMechanism’s control_signals and its combine_costs function. The EVC is determined by its compute_net_outcome function (assigned to its net_outcome attribute), which is computed for a given state by the OptimizationControlMechanism’s evaluate_agent_rep method. In these respects, optimization of a Composition’s performance by its OptimizationControlMechanism – as indexed by its net_outcome attribute – implement a form of Bounded Rationality, also referred to as Resource Rationality, in which the constraints imposed by the “bounds” or resources are reflected in the costs of the ControlSignals (also see Computational Rationality and Toward a Rational and Mechanistic Account of Mental Effort).

Agent Representation and Types of Optimization

Much of the functionality described above is supported by a ControlMechanism (the parent class of an OptimizationControlMechanism). The defining characteristic of an OptimizationControlMechanism is its agent representation, that is used to determine the net_outcome for a given state, and find the control_allocation that optimizes this. The agent_rep can be the Composition to which the OptimizationControlMechanism belongs (and controls), another (presumably simpler) one, or a CompositionFunctionApproximator that is used to estimate the net_outcome of the Composition of which the OptimizationControlMechanism is the controller. These different types of agent representation correspond closely to the distinction between model-based and model-free optimization in the machine learning and cognitive neuroscience literatures, as described below.

OptimizationControlMechanism

Functional Anatomy of an OptimizationControlMechanism. Panel A: Examples of use in fully model-based and model-free optimization. Note that in the example of model-based optimization (left), the OptimizationControlMechanism uses the entire Composition that it controls as its agent_rep, whereas in the example of model-free optimization (right) the the agent_rep is a CompositionFunctionApproximator. The agent_rep can also be another (presumably simpler) Composition that can be used to implement forms of optimization intermediate between fully model-based and model-free. Panel B: Flow of execution during optimization. In both panels, faded items show process of adaptation when using a CompositionFunctionApproximator as the agent_rep.


Model-Based Optimization

The fullest form of this is implemented by assigning the Composition for which the OptimizationControlMechanism is the controller) as its a`agent_rep <OptimizationControlMechanism.agent_rep>` On each TRIAL, that Composition itself is provided with either the most recent inputs to the Composition, or ones predicted for the upcoming trial (as determined by the state_feature_values of the OptimizationControlMechanism), and then used to simulate processing on that trial in order to find the control_allocation that yields the best net_outcome for that trial. A different Composition can also be assigned as the agent_rep, that approximates in simpler form the dynamics of processing in the Composition for which the OptimizationControlMechanism is the controller, implementing a more restricted form of model-based optimization.

“Model-Free” Optimization

Note

The term model-free is placed in apology quotes to reflect the fact that, while this term is used widely (e.g., in machine learning and cognitive science) to distinguish it from model-based forms of processing, model-free processing nevertheless relies on some form of model – albeit usually a much simpler one – for learning, planning and decision making. In the context of a OptimizationControlMechanism, this is addressed by use of the term “agent_rep”, and how it is implemented, as described below.

This clearest form of this uses a CompositionFunctionApproximator, that learns to predict the net_outcome for a given state (e.g., using reinforcement learning or other forms of function approximation, such as a RegressionCFA). In each TRIAL the agent_rep is used to search over control_allocations, to find the one that yields the best predicted net_outcome of processing on the upcoming trial, based on the current or (expected) state_feature_values for that trial. The agent_rep is also given the chance to adapt in order to improve its prediction of its net_outcome based on the state and net_outcome of the prior TRIAL. A Composition can also be used to generate such predictions, permitting forms of optimization that are intermediate between the extreme examples of model-based and model-free, as noted above.

Creating an OptimizationControlMechanism

The constructor has the same arguments as a ControlMechanism, with the following exceptions/additions, which are specific to the OptimizationControlMechanism:

  • state_features – specifies the sources of input to the OptimizationControlMechanism’s agent_rep which, together with a selected control_allocation, are provided as input to it’s evaluate method when that is executed to estimate or predict the Composition’s net_outcome. Those sources of input are used to construct the OptimizationControlMechanism’s state_input_ports, one for each external InputPort of the agent_rep. The input to each state_input_port, after being processed by it function, is assigned as the corresponding value of state_feature_values, the values of which provided as the input to the corresponding InputPorts of the INPUT Nodes of the agent_rep each time it is evaluated. Accordingly, the specification requirements for state_features depend on whether the agent_rep is a Composition or a CompositionFunctionApproximator, as described in each of the two sections below.


    state_features for an agent_rep that is a Composition


    Automatic assignment. By default, if state_features, state_feature_default and state_feature_function are not specified, the state_input_ports are configured to shadow the inputs of every external InputPort of the agent_rep Composition; as a result, each time agent_rep is evaluated, it receives the same external input) it received during its last TRIAL of execution.


    Explicit specification. Specifying the state_features, state_feature_default and/or state_feature_function arguments explicitly can be useful if: values need to be provided as input to the agent_rep when it is evaluated other than its external inputs; to restrict evaluation to a subset of its inputs (while others are held constant); and/or to assign specific functions to one or more state_input_ports (see below) that allow them to process the inputs (e.g., modulate and/or integrate them) before they are assigned to state_feature_values and passed to the agent_rep. Assignments can be made to state_features corresponding to any or all InputPorts of the agent_rep's INPUT Nodes, as described below. Any that are not specified are assigned the value specified for state_feature_default (SHADOW_INPUTS by default; see state_feature_default for additional details). A single assignment can be made for all state_features, or they can be specified individually for each INPUT Nodes InputPort, as descdribed below.

    Note

    If state_features are specified explicitly, the values of the specified Components must match the input_shape of the corresponding InputPorts of the agent_rep's INPUT Nodes. Those InputPorts are listed in the agent_rep's external_input_ports_of_all_input_nodes attribute and, together with examples of their values, in the OptimizationControlMechanism’s state_feature_values attribute. A failure to properly meet these requirements produces an error.

    The state_features argument can be specified using any of the following formats:

    • List – a list of individual state_feature specifications, that can be any of the forms of individual input specifications listed below. The items correspond to all of the external InputPorts of the agent_rep, and must be specified in the order they are listed in the agent_rep's external_input_ports_of_all_input_nodes attribute. If the list is incomplete, the remaining InputPorts are assigned state_feature_default as their state_feature specification, which by default is SHADOW_INPUTS (see below. Items can be included in the list that have not yet been added to the OptimizationControlMechanism’s Composition or its agent_rep. However, these must be added before the Composition is executed, and must appear in the list in the same position that the InputPorts to which they pertain are listed in the agent_rep's external_input_ports_of_all_input_nodes attribute, once construction of the agent_rep is complete.

    • Set – a set of INPUT Nodes of the agent_rep that are assigned SHADOW_INPUTS as their state_feature – that is, that should receive the same inputs during evaluation as when the Composition of which the OptimizationControlMechanism is the controller is fully executed (see below). The order of their specification does not matter; however, any of the agent_rep's INPUT Nodes that are not included in the set are assigned state_feature_default as their state_feature specification. Note that, since the default for state_feature_default is SHADOW_INPUTS, unless this is specified otherwise omitting items from a set has no effect (i.e., they too are assigned SHADOW_INPUTS); for omitted items to be treated differently, state_feature_default must be specified; for example by assigning it None so that items omitted from the set are assigned their default input value (see below.


    state_features for an agent_rep that is a CompositionFunctionApproximator


    The state_features specify the feature_values argument to the CompositionFunctionApproximator's evaluate method. These cannot be determined automatically and so they must be specified explicitly, in a list, with the correct number of items in the same order and with the same shapes they are expected have in the array passed to the feature_values argument of the evaluate method (see warning below).

    Warning

    The state_features for an agent_rep that is a CompositionFunctionApproximator cannot be created automatically nor can they be validated; thus specifying the wrong number or invalid state_features, or specifying them in an incorrect order may produce errors that are unexpected or difficult to interpret.

    The list of specifications can contain any of the forms of specification used for an agent_rep that is a Composition as described above, with the following exception: if a Mechanism is specified, its primary OutputPort is used (rather than shadowing its primary InputPort), since that is more typical usage, and there are no assumptions made about the state features of a CompositionFunctionApproximator (as there are about a Composition as agent_rep); if the input to the Mechanism is to be shadowed, then its InputPort must be specified explicitly (as described above).


  • state_feature_function – specifies a function to be used as the default function for state_input_ports. This is assigned as the function to any state_input_ports for which no other Function is specified – that is, in either an InputPort specification dictionary or a 2-item tuple in the state_features argument (see state_features). If either of the latter is specified, they override the specification in state_feature_function. If state_feature_function is not specified, then LinearCombination (the standard default Function for an InputPort) is assigned to any state_input_ports that are not otherwise assigned a Function. Specifying functions for state_input_ports can be useful, for example to provide an average or integrated value of prior inputs to the agent_rep's evaluate method during the optimization process, or to use a generative model of the environment to provide those inputs.

    Note

    The value returned by a function assigned to the state_feature_function argument must preserve the shape of its input, and must also accommodate the shape of the inputs to all of the state_input_ports to which it is assigned (see note above).

  • Outcome arguments – these specify the Components, the values of which are assigned to the outcome attribute, and used to compute the net_outcome for a given control_allocation (see Execution). As with a ControlMechanism, these can be sepcified directly in the monitor_for_control argument, or through the use of ObjectiveMechanism specified in the objecctive_mechanism argument (see ControlMechanism_Monitor_for_Control for additional details). However, an OptimizationControlMechanism places some restrictions on the specification of these arguments that, as with specification of state_features, depend on the nature of the agent_rep, as described below.

    • agent_rep is a Composition – the items specified to be monitored for control must belong to the agent_rep, since those are the only ones that will be executed when the evaluate_agent_rep is called; an error will be generated identifying any Components that do not belong to the agent_rep.

    • agent_rep is a CompositionFunctionApproximator – the items specified to be monitored for control can be any within the Composition for which the OptimizationControlMechanism is the controller; this is because their values during the last execution of the Composition are used to determine the net_outcome that the agent_rep's adapt method – if it has one – seeks to predict. Accordingly, the values of the items specified to be monitored control must match, in shape and order, the net_outcome of that adapt method.

  • Optimization arguments – these specify parameters that determine how the OptimizationControlMechanism’s function searches for and determines the optimal control_allocation (see Execution); this includes specification of the num_estimates and num_trials_per_estimate parameters, as well as the random_variables, initial_seed and same_seed_for_all_allocations Parameters, which determine how the net_outcome is estimated for a given control_allocation (see Randomization of Estimation for additional details).

Structure

An OptimizationControlMechanism conforms to the structure of a ControlMechanism, with the following exceptions and additions.

Agent Representation

The defining feature of an OptimizationControlMechanism is its agent representation, specified in the agent_rep argument of its constructor, and assigned to its agent_rep attribute. This designates a representation of the Composition (or parts of it) that the OptimizationControlMechanism uses to evaluate sample control_allocations in order to find one that optimizes the the net_outcome of the Composition when it is fully executed. The agent_rep can be the Composition itself for which the OptimizationControlMechanism is the controller (fully model-based optimization, or another one model-free optimization), that is usually a simpler Composition or a CompositionFunctionApproximator used to estimate the net_outcome for the full Composition (see above). The evaluate method of the agent_rep is assigned as the evaluate_agent_rep method of the OptimizationControlMechanism. If the agent_rep is not the Composition for which the OptimizationControlMechanism is the controller, then it must meet the following requirements:

State

The current state of the OptimizationControlMechanism – or, more properly, of its agent_rep – is determined by the OptimizationControlMechanism’s current state_feature_values (see below) and control_allocation. These are used by the evaluate_agent_rep method, the results of which are combined with the costs associated with the control_allocation, to evaluate the net_outcome for that state. The current state is listed in the OptimizationControlMechanism’s state attribute, and state_dict contains the Components associated with each value of state.

Input

An OptimizationControlMechanism has two types of input_ports, corresponding to the two forms of input it requires: state_input_ports that provide the values of the Components specified as its state_features, and that are used as inputs to the agent_rep when its evaluate method is used to execute it; and outcome_input_ports that provide the outcome of executing the agent_rep, that is used to compute the net_outcome for the control_allocation under which the execution occurred. Each of these is described below.

state_input_ports

The state_input_ports receive Projections from the Components specified as the OptimizationControlMechanism’s state_features, the values of which are assigned as the state_feature_values, and conveyed to the agent_rep's evaluate method when it is executed. The OptimizationControlMechanism has a state_input_port for every specification in the state_features arg of its constructor (see above).

outcome_input_ports

The outcome_input_ports comprise either a single OutputPort that receives a Projection from the OptimizationControlMechanism’s objective_mechanism if has one; or, if it does not, then an OutputPort for each Component it monitors to determine the net_outcome of executing its agent_rep (see outcome arguments for how these are specified). The value(s) of the outcome_input_ports are assigned to the OptimizationControlMechanism’s outcome attribute.

objective_mechanism

If an OptimizationControlMechanism has an objective_mechanism, it is assigned a single outcome_input_port, named OUTCOME, that receives a Projection from the objective_mechanism’s OUTCOME OutputPort. The OptimizationControlMechanism’s objective_mechanism is used to evaluate the outcome of executing its agent_rep for a given state. This passes the result to the OptimizationControlMechanism’s OUTCOME InputPort, that is placed in its outcome attribute.

Note

An OptimizationControlMechanism’s objective_mechanism and the function of that Mechanism, are distinct from and should not be confused with the objective_function parameter of the OptimizationControlMechanism’s function. The objective_mechanism's function evaluates the outcome of processing without taking into account the costs of the OptimizationControlMechanism’s control_signals. In contrast, its evaluate_agent_rep method, which is assigned as the objective_function parameter of its function, takes the costs of the OptimizationControlMechanism’s control_signals into account when calculating the net_outcome that it returns as its result.

monitor_for_control

If an OptimizationControlMechanism is not assigned an objective_mechanism, then its outcome_input_ports are determined by its monitor_for_control and outcome_input_ports_option attributes, specified in the corresponding arguments of its constructor (see Outcomes arguments). The value(s) of the specified Components are assigned as the OptimizationControlMechanism’s outcome attribute, which is used to compute the net_outcome of executing its agent_rep.

Note

If a Node other than an OUTPUT of a nested Composition is specified to be monitored, it is assigned as a PROBE of that nested Composition. Although PROBE Nodes are generally treated like OUTPUT Nodes (since they project out of the Composition to which they belong), their value is not included in the output_values or results attributes of the Composition for which the OptimizationControlMechanism is the controller, unless that Composition’s include_probes_in_output attribute is set to True (see Probes Composition_Probes for additional information).

Function

The function of an OptimizationControlMechanism is used to find the control_allocation that optimizes the net_outcome for the current (or expected) state. It is generally an OptimizationFunction, which in turn has objective_function, search_function and search_termination_function methods, as well as a search_space attribute. The objective_function is automatically assigned the OptimizationControlMechanism’s evaluate_agent_rep method, that is used to evaluate each control_allocation sampled from the search_space by the search_function until the search_termination_function returns True. The net_outcome returned by the call to evaluate_agent_rep is assigned to the OptimizationControlMechanism’s optimal_net_outcome attribute, and the control_allocation that yielded it is assigned to the OptimizationControlMechanism’s optimal_control_allocation attribute (see Execution for additional details).

Custom Function

A custom function can be assigned as the OptimizationControlMechanism’s function, however it must meet the following requirements:

  • It must accept as its first argument and return as its result an array with the same shape as the OptimizationControlMechanism’s control_allocation.

  • It must implement a reset method that can accept as keyword arguments objective_function, search_function, search_termination_function, and search_space, and implement attributes with corresponding names.

Search Function, Search Space and Search Termination Function

Subclasses of OptimizationControlMechanism may implement their own search_function and search_termination_function methods, as well as a control_allocation_search_space attribute, that are passed as parameters to the OptimizationFunction when it is constructed. These can be specified in the constructor for an OptimizationFunction assigned as the function argument in the OptimizationControlMechanism’s constructor, as long as they are compatible with the requirements of the OptimizationFunction and OptimizationControlMechanism. If they are not specified, then defaults specified either by the OptimizationControlMechanism or the OptimizationFunction are used.

Default Function: GridSearch

If the function argument is not specified, the GridSearch OptimizationFunction is assigned as the default, which evaluates the net_outcome using the OptimizationControlMechanism’s control_allocation_search_space as its search_space, and returns the control_allocation that yields the greatest net_outcome, thus implementing a computation of EVC.

Output

The output of OptimizationControlMechanism are its control_signals that implement the control_allocations it evaluates and optimizes. These their effects are estimated over variation in the values of Components with random variables, then the OptimizationControlMechanism’s control_signals include an additional RANDOMIZATION_CONTROL_SIGNAL that implements that variablity for the relevant Components, as described below.

Randomization ControlSignal

If num_estimates is specified (that is, it is not None), and agent_rep has any Components with random variables (that is, that call a randomization function) specified in the OptimizationControlMechanism’s random_variables attribute, then a ControlSignal is automatically added to the OptimizationControlMechanism’s control_signals, named RANDOMIZATION_CONTROL_SIGNAL, that randomizes the values of the random variables over estimates of its net_outcome for each control_allocation If num_estimates is specified but agent_rep has not random variables, then a warning is issued and no RANDOMIZATION_CONTROL_SIGNAL is constructed. The initial_seed and same_seed_for_all_allocations Parameters can also be used to further refine randomization (see Randomization of Estimation for additional details).

The RANDOMIZATION_CONTROL_SIGNAL ControlSignal sends a ControlProjection to the ParameterPort for the see Parameter of Components specified either in the OptimizationControlMechanism’s random_variables attribute or that of the agent_rep (see above). The RANDOMIZATION_CONTROL_SIGNAL is also included when constructing the control_allocation_search_space passed to the constructor for OptimizationControlMechanism’s function, as its search_space argument, along with the index of the RANDOMIZATION_CONTROL_SIGNAL as its randomization_dimension argument.

Execution

When an OptimizationControlMechanism is executed, the OptimizationFunction assigned as it’s function is used evaluate the effects of different control_allocations to find one that optimizes the net_outcome; that control_allocation is then used when the Composition controlled by the OptimizationControlMechanism is next executed. The OptimizationFunction does this by either simulating performance of the Composition or executing the CompositionFunctionApproximator that is its agent_rep.

Timing of Execution

When the OptimizationControlMechanism is executed is determined by the controller_mode of the Composition for which the OptimizationControlMechanism is the controller: if it is set to AFTER (the default), the OptimizationControlMechanism is executed at the end of a TRIAL, after the Composition has executed, using state_feature_value (including any inputs to the Composition) for that TRIAL; if the controller_mode is BEFORE, then the OptimizationControlMechanism is executed before the Composition that it controls, using state_feature_value (including any inputs to the Composition) from the previous TRIAL.

Optimization Procedure

When an OptimizationControlMechanism is executed, it carries out the following steps to find a control_allocation that optmimzes performance of the Composition that it controls:

Randomization of Estimation

If num_estimates is specified (i.e., it is not None), then each control_allocation is independently evaluated num_estimates times (i.e., by that number of calls to the OptimizationControlMechanism’s evaluate_agent_rep method). The values of Components listed in the OptimizationControlMechanism’s random_variables attribute are randomized over those estimates. By default, this includes all Components in the agent_rep with random variables (listed in its random_variables attribute). However, if particular Components are specified in the random_variables argument of the OptimizationControlMechanism’s constructor, then randomization is restricted to their values. Randomization over estimates can be further configured using the initial_seed and same_seed_for_all_allocations attributes. The results of all the estimates for a given control_allocation are aggregated by the aggregation_function of the OptimizationFunction assigned to the OptimizationControlMechanism’s function, and used to compute the net_outcome over the estimates for that control_allocation (see Execution for additional details).

Class Reference

class psyneulink.core.components.mechanisms.modulatory.control.optimizationcontrolmechanism.OptimizationControlMechanism(agent_rep=None, state_features='shadow_inputs', state_feature_default='shadow_inputs', state_feature_function=None, function=None, num_estimates=None, random_variables=None, initial_seed=None, same_seed_for_all_allocations=None, num_trials_per_estimate=None, search_function=None, search_termination_function=None, search_statefulness=None, return_results=False, data=None, context=None, **kwargs)

Subclass of ControlMechanism that adjusts its ControlSignals to optimize performance of the Composition to which it belongs. See ControlMechanism for arguments not described here.

Parameters
agent_rep

determines the Composition used by the evaluate_agent_rep method to predict the net_outcome for a given state; see Agent Representation for additional details.

Type

Composition

agent_rep_type

identifies whether the agent_rep is a Composition, a CompositionFunctionApproximator or one of its subclasses, or it has not been assigned (None) (see Agent Representation and Types of Optimization for additional details).

Type

None, COMPOSITION or COMPOSITION_FUNCTION_APPROXIMATOR

state_features

dictionary in which keys are all external InputPorts for agent_rep, and values are the sources of their input specified in state_features. These are provided as the inputs to state_input_ports, the values of which are assigned to state_feature_values and provided to the agent_rep's evaluate method when it is executed (see state_features and state_input_ports for additional details).

Type

Dict[Node:source]

state_feature_default

determines the default used if the state_feature (i.e. source) is not otherwise specified for the InputPort of an INPUT Node of agent_rep. If it is None, then no corresponding state_input_port is created for that InputPort, and its default variable is used as its input when the agent_rep's evaluate method is executed (see state_features for additional details).

Type

Mechanism, InputPort, OutputPort, Projection, dict, SHADOW_INPUTS, numeric value

state_feature_values

a dict containing the current values assigned as the input to the InputPorts of the INPUT Nodes of the agent_rep when its evaluate method is executed. For each such InputPort, if a state_feature has been specified for it, then its value in state_feature_values is the value of the corresponding state_input_port. There are no entries for InputPorts for which the state_features specification is None or it has not been otherwise specified; for those InputPorts, their default_variable is assigned directly as their input when agent_rep is evaluated (see state_input_ports for additional details).

Type

2d array

state_feature_function

determines the function used as the default function for state_input_ports (see state_feature_function for additional details).

Type

Function of function

state_input_ports

lists the OptimizationControlMechanism’s InputPorts that receive Projections from the items specified in the state_features argument in the OptimizationControlMechanism’s constructor, or constructed automatically (see state_features), the values of which are assigned to state_feature_values and provided as input to the agent_rep method (see state_input_ports for additional details).

Type

ContentAddressableList

num_state_input_ports

contains the number of state_input_ports.

Type

int

outcome_input_ports

lists the OptimizationControlMechanism’s OutputPorts that receive Projections from either its objective_mechanism or the Components listed in its monitor_for_control attribute, the values of which are used to compute the net_outcome of executing the agent_rep in a given State (see objective_mechanism and outcome_input_ports for additional details).

Type

ContentAddressableList

state

lists the values of the current state – a concatenation of the state_feature_values and control_allocation following the last execution of agent_rep.

Type

ndarray

state_dict

dictionary containing information about the Components corresponding to the values in state. Keys are (Port, Mechanism, Composition, index) tuples, identifying the source of the value for each item at the corresponding index in state, and values are its value in state. The initial entries are for the OptimizationControlMechanism’s state features, that are the sources of its state_feature_values; they are followed by entries for the parameters modulated by the OptimizationControlMechanism’s control_signals with the corresponding control_allocation values.

Type

Dict[(Port, Mechanism, Composition, index)):value]

num_estimates

determines the number independent runs of agent_rep (i.e., calls to evaluate_agent_rep) used to estimate the net_outcome of each control_allocation evaluated by the OptimizationControlMechanism’s function (i.e., that are specified by its search_space); see Randomization of Estimation for additional details.

Type

int

random_variables

list of the Parameters in agent_rep with random variables (that is, ones that call a randomization function) that are randomized over estimates for a given control_allocation; by default, all Components in the agent_rep with random variables are included (listed in its random_variables attribute); see Randomization of Estimation for additional details.

Type

Parameter or List[Parameter]

initial_seed

determines the seed used to initialize the random number generator at construction. If it is not specified then then the seed is set to a random value, and different runs of a Composition containing the OptimizationControlMechanism will yield different results, which should be roughly comparable if the estimation process is stable. If initial_seed is specified, then running the Composition should yield identical results for the estimation process, which can be useful for debugging.

Type

int or None

same_seed_for_all_allocations

determines whether the random number generator used to select seeds for each estimate of the agent_rep's net_outcome is re-initialized to the same value for each control_allocation evaluated. If same_seed_for_all_allocations is True, then any differences in the estimates made of net_outcome for each control_allocation will reflect exclusively the influence of the different control_allocations on the execution of the agent_rep, and not any variability intrinsic to the execution of the Composition itself (e.g., any of its Components). This can be confirmed by identical results for repeated executions of the OptimizationControlMechanism’s evaluate_agent_rep method for the same control_allocation. If same_seed_for_all_allocations is False, then each time a control_allocation is estimated, it will use a different set of seeds. This can be confirmed by differing results for repeated executions of the OptimizationControlMechanism’s evaluate_agent_rep method with the same control_allocation). Small differences in results suggest stability of the estimation process across control_allocations, while substantial differences indicate instability, which may be helped by increasing num_estimates.

Type

bool

num_trials_per_estimate

imposes an exact number of trials to execute in each run of agent_rep used to evaluate its net_outcome by a call to the OptimizationControlMechanism’s evaluate_agent_rep method. If it is None (the default), then either the number of inputs or the value specified for num_trials in the Composition’s run method used to determine the number of trials executed (see number of trials for additional information).

Type

int or None

function

takes current control_allocation (as initializer), uses its search_function to select samples of control_allocation from its search_space, evaluates these using its evaluate_agent_rep method by calling it num_estimates times to estimate its net_outcome `net_outcome for a given control_allocation, and returns the one that yields the optimal net_outcome (see Function for additional details).

Type

OptimizationFunction, function or method

evaluate_agent_rep

returns the net_outcome(s) for a given state (i.e., combination of state_feature_values and control_allocation). It is assigned as the objective_function parameter of function, and calls the evaluate method of the OptimizationControlMechanism’s agent_rep with the current state_feature_values and a specified control_allocation, which runs of the agent_rep for num_trials_per_estimate trials. It returns an array containing the net_outcome of the run and, if the return_results argument is True, an array containing the results of the run. This method is num_estimates times by the OptimizationControlMechanism’s function, which aggregates the net_outcome over those in evaluating a given control_allocation (see Function for additional details).

Type

function or method

search_function

search_function assigned to function; used to select samples of control_allocation to evaluate by evaluate_agent_rep.

Type

function or method

search_termination_function

search_termination_function assigned to function; determines when to terminate the optimization process.

Type

function or method

control_signals

list of the ControlSignals for the OptimizationControlMechanism for the Parameters being optimized by the OptimizationControlMechanism, including any inherited from the Composition for which it is the controller (this is the same as ControlMechanism’s output_ports attribute). Each sends a ControlProjection to the ParameterPort for the Parameter it controls when evaluating a control_allocation. If num_estimates is specified (that is, it is not None), a ControlSignal is added to control_signals, named RANDOMIZATION_CONTROL_SIGNAL, that is used to randomize estimates of outcome for a given control_allocation (see Randomization of Estimation for details.)

Type

ContentAddressableList[ControlSignal]

control_allocation_search_space

search_space assigned by default to the OptimizationControlMechanism’s function, that determines the samples of control_allocation evaluated by the evaluate_agent_rep method. This is a property that, unless overridden, returns a list of the SampleIterators generated from the allocation_samples specifications for each of the OptimizationControlMechanism’s control_signals, and includes the RANDOMIZATION_CONTROL_SIGNAL used to randomize estimates of each control_allocation (see note above).

Type

list of SampleIterators

optimal_control_allocation

the control_allocation that yielded the optimal net_outcome in call to evaluate_agent_rep.

Type

1d array

optimal_net_outcome

the net_outcome for the optimal_control_allocation returned by call to evaluate_agent_rep.

Type

float

saved_samples

contains all values of control_allocation sampled by function if its save_samples parameter is True; otherwise list is empty.

Type

list

saved_values

contains values of net_outcome associated with all samples of control_allocation evaluated by by function if its save_values parameter is True; otherwise list is empty.

Type

list

search_statefulness

if True (the default), calls to evaluate_agent_rep by the OptimizationControlMechanism’s function for each control_allocation will run as simulations in their own execution contexts. If search_statefulness is False, calls for each control_allocation will not be executed as independent simulations; rather, all will be run in the same (original) execution context.

Type

bool : True

value

the optimal_control_allocation returned by the OptimizationFunction assigned as the OptimizationControlMechanism’s function, which is the net_outcome of the agent_rep.

This uses only the first value returned by the OptimizationFunction which also may return the value associated with the optimal_control_allocation as well as the full set of control_allocations and corresponding values (if the save_samples and/or save_values arguments of the OptimizationControlMechanism’s constructor are True); these are stored in the OptimizationControlMechanism’s saved_samples and saved_values attributes, respectively.

Type

2d np.array

return_results

if True, the complete simulation results are returned when invoking evaluate_agent_rep calls. This is nescessary when using a ParameterEstimationCompostion for parameter estimation via data fitting.

Type

bool : False

_validate_params(request_set, target_set=None, context=None)

Insure that specification of ObjectiveMechanism has projections to it

_instantiate_input_ports(context=None)

Instantiate InputPorts for state_features (with state_feature_function if specified).

This instantiates the OptimizationControlMechanism’s state_input_ports;

these are used to provide input to the agent_rep when its evaluate method is called

The OptimizationControlMechanism’s outcome_input_ports are instantiated by

ControlMechanism._instantiate_input_ports in the call to super().

InputPorts are constructed for state_features by calling _parse_state_feature_specs()

with them and state_feature_function arguments of the OptimizationControlMechanism constructor.

The constructed state_input_ports are passed to ControlMechanism_instantiate_input_ports(),

which appends them to the InputPort(s) that receive input from the objective_mechanism* (if specified) or **monitor_for_control ports (if objective_mechanism is not specified).

Also ensures that:

  • every state_input_port has only a single Projection;

  • every outcome_input_ports receive Projections from within the agent_rep if it is a Composition.

If no state_features are specified in the constructor, assign ones for INPUT Nodes of owner.

  • warn for use of CompositionFunctionApproximator as agent_rep;

  • ignore here for Composition as agent_rep (handled in _update_state_input_ports_for_controller).

See state_features and state_input_ports for additional details.

_validate_input_nodes(nodes, enforce=None)

Check that nodes are INPUT Nodes of agent_rep INPUT Nodes are those at the top level of agent_rep as well as those of any Compositions nested within it

that are themselves INPUT Nodes of their enclosing Composition.

Raise exception for non-INPUT Nodes if enforce is specified; otherwise just issue warning.

_parse_state_feature_specs(context=None)

Parse entries of state_features specifications used to construct state_input_ports.

Called from _instantiate_input_ports()

Parse state_features arg of constructor for OptimizationControlMechanism, assigned to state_feature_specs.

state_feature_specs lists sources of inputs to all INPUT Nodes of agent_rep, at all levels of nesting; there

is one entry for every INPUT Node in agent_rep, and every INPUT Node of any nested Composition that is itself an INPUT Node at any level of nesting.

Construct a state_input_port for every entry in state_feature_specs that is not None:
the value of those state_input_ports comprise the state_feature_values attribute, and are provided as the

input to the INPUT Nodes of agent_rep when its evaluate() method is executed (as the predicted_inputs argument if agent_rep is a Composition, and the feature_values argument if it is a CompositionFunctionApproximator); for INPUT;

for None entries in state_feature_specs, the corresponding INPUT Nodes are provided their

default_external_input_shape as their input when agent_rep.evaluate() executes.

Projection(s) to state_input_ports from sources specified in state_feature_specs can be direct,

or indirect by way of a CIM if the source is in a nested Composition.

Handle four formats:

  • dict {INPUT Node: source or None, INPUT Node or InputPort: source or None…}:
    • every key must be an INPUT Node of agent_rep or an INPUT Node of a nested Composition within it that is

      itself an INPUT Node of its enclosing Composition, or the external InputPort of one, at any level of nesting;

    • if a Mechanism is specified as a key, construct a state_input_port for each of its external InputPorts,

      and assign the value of the dict entry as the source for all of them;

    • if a Composition is specified as a key, construct a state_input_port for each external InputPort of each

      of its INPUT Nodes, and those of any Compositions nested within it at all levels of nesting, and assign the the value of the dict entry as the source for all of them;

    • for INPUT Nodes not specified or assigned None as their value, assign corresponding entries in

      state_feature_specs as state_feature_default

    • if only one or some of the INPUT Nodes of a nested Composition are specified,

      for the remaining ones assign the corresponding entries in state_feature_specs as state_feature_default

    • if None is specified, don’t construct a state_input_port

  • list [source, None, source…]: specifies source specs for INPUT Node external InputPorts:

    • must be listed in same order as expanded list of agent_rep INPUT Node external InputPorts to which they correspond (i.e., nested Compositions that are INPUT Nodes replaced by their INPUT Nodes, for all levels of nesting);

    • if there are fewer sources listed than INPUT Node external InputPorts, assign state_feature_default to

      the entries in state_feature_specs corresponding to the remaining INPUT Node external InputPorts

    • if there more sources listed than INPUT Nodes, leave the excess ones, and label them as

      ‘EXPECT <specified INPUT Node InputPort name>’ for later resolution (see below).

  • set {INPUT Node, Input Node…}: specifies INPUT Nodes to be shadowed
    • every item must be an INPUT Node of agent_rep or an INPUT Node of a nested Composition within it that

      is itself an INPUT Node of its enclosing Composition, at any level of nesting;

    • if a Composition is specified, construct a state_input_port for each of its INPUT Node extenal InputPorts,

      and those of any Compositions nested within it at all levels of nesting, each of which shadows the input of the corresponding INPUT Node (see _InputPort_Shadow_Inputs).

    • if only one or some of the INPUT Nodes of a nested Composition are specified, use state_feature_default.

IMPLEMENTATION NOTE: this is a legacy format for consistency with generic specification of shadowing inputs - SHADOW_INPUTS dict {“SHADOW_INPUTS”:[shadowable input, None, shadowable input…]}:

  • all items must be a Mechanism (or one of its external InputPorts) that is an INPUT Node of agent_rep or

    of a nested Composition within it that is itself an INPUT Node;

  • must be listed in same order as expanded list of agent_rep INPUT Nodes to which they correspond

    (see list format above);

  • construct a state_input_port for each non-None spec, and assign it a Projection that shadows the spec.

    (see _InputPort_Shadow_Inputs).

If shadowing is specified for an INPUT Node InputPort, set INTERNAL_ONLY to True in entry of params dict in

specification dictionary for corresponding state_input_port (so that inputs to Composition are not required if the specified source is itself an INPUT Node).

If an INPUT Node (or one of its external InputPorts) is specified that is not (yet) in agent_rep,

and/or a source is specified that is not yet in self.composition, warn and defer creating a state_input_port; final check is made, and error(s) generated for unresolved specifications at run time.

Assign functions specified in state_feature_function to InputPorts for all state_features

Return list of InputPort specification dictionaries for state_input_ports

_assign_state_feature_function(specification_dict, idx=None)

Assign any specified state_feature_function to corresponding state_input_ports idx is index into self._state_feature_functions; if None, use self.state_feature_function specified by user Specification in InputPort specification dictionary or state_features tuple

takes precedence over state_feature_function specification.

Assignment of function to dict specs handled above, so skip here Return state_input_port_dicts with FUNCTION entries added as appropriate.

_update_state_input_ports_for_controller(context=None)

Check and update state_input_ports at run time if agent_rep is a Composition

If no agent_rep has been specified or it is a CompositionFunctionApproximator, return

(note: validation of state_features specified for CompositionFunctionApproximator optimization is up to the CompositionFunctionApproximator)

If agent_rep is a Composition:
  • if has any new INPUT Node InputPorts:

    • construct state_input_ports for them

    • add to _specified_INPUT_Node_InputPorts_in_order

  • call _validate_state_features()

  • call _update_state_input_port_names()

_update_state_input_port_names(context=None)

Update names of state_input_port for any newly instantiated INPUT Node InputPorts

If its instantiation has NOT been DEFERRED, assert that:

  • corresponding agent_rep INPUT Node InputPort is in Composition

  • state_input_port either has path_afferents or it is for a numeric spec

If it’s instantiation HAS been DEFERRED, for any newly added agent_rep INPUT Node InputPorts:

  • add agent_rep INPUT Node InputPort to _specified_INPUT_Node_InputPorts_in_order

  • if state_input_port:

    • HAS path_afferents, get source and generate new name

    • does NOT have path_afferents, assert it is for a numeric spec and generate new name

  • assign new name

_validate_state_features(context)

Validate that state_features are legal and consistent with agent_rep.

Called by _update_state_input_ports_for_controller, - after new Nodes have been added to Composition - and/or in run() as final check before execution.

Ensure that: - the number of state_feature_specs equals the number of external InputPorts of INPUT Nodes of agent_rep; - if state_feature_specs are specified as a user dict, keys are valid INPUT Nodes of agent_rep; - all InputPorts shadowed by specified state_input_ports are in agent_rep or one of its nested Compositions; - any Projections received from output_ports are from Nodes in agent_rep or its nested Compositions; - all InputPorts shadowed by state_input_ports reference INPUT Nodes of agent_rep or Compositions nested in it; - state_features are compatible with input format for agent_rep Composition

_validate_monitor_for_control(nodes)

Ensure all of the Components being monitored for control are in the Composition being controlled If monitor_for_control is specified as an ObjectiveMechanism, warn and move to objective_mecahnism arg

_instantiate_output_ports(context=None)

Assign CostFunctions.DEFAULTS as default for cost_option of ControlSignals.

_instantiate_control_signals(context)

Subclasses can override for class-specific implementation (see OptimizationControlMechanism for example)

_set_mechanism_value(context)

Set Mechanism’s value from control_allocation. OCM uses optimal_control_allocation (returned by its _execute() method), which is isomorphic to self.control_allocation, as its value. This is needed because the OCM’s:

  • function (an OptimizationFunction) can return additional information (e.g., GridSearch)

  • _execute() method processes the value returned by the OptimizationFunction (to incorporate costs)

_instantiate_function(function, function_params=None, context=None)

Assign weights and exponents if specified in input_ports

_instantiate_attributes_after_function(context=None)

Instantiate OptimizationControlMechanism’s OptimizationFunction attributes

_execute(variable=None, context=None, runtime_params=None)

Return control_allocation that optimizes net_outcome of agent_rep.evaluate().

Return type

ndarray

evaluate_agent_rep(control_allocation, context=None)

Call evaluate method of agent_rep

Assigned as the objective_function for the OptimizationControlMechanism’s function.

Evaluates agent_rep by calling its evaluate method, which executes its agent_rep using the current state_feature_values as the input and the specified control_allocation.

If the agent_rep is a Composition, each execution is a call to its run method that uses the num_trials_per_estimate as its num_trials argument, and the same state_feature_values and control_allocation but a different randomly chosen seed for the random number generator for each run. It then returns an array of length number_estimates containing the net_outcome of each execution and, if return_results is True, also an array with the results of each run.

If the agent_rep is a CompositionFunctionApproximator, then num_estimates is passed to it to handle execution and estimation as determined by its implementation, and returns a single estimated net_outcome.

(See evaluate for additional details.)

_apply_control_allocation(control_allocation, runtime_params, context)

Update values to control_signals based on specified control_allocation

_gen_llvm_function(*, ctx, tags)

Overloaded main function LLVM generation method.

Mechanisms need to support “is_finished” execution variant (used by scheduling conditions) on top of the variants supported by Component.

property state_features

source name} for all state_features. If state_feature_spec is numeric for a Node, assign its value as the source If existing INPUT Node is not specified in state_feature_specs, assign state_feature_default as source If an InputPort is referenced in state_feature_specs that is not yet in agent_rep,

assign “DEFERRED INPUT NODE <InputPort name> OF <agent_rep>” as key for the entry; (it should be resolved by runtime, or an error is generated).

If a state_feature_spec is referenced that is not yet in ocm.composition,

assign “<InputPort name> NOT (YET) IN <agent_rep>” as the value of the entry; (it should be resolved by runtime, or an error is generated).

Type

Return {InputPort name

property state

Array that is concatenation of state_feature_values and control_allocations

property state_distal_sources_and_destinations_dict

Return dict with (Port, Node, Composition, index) tuples as keys and corresponding state[index] as values. Initial entries are for sources of the state_feature_values (i.e., distal afferents for state_input_ports) and subsequent entries are for destination parameters modulated by the OptimizationControlMechanism’s ControlSignals (i.e., distal efferents of its ControlProjections). Note: the index is required, since a state_input_port may have more than one afferent Projection

(that is, a state_feature_value may be determined by Projections from more than one source), and a ControlSignal may have more than one ControlProjection (that is, a given element of the control_allocation may apply to more than one Parameter). However, for state_input_ports that shadow a Node[InputPort], only that Node[InputPort] is listed in state_dict even if the Node[InputPort] being shadowed has more than one afferent Projection (this is because it is the value of the Node[InputPort] (after it has processed the value of its afferent Projections) that determines the input to the state_input_port.

property state_feature_sources

source} for all INPUT Nodes of agent_rep, and sources in state_feature_specs. Used by state_distal_sources_and_destinations_dict()

Type

Dict with {InputPort

property _model_spec_parameter_blacklist

A set of Parameter names that should not be added to the generated constructor string

_initialize_composition_function_approximator(context)

Initialize CompositionFunctionApproximator