# OptimizationControlMechanism¶

## Overview¶

An OptimizationControlMechanism is a ControlMechanism that uses an OptimizationFunction to find an optimal control_allocation for a given state. The OptimizationFunction uses the OptimizationControlMechanism’s evaluation_function <OptimizationControlMechanism.evalutate_function> to evaluate control_allocation samples, and then implements the one that yields the best predicted result. The result returned by the evaluation_function <OptimizationControlMechanism.evalutate_function> is ordinally the net_outcome computed by the OptimizationControlMechanism for the Composition (or part of one) that it controls, and its ObjectiveFunction seeks to maximize this, which corresponds to maximizing the Expected Value of Control, as described below.

Expected Value of Control

The net_outcome of an OptmizationControlMechanism, like any Control Mechanisms is computed as the difference between the outcome computed by its objective_mechanism and the costs of its control_signals for a given state (i.e., set of feature_values and control_allocation. If the outcome is configured to measure the value of processing (e.g., reward received, time taken to respond, or a combination of these, etc.), and the OptimizationFunction assigned as the OptimizationControlMechanism’s function is configured find the control_allocation that maximizes its net_outcome, then the OptimizationControlMechanism is said to be maximizing the Expected Value of Control (EVC). That is, it implements a cost-benefit analysis that weighs the costs of the ControlSignal values specified by a control_allocation against the outcome expected to result from it. The costs are computed based on the cost_options specified for each of the OptimizationControlMechanism’s control_signals and its combine_costs function. The EVC is determined by its compute_net_outcome function (assigned to its net_outcome attribute), which is computed for a given state by the OptimizationControlMechanism’s evaluation_function.

Agent Representation and Types of Optimization

The defining characteristic of an OptimizationControlMechanism is its agent representation, that is used to determine the net_outcome and find the control_allocation that optimizes this. The agent_rep can be either the Composition to which the OptimizationControlMechanism belongs (and controls) or another one that is used to estimate the net_outcome for that Composition. This distinction corresponds closely to the distinction between model-based and model-free optimization in the machine learning and cognitive neuroscience literatures, as described below.

Model-Free Optimization

This is achieved by assigning as the agent_rep a Composition other than the one to which the OptimizationControlMechanism belongs (and for which it is the controller). In each trial, the agent_rep is given the chance to adapt, by adjusting its parameters in order to improve its prediction of the net_outcome for the Composition (or part of one) that is controlled by the OptimizationControlMechanism (based on the state and net_outcome of the prior trial). The agent_rep is then used to predict the net_outcome for control_allocation samples to find the one that yields the best predicted net_outcome of processing on the upcoming trial, based on the current or (expected) feature_values for that trial.

Model-Based Optimization

This is achieved by assigning as the agent_rep the Composition to which the OptimizationControlMechanism belongs (and for which it is the controller). On each trial, that Composition itself is used to simulate processing on the upcoming trial, based on the current or (expected) feature_values for that trial, in order to find the <ControlMechanism.control_allocation> that yields the best net_outcome <ControlMechanism.net_outcome> for that trial.

## Creating an OptimizationControlMechanism¶

An OptimizationControlMechanism is created in the same was as any ControlMechanism. The following arguments of its constructor are specific to the OptimizationControlMechanism:

• features – takes the place of the standard input_states argument in the constructor for a Mechanism, and specifies the values used by the OptimizationControlMechanism, together with a control_allocation, to calculate a net_outcome. Features can be specified using any of the following, singly or combined in a list:

• InputState specification – this can be any form of InputState specification that resolves to an OutputState from which the InputState receives a Projection; the value of that OutputState is used as the feature. Each of these InputStates is marked as internal_only = True.

Features can also be added to an existing OptimizationControlMechanism using its add_features method. If the features argument is not specified, then the input to the Composition on the last trial of its execution is used to predict the net_outcome for the upcoming trial.

• feature_function – specifies function of the InputState created for each item listed in features. By default, this is the identity function, that assigns the current value of the feature to the OptimizationControlMechanism’s feature_values attribute. However, other functions can be assigned, for example to maintain a record of past values, or integrate them over trials.
• agent_rep – specifies the Composition used by the OptimizationControlMechanism’s evaluation_function to calculate the predicted net_outcome for a given state (see below for additional details). If it is not specified, the Composition to which the OptimizationControlMechanism belongs is assigned, and the OptimizationControlMechanism is assigned as that Composition’s controller, implementing model-based optimization. If that Composition already has a controller specified, the OptimizationControlMechanism is disable. If another Composition is specified, it must conform to the specifications for an agent_rep as described below.

## Structure¶

In addition to the standard Components associated with a Control Mechanisms, including a Projection to its OUTCOME InputState from its objective_mechanism, and a function used to carry out the optimization process, it has several other constiuents, as described below.

### ObjectiveMechanism¶

Like any Control Mechanisms, an OptimizationControlMechanism has an associated objective_mechanism that is used to evaluate the outcome of processing for a given trial and pass the result to the OptimizationControlMechanism, which it places in its outcome attribute. This is used by its compute_net_outcome function, together with the costs of its control_signals, to compute the net_outcome of processing for a given state, and that is returned by evaluation method of the OptimizationControlMechanism’s agent_rep.

Note

The objective_mechanism is distinct from, and should not be confused with the objective_function parameter of the OptimizationControlMechanism’s function. The objective_mechanism evaluates the outcome of processing without taking into account the costs of the OptimizationControlMechanism’s control_signals. In contrast, its evaluation_function, which is assigned as the objective_function parameter of its function, takes the costs of the OptimizationControlMechanism’s control_signals into account when calculating the net_outcome that it returns as its result.

### Features¶

In addition to its primary InputState (which receives a projection from the OUTCOME OutpuState of the objective_mechanism, an OptimizationControlMechanism also has an InputState for each of its features. By default, these are the current input for the Composition to which the OptimizationControlMechanism belongs. However, different values can be specified, as can a feature_function that transforms these. For OptimizationControlMechanisms that implement model-free optimization, its feature_values are used by its evaluation_function to predict the net_outcome for a given control_allocation. For OptimizationControlMechanisms that implement model-based optimization, the feature_values are used as the Composition’s input when it is executed to evaluate the net_outcome for a given control_allocation.

Features can be of two types:

• Input Features – these are values received as input by other Mechanisms in the Composition. They are specified as shadowed inputs in the features argument of the OptimizationControlMechanism’s constructor (see Creating an OptimizationControlMechanism). An InputState is created on the OptimziationControlMechanism for each feature, that receives a Projections paralleling the input to be shadowed.
• Output Features – these are the value of an OutputState of some other Mechanisms in the Composition. These too are specified in the features argument of the OptimizationControlMechanism’s constructor (see Creating an OptimizationControlMechanism), and each is assigned a Projections from the specified OutputState(s) to the InputState of the OptimizationControlMechanism for that feature.

The current value of the InputStates for the features are listed in the feature_values attribute.

### State¶

The state of the Composition (or part of one) controlled by an OptimizationControlMechanism is defined by a combination of feature_values (see above) and a control_allocation.

### Agent Representation¶

The defining feature of an OptimizationControlMechanism is its agent representation, specified in the agent_rep argument of its constructor and assigned to its agent_rep attribute. This designates a representation of the Composition (or parts of one) that the OptimizationControlMechanism controls, that is used to evaluate sample control_allocations in order to find the one that optimizes the net_outcome. The agent_rep is always itself a Composition, that can be either the same one that the OptimizationControlMechanism controls or another one that is used to estimate the net_outcome for that Composition (see above). The evaluate method of the Composition is assigned as the evaluation_function of the OptimizationControlMechanism. If the agent_rep is not the Composition for which the OptimizationControlMechanism is the controller, then it must meet the following requirements:

• Its evaluate method must accept as its first three arguments, in order, values that correspond in shape to the feature_values, control_allocation and num_estimates attributes of the OptimizationControlMechanism, respectively.
• If it has an adapt method, that must accept as its first three arguments, in order, values that corresopnd to the shape of the feature_values, control_allocation and net_outcome attributes of the OptimizationControlMechanism. respectively.

### Function¶

The function of an OptimizationControlMechanism is used to find the control_allocation that optimizes the net_outcome for the current (or expected) features. It is generally an OptimizationFunction, which in turn has objective_function, search_function and search_termination_function methods, as well as a search_space attribute. The OptimizationControlMechanism’s evaluation_function is automatically assigned as the OptimizationFunction’s objective_function, and is used to evaluate each control_allocation sampled from the search_space by the search_function search_function until the search_termination_function returns True. A custom function can be assigned as the function of an OptimizationControlMechanism, however it must meet the following requirements:

• It must accept as its first argument and return as its result an array with the same shape as the OptimizationControlMechanism’s control_allocation.
• It must implement a reinitialize method that accepts objective_function as a keyword argument and implements an attribute with the same name.

If function argument is not specified, the GridSearch OptimiziationFunction is assigned as the default, which evaluates the net_outcome using the OptimizationControlMechanism’s control_allocation_search_space as its search_space, and returns the control_allocation that yields the greatest net_outcome, thus implementing a computation of EVC.

## Execution¶

When an OptimizationControlMechanism is executed, it carries out the following steps:

• Calls adapt method of its agent_rep to give that a chance to modify its parameters in order to better predict the net_outcome for a given state, based the state and net_outcome of the previous trial.

## Class Reference¶

class psyneulink.core.components.mechanisms.adaptive.control.optimizationcontrolmechanism.OptimizationControlMechanism(objective_mechanism=None, monitor_for_control=None, objective_mechanism=None, origin_objective_mechanism=False terminal_objective_mechanism=False features=None, feature_function=None, function=None, agent_rep=None, search_function=None, search_termination_function=None, search_space=None, control_signals=None, modulation=ModulationParam.MULTIPLICATIVE, combine_costs=np.sum, compute_reconfiguration_cost=None, compute_net_outcome=lambda x, y:x-y, params=None, name=None, prefs=None)

Subclass of ControlMechanism that adjusts its ControlSignals to optimize performance of the Composition to which it belongs

Parameters: objective_mechanism (ObjectiveMechanism or List[OutputState specification]) – specifies either an ObjectiveMechanism to use for the OptimizationControlMechanism, or a list of the OutputStates it should monitor; if a list of OutputState specifications is used, a default ObjectiveMechanism is created and the list is passed to its monitored_output_states argument. features (Mechanism, OutputState, Projection, dict, or list containing any of these) – specifies Components, the values of which are assigned to feature_values and used to predict net_outcome. Any InputState specification can be used that resolves to an OutputState that projects to the InputState. feature_function (Function or function : default None) – specifies the function for the InputState assigned to each feature. agent_rep (Composition : default Composition to which the OptimizationControlMechanism belongs) – specifies the Composition used by the evalution_function to predictd the net_outcome for a given state. If a Composition other than the default is assigned, it must be suitably configured (see above for additional details). If the default is used, the OptimizationControlMechanism is assigned as the Composition’s controller unless one has already been assigned, in which case the OptimizationControlMechanismit is disabled. search_function (function or method) – specifies the function assigned to function as its search_function parameter, unless that is specified in a constructor for function. It must take as its arguments an array with the same shape as control_allocation and an integer (indicating the iteration of the optimization process), and return an array with the same shape as control_allocation. search_termination_function (function or method) – specifies the function assigned to function as its search_termination_function parameter, unless that is specified in a constructor for function. It must take as its arguments an array with the same shape as control_allocation and two integers (the first representing the net_outcome for the current control_allocation, and the second the current iteration of the optimization process); it must return True or False. search_space (list or ndarray) – specifies the search_space parameter for function, unless that is specified in a constructor for function. Each item must have the same shape as control_allocation. function (OptimizationFunction, function or method) – specifies the function used to optimize the control_allocation; must take as its sole argument an array with the same shape as control_allocation, and return a similar array (see Function for additional details). params (Dict[param keyword: param value] : default None) – a parameter dictionary that can be used to specify the parameters for the OptimizationControlMechanism, its function, and/or a custom function and its parameters. Values specified for parameters in the dictionary override any assigned to those parameters in arguments of the constructor. name (str : default see name) – specifies the name of the OptimizationControlMechanism. prefs (PreferenceSet or specification dict : default Mechanism.classPreferences) – specifies the PreferenceSet for the OptimizationControlMechanism; see prefs for details.
feature_values

2d array – the current value of each of the OptimizationControlMechanism’s features (each of which is a 1d array).

agent_rep

Composition – determines the Composition used by the evalution_function to predict the net_outcome for a given state (see above <OptimizationControlMechanism_Agent_Rep>for additional details).

function

OptimizationFunction, function or method – takes current control_allocation (as initializer), uses its search_function to select samples of control_allocation from its search_space, evaluates these using its evaluation_function, and returns the one that yields the optimal net_outcome (see Function for additional details).

evaluation_function

function or method – returns net_outcome for a given state (i.e., combination of feature_values and control_allocation. It is assigned as the objective_function parameter of function, and calls the evaluate method of the OptimizationControlMechanism’s agent_rep with a control_allocation, the OptimizationControlMechanism’s num_estimates attribute, and the current feature_values.

control_allocation_search_space

list of SampleIteratorssearch_space assigned by default to function, that determines the samples of control_allocation evaluated by the evaluation_function. This is a proprety that, unless overridden, returns a list of the SampleIterators generated from the allocation_sample specifications for each of the OptimizationControlMechanism’s control_signals.

saved_samples

list – contains all values of control_allocation sampled by function if its save_samples parameter is True; otherwise list is empty.

saved_values

list – contains values of net_outcome associated with all samples of control_allocation evaluated by by function if its save_values parameter is True; otherwise list is empty.

name

str – name of the OptimizationControlMechanism; if it is not specified in the name argument of the constructor, a default is assigned by MechanismRegistry (see Naming for conventions used for default and duplicate names).

prefs

PreferenceSet or specification dict – the PreferenceSet for the OptimizationControlMechanism; if it is not specified in the prefs argument of the constructor, a default is assigned using classPreferences defined in __init__.py (see PreferenceSet for details).

evaluation_function(control_allocation, execution_id=None)

Compute net_outcome for current set of feature_values and a specified control_allocation.

Assigned as the objective_function for the OptimizationControlMechanism’s function.

Calls agent_rep’s evalute method.

Returns a scalar that is the predicted net_outcome for the current feature_values and specified control_allocation.

apply_control_allocation(control_allocation, runtime_params, context, execution_id=None)

Update values of control_signals based on specified control_allocation.

Called by evaluate method of Composition when it is assigned as agent_rep.