# OptimizationControlMechanism¶

## Overview¶

An OptimizationControlMechanism is a ControlMechanism that uses an OptimizationFunction to find an optimal control_allocation for a given state. The OptimizationFunction uses the OptimizationControlMechanism’s evaluation_function <OptimizationControlMechanism.evalutate_function> to evaluate control_allocation samples, and then implements the one that yields the best predicted result. The result returned by the evaluation_function <OptimizationControlMechanism.evalutate_function> is ordinarily the net_outcome computed by the OptimizationControlMechanism for the Composition (or part of one) that it controls, and its ObjectiveFunction seeks to maximize this, which corresponds to maximizing the Expected Value of Control, as described below.

Expected Value of Control

The net_outcome of an OptimizationControlMechanism, like any ControlMechanism is computed as the difference between the outcome computed by its objective_mechanism and the costs of its control_signals for a given state (i.e., set of state_feature_values and control_allocation. If the outcome is configured to measure the value of processing (e.g., reward received, time taken to respond, or a combination of these, etc.), and the OptimizationFunction assigned as the OptimizationControlMechanism’s function is configured find the control_allocation that maximizes its net_outcome, then the OptimizationControlMechanism is said to be maximizing the Expected Value of Control (EVC). That is, it implements a cost-benefit analysis that weighs the costs of the ControlSignal values specified by a control_allocation against the outcome expected to result from it. The costs are computed based on the cost_options specified for each of the OptimizationControlMechanism’s control_signals and its combine_costs function. The EVC is determined by its compute_net_outcome function (assigned to its net_outcome attribute), which is computed for a given state by the OptimizationControlMechanism’s evaluation_function.

Agent Representation and Types of Optimization

The defining characteristic of an OptimizationControlMechanism is its agent representation, that is used to determine the net_outcome and find the control_allocation that optimizes this. The agent_rep can be either the Composition to which the OptimizationControlMechanism belongs (and controls) or another one that is used to estimate the net_outcome for that Composition. This distinction corresponds closely to the distinction between model-based and model-free optimization in the machine learning and cognitive neuroscience literatures, as described below.

“Model-Free” Optimization

Note

The term “model-free” is used here, but placed in “apology quotes,” to reflect the fact that, while this term is used widely (e.g., in machine learning and cognitive science) to distinguish it from “agent_rep-based” forms of processing, “model-free” processing nevertheless relies on some form of agent_rep – albeit a much simpler one – for learning, planning and decision making. Here, the distinction is between the use of the most complete form of agent_rep (referred to as “agent_rep-based”), which the agent (i.e., Composition itself) serves as the “agent_rep,” and simpler forms of models that (can learn to) approximate the agent’s behavior (e.g., reinforcement learning algorithms or other forms of function approximation) that can be assigned as the OptimizationControlMechanism’s agent_rep.

This is implemented by assigning as the agent_rep a Composition other than the one to which the OptimizationControlMechanism belongs (and for which it is the controller). In each TRIAL, the agent_rep is given the chance to adapt, by adjusting its parameters in order to improve its prediction of the net_outcome for the Composition (or part of one) that is controlled by the OptimizationControlMechanism (based on the state and net_outcome of the prior trial). The agent_rep is then used to predict the net_outcome for control_allocation samples to find the one that yields the best predicted net_outcome of processing on the upcoming trial, based on the current or (expected) state_feature_values for that trial.

Model-Based Optimization

This is achieved by assigning as the agent_rep the Composition to which the OptimizationControlMechanism belongs (and for which it is the controller). On each TRIAL, that Composition itself is used to simulate processing on the upcoming trial, based on the current or (expected) state_feature_values for that trial, in order to find the control_allocation that yields the best net_outcome for that trial.

## Creating an OptimizationControlMechanism¶

An OptimizationControlMechanism is created in the same was as any ControlMechanism. The following arguments of its constructor are specific to the OptimizationControlMechanism:

• state_feature_function – specifies function of the InputPort created for each item listed in state_features. By default, this is the identity function, that assigns the current value of the feature to the OptimizationControlMechanism’s state_feature_values.

## Structure¶

In addition to the standard Components associated with a ControlMechanism, including a Projection to its OUTCOME InputPort from its objective_mechanism, and a function used to carry out the optimization process, it has several other constiuents, as described below.

### ObjectiveMechanism¶

Like any ControlMechanism, an OptimizationControlMechanism may be assigned an objective_mechanism that is used to evaluate the outcome of processing for a given trial (see ControlMechanism_Objective_ObjectiveMechanism). This passes the result to the OptimizationControlMechanism, which it places in its outcome attribute. This is used by its compute_net_outcome function, together with the costs of its control_signals, to compute the net_outcome of processing for a given state, and that is returned by evaluation method of the OptimizationControlMechanism’s agent_rep.

Note

The objective_mechanism is distinct from, and should not be confused with the objective_function parameter of the OptimizationControlMechanism’s function. The objective_mechanism evaluates the outcome of processing without taking into account the costs of the OptimizationControlMechanism’s control_signals. In contrast, its evaluation_function, which is assigned as the objective_function parameter of its function, takes the costs of the OptimizationControlMechanism’s control_signals into account when calculating the net_outcome that it returns as its result.

### State Features¶

In addition to its primary InputPort (which typically receives a projection from the OUTCOME OutputPort of the objective_mechanism, an OptimizationControlMechanism also has an InputPort for each of its state_features. By default, these are the current input for the Composition to which the OptimizationControlMechanism belongs. However, different values can be specified, as can a state_feature_function that transforms these. For OptimizationControlMechanisms that implement model-free optimization, its state_feature_values are used by its evaluation_function to predict the net_outcome for a given control_allocation. For OptimizationControlMechanisms that implement fully agent_rep-based optimization, the state_feature_values are used as the Composition’s input when it is executed to evaluate the net_outcome for a given control_allocation.

State features can be of two types:

• Input Features – these are values received as input by other Mechanisms in the Composition. They are specified as shadowed inputs in the state_features argument of the OptimizationControlMechanism’s constructor (see Creating an OptimizationControlMechanism). An InputPort is created on the OptimizationControlMechanism for each feature, that receives a Projection paralleling the input to be shadowed.

The current value of the InputPorts for the state_features are listed in the state_feature_values attribute.

### State¶

The state of the Composition (or part of one) controlled by an OptimizationControlMechanism is defined by a combination of state_feature_values for its state_features <state_features <Composition.state_features> (see above) and a control_allocation.

### Agent Representation¶

The defining feature of an OptimizationControlMechanism is its agent representation, specified in the agent_rep argument of its constructor and assigned to its agent_rep attribute. This designates a representation of the Composition (or parts of it) that the OptimizationControlMechanism controls, and that it uses to evaluate sample control_allocations in order to find the one that optimizes the net_outcome when the Composition is fully executed. The agent_rep is always itself a Composition, that can be either the same one that the OptimizationControlMechanism controls (fully model-based optimization, or another one (model-free optimization) that is used to estimate the net_outcome for that Composition (see above). The evaluate method of the Composition is assigned as the evaluation_function of the OptimizationControlMechanism. If the agent_rep is not the Composition for which the OptimizationControlMechanism is the controller, then it must meet the following requirements:

### Function¶

The function of an OptimizationControlMechanism is used to find the control_allocation that optimizes the net_outcome for the current (or expected) state. It is generally an OptimizationFunction, which in turn has objective_function, search_function and search_termination_function methods, as well as a search_space attribute. The OptimizationControlMechanism’s evaluation_function is automatically assigned as the OptimizationFunction’s objective_function, and is used to evaluate each control_allocation sampled from the search_space by the search_function search_function until the search_termination_function returns True. A custom function can be assigned as the function of an OptimizationControlMechanism, however it must meet the following requirements:

• It must accept as its first argument and return as its result an array with the same shape as the OptimizationControlMechanism’s control_allocation.

• It must implement a reset method that accepts objective_function as a keyword argument and implements an attribute with the same name.

If function argument is not specified, the GridSearch OptimiziationFunction is assigned as the default, which evaluates the net_outcome using the OptimizationControlMechanism’s control_allocation_search_space as its search_space, and returns the control_allocation that yields the greatest net_outcome, thus implementing a computation of EVC.

## Execution¶

When an OptimizationControlMechanism is executed, it carries out the following steps:

## Class Reference¶

class psyneulink.core.components.mechanisms.modulatory.control.optimizationcontrolmechanism.OptimizationControlMechanism(objective_mechanism=None, monitor_for_control=None, objective_mechanism=None, origin_objective_mechanism=False terminal_objective_mechanism=False state_features=None, state_feature_function=None, function=GridSearch, agent_rep=None, search_function=None, search_termination_function=None, search_space=None, control_signals=None, modulation=MULTIPLICATIVE, combine_costs=np.sum, compute_reconfiguration_cost=None, compute_net_outcome=lambda x, y:x-y)

Subclass of ControlMechanism that adjusts its ControlSignals to optimize performance of the Composition to which it belongs. See parent class for additional arguments.

Parameters
state_feature_values

the current value of each item of the OptimizationControlMechanism’s state_features (each of which is a 1d array).

Type

2d array

agent_rep

determines the Composition used by the evalution_function to predict the net_outcome for a given state (see above <OptimizationControlMechanism_Agent_Rep>for additional details).

Type

Composition

function

takes current control_allocation (as initializer), uses its search_function to select samples of control_allocation from its search_space, evaluates these using its evaluation_function, and returns the one that yields the optimal net_outcome (see Function for additional details).

Type

OptimizationFunction, function or method

evaluation_function

returns net_outcome for a given state (i.e., combination of state_feature_values and control_allocation. It is assigned as the objective_function parameter of function, and calls the evaluate method of the OptimizationControlMechanism’s agent_rep with a control_allocation, the OptimizationControlMechanism’s num_estimates attribute, and the current state_feature_values.

Type

function or method

control_allocation_search_space

search_space assigned by default to function, that determines the samples of control_allocation evaluated by the evaluation_function. This is a proprety that, unless overridden, returns a list of the SampleIterators generated from the allocation_sample specifications for each of the OptimizationControlMechanism’s control_signals.

Type

list of SampleIterators

saved_samples

contains all values of control_allocation sampled by function if its save_samples parameter is True; otherwise list is empty.

Type

list

saved_values

contains values of net_outcome associated with all samples of control_allocation evaluated by by function if its save_values parameter is True; otherwise list is empty.

Type

list

search_statefulness

if set to False, an OptimizationControlMechanism‘s evaluation_function will never run simulations; the evaluations will simply execute in the original execution context.

if set to True, simulations will be created normally for each control allocation.

Type

bool : True

_validate_params(request_set, target_set=None, context=None)

Insure that specification of ObjectiveMechanism has projections to it

_instantiate_input_ports(context=None)

Instantiate input_ports for Projections from state_features and objective_mechanism.

Inserts InputPort specification for Projection from ObjectiveMechanism as first item in list of InputPort specifications generated in _parse_state_feature_specs from the state_features and state_feature_function arguments of the OptimizationControlMechanism constructor.

_instantiate_output_ports(context=None)

Assign CostFunctions.DEFAULTS as default for cost_option of ControlSignals. OptimizationControlMechanism requires use of at least one of the cost options

_instantiate_control_signals(context)

Size control_allocation and assign modulatory_signals Set size of control_allocadtion equal to number of modulatory_signals. Assign each modulatory_signal sequentially to corresponding item of control_allocation.

_instantiate_function(function, function_params=None, context=None)

Assign weights and exponents if specified in input_ports

_instantiate_attributes_after_function(context=None)

Instantiate OptimizationControlMechanism’s OptimizatonFunction attributes

_update_input_ports(runtime_params=None, context=None)

Update value for each InputPort in self.input_ports:

Call execute method for all (MappingProjection) Projections in Port.path_afferents Aggregate results (using InputPort execute method) Update InputPort.value

_execute(variable=None, context=None, runtime_params=None)

Find control_allocation that optimizes result of agent_rep.evaluate .

evaluation_function(control_allocation, context=None, return_results=False)

Compute net_outcome for current set of state_feature_values and a specified control_allocation.

Assigned as the objective_function for the OptimizationControlMechanism’s function.

Calls agent_rep's evalute method.

Returns a scalar that is the predicted net_outcome for the current state_feature_values and specified control_allocation.

add_state_features(features, context=None)

Add InputPorts and Projections to OptimizationControlMechanism for state_features used to predict net_outcome

state_features argument can use any of the forms of specification allowed for InputPort(s)

_parse_state_feature_specs(input_ports, feature_function, context=None)

Parse entries of state_features into InputPort spec dictionaries Set INTERNAL_ONLY entry of params dict of InputPort spec dictionary to True

(so that inputs to Composition are not required if the specified state is on an INPUT Mechanism)

Assign functions specified in state_feature_function to InputPorts for all state_features Return list of InputPort specification dictionaries

property _model_spec_parameter_blacklist

A set of Parameter names that should not be added to the generated constructor string

_initialize_composition_function_approximator(context)

Initialize CompositionFunctionApproximator

exception psyneulink.core.components.mechanisms.modulatory.control.optimizationcontrolmechanism.OptimizationControlMechanismError`(error_value)