EVCControlMechanism

Overview

An EVCControlMechanism is a ControlMechanism that regulates it ControlSignals in order to optimize the performance of the System to which it belongs. EVCControlMechanism is one of the most powerful, but also one of the most complex components in PsyNeuLink. It is designed to implement a form of the Expected Value of Control (EVC) Theory described in Shenhav et al. (2013), which provides useful background concerning the purpose and structure of the EVCControlMechanism.

An EVCControlMechanism is similar to a standard Control Mechanism, with the following exceptions:

  • it has several specialized functions that are used to search over the allocationss of its its ControlSignals, and evaluate the performance of its system; by default, it simulates its system and evaluates its performance under all combinations of ControlSignal values to find the one that optimizes the Expected Value of Control, however its functions can be customized or replaced to implement other optimization procedures.
  • it creates a specialized set of prediction Mechanisms EVCControlMechanism_Prediction_Mechanisms` that are used to simulate the performnace of its system.

Expected Value of Control (EVC)

The EVCControlMechanism uses it function to select an allocation_policy for its system. In the default configuration, an EVCControlMechanism carries out an exhaustive evaluation of allocation policies, simulating its system under each, and using an ObjectiveMechanism and several auxiliary functions to calculate the expected value of control (EVC) for each allocation_policy: a cost-benefit analysis that weighs the cost s performance for a given allocation_policy. The EVCControlMechanism selects the allocation_policy that generates the maximum EVC, and implements that for the next TRIAL. Each step of this procedure can be modified, or replaced entirely, by assigning custom functions to corresponding parameters of the EVCControlMechanism, as described below.

Creating an EVCControlMechanism

An EVCControlMechanism can be created in any of the ways used to create a ControlMechanism; it is also created automatically when a System is created and the EVCControlMechanism class is specified in the controller argument of the System’s constructor (see Creating a System). The ObjectiveMechanism, the OutputStates it monitors and evaluates, and the parameters controlled by an EVCControlMechanism can be specified in the standard way for a ControlMechanism (see ObjectiveMechanism and Monitored OutputStates and Specifying Parameters to Control, respectively).

Note

Although an EVCControlMechanism can be created on its own, it can only be assigned to, and executed within a System as the System’s controller.

When an EVCControlMechanism is assigned to, or created by a System, it is assigned the OutputStates to be monitored and parameters to be controlled specified for that System (see Control), and a prediction Mechanism is created for each ORIGIN Mechanism in the system. The prediction Mechanisms are assigned to the EVCControlMechanism’s prediction_mechanisms attribute. The OutputStates used to determine an EVCControlMechanism’s allocation_policy and the parameters it controls can be listed using its show method. The EVCControlMechanism and the Components associated with it in its system can be displayed using the System’s System.show_graph method with its show_control argument assigned as True

An EVCControlMechanism that has been constructed automatically can be customized by assigning values to its attributes (e.g., those described above, or its function as described under `EVC_Default_Configuration `below).

Structure

An EVCControlMechanism must belong to a System (identified in its system attribute). In addition to the standard Components of a Control Mechanism, has a specialized set of prediction mechanisms and functions that it uses to simulate and evaluate the performance of its system under the influence of different values of its ControlSignals. Each of these specialized Components is described below.

Input

ObjectiveMechanism

Like any ControlMechanism, an EVCControlMechanism receives its input from the OUTCOME OutputState of an ObjectiveMechanism, via a MappingProjection to its primary InputState. The ObjectiveFunction is listed in the EVCControlMechanism’s objective_mechanism attribute. By default, the ObjectiveMechanism’s function is a LinearCombination function with its operation attribute assigned as PRODUCT; this takes the product of the values of the OutputStates that it monitors (listed in its monitored_output_states attribute. However, this can be customized in a variety of ways:

  • using a list to specify the OutputStates to be monitored (and the tuples format to specify weights and/or exponents for them) in the objective_mechanism argument of the EVCControlMechanism’s constructor;
  • using the monitored_output_states argument for an ObjectiveMechanism specified in the objective_mechanism argument of the EVCMechanism’s constructor;
  • specifying a different ObjectiveMechanism in the objective_mechanism argument of the EVCControlMechanism’s constructor. The result of the objective_mechanism’s function is used as the outcome in the calculations described below.

Note

If a constructor for an ObjectiveMechanism is used for the objective_mechanism argument of the EVCControlMechanism’s constructor, then the default values of its attributes override any used by the EVCControlMechanism for its objective_mechanism. In particular, whereas an EVCControlMechanism uses the same default function as an ObjectiveMechanism (LinearCombination), it uses PRODUCT rather than SUM as the default value of the operation attribute of the function. As a consequence, if the constructor for an ObjectiveMechanism is used to specify the EVCControlMechanism’s objective_mechanism argument, and the operation argument is not specified, SUM rather than PRODUCT will be used for the ObjectiveMechanism’s function. To ensure that PRODUCT is used, it must be specified explicitly in the operation argument of the constructor for the ObjectiveMechanism (see 1st example under Specifying Control for a System).

The result of the EVCControlMechanism’s objective_mechanism is used by its function to evaluate the performance of its system when computing the EVC.

Prediction Mechanisms

These are used to provide input to the system when the EVCControlMechanism’s default function (ControlSignalGridSearch) simulates its execution to evaluate the EVC for each allocation_policy. When an EVCControlMechanism is created, a prediction Mechanism is created for each ORIGIN Mechanism in its system, and for each Projection received by an ORIGIN Mechanism, a MappingProjection from the same source is created that projects to the corresponding prediction Mechanism. The type of Mechanism used for the prediction Mechanisms is specified by the EVCControlMechanism’s prediction_mechanism_type attribute, and their parameters can be specified with the prediction_mechanism_params attribute. The default type is an ‘IntegratorMechanism`, that calculates an exponentially weighted time-average of its input. The prediction mechanisms for an EVCControlMechanism are listed in its prediction_mechanisms attribute.

Function

By default, the primary function is ControlSignalGridSearch (see EVC_Default_Configuration), that systematically evaluates the effects of its ControlSignals on the performance of its system to identify an allocation_policy that yields the highest EVC. However, any function can be used that returns an appropriate value (i.e., that specifies an allocation_policy for the number of ControlSignals in the EVCControlMechanism’s control_signals attribute, using the correct format for the allocation value of each ControlSignal). In addition to its primary function, an EVCControlMechanism has several auxiliary functions, that can be used by its function to calculate the EVC to select an allocation_policy with the maximum EVC among a range of policies specified by its ControlSignals. The default set of functions and their operation are described in the section that follows; however, the EVCControlMechanism’s function can call any other function to customize how the EVC is calcualted.

Default Configuration of EVC Function and its Auxiliary Functions

In its default configuration, an EVCControlMechanism simulates and evaluates the performance of its system under a set of allocation_policies determined by the allocation_samples attributes of its ControlSignals, and implements (for the next TRIAL of execution) the one that generates the maximum EVC. This is carried out by the EVCControlMechanism’s default function and three auxiliary functions, as described below.

The default function of an EVCControlMechanism is ControlSignalGridSearch. It identifies the allocation_policy with the maximum EVC by a conducting an exhaustive search over every possible allocation_policy— that is, all combinations of allocation values for its ControlSignals, where the allocation values sampled for each ControlSignal are determined by its allocation_samples attribute. For each allocation_policy, the EVCControlMechanism executes the system, evaluates the EVC for that policy, and returns the allocation_policy that yields the greatest EVC value. The following steps are used to calculate the EVC in each allocation_policy:

  • Implement the policy and simulate the System - assign the allocation that the selected allocation_policy specifies for each ControlSignal, and then simulate the system using the corresponding parameter values.


In addition to modifying the default functions (as noted above), any or all of them can be replaced with a custom function to modify how the allocation_policy is determined, so long as the custom function accepts arguments and returns values that are compatible with any other functions that call that function (see note below).

Note

The EVCControlMechanism auxiliary functions described above are all implemented as PsyNeuLink Functions. Therefore, to call a function itself, it must be referenced as <EVCControlMechanism>.<function_attribute>.function. A custom function assigned to one of the auxiliary functions can be either a PsyNeuLink Function, or a generic python function or method (including a lambda function). If it is one of the latter, it is automatically “wrapped” as a PsyNeuLink Function (specifically, it is assigned as the function attribute of a UserDefinedFunction object), so that it can be referenced and called in the same manner as the default function assignment. Therefore, once assigned, it too must be referenced as <EVCControlMechanism>.<function_attribute>.function.

ControlSignals

The OutputStates of an EVCControlMechanism (like any Control Mechanism) are a set of ControlSignals, that are listed in its control_signals attribute (as well as its output_states attribute). Each ControlSignal is assigned a ControlProjection that projects to the ParameterState for a parameter controlled by the EVCControlMechanism. Each ControlSignal is assigned an item of the EVCControlMechanism’s allocation_policy, that determines its allocation for a given TRIAL of execution. The allocation is used by a ControlSignal to determine its intensity, which is then assigned as the value of the ControlSignal’s ControlProjection. The value of the ControlProjection is used by the ParameterState to which it projects to modify the value of the parameter (see Modulation for description of how a ControlSignal modulates the value of a parameter it controls). A ControlSignal also calculates a cost, based on its intensity and/or its time course. The cost is included in the evaluation that the EVCControlMechanism carries out for a given allocation_policy, and that it uses to adapt the ControlSignal’s allocation in the future. When the EVCControlMechanism chooses an allocation_policy to evaluate, it selects an allocation value from the ControlSignal’s allocation_samples attribute.

Execution

An EVCControlMechanism must be the controller of a System, and as a consequence it is always the last Mechanism to be executed in a TRIAL for its system (see System Control and Execution). When an EVCControlMechanism is executed, it updates the value of its prediction_mechanisms and objective_mechanism, and then calls its function, which determines and implements the allocation_policy for the next TRIAL of its system s execution. The default function executes the following steps (described in greater detailed above):

This procedure can be modified by specifying a custom function for any or all of the functions referred to above.

Example

The following example implements a System with an EVCControlMechanism (and two processes not shown):

>>> import psyneulink as pnl                                                        
>>> myRewardProcess = pnl.Process(...)                                              
>>> myDecisionProcess = pnl.Process(...)                                            
>>> mySystem = pnl.System(processes=[myRewardProcess, myDecisionProcess],           
...                       controller=pnl.EVCControlMechanism,                       
...                       monitor_for_control=[Reward,                              
...                                            pnl.DDM_OUTPUT.DECISION_VARIABLE,    
...                                            (pnl.RESPONSE_TIME, 1, -1)],         

It uses the System’s monitor_for_control argument to assign three OutputStates to be monitored. The first one references the Reward Mechanism (not shown); its primary OutputState will be used by default. The second and third use keywords that are the names of outputStates of a DDM Mechanism (also not shown). The last one (RESPONSE_TIME) is assigned a weight of 1 and an exponent of -1. As a result, each calculation of the EVC computation will multiply the value of the primary OutputState of the Reward Mechanism by the value of the DDM_DECISION_VARIABLE OutputState of the DDM Mechanism, and then divide that by the value of the RESPONSE_TIME OutputState of the DDM Mechanism.

See ObjectiveMechanism for additional examples of how to specify it’s monitored_output_states argument, ControlMechanism for additional examples of how to specify ControlMechanisms, and System for how to specify the controller of a System.

Class Reference

class psyneulink.library.subsystems.evc.evccontrolmechanism.EVCControlMechanism(system=True, objective_mechanism=None, prediction_mechanism_type=IntegratorMechanism, prediction_mechanism_params=None, function=ControlSignalGridSearch value_function=ValueFunction, cost_function=LinearCombination(operation=SUM), combine_outcome_and_cost_function=LinearCombination(operation=SUM) save_all_values_and_policies:bool=:keyword:`False`, control_signals=None, params=None, name=None, prefs=None)

Subclass of ControlMechanism that optimizes the ControlSignals for a System.

Parameters:
  • system (System : default None) – specifies the System for which the EVCControlMechanism should serve as a controller; the EVCControlMechanism will inherit any OutputStates specified in the monitor_for_control argument of the system’s constructor, and any ControlSignals specified in its control_signals argument.
  • objective_mechanism (ObjectiveMechanism, List[OutputState or Tuple[OutputState, list or 1d np.array, list or 1d) –
  • np.array]] (default MonitoredOutputStatesOptions.PRIMARY_OUTPUT_STATES) – specifies either an ObjectiveMechanism to use for the EVCControlMechanism or a list of the OutputStates it should monitor; if a list of OutputState specifications is used, a default ObjectiveMechanism is created and the list is passed to its monitored_output_states argument.
  • prediction_mechanism_type (CombinationFunction: default IntegratorMechanism) – the Mechanism class used for prediction Mechanism(s). Each instance is named using the name of the ORIGIN Mechanism + “PREDICTION_MECHANISM” and assigned an OutputState with a name based on the same.
  • prediction_mechanism_params (Dict[param keyword: param value] : default None) – a parameter dictionary passed to the constructor for a Mechanism of prediction_mechanism_type. The same parameter dictionary is passed to all prediction mechanisms created for the EVCControlMechanism.
  • function (function or method : ControlSignalGridSearch) – specifies the function used to determine the allocation_policy for the next execution of the EVCControlMechanism’s system (see function for details).
  • value_function (function or method : value_function) – specifies the function used to calculate the EVC for the current allocation_policy (see value_function for details).
  • cost_function (function or method : LinearCombination(operation=SUM)) – specifies the function used to calculate the cost associated with the current allocation_policy (see cost_function for details).
  • combine_outcome_and_cost_function (function or method : LinearCombination(operation=SUM)) – specifies the function used to combine the outcome and cost associated with the current allocation_policy, to determine its value (see combine_outcome_and_cost_function for details).
  • save_all_values_and_policies (bool : default False) – specifes whether to save every allocation_policy tested in EVC_policies and their values in EVC_values.
  • control_signals (ControlSignal specification or List[ControlSignal specification, ..]) – specifies the parameters to be controlled by the EVCControlMechanism (see Specifying ControlSignals for details of specification).
  • params (Dict[param keyword: param value] : default None) – a parameter dictionary that can be used to specify the parameters for the Mechanism, its function, and/or a custom function and its parameters. Values specified for parameters in the dictionary override any assigned to those parameters in arguments of the constructor.
  • name (str : default see name) – specifies the name of the EVCControlMechanism.
  • prefs (PreferenceSet or specification dict : default Mechanism.classPreferences) – specifies the PreferenceSet for the EVCControlMechanism; see prefs for details.
system

System_Base – the System for which EVCControlMechanism is the controller; the EVCControlMechanism inherits any OutputStates specified in the monitor_for_control argument of the system’s constructor, and any ControlSignals specified in its control_signals argument.

prediction_mechanisms

List[ProcessingMechanism] – list of predictions mechanisms generated for the EVCControlMechanism’s system when the EVCControlMechanism is created, one for each ORIGIN Mechanism in the System.

origin_prediction_mechanisms

Dict[ProcessingMechanism, ProcessingMechanism] – dictionary of prediction mechanisms added to the EVCControlMechanism’s system, one for each of its ORIGIN Mechanisms. The key for each entry is an ORIGIN Mechanism of the System, and the value is the corresponding prediction Mechanism.

prediction_mechanism_type

ProcessingMechanism : default IntegratorMechanism – the ProcessingMechanism class used for prediction Mechanism(s). Each instance is named based on ORIGIN Mechanism + “PREDICTION_MECHANISM”, and assigned an OutputState with a name based on the same.

prediction_mechanism_params

Dict[param key, param value] : default None – a parameter dictionary passed to prediction_mechanism_type when the prediction Mechanism is created. The same dictionary will be passed to all instances of prediction_mechanism_type created.

predicted_input

Dict[ProcessingMechanism, value] – dictionary with the value of each prediction Mechanism listed in prediction_mechanisms corresponding to each ORIGIN Mechanism of the System. The key for each entry is the name of an ORIGIN Mechanism, and its value the value of the corresponding prediction Mechanism.

objective_mechanism

ObjectiveMechanism – the ‘ObjectiveMechanism’ used by the EVCControlMechanism to evaluate the performance of its system. If a list of OutputStates is specified in the objective_mechanism argument of the EVCControlMechanism’s constructor, they are assigned as the monitored_output_states attribute for the objective_mechanism.

monitored_output_states

List[OutputState] – list of the OutputStates monitored by objective_mechanism (and listed in its monitored_output_states attribute), and used to evaluate the performance of the EVCControlMechanism’s system.

monitored_output_states_weights_and_exponents

List[Tuple[scalar, scalar]] – a list of tuples, each of which contains the weight and exponent (in that order) for an OutputState in monitored_outputStates, listed in the same order as the outputStates are listed in monitored_outputStates.

function

function : default ControlSignalGridSearch – determines the allocation_policy to use for the next round of the System’s execution. The default function, ControlSignalGridSearch, conducts an exhaustive (grid) search of all combinations of the allocation_samples of its ControlSignals (and contained in its control_signal_search_space attribute), by executing the System (using run_simulation) for each combination, evaluating the result using value_function, and returning the allocation_policy that yielded the greatest EVC value (see Default Configuration of EVC Function and its Auxiliary Functions for additional details). If a custom function is specified, it must accommodate a controller argument that specifies an EVCControlMechanism (and provides access to its attributes, including control_signal_search_space), and must return an array with the same format (number and type of elements) as the EVCControlMechanism’s allocation_policy attribute.

value_function

function : default ValueFunction – calculates the EVC for a given allocation_policy. It takes as its arguments an EVCControlMechanism, an outcome value and a list or ndarray of costs, uses these to calculate an EVC, and returns a three item tuple with the calculated EVC, and the outcome value and aggregated value of costs used to calculate the EVC. The default, ValueFunction, calls the EVCControlMechanism’s cost_function to aggregate the value of the costs, and then calls its combine_outcome_and_costs to calculate the EVC from the outcome and aggregated cost (see Default Configuration of EVC Function and its Auxiliary Functions for additional details). A custom function can be assigned to value_function so long as it returns a tuple with three items: the calculated EVC (which must be a scalar value), and the outcome and cost from which it was calculated (these can be scalar values or None). If used with the EVCControlMechanism’s default function, a custom value_function must accommodate three arguments (passed by name): a controller argument that is the EVCControlMechanism for which it is carrying out the calculation; an outcome argument that is a value; and a costs argument that is a list or ndarray. A custom function assigned to value_function can also call any of the helper functions that it calls (however, see note above).

cost_function

function : default LinearCombination(operation=SUM) – calculates the cost of the ControlSignals for the current allocation_policy. The default function sums the cost of each of the EVCControlMechanism’s ControlSignals. The weights and/or exponents parameters of the function can be used, respectively, to scale and/or exponentiate the contribution of each ControlSignal cost to the combined cost. These must be specified as 1d arrays in a WEIGHTS and/or EXPONENTS entry of a parameter dictionary assigned to the params argument of the constructor of a LinearCombination function; the length of each array must equal the number of (and the values listed in the same order as) the ControlSignals in the EVCControlMechanism’s control_signals attribute. The default function can also be replaced with any custom function that takes an array as input and returns a scalar value. If used with the EVCControlMechanism’s default value_function, a custom cost_function must accommodate two arguments (passed by name): a controller argument that is the EVCControlMechanism itself; and a costs argument that is a 1d array of scalar values specifying the cost for each ControlSignal listed in the control_signals attribute of the ControlMechanism specified in the controller argument.

combine_outcome_and_cost_function

function : default LinearCombination(operation=SUM) – combines the outcome and cost for given allocation_policy to determine its EVC. The default function subtracts the cost from the outcome, and returns the difference. This can be modified using the weights and/or exponents parameters of the function, as described for the cost_function. The default function can also be replaced with any custom function that returns a scalar value. If used with the EVCControlMechanism’s default value_function, a custom If used with the EVCControlMechanism’s default value_function, a custom combine_outcome_and_cost_function must accomoudate three arguments (passed by name): a controller argument that is the EVCControlMechanism itself; an outcome argument that is a 1d array with the outcome of the current allocation_policy; and a cost argument that is 1d array with the cost of the current allocation_policy.

control_signal_search_space

2d np.array – an array each item of which is an allocation_policy. By default, it is assigned the set of all possible allocation policies, using np.meshgrid to construct all permutations of ControlSignal values from the set specified for each by its allocation_samples attribute.

EVC_max

1d np.array with single value – the maximum EVC value over all allocation policies in control_signal_search_space.

EVC_max_state_values

2d np.array – an array of the values for the OutputStates in monitored_output_states using the allocation_policy that generated EVC_max.

EVC_max_policy

1d np.array – an array of the ControlSignal intensity <ControlSignal.intensity> values for the allocation policy that generated `EVC_max.

save_all_values_and_policies

bool : default False – specifies whether or not to save every allocation_policy and associated EVC value (in addition to the max). If it is specified, each `allocation_policy tested in the control_signal_search_space is saved in EVC_policies, and their values are saved in EVC_values.

EVC_policies

2d np.array – array with every allocation_policy tested in control_signal_search_space. The EVC value of each is stored in EVC_values.

EVC_values

1d np.array – array of EVC values, each of which corresponds to an allocation_policy in EVC_policies;

allocation_policy

2d np.array : defaultControlAllocation – determines the value assigned as the variable for each ControlSignal and its associated ControlProjection. Each item of the array must be a 1d array (usually containing a scalar) that specifies an allocation for the corresponding ControlSignal, and the number of items must equal the number of ControlSignals in the EVCControlMechanism’s control_signals attribute.

control_signals

ContentAddressableList[ControlSignal] – list of the EVCControlMechanism’s ControlSignals, including any that it inherited from its system (same as the EVCControlMechanism’s output_states attribute); each sends a ControlProjection to the ParameterState for the parameter it controls

name

str – the name of the EVCControlMechanism; if it is not specified in the name argument of the constructor, a default is assigned by MechanismRegistry (see Naming for conventions used for default and duplicate names).

prefs

PreferenceSet or specification dict – the PreferenceSet for the EVCControlMechanism; if it is not specified in the prefs argument of the constructor, a default is assigned using classPreferences defined in __init__.py (see PreferenceSet for details).

run_simulation(inputs, allocation_vector, runtime_params=None, context=None)

Run simulation of System for which the EVCControlMechanism is the controller.

Parameters:
  • inputs (List[input] or ndarray(input) : default default_variable) – the inputs used for each in a sequence of executions of the Mechanism in the System. This should be the value listed in the prediction_mechanisms attribute. The inputs are available from the predicted_input attribute.
  • allocation_vector ((1D np.array)) – the allocation policy to use in running the simulation, with one allocation value for each of the EVCControlMechanism’s ControlSignals (listed in control_signals).
  • runtime_params (Optional[Dict[str, Dict[str, Dict[str, value]]]]) – a dictionary that can include any of the parameters used as arguments to instantiate the mechanisms, their functions, or Projection(s) to any of their states. See Runtime Parameters for a full description.