# ObjectiveMechanism¶

## Overview¶

An ObjectiveMechanism is a ProcessingMechanism that monitors the OutputStates of one or more other ProcessingMechanisms specified in its monitor attribute, and evaluates them using its function. The result of the evaluation is placed in the ObjectiveMechanism’s OUTCOME (primary) OutputState. ObjectiveMechanisms are typically used closely with (and often created automatically by) AdaptiveMechanisms.

## Creating an ObjectiveMechanism¶

ObjectiveMechanisms are often created automatically when other PsyNeuLink components are created (in particular, AdaptiveMechanisms, such as LearningMechanisms and ControlMechanisms). An ObjectiveMechanism can also be created directly by calling its constructor. The primary attribute used to define an ObjectiveMechanism is its monitor attribute, that is specified using the corresponding argument of its constructor as described below.

### Monitor¶

The monitor argument of the constructor specifies the OutputStates it monitors. This takes the place of the input_states argument used by most other forms of Mechanism, and is used by the ObjectiveMechanism to create an InputState for each OutputState it monitors, along with a MappingProjection from the OutputState to that InputState. The monitor argument takes a list of items that can include any of the forms of specification used in a standard input_states argument. For the monitor argument, this is usually a list of OutputStates to be monitored. However, as with a standard input_states argument, the monitor argument can include Mechanisms (in which case their primary Outputstate is used) or the InputState(s) of other Mechanisms (in which case the ObjectiveMechanism will be assigned Projections from all of the OutputStates that project to the specified InputState – that is, it will shadow their inputs). Items in the monitor argument can also be used to specify attributes of the InputState and/or MappingProjection(s) to it, that the ObjectiveMechanism creates to monitor the specified OutputState. In general, the value of each specified OutputState determines the format of the variable of the InputState that is created for it by the ObjectiveMechanism. However, this can be overridden using the ObjectiveMechanism’s default_variable or size attributes (see Mechanism InputState specification), or by specifying a Projection from the OutputState to the InputState (see Input Source Specification). If an item in the monitor argument specifies an InputState for the ObjectiveMechanism, but not the OutputState to be monitored, the InputState is created but will be ignored until an OutputState (and MappingProjection from it) are specified for that InputState.

The OutputStates monitored by the ObjectiveMechanism are listed in its monitor attribute. When an ObjectiveMechanism is created by a Control Mechanisms, or a System for its controller, these may pass a set of OutputStates to be monitored to the ObjectiveMechanism. A ControlMechanism passes OutputState specifications listed in its objective_mechanism argument (see ObjectiveMechanism), and a System passes any listed in its monitor_for_control argument (see Specifying Control).

## Structure¶

### Input¶

An ObjectiveMechanism has one InputState for each of the OutputStates specified in its monitor argument (see Monitor). Each InputState receives a MappingProjection from the corresponding OutputState, the values of which are used by the ObjectiveMechanism’s function to generate the value of its OUTCOME OutputState. The InputStates are listed in the ObjectiveMechanism’s input_states attribute, and the monitored OutputStates from which they receive projections are listed in the same order its monitor attribute.

By default, the format of the variable for each InputState is determined by the value of the monitored OutputState(s) to which it corresponds. However, if either the default_variable or size argument is specified in an Objective Mechanism’s constructor, or a variable is specified for an InputState for one or more of the items in its monitor argument, then that is used as the format for the corresponding InputState(s). This can be used to transform the value of a monitored OutputState into different form for the variable of the InputState (see the first example below).

If the weight and/or exponent is specified for ay item in the monitor argument of the ObjectiveMechanism’s constructor, it is assigned to the corresponding InputState. If the ObjectiveMechanism’s function implements a weights and/or exponents attribute, the values specified is assigned to the corresponding attribute, and applied to the value of the InputState before it is combined with that of the other InputStates to generate the ObjectiveMechanism’s output.

### Function¶

The ObjectiveMechanism’s function uses the values of its input_states to compute an objective (or “loss”) function, that is assigned as the value of its OUTCOME OutputState. By default, it uses a LinearCombination function to sum the values of the values of the items in its variable. However, by assigning values to the ‘weight <InputState.weight> and/or ‘exponent <InputState.exponent> attributes of the corresponding InputStates, it can be configured to calculate differences, ratios, etc. (see example below). The function can also be replaced with any CombinationFunction, or any python function that takes an ndarray as its input (with a number of items in axis 0 equal to the number of the ObjectiveMechanism’s InputStates), and generates a 1d array as its result. If it implements weight and/or exponent attributes, those are assigned from weight and exponent attributes of its input_states (also listed in the monitor_weights_and_exponents attribute); otherwise, they are ignored.

### Output¶

The primary OutputState of an Objective mechanism is a 1d array, named OUTCOME, that is the result of its function (as described above).

## Execution¶

When an ObjectiveMechanism is executed, it updates its input_states with the values of the OutputStates listed in its monitor attribute, and then uses its function to evaluate these. The result is assigned as to its value attribute as the value of its OUTCOME (primary) OutputState.

Examples

Specifying the variable for the InputStates of an ObjectiveMechanism

This can be useful in some situations, and there are several ways it can be done. For example, for Reinforcement Learning, an ObjectiveMechanism is used to monitor the rewards by an action selection Mechanism (and used by a LearningMechanism to improve those predictions). However, whereas the action selection Mechanism generates a vector indicating the reward predicted by the selected action, the ObjectiveMechanism for RL simply needs to know the magnitude of the reward predicted, irrespective of the action taken; that is, it simply requires a single scalar value indicating the magnitude of the predicted reward. Thus, the vector of action-related reward predictions needs to be condensed to a single predicted value for the ObjectiveMechanism. This can be accomplished in several ways, that are illustrated in the examples below. In the first example, a TransferMechanism with the SoftMax function (and the PROB as its output format) to select an action and represent its reward prediction. This generates a vector with a single non-zero value, which designates the predicted reward for the selected action. Because the output is a vector, by default the InputState of the ObjectiveMechanism created to monitor it will also be a vector. However, the ObjectiveMechanism requires this to be a single value, that it can compare with the value of the reward Mechanism (monitoring the feedback provided by the environment). In the example below, this is accomplished by using default_variable in the constructor of the ObjectiveMechanism to force the InputState for the ObjectiveMechanism to have a single value:

>>> import psyneulink as pnl
>>> my_action_select_mech = pnl.TransferMechanism(default_variable=[0, 0, 0],
...                                               function=pnl.SoftMax(output=pnl.PROB),
...                                               name='Action Selection Mech')

>>> my_reward_mech = pnl.TransferMechanism(default_variable=[0],
...                                        name='Reward Mech')

>>> my_objective_mech = pnl.ObjectiveMechanism(default_variable=[[0],[0]],
...                                            monitor=[my_action_select_mech, my_reward_mech])


Note that the OutputStates for the my_action_selection and my_reward_mech are specified in monitor. If that were the only specification, the InputState created for my_action_select_mech would be a vector of length 3. This is overridden by specifying default_variable as an array with two single-value arrays (one corresponding to my_action_select_mech and the other to my_reward_mech). This forces the InputState for my_action_select_mech to have only a single element which, in turn, will cause a MappingProjection to be created from my_action_select_mech to the ObjectiveMechanism’s InputState using a FULL_CONNECTIVITY_MATRIX (the one used for AUTO_ASSIGN_MATRIX when the sender and receiver have values of different lengths). This produces the desired effect, since the action selected is the only non-zero value in the output of my_action_select_mech, and so the FULL_CONNECTIVITY_MATRIX will combine it with zeros (the other values in the vector), and so its value will be assigned as the value of the corresponding InputState in the ObjectiveMechanism.

An alternative would be to explicitly specify the variable attribute for the InputState created for my_action_select_mech using a InputState specification dictionary in the monitor argument of my_objective_mech, as follows:

>>> my_objective_mech = pnl.ObjectiveMechanism(monitor=[{pnl.MECHANISM: my_action_select_mech,
...                                                      pnl.VARIABLE: [0]},
...                                                      my_reward_mech])


Note that the VARIABLE entry here specifies the variable for the InputState of the ObjectiveMechanism created to receive a Projection from my_action_select_mech, and not my_action_select_mech itself (see Input for a full explanation).

Another way to specify the variable for the InputState of an ObjectiveMechanism is to specify the Projections it receives from the OutputState it monitors. The following example uses a tuple specification to assign the matrix for the MappingProjection from my_action_select_mech to the corresponding InputState of my_objective_mech:

>>> import numpy as np
>>> my_objective_mech = pnl.ObjectiveMechanism(monitor=[(my_action_select_mech, np.ones((3,1))), my_reward_mech])


Since the matrix specified has three rows (for its inputs) and one col (for the output), it will take the length three vector provided as the output of my_action_select_mech and combine its elements into a single value that is provided to the InputState of the ObjectiveMechanism.

A Connection tuple could also be used to specify the matrix, but this would require that additional entries (for the weight and exponent of the InputState) which, in this case, are not necessary (see example below for how these can be used).

By default, an ObjectiveMechanism simply adds the values received by each of its InputStates to generate its output. However, this too can be customized in a variety of ways.

The simplest way is to assign values to the weight and/or exponent attributes of its InputStates. This can be done by placing them in a tuple specification for the OutputState that provides value for the InputState. In the example below, the ObjectiveMechanism used in the previous example is further customized to subtract the value of the action selected from the value of the reward:

>>> my_objective_mech = pnl.ObjectiveMechanism(default_variable = [[0],[0]],
...                                            monitor = [(my_action_select_mech, -1, 1), my_reward_mech])


This specifies that my_action_select_mech should be assigned a weight of -1 and an exponent of 1 when it is submitted to the ObjectiveMechanism’s function. Notice that the exponent had to be included, even though it is the default value; when a tuple is used, the weight and exponent values must both be specified. Notice also that my_reward_mech does not use a tuple, so it will be assigned defaults for both the weight and exponent parameters.

An ObjectiveMechanism can also be configured to monitor multiple OutputStates of the same Mechanism. In the following example, an ObjectiveMechanism is configured to calculate the reward rate for a DDM Mechanism, by specifying OutputStates for the DDM that report its response time and accuracy:

>>> my_decision_mech = pnl.DDM(output_states=[pnl.RESPONSE_TIME,
...                                           pnl.PROBABILITY_UPPER_THRESHOLD])

>>> my_objective_mech = pnl.ObjectiveMechanism(monitor=[
...                                              my_reward_mech,
...                                              my_decision_mech.output_states[pnl.PROBABILITY_UPPER_THRESHOLD],
...                                              (my_decision_mech.output_states[pnl.RESPONSE_TIME], 1, -1)])


This specifies that the ObjectiveMechanism should multiply the value of my_reward_mech’s primary OutputState by the value of my_decision_mech’s PROBABILITY_UPPER_THRESHOLD, and divide the result by my_decision_mech’s RESPONSE_TIME value. The two OutputStates of my_decision_mech are referenced as items in the output_states list of my_decision_mech. However, a 2-item (State name, Mechanism) tuple can be used to reference them more simply, as follows:

>>> my_objective_mech = pnl.ObjectiveMechanism(monitor=[
...                                           my_reward_mech,
...                                           (pnl.PROBABILITY_UPPER_THRESHOLD, my_decision_mech),
...                                           ((pnl.RESPONSE_TIME, my_decision_mech), 1, -1)])


Customizing the ObjectiveMechanism’s function

In the examples above, the weights and exponents assigned to the InputStates are passed to the ObjectiveMechanism’s function for use in combining their values. The same can be accomplished by specifying the relevant parameter(s) of the function itself, as in the following example:

>>> my_objective_mech = pnl.ObjectiveMechanism(default_variable = [[0],[0]],
...                                            monitor = [my_action_select_mech, my_reward_mech],
...                                            function=pnl.LinearCombination(weights=[[-1], [1]]))


Here, the weights parameter of the LinearCombination function is specified directly, with two values [-1] and [1] corresponding to the two items in monitor (and default_variable). This will multiply the value from my_action_select_mech by -1 before adding it to (and thus subtracting it from) the value of my_reward_mech. Notice that the weight for my_reward_mech had to be specified, even though it is using the default value (1); whenever a weight and/or exponent parameter is specified, there must be an entry for every item of the function’s variable. However, the exponents did not need to be specified, as it is not used. However it, and the operation parameters of LinearCombination can also be used to multiply and divide quantities.

## Class Reference¶

class psyneulink.core.components.mechanisms.processing.objectivemechanism.OBJECTIVE_OUTPUT
OUTCOME : 1d np.array
the value of the objective or “loss” function computed based on the ObjectiveMechanism’s function
class psyneulink.core.components.mechanisms.processing.objectivemechanism.ObjectiveMechanism(monitor, default_variable, size, function=LinearCombination, output_states=OUTCOME, params=None, name=None, prefs=None)

Subclass of ProcessingMechanism that evaluates the value(s) of one or more OutputStates.

Parameters: monitor (List[OutputState, ‘InputState, Mechanisms, str, value, dict, MonitoredOutputStatesOption] or dict) – specifies the OutputStates, the values of which will be monitored, and evaluated by function (see Monitor for details of specification). default_variable (number, list or np.ndarray : default monitor) – specifies the format of the variable for the InputStates of the ObjectiveMechanism (see Mechanism InputState specification for details). size (int, list or np.ndarray of ints) – specifies default_variable as array(s) of zeros if default_variable is not passed as an argument; if default_variable is specified, it takes precedence over the specification of size. As an example, the following mechanisms are equivalent: T1 = TransferMechanism(size = [3, 2]) T2 = TransferMechanism(default_variable = [[0, 0, 0], [0, 0]])  function (CombinationFunction, ObjectiveFunction, function or method : default LinearCombination) – specifies the function used to evaluate the values listed in monitor  (see function for details). output_states (List[OutputState, value, str or dict] or Dict[] : default [OUTCOME]) – specifies the OutputStates for the Mechanism; role (Optional[LEARNING, CONTROL]) – specifies if the ObjectiveMechanism is being used for learning or control (see role for details). params (Dict[param keyword: param value] : default None) – a parameter dictionary that can be used to specify the parameters for the Mechanism, its function, and/or a custom function and its parameters. Values specified for parameters in the dictionary override any assigned to those parameters in arguments of the constructor. name (str : default see name) – specifies the name of the ObjectiveMechanism. prefs (PreferenceSet or specification dict : default Mechanism.classPreferences) – specifies the PreferenceSet for the ObjectiveMechanism; see prefs for details.
variable

2d ndarray : default array of values of OutputStates in monitor_output_states – the input to Mechanism’s function.

monitor

ContentAddressableList[OutputState] – determines the OutputStates, the values of which are monitored, and evaluated by the ObjectiveMechanism’s function. Each item in the list refers to an OutputState containing the value to be monitored, with a MappingProjection from it to an corresponding InputState listed in the input_states attribute

Note

If any of the ObjectiveMechanism’s input_states were specified to shadow the InputState of another Mechanism, and any of those shadowed InputStates receives more than one Projections, then the list of monitored OutputStates in monitor will be longer than the list of the ObjectiveMechanism’s input_states.

monitor_weights_and_exponents

List[Tuple(float, float)] – each tuple in the list contains a weight and exponent associated with a corresponding InputState listed in the ObjectiveMechanism’s input_states attribute; these are used by its function to parametrize the contribution that the values of each of the OuputStates monitored by the ObjectiveMechanism makes to its output (see Function)

input_states

ContentAddressableList[InputState] – contains the InputStates of the ObjectiveMechanism, each of which receives a MappingProjection from the OutputStates specified in its monitor attribute.

function

CombinationFunction, ObjectiveFunction, function, or method – the function used to evaluate the values monitored by the ObjectiveMechanism. The function can be any PsyNeuLink CombinationFunction or a Python function that takes a 2d array with an arbitrary number of items or a number equal to the number of items in the ObjectiveMechanism’s variable (i.e., its number of input_states) and returns a 1d array.

role

None, LEARNING or CONTROL – specifies whether the ObjectiveMechanism is used for learning in a Process or System (in conjunction with a ObjectiveMechanism), or for control in a System (in conjunction with a ControlMechanism).

value

1d np.array – the output of the evaluation carried out by the ObjectiveMechanism’s function.

output_state

OutputState – contains the primary OutputState of the ObjectiveMechanism; the default is its OUTCOME OutputState, the value of which is equal to the value attribute of the ObjectiveMechanism.

output_states

ContentAddressableList[OutputState] – by default, contains only the OUTCOME (primary) OutputState of the ObjectiveMechanism.

output_values

2d np.array – contains one item that is the value of the OUTCOME OutputState.

name

str – the name of the ObjectiveMechanism; if it is not specified in the name argument of the constructor, a default is assigned by MechanismRegistry (see Naming for conventions used for default and duplicate names).

prefs

PreferenceSet or specification dict – the PreferenceSet for the ObjectiveMechanism; if it is not specified in the prefs argument of the constructor, a default is assigned using classPreferences defined in __init__.py (see PreferenceSet for details).

add_to_monitor(monitor_specs, context=None)

Instantiate OutputStates to be monitored by the ObjectiveMechanism.

Used by other Components to add a States or list of States to be monitored by the ObjectiveMechanism. The monitor_spec can be any of the following: - MonitoredOutputStateTuple - Mechanisms; - OutputState; - tuple specification; - State specification dictionary; - list with any of the above. If the item is a Mechanism, its primary OutputState is used.