ObjectiveMechanism

Overview

An ObjectiveMechanism is a ProcessingMechanism that monitors the OutputStates of one or more other ProcessingMechanisms specified in its monitor attribute, and evaluates them using its function. The result of the evaluation is placed in the ObjectiveMechanism’s primary OutputState. ObjectiveMechanisms are typically used closely with (and often created automatically by) AdaptiveMechanisms.

Creating an ObjectiveMechanism

ObjectiveMechanisms are often created automatically when other PsyNeuLink components are created (in particular, AdaptiveMechanisms, such as LearningMechanisms and ControlMechanisms). An ObjectiveMechanism can also be created directly by calling its constructor. The primary attribute used to define an ObjectiveMechanism is its monitored_output_states, that is specified using the corresponding argument of its constructor as described below.

Monitored OutputStates

The monitored_output_states argument of the constructor specifies the OutputStates it monitors. This takes the place of the input_states argument used by most other forms of Mechanism, and is used by the ObjectiveMechanism to create an InputState for each OutputState it monitors, along with a MappingProjection from the OutputState to that InputState. The monitored_output_states argument takes a list of items that can include any of the forms of specification used in a standard input_states argument. For the monitored_output_states argument, this is usually a list of OutputStates to be monitored. However, as with a standard input_states argument, items in the monitored_output_states argument can also be used to specify attributes of the InputState and/or MappingProjection to it created that the ObjectiveMechanism creates to monitor the specified OutputState. In general, the value of each specified OutputState determines the format of the variable of the InputState that is created for it by the ObjectiveMechanism. However, this can be overridden using the ObjectiveMechanism’s default_variable or size attributes (see Mechanism InputState specification), or by specifying a Projection from the OutputState to the InputState (see Input Source Specification). If an item in the monitored_output_states argument specifies an InputState for the ObjectiveMechanism, but not the OutputState to be monitored, the InputState is created but will be ignored until an OutputState (and MappingProjection from it) are specified for that InputState.

The list of OutputStates monitored by the ObjectiveMechanism are listed in its monitored_output_states attribute. When an ObjectiveMechanism is created by a Control Mechanism, or a System for its controller, these may pass a set of OutputStates to be monitored to the ObjectiveMechanism. A ControlMechanism passes OutputState specifications listed in its objective_mechanism argument (see ObjectiveMechanism and Monitored OutputStates), and a System passes any listed in its monitor_for_control argument (see Specifying Control).

Structure

Input

An ObjectiveMechanism has one InputState for each of the OutputStates specified in its monitored_output_states argument (see Monitored OutputStates). Each InputState receives a MappingProjection from the corresponding OutputState, the values of which are used by the ObjectiveMechanism’s function to generate the value of its OUTCOME OutputState. The InputStates are listed in the ObjectiveMechanism’s input_states attribute, and the monitored OutputStates from which they receive projections are listed in the same order its monitored_output_states attribute.

By default, the format of the variable for each InputState is determined by the value of the monitored OutputState to which it corresponds. However, if either the default_variable or size argument is specified in an Objective Mechanism’s constructor, or a variable is specified for an InputState for one or more of the items in its monitored_output_states argument, then that is used as the format for the corresponding InputState(s). This can be used to transform the value of a monitored OutputState into different form for the variable of the InputState (see the first example below).

If the weight and/or exponent is specified for ay item in the monitored_output_states argument of the ObjectiveMechanism’s constructor, it is assigned to the corresponding InputState. If the ObjectiveMechanism’s function implements a weights and/or exponents attribute, the values specified is assigned to the corresponding attribute, and applied to the value of the InputState before it is combined with that of the other InputStates to generate the ObjectiveMechanism’s output.

Function

The ObjectiveMechanism’s function uses the values of its input_states to compute an objective (or “loss”) function, that is assigned as the value of its OUTCOME OutputState. By default, it uses a LinearCombination function to sum the values of the values of the OutputStates listed in monitored_output_states. However, by assigning values to the ‘weight <InputState.weight>` and/or ‘exponent <InputState.exponent>` attributes of the corresponding InputStates, it can be configured to calculate differences, ratios, etc. (see example below). The function can also be replaced with any CombinationFunction, or any python function that takes an ndarray as its input (with a number of items in axis 0 equal to the number of the ObjectiveMechanism’s InputStates), and generates a 1d array as its result. If it implements weight and/or exponent attributes, those are assigned from weight and exponent attributes of its input_states (also listed in the monitored_output_states_weights_and_exponents attribute); otherwise, they are ignored.

Output

The primary OutputState of an Objective mechanism is a 1d array, named OUTCOME, that is the result of its function (as described above).

Execution

When an ObjectiveMechanism is executed, it updates its input_states with the values of the OutputStates listed in its monitored_output_states attribute, and then uses its function to evaluate these. The result is assigned as to its value attribute as the value of its OUTCOME (primary) OutputState.

Examples

Specifying the variable for the InputStates of an ObjectiveMechanism

This can be useful in some situations, and there are several ways it can be done. For example, for Reinforcement Learning, an ObjectiveMechanism is used to monitor the rewards by an action selection Mechanism (and used by a LearningMechanism to improve those predictions). However, whereas the action selection Mechanism generates a vector indicating the reward predicted by the selected action, the ObjectiveMechanism for RL simply needs to know the magnitude of the reward predicted, irrespective of the action taken; that is, it simply requires a single scalar value indicating the magnitude of the predicted reward. Thus, the vector of action-related reward predictions needs to be condensed to a single predicted value for the ObjectiveMechanism. This can be accomplished in several ways, that are illustrated in the examples below. In the first example, a TransferMechanism with the SoftMax function (and the PROB as its output format) to select an action and represent its reward prediction. This generates a vector with a single non-zero value, which designates the predicted reward for the selected action. Because the output is a vector, by default the InputState of the ObjectiveMechanism created to monitor it will also be a vector. However, the ObjectiveMechanism requires this to be a single value, that it can compare with the value of the reward Mechanism (monitoring the feedback provided by the environment). In the example below, this is accomplished by using default_variable in the constructor of the ObjectiveMechanism to force the InputState for the ObjectiveMechanism to have a single value:

>>> import psyneulink as pnl
>>> my_action_select_mech = pnl.TransferMechanism(default_variable=[0, 0, 0],
...                                               function=pnl.SoftMax(output=pnl.PROB),
...                                               name='Action Selection Mech')

>>> my_reward_mech = pnl.TransferMechanism(default_variable=[0],
...                                        name='Reward Mech')

>>> my_objective_mech = pnl.ObjectiveMechanism(default_variable=[[0],[0]],
...                                            monitored_output_states=[my_action_select_mech,
...                                                                     my_reward_mech])

Note that the OutputStates for the my_action_selection and my_reward_mech are specified in monitored_output_states. If that were the only specification, the InputState created for my_action_select_mech would be a vector of length 3. This is overridden by specifying default_variable as an array with two single-value arrays (one corresponding to my_action_select_mech and the other to my_reward_mech). This forces the InputState for my_action_select_mech to have only a single element which, in turn, will cause a MappingProjection to be created from my_action_select_mech to the ObjectiveMechanism’s InputState using a FULL_CONNECTIVITY_MATRIX (the one used for AUTO_ASSIGN_MATRIX when the sender and receiver have values of different lengths). This produces the desired effect, since the action selected is the only non-zero value in the output of my_action_select_mech, and so the FULL_CONNECTIVITY_MATRIX will combine it with zeros (the other values in the vector), and so its value will be assigned as the value of the corresponding InputState in the ObjectiveMechanism.

An alternative would be to explicitly specify the variable attribute for the InputState created for my_action_select_mech using a InputState specification dictionary in the monitored_output_states argument of my_objective_mech, as follows:

>>> my_objective_mech = pnl.ObjectiveMechanism(monitored_output_states=[{pnl.MECHANISM: my_action_select_mech,
...                                                                      pnl.VARIABLE: [0]},
...                                                                     my_reward_mech])

Note that the VARIABLE entry here specifies the variable for the InputState of the ObjectiveMechanism created to receive a Projection from my_action_select_mech, and not my_action_select_mech itself (see Input for a full explanation).

Another way to specify the variable for the InputState of an ObjectiveMechanism is to specify the Projections it receives from the OutputState it monitors. The following example uses a tuple specification to assign the matrix for the MappingProjection from my_action_select_mech to the corresponding InputState of my_objective_mech:

>>> import numpy as np
>>> my_objective_mech = pnl.ObjectiveMechanism(monitored_output_states=[(my_action_select_mech, np.ones((3,1))),
...                                                                     my_reward_mech])

Since the matrix specified has three rows (for its inputs) and one col (for the output), it will take the length three vector provided as the output of my_action_select_mech and combine its elements into a single value that is provided to the InputState of the ObjectiveMechanism.

A Connection tuple could also be used to specify the matrix, but this would require that additional entries (for the weight and exponent of the InputState) which, in this case, are not necessary (see example below for how these can be used).

By default, an ObjectiveMechanism simply adds the values received by each of its InputStates to generate its output. However, this too can be customized in a variety of ways.

The simplest way is to assign values to the weight and/or exponent attributes of its InputStates. This can be done by placing them in a tuple specification for the OutputState that provides value for the InputState. In the example below, the ObjectiveMechanism used in the previous example is further customized to subtract the value of the action selected from the value of the reward:

>>> my_objective_mech = pnl.ObjectiveMechanism(default_variable = [[0],[0]],
...                                            monitored_output_states = [(my_action_select_mech, -1, 1),
...                                                                       my_reward_mech])

This specifies that my_action_select_mech should be assigned a weight of -1 and an exponent of 1 when it is submitted to the ObjectiveMechanism’s function. Notice that the exponent had to be included, even though it is the default value; when a tuple is used, the weight and exponent values must both be specified. Notice also that my_reward_mech does not use a tuple, so it will be assigned defaults for both the weight and exponent parameters.

An ObjectiveMechanism can also be configured to monitor multiple OutputStates of the same Mechanism. In the following example, an ObjectiveMechanism is configured to calculate the reward rate for a DDM Mechanism, by specifying OutputStates for the DDM that report its response time and accuracy:

>>> my_decision_mech = pnl.DDM(output_states=[pnl.RESPONSE_TIME,
...                                           pnl.PROBABILITY_UPPER_THRESHOLD])

>>> my_objective_mech = pnl.ObjectiveMechanism(monitored_output_states=[
...                                              my_reward_mech,
...                                              my_decision_mech.output_states[pnl.PROBABILITY_UPPER_THRESHOLD],
...                                              (my_decision_mech.output_states[pnl.RESPONSE_TIME], 1, -1)])

This specifies that the ObjectiveMechanism should multiply the value of my_reward_mech’s primary OutputState by the value of my_decision_mech’s PROBABILITY_UPPER_THRESHOLD, and divide the result by my_decision_mech’s RESPONSE_TIME value. The two OutputStates of my_decision_mech are referenced as items in the output_states list of my_decision_mech. However, a 2-item (State name, Mechanism) tuple can be used to reference them more simply, as follows:

>>> my_objective_mech = pnl.ObjectiveMechanism(monitored_output_states=[
...                                           my_reward_mech,
...                                           (pnl.PROBABILITY_UPPER_THRESHOLD, my_decision_mech),
...                                           ((pnl.RESPONSE_TIME, my_decision_mech), 1, -1)])

Customizing the ObjectiveMechanism’s function

In the examples above, the weights and exponents assigned to the InputStates are passed to the ObjectiveMechanism’s function for use in combining their values. The same can be accomplished by specifying the relevant parameter(s) of the function itself, as in the following example:

>>> my_objective_mech = pnl.ObjectiveMechanism(default_variable = [[0],[0]],
...                                            monitored_output_states = [my_action_select_mech, my_reward_mech],
...                                            function=pnl.LinearCombination(weights=[[-1], [1]]))

Here, the weights parameter of the LinearCombination function is specified directly, with two values [-1] and [1] corresponding to the two items in monitored_output_states (and default_variable). This will multiply the value from my_action_select_mech by -1 before adding it to (and thus subtracting it from) the value of my_reward_mech. Notice that the weight for my_reward_mech had to be specified, even though it is using the default value (1); whenever a weight and/or exponent parameter is specified, there must be an entry for every item of the function’s variable. However, the exponents did not need to be specified, as it is not used. However it, and the operation parameters of LinearCombination can also be used to multiply and divide quantities.

Class Reference

class psyneulink.components.mechanisms.processing.objectivemechanism.OBJECTIVE_OUTPUT

Standard OutputStates for ObjectiveMechanism:

OUTCOME : 1d np.array
the value of the objective or “loss” function computed based on the ObjectiveMechanism’s function
class psyneulink.components.mechanisms.processing.objectivemechanism.ObjectiveMechanism(monitored_output_states, default_variable, size, function=LinearCombination, output_states=OUTCOME, params=None, name=None, prefs=None)

Subclass of ProcessingMechanism that evaluates the value(s) of one or more OutputStates.

Parameters:
  • monitored_output_states (List[OutputState, Mechanisms, str, value, dict, MonitoredOutputStatesOption] or dict) – specifies the OutputStates, the values of which will be monitored, and evaluated by the ObjectiveMechanism’s function (see Monitored OutputStates for details of specification).
  • default_variable (number, list or np.ndarray : default monitored_output_states) – specifies the format of the variable for the InputStates of the ObjectiveMechanism (see Mechanism InputState specification for details).
  • size (int, list or np.ndarray of ints) –

    specifies default_variable as array(s) of zeros if default_variable is not passed as an argument; if default_variable is specified, it takes precedence over the specification of size. As an example, the following mechanisms are equivalent:

    T1 = TransferMechanism(size = [3, 2])
    T2 = TransferMechanism(default_variable = [[0, 0, 0], [0, 0]])
    
  • function (CombinationFunction, ObjectiveFunction, function or method : default LinearCombination) – specifies the function used to evaluate the values listed in monitored_output_states (see function for details.
  • output_states (List[OutputState, value, str or dict] or Dict[] : default [OUTCOME]) – specifies the OutputStates for the Mechanism;
  • role (Optional[LEARNING, CONTROL]) – specifies if the ObjectiveMechanism is being used for learning or control (see role for details).
  • params (Dict[param keyword: param value] : default None) – a parameter dictionary that can be used to specify the parameters for the Mechanism, its function, and/or a custom function and its parameters. Values specified for parameters in the dictionary override any assigned to those parameters in arguments of the constructor.
  • name (str : default see name) – specifies the name of the ObjectiveMechanism.
  • prefs (PreferenceSet or specification dict : default Mechanism.classPreferences) – specifies the PreferenceSet for the ObjectiveMechanism; see prefs for details.
variable

2d ndarray : default array of values of OutputStates in monitor_output_states – the input to Mechanism’s function.

monitored_output_states

ContentAddressableList[OutputState] – determines the OutputStates, the values of which are monitored, and evaluated by the ObjectiveMechanism’s function. Each item in the list refers to an OutputState containing the value to be monitored, with a MappingProjection from it to the corresponding InputState listed in the input_states attribute.

monitored_output_states_weights_and_exponents

List[Tuple(float, float)] – each tuple in the list contains a weight and exponent associated with a corresponding InputState listed in the ObjectiveMechanism’s input_states attribute; these are used by its function to parametrize the contribution that the values of each of the OuputStates monitored by the ObjectiveMechanism makes to its output (see Function)

input_states

ContentAddressableList[InputState] – contains the InputStates of the ObjectiveMechanism, each of which receives a MappingProjection from the OutputStates specified in its monitored_output_states attribute.

function

CombinationFunction, ObjectiveFunction, function, or method – the function used to compare evaluate the values monitored by the ObjectiveMechanism. The function can be any PsyNeuLink CombinationFunction or a Python function that takes a 2d array with an arbitrary number of items or a number equal to the number of items in the ObjectiveMechanism’s variable (and its number of input_states), and returns a 1d array.

role

None, LEARNING or CONTROL – specifies whether the ObjectiveMechanism is used for learning in a Process or System (in conjunction with a ObjectiveMechanism), or for control in a System (in conjunction with a ControlMechanism).

value

1d np.array – the output of the evaluation carried out by the ObjectiveMechanism’s function.

output_state

OutputState – contains the primary OutputState of the ObjectiveMechanism; the default is its OUTCOME OutputState, the value of which is equal to the value attribute of the ObjectiveMechanism.

output_states

ContentAddressableList[OutputState] – by default, contains only the OUTCOME (primary) OutputState of the ObjectiveMechanism.

output_values

2d np.array – contains one item that is the value of the OUTCOME OutputState.

name

str – the name of the ObjectiveMechanism; if it is not specified in the name argument of the constructor, a default is assigned by MechanismRegistry (see Naming for conventions used for default and duplicate names).

prefs

PreferenceSet or specification dict – the PreferenceSet for the ObjectiveMechanism; if it is not specified in the prefs argument of the constructor, a default is assigned using classPreferences defined in __init__.py (see PreferenceSet for details).

add_monitored_output_states(monitored_output_states_specs, context=None)

Instantiate OutputStates to be monitored by the ObjectiveMechanism.

Used by other Components to add a States or list of States to be monitored by the ObjectiveMechanism. The monitored_output_states_spec can be any of the following: - MonitoredOutputStateTuple - Mechanisms; - OutputState; - tuple specification; - State specification dictionary; - list with any of the above. If the item is a Mechanism, its primary OutputState is used.