PredictionErrorMechanism

Overview

A PredictionErrorMechanism is a subclass of ComparatorMechanism that receives two inputs (a sample and a target), and calculates the temporal difference prediction error as found in Montague, Dayan, and Sejnowski (1996) using its function, and places the delta values (the difference between the actual and predicted reward) in its OUTCOME OutputState.

Creating a PredictionErrorMechanism

A PredictionErrorMechanism is usually created automatically when a Learning Mechanism is created using the TDLearning function). A PredictionErrorMechanism can also be created directly by calling its constructor. Its sample and target arguments are used to specify the OutputStates that provide the sample and target inputs, respectively (see ObjectiveMechanism Monitored Output States for details). When the PredictionErrorMechanism is created, two InputStates are created, one each for its sample and target inputs (and named, by default SAMPLE and TARGET). Each is assigned a MappingProjection from the corresponding OutputState specified in the sample and target arguments.

It is important to recognize that the value of the SAMPLE and TARGET InputStates must have the same length and type, so that they can be compared using the PredictionErrorMechanism’s function By default, they use the format of the OutputStates specified in the sample and target arguments, respectively, and the MappingProjection to each uses an IDENTITY_MATRIX. Therefore, for the default configuration, the OutputStates specified in the sample and target arguments must have values of the same length and type. If these differ, the input_states argument can be used to explicitly specify the format of the PredictionErrorMechanism’s SAMPLE and TARGET InputStates, to insure they are compatible with one another (as well as to customize their names, if desired). If the input_states argument is used, both the sample and target InputStates must be specified. Any of the formats for specifying InputStates can be used in the argument. If values are assigned for the InputStates, they must be of equal length and type. Their types must also be compatible with the value of the OutputStates specified in the sample and target arguments. However, the length specified for an InputState can differ from its corresponding OutputState; in that case, by default, the MappingProjection created uses a FULL_CONNECTIVITY matrix. Thus, OutputStates of differing lengths can be mapped to the sample and target InputStates of a PredictionErrorMechanism (see the example below), so long as the latter of of the same length. If a projection other than a FULL_CONNECTIVITY matrix is needed, this can be specified using the PROJECTION entry of a State specification dictionary for the InputState in the input_states argument.

Structure

A PredictionErrorMechanism has two input_states, each of which receives a MappingProjection from a corresponding OutputState specified in the sample and target arguments of its constructor. The InputStates are listed in the Mechanism’s input_states attribute and named, respectively, SAMPLE and TARGET. The OutputStates from which they receive their projections (specified in the the sample and target arguments) are listed in the Mechanism’s sample and target attributes as well as in its monitored_output_states attribute. The PredictionErrorMechanism’s function calculates the difference between the predicted reward and the true reward at each timestep in SAMPLE. By default, it uses a PredictionErrorDeltaFunction. However, the function can be customized, so long as it is replaced with one that takes two arrays with the same format as its inputs and generates a similar array as its result. The result is assigned as the value of the PredictionErrorMechanism’s OUTCOME (primary) OutputState.

Execution

When a PredictionErrorMechanism is executed, it updates its input_states with the values of the OutputStates specified in its sample and target arguments, and then uses its function to compare these. By default, the result is assigned to the value of its OUTCOME output_state, and as the first item of the Mechanism’s output_values attribute.

Example

Formatting InputState values

The default_variable argument can be used to specify a particular format for the SAMPLE and/or TARGET InputStates of a PredictionErrorMechanism. This can be useful when one or both of these differ from the format of the OutputState(s) specified in the sample and target arguments. For example, for Temporal Difference Learning, a PredictionErrorMechanism is used to compare the predicted reward from the sample with the true reward (the target). In the example below, the sample Mechanism is a TransferMechanism that uses the Linear function to output the sample values. Because the output is a vector, specifying it as the PredictionErrorMechanism’s sample argument will generate a corresponding InputState with a vector as its value. This should match the the reward signal specified in the PredictionErrorMechanism’s target argument, the value of which is a vector of the same length as the output of sample.

>>> import psyneulink as pnl
>>> sample_mech = pnl.TransferMechanism(size=5,
...                                     function=pnl.Linear())
>>> reward_mech = pnl.TransferMechanism(size=5)
>>> prediction_error_mech = pnl.PredictionErrorMechanism(sample=sample_mech,
...                                                      target=reward_mech)

Note that sample_mech is specified to take an array of length 5 as its input, and therefore generate one of the same length as its primary output. Since it is assigned as the sample of the PredictionErrorMechanism, by default this will create a SAMPLE InputState of length 5, that will match the length of the TARGET InputState.

Currently the default method of implementing temporal difference learning in PsyNeuLink requires the values of SAMPLE and TARGET to be provided as an array representing a full time series as an experiment. See MontagueDayanSejnowski.py in the Scripts folder for an example.

Class Reference

class psyneulink.library.mechanisms.processing.objective.predictionerrormechanism.PredictionErrorMechanism( sample, target, function=PredictionErrorDeltaFunction, output_states=[OUTCOME], params=None, name=None, prefs=None)

Calculates the prediction error between the predicted reward and the target

Parameters:
  • sample (OutputState, Mechanism_Base, dict, number, or str) – specifies the SAMPLE InputState, which will be evaluated by the function
  • target (OutputState, Mechanism_Base, dict, number, or str) – specifies the TARGET InputState, which will be used by the function to evaluate the sample
  • function (CombinationFunction, ObjectiveFunction, function, or method : default PredictionErrorDeltaFunction) – the function used to evaluate the sample and target inputs.
  • output_states (str, Iterable : default OUTCOME) – by default, contains only the OUTCOME (primary) OutputState of the PredictionErrorMechanism.
  • learning_rate (Number : default 0.3) – controls the weight of later timesteps compared to earlier ones. Higher rates weight later timesteps more heavily than previous ones.
  • name (str) – the name of the PredictionErrorMechanism; if it is not specified in the name argument of the constructor, a default is assigned by MechanismRegistry (see Naming for conventions used for default and duplicate names).
sample

OutputState, Mechanism_Base, dict, number, or str – specifies the SAMPLE InputState, which will be evaluated by the function

target

OutputState, Mechanism_Base, dict, number, or str – specifies the TARGET InputState, which will be used by the function to evaluate the sample

function

CombinationFunction, ObjectiveFunction, Function, or method : default PredictionErrorDeltaFunction – the function used to evaluate the sample and target inputs.

output_states

str, Iterable : default OUTCOME – by default, contains only the OUTCOME (primary) OutputState of the PredictionErrorMechanism.

learning_rate

Number : default 0.3 – controls the weight of later timesteps compared to earlier ones. Higher rates weight later timesteps more heavily than previous ones.

name

str – the name of the PredictionErrorMechanism; if it is not specified in the name argument of the constructor, a default is assigned by MechanismRegistry (see Naming for conventions used for default and duplicate names).