Table of Contents

PredictionErrorMechanism

Contents

Overview

A PredictionErrorMechanism is a subclass of ComparatorMechanism that receives two inputs (a sample and a target), and calculates the temporal difference prediction error as found in Montague, Dayan, and Sejnowski (1996) using its function, and places the delta values (the difference between the actual and predicted reward) in its OUTCOME OutputPort.

Creating a PredictionErrorMechanism

A PredictionErrorMechanism is usually created automatically when a LearningMechanism is created using the TDLearning function). A PredictionErrorMechanism can also be created directly by calling its constructor. Its sample and target arguments are used to specify the OutputPorts that provide the sample and target inputs, respectively (see ObjectiveMechanism Monitored Output Ports for details). When the PredictionErrorMechanism is created, two InputPorts are created, one each for its sample and target inputs (and named, by default SAMPLE and TARGET). Each is assigned a MappingProjection from the corresponding OutputPort specified in the sample and target arguments.

It is important to recognize that the value of the SAMPLE and TARGET InputPorts must have the same length and type, so that they can be compared using the PredictionErrorMechanism’s function <PredictionErrorMechanism.function>. By default, they use the format of the OutputPorts specified in the sample and target arguments, respectively, and the MappingProjection to each uses an IDENTITY_MATRIX. Therefore, for the default configuration, the OutputPorts specified in the sample and target arguments must have values of the same length and type. If these differ, the input_ports argument can be used to explicitly specify the format of the PredictionErrorMechanism’s SAMPLE and TARGET InputPorts, to insure they are compatible with one another (as well as to customize their names, if desired). If the input_ports argument is used, both the sample and target InputPorts must be specified. Any of the formats for specifying InputPorts can be used in the argument. If values are assigned for the InputPorts, they must be of equal length and type. Their types must also be compatible with the value of the OutputPorts specified in the sample and target arguments. However, the length specified for an InputPort can differ from its corresponding OutputPort; in that case, by default, the MappingProjection created uses a FULL_CONNECTIVITY matrix. Thus, OutputPorts of differing lengths can be mapped to the sample and target InputPorts of a PredictionErrorMechanism (see the example below), so long as the latter of of the same length. If a projection other than a FULL_CONNECTIVITY matrix is needed, this can be specified using the PROJECTION entry of a Port specification dictionary for the InputPort in the input_ports argument.

Structure

A PredictionErrorMechanism has two input_ports, each of which receives a MappingProjection from a corresponding OutputPort specified in the sample and target arguments of its constructor. The InputPorts are listed in the Mechanism’s input_ports attribute and named, respectively, SAMPLE and TARGET. The OutputPorts from which they receive their projections (specified in the sample and target arguments) are listed in the Mechanism’s sample and target attributes as well as in its monitored_output_ports attribute. The PredictionErrorMechanism’s function calculates the difference between the predicted reward and the true reward at each timestep in SAMPLE. By default, it uses a PredictionErrorDeltaFunction. However, the function can be customized, so long as it is replaced with one that takes two arrays with the same format as its inputs and generates a similar array as its result. The result is assigned as the value of the PredictionErrorMechanism’s OUTCOME (primary) OutputPort.

Execution

When a PredictionErrorMechanism is executed, it updates its input_ports with the values of the OutputPorts specified in its sample and target arguments, and then uses its function to compare these. By default, the result is assigned to the value of its OUTCOME output_port, and as the first item of the Mechanism’s output_values attribute.

Example

Formatting InputPort values

The default_variable argument can be used to specify a particular format for the SAMPLE and/or TARGET InputPorts of a PredictionErrorMechanism. This can be useful when one or both of these differ from the format of the OutputPort(s) specified in the sample and target arguments. For example, for Temporal Difference Learning, a PredictionErrorMechanism is used to compare the predicted reward from the sample with the true reward (the target). In the example below, the sample Mechanism is a TransferMechanism that uses the Linear function to output the sample values. Because the output is a vector, specifying it as the PredictionErrorMechanism’s sample argument will generate a corresponding InputPort with a vector as its value. This should match the reward signal specified in the PredictionErrorMechanism’s target argument, the value of which is a vector of the same length as the output of sample.

>>> import psyneulink as pnl
>>> sample_mech = pnl.TransferMechanism(input_shapes=5,
...                                     function=pnl.Linear())
>>> reward_mech = pnl.TransferMechanism(input_shapes=5)
>>> prediction_error_mech = pnl.PredictionErrorMechanism(sample=sample_mech,
...                                                      target=reward_mech)

Note that sample_mech is specified to take an array of length 5 as its input, and therefore generate one of the same length as its primary output. Since it is assigned as the sample of the PredictionErrorMechanism, by default this will create a SAMPLE InputPort of length 5, that will match the length of the TARGET InputPort.

Currently the default method of implementing temporal difference learning in PsyNeuLink requires the values of SAMPLE and TARGET to be provided as an array representing a full time series as an experiment. See MontagueDayanSejnowski.py in the Scripts folder for an example.

Class Reference

class psyneulink.library.components.mechanisms.processing.objective.predictionerrormechanism.PredictionErrorMechanism(sample=None, target=None, function=None, output_ports=None, learning_rate=None, params=None, name=None, prefs=None, **kwargs)

Subclass of ComparatorMechanism that calculates the prediction error between the predicted reward and the target. See ComparatorMechanism for additional arguments and attributes.

Parameters

  • sample (OutputPort, Mechanism_Base, dict, number, or str) – specifies the SAMPLE InputPort, that is evaluated by the function.

  • target (OutputPort, Mechanism_Base, dict, number, or str) – specifies the TARGET InputPort used by the function to evaluate sample.

  • function (TransformFunction, ObjectiveFunction, function, or method : default PredictionErrorDeltaFunction) – the function used to evaluate the SAMPLE and TARGET inputs.

  • learning_rate (Number : default 0.3) – controls the weight of later timesteps compared to earlier ones; higher rates weight later timesteps more heavily than previous ones.

sample

the SAMPLE InputPort, the value of which will be evaluated by the function.

Type

OutputPort, Mechanism_Base, dict, number, or str

target

the TARGET InputPort, the value of which will be used to evaluate sample.

Type

OutputPort, Mechanism_Base, dict, number, or str

function

the function used to evaluate the sample and target inputs.

Type

TransformFunction, ObjectiveFunction, Function, or method : default PredictionErrorDeltaFunction

output_ports

by default, contains only the OUTCOME (primary) OutputPort of the PredictionErrorMechanism.

Type

str, Iterable : default OUTCOME

learning_rate

controls the weight of later timesteps compared to earlier ones; higher rates weight later timesteps more heavily than previous ones.

Type

Number : default 0.3

_parse_function_variable(variable, context=None)

Parses the variable passed in to a Component into a function_variable that can be used with the Function associated with this Component

exception psyneulink.library.components.mechanisms.processing.objective.predictionerrormechanism.PredictionErrorMechanismError(error_value)