# Learning Mechanism¶

## Overview¶

A LearningMechanism is a ModulatoryMechanism that modifies the matrix parameter of one or more MappingProjections. Its function takes one or more error_signals (usually the output of a ComparatorMechanism or one or more other LearningMechanisms), as well as information about the MappingProjection(s) and activity that generated the error(s), and calculates a learning_signal that is used to modify the MappingProjection(s) by way of its LearningProjection(s). Typically, a LearningMechanism is used to “train” a single MappingProjection (its primary_learned_projection), using the output of the Mechanism to which that MappingProjection projects (its output_source) as the source of the error it attempts to reduce. A LearningMechanism can be used to train multiple MappingProjections, by assigning it additional LearningProjections; however, these will all use the same learning_signal, generated by the primary_learned_projection and its associated output_source. All of the MappingProjection(s) modified by a LearningMechanism must project from one ProcessingMechanism to another in the same Composition. The learning components of a Composition can be displayed using the System’s show_graph method with its show_learning argument assigned True or ALL.

### A Note about the Implementation of Learning¶

The implementation of learning in PsyNeuLink was designed for exposition rather than efficiency. Unlike its implementation in most other environments – where the learning algorithm is tightly integrated with the elements of processing that it modifies – PsyNeuLink separates it into three constituent components: An ObjectiveMechanism used to evaluate the most proximal source of error, a Learning Mechanism that uses that error (or one derived from it by another LearningMechanism) to calculate a learning signal; and LearningProjection(s) that use that learning signal to modify the weight matrix of the MappingProjection(s) being learned. This has the advantage of isolating and exposing the constituent computations, making it clearer to students what these are and how they operate, and also making each individually accessible for reconfiguration. However, it comes at the cost of efficiency. For efficient execution of supervised forms of learning (e.g., reinforcement learning and backpropagation), the AutodiffComposition can be used, which allows the model to be specified using PsyNeuLink, but actually executes learning using PyTorch.

## Creating a LearningMechanism¶

A LearningMechanism can be created in any of the ways used to create Mechanisms. More commonly, however, LearningMechanisms are created automatically.

### Automatic Creation¶

A LearningMechanism is created automatically when a Composition’s learning method is called. In that case, a LearningSignal, LearningProjection, a ComparatorMechanism (in the case of supervised learning), and any additional Projections required to implement learning that do not already exist are also instantiated. This is described below, under Learning Configurations. A LearningMechanism is also created automatically when either the tuple specification is used to specify learning for a MappingProjection, or a LearningProjection is created without specifying its sender attribute. However, this is not advised, and should only used in special circumstances, as properly configuring learning generally requires the instantiation of several other closely related Components, as described below.

### Explicit Creation¶

If a LearningMechanism is created explicitly (using its constructor), then its variable and error_sources arguments must be specified. The variable must have at leaset three items that are compatible (in number and type) with the value of the LearningMechanism’s InputPorts. Each item in error_sources must be one of the following: a ComparatorMechanism, for single layer learning or for the last MappingProjection in a learning Pathway for multilayer learning; or a Learning Mechanism.

When a LearningMechanism is created explicitly, it can also be assigned existing LearningSignals and/or specified to create these, as well as LearningProjections from these to specified MappingProjections. These are specified in the learning_signals argument of the LearningMechanism’s constructor, using any of the forms allowed for specifying a LearningSignal.

## Structure¶

A LearningMechanism has three types of InputPorts, a learning function, and two types of OutputPorts. These are used, respectively, to receive, compute, and transmit the information needed to modify the MappingProjection(s) for which the LearningMechanism is responsible. In addition, it has several attributes that govern and provide access to its operation. These are described below.

### InputPorts¶

These receive the information required by the LearningMechanism’s function. They are listed in the LearningMechanism’s input_ports attribute. They have the following names and roles (shown in the figure below):

The Mechanisms from the which the values above are received are listed in the LearningMechanism’s input_source, output_source, and error_sources attributes, respectively (see Additional Attributes for additional details).

### Learning Function¶

The function of a LearningMechanism uses the values received by the Mechanism’s InputPorts (described above) to calculate the value of its learning_signal and error_signal attributes.

The default function of a LearningMechanism is BackPropagation (also known as the Generalized Delta Rule; see Rumelhart et al., 1986). However, it can be assigned any other PsyNeuLink LearningFunction, or any other Python function that takes as its input a list or np.array containing three lists or 1d np.arrays of numeric values, and returns two lists or 1d np.arrays. The two values it returns are assigned to the LearningMechanism’s learning_signal and error_signal attributes, respectively, as well as to its two OutputPorts, as described below.

### OutputPorts¶

By default, a LearningMechanism has two OutputPorts, the first of which is named ERROR_SIGNAL and is assigned the value of the error_signal returned by the LearningMechanism’s function, and the second of which is a LearningSignal and is assigned the value of the learning_signal returned by the function. They are each described below:

Multiple LearningSignals and LearningProjections. Though not common, it is possible for a LearningMechanism to be assigned more than one LearningSignal, and/or more than one LearningProjection to its LearningSignal(s). This allows multiple MappingProjections to be trained by a single LearningMechanism. Note, however, that all of the LearningSignals of a LearningMechanism (and therefore its LearningProjections) use the same learning_signal, that is calculated based on the LearningMechanism’s primary_learned_projection and its associated output_source. This can be useful in some settings, such as for certain forms of convolutional neural networks.

If all of the LearningProjections are used to implement the same form of modulation, (determined by their LearningSignals’ modulation attribute), then they should be assigned to a single LearningSignal. Multiple LearningProjections can be assigned to a LearningSignal by specifying them in the projections argument of its constructor, or the PROJECTIONS entry of a dictionary assigned to its params argument); however, the matrix parameter for the MappingProjection to which they project must have the same shape for all of them. If different forms of modulation are required to train different MappingProjections, then multiple LearningSignals should be assigned to the LearningMechanism, each specified for one of the required types of modulation, and then LearningProjections assigned to them for the corresponding MappingProjections. Multiple LearningSignals can be specified for a LearningMechanism by including them in a list assigned to the learning_signals argument of the LearningMechanism’s constructor.

The learning_rate for each LearningSignal, and the learning_rate for each of its LearningProjections, can all be assigned different values (with the latter taking precedence over the former). If none of these are specified, the learning_rate of the LearningMechanism is used (see below).

All of the LearningSignals of a LearningMechanism are listed in its learning_signals attribute. Because these are OutputPorts, they are also listed in the output_ports attribute, after the ERROR_SIGNAL OutputPort. All of the LearningMechanism’s LearningProjections (that is, those belonging to all of its LearningSignals) are listed in its learning_projections attribute.

In addition to its InputPorts, function and OutputPorts, a LearningMechanism has the following attributes that refer to the Components being learned and/or its operation:

### Learning Configurations¶

When learning is enabled for a RecurrentTransferMechanism (for unsupervised learning) or using the learning method of a Composition, all of the Components required for learning are created automatically. The types of Components that are generated depend on the type of learning specified and the configuration of the Composition, as described below. All of the learning Components of a Composition can be displayed using its show_graph method with the show_learning argument assigned True or ALL.

#### Single layer learning¶

This configuration occurs when unsupervised learning is used, or supervised learning is used for a pathway in a Composition with only two Mechanisms (i.e., for the Projection between them). In this case, a single ComparatorMechanism and LearningMechanism are created (if they do not already exist) as well as the following MappingProjections:

• from the ComparatorMechanism’s OUTCOME OutputPort to the LearningMechanism’s ERROR_SIGNAL InputPort.

In addition, a LearningProjection is created from the LearningSignal for the primary_learned_projection to the ParameterPort for the matrix of the primary_learned_projection. Because this configuration involves only a single layer of learning, no Projection is created or assigned to the LearningMechanism’s ERROR_SIGNAL OutputPort.

Components for Single Layer Learning

#### Multilayer learning¶

This refers to learning in a pathway that has three or more Mechanisms in a sequence, and therefore two or more MappingProjections that are being learned. The learning components for such a configuration are created automatically when a learning method of a Composition is used that supports multilayer learning (at present, this is only add_backpropagation_learning_pathway). If the learning components are constructed explicitly for a multilayer learning configuration, then each LearningMechanism must use a LearningFunction that can calculate the influence that each MappingProjection and its output have on the error that the LearningMechanism receives from the previous one(s) in the sequence (e.g., BackPropagation). Furthermore, the construction of the other learning-related Components associated with each LearningMechanism depend on the position of its primary_learned_projection and output_source in the sequence. If it is the last one in the sequence, it is treated in the same way the LearningMechanism in a single layer learning configuration configuration. This is the case if the output_source is a standalone Mechanism or the TERMINAL Mechanism of a Composition. In these cases, as for single layer learning, a ComparatorMechanism is created that receives the output of the output_source as well as the target for learning (see TARGET Mechanisms below), and projects to a LearningMechanism that is created for the primary_learned_projection. For all other MappingProjections being learned in the sequence, the following additional MappingProjections are created for learning (shown in the figure below):

• from the ERROR_SIGNAL OutputPort of each of the LearningMechanism’s error_sources to each of its corresponding ERROR_SIGNAL InputPort(s).

In addition, a LearningProjection is created from the LearningSignal for the primary_learned_projection of each LearningMechanism in the sequence, to the ParameterPort for the matrix of the primary_learned_projection. If the primary_learned_projection is the first in the sequence, then no Projection is created or assigned to its LearningMechanism’s ERROR_SIGNAL OutputPort.

Components for Multilayer Learning

TARGET and OBJECTIVE Mechanisms**. When supervised learning is implemented using one of a Composition’s learning methods, it automatically creates a number of learning-related Components. This includes a TARGET_MECHANISM, that receives the target stimulus specifed in the inputs argument of the Composition’s learn method; and a OBJECTIVE_MECHANISM, that computes the error_signal for the sequence. The output of the OBJECTIVE_MECHANISM is the error_source for the last MappingProjection in the learning pathway. If a multilayer learning pathway is implemented explicitly, it must include these Components.

## Execution¶

LearningMechanisms that implement unsupervised learning (AutoassociativeLearningMechanisms) execute when the RecurrentTransferMechanism with which they are associated executes. LearningMechanisms that are part of a supervised learning pathway execute after all of the ProcessingMechanisms in the Composition to which they belong have executed. When a LearningMechanism is executed, it uses the value to calculate changes to the weights of the matrix parameter of its primary_learned_projection and any of its other learned_projections. Those weight changes are assigned as the LearningMechanism’s learning_signal attribute, the value of each of its LearningSignals, and as the value of each of their LearningProjections. That value is used, in turn, to modify the value of the MATRIX ParameterPort of each of the MappingProjections being learned (listed in the LearningMechanism’s learned_projections attribute).

Each ParameterPort uses the value it receives from the LearningProjection that projects to it to modify the parameter of its function, in a manner specified by the modulation attribute of the LearningSignal from which it receives the LearningProjection (see Modulation for a description of how modulation operates). By default, the modulation attribute of a LearningSignal is ADDITIVE, the function of a MATRIX ParameterPort for a MappingProjection is AccumulatorIntegrator, and the parameter it uses for additive modulation is its increment parameter. These assignments cause the value of the LearningProjection to be added to the previous value of the MATRIX ParameterPort, thus incrementing the weights by an amount specified by the LearningMechanism’s learning_signal. Note, that the changes to the matrix parameter itself do not take effect until the next time the learned_projection is executed (see Lazy Evaluation for an explanation of “lazy” updating).

A LearningMechanism’s function also computes an error_signal that is assigned as the value of its ERROR_SIGNAL OutputPort; in a multilayer learning configuration, that value is provided to the ERROR_SIGNAL InputPort(s) of the LearningMechanism(s) for the preceding MappingProjection(s) being learned in the sequence.

## Class Reference¶

psyneulink.core.components.mechanisms.modulatory.learning.learningmechanism.DefaultTrainingMechanism
class psyneulink.core.components.mechanisms.modulatory.learning.learningmechanism.LearningMechanism(variable, error_sources, function=BackPropagation, learning_rate=None, learning_signals=LEARNING_SIGNAL, modulation=ADDITIVE, learning_enabled=True)

Subclass of ModulatoryMechanism that modifies the matrix parameter of a MappingProjection. See Mechanism for additional arguments and attributes.

Parameters
variable

has three items that serve as the template for the three inputs required by the LearningMechanism’s function (corresponding to its three InputPorts: the input to the primary_learned_projection (from input_source), the output of the Mechanism to which that projects (i.e., of output_source); and the error signal (from LearningMechanism.error_sources).

Type

2d np.array

input_ports

list containing the LearningMechanism’s three InputPorts: ACTIVATION_INPUT, ACTIVATION_OUTPUT, and ERROR_SIGNAL.

Type

error_signal_input_ports

list of InputPorts that receive error_signals from the LearningMechanism’s error_sources.

Type

list[InputPorts]

input_source

the Mechanism that sends the primary_learned_projection, and projects to the LearningMechanism’s ACTIVATION_INPUT InputPort.

Type

ProcessingMechanism

output_source

the Mechanism that receives the primary_learned_projection, and projects to the LearningMechanism’s ACTIVATION_OUTPUT InputPort.

Type

ProcessingMechanism

error_sources

the Mechanism(s) that calculate the error signal(s) provided to the LearningMechanism’s ERROR_SIGNAL(s) InputPort(s).

Type
error_matrices

the matrices of the Projections associated with the error_sources, (i.e., for the next Projection(s) in the learning_sequence, or to the ComparatorMechanism); note: these are not for the LearningMechanism’s learned_projections.

Type

list[ParameterPort]

primary_learned_projection

the Projection with the matrix parameter used to generate the LearningMechanism’s error_signal and learning_signal attributes. It is always the first Projection listed in the LearningMechanism’s learned_projections attribute.

Type

MappingProjection

learned_projections

all of the MappingProjections modified by the LearningMechanism; the first item in the list is always the primary_learned_projection.

Type

List[MappingProjection]

function

specifies the function used to calculate the learning_signal (assigned to the LearningMechanism’s LearningSignal(s)), and the error_signal (passed to the LearningMechanism for the preceding MappingProjection in a multilayer learning pathway). It takes the following three arguments, each of which must be a list or 1d array: input, output, and error (see Learning Function for additional details).

Type

LearningFunction or function : default BackPropagation

learning_rate

determines the learning rate for the LearningMechanism. It is used to specify the learning_rate parameter for the LearningMechanism’s learning function (see description of learning_rate for additional details).

Type

float : None

error_signal

one of two values returned by the LearningMechanism’s function. For single layer learning, this is the same as the value received in the LearningMechanism’s ERROR_SIGNAL InputPort; for multilayer learning, it is a modified version of the value received, that takes account of the contribution made by the learned_projection and its input to the error signal received. This is assigned as the value of the LearningMechanism’s ERROR_SIGNAL OutputPort.

Type

1d np.array

learning_signal

one of two values returned by the LearningMechanism’s function, that specifies the changes to the weights of the matrix parameter for the LearningMechanism’s learned_projections; it is calculated to reduce the error signal associated with the primary_learned_projection and received from the LearningMechanism’s error_sources. It is assigned as the value of the LearningMechanism’s LearningSignal(s) and, in turn, its LearningProjection(s).

Type

number, ndarray or matrix

learning_signals

list of all of the LearningSignals for the LearningMechanism, each of which sends one or more LearningProjections to the ParameterPort(s) for the matrix parameter of the MappingProjection(s) trained by the LearningMechanism. The value of each LearningSignal is the LearningMechanism’s learning_signal attribute. Since LearningSignals are OutputPorts, they are also listed in the LearningMechanism’s output_ports attribute, after it ERROR_SIGNAL OutputPort.

Type

learning_projections

list of all of the LearningProjections <LearningProject> from the LearningMechanism, listed in the order of the LearningSignals to which they belong (that is, in the order they are listed in the learning_signals attribute).

Type

List[LearningProjection]

learning_enabled

determines whether and when the learning_projections are executed. If set to False, they are never updated; however, the LearningMechanism is still executed in any Composition to which it belongs, so that the error signals it calculates can be passed to any other LearningMechanism(s) to which it projects (see Multilayer learning). If set to True or ONLINE, learning_projections are updated when the LearningMechanism executes. If set to AFTER, learning_projections are updated at the end of each TRIAL of execution of the Composition to which the LearningMechanism belongs.

Note

the learning_enabled attribute of a LearningMechanism determines the default behavior of its learning_projections. However, this can be overridden for individual LearningProjections by assigning their learning_enabled attributes either at or after construction.

Type

bool or Enum[ONLINE|AFTER]

output_ports

list of the LearningMechanism’s OutputPorts, including its ERROR_SIGNAL OutputPort, followed by its LearningSignal(s), and then any additional (user-specified) OutputPorts.

Type

output_values

the first item is the value of the LearningMechanism’s ERROR_SIGNAL OutputPort, followed by the value (s) of its LearningSignal(s), and then those of any additional (user-specified) OutputPorts.

Type

2d np.array

modulation

the default form of modulation used by the LearningMechanism’s LearningSignal(s), unless they are individually specified.

Type

str

outputPortTypes
_parse_function_variable(variable, context=None)

Parses the variable passed in to a Component into a function_variable that can be used with the Function associated with this Component

_validate_variable(variable, context=None)

Validate that variable has exactly three items: activation_input, activation_output and error_signal

_validate_params(request_set, target_set=None, context=None)

Validate error_sources

error_sources argument must be an ObjectiveMechanism, another Learning Mechanism, an ERROR_SIGNAL OutputPort of a LearningMechanism, or a list of these, and there must be the same number as there are ERROR_SIGNAL InputPorts.

_instantiate_attributes_before_function(function=None, context=None)

Instantiates MappingProjection(s) from error_sources (if specified) to LearningMechanism

Also determines and assigns error_matrices from the error_sources, identified as the matrix for the

Projection with which each error_source is associated. :param function:

_instantiate_output_ports(context=None)

Call Port._instantiate_output_ports to instantiate orderedDict of OutputPort(s)

This is a stub, implemented to allow Mechanism subclasses to override _instantiate_output_ports

or process InputPorts before and/or after call to _instantiate_output_ports

add_ports(error_sources, context=None)

remove_ports(ports)

Keep error_signal_input_ports and error_matrices in sych with error_signals in input_ports

_execute(variable=None, context=None, runtime_params=None)

Execute LearningMechanism function and return learning_signal

Identify error_signals received from LearningMechanisms currently being executed Assign them, and the corresponding error_matrices to a pair of arrays Execute function for each error_signal, error_matrix pair Sum the learning_signal and error_signal values received from each execution

Returns

List[ndarray, ndarray]

Return type

summed learning_signal, summed error_signal

exception psyneulink.core.components.mechanisms.modulatory.learning.learningmechanism.LearningMechanismError`(error_value)