Learning Mechanism

Overview

A LearningMechanism is an AdaptiveMechanism that modifies the matrix parameter of one or more MappingProjections. Its function takes one or more error_signals (usually the output of a ComparatorMechanism or one or more other LearningMechanisms), as well as information about the MappingProjection(s) and activity that generated the error(s), and calculates a learning_signal that is used to modify the MappingProjection(s) by way of its LearningProjection(s). Typically, a LearningMechanism is used to “train” a single MappingProjection (its primary_learned_projection), using the output of the Mechanism to which that MappingProjection projects (its output_source) as the source of the error it attempts to reduce. A LearningMechanism can be used to train multiple MappingProjections, by assigning it additional LearningProjections; however, these will all use the same learning_signal, generated by the primary_learned_projection and its associated output_source.

All of the MappingProjection(s) modified by a LearningMechanism must project from one ProcessingMechanism to another in the same System or Process. The learning components of a System can be displayed using the System’s show_graph method with its show_learning argument assigned True or ALL. LearningMechanisms are executed after all of the ProcessingMechanisms in a System or Process have been executed, and before any ControlMechanisms of the System have been executed (see System Execution).

A Note about the Implementation of Learning

The implementation of learning in PsyNeuLink was designed for flexibility and “transparency” rather than efficiency. Unlike its implementation in most other environments – where the learning algorithm is tightly integrated with the elements of processing that it modifies – PsyNeuLink separates it into three constituent components: An ObjectiveMechanism used to evaluate the most proximal source of error, a Learning Mechanism that uses that error (or one derived from it by another LearningMechanism) to calculate a learning signal; and LearningProjection(s) that use that learning signal to modify the weight matrix of the MappingProjection(s) being learned. This has the advantage of isolating and exposing the constituent computations, making it clearer to students what these are and how they operate, and also making each individually accessible for reconfiguration. However, it comes at the cost of efficiency. Therefore, in its present form, PsyNeuLink is not suitable for conducting large scale learning models (e.g, deep learning networks). However, once trained, the parameters of such models can easily be incorporated into a PsyNeuLink model for use as part of a System. Future extensions of PsyNeuLink will provide an interface for the direct construction and integration within the PsyNeuLink environment of objects created in other environments (e.g., TensorFlow or Emergent).

Creating a LearningMechanism

A LearningMechanism can be created in any of the ways used to create Mechanisms. More commonly, however, LearningMechanisms are created automatically.

Automatic Creation

A LearningMechanism is created automatically when:

  • the learning argument is specified in the constructor for a System or Process;

In these instances, a ComparatorMechanism, LearningSignal, LearningProjection, and any additional Projections required to implement learning that do not already exist are also instantiated. This is described below, under Learning Configurations.

Explicit Creation

If a LearningMechanism is created explicitly (using its constructor), then its variable and error_sources arguments must be specified. The variable must have at leaset three items that are compatible (in number and type) with the value of the the LearningMechanism’s InputStates. Each item in error_sources must be one of the following: a ComparatorMechanism, for single layer learning or for the last MappingProjection in a learning sequence for multilayer learning; or a Learning Mechanism.

When a LearningMechanism is created explicitly, it can also be assigned existing LearningSignals and/or specified to create these, as well as LearningProjections from these to specified MappingProjections. These are specified in the learning_signals argument of the LearningMechanism’s constructor, using any of the forms allowed for specifying a LearningSignal.

Structure

A LearningMechanism has three types of InputStates, a learning function, and two types of OutputStates. These are used, respectively, to receive, compute, and transmit the information needed to modify the MappingProjection(s) for which the LearningMechanism is responsible. In addition, it has several attributes that govern and provide access to its operation. These are described below.

InputStates

These receive the information required by the LearningMechanism’s function. They are listed in the LearningMechanism’s input_states attribute. They have the following names and roles (shown in the figure below):

  • ACTIVATION_INPUT - receives the value of the input to the primary_learned_projection; that is, the value of that MappingProjection’s sender. The value is assigned as the first item of the LearningMechanism’s variable attribute.
  • ERROR_SIGNAL - this receives the value from the OUTCOME OutputState of a ComparatorMechanism, or of the ERROR_SIGNAL OutputState of another LearningMechanisms. If the primary_learned_projection projects to the TERMINAL Mechanism of the Process or System being learned, or is not part of a multilayer learning sequence, then the LearningMechanism has a single ERROR_SIGNAL InputState, that receives its input from a ComparatorMechanism. If the primary_learned_projection is part of a multilayer learning sequence, then the LearningMechanism will have one or more ERROR_SIGNAL InputStates, that receive their input from the next LearningMechanism(s) in the sequence; that is, the one(s) associated with the efferents (outgoing Projections) of its output_source, with one ERROR_SIGNAL InputState for each of those Projections. The values of the ERROR_SIGNAL InputStates are summed by the LearningMechanism’s function to calculate the learning_signal (see below); note that the value of the ERROR_SIGNAL InputState may not be the same as that of the LearningMechanism’s error_signal attribute or ERROR_SIGNAL OutputState (see note below). If a LearningMechanism has more than one ERROR_SIGNAL InputState, their names are suffixed with a hyphenated index, that is incremented for each additional InputState (e.g., error_signal-1, error_signal-2, etc.). These are listed in the LearningMechanism’s error_signal_input_states attribute, and the value of each is assigned as an item of the LearningMechanism’s variable attribute, beginning with its third item (i.e., following the value of the ACTIVATION_INPUT and ACTIVATION_VALUE InputStates).

The Mechanisms from the which the values above are received are listed in the LearningMechanism’s input_source, output_source, and error_sources attributes, respectively (see Additional Attributes for additional details).

Learning Function

The function of a LearningMechanism uses the values received by the Mechanism’s InputStates (described above) to calculate the value of its learning_signal and error_signal attributes.

The default function of a LearningMechanism is BackPropagation (also known as the Generalized Delta Rule; see Rumelhart et al., 1986). However, it can be assigned any other PsyNeuLink LearningFunction, or any other Python function that takes as its input a list or np.array containing three lists or 1d np.arrays of numeric values, and returns two lists or 1d np.arrays. The two values it returns are assigned to the LearningMechanism’s learning_signal and error_signal attributes, respectively, as well as to its two OutputStates, as described below.

OutputStates

By default, a LearningMechanism has two OutputStates, the first of which is named ERROR_SIGNAL and is assigned the value of the error_signal returned by the LearningMechanism’s function, and the second of which is a LearningSignal and is assigned the value of the learning_signal returned by the function. They are each described below:

  • ERROR_SIGNAL - this is the primary OutputState of a LearningMechanism, and receives the value of the error_signal used to calculate the learning_signal. Its value is assigned as the first item of the LearningMechanism’s output_values attribute. If the LearningMechanism is part of a multilayer learning sequence, the ERROR_SIGNAL OutputState is assigned a Projection to the LearningMechanism for the preceding MappingProjection in the sequence being learned - see figure below). Note that the value of the ERROR_SIGNAL OutputState may not be the same as that of the LearningMechanism’s ERROR_SIGNAL InputStates (see error_signal).
  • LearningSignal(s) - by default, a LearningMechanism has a single LearningSignal, which is a special type of OutputState that receives the learning_signal generated by the LearningMechanism’s function, and used to modify the matrix parameter of the primary_learned_projection. The LearningSignal is assigned as the second item in the list of the LearningMechanism’s OutputStates (i.e., of its output_states attribute), and its value is assigned as the second item of the LearningMechanism’s output_values attribute.

Multiple LearningSignals and LearningProjections. Though not common, it is possible for a LearningMechanism to be assigned more than one LearningSignal, and/or more than one LearningProjection to its LearningSignal(s). This allows multiple MappingProjections to be trained by a single LearningMechanism. Note, however, that all of the LearningSignals of a LearningMechanism (and therefore its LearningProjections) use the same learning_signal, that is calculated based on the LearningMechanism’s primary_learned_projection and its associated output_source. This can be useful in some settings, such as for certain forms of convolutional neural networks.

If all of the LearningProjections are used to implement the same form of modulation, (determined by their LearningSignals’ modulation attribute), then they should be assigned to a single LearningSignal. Multiple LearningProjections can be assigned to a LearningSignal by specifying them in the projections argument of its constructor, or the PROJECTIONS entry of a dictionary assigned to its params argument); however, the matrix parameter for the MappingProjection to which they project must have the same shape for all of them. If different forms of modulation are required to train different MappingProjections, then multiple LearningSignals should be assigned to the LearningMechanism, each specified for one of the required types of modulation, and then LearningProjections assigned to them for the corresponding MappingProjections. Multiple LearningSignals can be specified for a LearningMechanism by including them in a list assigned to the learning_signals argument of the LearningMechanism’s constructor.

The learning_rate for each LearningSignal, and the learning_rate for each of its LearningProjections, can all be assigned different values (with the latter taking precedence over the former). If none of these are specified, the learning_rate of the LearningMechanism is used (see below).

All of the LearningSignals of a LearningMechanism are listed in its learning_signals attribute. Because these are OutputStates, they are also listed in the output_states attribute, after the ERROR_SIGNAL OutputState. All of the LearningMechanism’s LearningProjections (that is, those belonging to all of its LearningSignals) are listed in its learning_projections attribute.

Additional Attributes

In addition to its InputStates, function and OutputStates, a LearningMechanism has the following attributes that refer to the Components being learned and/or its operation:

  • learning_enabled - if set to False, learning is disabled for all of its LearningProjections; however, the LearningMechanism is still executed during the learning phase of execution of a Process or System, so that the error signals it calculates can be passed to any other LearningMechanism(s) to which it projects (see Multilayer learning).
  • learning_rate - specifies the learning_rate parameter used by the LearningMechanism’s function, which uses it to multiply the weight change matrix before returning it as the learning_signal. This can be specified in the learning_rate argument of the LearningMechanism’s constructor (or the constructor for its function; doing so supersedes specification of the learning_rate for any Process, and/or the learning_rate for any System to which the LearningMechanism belongs. The default value for a LearningMechanism’s learning_rate attribute is None, in which case the LearningMechanism (and its function) inherit the specification of the learning_rate for the Process in which the LearningMechanism is executed. If that is None, then it inherits the learning_rate specification of the System in which it is executed. If that is also None, then it uses the value of the default_learning_rate parameter of its function. A learning_rate parameter can also be specified for individual LearningSignals and/or their associated LearningProjections. Those have a direct multiplicative effect on the learning_signal of the LearningSignal and/or it LearningProjections (see LearningSignal learning_rate for additional details).

Learning Configurations

When learning is specified for a MappingProjection, a Process, or a System, all of the Components required for learning are created automatically. The types of Components that are generated depend on the LearningFunction specified, and the configuration of the Composition, as described below. If learning is specified for individual Projections in the pathway of a Process, it takes effect only if that Process is executed on its own (i.e., using the Process’ execute or run methods. For the learning Components associated with a Process to be implemented by a System, learning must be specified for the entire Process. All of the learning Components of a System can be displayed using the System’s show_graph method with its show_learning argument assigned True or ALL.

Single layer learning

This configuration occurs when only a single MappingProjection is specified for learning, or the LearningMechanism’s function only considers the error on its output_source (and not any additional sources of error) when calculating the learning_signal (e.g., for Reinforcement learning). In this case, a single ComparatorMechanism and LearningMechanism are created (if they do not already exist); in addition, the following MappingProjections are created:

  • from the Process or System to the ComparatorMechanism’s TARGET InputState;
  • from the ComparatorMechanism’s OUTCOME OutputState to the LearningMechanism’s ERROR_SIGNAL InputState.

In addition, a LearningProjection is created from the LearningSignal for the primary_learned_projection to the ParameterState for the matrix of the primary_learned_projection. Because this configuration involves only a single layer of learning, no Projection is created or assigned to the LearningMechanism’s ERROR_SIGNAL OutputState.

Components for Single Layer Learning

Schematic of Mechanisms and Projections involved in learning for a single MappingProjection

ComparatorMechanism, LearningMechanism and associated Projections created for the primary_learned_projection and output_source. Each Mechanism is labeled by its type (upper line, in bold) and its designated status in the Process and/or System to which it belongs (lower line, capitalized). Italicized label beside a component indicates the attribute of the LearningMechanism with which it is associated.

Multilayer learning

This configuration occurs when a set of MappingProjections being learned are in a sequence (such as the pathway of a Process); that is, in which each projects to a ProcessingMechanism that is the sender for the next MappingProjection(s) in the sequence (see the figure below). This requires the use of a learning function that can calculate the influence that each MappingProjection and its output have on the error that the LearningMechanism receives from the next one(s) in the sequence (e.g., BackPropagation). In multilayer learning, the Components created depend on the position of the primary_learned_projection and output_source in the sequence. If these are the last ones in the sequence, they are treated in the same way as single layer learning. This is the case if the output_source is a standalone Mechanism (one not in a Process or System), the TERMINAL Mechanism of a standalone Process (i.e., one not in a System), or the TERMINAL of all of the Processes to which it belongs in a System (and therefore a TERMINAL for the System). In these cases, as for single layer learning, a ComparatorMechanism is created that receives the output of the output_source as well as the target for learning (see TARGET Mechanisms below), and projects to a LearningMechanism that is created for the primary_learned_projection. For all other MappingProjections being learned in the sequence, the following additional MappingProjections are created for learning (shown in the figure below):

  • from the ERROR_SIGNAL OutputState of each of the LearningMechanism’s error_sources to each of its corresponding ERROR_SIGNAL InputState(s).

In addition, a LearningProjection is created from the LearningSignal for the primary_learned_projection of each LearningMechanism in the sequence, to the ParameterState for the matrix of the primary_learned_projection. If the primary_learned_projection is the first in the sequence, then no Projection is created or assigned to its LearningMechanism’s ERROR_SIGNAL OutputState.

Components for Multilayer Learning

Schematic of Mechanisms and Projections involved in learning for a sequence of MappingProjections

ComparatorMechanism and LearningMechanisms and associated Projections created for a sequence of two MappingProjections specified for learning. Each Mechanism is labeled by its type (upper line, in bold) and its designated status in the Process and/or System to which it belongs (lower line, capitalized). Italicized label beside a component indicates the attribute of the LearningMechanism with which it is associated.

TARGET Mechanisms. When a LearningFunction is specified for a LearningMechanism that requires a target (e.g., BackPropagation or Reinforcement), a ComparatorMechanism must be specified to receive the target. For multilayer learning, this is the error_source for the last MappingProjection in each learning sequence. When learning is specified for a Composition (i.e., a Process or a System), the ComparatorMechanism(s) that receive the targets are identified and designated as TARGET Mechanisms, and are listed in the Composition’s target_mechanisms attribute. If a TERMINAL Mechanism of a Composition receives a MappingProjection that is specified for learning, then it always projects to a TARGET Mechanism in that Composition. It is important to note, in this context, the status of a Mechanism in a System takes precedence over its status in any of the Processes to which it belongs. This means that even if a Mechanism is the TERMINAL of a particular Process, if that Process is combined with others in a System, and the Mechanism appears in any of those other Processes, and it is not the TERMINAL of all of them, then it will not be the TERMINAL for the System. As a consequence, although it will project to a TARGET Mechanism in the Process for which it is the TERMINAL, it will not do so in the System, and its LearningMechanism will receive its error_signal from another LearningMechanism rather than a ComparatorMechanism (see figure below for an example). Finally, if a Mechanisms is the TERMINAL for more than one Process used to create a System (that is, the pathways for those Processes converge on that Mechanism), only one ComparatorMechanism will be created for it in the System.

TERMINAL and TARGET Mechanisms in Learning

Schematic of Mechanisms and Projections involved in learning

Mechanism 3 is the TERMINAL Mechanism for Process A, However, it is also an INTERNAL Mechanism of Process B. Therefore, Mechanism 3 is designated as an INTERNAL Mechanism for the System, and Mechanism 4 is its TERMINAL Mechanism. As a consequence, if BackPropagation is used for learning, then Mechanism 4 will project to a TARGET Mechanism of the System, while Mechanism 3 will not.

Execution

LearningMechanisms are executed after all of the ProcessingMechanisms in the Process or System to which they belong have been executed. When a LearningMechanism is executed, it uses the value to calculate changes to the weights of the matrix parameter of its primary_learned_projection and any of its other learned_projections. Those weight changes are assigned as the LearningMechanism’s learning_signal attribute, the value of each of its LearningSignals, and as the value of each of their LearningProjections. That value is used, in turn, to modify the value of the MATRIX ParameterState of each of the MappingProjections being learned (listed in the LearningMechanism’s learned_projections attribute).

Each ParameterState uses the value it receives from the LearningProjection that projects to it to modify the parameter of its function, in a manner specified by the modulation attribute of the LearningSignal from which it receives the LearningProjection (see Modulation for a description of how modulation operates). By default, the modulation attribute of a LearningSignal is ModulationParam.ADDITIVE, the function of a MATRIX ParameterState for a MappingProjection is AccumulatorIntegrator, and the parameter it uses for additive modulation is its increment parameter. These assignments cause the value of the LearningProjection to be added to the previous value of the MATRIX ParameterState, thus incrementing the weights by an amount specified by the LearningMechanism’s learning_signal. Note, that the changes to the matrix parameter itself do not take effect until the next time the learned_projection is executed (see Lazy Evaluation for an explanation of “lazy” updating).

A LearningMechanism’s function also computes an error_signal that is assigned as the value of its ERROR_SIGNAL OutputState; in a multilayer learning configuration, that value is provided to the ERROR_SIGNAL InputState(s) of the LearningMechanism(s) for the preceding MappingProjection(s) being learned in the sequence.

Class Reference

psyneulink.components.mechanisms.adaptive.learning.learningmechanism.DefaultTrainingMechanism

alias of ObjectiveMechanism

class psyneulink.components.mechanisms.adaptive.learning.learningmechanism.LearningMechanism(variable, error_sources, function=BackPropagation, learning_rate=None, learning_signals=LEARNING_SIGNAL, modulation=ModulationParam.ADDITIVE, params=None, name=None, prefs=None)

Implements a Mechanism that modifies the matrix parameter of a MappingProjection.

Parameters:
  • variable (List or 2d np.array) – it must have three items that correspond to the three values required by the LearningMechanism’s function; they must each be compatible (in number and type) with the value of the corresponding InputState (see variable for additional details).
  • error_sources (ComparatorMechanism, LearningMechanism or list of them) – specifies the source(s) of the error signal(s) used by the LearningMechanism’s function. Each must be a ComparatorMechanism for single layer learning, or for the last MappingProjection in a learning sequence in multilayer learning; otherwise they must be a Learning Mechanism.
  • learning_signals (List[parameter of Projection, ParameterState, Projection, tuple[str, Projection] or dict] :) –
  • LEARNING_SIGNAL (default) – specifies the parameter(s) to be learned (see learning_signals for details).
  • modulation (ModulationParam : default ModulationParam.ADDITIVE) – specifies the default form of modulation used by the LearningMechanism’s LearningSignals, unless they are individually specified.
  • function (LearningFunction or function : default BackPropagation) – specifies the function used to calculate the LearningMechanism’s learning_signal and error_signal attributes. It’s variable must have three items, each of which must be a list or 1d array of numeric values, corresponding to values provided by the LearningMechanism’s ACTIVATION_INPUT, ACTIVATION_OUTPUT, and ERROR_SOURCES InputStates, respectively (see LearningMechanism_InputStates `LearningMechanism_Function and InputStates for additional details).
  • learning_rate (float : default None) – specifies the learning rate for the LearningMechanism (see learning_rate for details).
  • params (Dict[param keyword: param value] : default None) – a parameter dictionary that specifies the parameters for the Projection, its function, and/or a custom function and its parameters. By default, it contains an entry for the Projection’s default function and parameter assignments. Values specified for parameters in the dictionary override any assigned to those parameters in arguments of the constructor.
  • name (str : default see name) – specifies the name of the LearningMechanism.
  • prefs (PreferenceSet or specification dict : default Mechanism.classPreferences) – specifies the PreferenceSet for the LearningMechanism; see prefs for details.
variable

2d np.array – has three items that serve as the template for the three inputs required by the LearningMechanism’s function (corresponding to its three InputStates: the input to the primary_learned_projection (from input_source), the output of the Mechanism to which that projects (i.e., of output_source); and the error signal (from LearningMechanism.error_sources).

input_states

ContentAddressableList[OutputState] – list containing the LearningMechanism’s three InputStates: ACTIVATION_INPUT, ACTIVATION_OUTPUT, and ERROR_SIGNAL.

error_signal_input_states

list[InputStates] – list of InputStates that receive error_signals from the LearningMechanism’s error_sources.

input_source

ProcessingMechanism – the Mechanism that sends the primary_learned_projection, and projects to the LearningMechanism’s ACTIVATION_INPUT InputState.

output_source

ProcessingMechanism – the Mechanism that receives the primary_learned_projection, and projects to the LearningMechanism’s ACTIVATION_OUTPUT InputState.

error_sources

list[ComparatorMechanism or LearningMechanism] – the Mechanism(s) that calculate the error signal(s) provided to the LearningMechanism’s ERROR_SIGNAL(s) InputState(s).

error_matrices

list[ParameterState] – the matrices of the Projections associated with the error_sources, (i.e., for the next Projection(s) in the learning_sequence, or to the ComparatorMechanism); note: these are not for the LearningMechanism’s learned_projections.

primary_learned_projection

MappingProjection – the Projection with the matrix parameter used to generate the LearningMechanism’s error_signal and learning_signal attributes. It is always the first Projection listed in the LearningMechanism’s learned_projections attribute.

learned_projections

List[MappingProjection] – all of the MappingProjections modified by the LearningMechanism; the first item in the list is always the primary_learned_projection.

learning_enabled

bool : True – if set to False, learning is disabled for all of its LearningProjections; however, the LearningMechanism is still executed during the learning phase of execution of a Process or System, so that the error signals it calculates can be passed to any other LearningMechanism(s) to which it projects (see Multilayer learning).

function

LearningFunction or function : default BackPropagation – specifies the function used to calculate the learning_signal (assigned to the LearningMechanism’s LearningSignal(s)), and the error_signal (passed to the next LearningMechanism in a learning sequence for multilayer learning). It takes the following three arguments, each of which must be a list or 1d array: input, output, and error (see Learning Function for additional details).

learning_rate

float : None – determines the learning rate for the LearningMechanism. It is used to specify the learning_rate parameter for the LearningMechanism’s learning function (see description of learning_rate for additional details).

error_signal

1d np.array – one of two values returned by the LearningMechanism’s function. For single layer learning, this is the same as the value received in the LearningMechanism’s ERROR_SIGNAL InputState; for multilayer learning, it is a modified version of the value received, that takes account of the contribution made by the learned_projection and its input to the error signal received. This is assigned as the value of the LearningMechanism’s ERROR_SIGNAL OutputState.

learning_signal

number, ndarray or matrix – one of two values returned by the LearningMechanism’s function, that specifies the changes to the weights of the matrix parameter for the LearningMechanism’s learned_projections; it is calculated to reduce the error signal associated with the primary_learned_projection and received from the LearningMechanism’s error_sources. It is assigned as the value of the LearningMechanism’s LearningSignal(s) and, in turn, its LearningProjection(s).

learning_signals

ContentAddressableList[LearningSignal] – list of all of the LearningSignals for the LearningMechanism, each of which sends one or more LearningProjections to the ParameterState(s) for the matrix parameter of the MappingProjection(s) trained by the LearningMechanism. The value of each LearningSignal is the LearningMechanism’s learning_signal attribute. Since LearningSignals are OutputStates, they are also listed in the LearningMechanism’s output_states attribute, after it ERROR_SIGNAL OutputState.

learning_projections

List[LearningProjection] – list of all of the LearningProjections <LearningProject>` from the LearningMechanism, listed in the order of the LearningSignals to which they belong (that is, in the order they are listed in the learning_signals attribute).

output_states

ContentAddressableList[OutputState] – list of the LearningMechanism’s OutputStates, including its ERROR_SIGNAL OutputState, followed by its LearningSignal(s), and then any additional (user-specified) OutputStates.

output_values

2d np.array – the first item is the value of the LearningMechanism’s ERROR_SIGNAL OutputState, followed by the value (s) of its LearningSignal(s), and then those of any additional (user-specified) OutputStates.

modulation

ModulationParam – the default form of modulation used by the LearningMechanism’s LearningSignal(s), unless they are individually specified.

name

str – the name of the LearningMechanism; if it is not specified in the name argument of the constructor, a default is assigned by MechanismRegistry (see Naming for conventions used for default and duplicate names).

prefs

PreferenceSet or specification dict – the PreferenceSet for the LearningMechanism; if it is not specified in the prefs argument of the constructor, a default is assigned using classPreferences defined in __init__.py (see PreferenceSet for details).

outputStateType

alias of LearningSignal

add_states(states, context=None)

Add error_source and error_matrix for each InputState added