The implementation of learning in PsyNeuLink was designed for exposition rather than efficiency. Unlike its
implementation in most other environments – where the learning algorithm is tightly integrated with the
elements of processing that it modifies – PsyNeuLink separates it into three constituent components, often including:
an ObjectiveMechanism used to evaluate the most proximal source of error; a LearningMechanism that uses that error
(or one derived from it by another LearningMechanism) to calculate a learning signal; and one or more
LearningProjections that use that learning signal to modify the weight matrix of the MappingProjection(s) being learned. This has the advantage
of isolating and exposing the constituent computations, making it clearer what these are and how they operate, and
also making each individually accessible for reconfiguration. However, it comes at the cost of efficiency. For
efficient execution of supervised forms of learning (e.g., reinforcement learning and backpropagation),
an AutodiffComposition can be used, which allows the model to be specified using PsyNeuLink, but actually executes
learning either in a compiled form or using PyTorch (see Compilation for additional details).
A LearningMechanism is created automatically when a Composition’s learning method is
called. In that case, a LearningSignal, LearningProjection, a ComparatorMechanism (in the case of supervisedlearning), and any additional Projections required to implement learning that do not already exist are also
instantiated. This is described below, under Learning Configurations.
A LearningMechanism is also created automatically when either the tuple specification is used to specify learning for a MappingProjection, or a
LearningProjection is created without specifying its sender attribute. However, this
is not advised, and should only used in special circumstances, as properly configuring learning generally requires
the instantiation of several other closely related Components, as described below.
When a LearningMechanism is created explicitly, it can also be assigned existing LearningSignals and/or specified to
create these, as well as LearningProjections from these to specified MappingProjections. These
are specified in the learning_signals argument of the LearningMechanism’s constructor, using any of the forms
allowed for specifying a LearningSignal.
A LearningMechanism has three types of InputPorts, a learning function,
and two types of OutputPorts. These are used, respectively, to receive, compute, and transmit the
information needed to modify the MappingProjection(s) for which the LearningMechanism is responsible. In addition,
it has several attributes that govern and provide access to its operation. These are described below.
These receive the information required by the LearningMechanism’s function. They are
listed in the LearningMechanism’s input_ports attribute. They have the following
names and roles (shown in the figure below):
ACTIVATION_INPUT - receives the value of the input to the primary_learned_projection; that is, the value of that MappingProjection’s sender. The value is assigned
as the first item of the LearningMechanism’s variable attribute.
ACTIVATION_OUTPUT - receives the value of the LearningMechanism’s output_source;
that is, the value of the OutputPort of the ProcessingMechanism to which the
primary_learned_projection projects. By default, the output_source’s
primary OutputPort is used. However, a different OutputPort can be designated in
the constructor for the output_source, by assigning a parameter specification
dictionary to the params argument of its constructor, with an entry that uses
MONITOR_FOR_LEARNING as its key and a list containing the desired OutputPort(s) as its value. The value of the ACTIVATION_OUTPUT InputPort is assigned as the second item of the LearningMechanism’s
variable attribute.
ERROR_SIGNAL - this receives the value from the OUTCOMEOutputPort of a ComparatorMechanism, or of the ERROR_SIGNAL OutputPort of another
LearningMechanisms. If the primary_learned_projection projects
to the TERMINAL Mechanism of the Composition to which it belongs, or is not part of a multilayer learning
sequence, then the LearningMechanism has a single ERROR_SIGNAL InputPort,
that receives its input from a ComparatorMechanism. If the primary_learned_projection is part of a multilayer
learning pathway, then the LearningMechanism will have one or more
ERROR_SIGNAL InputPorts, that receive their input from the next LearningMechanism(s) in the sequence; that is,
the one(s) associated with the efferents (outgoing Projections) of its output_source,
with one ERROR_SIGNAL InputPort for each of those Projections. The values of the
ERROR_SIGNAL InputPorts are summed by the LearningMechanism’s function to
calculate the learning_signal (see below);
note that the value of the ERROR_SIGNAL InputPort may not be the same as that of the LearningMechanism’s
error_signal attribute or ERROR_SIGNALOutputPort (see note below). If a LearningMechanism
has more than one ERROR_SIGNAL InputPort, their names are suffixed with a hyphenated index, that is incremented for
each additional InputPort (e.g., error_signal-1, error_signal-2, etc.). These are listed in the
LearningMechanism’s error_signal_input_ports attribute, and the value of each is assigned
as an item of the LearningMechanism’s variable attribute, beginning with its third
item (i.e., following the value of the ACTIVATION_INPUT and ACTIVATION_VALUE InputPorts).
COVARIATES - one or more InputPorts that receive the value of the inputs to output_source(i.e.,the*ProcessingMechanism*towhichthe`primary_learned_projection projects) other than primary_learned_projection, if the output_source’s
function takes more than one argument and they affect its derivative. These are
assigned as a list to the fourth item in the LearningMechanism’s variable
attribute, and are used to calculate the learning_signal (see below).
Note
The LearningMechanism has a separate InputPort for each covariate, however their
values are combined into a single item of the LearningMechanism’s variable
that is provided to its function at the fourth position.
Although the COVARIATES InputPorts receive Projections from the same senders as the corresponding
Projections to the output_source, these are for display only,
and are not to actually compute the LearningSignal, since they are not subject to learning and therefore
do not provide the same value as the actual Projections they shadow. Rather, those values are accessed
directly by the LearningMechanism from the InputPorts of the output_source,
in a manner similar to its access of the error_matrices from the
ParameterPort`softhe`error_sources.
The function of a LearningMechanism uses the values received by the Mechanism’s
InputPorts (described above) to calculate the value of its learning_signal and error_signal attributes.
error_signal - the contribution made by the primary_learned_projection to the
error_signal(s) received by the LearningMechanism’s ERROR_SIGNALInputPort(s). It is used by the LearningMechanism’s function
to calculate the learning_signal. Depending upon the context and specific
LearningFunction used, it may also take account of the value of its
output_source, as well as the matrix parameter of any of the output_source’s
outgoing Projections that are also being learned (these are listed in the LearningMechanism’s error_matrices attribute). The value of the error_signal
is assigned as the value of the LearningMechanism’s ERROR_SIGNALOutputPort.
The default function of a LearningMechanism is BackPropagation (also known as the
Generalized Delta Rule; see
Rumelhart et al., 1986). However, it can be
assigned any other PsyNeuLink LearningFunction, or any other Python function that takes as its
input a list or np.array containing three lists or 1d np.arrays of numeric values, and returns two lists or 1d
np.arrays. The two values it returns are assigned to the LearningMechanism’s learning_signal and error_signal attributes, respectively,
as well as to its two OutputPorts, as described below.
By default, a LearningMechanism has two OutputPorts, the first of which is named ERROR_SIGNAL and
is assigned the value of the error_signal returned by the LearningMechanism’s
function, and the second of which is a LearningSignal and is assigned the value of the
learning_signal returned by the function.
They are each described below:
ERROR_SIGNAL - this is the primary OutputPort of a LearningMechanism, and receives the
value of the error_signal used to calculate the learning_signal. Its value is assigned as the first item of the LearningMechanism’s
output_values attribute. If the LearningMechanism is part of a multilayer
learning pathway, the ERROR_SIGNAL OutputPort is assigned a Projection
to the LearningMechanism for the preceding MappingProjection in the sequence being learned - see figure below). Note that the value of the
ERROR_SIGNAL OutputPort may not be the same as that of the LearningMechanism’s ERROR_SIGNALInputPorts (see error_signal).
LearningSignal(s) - by default, a LearningMechanism has a single LearningSignal, which is a
special type of OutputPort that receives the learning_signal generated by the
LearningMechanism’s function, and used to modify the matrix parameter of the primary_learned_projection. The LearningSignal is assigned as the
second item in the list of the LearningMechanism’s OutputPorts (i.e., of its output_ports attribute), and its value is assigned as the second
item of the LearningMechanism’s output_values attribute.
Multiple LearningSignals and LearningProjections. Though not common, it is possible for a LearningMechanism to
be assigned more than one LearningSignal, and/or more than one LearningProjection to its LearningSignal(s). This
allows multiple MappingProjections to be trained by a single LearningMechanism. Note, however, that all of the
LearningSignals of a LearningMechanism (and therefore its LearningProjections) use the same
learning_signal, that is calculated based on the LearningMechanism’s
primary_learned_projection and its associated output_source. This can be useful in some settings, such as for
certain forms of convolutional neural networks.
If all of the LearningProjections are used to implement the same form of modulation,
(determined by their LearningSignals’ modulation attribute), then they should be
assigned to a single LearningSignal. Multiple LearningProjections can be assigned to a LearningSignal by specifying
them in the projections argument of its constructor, or the PROJECTIONS entry of a dictionary assigned to its
params argument); however, the matrix parameter for the MappingProjection to which
they project must have the same shape for all of them. If different forms of modulation are required to train
different MappingProjections, then multiple LearningSignals should be assigned to the LearningMechanism, each
specified for one of the required types of modulation, and then LearningProjections assigned to them for the
corresponding MappingProjections. Multiple LearningSignals can be specified for a LearningMechanism by including
them in a list assigned to the learning_signals argument of the LearningMechanism’s constructor.
All of the LearningSignals of a LearningMechanism are listed in its learning_signals attribute. Because these
are OutputPorts, they are also listed in the output_ports
attribute, after the ERROR_SIGNAL OutputPort. All of the LearningMechanism’s LearningProjections (that is, those
belonging to all of its LearningSignals) are listed in its learning_projections attribute.
In addition to its InputPorts, function and
OutputPorts, a LearningMechanism has the following attributes that
refer to the Components being learned and/or its operation:
learning_enabled - determines whether and when the LearningMechanism’s
learning_projections are executed (see learning_enabled for additional details).
modulation - the default value used for the modulation attribute of
LearningMechanism’s LearningSignals (i.e. those for which it is not explicitly specified).
This determines the way in which the learning_signal is used to modify the
matrix parameter of the learned_projections. By default its value is
Modulation.ADD, which causes the weight changes in the learning_signal to be added to the current value of the
matrix parameter (see Execution for a description of how the
modifications are executed).
learning_rate - specifies the learning_rate
parameter used by the LearningMechanism’s function which, for most forms of learning,
uses it to multiply the weight change matrix before returning it as the learning_signal. The value can be specified in the learning_rate argument of the
LearningMechanism’s constructor, or in the constructor for its function. If both
are specified, the specification for the function takes precedence; in either case, the value of the learning_rate
parameter is always the same for the LearningMechanism and its function. If neither is specified, then it inherits
its value from any specifications made for the Composition or a learning Pathway
of the Composition to which it belongs (see Learning Rate for additional details). However, after the
Composition and/or its learning pathways have been constructed, specifying the
learning_rate for a LearningMechanism (or its function) overrides any
specifications made for the Composition or its learning pathways, including in calls the Composition’s learn method. In this way, individual LearningMechanisms can be assigned specific learning rates
that apply whenever those are executed. If the learning_rate is not explicitly specified anywhere, the default
value for the LearningMechanism’s function is used.
When learning is enabled for a RecurrentTransferMechanism (for unsupervised
learning) or using the learning method of a
Composition, all of the Components required for learning are created automatically. The types of Components that are
generated depend on the type of learning specified and the configuration of the Composition, as
described below. All of the learning Components of a Composition can be displayed using its show_graph<ShowGraph.show_graph method with the show_learning argument assigned True or ALL.
This configuration occurs when unsupervised learning is used, or supervised
learning is used for a pathway in a Composition with only two Mechanisms (i.e.,
for the Projection between them). In this case, a single ComparatorMechanism and LearningMechanism are created
(if they do not already exist) as well as the following MappingProjections:
Components generated by a call to a learning method of a Composition
(e.g., add_backpropagaption_learning_pathway(pathway=[X,Y]), labeled by type of Component (in bold,
camelCase), NodeRole(s) assigned (italics), learning component type used as key in dictionary returned by
the learning method (UPPER_CASE italics, outside of object) and, where relevant, the name of the attribute
of the LearningMechanism with which it is associated (italicized lower case, outside of object). The
ComparatorMechanism and LearningMechanism are shown with their InputPorts, OutputPorts and functions
diagrammed.¶
This refers to learning in a pathway that has three or more Mechanisms in a sequence, and therefore two or more
MappingProjections that are being learned. The learning components for such a configuration are created
automatically when a learning method of a Composition is used that supports
multilayer learning (at present, this is only add_backpropagation_learning_pathway). If the learning components
are constructed explicitly for a multilayer learning configuration,
then each LearningMechanism must use a LearningFunction that can calculate the influence
that each MappingProjection and its output have on the error that the LearningMechanism receives from the previous
one(s) in the sequence (e.g., BackPropagation). Furthermore, the construction of the other learning-related
Components associated with each LearningMechanism depend on the position of its
primary_learned_projection and output_source in the sequence. If it is the
last one in the sequence, it is treated in the same way the LearningMechanism in a single layer learning configuration
configuration. This is the case if the output_source is a standalone
Mechanism or the TERMINAL Mechanism of a Composition. In these cases, as for single layer learning, a
ComparatorMechanism is created that receives the output of the output_source as well as the target for learning
(see TARGET Mechanisms below), and projects to a LearningMechanism that is created for
the primary_learned_projection. For all other MappingProjections being learned in the
sequence, the following additional MappingProjections are created for learning (shown in the figure below):
from the ERROR_SIGNALOutputPort of each of the LearningMechanism’s
error_sources to each of its corresponding ERROR_SIGNALInputPort(s);
from the covariates_source to the LearningMechanism’s COVARIATESInputPort`(s)ifthe`function of the output_source
takes more than one argument and they affect its derivative (see above).
Components generated by a call to a learning method of a Composition for a
sequence with three ProcessingMechanisms (and therefore two MappingProjections to be learned; e.g.,
add_backpropagaption_learning_pathway(pathway=[X,Y,Z]). Components are labeled by: type of Component (in
bold, camelCase); NodeRole(s) assigned (italics) to it; name of learning component type (used as key in
dictionary returned by the learning method; UPPER_CASE italics, outside of object); and, where relevant,
the name of the attribute of the LearningMechanism with which it is associated (italicized lower case,
outside of object). The ComparatorMechanism and LearningMechanism are shown with their InputPorts, OutputPorts
and functions diagrammed.¶
TARGET and OBJECTIVE Mechanisms**. When supervised learning is
implemented using one of a Composition’s learning methods, it automatically creates a
number of learning-related Components. This includes a TARGET_MECHANISM, that
receives the target stimulus specifed in the inputs argument of the Composition’s learn
method; and a OBJECTIVE_MECHANISM, that computes the error_signal for the sequence. The output of the
OBJECTIVE_MECHANISM is the error_source for the last MappingProjection in the
learning pathway. If a multilayer learning pathway is implemented explicitly, it must include these Components.
LearningMechanisms that implement unsupervised learning
(AutoassociativeLearningMechanisms) execute when the RecurrentTransferMechanism
with which they are associated executes. LearningMechanisms that are part of a supervised learning pathway execute after all of the ProcessingMechanisms in the Composition to which they
belong have executed. When a LearningMechanism is executed, it uses the value to calculate changes to the weights of the
matrix parameter of its primary_learned_projection and any of its other learned_projections. Those weight changes are assigned as the LearningMechanism’s
learning_signal attribute, the value of each of its
LearningSignals, and as the value of each of their
LearningProjections. That value is used, in turn, to modify the value of the MATRIXParameterPort of each of the MappingProjections being learned (listed in the LearningMechanism’s
learned_projections attribute).
Each ParameterPort uses the value it receives from the LearningProjection that projects to it to modify the
parameter of its function, in a manner specified by the modulation attribute of the LearningSignal from which it receives the LearningProjection (see
Modulation for a description of how modulation operates). By default, the
modulation attribute of a LearningSignal is ADDITIVE, the function of a MATRIX ParameterPort for a MappingProjection is AccumulatorIntegrator,
and the parameter it uses for additive modulation is its increment parameter.
These assignments cause the value of the LearningProjection to be added to the previous value of the MATRIX
ParameterPort, thus incrementing the weights by an amount specified by the LearningMechanism’s learning_signal. Note, that the changes to the matrix parameter itself do not take effect until the next time the
learned_projection is executed (see Lazy Evaluation for an explanation of
“lazy” updating).
Subclass of ModulatoryMechanism that modifies the matrix parameter of a
MappingProjection. See Mechanism for additional arguments and attributes.
Parameters
variable (List or 2d np.array) – it must have three items that correspond to the three values required by the LearningMechanism’s function; they must each be compatible (in number and type) with the value of the corresponding InputPort (see variable for additional details).
function (LearningFunction or function : default BackPropagation) – specifies the function used to calculate the LearningMechanism’s learning_signal and error_signal attributes. It’s
variable must have three items, each of which must be a list or 1d array of
numeric values, corresponding to values provided by the LearningMechanism’s ACTIVATION_INPUT,
ACTIVATION_OUTPUT, and ERROR_SOURCES InputPorts, respectively (see LearningMechanism_InputPorts`LearningMechanism_Function and InputPorts for additional details). If supports an
activation function that takes more than one argument that impact its derivative, then it must also accept a
potential fourth keyword argument (covariates), that is a list or 1d array of numeric values provided by
the LearningMechanism’s COVARIATES InputPort(s) (see LearningMechanism_Covariates for additional details).
learning_rate (float : default None) – specifies the learning rate for the LearningMechanism (see learning_rate
for details).
learning_signals (List[parameter of Projection, ParameterPort, Projection, tuple[str, Projection] or dict] :) – default LEARNING_SIGNAL specifies the parameter(s) to be learned (see learning_signals for details).
modulation (str : default ADDITIVE) – specifies the default form of modulation used by the LearningMechanism’s LearningSignals,
unless they are individually specified.
learning_enabled (bool or Enum[ONLINE|AFTER] : True) – specifies whether and when the LearningMechanism’s LearningProjections are executed
(see learning_enabled for additional details).
the values of the InputPorts to which the covariates_sources project;
passed to the LearningMechanism’s function as the COVARIATES item of its
variable, and assigned as the value of the LearningMechanism’s
COVARIATESInputPorts.
the matrices of the Projections associated with the error_sources,
(i.e., for the next Projection(s) in the learning_sequence, or to the ComparatorMechanism);
note: these are not for the LearningMechanism’s learned_projections.
the Projection with the matrix parameter used to generate the
LearningMechanism’s error_signal and learning_signal attributes. It is always the first Projection listed in the
LearningMechanism’s learned_projections attribute.
determines the learning rate for the LearningMechanism. It is used to specify the learning_rate
parameter for the LearningMechanism’s learningfunction
(see description of learning_rate for additional details).
one of two values returned by the LearningMechanism’s function. For
single layer learning, this is the same as the value received in
the LearningMechanism’s ERROR_SIGNALInputPort; for multilayer
learning, it is a modified version of the value received, that takes
account of the contribution made by the learned_projection and its input to the error signal received. This
is assigned as the value of the LearningMechanism’s ERROR_SIGNALOutputPort.
one of two values returned by the LearningMechanism’s function, that specifies
the changes to the weights of the matrix parameter for the LearningMechanism’s
learned_projections; it is calculated to reduce the error signal
associated with the primary_learned_projection and received
from the LearningMechanism’s error_sources. It is assigned as the value of the LearningMechanism’s
LearningSignal(s) and, in turn, its LearningProjection(s).
list of all of the LearningProjections <LearningProject>` from the LearningMechanism, listed in the order of
the LearningSignals to which they belong (that is, in the order they are listed in
the learning_signals attribute).
determines whether and when the learning_projections are executed.
If set to False, they are never updated; however, the LearningMechanism is still executed in any Composition
to which it belongs, so that the error signals it calculates can be passed to any other LearningMechanism(s)
to which it projects (see Multilayer learning). If set to True or ONLINE,
learning_projections are updated when the LearningMechanism
executes. If set to AFTER, learning_projections are updated at the
end of each TRIAL of execution of the Composition to which the LearningMechanism belongs.
the first item is the value of the LearningMechanism’s ERROR_SIGNALOutputPort, followed by the value (s) of its
LearningSignal(s), and then those of any additional (user-specified)
OutputPorts.
error_sources argument must be an ObjectiveMechanism, another LearningMechanism, an ERROR_SIGNAL
OutputPort of a LearningMechanism, or a list of these, and there must be the same number as there are
ERROR_SIGNAL InputPorts.
Execute LearningMechanism function and return learning_signal
Identify error_signals received from LearningMechanisms currently being executed
Assign them, and the corresponding error_matrices to a pair of arrays
Execute function for each error_signal, error_matrix pair
Sum the learning_signal and error_signal values received from each execution
