|
Backpropagation is a technique used for training neural networks. It is most useful for feed-forward networks (networks that have no feedback, or simply, that have no connections that loop). The term is an abbreviation for "backwards propagation of errors". Backpropagation requires that the transfer function used for the units (sometimes called neurons) be differentiable.
The gist of the technique is as follows -
- Present a training sample to the neural network.
- Compare the NN's output to the required output from that sample pair. Calculate the error in each output neuron.
- For each neuron, calculate from the error, the actual output, and a scaling factor, how much lower or higher it should be. This is the local error.
- Using the neurons weights on its incoming connections, assign "blame" for the local error to neurons at the previous level.
- Repeat the steps above on the neurons at the previous level, using each ones "blame" as their error.
As the algorithm's name implies, the errors (and therefore the learning) propagate backwards from the output nodes to the inner nodes. So technically speaking, backpropagation is used to calculate the gradient of the error of the network with respect to the network's modifiable weights. This gradient is almost always then used in a simple stochastic gradient descent algorithm to find weights that minimize the error. Often the term "backpropagation" is used in a more general sense, to refer to the entire procedure encompasing both the calculation of the gradient and its use in stochastic gradient descent. Backpropagation usually allows quick convergence on satisfactory local minima in the kind of networks to which it is suited.
The backpropagation algorithm for calculating a gradient has been rediscovered a number of times, and is a special case of a more general technique called algorithmic differentiation in the reverse accumulation mode.
|
