Anticipation Model for Sequential Learning of Complex Sequences

One of the fundamental aspects of human intelligence is the ability to process temporal information Lashley, 1951. Learning and reproducing temporal sequences are closely associated with our ability to perceive and generate body movements, speech and language, music, etc. A considerable body of neural network literature is devoted to temporal pattern generation see Wang, 2001, for a recent review. These models generally treat a temporal pattern as a sequence of discrete patterns, called a temporal sequence. Most of the models are based on either multilayer perceptrons with backpropagation training or on the Hopfield model of associative recall. The basic idea for the former class of models is to view a temporal sequence as a set of associations between consecutive components, and learn these associations as input-output transformations Jordan, 1986 Elman, 1990 Mozer, 1993. To deal with temporal dependencies beyond consecutive components, part of the input layer is used to keep a trace of history, behaving as short-term memory STM. Similarly, for temporal recall based on the Hopfield associative memory, a temporal sequence is viewed as associations between consecutive components. These associations are store in extended versions of the Hopfield model that includes some time delays Sompolinsky Kanter, 1986 Buhmann Schulten, 1987 Heskes Gielen, 1992. To deal with longer temporal dependencies, high-order networks have been proposed Guyon et al., 1988.