Deep Gate Recurrent Neural Network
University of Waikato Hamilton New Zealand
Pagination or Media Count:
This paper explores the possibility of using multiplicative gate to build two recurrent neural network structures. These two structures are called Deep Simple Gated Unit DSGU and Simple Gated Unit SGU, which are structures for learning long-term dependencies. Compared to traditional Long Short-Term Memory LSTM and Gated Recurrent Unit GRU, both structures require fewer parameters and less computation time in sequence classification tasks. Unlike GRU and LSTM, which require more than one gate to control information flow in the network, SGU and DSGU only use one multiplicative gate to control the flow of information. We show that this difference can accelerate the learning speed in tasks that require long dependency information. We also show that DSGU is more numerically stable than SGU. In addition, we also propose a standard way of representing the inner structure of RNN called RNN Conventional Graph RCG, which helps to analyze the relationship between input units and hidden units of RNN.
- Information Science