Dynamic Neural Regression Models

We consider sequential or online learning in dynamic neural
regression models. By using a state space representation for the
neural network' s parameter evolution in time we obtain
approximations to the unknown posterior by either deriving
posterior modes via the Fisher scoring algorithm or by deriving
approximate posterior means with the importance sampling method.
Furthermore, we replace the commonly used Gaussian noise assumption
in the neural regression model by a more flexible noise model based
on the Student t-density. Since the t-density can be interpreted as
being an infinite mixture of Gaussians, hyperparameters such as the
degrees of freedom of the t-density can be learned from the data
based on an online EM-type algorithm. We show experimentally that
our novel methods outperform state-of-the art neural network online
learning algorithms like the extended Kalman filter method for
both, situations with standard Gaussian noise terms and situations
with measurement outliers.