Generalized semiparametrically structured mixed models

Generalized linear mixed models are a common tool in statistics
which extends generalized linear models to situations where data
are hierarchically clustered or correlated. In this article the
simple but often inadequate restriction to a linear form of the
predictor variables is dropped. A class of semiparametrically
structured models is proposed in which the predictor decomposes
into components that may be given by a parametric term, an additive
form of unspecified smooth functions of covariates,
varying-coefficient terms or terms which vary smoothly (or not)
across the repetitions in a repeated measurement design. The class
of models is explicitly designed as an extension of multivariate
generalized mixed linear models such that ordinal responses may be
treated within this framework. The modelling of smooth effects is
based on basis functions like e.g. B-splines or radial basis
functions. For the estimation of parameters a penalized marginal
likelihood approach is proposed which may be based on integration
techniques like Gauss-Hermite quadrature but may as well be used
within the more recently developed nonparametric maximum likelihood
approaches. For the maximization of the penalized marginal
likelihood the EM-algorithm is adapted. Moreover, an adequate form
of cross-validation is developed and shown to work satisfactorily.
Various examples demonstrate the flexibility of the class of
models.