A robust procedure for comparing multiple means under heteroscedasticity in unbalanced designs.

Investigating differences between means of more than two groups or
experimental conditions is a routine research question addressed in
biology. In order to assess differences statistically, multiple
comparison procedures are applied. The most prominent procedures of
this type, the Dunnett and Tukey-Kramer test, control the
probability of reporting at least one false positive result when
the data are normally distributed and when the sample sizes and
variances do not differ between groups. All three assumptions are
non-realistic in biological research and any violation leads to an
increased number of reported false positive results. Based on a
general statistical framework for simultaneous inference and robust
covariance estimators we propose a new statistical multiple
comparison procedure for assessing multiple means. In contrast to
the Dunnett or Tukey-Kramer tests, no assumptions regarding the
distribution, sample sizes or variance homogeneity are necessary.
The performance of the new procedure is assessed by means of its
familywise error rate and power under different distributions. The
practical merits are demonstrated by a reanalysis of fatty acid
phenotypes of the bacterium Bacillus simplex from the "Evolution
Canyons" I and II in Israel. The simulation results show that even
under severely varying variances, the procedure controls the number
of false positive findings very well. Thus, the here presented
procedure works well under biologically realistic scenarios of
unbalanced group sizes, non-normality and heteroscedasticity.