Traditional multilevel models are commonly used to analyze multilevel data with a pure nested structure. However, cross-classified data structures are present when observations are nested within two higher-level structures which in turn are not nested within each other. To capture the variations from both clustering dimensions, cross-classified random effects models (CCREMs) were developed as an extension to standard multilevel modeling. Alternatively, simpler linear regression models with cluster-robust standard errors (CRSEs) can also be effective when analyzing cross-classified data structures. However, CRSEs can be underestimated when the number of clusters is small. The present chapter extends the use of a small sample adjustment approach (i.e., CR2) of CRSEs when used with cross-classified data structures (i.e., two-way OLS-CR2) and investigates its performance under various conditions. Simulation results showed that two-way OLS-CR2 can be used for both complete and partial cross-classified data with at least 30 clusters in one of two clustering dimensions. An applied data example showed similar results using the two-way OLS-CR2 and CCREMs.

Understanding causal mechanisms is a central goal in the behavioral, developmental, and social sciences. When estimating and probing causal effects using observational data, covariate adjustment is a crucial element to remove dependencies between focal predictors and the error term. Covariate selection, however, constitutes a challenging task because availability alone is not an adequate criterion to decide whether a covariate should be included in the statistical model. The present study introduces a non-Gaussian method for covariate selection and provides a forward selection algorithm for linear models (i.e., non-Gaussian Forward Selection; nGFS) to select appropriate covariates from a set of potential control variables to avoid inconsistent and biased estimators of the causal effect of interest. Further, we demonstrate that the forward selection algorithm has properties compatible with principles of direction of dependence, i.e., probing whether the causal target model is correctly specified with respect to the causal direction of effects. Results of a Monte-Carlo simulation study suggest that the selection algorithm performs well, in particular, when sample sizes are large (i.e., n ≥ 250), and data strongly deviate from Gaussianity (e.g., distributions with skewnesses beyond 1.5). An empirical example is given for illustrative purposes.

The usefulness of mean aggregates in the analysis of intervention effectiveness is a matter of considerable debate in the psychological, educational, and social sciences. In addition to studying “average treatment effects”, the evaluation of “distributional treatment effects”, (i.e., effects that go beyond means), has been suggested to obtain a broader picture of how an intervention affects the study outcome. We continue this discussion by considering distributional causal effects. We present formal definitions of causal effects that go beyond means and utilize a distributional regression framework known as generalized additive models for location, scale, and shape (GAMLSS). GAMLSS allows one to characterize an intervention effect in its totality through simultaneously modeling means, variances, skewnesses, kurtoses, as well as ceiling and floor effects of outcome distributions. Based on data from a large-scale randomized controlled trial, we use GAMLSS to evaluate the impact of a teacher classroom management program on student academic performance. Results suggest the teacher classroom management training increased mean academic competence as well as the chance to obtain the maximum score on the academic competence scale. These effects would have been completely overlooked in a traditional evaluation of mean aggregates.

Homogeneity of variance (HOV) is a well-known but often untested assumption in the context of multilevel models (MLMs). However, depending on how large the violation is, how different group sizes are, and the variance pairing, standard errors can be over or underestimated even when using MLMs, resulting in questionable inferential tests. We evaluate several tests (e.g., the H statistic, Breusch Pagan, Levene’s test) that can be used with MLMs to assess violations of HOV. Although the traditional robust standard errors used with MLMs require at least 50 clusters to be effective, we assess a robust standard error adjustment (i.e., the CR2 estimator) that can be used even with a few clusters. Findings are assessed using a Monte Carlo simulation and are further illustrated using an applied example. We show that explicitly modeling the heterogenous variance structures or using the CR2 estimator are both effective at ameliorating the issues associated with the fixed effects of the regression model related to violations of HOV resulting from between-group differences.