One method for determining correct denominators in analysis of variance is the Cornfield-Tukey method. This FAQ presents a modified version of the Cornfield-Tukey method for manually deriving the symbolic values for the expected mean squares. It is from these expected mean squares that one can determine appropriate error terms.
Please note that this approach to deriving expected mean squares assumes that the interaction of the fixed and random effects sum to zero over the fixed effect levels. This approach can be found in a number of classical ANOVA textbooks, such as, Kirk, Winer and Keppel. This assumption, however, is not universal and is not used in most mixed programs (proc mixed, xtmixed, etc).
If you would like to try a program that automates much of the computation for this algorithm, go to How can I determine the correct term in an anova using Stata?.
Steps in deriving expected mean squares
Step 1 - Write the linear model for the design. Step 2 - Construct a table with three parts. Step 3 - The row headings in part 1 contain each of the terms from the linear model including their subscripts but leaving out μ. Step 4 - The column heading in part 2 contain the subscripts from the linear model, the symbol for the number of levels along with the sampling coefficient. Sampling coefficients are coded 1 for random variables and 0 for fixed. Step 5 - If a column heading appears as a row subscript in parentheses enter a 1 in part 2. Step 6 - If a column heading appears as a row subscript, not in parentheses, enter the appropriate sampling coefficient (0 or 1). Step 7 - If a column heading does not appear as a row subscript enter the letter for the number of levels Step 8 - In part 3 list a variance for each term in the linear model that contains all the row subscripts. Step 9 - Coefficients for variances are obtained by covering the column headed by subscripts that appear in the row but not including subscripts in parentheses. Obviously, terms with zero coefficients drop out.
Example three-way factorial design
In this example, A & C are fixed and B is random. The subscript for ε is i(jkl) because the subjects are nested in the A*B*C cells. The subjects themselves are also random. The term, εi(jkl), is known as error, within cell or residual.
Step 1 - Yijkl = μ + αj + βk + γl + αβjk + αγjl + βγkl + αβγjkl + εi(jkl) Part 1 Part 2 Part 3 subscript i j k l levels n p q m sampling coef 1 0 1 0 ------------------------------------------------------------------- αj n 0 q m σ2ε + 0σ2αβγ + 0σ2αγ + nmσ2αβ + nqmσ2α σ2ε + nmσ2αβ + nqmσ2α βk n p 1 m σ2ε + 0σ2αβγ + 0σ2βγ + 0σ2αβ + npmσ2β σ2ε + npmσ2β γl n p q 0 σ2ε + 0σ2αβγ + npσ2βγ + 0σ2αγ + npqσ2γ σ2ε + npσ2βγ + npqσ2γ αβjk n 0 1 m σ2ε + 0σ2αβγ + nmσ2αβ σ2ε + nmσ2αβ αγjl n 0 q 0 σ2ε + nσ2αβγ + nqσ2αγ βγkl n p 1 0 σ2εσ2ε + 0σ2αβγ + npσ2βγ σ2ε + npσ2βγ αβγjkl n 0 1 0 σ2ε + nσ2αβγ εi(jkl) 1 1 1 1 σ2ε
A correctly formed F-ratio will have one more term in the numerator than in the denominator. The additional
term in the numerator is the effect of interest. Thus, the F-ration for A main effect would
look something like this:
σ2ε + nmσ2αβ + nqmσ2α MS(A) F(A) = ------------------------ = --------- σ2ε + nmσ2αβ MS(A*B)
Here are the terms that go into each of the F-ratios for the above model:
Effect Error Term numerator denominator MS(A) MS(A*B) MS(B) MS(residual) MS(C) MS(B*C) MS(A*B) MS(residual) MS(A*C) MS(A*B*C) MS(B*C) MS(residual) MS(A*B*C) MS(residual)
Kirk, Roger E. (1998) Experimental Design: Procedures for the Behavioral Sciences, Third Edition. Monterey, California: Brooks/Cole Publishing. ISBN 0-534-25092-0