However in the absence of knowing what these are we run the same experiment as in case a. This may be an impossible generalization anyway .sx Problems of differences between the selection of experimental and real groups may invalidate it .sx The unknown numbers in each category in the target population make it impossible to estimate the main effects when an interaction is present .sx If the effect of one of the variables depends on the level of the other then the overall effect of the first variable could be any weighted average of its effects at each level on the second variable .sx Indeed it should be the weighted average of its effects over all naturally occurring levels of the second variable including those not used in the experiment .sx Since this design is not OSCAR , the numbers of observations in the experimental conditions do not correspond to population frequencies .sx Thus an EWO analysis is inappropriate .sx For the main effects we shall have to confine ourselves to a simple effects analysis .sx The alternative is to give up attempting to generalize to existing groups and analyse the data as for case a. Case c .sx The group study .sx black-square We are interested in the effects of age and sex on memory loss in head injured patients .sx Samples of male and female head injured patients in each of a number of age groups are obtained .sx Here the design could be regarded as OSCAR .sx If , because of sampling bias , we were not prepared to make the assumption that the design was OSCAR , it would be difficult to justify the assumption that the design was ROWT .sx It is hard to imagine that there is a bias in the sample which affects only the age and sex of the sampled subjects and not any other related characteristics .sx The numbers in the different groups reflect differences in that real population .sx The EWM analysis of main effects depends arbitrarily on the other factors in the study .sx We wish to test whether the mean of male patients differs from the female mean and whether the means in the different age groups differ .sx Here and EWO analysis is appropriate for both main effects .sx An EWM analysis would only be of interest if one wished to generalize to the hypothetical results of a study on this population which used a balanced design .sx Case d .sx The first hybrid case .sx black-square We are interested in differences in reading between a dyslexic group ( assumed to be a random sample from a population of dyslexics ) and a control group , together with the effects of social class on the two groups .sx The control group is assumed to be a random sample of children from the same age cohorts .sx Here we assume we have a design which is OSCAR with respect to the social class factor but is ROWT with respect to the other factor .sx The unequal numbers in each social class would therefore reflect the true population differences .sx In this case the EWM main effect of dyslexics versus controls would be arbitrarily related to the categories of social class we had defined .sx Only an EWO analysis of the differences between the dyslexics and normals generalizes to the population of dyslexics .sx An EWM analysis refers to a population of dyslexics equally distributed across the social classes .sx As the relative sizes of the dyslexic and control groups are arbitrary , a simple effects analysis of social class effects in dyslexics and normals seems appropriate , or perhaps an EWM analysis of the differences between classes generalizing to the results of balanced design replications of this study .sx Case e .sx A hybrid study with a matched design .sx black-square We are again interested in differences in reading between a dyslexic group ( assumed to be a random sample from a population of dyslexics ) and a control group , together with the effects of social class on the two groups .sx Here the control group is a sample of children from the same age cohorts matched for social class .sx Here the design is OSCAR for the dyslexic group with respect to social class but in other respects it is ROWT .sx An EWO analysis of dyslexics versus controls tells us whether dyslexics differ from a sample of matched normals , whereas an EWM analysis tells us whether we could expect this difference in a balanced replication .sx Again a simple effects analysis or an EWM analysis should be used to investigate social class .sx To sum up , in two factor designs EWO tests of main effects are recommended for OSCAR designs generalizing to natural populations .sx In two factor balanced designs or where the interaction is assumed to be negligible , EWM tests of main effects can be used .sx Here one is generalizing to hypothetical results from balanced replications of the experiment .sx In all other cases no overall test of the main effects is possible and one should resort to simple effects analyses .sx A WORKED EXAMPLE .sx Suppose we obtained a random sample of students at a particular university and classified them by sex and whether they were studying predominantly arts or science subjects .sx The attitude of these students to proposed changes at their university was then measured on a twenty point scale ; the results are in Table 3 .sx tables&captions .sx This is an OSCAR design and an EWO analysis of main effects and an EWM analysis of the interaction is appropriate .sx The results of the analysis appear in Table 4 .sx It can be seen that both the main effects are significant ; sex at the 0.01 level and faculty at the 0.05 level .sx This means that we can conclude that in this university population males have greater attitude scores than females and science students have greater attitude scores than arts students .sx Since the imbalance in the study implies that the hypotheses being tested are related one could ask the question :sx " Suppose we performed a balanced replication could we still expect the differences to be present ?sx " This question could be answered by applying an EWM analysis to the main effects .sx The results of the analysis appear in Table 5 .sx It now appears that the main effect of the subject area is no longer significant .sx The F ( 1 , 32 ) testing for an effect of faculty has changed from 7.34 in the EWO analysis to 1.22 in the EWM analysis , from an effect almost significant at the 0.01 level to barely a trace of an effect .sx The conclusions of the analyses are that sex differences have been established in both OSCAR designs and ROWT designs balanced with respect to sex and subject area ; whereas the subject area difference has only been established with respect to OSCAR designs .sx It has not been demonstrated that we could expect to find a subject area difference in samples balanced with respect to sex .sx Now suppose the same data had been obtained but this time the arts and science groups were random samples of arbitrary size from the two faculties .sx This time the design is OSCAR with respect to sex but ROWT with respect to subject area .sx Using the ROWT analysis in Table 5 to generalize to the results of balanced designs is still legitimate .sx If we want to generalize to the population of arts and science students the tests applied to the subject area effect and the interaction in Table 4 still stand .sx However the main effect of sex cannot be tested without assuming the interaction is zero .sx Instead , separate simple effects analyses of sex in the arts and in the science groups should be performed .sx These appear in Table 6 .sx From the analysis we can conclude that there is a difference in sexes in both the science and the arts groups at the 0.01 level .sx From this we can be sure that there will be an overall effect of sex regardless of the distribution of arts and science students .sx If the sex effect had not been significant in both the groups we could not have concluded that there would be an overall effect of sex unless we assumed that the interaction between sex and faculty was zero .sx table&caption .sx The above analyses were performed using BMDP4V which has facilities for all the tests discussed so far .sx SPSSX also performs such analyses .sx If you do not have a big statistical package to hand , the EWO analysis of main effects can be performed by treating the analysis as one way and ignoring the other factors .sx Several methods exist for approximating to the EWM analysis when the design is unbalanced .sx For example , Bartlett ( 1937 ) used a procedure involving a multiple covariance analysis , and Winer ( 1971 ) gives a harmonic means analysis .sx These procedures may be useful where one does not have access to a modern computing environment but the EWM analysis is available on SPSSX , BMDP and SAS and is to be performed .sx FILLING IN FOR MISSING DATA .sx So far we have assumed that the only results we have are the observed data .sx Sometimes we know in addition that it was intended to make more observations in a particular condition but because of such factors as experimenter error , equipment failure , subject refusal , or failure to understand instructions , a known amount of data was lost .sx Missing data can only be handled by knowing the nature of the mechanism giving rise to it .sx The simplest case is where the data are Missing Completely At Random ( MCAR) .sx Here each observed value is equally likely to go missing .sx A more complex case occurs when some conditions are more likely to produce missing data than others but within each condition each observation is equally likely to be missing .sx This we have called Missing At Random Only Within Treatment ( MROWT) .sx Yet more complex cases arise when the missing data depend on the observed score ( for example , high values are more likely to go missing) .sx This changes the nature of the distribution of scores and one must either use nonparametric analyses or specify the distribution of the missing data .sx The case where missing data depend on the combination of the treatment and the value of the score makes the concept of average differences , in both observed and missing data , between conditions , exceedingly hypothetical .sx The more complex the assumptions necessary for the analysis the better it is to restrict oneself to inferences about observable data .sx Little & Rubin ( 1987 ) describe some such models which make complex assumptions about missing data .sx These will be discussed below .sx All the analyses we have considered assume that the data are MCAR or MROWT .sx This being so the missing data are ignorable , so why not just analyse the data using the procedures suggested earlier ?sx There are three answers .sx The analyses of unbalanced designs are less robust against violations in the underlying assumptions ( Milligan et al .sx , 1987 ) and imputing new data to restore the balance would also restore the robustness .sx Secondly , in the analysis of OSCAR designs , we may have tried to obtain data from a known number of subjects in each condition but have been unsuccessful in certain cases .sx Here it may be preferable to use the known number of subjects in each condition rather than the number of observations for which data exist , since this is a more accurate estimate of the relative sizes of the conditions in the target population .sx Finally , in appropriate circumstances , knowledge of a covariate can tell us something about the likely values of missing data .sx This information can be used to increase the precision of the analysis .sx The simplest way of handling missing data is to replace them with something else .sx Perhaps the most obvious way is to collect more data .sx Less desirable ad hoc methods include selecting data at random from previous studies or randomly sampling the current data and duplicating the selected points .sx An alternative procedure is to substitute the mean of the condition where the missing data occurred .sx This method reduces the error variance but this can be corrected using the actual numbers of observed data points .sx When this is done the analysis is equivalent to weighting the sample means with the frequencies of missing and observed rather than just the frequency of the observed data .sx