Balancing Power to Generalize and Power to Convince in Education Research

1. Experimental, Quasi- experimental, and Single Subject Research 774/801 Sept 1, 2004 John Hattie & Tony Hunt

2. It is simple: There is no perfect experiment in education There is nearly always a trade off between the Power to generalise - from sample to population - from items to the behaviour domain - from conditions in the study to all intended conditions and the Power to convince - there are many audiences PG PC

3. Power to Generalise How confident can be generalise from the study to all similar situations Is the design replicable/reproducible/exchangeable? Is the evidence/conclusions unique to this study? Have the generalisations taken into account all possible competing views plausible alternative rival explanations (PARE) PG PC

4. Power to convince Who are we trying to convince If it is a colleague(s) then more situation specificity may be convincing (kids/classrooms/schools like mine) If it is the educational community, then situation needs to be less critical PC PG

5. Resolution: Linking Power Experimental design consists of a series of links: It is as strong as the weakest link Each link influences the next link Desirable to have equal strength Does each link have explanatory power Are conclusions credible to the intended audience

6. History Stanley and Campbell (1963) Cook and Campbell (1979) Shadish, Cook & Campbell (2002) Evidence based All based on designing studies that can lead to explanation and claims of causality

7. Explanation and Cause 1. Cause and effect must be related (e.g., self-concept & achievement) 1. There needs to be temporal order (cause before effect) 2. Need to rule out other explanations/ other Plausible Alternative Rival Explanations (PARE)

8. Campbell & Stanley (1963) Pretest-Posttest Control Group Design Pre Treatment Post R O X O R O O Randomisation aiming for representativeness

9. But can we randomise No Child Left Behind Tennessee Class Size Study

10. Quasi-experimentation: When you do not have so much control over allocation of treatment, conditions, sample When you have non-equivalent groups In quasi-experimentation, the researcher has to enumerate alternative explanations one by one, decide which are plausible, and then use logic, design, and measurement to assess whether each one is operating in a way that might explain any observed effect (Shadish, Cook & Campbell, 2002, p. 14) Relates to the Popper notion of falsification: What evidence would you accept that you are wrong?

11. Examples of Quasi-experimental designs Divorce Laws Ozdowski, S.A. & Hattie, J.A. (1981). The impact of divorce laws on divorce rate in Australia: A time series analysis. Australian Journal of Social Issues , 16 , 3- 17. a. Time Series O 1 O 2 O 3 O 4 O 5 X O 6 O 7 O 8 O 9

12. ABA design A B A O 1 O 2 O 3 X 4 X 5 X 6 O 7 O 8 O 9 Le Fevre, et.al. (2002). Adequate Decoders

13. Multilevel Design: Hierarchical Linear Modelling Students within classes within schools E.g., Tracking/Streaming School 1 School 2 Teacher 1 Teacher 2 Class 1 Class 2 Class 1 Class 2

14. Structural Equation Modelling

15. Minimal requirements for Studies Sampling Items to behaviour domains People to all possible people Conditions to all possible conditions Representative sampling via Random sampling Stratified random sampling

16. Variables At the end of your study, can I say Aha, so that is what you mean, now I am clear Open constructs NOT Definitions No such thing as immaculate perception Dependent - Manipulable Independent - Nonmanipulable

17. Dependability How reliable/consistent/replicable are your measures/ observations

18. Validity = Interpretations Validity - "an integrated evaluative judgement of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment". Not validity of a test, but validity of interpretations

19. Validity of your study Is related to having ruled out Plausible Alternative Rival Explanations (PARE) CONTROL CONTROL CONTROL some examples

20. 1. PARE: Power Is your study POWERFUL enough to detect the effect you are investigating

21. 1. PARE: Power Is your study POWERFUL enough to detect the effect you are investigating Do chickens have lips?

23. 2. PARE: Chance Did the effect/conclusion occur by chance E.g., That two means are the same the hypothesis of no difference Setting a rejection level, say =.05

24. 3. PARE: Type II errors Type I errors Rejecting a claim when it is true ( =.05) Type II errors Accepting a claim when it is false (e.g., chickens do not have lips, if it is indeed true)

25. 4. PARE Reliability of your measures If the reliability is low, then the scores wobble and no guarantee you will get same results using these instruments (tests, observations, interviews, etc.) Was the treatment consistent in the various classes/implementations?

26. 5. PARE: Was the treatment implemented? Degree of implementation The Hong Kong Practical Science Study (Cheung, Hattie, & Bucat, 1997)

27. 6. PARE: Maturation Showing change may not be enough as kids improve anyway (e.g., by maturation) Method to measure change = Effect-sizes Post-Pre/spread = Effect-size X2 X1 sd diff e.g., Before = 12, After = 15, spread = 6 15-12 = .5 6

28. Distribution of effects Zero achievement Average effect

29. Distribution of effects Maturation

30. The disasters 71 programmed instruction 801 .14 72 finances 1634 .14 73 problem based learning 41 .12 74 diet 255 .12 75 gender (female-male) 9020 .09 76 inductive teaching 570 .06 77 team teaching 41 .06 78 ability grouping 3355 .05 79 class size 2559 .05 80 open vs. traditional 3426 -.01 81 summer vacation 269 -.06 82 retention 3626 -.17 83 transfer of school 354 -.26 84 disruptive students 1511 -.78

31. The also rans 56 metacognitive intervention 921 .29 57 math programs 3326 .27 58 audio-visual 2699 .26 59 gifted programs 47 .25 60 coaching 1076 .24 61 behavior objectives 157 .24 62 calculators 238 .24 63 mainstreaming 1641 .21 64 questioning 493 .20 65 learning hierarchies 168 .19 66 attitude to math 1122 .19 67 desegregation 1590 .18 68 play 129 .16 69 television 4337 .15

32. Almost there 42 tutoring 136 .35 43 activity-based programs 674 .35 44 remedial programs 1438 .35 45 classroom climate 2726 .35 46 social skills training 5472 .35 47 time 1680 .34 48 CAI 18231 .32 49 inquiry based teaching 2740 .32 50 preschool 242 .32 51 whole language 198 .31 52 within class grouping 2359 .31 53 testing 1463 .31 54 problem solving 1141 .30 55 background 692 .30

33. In the middle 29 parent involvement 2597 .46 30 bilingual programs 1501 .46 31 adjunct aids 659 .45 32 concept mapping 18 .45 33 advance organizers 2106 .44 34 hypermedia instruction 317 .44 35 socio economic status 1657 .44 36 perceptual-motor skills 7592 .42 37 individualised instruction 5948 .42 38 homework 568 .41 39 competitive learning 144 .41 40 simulations 972 .37 41 expectations 912 .36

34. Worth having 14 self-assessment 152 .54 15 mastery learning 1933 .53 16 creativity programs 2340 .52 17 interactive video 1152 .52 18 psycho-linguistics 4404 .51 19 goals 959 .51 20 peer influence 366 .50 21 early intervention 30971 .49 22 outdoor education 294 .49 23 science 4124 .49 24 inservice ed 18644 .48 25 acceleration 371 .47 26 motivation 2196 .47

35. The MAJOR Influences Influence # effects Mean 1 Direct instruction 1925 .93 2 Reciprocal teaching 52 .86 3 Feedback 13209 .81 4 Cognitive strategy training 7649 .80 5 Classroom behaviour 361 .71 6 Prior achievement 2094 .71 7 Phonological awareness 2630 .70 8 Home encouragement 25706 .69 9 Piagetian programs 786 .63 10 Cooperative learning 1153 .59 11 Reading programs 14945 .58 12 Quality of teaching 808 .55 13 Study skills 3224 .54

36. Identifying that what matters

37. 7. PARE: Testing People become test wise and/or may respond different when under test conditions White space and testing in asTTle Testwiseness

38. Test of Objective Evidence Each of the questions in the following set has a logical or "best" answer from its corresponding multiple choice answer set. Please record your eight answers. 1. The purpose of the cluss in 2 Trassig is true when furmpaling is to remove A clump trasses the von A cluss-prags B the viskal flans, if the viskal is B tremails donwil or zortil C cloughs C the belgo fruls D pluomots D dissels lisk easily 3 The sigia frequently overfesks the 4. The fribbled breg will minter best trelsum because with an A all sigias are mellious A derst B all sigias are always votial B morst C the trelsum is usually tarious C sortar D no trelsa are feskable D ignu

39. Test of Objective Evidence, Part II 5 The reasons for tristal doss are 6 Which of the following is/are always present when trossels are being gruven? A the sabs foped and the doths tinzed A rint and vost B the dredges roted with the crets B vost C few rakobs were accepted in sluth C shum and vost D most of the polats were thonced D vost and plone 7 The mintering function of the ignu is most 8 effectively carried out in connection with A a razma toi A B the groshing stantol B C the fribbled breg C D a frailly sush D

40. 8. PARE Statistical Regression When taking extreme groups the means tend to move to the middle. Why do the tallest fathers have shorter sons, and the shortest fathers have taller sons?

41. . Regression to the Mean Special Education (e.g., Sesame Street) Effective schools Gifted education

42. 9. PARE Response rates The returns of questionnaires/tests/interviews should be high What is typical?

43. Meta-analyses of Response Rates Typical return is 50% Three major factors: 1. Salience (77% vs 42%) 2. Number of follow ups (halve each time) 3. Lack of clutter/ orderliness Not length (ave 7 pages, 72 questions), colour,

44. 10. Change scores The difference between post-pre scores Problems 1 Unreliable 2 Are you measuring same thing both times 3 Regression to the mean

45. 11 PARE: Experimenter effects Hawthorne effect: Because we know we are in an experiment this alters our responses Hans the Horse Pygmalion in the classroom Christine Rubies thesis Stanley Milgrims experiment

46. 12. PARE: Restriction of range When you choose/focus on a narrow range of abilities (etc.) this can be misleading Picture

49. 13. PARE: Specification of target and accessible sample/population Most experiments are highly local but have general aspirations Often, there are two groups you are generalising to: e.g., all secondary students in NZ, and to all secondary students you have access -- from which to sample

51. 14: Interactions The model of individual differences indicates that we should modify our teaching methods to allow for individual differences in the class

52. The art of research design is to devise experiments to identify the explanation and cause of effects by Maximising the chance that the conclusions are defensible and Minimising the PAREs Such that you have Power to Generalise and Power to Convince

53. Unobtrusive measures Which painting do most people watch? Friendship in cities Racism in suburbs/cities

54. Statistical Methods to assist Correlation Analysis of variance (anova) Cross-tabulation

57. Comparing means: Magnitude and Chance Magnitude Effect-sizes Chance Analysis of Variance

58. Well-being What are the differences in levels of WELL- BEING among males and females, and between Australia and New Zealand Country * GENDER Cross tabulation Count GENDER MALE FEMALE Total Country New Zealand 516 644 1160 Australia 421 694 1115 Total 937 1338 2275

59. Australia Mn sd Effect-size Male 45.7 10.6 Female 46.2 10.6 0.04 Total 46.0 10.6 New Zealand Male 53.6 7.5 Female 54.3 7.4 0.08 54.0 7.5 NZ - Australia 0.89

60. anova Source df MS F p Country 1 35211.9 416.71 <.001 Gender 1 151.8 1.80 .180 Country * Gender 1 6.1 0.07 .787 Error 2271 84.5