Principles of Measurement: Understanding the Process and Examples

1. Principles of Measurement Lunch & Learn Oct 16, 2013 J Tobon & M Boyle

2. What is Measurement? the process of assigning numerals to variables to represent quantities of characteristics according to certain rules. Objective directly observable (e.g., length) Subjective inferred from experience (e.g., sensations, perceptions, behaviour)

3. What are some examples of concepts you have/would like to measure?

4. Measurement Steps 1. Identify and define the concept 2. Specify the objective and conditions of measurement 3. Determine if a useful measure exists 4. Decide whether to use existing measure or develop a new measure Use Develop

5. Measurement Steps 5. Create a question/item pool and evaluate the relevance/importance of each item using a formal judgment approach and trim item pool 6. Choose an approach to scaling 7. Pre-test instrument with target respondents (interviews, focus groups, pilot data) 8. Conduct reliability and validity studies

6. Measurement Steps Reliability and Validity studies

7. Basic Measurement Concepts Development Evaluation

8. Basic Concepts What is a Concept? A mental representation, an abstract idea, the invented name for an attribute that we cannot observe directly but infer from the data of experience (indicators) e.g. SES, I.Q., anger Concepts are usually complex and need to be represented by multiple indicators

9. Basic Concepts What are the Objectives of Measurement? 1. To discriminate between individuals 2. To evaluate change within individuals 3. To classify individuals What are the Conditions of Measurement? Practical considerations associated with assessment and data collection (cost/burden)

10. Basic Concepts What is a Useful Measure? 1. Adequately represents the concept of interest 2. Satisfies the reliability/validity (empirical) requirements associated with the objectives of measurement 3. Meets practical requirements

11. Basic Concepts What is Scaling? The process of scaling is an attempt to upgrade measurement the numerical representation of attributes (e.g., increasing reliable variance or the meaning of score intervals)

12. Basic Concepts What are the Levels of Measurement? Nominal, ordinal, interval and ratio These levels determine the numerical operations available in an analysis Strength and form of association between variables will depend on capturing their underlying nature

13. Levels of Measurement NO SCHOOLING SOME PRIMARY COMPLETED PRIMARY SOME SECONDARY SECONDARY COMPLETED SOME COMMUNITY ETC COMPLETED COMMUNITY SOME UNIVERSITY COMPLETED UNIVERSITY 1 2 3 4 5 6 7 8 9 0 3 6 10 13 14 15 16 17 Interval Ordinal 0 1 Nominal

14. Rate your agreement with the following statements SA A N D SD I am knowledgeable about the signs and symptoms of student mental health issues 1 2 3 4 5 I am knowledgeable about appropriate actions to take to support student mental health at school 1 2 3 4 5 I am knowledgeable about legislation related to mental health issues (confidentiality, consent to treatment, etc.) 1 2 3 4 5 I am knowledgeable about school system services and resources for helping students with mental health issues 1 2 3 4 5 What is an Additive Scale? multiple items, same response, added up Basic Concepts

15. Single Item Multiple Items Signs & Symptoms MH Literacy

16. Basic Concepts What is a Good Additive Scale? It will be unidimensional It will cover the full range of meaningful values It will be scaled to the highest level of measurement It will be reliable and valid for its intended use

17. Basic Concepts What is reliability? the extent to which measures provide similar values on replication What are the types of reliability? 1. Internal consistency alpha (item sampling) 2. Test-retest (b/w respondent) 3. Inter-observer (b/w observer)

18. Levels of reliability constrain the magnitude of associations (Spearman Brown Prophecy) (Predicted) = (Observed) / (Rel_x)(Rel_y) 0.24 r XY = = 0.50 0.60 x 0.80 Observed Predicted Reliabilities of x & y Importance of Reliability

19. Basic Concepts What is validity? the extent to which measures adequately represent a concept and are useful for its intended objective What are the types of validity? Old: face, content, criterion, construct New: construct test hypotheses based on theoretical linkages with other measured concepts (nomological net)

20. Types of Hypotheses Differences between groups (discrimination) Sensitivity to change (evaluation) Accuracy of classification (prediction) Correlations with similar measures Types of hypotheses should align with the objective of measurement

21. Summary Concepts define the subject matter (boundary) of different scientific disciplines Measurement quality (level) defines the sophistication scientific disciplines Measurement is a two-step process that works within practical constraints (objectives and assessment conditions) and scientific prerequisites (reliability and validity)