Developing Indicators of Quality for AT Reuse Programs

1. 1 Developing Indicators of Quality for AT Reuse Programs Developing Indicators of Quality for AT Reuse Programs Lindsey Bean Kampwerth and Trish Redmon Moderators: Carolyn Phillips, Joy Kniskern and Liz Persaud Pass It On Center Pass It On Center www.passitoncenter.org www.passitoncenter.org

2. 2 Agenda Agenda – Vision and goals of Indicators of Quality – Vision and goals of Indicators of Quality – Overview of the development process – Overview of the development process – Themes of feedback on first two versions – Themes of feedback on first two versions – Design of Prototype IQ - ATR Online Assessment Tool – Design of Prototype IQ - ATR Online Assessment Tool – Next steps for IQ and Online Tool – Next steps for IQ and Online Tool

3. 3 Vision and Goals for Indicators of Quality for AT Reuse IQ - ATR Vision and Goals for Indicators of Quality for AT Reuse IQ - ATR • IQ - ATR is a living document that grows and develops as AT Reuse Programs develop. • IQ - ATR is a living document that grows and develops as AT Reuse Programs develop. • IQ - ATR Online will be a dynamic, interactive tool to assess an AT Reuse Program. • IQ - ATR Online will be a dynamic, interactive tool to assess an AT Reuse Program. • IQ - ATR will link to Pass It On Knowledge Base to provide references to information and models to assist with improving the Reuse Program. • IQ - ATR will link to Pass It On Knowledge Base to provide references to information and models to assist with improving the Reuse Program. 3

4. 4 Vision and Goals for IQ - ATR Vision and Goals for IQ - ATR 4 Knowledge Base supports IQ-ATR Self-Assessment Tool IQ-ATR and Knowledge Base inform Successful Practices Successful Practices inform Knowledge Base IQ - ATR IQ - ATR

5. 5 This is not the first project: Other Related QI Initiatives This is not the first project: Other Related QI Initiatives • The SMART Exchange • The SMART Exchange – Led by Shelley Kaplan, UCP in Atlanta – Led by Shelley Kaplan, UCP in Atlanta • 1989-1991, NIDRR grant • 1989-1991, NIDRR grant • Developed indicators for AT use • Developed indicators for AT use The rationale for many of the SMART Exchange indicators are relevant to AT reuse indicators. The rationale for many of the SMART Exchange indicators are relevant to AT reuse indicators.

6. 6 Other Related QI Initiatives Other Related QI Initiatives • QIAT • QIAT – Quality Indicators for Assistive Technology Services in School Settings – Quality Indicators for Assistive Technology Services in School Settings – Developed in 1998 – Developed in 1998 QIAT is an outstanding working model of quality indicators for AT use in a specific environment. QIAT is an outstanding working model of quality indicators for AT use in a specific environment.

7. 7 Validating the IQ-ATR Validating the IQ-ATR 1. Have we defined a set of factors or processes known to lead to successful outcomes? 1. Have we defined a set of factors or processes known to lead to successful outcomes? 2. Can we create a review instrument to determine if the factors/processes are in place? 2. Can we create a review instrument to determine if the factors/processes are in place? 3. Is the indicator useful for internal quality improvements? 3. Is the indicator useful for internal quality improvements? 4. Can the indicator be used as an external measure of quality across programs of the same type and activities? 4. Can the indicator be used as an external measure of quality across programs of the same type and activities?

8. 8 Developing the IQ - ATR Developing the IQ - ATR • Two Webinars on Quality Indicators • Two Webinars on Quality Indicators • A meeting of program professionals and PIOC team members in Atlanta to develop initial draft • A meeting of program professionals and PIOC team members in Atlanta to develop initial draft • Distribution of draft to developers and interested parties • Distribution of draft to developers and interested parties • Editing of draft based on feedback, including alteration of restrictive format • Editing of draft based on feedback, including alteration of restrictive format • Development of assessment tool prototype • Development of assessment tool prototype

9. 9 Developing the first draft Developing the first draft • Reviewed past quality indicators • Reviewed past quality indicators • Reviewed guidelines for writing quality indicators • Reviewed guidelines for writing quality indicators • Small groups edited and added to draft of quality indicators • Small groups edited and added to draft of quality indicators

10. 10 First draft done: Critique time! First draft done: Critique time! QI for AT Reuse Development Meeting in Atlanta, March 23 & 24 with occupational therapists, program directors and other key informants. Thank you! QI for AT Reuse Development Meeting in Atlanta, March 23 & 24 with occupational therapists, program directors and other key informants. Thank you!

11. 11 Categories of Indicators Categories of Indicators • Program Operations • Program Operations • User Services • User Services • Sustainability • Sustainability • Management • Management • Human Resources • Human Resources • Marketing • Accounting • Organizational Structure • Emergency Preparedness • Partnerships and Collaboration

12. 12 Suggestions from reviewers: Suggestions from reviewers: • Reorganize format: categories and sequence • Reorganize format: categories and sequence • Combine some topics; separate some topics • Combine some topics; separate some topics • Consider strictures of tabular format • Consider strictures of tabular format • Add rationale for categories • Add rationale for categories • Reorganize into “core” indicators and options based on program type • Reorganize into “core” indicators and options based on program type • Add specific AT State program indicators • Add specific AT State program indicators

13. 13 Version 2 Version 2 • Restrictive five-cell response format was abandoned in favor of a flexible checklist of key factors for consideration • Restrictive five-cell response format was abandoned in favor of a flexible checklist of key factors for consideration • Indicators were combined or split as needed • Indicators were combined or split as needed • New version was presented to National Task Force – which recommended a thorough language review • New version was presented to National Task Force – which recommended a thorough language review

14. 14 Version 2.1 Version 2.1 • NTF sub-committee (Rik Opstelten, Brian Bard, Kerri Morgan and Lindsey Bean) reviewed the language and recommended changes. • NTF sub-committee (Rik Opstelten, Brian Bard, Kerri Morgan and Lindsey Bean) reviewed the language and recommended changes. • Oops! Name change! QIAT recommended against expansion of their name and possible confusion. (Hence, IQ-ATR.) • Oops! Name change! QIAT recommended against expansion of their name and possible confusion. (Hence, IQ-ATR.) • Language updated, and some major issues were confronted before building the Online Tool. • Language updated, and some major issues were confronted before building the Online Tool.

15. 15 Major Challenges Major Challenges • Organizing the indicators into CORE indicators and implementing appropriate ‘branching’ based on program type • Organizing the indicators into CORE indicators and implementing appropriate ‘branching’ based on program type • Weighting the categories and the indicators to create an appropriate “result” measure for the program • Weighting the categories and the indicators to create an appropriate “result” measure for the program • Implementing online assessment tool and links to Knowledge Base • Implementing online assessment tool and links to Knowledge Base

16. 16 Organizing the Indicators Organizing the Indicators Variables affecting grouping: Variables affecting grouping: • Type of organization • Type of organization • AT reuse activities (reassign, refurbish, exchange, recycle) • AT reuse activities (reassign, refurbish, exchange, recycle) • Types of equipment (DME, computers, all?) • Types of equipment (DME, computers, all?) How to deal with the possible combinations? How to deal with the possible combinations?

17. 17 Weighting Categories & Indicators Weighting Categories & Indicators 1. Survey designed and individuals asked to rank: 1. Survey designed and individuals asked to rank: • Categories in order of importance • Categories in order of importance • Indicators in order of importance within categories • Indicators in order of importance within categories 2. Ph.D. statistician recommended weighting methodology based on ranking, but it required a standard number of responses. 2. Ph.D. statistician recommended weighting methodology based on ranking, but it required a standard number of responses.

18. 18 To weight or not to weight? To weight or not to weight? • Based on the ranking of categories and indicators, it is possible to produce a numeric “score” from the online tool. • Based on the ranking of categories and indicators, it is possible to produce a numeric “score” from the online tool. • For the initial assessment tool, it was decided not to produce numeric results, but advisory messages supported by lists of resources for each indicator. • For the initial assessment tool, it was decided not to produce numeric results, but advisory messages supported by lists of resources for each indicator.

19. 19 Version 3 Version 3 • Challenge: A return to the five-cell format would lose ease-of-use that made the flexible checklist of key factors so attractive • Challenge: A return to the five-cell format would lose ease-of-use that made the flexible checklist of key factors so attractive • Challenge: Too many variables to group indicators by program type • Challenge: Too many variables to group indicators by program type Solution: Solution:  Retain the flexible lists of key factors for consideration.  Retain the flexible lists of key factors for consideration.  Create standard set of responses to assess compliance.  Create standard set of responses to assess compliance.  Allow program to determine if indicator applies.  Allow program to determine if indicator applies.  No numeric scores – even hidden – at this time  No numeric scores – even hidden – at this time

20. 20 IQ-ATR and the Online Program Assessment Tool IQ-ATR and the Online Program Assessment Tool • Sharon Meek and the PIOC web developers from REACH, Inc. developed the prototype of the interactive tool • Sharon Meek and the PIOC web developers from REACH, Inc. developed the prototype of the interactive tool www.passitoncenter.org/iq-atreuse www.passitoncenter.org/iq-atreuse

21. 21 Home Page: A brief introduction to how the IQ-ATR tool works

22. 22 Registration Registration Create an account for personal use or register as a guest.

23. 23 Directions explain the response choices.

24. 24 Category in the yellow bar. The name of the indicator and the text of the indicator are followed by the Rationale and the Key Factors for Consideration. The standard set of responses follows. User selects a response and clicks Next.

25. 25 User receives a results page for each category and may stop or continue to the next category. KB or other sources are suggested for indicators that do not meet all Key Factors for the promising practice.

26. 26 Next Steps Next Steps IQ-ATR: IQ-ATR:  Share and solicit additional feedback  Share and solicit additional feedback  Revise IQ - ATR based on recommendations  Revise IQ - ATR based on recommendations  Design a validation process  Design a validation process

27. 27 Next Steps Next Steps Online Program Assessment Tool: Online Program Assessment Tool:  Gather feedback from pilot users  Gather feedback from pilot users  Revise online tool and make “updatable” to keep synchronized with Knowledge Base (KB)  Revise online tool and make “updatable” to keep synchronized with Knowledge Base (KB)  Allow user to choose sequence of categories  Allow user to choose sequence of categories  Have NTF review KB content for appropriateness and usefulness as recommended at May meeting  Have NTF review KB content for appropriateness and usefulness as recommended at May meeting  Identify or create content to fill reference gaps in Knowledge Base  Identify or create content to fill reference gaps in Knowledge Base  Decide whether responses should be weighted and tallied for a quantitative (rather than general) result.  Decide whether responses should be weighted and tallied for a quantitative (rather than general) result.

28. 28 Not enough knowledge. . . Not enough knowledge. . . • Referencing Indicators of Quality to content in the Knowledge Base highlighted the gaps in supporting information. • Referencing Indicators of Quality to content in the Knowledge Base highlighted the gaps in supporting information. • The content that does exist needs to be strengthened with more comprehensive presentation of needed policies and procedures. • The content that does exist needs to be strengthened with more comprehensive presentation of needed policies and procedures. • The existing content has not been reviewed by NTF panels as recommended in May 2009. • The existing content has not been reviewed by NTF panels as recommended in May 2009.

29. 29 How you can help How you can help • Review the IQ-ATR and send comments • Review the IQ-ATR and send comments http://passitoncenter.org/content http://passitoncenter.org/content • Try the online tool and send feedback • Try the online tool and send feedback http://passitoncenter.org/IQATReuse http://passitoncenter.org/IQATReuse • Contribute content or suggest resources for the Knowledge Base • Contribute content or suggest resources for the Knowledge Base Please send comments and suggestions to: Please send comments and suggestions to: Lindsey@passitoncenter.org Lindsey@passitoncenter.org Trish@passitoncenter.org Trish@passitoncenter.org

30. 30 DISCLAIMER This work is supported under a five-year cooperative agreement # H235V060016 awarded by the U.S. Department of Education, Office of Special Education and Rehabilitative Services, and is administered by the Pass It On Center of the Georgia Department of Labor – Tools for Life. However, the contents of this publication do not necessarily represent the policy or opinions of the Department of Education, or the Georgia Department of Labor, and you should not assume endorsements of this document by the Federal government or the Georgia Department of Labor.