Competency-Based Education Quality Principle #11: Establish Mechanisms to Ensure Consistency and Reliability

This is the twelfth article in a series based on the book Quality Principles for Competency-Based Education. You can find the section on Principle #11 Establish Mechanisms to Ensure Consistency and Reliability on page 77. The links to the other articles can be found at the bottom of this page and will be updated as they are posted.

In the midst of writing all the papers for the Summit, a colleague said to me, “Pick consistency or reliability but not both.” I thought about it for a bit, but after looking up the definitions, I ended up feeling that both were actually important.

Let me start with reliability. It is a characteristic that is tragically lacking in the traditional system and has led us on the tumultuous path toward No Child Left Behind, Common Core of State Standards, and state accountability tests that are effective in identifying inequity but have been used to create a culture of shame, blame, and fear in our schools.

A dictionary definition of reliability includes “the quality of being trustworthy.” The standards movement and the drive toward top-down accountability policies grew out of the fact that our schools simply were not trustworthy. Students graduated from high school only to find they needed remediation courses at college. Parents were told that their A and B students were doing great only to find out later that their children were reading two or three levels below their grade. The mixed messages and false signals of our grading systems and varied expectations leads to low achievement, inequity, and this sense of mistrust that creates tension at the local level and has led to the voucher and charter policies as well as this idea that somehow school can and should be “teacher-proofed.”

Competency-based education seeks to and requires us to counter this trend toward policies in response to mistrust, instead building cultures and daily practices that create trust. And one of the big steps is to be reliable and, if there is variability, to be honest about it so solutions can be found.

Reliability is indeed built on the idea of consistency. Consistency is important in the traditional system as well, but on a different set of factors. The traditional system seeks consistency in terms of curriculum, scope and sequence, and pacing guides. At the extreme, every student should be on the same chapter in the same textbook on the same day — regardless of whether they already have mastered the content or have yet to master the prerequisite knowledge needed to be successful. Personalized, competency-based education demands that the learning expectations (competencies based on applying knowledge and skills and standards needed to develop the competencies) be held to a high degree of consistency while creating room for personalized pathways in terms of how students learn, how they demonstrate their learning, and the amount of instructional support and resources (including time) that is needed to be successful.

In personalized, competency-based education, the idea of reliability means “The degree to which the result of a measurement, calculation, or specification can be depended on to be accurate.” In other words, competency-based schools should be able to consistently determine or certify that students have ‘mastered’ the content and knowledge. When a competency-based school says that a student has mastered eighth grade math standards, parents and high schools should have complete trust that that is in fact the case. Maybe a student will forget some of it by the time they enter ninth grade, but after a refresher, they should be able to demonstrate that they can apply those standards.

During a site visit, Susan Bell, former superintendent of Windsor Locks School District, and David Prinstein, principal of Windsor Locks Middle School, highlighted the importance of reliability and consistency: “In the traditional system, it can mistakenly feel more precise because we use mathematics to determine the grade. In the mastery-based system, we have to make sure we are as objective as possible… We used to have teachers say that they wanted to give students who had worked hard the benefit of the doubt. Why is there any doubt? We need to have a system in which we can be confident of what students know.”

There are three aspects to building consistent determination or certification of learning:

• The level of depth of knowledge is aligned with the way the standard is written.
• Students successfully learn how to apply standards, both knowledge and skills, to new and challenging contexts. This is the process of transferring knowledge and skills. This means drawing on interdisciplinary knowledge to figure out which knowledge and skills are needed. It also means that students use and develop transferable skills such as analysis, creativity, problem-solving, collaboration, and communication.
• Gaps and fluency are addressed so students have prerequisite knowledge and skills, and can access them with ease so they can direct working memory to the complexity of applying knowledge.

There is in fact a fourth feature related to ensuring that students are developing the building blocks of learning and becoming lifelong learners. The challenge is that there is not enough clarity in the field of the developmental stages related to perseverance, mindsets, and agency to expect that schools can be accurate in measuring the learning and achievement levels. This will be a set of work that all of us will be involved in over the coming decade.

In order to ensure consistency in determining proficiency or certifying learning, teachers need to be fully familiar with what it means to be proficient at every standard that is included in the range of their student continuum of learning. This means that an eighth grade math teacher might need to be able to understand what proficiency looks like for third through tenth grade standards. As described above, they also need to understand proficiency in terms of the  application of learning at different performance levels.  This requires states districts and schools to create mechanisms for moderation. This has been a missing piece in our education system and has allowed the troublesome variability to continue. It is also helpful to have calibration processes that ensure consistency in “grading” so that students get similar feedback about what they need to do to reach proficiency.

An additional capacity that needs to be developed is performance-based tasks and assessment. Again, teachers need to be able to provide consistent feedback and have a moderated understanding of the higher order skills involved in applying knowledge and skills. States and districts can learn how to build the capacity for moderated performance-based assessments from New Hampshire’s PACE. They’ve done an outstanding job in building statewide capacity in a way that builds trust and shared understanding across multiple districts.

Given that the capacities related to this quality principle are essentially new capacities that have been missing in our education system, there is a lot of work to be done that will involve multiple levels of governance. In the short run, schools can start by building internal consistency, and districts can play a powerful role in creating consistency across their schools and in moderating the expectations between elementary and middle and between middle and high school.

Other partners will be needed to create moderating capacity throughout the education system. We’ll need higher education to work with districts to align between twelfth grade and college readiness. Leading states can co-design moderation processes with districts so we begin to build consistency of expectations between high income and low income communities. It’s likely that some type of organization will need to be created (or perhaps current organizations can create the capacity) to serve as intermediaries that support moderation. Wouldn’t it be interesting if accreditation agencies or other organizations actually sampled student work, similar to the role of New Zealand’s Qualification Authority in ensuring consistency in credentialing achievement in the second school National Certification of Educational Achievement?

Read the Entire Series:

1. Quality Principles for Competency-Based Education
2. Purpose-Driven
3. Commit to Equity
4. Nurture a Culture of Learning and Inclusivity
5. Foster the Development of a Growth Mindset
6. Cultivate Empowering and Distributed Leadership
7. Base School Design and Pedagogy on Learning Sciences
8. Activate Student Agency and Ownership
9. Design for the Development of Rigorous Higher-Level Skills
10. Ensure Responsiveness
11. Seek Intentionality and Alignment
12. Establish Mechanisms to Ensure Consistency and Reliability
13. Maximize Transparency
14. Invest in Educators as Learners
15. Increase Organizational Flexibility
16. Develop Processes for Ongoing Continuous Improvement and Organizational Learning
17. Advance Upon Demonstrated Mastery