The End of the Big Test: Moving to Competency-Based Policy

This post originally appeared at Getting Smart on May 19, 2015.

What’s next? With all the frustration surrounding NCLB and big end of year tests, what’s the new policy framework that will replace grade level testing? For a state ready to embrace personalized and competency-based learning, what are the next steps?

This post suggests the use of assessment pilots and innovation zones where groups of schools apply and become authorized to operate alternative assessment systems. But first, some background.

Jobs to be done. To get at the heart of value creation, Clayton Christensen taught us to think about the job to be done. Assessment plays four important roles in school systems:

1. Inform learning: continuous data feed that informs students, teachers, and parents about the learning process.
2. Manage matriculation: certify that students have learned enough to move on and ultimately graduate.
3. Evaluate educators: data to inform the practice and development of educators.
4. Check quality: dashboard of information about school quality particularly what students know and can do and how fast they are progressing.

Initiated in the dark ages of data poverty, state tests were asked to do all these jobs. As political stakes grew, psychometricians and lawyers pushed for validity and reliability and the tests got longer in an attempt to fulfill all four roles.

With so much protest, it may go without saying but the problem with week long summative tests is that they take too much time to administer; they don’t provide rapid and useful feedback for learning and progress management (jobs 1&2); and test preparation rather than preparation for college, careers, and citizenship has become the mission of school. And, with no student benefit many young people don’t try very hard and increasingly opt out.

But it is no longer necessary or wise to ask one test to do so many jobs when better, faster, cheaper data is available from other sources.

What’s new? There have been six important assessment developments since NCLB was enacted in 2002:

• Student internet access has improved sufficiently to support an expectation of frequent online learning and assessment.
• Performance assessment tools make it easier to construct, manage, and assess projects and standards-aligned prompts (see features on LDC CoreTools, and Buck Institute).
• Embedded assessments are incorporated into many forms of digital content.
• Formative assessment systems have improved dramatically. Platforms like MasteryConnect, Acuity, Edmodo, OpenEd, and Schoology make it easy to build, administer, and share standards-aligned assessments.
• Adaptive assessment, such as MAPS from NWEA, is widely used. Adaptive learning, which combines adaptive assessment and targeted tutoring, is gaining widespread use in blended learning models. Providers include DreamBox (K-8 math) i-Ready from Curriculum Associates (k-8 math and reading), ALEKS from McGraw Hill (mostly secondary use).
• Broader aims of student success, including self management and relational skills, are widely recognized as important and are being incorporated into state and district goals. The hard to measure skills and dispositions require broader feedback systems than traditional standardized testing.

Progress has been faster on these individual vectors than on combining formative assessment from various sources into useful information for students, teachers and parents. Simply tracking a checklist of skills won’t cut it, solutions will need to include micro-standard tagging grouped into skill clusters, a two or three layer hierarchy supporting the ability to combine fine grain and broader performance assessments. Examples of progress on alignment through tagging include:

• Houston ISD has had some success requiring content and assessment partners to use the IMS Global Thin Common Cartridge, which enables content interoperability using Learning Tools Interoperability (LTI).
• Some learning platforms combine multiple forms of assessment around units of instruction (see feature on Buzz from Agilix).
• Scootle is an Australian library of tagged learning resources suggesting that a state or network could encourage strong alignment.

And, as noted last month, Without going through the trouble of tagging data or putting up with the limitations of a closed system, it is increasingly possible to subject to correlate data sequences from different digital learning applications after the fact by analyzing thousands of data points. IMS’s Caliper Analytics standards support both post hoc and real-time data feeds for millions of students daily.

Next steps. The first step for a state, according to the The iNACOL State Policy Frameworks is for the state board and/or legislature to, “Redefine Carnegie Units or credits as competencies aligned to state academic standards.” The Great Schools Partnership has supported movement towards “proficiency-based diplomas” in Connecticut, New Hampshire, Maine, Vermont, (see a summary of state policies).

For states ready to embrace personalized and competency-based learning, CompetencyWorks, an online community and resources supported by iNACOL, outlines five components of competency-based education (CBE):

• Students advance upon mastery.
• Competencies include explicit, measurable, transferable objectives that empower students.
• Assessment is meaningful and positive learning experience for students.
• Students receive timely, differentiated support based on their individual needs.
• Outcomes include application of knowledge and development of important skills and dispositions.

The definition sets a high bar by requiring well stated learning targets, powerful learning experiences, better reporting systems, and new rules for matriculation management. It focuses primarily on the first two jobs: student learning and progress management.

High school options. There are two test-based options to replace an end of the year standardized test in high school:

• End of course exams. A system of end of course exams has advantages during secondary education as course taking broadens and rates of progress diverge.They can be offered on a frequently scheduled basis so students can take them when ready. Like the New York Regents and Advanced Placement exams, end of course test have been most common in high school but could be used in middle grades. They facilitate incorporation of online, AP, and dual enrollment courses and promote reliability but lock in the structure of courses and credits and lack the validity and authenticity of student work.
• College entrance exams. Since many college bound students take the SAT or ACT, they are used as part of the state testing system. Eighteen states already administer the ACT and three more make it available to districts.

The Great Schools Partnership suggests two curriculum-aligned approaches:

• Body of evidence: using common criteria, educators review a portfolio of student work. It encourages students to take greater ownership over the learning process but is time-consuming and can be inconsistently executed.
• Compilation of assessments: using a common formula to aggregate assessment results on performance indicators over time. Results are relatively straightforward and easy to calculate but it may encourage students to narrowly focus on grades and numerical indicators of success.

The test based options tend to be more reliable while the student work product approaches are more valid and authentic. A jurying process for portfolios can boost reliability but adds cost and complexity. A state could combine both approaches by requiring a series of several ACT tests (Plan, Explore, Compass) and incorporating them into a body-of evidence or complication of assessments approach. A state could also combine short end of course exams with a body of evidence approach to gain affordable validity and reliability.

New K-8 systems. When federal education legislation is reauthorized it is likely to retain the annual testing requirement in grades 3-8. That’s a good thing because we don’t have alternative ways to comparably measure student learning levels and growth rates to verify school quality (job #4).

But the beginning of the end of the big test is in sight and two replacement options are being contemplated. The first is assessment flexibility pilots. New Hampshire’s Performance Assessment of Competency Education (PACE) is the first federally approved accountability system that offers a reduced level of standardized testing together with locally developed common performance assessments designed to support deeper learning through competency education.

This year, four PACE implementing districts gave the Smarter Balanced assessment once in elementary school, once in middle school and once in high school, in three grades instead of seven. The state is also conducting a readiness survey to identify the next eight districts to include in the pilot. Next year PACE districts will administer common and local performance assessments developed by the districts. For comparability, Smarter Balanced Achievement Level Descriptors are the basis for establishing cut scores. Regional calibration scoring sessions will be conducted to build inter-rater reliability and consistency in scoring across districts.

Maria Worthen, iNACOL explains, “Far from fewer assessments, New Hampshire must continue to administer the SBAC in three grade spans to evaluate and validate the new system of assessments. No other state is close to gaining approval for a similar assessment for accountability purposes.” Federal reauthorization discussions have contemplated inclusion of an innovative assessment pilot in the bill.

The other alternative path forward is the use of innovation zones where systems earn the right to opt out of testing by demonstrating that their proposed alternative assessment system can report comparable results. Districts, networks, and consortia could apply to become a ‘certified assessment system.’ These systems would need to certify that they could report learning levels and growth rates for students in ways comparable to (but not limited to) state administered tests before being admitted into the innovation zone.

As noted last fall, the innovation zone strategy could be implemented by an a statewide authorizer using performance contracting to engage districts and networks in experiments with new delivery strategies, talent development structures, and measurement systems. With a combination of pressure and incentives over five to 10 years, an innovation zone could become the primary delivery and accountability system for the state.

In summary, assessment pilots would run parallel systems to demonstrate comparability in action while innovation zones would require demonstrated comparability before receiving a waiver. Federal reauthorization of elementary and secondary act should include provisions for both of these options. Susan Patrick, iNACOL, said, “States desperately need the flexibility to be able to do this to innovate.” After a few dozen districts in a handful of states demonstrate new lightweight testing systems the rest of the states are likely to follow suit.

Tom Vander Ark is author of Getting Smart: How Digital Learning is Changing the World and founder of GettingSmart.com. Tom is also CEO of Open Education Solutions and a partner in Learn Capital, a venture capital firm investing in learning content, platforms, and services with the goal of transforming educational engagement, access, and effectiveness.