Not as Simple as It Seems: Generating Evidence for the Claims of Competency-Based Education

This is the fourth post attempting to capture the conversation and lessons learned shared at CBExchange. Other posts include: (Understanding Variation in Higher Ed CBE, It’s a Crazy Hot Mess…or Everything You Need to Know about Financial Aid, and The Magic of the Seal: Developing a New Student Record and Extended Transcript).

FYI – Although I did my best to capture the discussion, I’m sure I left gaps along the way. As always, we invite comments or blogs to clarify, correct, or expand on these issues. It’s all about learning.

At the session on Continuous Improvement through Innovation: Leveraging Data to Drive Results and a Research Agenda at CBExchange, Becky Klein-Collins, CAEL, opened the conversation with the question, “How can we collect evidence to support the growth of our field and the improvement of our programs?” She explained that as a movement, institutions of higher education (IHE) need evidence of the claims that are being made to advance significant policy change.

Co-presenter Matt Soldner, AIR, provided a quick look at the key claims about the value of competency-based education in higher education:

1. Time to degree is shortened
2. Graduates better prepared for workforce
3. Assessment loop bolsters student learning
4. New approaches reach unserved student populations (older adults who are working)
5. Approach is more cost-effective

He explained that as true believers (I still haven’t stopped laughing at his reference to those of us who are true believers walking around with with a tattoo of Mike Offerman on one shoulder and Deb Bushway on the other), we need to think carefully about the most important value propositions about CBE and how to generate the data that can support them. In other words – our passion is powerful, but not sufficient.

Klein-Collins and Soldner then guided us through a thoughtful conversation about the challenges IHE face. First, the variability of programs and lack of common definitions make generalizing hard. Given the fact that no CBE program is quite like the others, IHE have to understand the similarities and differences in order to use research and evaluation data to inform program development. Second, all claims are not as simple as they seem. In order to help IHE understand the complexity, they walked us through the claims.

Claim #1: Time to Degree is Shortened

They used the claim “Time to Degree is Shortened ” as an example. What else should we consider in thinking about this claim? Students bring different sets of experiences, different sets of skills, and different demands in their lives to any program – each of which will have impact on the amount of time it takes to complete a degree. Program design will also have impact. Think about the differences between designing accelerated programming as compared to self-paced, or the structure of standard or non-standard term to no-term programs. Students also have different goals, different work experiences, different skills, and different demands in their lives. Furthermore, many of the CBE programs are designed to attract working adults in markets that haven’t even been served in the past – so it’s a bit difficult to think about how to say the time is shortened.

Soldner then introduced a “Dork-Out” moment by asking the question, Compared to Who? He emphasized this with, “What will tank your research is to make a comparison to students who have nothing in similar to your CBE students.” He explained that IHE “need to show internal validity that CBE is the main reason why time to degree is lessened.” He then refreshed our knowledge about evaluation by walking us through three approaches.

• Random assignment: It’s the strongest method but rarely realistic.
• Propensity score matching: This approach matches CBE students with one or more non-CBE student not like them, but will require some statistical expertise to do.
• Coarsened matching or blocking: It’s the easiest to implement but weakest approach to reduce bias.

Other issues raised included:

• How tricky it’s going to be to handle prior knowledge in any comparison.
• Some programs are directing students to online or CBE-online program based on their prior knowledge and ability for self-directed learning.

Claim #2: Competency-Based Education Graduates are Better Prepared for The Workplace

Klein-Collins and Soldner pointed out that this claim implies a comparison group of graduates from non-CBE programs, which requires IHE to engage employers and graduates to be able to support it. Again, given that CBE programs are often designed for non-traditional students who wouldn’t otherwise be in any other program to complete their college degree, it may be difficult to find comparison groups. This really does put IHE in a bit of quandary to be able to create an adequate comparison group for this claim.

Examples of indicators for workplace preparedness might include:

• Employer satisfaction with new hires
• Time to promotion or pay raise
• Workplace specific/job-specific performance measures
• Employers start to contact your institution to hire graduates

Data sources for this claim might include:

• Survey of new graduates
• Survey/interview of employers
• Jobs placement information from career services
• Long term employment and salary data

Claim #3: Assessment Loop Bolsters Student Learning

This claim was described as “mushy” and too large. The recommendation was to try to reduce it to a more manageable size and scope. For example, IHE could create a smaller, self-contained study such as thirty students receiving rich feedback through assessment (treatment group) with thirty student not receiving rich feedback (control group), and then look at which students performed better. However, as one participant pointed out, there is a risk in such a study that “if you don’t give feedback, students might not build their skills adequately and would slow down their progress.”

Claim #4: CBE is More Inclusive

Klein-Collins and Soldner pointed out that this claim would have to be clarified by asking, “Is this a claim about access (more students will be interested in participating), success (all students will do better), or both?” The data you would need would depend on the answer to that question. For example, access would require IHE to look at how demographics change as they expand CBE programs and whether demographics of CBE programs differ from non-CBE students.

Claim #5: CBE is More Cost Effective

Klein-Collins and Soldner explained that the first step for this question is to clarify whether you are looking at cost or price. The cost is how much you deliver and the price you charge – and its not always easy to put your finger on either one. Resources are hard to cost out across the institution, as there are different perceived values and different amounts of learning taking place.

Continuous Quality Improvement

By this time, I have to say that I was feeling like any evaluation was going to be really difficult. And then Klein-Collins and Soldner turned the conversation to continuous improvement, introducing the simple and powerful Plan-Do-Study-Act process to improve processes and results.

They suggested that potential quality improvement metrics might include:

• Time for students to complete competency units
• Pass rates on specific authentic assessments
• Degree of student engagement
• Overall cost of the degree to the student
• Faculty roles and responsibliities
• Turn-around time for assessment scoring
• Student and employer satisfaction

They also provided a easy to use six-question tool for getting started on research and evaluation:

1. What questions do I need to be able to answer about CBE on my campus?
2. What data about CBE participants would I need to answer my question?
3. What data aobut non-CBE students would I need to answer my question?
4. What partnerships and resources could I leverage on campus to answer my question?
5. What challenges exist in answering my question?
6. Once I have an answer, how will I use it?

Key Collaborators

Finally, think about who you are going to need to involve to frame the question and access the data. A potentially untapped resource for looking at any of these claims are assistant professors (and their students) who would like to do research. Within IHE, the registrar and marketing (they could be skewing who is enrolling based on their strategies) are going to be important to involve. Some of these questions will also require IHE to involve graduates and employers.

Thanks to Becky Klein-Collins and Matt Soldner for a great conversation!

• Becky Klein-Collins is the associate vice president of research and policy development for the Council for Adult and Experiential Learning (CAEL), with responsibilities in overseeing and conducting new research to benefit adult learners. Special areas of focus include prior learning assessment and competency-based education. Her work also focuses on mature learners, student veterans, workforce development, and new approaches for policy change at both the federal and state levels.
• Matt Soldner is a senior researcher in the Higher Education practice area at the American Institutes for Research. Current areas of focus include college and career readiness and success, innovations in student financial assistance, postsecondary models of competency-based education, and the strengthening of institution-level performance metrics. Prior to joining AIR, Dr. Soldner was a senior technical advisor for the U.S. Department of Education’s National Center for Education Statistics. His Ph.D. is from the University of Maryland.

See also:

• Department of Education Wants to Talk CBE
• Moving Past Summative vs. Formative Assessments
• What Comes after “Next”?