Data analytics in K-12 online learning – Guest blogger Dr. Joe Cozart

Dr. Joe Cozart is guest blogging today. Joe is the Associate Director of Strategic Planning at Georgia Virtual School. Here he discusses learning analytics. He also recently did an iNACOL Research in Review Webinar. You can see the recorded webinar at HERE.

Using Data to Improve the Management and Operations of an Online School

Administrators of online and blended schools have access to more data than ever before, but the time and expertise required to use this data is not as easy to come by.  However, using readily available tools, there are questions administrators can easily use data to answer.  Georgia Virtual School is a supplemental program of the Georgia Department of Education offering online courses to students in grades 6-12.  The school provides example that may apply to other schools as well in the areas of: learning analytics, course survey analysis, budget analysis, and course enrollment forecasting.

Learning analytics

Learning analytics has helped Georgia Virtual School better understand what how successful students utilize the learning management system compared to underperforming students.  The decision tree image below, specifically node seven in the bottom right, shows that students who utilize the discussion tool more than 20 times per semester have a statistically significant higher final average than those who do not.  This was interesting to our administrative team because discussions only count five percent of the course grade, yet there is more than a five-point change in mean final grade when using the discussion large amounts.  It provides data to support the belief that students who actively engage with the content and with classmates do better in their courses.  Thus, our school continues to place emphasis on teachers fostering rich discussions even though it is a relatively small part of the overall course grade.

A second benefit to our school from learning analytics is increased efficiency.  When staff communicate with stakeholders on student progress, it is key to have information on each student readily available.  While reports in the learning management system and student information system show info for students one at a time, our analytics allow for data on entire classes to be seen in a single simple visual like the one below.  A teacher can call parents in an entire class while viewing information on how active each student has been in each area of the course, all with a single graphic.

The third focus in analytics is on course design.  The gradebook heatmap below shows all student grades for all students in a course, though this image is zoomed to a portion of that report.  The grade items are auto-sorted so the lowest overall scores are to the left.  Administrators can analyze images like this to see if certain units tend to always show up at the low end.  For problem areas, the pacing of the course can give more time to it or more focused content can be developed.  There is also the potential here for cost-savings by targeting course development only in weak areas instead of the entire course.  In terms of assignment, it is noteworthy which assignments have a variety of grades given and discriminate which students know the content from those who do not.  If the heatmap shows only 100s and zeros, the assignment may primarily be a participation grade and not particularly valuable to the learning process.

Course surveys

While many institutions use course surveys, not all schools administer the surveys in a way that makes the data easily aggregated and broken down for teachers.  Georgia Virtual School recently began using a new survey tool that automatically generates reports for teachers on their student survey data.  It also lets administrators see data for the area they oversee, whether it is a department or two, or the entire school. Initial analysis is showing that the student reports on teacher quality have a much better correlation with our own evaluations of staff than other measures like pass rates. Below is an example of part of a report a teacher could see.  Note the boxes below the graphs and how easy it is to compare the mean on each question for the teacher to overall department and school averages.

Budget analysis

Using simple excel spreadsheets, broke down teacher quality spending to determine the amount spent per teacher per year.  Showed the rate was higher than expected and not sustainable for future growth.  To address the issue, Georgia Virtual School developed a new teacher training course and moved it to a pre-hire process.  The course is free and open to the public at https://www.openteachertraining.org.  The training course breaks down the basics of online and blended teaching into five skills.  Users complete quests in each skill to earn a badge in the respective areas shown below.

Enrollment forecasting

Georgia Virtual School is a part-time, supplemental program used by hundreds of high schools across the state of Georgia and beyond.  Each semester, thousands of students take hundreds of courses, but the enrollments in individual courses vary greatly and are difficult to predict.  This makes it tough to ensure an adequate number of teachers is available in each course.  Specifically, Georgia Virtual School uses IBM’s statistical program SPSS with the forecasting package that includes time series analysis. In addition to providing a forecast, the program also reports the confidence level of the forecast.  The image below shows the forecasts for a few courses as well as the upper confidence level (UCL) and lower confidence level (LCL). While these are broad ranges currently, they improve each semester as more data is added and the model is refined.

– See more here