In my last post, I talked about why we should revamp and pivot our current data systems to respond better to our new, virtual landscape. Today I’ll be sharing some of my Do’s and Don’ts for pandemic data systems.
The best way to kill any new data practices or systems is to build in elements that everyone hates or serve no purpose. Here are some pitfalls to avoid:
- Obsession with absolute numbers: Weird suggestion coming from a self-professed data nerd, I know. Being data-driven doesn’t mean that users are just trying to hit a quota or a certain milestone number. It’s about using the numbers as guidance and confirmation that you’re heading in the right direction. I was happy to see that GreatSchools changed their rating formula to focus more on growth, and hope we all focus more on student growth instead of absolute numbers.
- Punitive measures: This summer there was a horrifying case where a Michigan student was sent to juvenile detention for missing virtual classes. In these trying times, both students and teachers are highly stressed and navigating this great unknown without existing precedents. Let’s use the data to celebrate growth and progress rather than punish people for not meeting pre-COVID targets.
- High-stakes testing: Tying SBAC scores to pay or anything would be a recipe for disaster right now. Everything is new, up in the air, and constantly changing. Imagine a 3rd grade class is extremely unlucky and gets hit with coronavirus at 3 different times during the school year, and instruction moves back into remote teaching each time. Multiple kids would be out sick at different times, resulting in multiple weeks of lost instruction. It would be inaccurate to judge a teacher’s performance based on high stakes testing during this time of crisis.
Data-savvy organizations (think Aspire or OUSD) have infrastructure and knowledge to transition to pandemic-responsive data practices and systems. (Shout-out to RAD at OUSD for this attendance dashboard!) But smaller charter organizations, single-site charters, or even individual district schools may be worried about increasing data transparency because it involves huge amount of additional work or resources. But I think it’s more about HOW schools leverage what’s already in place and getting the most mileage out of existing data practices and systems. Here are some ideas:
- Track physical barriers to remote learning: Schools don’t exist in a vacuum, and during this economic downturn, physical barriers are real and present for our students —sufficient device access, internet stability, quiet space, paper & pencils, housing, food security, etc. Some of these may be outside a school’s control or ability to address, but regularly tracking data around physical barriers enables a school to know who to refer to community partners for additional help. Given the economic uncertainty, up-to-date data is critical to identify aid as soon as possible.
- Find new ways to track mastery: Although the annual state testing in the Spring (SBAC) did not take place last spring (rightly so), we shouldn’t accept the lack of public insight into students’ progress. Data around students’ content mastery is critical in helping schools realize their primary mission. Without standardized tests, could we substitute on a dashboard of existing ed-tech apps as proxy for assessments used in the past?
- Incorporate new measures beyond just test scores: Let’s track non-academic indicators beyond just test scores, like log-in/attendance rates, assignment participation, or assignment completion as self-report measures of how much a student feels in community with their peers and how often a student feels safe in their (virtual) classrooms. And we’re not limited to just student interactions online; for example, with wellness checks, we can make them non-digital dependent by recording how often and how long someone from the school checks-in with a student. Over time, how many minutes of interaction did they get with the school? Did increased interaction help with engagement? Does the amount of minutes of interaction align with the school’s intention to spend more time engaging the most disengaged and vulnerable students?
- Double-down on tracking for inequities: Historically, Oakland has underserved certain students, particularly black and brown children. Unless we disaggregate and track data from the start with these subgroups in mind, it’s easy to hide the inequities by only looking at the whole group. Let’s use example from previous bullet point around tracking minutes of interaction. With disaggregated data, you could check whether the neediest students are getting the increased attention that they need. I believe that all educators are checking in with their students regularly in these trying times because they care about their kids. A built-in system to log and track these interactions doubly ensures that those students don’t fall through the crack.
- Get creative and use what you already have (or go free): We’re entering a new era of distance learning without many of the old forms of data that were available to us (bye SBAC). Pivoting existing data systems doesn’t need to take up a lot of work or time – schools can tweak how they use what they already have or look for free resources. Many existing classroom online platforms (that you were already using pre-pandemic) have built-in data tools, like assessments to track mastery or engagement rates. There are free and existing tools on the internet, like Khan Academy for Math, that can act as assessment tools.
Figure 1. Whole-class report from ClassDojo could be a proxy for engagement rates.
- Create a citywide data clearinghouse: Right now, there isn’t a place that stores data for all Oakland public schools. OUSD has their own set of data dashboards for their district-run schools, but each charter’s data is scattered across different corners of Oakland. It would be wise if there was a dedicated education leader (OUSD’s Office of Charter Schools?) that could work with all schools in Oakland, district-run and charter, to get everyone’s data in one place.
- Share the data publicly – with partners, parents and community members: This one is, I think, the most important element. OUSD is a local leader and have historically been transparent with their data. (Shout-out to Jean Wing, who retired this summer, for building such great data culture and infrastructure at OUSD!) OUSD’s public dashboards are my usual go-to data transparency standard, and I was excited to see that they have a few new COVID-related dashboards. But not every school has to have a data analyst creating Tableau dashboards to keep their parents and community supports in the loop! Public data sharing can in any form, even as simple as sharing out daily log-in rates in a monthly school newsletter or school portal page. It just needs to be easily accessible and digestible. Frequent communication is key!
If you’re looking for more rigorous, fleshed out principles for designing effective assessments during COVID-19 pandemic times, check out CPRE’s report, which is what inspired me to think about how a revamped data system might look like in Oakland.
I know the words “data tracking systems” invoke strong feelings of aversion for some and drowsiness in others, but I think it would be foolhardy to ignore all data entirely during this era of remote learning, where all students are online (guaranteed for at least part of the year and some will opt to go digital all year) and there are multiple opportunities to collect data. Done right, new forms of assessment and data cycles can help educators not only improve their teaching practices, but also help identify inequitable outcomes (intentional or unintentional) towards directing more interventions, resources, and dedicated time to the students who need it the most.