A Fairbanks high school math teacher and mom has begun to fill in the gap for those who have asked for analysis of the recent charter school study touted endlessly by Gov. Mike Dunleavy, Education Commissioner Deena Bishop and many Republican legislators.

West Valley High School teacher Beth Zirbes, who has master’s degrees in math and statistics from UAF, makes a clear case that the situation is far more complex than the simplistic pitch from Dunleavy and Bishop—who act as if creating new charter schools is a magic bullet that will solve every education problem.

I commend Zirbes for devoting many hours of her own time to try and put this study in context, showing its strengths and weaknesses and why more work needs to be done.

When legislators questioned the Dunleavy administration this year about charter schools, the education department couldn’t answer the most basic questions about demographics and performance. Bishop repeated false information about waiting lists based on anecdotes, not statistics.

I wrote here April 9 that Alaskans deserve an intellectually honest report from the state on the performance of charter schools, the makeup of the student population as compared to the general population, the reasons for different academic results and details on how much an expansion of charter schools could widen the gap between the haves and the have-nots.

Thanks to Zirbes, we are starting to get some of the information necessary for an informed debate. A presentation like the one that follows—based entirely on reports published by the state—should have been given to legislators, but the education department never came close.

This work needs to be followed up by others. The education commissioner has been singularly focused not on research and facts, but on selling Dunleavy’s plan to allow his appointees to the state school board to create new charter schools.

Here is the important column from Zirbes, first published in the latest newsletter of the Alaska Association of School Boards.

By BETH ZIRBES

When I first saw the results of the Harvard study concerning charter schools I was simultaneously unsurprised and skeptical.

I was unsurprised as I have seen many very bright young students in my Advanced Placement classes who have come from charter schools. I was skeptical as I suspected much of this performance could be attributed to the type of student attending Alaska’s charter schools.

As a comparative analysis of Alaska’s charter schools and neighborhood schools had not been done, I set out to do one myself.

To determine whether charter schools outperform neighborhood schools I looked at the performance of all schools on the 2018-2019 PEAKS ELA (English Language Arts) assessment from the Alaska Department of Early Childhood Education and Development’s (DEED) report card to the public, as this year was within the same time frame as the data from the Harvard study.

The performance of each school is given under the “2018-2019 Performance Evaluation for Alaska’s Schools (PEAKS)” tab.

I used the data on this page for every school in the state, which allowed me to analyze test scores and demographic characteristics such as the proportion of the school who are economically disadvantaged, English language learners (ELL), and in special education (SPED).

Demographic characteristics are only given for the set of test-takers and thus all summaries and analyses are for students in grades 3-9 during the 2018-2019 school year.

I also removed all correspondence schools from my dataset as these students were not included in the Harvard study, do not take the National Assessment of Educational Progress tests, and have very low participation rates on state tests.

For my analysis on performance, I also restricted my dataset to include schools only in districts where charters are an option to ensure that the student populations were as similar as possible.

At first glance, it appears that charter schools are more successful than neighborhood schools.

At charter schools 52.5% (1,866 out of 3,554) of students were proficient on the language arts assessment versus 40.1% (18,655 out of 46,574) of students at neighborhood schools.

However, these differences could be explained by the differences in demographics of the student bodies at these schools. To rule this out as a potential issue, statisticians control for these variables in their mathematical models.

We can then ask, do the charter schools outperform neighborhood schools that have similar characteristics?

Or do charter schools do any better than we would expect, given their student populations? In short, the answer is no, they do not.

When I fit a model which controlled for socioeconomic status alone, the type of school (charter versus neighborhood) was not significant.

(For anyone who knows statistics, the p-value associated with type of school was 0.57. It wasn’t even close.) In summary, there is no evidence that charter schools outperform neighborhood schools in terms of English Language Arts proficiency once we consider their socioeconomic make-up.

During my data exploration, I discovered that charter schools, on average, have very different student bodies than neighborhood schools.

Charter schools have far fewer economically disadvantaged students, far fewer students who are English Language Learners, and are comparable to neighborhood schools in terms of special education populations.

Here is a summary of how these populations differ for all Alaska students in the relevant grades in all of Alaska’s brick and mortar schools for the 2018-2019 school year:

•  Neighborhood schools were 52.2% economically disadvantaged (30,780 out of 58,929 students) compared to 31.3% in charter schools (1,219 out of 3895 students).

• Neighborhood schools were 15.5% ELL (9,150 out of 58,929 students) compared to 9.3% in charter schools (363 out of 3895 students).

• Neighborhood schools were 16.3% SPED (9,162 out of 58,929 students) compared to 13.7% in charter schools (532 out of 3895 students).

However, these summaries are highly influenced by a few outliers and obscure some large discrepancies, especially in terms of the economically disadvantaged and ELL students.

• Of the charter schools, 46.4% (13 out of 28) have economically disadvantaged rates below 20%, compared to just 3.5% of neighborhood schools (15 out of 426).

• Only 10.7% of charter schools (3 out of 28) have ELL percentages above 10%, compared to 36.9% of neighborhood schools (157 out of 426).

Much of the local reporting on the results of the Harvard study focuses on how Alaska’s charter schools rank at the top of the rankings presented in the paper but at the bottom with regards to NAEP rankings, which include all schools (but not correspondence schools). 

It is important to note that the Harvard charter school rankings and the rankings presented by NAEP are not comparable.

The Harvard charter school study ranked charter schools of each state while controlling for various factors, such as socioeconomic status, along with many other characteristics.

Comparing the rankings presented in the Harvard paper to those reported by NAEP is then meaningless, as the NAEP rankings do not control for any of these factors.

If one wanted to compare the performance of charter schools to neighborhood schools state by state, one would need to expand the dataset of the Harvard study to include all neighborhood schools in those states and then apply the methodology presented in the Harvard paper to determine the rankings.

Even if we did a comparison of charter schools and neighborhood schools and found that charter schools did better, we still cannot conclude charter schools are causing the performance difference we observe.

A comparative study like this is an example of an observational study and because it is impossible to control for all confounding factors, such as parental involvement, we can’t conclude success is caused by the school type.

To definitively conclude that charter schools were causing the observed difference in success compared to neighborhood schools we would have to randomly assign some students to go to a charter school and some students to go to neighborhood schools.

After some time, we would then compare the results. Obviously, this is impractical as many charter schools do not have busing, require parents to volunteer a certain number of hours, can remove students for poor attendance, and some do not even have lunch services.

Before the state uses the results of the Harvard study to change the approval process for charter schools we need to understand if charters are better and, if so, why.

So far, I have not seen convincing evidence that charter schools outperform neighborhood schools when we control for various student characteristics.

I have an idea for further study which I believe should be completed before any changes to policy are made.

We can examine the performance of students who got admitted to charter schools via the lottery to those who applied but did not get in and attended their neighborhood schools instead.

The group that was admitted is likely similar to those who applied but were not. This would be as close to a randomized experiment as one could hope to have.

From this one could determine whether various factors were causing differences in performance, such as class size and teaching methodology. Additionally, we could use results of such a study to determine which factors correlate with success and apply these strategies in all our schools.

As more than 90% of the students at our in-person schools are in neighborhood schools, such reforms will be more wide-reaching than simply adding a few more charter schools.

Beth Zirbes has a bachelor’s degree in mathematics from Gustavus Adolphus College in St. Peter, Minnesota and master’s degrees in both mathematics and statistics from UAF. During her maternity leave this spring she applied some of her skills to compare the performance of neighborhood and charter schools in response to the much-touted Harvard Charter School Study.

Your contributions help support independent analysis and political commentary by Alaska reporter and author Dermot Cole. Thank you for reading and for your support. Either click here to use PayPal or send checks to: Dermot Cole, Box 10673, Fairbanks, AK 99710-0673