First year evaluation results show promise

Feedback is central to teaching and learning at Rising Academies. Students and teachers learn to give and receive feedback using techniques like Two Stars and a Wish or What Went Well...Even Better If. The Rising Academy Creed reminds us that "Our first draft is never our final draft." Given that, it would be pretty strange if Rising as an organisation didn't also embrace feedback on how well we are doing at enabling more children to access quality learning.

That's why, even as a new organisation, we've made rigorous, transparent monitoring and evaluation a priority from the outset. Internally, we've invested in our assessment systems and data. But my focus here is on external evaluation, because I'm excited to report that we have just received the first annual report from our external evaluators. If you want to understand the background to the study and our reactions to the first annual report, read on. If you're impatient and want to jump straight into the report itself, it's here.

Background

Last year, we commissioned a team led by Dr David Johnson from the Department of Education at Oxford University to conduct an independent impact evaluation of our schools in Sierra Leone.

The evaluation covers three academic years:

• (The abridged) School Year 2016 (January-July)
• School Year 2016-17 (September-July)
• School Year 2017-18 (September-July)

The evaluation will track a sample of Rising students over those three years, and compare their progress both to a comparison group of students drawn from other private schools and government schools.

The overall evaluation will be based on a range of outcome measures, including standardised tests of reading and maths, a measure of writing skills, and a mixed-methods analysis of students' academic self-confidence and other learning dispositions.

The evaluation is based on what is known as a 'quasi-experimental' design rather than a randomised controlled trial (unlike our schools in Liberia, where we are part of a much larger RCT). But by matching the schools (on things like geography, fee level, and primary school exam scores), randomly selecting students within schools, and collecting appropriate student-level control variables (such as family background and socio-economic status) the idea is that it will ultimately be possible to develop an estimate of our impact over these 3 years that is relatively free of selection bias.

Figure 1: How the evaluation defines impact

BASELINE

To make sure any estimate of learning gains is capturing the true impact of our schools, one of the most important control variables to capture is students' ability levels at baseline (i.e. at the start of the three-year evaluation period). This allows for an estimate of the 'value-added' by the student's school, controlling for differences in cognitive ability among students when they enrolled. Baselining for the evaluation took place in January and February 2016. The baseline report is available here. It showed:

• That on average both Rising students (the treatment group) and students in the other schools (the comparison group) began their junior secondary school careers with similar ability levels in reading and maths. The two groups were, in other words, well matched;
• That these averages were extremely low - for both reading and maths, approximately five grades below where they would be expected to be given students' chronological age.

YEAR ONE PROGRESS REPORT: RESULTS

The Year One Progress Report covers Academic Year 2016. The Ebola Crisis of 2014-15 disrupted the academic calendar in Sierra Leone. Students missed two full terms of schooling. The Government of Sierra Leone therefore introduced a temporary academic calendar, with the school year cut from three terms to two in 2015 (April-December) and again in 2016 (January-July). The normal (September-July) school year will resume in September 2016.

The Progress Report therefore covers a relatively short period - essentially 4.5 months from late January when baselining was undertaken to late June when the follow-up assessments took place. It would be unrealistic to see major impacts in such a short period, and any impacts that were identified would need to be followed-up over the next two academic years to ensure they were actually sustained. As the authors note, "it is a good principle to see annual progress reports as just that – reports that monitor progress and that treat gains as initial rather than conclusive. A more complete understanding of the extent to which learning in the Rising Academy Network has improved is to be gained towards the end of the study."

Nevertheless, this report represents an important check-in point and an opportunity for us to see whether things looking to be heading in the right direction.

Our reading of the Year One report is that, broadly speaking, they are. To summarise the key findings:

• The report finds that Rising students made statistically significant gains in both reading and maths, even in this short period. Average scaled scores rose 35 points in reading (from 196 to 231) and 36 points in maths (from 480 to 516). To put these numbers in context, this change in reading scores corresponds to 4 months' worth of progress (based on the UK student population on which these tests are normed) in 4.5 months of instruction.
• These gains were higher than for students in comparison schools. The differences were both statistically significant and practically important: in both reading and maths, Rising students gained more than twice as much as their peers in other private schools (35 points versus 13 points in reading, and 36 points versus 4 points in maths). Students in government schools made no discernible progress at all in either reading or maths. (For the more statistically inclined, this represents an effect size of 0.39 for reading and 0.38 for maths relative to government schools, or 0.23 for reading and 0.29 for maths relative to private schools, which is pretty good in such a short timespan.) 
• The gains were also equitably distributed, in that the students who gained most were the students who started out lowest, and there were no significant differences between boys and girls.
• Finally, there are early indications that students' experience of school is quite different at Rising compared to other schools. Rising students were more likely to report spending time working together and supporting each others' learning, and more likely to report getting praise, feedback and help when they get stuck from their teachers.

That's the good news. What about the bad news? The most obvious point is that in absolute terms our students' reading and maths skills are still very low. They are starting from such a low base that one-off improvements in learning levels are not good enough. To catch-up, we need to sustain and accelerate these gains over the next few years.

That's why, for example, we've recently been partnering with Results for Development to prototype and test new ways to improve the literacy skills of our most struggling readers, including a peer-to-peer reading club.

So what are my two stars and a wish?

• My first star is that our students are making much more rapid progress in our schools than they did in their previous schools, or than that their peers are making in other schools they might have chosen to attend;
• My second star is that these gains are not concentrated in a single subset of higher ability students but widely and equitably shared across our intake;
• My wish is that we find ways to sustain these gains next year (particularly as we grow, with 5 new schools joining our network in September 2016) and accelerate them through innovations like our reading club. If we can do that, and with the benefit of 50% more instructional time (as the school year returns to its normal length), we can start to be more confident we are truly having the impact we're aiming for.

Take a look at the report yourself, and let us know what you think. Tweet me @pjskids or send me an email.

Rising Academy Partnership Schools launched in Liberia

Monday September 5th marks the start of the new school year in Liberia - and with it the start of a new relationship between Rising Academies and the Government of Liberia.

From today, five government elementary schools – three in Bomi County and two in Montserrado County – will become Rising Academy Partnership Schools. They will remain in public ownership, free to attend and non-selective, using qualified government teachers on the government payroll, observing the Liberian National Curriculum, and with government retaining responsibility for the physical upkeep of the school buildings. But responsibility for the day-to-day management of the schools and for improving the quality of teaching and learning will pass to Rising.

This effort is part of Partnership Schools for Liberia (PSL), a bold and deliberately experimental pilot programme to explore whether bringing in operators from outside government can help address the chronic crisis of education quality in the public system.

The case for change is compelling: for every 100 children of primary school age in Liberia, only 38 attend primary school, of whom only 23 will complete grade 6, of whom only about 8 will make it through secondary school and sit the WAEC exams at the end of grade 12, of whom only about 4 will pass. Along the way, even the kids in school aren’t learning what they need. To be considered fluent readers, Grade 3 students given an age-appropriate text ought to be able to read 45 to 60 words per minute correctly; in Liberia, the average is less than 20 words per minute.

Much has been written about the PSL programme – in The New York Times, The Guardian, Vox and HuffPo among others. Unfortunately, early (mis)conceptions about the programme have proved hard to shake. For a clear and comprehensive account of the programme from two people who actually know what they are talking about, this piece by Susannah Hares and Justin Sandefur is the best place to start.

Here’s the short version: under PSL, 90 primary schools (less than 3% of the total) will be handed over to outside operators to run in 2016-17. As well as Rising, these operators include international NGOs like BRAC, Streetchild and More Than Me, private school operators like Omega Schools and Bridge International Academies, and Liberian organisations Stella Maris Polytechnic and the Liberia Youth Network. As a relatively new organisation, Rising is proud to be among such distinguished company.

Operators are paid a fixed per capita grant for each student enrolled, and then held accountable for using this money to improve learning outcomes for students. If operators do well, they might be allowed to expand to manage additional schools in future; if they fail, they might be stripped of the schools they are running. If the programme as a whole shows it can make a difference to the quality of schooling, it might be expanded; if the programme as a whole fails, it will be shut down.

The key point is that these decisions will be based on rigorous evidence. A major attraction of the programme is that it gets beyond unhelpful and ideological debates about who should run schools and focuses on getting the data. PSL has been designed as a ‘gold standard’ randomised controlled trial, with schools randomly assigned to a treatment condition where they are run by a PSL operator or a control condition where they are not. Comparing what happens in these two groups of schools over time should therefore provide a reliable estimate of what difference (if any) it makes to have an outside operator. Formal details of the evaluation are on the AEA’s RCT Registry here.

The speed at which the PSL programme has moved from idea to implementation is staggering, particularly in a country where, as Liberia’s President Ellen Johnson Sirleaf has admitted, ambition is often thwarted by a lack of capacity in the system to get things done. Some operators have had a head-start, but in Rising’s case we only submitted our initial Expression of Interest on 4th May, only got the green light from government in mid-July, and were only notified which schools we were being asked to run in early August (and in one case even later than that).

Nevertheless, it was absolutely right for the Government to try to move quickly, rather than delay another whole year and risk losing momentum, and we have been doing our best to make the most of limited time available.

Candidates attend Rising's assessment centre

A major focus has been on staffing, screening existing staff and undertaking an urgent recruitment exercise. Our initial assessment showed that our schools had less than half the number of teachers they were supposed to have. In one case, a Principal had been assigned a teacher who had collected salary but never showed up for 3 years. That’s meant On August 23rd, we held an assessment centre for more than 200 recent teacher training graduates, with 12 candidates appointed to take up positions in our schools.

Teachers work through a card exercise during training

A second major focus has been on training. Through our work in Sierra Leone, we have developed an effective teacher pre-service training programme. Rather than lots of theory, teachers are given a small number of specific, high impact, practical skills to practise and receive feedback on. Rapid improvements in the level and quality of student engagement are possible as a result. Without the time to do the programme in full, 28 existing teachers received an abridged version of the training, with intensive in-service training taking place each afternoon for the first two weeks of term, and further training scheduled for later in the term.

Many schools lack basic infrastructure

A third focus has been on fixing some of the basic infrastructure in the schools. Bigger issues like leaky roofs remain the responsibility of government. But most of the schools didn’t have enough desks and chairs, and those they had were in disrepair, so we’ve procured more than 250 items of furniture to address the most urgent needs.

Finally, one of the freedoms granted to operators under PSL is to innovate with how the National Curriculum gets delivered. Some elements of our approach, like our highly effective phonics programme in partnership with Phonics International, remain just as relevant in Liberia as they do in Sierra Leone. But in other areas our international team of curriculum writers have been hard at work producing lesson plans and materials that will be appropriate for this new context.

With PSL, Rising embarks on a new and unfamiliar journey: a new country, and a new way of working. Among the teachers and principals, there is already a sense of excitement about what we might achieve together. Among parents too: student enrolment is up as parents get to hear about our new role in their local school.

Unlike so many traditional education programs which seek to raise the quality of outputs simply by increasing the number of inputs, PSL starts by correctly identifying the source of the education crisis as the way that schools are managed and held to account. With the right management and accountability, rapid improvements in student achievement are possible; without them, the system will continue to fail the children who need it most.

We wanted to be part of PSL because, when the history of this brave reform initiative is written, we want the world to know we did our best to make it a success.

If you are interested in learning more about Rising Academy Partnership Schools, tweet me @pjskids or send me an email.

Evaluation baseline report available

Rising Academies has commissioned a team from Oxford University to complete an independent impact evaluation of our work in Sierra Leone. The study will track the progress of a sample of Rising Academy students over three academic years, and benchmark this against the progress of a comparison group of students from government and private schools in Sierra Leone. The main outcome measures are tests of reading and maths. The team is also looking at writing skills, as well as academic self-confidence and learning dispositions.

The team conducted baseline assessments for English and Maths earlier this year. You can read the full baseline report here or the executive summary here.

The first follow-up assessments were completed in July, and so an annual progress report will be available later this year.

RAN joins CEI database

We're pleased to announce that the Rising Academy Network is the latest organization to be profiled in the Center for Education Innovations online database. CEI "seeks to fill the gaps in global understanding about innovative education programs striving to increase access to quality education for students in low income communities." CEI's database features more than 650 innovative education programs across 145 countries, and attracts over 10,000 visitors a month.

You can find RAN's profile here.