Problem

The World Development Report on “Making Services Work for Poor People" (World Bank, 2004) illustrates well the essential tension in the public conversation about primary education in developing countries. On the one hand the report embraces the broad agreement, now enshrined in the Millennium Development Goals, that primary education should be universal. On the other hand, it describes in detail the dismal quality of the educational services that developing countries offer to the poor.

Migrant children, like this boy, are among the poorest performing students in China.

Migrant communities in China are an excellent example of this paradox. Among the country’s most vulnerable populations, migrant communities in China’s cities often lack access to basic services—including state-funded education—that regular urban dwellers enjoy. Because migrant children often have difficulty enrolling in urban public schools, unregulated and privately run migrant schools have emerged to offer them educational services. Despite the proliferation of these institutions, they still operate on the margins of the education sector, and are plagued by poor teaching, poor facilities, undeveloped curriculum, and high tuition (Ma et al., 2009). For these reasons, educational outcomes for the thousands of children who attend these schools are abysmal. What is more, the problem has the potential to explode. Migrant children account for the fastest growing segment of the education population; indeed, the sizes of the populations of elementary students in urban public schools and elementary students in rural public schools are falling. This trend will only accelerate as migration and urbanization increase in coming years, as is heavily emphasized in China’s twelfth Five Year Plan (Wang, 2010).  

Due at least in part to the poor educational services in migrant communities, China’s migrant children (liudong ertong) have been shown to be the poorest performing group of students in the entire country—poorer than the so-called “left-behind” children (liushou ertong) that stay in rural areas when their parents migrate to cities (Ma et al., 2009). Standardized test scores from a set of REAP benchmarking studies bear this out:

• Urban students in urban schools -- average score: 83.0
• Rural students in rural schools, well-off areas -- average score: 76.0
• Migrant students in urban public schools -- average score: 75.8
• Rural students in rural schools, poor areas -- average score: 70.1
• Migrant students in migrant schools -- average score: 68.9

Considering these numbers, REAP is interested in two principle questions:

1.    Why do migrant children perform so poorly in school?
2.    What can be done to improve educational performance among migrant children?

Fundamental Problems: Lack of Tutoring, Lack of Care

There are many possible reasons for this poor performance: inferior teaching facilities, poor teachers, or the curriculum disruption that occurs when students—or entire schools—move from one area to another. Anecdotal evidence suggests that one particularly serious problem that migrant children face is lack of care outside of the classroom. What happens, for example, to a student who does not understand a lesson? If a child needs extra time to learn a concept or a new idea, how can he or she catch up?

What happens if he falls behind?

Children in migrant schools are at a severe disadvantage because they cannot access resources that can help them if they get behind. Students enrolled in public schools often have before and after class review sessions in school, access to commercial tutoring classes, and interested and well-educated parents to help them with their homework. Migrant students often do not have any of these options. Concerned about liability issues and cost cutting, principals of migrant schools almost always open their gates at 8 am—immediately before school—and close them at 3:30 pm—immediately after school. Even if the school grounds were open, teachers—who are poorly paid and often live far away from school—have little incentive to provide review classes. Because migrant families are poor, they have trouble financing commercial tutoring classes. What is more, migrant parents are often busy working and have little time to devote to helping their children with homework and class projects; even if they were not too busy, the parents of migrant children are typically poorly educated themselves and are in little position to offer help at night to their children.

Computer Assisted Learning: A Tutoring Alternative?

What are some alternatives for providing the educational support that migrant students lack? Scrutiny of the international literature reveals that little is known about how to improve the quality of education for vulnerable children in developing countries in a cost-effective way. Worse still, a number of rigorous evaluations have demonstrated that spending more on inputs such as textbooks (Glewwe et al., 2002), flip charts (Glewwe et al., 2004) or additional teachers (Banerjee, et al., 2004) has little effect on children’s test scores (see Kremer, 2003, for discussions and more references).

These results have led to a general skepticism about the ability of interventions focusing on inputs to make a difference and have led many to advocate more systemic reforms designed to change the incentives faced by teachers, parents and children. The World Development Report, once again, embraces this view, and proceeds to propose various ways to improve incentives (most of which have either not been rigorously tested or when tested, have also proven rather disappointing).

We believe, however, that to give up on inputs is premature. One promising alternative to extracurricular tutoring and care is Computer Assisted Learning (CAL). If integrated CAL programs in schools are proven to effectively raise student performance inexpensively, we believe they be readily employed across wide areas of the nation’s school system.

Computer Assissted Learning may be a solution to the lack of remedial care many migrant students face.

What is the basis for our optimism? In the early 2000s, a large-scale, rigorously implemented experiment in rural India’s schools demonstrated quite convincingly (in our opinion) that educational game-based computer assisted learning has the potential to drastically improve basic competencies among poor and underperforming student populations. The 2005 study found that students who participated in the CAL intervention improved their standardized math scores by 0.35 standard deviations the first year, and 0.47 the second year across performance levels. These results are outstanding, and far better than have been attributed to costly and complicated interventions such as reducing class size. In layman terms, the results from the India study show that after participating in the CAL program, a C+ student became a B student (while other C+ students who were not in the program made no improvement). What is more encouraging is that the study showed these results persist over time—especially in student math classes. According to the study, students of all levels performed better in math if they were in schools where the math CAL program was implemented. These results suggest that it is possible to substantially improve the quality of education in developing countries with relatively inexpensive interventions, and do so in a way that makes a long-term difference in the lives of young students.

An Opportunity in China?

Given this evidence, access to computers with innovative educational software may offer a viable alternative to the more conventional extracurricular care that so many migrant students lack. Providing such access could also be a way to help other groups of vulnerable and poorly performing students in remote rural communities. Unfortunately, China’s migrant students lack regular access to almost all forms of computing technology. Only 1 out of 20 migrant schools have functioning computing facilities. What is more, because the families of migrant students often live in poor, suburban villages (in houses of Beijing residents that rent their spaces out, often as they await the future demolition of the building), less than 10 percent of the children of migrants have access to computers in or near their homes. The promise of CAL to inexpensively deliver improved educational outcomes among migrant children is real, however, and deserves a proper evaluation. If found to be effective, a CAL program in China’s school’s could serve as a foundation for a new educational platform for helping children in other underprivileged areas of the country and the world.

Goals

REAP aims to test the potential for CAL programs to improve educational outcomes among migrant students in China. The proposed intervention is based on the premises that well-designed educational games can sustain interest and curiosity in an otherwise unsupportive school environment, and good educational software can be reproduced at nominal cost.

The experiment will achieve objectives in three areas:

Intellectual objective:

• Provide clear, quantitative evidence about the linkage between CAL programs and the educational performance of underserved children.

Policy objective:

• Demonstrate that computer assisted learning programs lead to better educational outcomes and that such programs should be scaled up in areas where poor student performance is chronic.

Educational objectives:

• Develop a curriculum (and training manual for teaching the curriculum) for a computer assisted learning program for China’s disadvantaged youth.
• Build an institutional channel for providing opportunities to students and recent graduates (from Stanford/other universities) to go to China during the summers and become involved in CAL programs. The students and recent graduates can serve are trainers, assessors, technical support, observers and data analysts.

Approach

Nap time at a Beijing area migrant school

REAP will test the potential for CAL to improve educational outcomes among migrant children by conducting a randomized control trial among fifth grade students in Beijing area migrant schools. Beginning in the first half of 2010, REAP will randomly select 100 such schools to participate in a CAL intervention. 50 of these schools will serve as treatment group and 50 will serve as a control group. Also during this period, After developing and testing a new educational games-based curriculum, in September 2010 the treatment schools will receive four high quality computers each.  These computers will be “lent” to the school by REAP. Fifth grade children in each treatment school will be offered two hours of shared computer time per week. During this time they will play educational computer games that involve solving math problems and performing English language exercises at varying levels of difficulty. The games will be designed to emphasize basic competencies in these areas per the local curriculum. The student pairs will play the games under the supervision of a teacher for two hours per week outside of the school’s regularly scheduled classes (e.g., during the homeroom, study period; and during the lunch break). REAP will train the fifth grade schoolteachers in the migrant schools as the program’s manager. They will be responsible (with the project team’s aid) for scheduling students and monitoring their progress and keeping the computers running (by being in touch with an on-call repair team). Children will also complete simple worksheets designed to track their progress at the beginning of each session.

During the first segment of the project, REAP will design the educational software program to allow the children to learn as independently as possible. The teacher-supervisors will encourage each child to play games during their assigned time slots (e.g., 2 hours per week) and when necessary, help individual children understand the tasks required of them by each game. All interaction between the students and instructors are to be driven by the child’s use of the various games, and at no time will any of the instructors provide general instruction in mathematics or other subject areas. Evaluation and monitoring will be done at regular intervals and overseen by the REAP team.

Teaching and Learning Materials

Drawing from the REAP’s experience in the field and the expertise of both the Stanford Design School K-12 Lab and the Stanford Program for International Cross-cultural Education (SPICE), REAP will either develop or identify educational software appropriate for China’s state mandated curriculum for grade five. The software is completely interactive and game-based, and will aim at enhancing the child’s basic competencies, suc

A screenshot from a math education software

h as knowledge of numbers, number operations, geometry, recognition of vowels and consonants, word formation, English vocabulary, pronunciation, recognition of measurements, recognition of direction and other general knowledge.

The software will include animation, pictures, sounds and language that are relevant and contextual to China’s schooling environment, and so facilitate recognition and grasping of concepts. Different games and levels will be made compatible for students with no previous exposure to computers as well as for those with more know-how. The software will be designed to gradually build him/her to higher levels of competencies, by introducing the usage of multiple keys, increasing speed, more complex operations, etc. In addition, the software will allow flexibility of use by giving the instructors the freedom to focus on competency areas that they want to build or reinforce.

In July and August 2010, prior to the intervention (which will begin in September 2010), REAP will conduct extensive beta testing of the software among migrant children around Beijing. This testing will ensure that the software is properly adapted to student needs and competency levels. During the intervention itself, REAP will also arrange for a maintenance hotline to provide on-the-spot technical assistance for instructors should the need arise. 

Teacher Selection and Training

The teacher-coordinators will be the fifth grade teachers in each of the program’s 50 treatment schools. The treatment schools will be selected randomly (as described above)

Teachers receiving training for CAL

from a list of 100 migrant schools that themselves were selected randomly from a comprehensive list of Beijing migrant schools that REAP maintains as part of its migrant school data base.

During the first two weekends of September (shortly before the start of the autumn semester of the 2010-11 academic year), all teacher-coordinators will attend two weekends of rigorous training. During this time, participating teachers will be trained in three principle areas: a) the curriculum requirements for fifth grade; b) how to use/play REAP’s educational games; c) various aspects of basic computer literacy; and d) how to properly monitor the intervention on a day to day basis. Teachers will be paid, their transportation will be covered and they will be housed in a training center in western Beijing (a training center and guesthouse that is owned and operated by the Beijing Municipal government and is open to the public). The REAP project team and a Beijing Normal University curriculum consultant will conduct the training. 

Program Monitoring

After the program is initiated (that is, after the teachers have been trained, the computers set up and student assignments/schedules have been made) Treatment school instructors will collect the data during the intervention to monitor: a.) compliance

Students during a CAL session

(Are students getting two hours per week? Are they going through their assigned lessons/games?.); b.) Are their scores—as measured by weekly mini-tests—improving; stagnating or improving over time?). Reports based on the data will be prepared monthly by the REAP project teams. We also will keep a log of any computer hardware and software problems that are encountered.

Control schools

Fifty of the 100 participating schools will comprise the control group for this experiment. The methodology of randomization (with prebalancing) ensures that these schools will be statistically identical to the 50 treatment schools. No aspect of the intervention will be implemented in the control schools other than baseline and final testing. We will make several visits to the control schools in order to try to minimize any possibility of a Hawthorne effect. In this way, changes in the standardized test scores of students in the control schools will serve as a basis of comparison those of the students in the treatment schools.

Evaluation

The students in both intervention and control schools will be tested once at the beginning of the academic semester (before any intervention has begun in early September 2010) and again at the end (in early January 2011). Both tests will be identical in nature and will measure basic mathematics and English language competencies. The surveys will also collect information on family and student characteristics, attendance rates, subjective enthusiasm/student interest in school, etc.

Results

Academic performance

Our results show that CAL has significant beneficial effects on student academic outcomes. Two 40-minute CAL math sessions per week for thirteen weeks increased the student standardized math scores by 0.11 standard deviations. That is a large effect, comparable to—indeed, often exceeding— the effects of much more costly and complicated interventions such as reducing class sizes and providing extracurricular tutoring. Another useful way to think about the effect is that a .1 standard deviation improvement is often thought of as equivalent to roughly one semester’s worth of learning. Thinking about it that way, thirteen weeks of CAL improved test scores as if the intervention students had attended over a semester’s worth of school more than the control students. Remarkable!  

Heterogeneous effects

Interestingly, when we looked at the intervention’s heterogeneous effects (i.e. how CAL affected different types of students differently) we found that the biggest improvements in math scores occurred among the most vulnerable students—the students who performed least well on the baseline test and those with less-educated parents. These are exactly the students for whom CAL was designed to help most: the ones falling behind.  

Self-confidence and self-efficacy

REAP was also careful to measure non-academic outcomes related to the CAL intervention. Before and after the intervention we measured how students liked school, their self-confidence, and their self-efficacy in math—a characteristic that relates to their perceived ability to learn new math skills. In all three cases we found improvements in these variables for the intervention students when compares to the control. CAL actually raised these outcomes! There was also a modest spillover effect in which Chinese test scores improved in the intervention group versus the control, reinforcing the finding that CAL can instill a love of learning that can manifest outside the confines of CAL subject matter.

Project Video

Below is a video of our April pre-pilot in action!