'We’ve been criticised yet everybody uses our data': Director of influential Indian education survey

On January 16, the latest edition of the Annual State of Education Report, or Aser, was released. Few education initiatives by non-government organisations can boast of the kind of influence this report has had.

Conducted since 2005 by a centre that grew out of the educational non-profit Pratham, this household survey of learning levels is most familiar as the sobering annual reminder that many children in India, despite being enrolled in school, are not learning enough.

It tests their basic reading and arithmetic skills with a standard assessment tool. The Aser website explains that it consists of four levels – letters, words and a short paragraph of Class 1-level and a longer story, which is a Class 2-level text. “The child is marked at the highest level which she can do comfortably,” the website says.

Reports from this annual stock-taking exercise place startling findings in the public domain. Till 2016, the survey covered children between the ages of five and 16 with a special focus on the six to 14 age group. From the first survey in 2006, the public learnt that about half the surveyed children in Class 5 were unable to read a Class 2-level text. The survey in 2016, covering over five lakh children, stated that 27% children in Class 8 were unable to read it.

The findings of this year’s survey – titled Beyond Basics – are similarly grim. This survey sought to examine how teenagers between the ages of 14 and 18 are faring. It found over 85% of the 30,000 teenagers surveyed were enrolled in the formal education system and only 40.3% of them could calculate the price of a shirt being sold at a 10% discount. Only 42% could identify their own state on a map and less than 60% could read the time from an analogue clock. In addition to their performance on the basic Aser tool, Beyond Basics also reported on their activities, aspirations and general awareness.

The Aser reports are perhaps the most frequently-cited of all education surveys in India. Its findings are invoked to lobby for or resist a variety of policy decisions. They help buttress arguments in favour of removing the no-detention policy from the Right to Education Act, 2009, which mandated the promotion of all children between Classes one and eight every year regardless of performance. Its data comparing learning outcomes of children studying in government schools with those enrolled in private ones, without considering non-school factors such caste, income or education or parents, are invariably cited to push for deregulation of private schools and even to challenge the Right to Education Act. The Act’s insistence on what it calls “inputs” – that schools have basic infrastructure such as toilets and classrooms and resources like qualified teachers – to be recognised, are seen as a hindrance to private enterprise in education.

But despite their pervasiveness, Aser’s findings – and the methods of arriving at them – have been questioned too. Educationist Krishna Kumar, the former director of the National Council of Educational Research and Training, which advises the government on curriculum, has questioned the policy of having volunteers assess children at home. He wrote in 2015: “The global benchmarks for achievement testing assiduously stick to the school – the place where children are most familiar with formal learning and with testing”.

Aser started as a research exercise within another influential education non-governmental organisation, Pratham, which advises and executes projects on education for several state governments, including Delhi. In 2008, the Aser Centre became an “autonomous measurement arm” of Pratham, as its executive director Wilima Wadhwa described it, with separate staff and funding. Financed by grants, research studies and foundations, the Centre independently works with state governments too.

Wadhwa, an economist and econometrician, has been at the helm of Aser since its inception. She spoke to Scroll.in on the latest survey, the dozen conducted before, their impact and criticisms, and the plans ahead. Edited excerpts follow.

Why start an annual survey and why this design?
In 2005, a bunch of circumstances came together. Pratham had been working with young children for a very long time and we had a fairly good idea that learning levels were low. But nobody at that point was talking about learning levels. All focus, worldwide, was on enrolment or access. That was the focus of the Millennium Development Goals and even of the Right to Education Act, 2009. You cannot do anything without access but even in 2005, about 95% children [aged six to 13] were enrolled. We asked: “We have got our children into school, what is it they are learning?”

The Sarva Shiksha Abhiyan [a government scheme to universalise education from Class 1 to Class 8] had been launched [in 2000-’01]. The government had imposed the 2% cess specially earmarked for education and P Chidambaram [then Union finance minister] talked about shifting focus from outlays to outcomes.

What did we know about assessment then? Even today, most assessments by the government are school-based, pen-and-paper and pegged to the curriculum. This [format] has come from the advanced, western nations where enrollment translates one-to-one into attendance. That is not the case here. If you go to schools in Uttar Pradesh or Bihar on a random day, you may miss out on 50% of the students.

Then there is a variety of schools – government, private, religious schools. We do not have reliable sampling frames available.

Aser was designed for our ground reality. A household-based survey, it could capture all children, even those out of school and those who do not regularly attend. It is not a pen-and-paper test. If children are not fluent readers, what is the point of giving them pen-and-paper tests?

Educationists point out that children do not identify the home as where tests happen and that may have an impact on their performance.
But we do not see this as a test.

Do the children see it as a test?
I do not think so. There is so much engagement with the Aser tool. Our learning from Beyond Basics is that if you show them anything that is school-based, their eyes glaze over.

That it is a household test has been criticised over the years. But do we not care about all our children? We feel children are far more comfortable at home. If you think about it intuitively, you would be far more comfortable at home. So we agree to disagree on this. We do not see any evidence of kids being stressed out.

For 12 years, the results of the annual survey have been practically unchanged. Have you considered adding more nuance to the survey so you can better understand why?
Through the years, we have always added some things. To basic literacy, we added word problems, English, reading comprehension, but all simple.

A frequent criticism has been that Aser does not tell you what more children can do, beyond floor-level tasks. Aser is not built for that, but since 2012, Aser [Centre] has done a variety of studies with older kids and more academic tools pegged to the curriculum. But these are research studies in a controlled environment. It can be difficult to match the Aser architecture where volunteers are involved.

The volunteer-driven assessment has led to questions about how their biases may have influenced the data.
We have been criticised for so many things yet everybody uses our data. Other surveys found the same thing – the National Achievement Survey [conducted by the National Council of Educational Research and Training], in a completely different environment, has found drops in learning levels. Same with the India Human Development Surveys by the National Council of Applied Economic Research, and Young Lives, which is a longitudinal study. Our methodology is very transparent and documented.

We always have a team of two volunteers – one records, the other has a conversation. Since Aser is conducted in 22 languages, we need a local partner – local NGOs, colleges, District Institutes for Education and Training. We do not need a specific kind. One year, our partner for Leh and Kargil [in Jammu and Kashmir] could not do it so the Animal Husbandry Department did. They said: ‘If we can count sheep and cows, we can count children.’ We said: ‘You definitely can. We will train you for it.’

What sort of training do the volunteers get? Is any sensitivity training conducted?
There is not so much sensitivity-training – we tell them about that – but there is very rigorous training about the survey. The three-day workshop includes use of the survey instruments, a practice day on the field, opportunity to offer feedback.

Questions are asked about the nature of training as well.
You should talk to people about the training for government surveys. But our methods are all very well documented.

For the main Aser, we have cascading model of training [that is, some are trained first, and they then train others] because we work with over 500 institutes. With Beyond Basics we had just 35 partners and trained them directly.

Aser reports always seem to present a dismal picture. Whose responsibility do you think it is to fix this?
Everybody’s. The findings [of Beyond Basics] should not be surprising because these youths have come from the same system. If a child is left behind, that child’s problems are not addressed.

We know 50% children in rural India have mothers who have never been to school. And 30% have fathers who have never been to school.

If I am uneducated and I cannot afford a tutor or private school, what does my child get? It is very easy to blame the teacher. A government school teacher has children of all ages, abilities and socio-economic backgrounds in her class. But her job is to finish the curriculum and for that, she is teaching the top of the class. With automatic promotion, you can go from Class 1 to 8, even if you cannot read but how will you do algebra and geometry after? If that deficit is not addressed, then this [findings of Beyond Basics], is what you are going to see. 81% of this sample [of 30,000] had completed eight years of schooling.

What household data do you collect?
As far as possible, we collect information on household assets that you can observe. The type of house – kuccha [a hut], semi-pucca [partially built] or pucca [fully built] – is a very good proxy for affluence in rural areas. We check for an electricity connection and whether there was power at the time, for a toilet, a television and any vehicle. We gather information on parents’ education, if there is any reading material other than religious books, if someone in the household can use a computer even if they do not have one, and if anyone, other than parents, has completed high school.

Do you control for these factors when you compare government and private schools?
We do.

In introductions to Aser reports you acknowledge that the gaps in learning between government and private school students narrow significantly when you factor in family background but the data is presented without this qualification…
How many things are you going to do? To do that [present data after factoring background in] you have to run regressions. I have written extensively about that.

But data presented without that caveat, implying that private schools are more efficient, have many activists believing that Aser pushes for privatisation. How do you respond to this?
I have written about this extensively. But this is the reality. This is the data. And whether you like it or not, that is the perception the parents have and that is why more and more parents are going for private schools. But we are not pushing privatisation. The difference narrows and we have talked about that.

Can you not control for all the datasets in which you compare learning achievements of children in government and private schools?
But that is not data, that is a paper. You are between a rock and a hard place. If we give data for all schools combined, private schools complain. If we separate, we are accused of pushing privatisation. But for the data table you cannot control because that would make it an analysis.

Your reports are used to buttress arguments to remove the ban on detention in primary school from the Right to Education Act. Do you have a stand on this issue?
We have never had a stance on it. You know it will have certain consequences and one of them is this [low outcomes]. Am I saying holding back a child is a good thing? No! Of course not. But the survey results are an early warning sign.

Informally, many states already had a no-detention policy up to Class 5. We think in terms of watershed years – end of primary school [Class 5] was one and that is when many dropped out. The jumps in curriculum levels – from Class 5 to 6 and from Class 8 to 9 – are different from the changes from Class 4 to 5 and Class 7 to 8.

We know where they will face most difficulty. Do not hold the child back but there has to be some way of addressing the deficit. Learning levels were low and slow to change from 2005 to 2010, but after 2010, there was a drop. Suddenly the focus shifted to playgrounds and boundary walls. I am not saying those are not important but whatever little attention learning got also went away. The teachers had to deal with elections and Census [enumeration] along with continuous comprehensive evaluation [mandated by the Right to Education Act, it involves constant tracking of a child’s progress in place of a year-end exam] which took years to figure out.

We have also been accused of constantly testing a child leading to stress – but unless you know where the deficits are, how will you address them? The government’s policy-making is like a patchwork – you have a crisis somewhere, you put a patch over it.

Now there are discussions on extending the Right to Education to children below six and in the 14-16 age-group. What do you make of the policies being considered?
Early childhood is certainly important but all pieces have to fit together. It is under a different ministry altogether [the Ministry of Women and Child Development].

With the debate on detention, we have come full circle. First, the government removed exams because they were too stressful and now we are back to exams.

Suddenly the government is pushing vocational education. Just 5% kids in the 14 to 18 age-group [in Beyond Basics] were taking some courses. If you push vocational, it cannot be for those kids, who are not academic or are poor. They are aspirational too. It has to be recognised as a viable alternative to an academic route by everybody. How do kids know if they like it if it is not integrated into schooling from Class 6, 7 or 8?

Are other types of surveys planned? Also, will Beyond Basics be an annual survey?
The 14-18 [age] group is the first cohort of post-RTE [Right to Education] kids who were entitled to eight years of schooling and automatic promotion. Enrolment in Class 8 doubled from 11 million in 2004 to 22 million in 2014. Many children are going beyond Class 8. But if a young person in this age group drops out, they are unlikely to go back to continue. What is going on in these years?

We do not know yet if this will be an annual exercise. We hope to start a conversation. At some point we want to do a survey on early childhood.

What has been Aser’s impact?
For data to be useful and to feed back into policy, it has to be reliable, comparable and available frequently. When we went in, we planned to do it for five years.

In January 2009, at a consultative meeting, academics, government officials and other NGOs unanimously agreed that we should continue. Doing an assessment will not change things. But the objective, and we have been fairly successful, was to bring learning to the centre.