Children who met their math minutes gained more. What does that prove?

What the i-Ready brief shows

In June 2021 Curriculum Associates, the company that sells i-Ready, an online math and reading program, pub­lished a research brief on time and learning.¹ It sorted 1,167,430 stu­dents, from kinder­garten to eighth grade, by their use of i-Ready math in the 2017–2018 school year. It made four groups and ana­lyzed three, because the smallest held only 106 chil­dren. Children in the group the company recommends aver­aged 30 to 49 minutes a week and passed at least 70% of their lesson quizzes. A low group aver­aged 10 to 29 minutes and passed fewer than 70%. A high group aver­aged 50 minutes or more and passed at least 70%. The groups leave no place for a child who logged many minutes and failed often, or the reverse.

The measure was the gain from fall to spring on the i-Ready Diagnostic, the company’s own test. Third graders in the low group gained about 25 points on its scale, and those in the recommended group about 30. The brief sizes that gap at 0.22 of the usual spread among stu­dents, the unit the sec­tions below use. The gap was about five to seven points in every grade.

The brief presents this as evi­dence for using the program as recommended.¹ Its lim­i­ta­tions section adds that the study “is obser­va­tional in nature” and that its infer­ences “are cor­re­la­tional and not causal”.¹ A school that reads the brief as a reason to require minutes takes a step that section declines.

Two fea­tures of the com­par­ison show why the step is unsafe. The groups differ in two ways at once, since the recommended group logged more minutes and also passed more quizzes. A child who passes most of her quizzes is already learning from the lessons. So the groups were sorted partly by the thing the study set out to measure. The low group is also small. It held about 59,000 chil­dren, one in twenty of those studied, who both logged few minutes and passed fewer than 70% of their quizzes. So the com­par­ison sets ordi­nary users against chil­dren already in trouble, and the gap mea­sures that dif­fer­ence as much as the minutes.

A sugar pill shows the trap

The heart trial held a control that schools seldom have. The Coronary Drug Project assigned men who had sur­vived a heart attack, at random, to a choles­terol drug called clofi­brate or to a placebo, an inert pill. After five years, 20.0% of the 1,103 men on the drug had died and 20.9% of the 2,789 men on placebo.² Chance could explain a dif­fer­ence that small, so the drug had done little or nothing.

Then the researchers split the drug group by how faith­fully the men had taken it. Among those who took at least 80% of their pre­scribed pills, 15.0% died. Among the rest, 24.6% died.² Read alone, that gap says the drug saves lives in the men who take it.

The placebo group, split the same way, showed a wider gap, 13 points against 10. Of the men who took at least 80% of their inert pills, 15.1% died, against 28.3% of the rest.² No one con­cluded that sugar pills prevent death. The men who took their pills faith­fully dif­fered from the men who did not, in their health, their habits and their cir­cum­stances. Those dif­fer­ences, and nothing in the pill, account for the gap. Assignment by lot had made the two arms of the trial alike. Splitting an arm after­ward by what the men did sorted them again, this time by every­thing that makes a man take his pills.

The trial holds a second lesson, for anyone told that a report con­trolled for prior scores. The 1980 researchers adjusted for 40 of the men’s char­ac­ter­is­tics at the start of the trial, and a gap of about 10 points remained.³ Decades later, two epi­demi­ol­o­gists rean­a­lyzed the placebo group with methods that also account for what hap­pened to each man during the five years. The gap shrank to between −0.7 and 4.5 points, depending on the method, and each esti­mate was uncer­tain enough to include zero.⁴ A man who grows sicker may stop his pills, and he is also more likely to die. A snap­shot taken at the start cannot hold that.

The i-Ready anal­ysis adjusted for each child’s fall scores, and the brief describes no other adjust­ment.¹ A child’s fall score is the same kind of snap­shot. What follows it, a long illness, a new teacher, a concept that finally lands, moves her minutes and her spring score together.

The same test, run on a math program

A school has no placebo software, but it has some­thing close. A math program should do little for reading. So if chil­dren who log more math minutes also gain more in reading, the minutes prob­ably did not make that gain.

Researchers at Khan Academy ran this test on their own product. They studied 181,359 stu­dents in grades 3 to 8, in 649 schools that had adopted a math tool called MAP Accelerator, in the 2020–2021 school year.⁵ Teachers were encour­aged to give each child at least 30 minutes a week, and 5% of the chil­dren reached that. Math growth rose with each level of use. Then the authors looked at reading and found “a similar pattern” there. They called it “prob­lem­atic as it sug­gests the exis­tence of unob­served con­founding in the usage groups”.⁵

They sub­tracted the reading pattern from the math one and reported what remained. Growth was higher at 30 minutes or more than at under 15, by about a quarter of the usual spread among stu­dents. That spread is the unit of edu­ca­tion research, the stan­dard devi­a­tion, and the figure was 0.26. The sub­trac­tion removes what­ever lifts both sub­jects alike, such as atten­dance or a parent who checks home­work. It cannot remove what lifts math alone, such as a strong math teacher. And if the math tool did help reading a little, the sub­trac­tion takes away too much.

The same paper mea­sured where minutes come from. Teacher, school and dis­trict together explained 57% of the dif­fer­ences in how many minutes a child logged.⁵ So a child’s minutes are in large part a fact about her class­room. A teacher who gets every child logged in each week has prob­ably orga­nized other things as well, and her students’ scores carry all of it.

The bias need not flatter the program. A teacher may send the chil­dren who are falling behind to the software for extra time, and a child who under­stands quickly may finish in fewer minutes. Either habit would make heavy users look worse than the program made them.

Three questions hide in “it works”

A school that asks whether a program works is asking one of three ques­tions, and each needs its own evi­dence. The adop­tion ques­tion is whether taking up the program raises scores. A trial that assigns some schools the program and leaves others without it answers that. The amount ques­tion is whether more minutes raise scores further. Only a trial that assigns dif­ferent amounts answers that. The usage ques­tion is whether the chil­dren who use the program more score higher. A usage report answers that one and no other.

A minutes target is a claim about the amount ques­tion. Evidence on the adop­tion ques­tion, however strong, is silent on it.

A lottery for the program is not a lottery for the minutes

Zearn Math shows how far apart the adop­tion and usage ques­tions can sit. According to the RAND report described below, Zearn recommends 90 minutes and three lessons a week. The same report describes Zearn’s own studies. They matched stu­dents who met usage goals with similar stu­dents who did not, and found the first group ahead by 0.27 to 0.40 stan­dard devi­a­tions on three tests.⁶

Across 1,942 effects from 747 randomized trials of school pro­grams, the median was 0.10 stan­dard devi­a­tions. By that measure, 0.27 to 0.40 would be three to four times the usual effect, if the program caused it.⁷

Researchers at RAND designed a trial with Zearn and ran it by lottery on a federal grant. They assigned 64 schools in one Texas dis­trict at random, half to use Zearn and half to carry on as before, other software included, and fol­lowed about 10,600 stu­dents for two years. On the state test the Zearn schools fin­ished 0.07 stan­dard devi­a­tions ahead, with an interval from −0.04 to 0.19 that includes zero.⁶ The authors wrote that the earlier dif­fer­ences were likely “upwardly biased,” because “stu­dents who use Zearn Math more may be more motivated”.⁶

Yet the trial does not refute the usage studies, and the reason matters for targets. In its first year 18% of the Zearn stu­dents com­pleted the recommended lessons, and in its second year 50% did.⁶ Suppose, as an example, that Zearn raised those stu­dents by 0.33, the middle of the usage studies’ range, and the rest by nothing. The average over everyone offered it would be 0.06 to 0.17, which sits inside the trial’s interval. So does zero. The lottery assigned the program, and the chil­dren still chose their minutes. So the trial answers the adop­tion ques­tion and leaves the amount question open.

One trial assigned the amount

One large trial assigned two amounts of the same software. In Russia, researchers took one third-grade class in each of 343 schools and assigned the classes by lot to three groups for five months.⁸ One group kept its ordi­nary home­work. A second had some or all of that home­work replaced by 10 software exer­cises a week in math and 10 in lan­guage, about 20 to 25 minutes for each subject. A third got 20 exer­cises a week in each, about 40 to 50 minutes.

Both software groups gained in math. Against the classes with ordi­nary home­work, the classes on 10 exer­cises a week scored 0.099 stan­dard devi­a­tions higher and the classes on 20 exer­cises a week 0.087.⁸

The second 10 exer­cises bought nothing that the trial could detect. The dif­fer­ence between the two software groups was −0.012 stan­dard devi­a­tions, and its 95% interval runs from about −0.09 to 0.06. The gain on 20 exer­cises lies below twice the gain on 10, which would be about 0.20. But twice the first gain is itself an esti­mate. The paper’s own test of pro­por­tional dou­bling came to −0.111 with a stan­dard error of 0.067, and that cannot rule out pro­por­tional gains at the usual 95% standard.⁸

The result has other limits. The trial assigned exer­cises, and the minutes are esti­mates, because the company could not supply com­plete usage records.⁸ Its authors thank Yandex for data and support. And these were third graders doing home­work in two Russian prov­inces, so the result is no ruling on an American classroom.

A search on November 11 and 12, 2026, found no randomized trial in American schools that assigned dif­ferent weekly minutes of one software program. One small trial of a school’s tutoring varied the ses­sions a week together with the size of the group.⁹

The minutes come from somewhere

If the second 20 minutes bought little that a trial could see, what they cost matters more. Thirty minutes a week over a 36-week school year is 18 hours, and the hours come out of some­thing. Software that replaces a work­sheet is one trade, and software that replaces a teacher’s small group is another.

A trial in rural China made the trade explicit. About 4,000 boarding stu­dents in grades 4 to 6 took part, and their classes were assigned by lot to three groups. One had a weekly 40-minute session of math software, one had the same session in a paper work­book, and one had none.¹⁰ On the math test, the software group stood 0.033 stan­dard devi­a­tions above the group with no session and 0.059 above the work­book group. Neither dif­fer­ence could be told from zero.¹⁰

The work­book group scored about the same as the software group. I make print books, so I should not turn that into a win for paper. The trial did not show that paper was better, and failing to find a dif­fer­ence does not show two things equal. Its value is in how it asks. Hold the content and the time fixed, change one thing, and see what moves.

The case for the target

The strongest case against me has three parts, and I accept each of them as far as it goes.

Software has raised scores in trials that assigned it by lot. In Delhi a lottery offered chil­dren places in an after-school program that paired adap­tive software with an instructor. After four and a half months the winners scored 0.37 stan­dard devi­a­tions higher in math on tests the researchers designed.¹¹ A trial in Texas assigned teachers by lot to weekly coaching in how to run prac­tice on Khan Academy. Across grades 3 to 8 together the effect was not sig­nif­i­cant. In grades 3 to 6, though, their stu­dents scored 0.12 to 0.17 stan­dard devi­a­tions higher on the state test.¹² So pushing prac­tice up can raise scores. What the trial assigned was a coach for the teacher, so it answers the adop­tion ques­tion for coaching.

Usage reports describe, and the two described here state their own limits in plain words. A dash­board also tells a school what it could not oth­er­wise see, which is whether the program it bought ever reached the children.

And schools need targets, because a teacher cannot run a trial every Monday and needs a number to plan a week around.

My objec­tion begins where the number changes jobs. As a check that the schedule ran, a minutes target is sound. As evi­dence that the minutes taught, it is unproven, and as the goal itself it invites adults to extend a session to turn the dash­board green. A child who has mas­tered the lesson, or who is stuck and guessing, logs the same minute. That last worry is a guess about incen­tives, and no mea­sure­ment of how often it happens was found for this essay. I would keep the target as a floor that shows the program reached the child, and stop counting minutes beyond it as progress.

What I would ask before setting one

The label “evi­dence-based” will not settle it. Federal law counts a sig­nif­i­cant result from a “cor­re­la­tional study with sta­tis­tical con­trols for selec­tion bias” as evi­dence, the third of its research tiers, and its school-improve­ment funds accept that tier.¹³ So a usage com­par­ison can law­fully carry the label.

I would ask four things of any study behind a target. Was the program assigned, or the minutes, or neither? What did the com­par­ison chil­dren do with the same time? Whose test mea­sured the gain? Who ran the study, and who paid?

A dis­trict can also test a schedule, within limits. It can assign two rea­son­able schedules to class­rooms by lot for a term. Before it starts, it names a test the vendor did not write and the smallest gain worth the time. Gains of a tenth of a stan­dard devi­a­tion are the size to expect, and the Russian trial needed 343 classes to measure one with a stan­dard error of 0.039. So I reason that one district’s term will mostly show whether a schedule is work­able. Districts that share one design and pool their results could show which schedule teaches more. The cost is the lottery, a second test and someone to analyze it, and the risk is that some classes get less of some­thing that helps. The step can be undone, since a schedule can change next term.

Check

The strongest evi­dence against me comes from the same Delhi program. There the researchers made two esti­mates of what a day of atten­dance was worth. One came from the lottery. The other came from com­paring chil­dren who attended more with chil­dren who attended less, adjusted for earlier scores. They could not tell the two apart.¹¹ In that program the usage com­par­ison I dis­trust gave about the right answer, and the authors knew it only because they had a lottery to check it against. So usage com­par­isons can be right. They usually need more than a fall score, and the i-Ready brief describes adjusting for that alone.

The essay leaves things out. It weighs test scores and ignores what software may give a teacher, such as time with a small group while the others prac­tice. It ignores cost. The search for trials that assigned the amount was a search, not a sys­tem­atic review, and its main result rests on one country and one program.

Any school can run a check that could fail. Ask whoever pro­duced the usage report to rerun it twice for the same two groups of chil­dren. The first run takes reading as the outcome, and the second takes the pre­vious year’s growth, before the program began. If the chil­dren who met the target were already growing faster the year before, or gained as much in reading, the report mea­sured the chil­dren. If both gaps are absent, one expla­na­tion is gone and my doubt about that report should shrink.

I am con­fi­dent that such a com­par­ison cannot by itself show what requiring the target would do. It can be right, as in Delhi, but a school cannot know it is right without an assigned com­par­ison to check it. I am much less sure about the right number of minutes for any program, and I do not think the evi­dence yet supports one.

A trial would change my mind about a given target. Assign two schedules of one program by lot, in schools like the ones deciding, and measure on an independent test. If the longer schedule wins by a margin worth the hours, I would support it there. And if lot­teries and usage com­par­isons of the same pro­grams agreed as often as they did in Delhi, I would trust usage reports much further.

Sources

1. Curriculum Associates Research, June 2021. Impact of i-Ready Personalized Instruction Time and Lesson Pass Rates on Student Learning Gains, Research Report 2021-08. Written by the program’s publisher; the outcome is the publisher’s own test. Summary page; PDF ↑
2. Coronary Drug Project Research Group 1980, New England Journal of Medicine 303:1038–1041. Influence of adherence to treatment and response of cholesterol on mortality in the Coronary Drug Project. PubMed; doi:10.1056/NEJM198010303031804 ↑
3. Murray & Hernán 2016, Clinical Trials 13(4):372–378. Adherence adjustment in the Coronary Drug Project: a call for better per-protocol effect estimates in randomized trials. Reports the 1980 adjustment for 40 baseline characteristics. doi:10.1177/1740774516634335; PMC full text ↑
4. Murray & Hernán 2018, Trials 19:158. Improved adherence adjustment in the Coronary Drug Project: the placebo gap after accounting for events during follow-up. PMC full text ↑
5. Grimaldi, Weatherholtz & Hill 2022, Proceedings of the 15th International Conference on Educational Data Mining. Estimating the causal effects of Khan Academy MAP Accelerator across demographic subgroups. The authors work at Khan Academy; observational. ERIC full text ↑
6. Pane, Doss, Todd & Seaman, RAND, June 2025. Efficacy of Zearn Math over two years in grades 3 to 5: an experiment in Texas. EdWorkingPaper 25-1211, funded by the Institute of Education Sciences; it also reports Zearn’s recommendation and Zearn’s own matched studies. PDF ↑
7. Kraft 2020, Educational Researcher 49(4):241–253. Interpreting effect sizes of education interventions: benchmarks from 1,942 effects in 747 randomized trials. doi:10.3102/0013189X20912798; working-paper version, EdWorkingPaper 19-10 ↑
8. Bettinger, Fairlie, Kapuza, Kardanova, Loyalka & Zakharov 2023, Journal of Policy Analysis and Management 42(2):552–570. Diminishing marginal returns to computer-assisted learning: workloads and usage records, pp. 554–555; Table 4, p. 561. Author-hosted PDF; doi:10.1002/pam.22442 ↑
9. Shah 2026, EdWorkingPaper 26-1379. The trade-off between quality and quantity: evidence from a field experiment on tutoring. One Indiana charter middle school, 343 students assigned by lot; tutoring twice a week in groups of two or three times a week in groups of three. edworkingpapers.com; doi:10.26300/ma7z-9x69 ↑
10. Ma, Fairlie, Loyalka & Rozelle 2024, Economic Development and Cultural Change 72(4):1923–1962. Isolating the “tech” from EdTech: experimental evidence on computer-assisted learning in China. The figures here are from the working paper, Table 1. NBER working paper 26953; doi:10.1086/726064 ↑
11. Muralidharan, Singh & Ganimian 2019, American Economic Review 109(4):1426–1460. Disrupting education? Experimental evidence on technology-aided instruction in India: the lottery result, and the attendance estimates with their assumptions. doi:10.1257/aer.20171112; author-hosted PDF ↑
12. Oreopoulos, Gibbs, Jensen & Price 2026, Journal of Human Resources, advance publication. Teaching teachers to use computer assisted learning effectively: experimental and quasi-experimental evidence. doi:10.3368/jhr.1224-13977R2; NBER working paper 32388 ↑
13. 20 U.S.C. § 7801(21), the definition of “evidence-based,” with its stricter form for activities funded under section 6303. law.cornell.edu ↑