At least since Sputnik in 1957, various tests and reports such as the Trends in International Mathematics and Science Study (TIMSS) have asserted that students in the United States perform surprisingly poorly in math and science compared to students in other nations. The poor state of education is a perennial topic in the United States. Most people would like to see better teaching and education. Particularly with respect to math and science, this goal is often phrased in terms of producing more Einsteins or Edisons, exceptional or expert performers who would presumably, so the theory goes, make more scientific and technological breakthroughs, spurring progress and economic growth. President Obama returned to this time worn political cliche in his State of the Union 2011 “Sputnik moment” speech:

Maintaining our leadership in research and technology is crucial to America’s success. But if we want to win the future — if we want innovation to produce jobs in America and not overseas — then we also have to win the race to educate our kids.

Think about it. Over the next 10 years, nearly half of all new jobs will require education that goes beyond a high school education. And yet, as many as a quarter of our students aren’t even finishing high school. The quality of our math and science education lags behind many other nations. America has fallen to ninth in the proportion of young people with a college degree. And so the question is whether all of us — as citizens, and as parents — are willing to do what’s necessary to give every child a chance to succeed.

This article is a follow up to Debating Deliberate Practice about Malcolm Gladwell‘s bestseller Outliers and deliberate practice, a theory of expert performance developed by psychologist K. Anders Ericsson. Outliers has a great deal to say about mathematics and mathematics education in the chapters titled “Rice Paddies and Math Tests” and “Marita’s Bargain.” Gladwell argues for a reduction or elimination of summer vacation and more work and practice as a solution to the perpetual purported problems with math and general education in the United States. In “Marita’s Bargain” he showcases the KIPP (“Knowledge Is Power Program“) Academy in the South Bronx in New York City as an exemplar for a new educational system. Most people would like to see better teaching and education. Many people would like to know whether deliberate practice is the proper way to achieve expert performance. This article takes a closer look at KIPP and the controversies surrounding it with a special focus on mathematics and also the use of mathematics in public policy debates.

What is KIPP?

The Knowledge Is Power Program, often referred to by the acronym KIPP, is a national non-profit network of charter schools serving disadvantaged inner city neighborhoods such as the South Bronx. The vast majority of students at KIPP schools, most of which are middle schools (5th through 8th grades), are black and Hispanic children from poor families. KIPP seemingly has had excellent results based on standardized tests of reading and math skills. In addition to its appearance in Outliers, KIPP has been the subject of a laudatory book Work Hard, Be Nice by Washington Post education writer Jay Mathews. KIPP is also the subject of intense political controversy.

Charter schools are a special kind of public school designed, in principle at least, to avoid the bureaucracy and other limitations of traditional public schools. Charter schools can be organized by various groups, which include for-profit school companies, which apply for and receive a charter from the state along with funding for each student. The charter schools frequently receive additional funding from donations and other sources in addition to the public funding. The large majority of charter schools are organized and operate in poor, disadvantaged, “inner city” areas like the KIPP charter schools. Most students at charter schools are black and Hispanic like the KIPP charter schools. In many cases the teachers at charter schools do not belong to the powerful teachers unions: the American Federation of Teachers (AFT) and its affiliates or the National Education Association (NEA). A substantial number of charter schools are run by for-profit school companies, known as Education Management Organizations (EMOs). There are suspicions that KIPP is a stealth school company that will convert from its current non-profit status to a for-profit school company to be listed on the stock market when the time is right.

The education market in the United States is about $500 billion per year. The US spends about $10,000 per year per student on about 50 million students. The “School Enrollment in the United States — Social and Economic Characteristics of Students” report from the US Census Bureau (dated October 1999) gives a total of 49,339,000 students in public school in the United States (Table A, page3). A total of 8,556,000 public school students are identified as “Black” and 8,081,000 as “Hispanic (of any race)”. The National Center for Education Statistics (NCES) reports that the US spent $9,683 per pupil in the 2006-2007 school year ($10,041 in inflation adjusted 2007-2008 dollars according to the report). The United States probably spends around $160 billion on public education for black and Hispanic students. Most of this money is channeled through the traditional public school system run and administered by school teachers and administrators who are essentially civil servants with a high degree of job security (a tenure system) and most of whom belong to a teachers union.

Many conservative, libertarian, and business groups have embraced charter schools as a vehicle for school reform which often appears synonymous with bringing the “miracle of the market” to education and eliminating the unions. As one might imagine, the education unions and their allies are not happy about this and generally have a dim view of the charter school movement. Charter schools have gained significant support in both the Republican and Democratic parties, and in liberal as well as conservative circles. The Obama Administration has made expanding support for charter schools a part of its education program. Charter schools have a mixed performance record. Some are better than the traditional public schools in their district, some are about the same, and some are worse as measured by standardized test results. The KIPP schools, as well as a few other charter schools, however, appear to have a consistently good record compared to comparable traditional public schools, making them the flagship for proponents of charter schools and associated reforms, and therein lies the rub.

The first KIPP school was founded in inner city Houston, Texas in 1994 by David Levin and Mike Feinberg, recent alumni of Teach for America (TFA), which is closely associated with KIPP. Teach for America was founded by Wendy Kopp based on her 1989 Princeton senior thesis proposing a sort of Peace Corps in which recent Ivy League graduates would become teachers in disadvantaged inner city schools. Wendy Kopp and Malcolm Gladwell have appeared together to discuss education reform at the New York Public Library. Wendy Kopp worked closely with Richard Barth who eventually became her husband. Richard Barth left TFA to become an executive at Edison Schools, the controversial for-profit school company founded by Chris Whittle. After Edison, Richard Barth became the CEO and President of the KIPP Foundation. KIPP has grown substantially since 1994. KIPP acquired the substantial backing of the late Donald Fisher, a co-founder of The Gap clothing stores. Fisher was also an investor in Edison Schools. His son John Fisher is the current Chairman of the Board of Directors of the KIPP Foundation.

At one point, Edison Schools was the darling of conservative, libertarian, and business school-choice/school privatization advocates, arguing that a private for-profit company listed on the NASDAQ could run schools more cost-effectively and with better results than the inefficient, bureaucratic traditional public schools dominated by the teachers unions. Edison has had many problems. For example, it apparently cost more for Edison to run a school than a traditional public school. Although Edison still exists as Edison Learning, its star has dimmed and similar hopes appear to be now pinned on KIPP and other charter school operators.

KIPP’s Numbers

KIPP posts an annual “report card” on its web site for each KIPP school as well as other information on the KIPP program. There are also a number of reports, both independent and funded by KIPP, on the performance of KIPP. Figure 1 shows the relative test scores for the KIPP Academy New York, the school showcased in Outliers.

The “State Criterion-Referenced Test” seems clear to interpret. The “Norm-Referenced Test: Measures of Academic Progress (MAP)” is not shown (cut off). Although also positive, the author found the explanation of what was being measured incomprehensible.

The “report card” gives the following general figures about the KIPP Academy New York which may be helpful in setting the context.

KIPP ACADEMY NEW YORK

250 East 156th Street, 4th Floor
Bronx-New York, NY 10451
718-665-3555 | www.kippnyc.org

School leader: Blanca Ruiz
Year founded: 1995
Grades served: 5-8
Student enrollment: 268
Number of teachers (FTE): 21
Per pupil funding: $13,675
Facility type: Provided by district
Size of school: 25,650 sq. ft.

Note: FTE is an acronym for Full Time Equivalent as in equivalent to a full time employee. For example, two part-time employees could make 1 FTE.

Figure Two shows the student demographics from the “report card”:

There are a number of KIPP schools in New York City. The main KIPP NYC web site provides many overall statistics including some statistics on longer term outcomes (retrieved April 29, 2011):

To and Through College

• Over the last seven years, 93 percent of KIPP NYC alumni graduated from high school. By comparison, 46 percent of African American and Hispanic students across New York City graduated from high school during 2005-08.
• 87 percent of KIPP NYC alumni have matriculated to college. By comparison, 21 percent of African American and Hispanic male high school graduates attend college in New York State; 36 percent of all low income New Yorkers ages 18-24 matriculated to college during 2003-08.
• To date, 31 percent of KIPP NYC alumni have earned bachelors’ degrees within six years of high school graduation. Our objective is to have 75 percent of our students graduate from college. We’re currently on track to hit a 50 percent 6-year graduation rate in the next two years; nationally, the college completion rate in low income communities is 10 percent.

NOTE: The comparisons in the three bullet points differ. The first bullet point compares KIPP NYC alumni favorably to African American and Hispanic students across New York City. The second bullet point compares KIPP NYC alumni favorably to African American and Hispanic male high school graduates in New York State and all low income New Yorkers aged 18-24. The third bullet point compares KIPP NYC alumni favorably to the national college completion rate in low income communities. This sort of slicing and dicing of data should make one uneasy and it is very easy to do today with widely available powerful computers and statistical tools.

Nonetheless, KIPP schools generally report positive results compared to comparable traditional public schools. Has KIPP found the magic formula for better education? Is it deliberate practice as Outliers implies?

Pump Up The Volume

In the United States, there is a genre of movie that features a new teacher, frequently young and idealistic, at a school, frequently in a poor crime-ridden inner-city neighborhood, who, after some problems and setbacks and usually a battle with the heartless or narrow-minded school administration, manages to inspire his or her students, generally a bunch of losers, to perform spectacularly well. Some are fairly serious like Dangerous Minds in which Michelle Pfeiffer stars as a former US Marine who inspires (yes) a bunch of losers. Others are over the top like the Arnold Schwarzenegger film Kindergarten Cop or The Substitute, which stars Tom Berenger as a mysterious special forces soldier who substitutes for his teacher girlfriend, who has barely survived an assassination attempt by a high school gang, in a tough inner city school. The public relations stories of Teach for America and KIPP very much fall within this tradition as Malcolm Gladwell notes in Outliers.

The 1990 movie Pump Up The Volume (not a very good movie) features a star school principal who turns her high school around, achieving winning SAT scores, and the adoration of the clueless parents. The secret to success? The principal illegally engineers the expulsion or removal of the school’s misfits and losers who score badly on the SAT test — using trumped up disciplinary charges and other tactics. This is what is known as sample bias. If you only test the good students, then you will get good scores.

Critics of KIPP and other high performing charter schools frequently allege a Pump Up The Volume scenario, whether intentional or not, to explain the test scores. There are two sources of bias that are frequently suggested. First, students and their parents choose to apply for a position at a KIPP or other charter school. By default, they would simply stay at their traditional public school. So there may be a selection effect in which more motivated or capable students and families apply to the KIPP schools. This does not require any skullduggery on the part of the KIPP schools. Second, students may leave KIPP after being accepted, known euphemistically as attrition or student mobility. The education business has more than a little jargon. It is here that the KIPP schools could consciously, unconsciously, or accidentally push out the “losers”. Critics often cite a study by SRI International (the think tank in Menlo Park, California formerly known as the Stanford Research Institute) funded by the William and Flora Hewlett (as in Hewlett-Packard) Foundation of the KIPP schools in the San Francisco Bay Area that showed a high rate of attrition in these schools.

KIPP has retained Mathematica Policy Research Inc. to conduct a comprehensive longitudinal study of the KIPP program and schools, probably in no small measure to counter the suggestions that KIPP gets rid of the losers to pump up its numbers like the nefarious school principal in Pump Up The Volume. Mathematica has presented a fifty-one page working paper “Student Selection, Attrition, and Replacement in KIPP Middle Schools” at the 2011 Annual Meeting of the American Educational Research Association (AERA). Mathematica has also produced an interim preliminary report which is highlighted on the main KIPP web site. This is a war waged with advanced statistics, regression models, and other sophisticated, ostensibly scientific, objective tools.

Infinite Regress

Modern public policy debates are frequently waged, at least in public, by think tanks such as the libertarian Cato Institute or the progressive Center for Economic and Policy Research (CEPR) (two heavily quoted and cited think tanks in the major media). These think tanks routinely produce scholarly studies, books, reports, and so forth that usually seem quite sophisticated, conform to most of the standards and forms of scholarship or science, and frequently incorporate sophisticated statistical and mathematical analyses. The think tankers usually seem extremely intelligent, highly educated, sincere, passionate about their beliefs and committed to the truth and the betterment of the world. The only problem is that they almost always produce studies and rigorous statistical and scientific analyses that confirm their beliefs. With a little practice, one can usually anticipate where a Cato Insitute or a CEPR will come out on an issue: often in the form of scholarly studies with data, seemingly rigorous analyses, and often sophisticated statistics. It is also extremely rare to find a think tank producing a report or study that concludes “we don’t know what is going on”. Think tanks live in an alternate universe where the think tankers have all the answers even to extremely complex and puzzling scientific and technical problems.

Now, one cannot claim that think tankers never come to contrary conclusions and reverse their positions because they believe that the accumulation of evidence fails to support their original position. For example, in the current controversies over education, Dianne Ravitch, a scholar and former education official in the George H.W. Bush (Bush I) administration, is well known for reversing many of her views. Nonetheless, this is quite rare and it is almost always an isolated individual rather than an institution or organized group. In fact, the apostate will often withdraw from the now offended institutions or groups. It is extremely rare to encounter anything like a retraction or explicit reversal of doctrine from a think tank or similar group. Sometimes these organizations will simply stop talking about something. For example, many liberal and progressive groups simply stopped talking about school busing after the Reagan Administration mostly killed it in the 1980’s.

How does this happen? Aren’t scientific and statistical methods supposed to provide definitive answers in a rigorous objective way? In most of these controversies where advanced statistical methods are used in an attempt to settle public policy issues “scientifically”, the issue of sample bias or some sort of measurement error frequently arises. In practice it is often extremely hard to design an experiment, study, or survey that could not be strongly affected by some sort of bias or measurement error. The commonly used statistical methods have limited ability to unambiguously detect or eliminate bias or measurement error. Thus the typical “scientific” public policy debate goes something like this:

Free Market Think Tank: Our new study of widgets shows conclusively that the free market widget producers in (insert allegedly free-market oriented organization, county, state, or nation here) outproduce the lazy socialist widget producers in (insert allegedly socialist organization, county, state or nation here) by (insert percent) percentage points. This is statistically significant at a (insert confidence level here) level.

Progressive Think Tank: That sounds fishy. We will have our staff of passionate young research associates examine the raw data. (Suspicious) You do have the raw data, don’t you?

Free Market Think Tank: Er, um, well, yes, you can look up the data on (fill in obscure source of technical data such as the Widget Producers Trade Association here).

Progressive Think Tank: The…(long pause)…Widget Producers Trade Association? Couldn’t they be biased in favor of their greedy, self-serving corporate members?

Free Market Think Tank: Er, um, well, we’ll check on that.

(some time passes)

Progressive Think Tank: Our research associates have completed their analysis of your…flawed, biased data and found that the data from the (heavy sarcasm) Widget Producers Trade Association does not properly account for (fill in obscure technical issue that only widget manufacturing experts understand here).

Free Market Think Tank: Not so! We have reanalyzed the data from the Widget Producers Trade Association using the Blowhard Method (see path-breaking paper by Blowhard et al in the American Journal of Widget Manufacturing) and, well yes, a few numbers have changed slightly, but our study holds up and remains statistically significant at the (fill in confidence level here) level.

Progressive Think Tank: The Blowhard Method? Isn’t Blowhard…(long pause)…the Director of Research at Amalgamated Widgets, the notorious multinational anti-union widget manufacturer?

Free Market Think Tank: Well, yes, we go to the top experts and well, heh heh, they always seem to work for Amalgamated! They are the top widget company after all!

Progressive Think Tank: Well, we will have our staff of passionate young research associates take a further look at the data.

Free Market Think Tank: You do that. Meanwhile we will be putting out a new press release and appearing on CNBC, Fox, and the Colbert Report!

(some time passes)

Progressive Think Tank: Ahem, we have found that your reanalysis did not consider the well-known Sneezer Effect documented in Sneezer et al, in the International Journal of Widget Manufacturing. Sneezer demonstrated that when widgets are manufactured using the cold aluminum brazing method of Warner and Hancock, the Blowhard Method is inappropriate and biased. Once we account for the Sneezer Effect, your positive effect disappears and we find that the (insert euphemism for socialist here) widget producers outperform the corporate widget producers by (fill in percentage level here) percent at the (insert confidence level here) level. We are putting out a new press release and appearing on CNBC, Rachel Maddow, and the Daily Show!

This can and often does continue indefinitely as each side finds and/or rules out various scenarios under which some sort of bias or measurement error could have produced or invalidated the contested result (the free market failed, union workers are lazy etc.). These debates often get extremely technical and arcane as the specific potential sources of bias or measurement error frequently involve the technical details of the specific activity being debated. Frequently most third parties without detailed expertise in the subject matter (e.g. widget manufacturing) cannot follow the debate and lack the time and resources to come up to speed on the arcane issues of the topic. They frequently lose interest in the entire issue or simply revert to their ideological biases, preconceived notions and so forth.

The fancy statistical techniques such as regression analysis often have no or limited ability to deal with the potential sources of bias. Thus, the accuracy of the quoted statistical results depends on the original selection or measurement of the data. Even if one can rule out specific scenarios for bias or measurement error, one can usually (not always) come up with yet another scenario that requires a new experiment, study, or analysis to rule the new scenario out. And outright fraud or hoaxing is extremely difficult to rule out. One can almost always construct some sort of conspiracy theory to circumvent any seeming precautions against fraud or hoaxing. Of course, to maintain one’s serious scholarly reputation, one should never call it a “conspiracy theory.”

This infinite regress in which a new possible source of bias or error replaces each previously proposed source of bias or error as it is ruled out or demonstrated unlikely occurs in many scientific controversies, especially in fringe areas such as parapsychology or UFOs. Parapsychology is particularly relevant because laboratory parapsychology such as J.B. Rhine’s famous studies at Duke University involves extensive use of statistics. It actually proves extremely hard to design and carry out experiments in which there is no possible source of significant bias, measurement error, or fraud. Fraud or hoaxing. in particular, is often difficult to prove and even more difficult to conclusively rule out.

What to make of KIPP?

It is difficult to know what to make of KIPP. It is a small program with schools located exclusively in quite poor areas by US standards that are mostly out of the author and probably most readers’ experience. Both advocates and critics have potentially serious conflicts of interest; even a small fraction of $500 billion is a huge prize. In violent crime-ridden areas there are methods of encouraging poorly performing students to leave or otherwise rigging the results that probably could not be detected by conventional scholarly methods. This could happen without any actual intent on the part of the wealthy KIPP leaders. “Do whatever it takes” can have a very different meaning in the South Bronx than at Princeton or Yale.

It is reasonable to expect that the longer hours, Saturday classes, and longer school year, combined with more extensive drilling and so forth could produce substantial measurable benefits, especially on standardized tests. “Practice makes perfect.” and “Practice, practice, practice.” are ancient maxims.

If authentic, is KIPP’s success an exemplar of the theory of deliberate practice proposed by K. Anders Ericsson? In one sense of deliberate practice, it is clearly not. In some versions of deliberate practice, the need to practice relatively rare activities such as the tennis backhand is emphasized. The basic arithmetic and basic reading on standardized tests is generally not rare and does not involve rare trick questions. Ordinary practice or experience is probably all that is needed to improve performance on these tests. Indeed this may be one of the differences between math and science education at the K (kindergarten) through 12 (12th grade) level as opposed to college and university. Trick problems actually seem rare in K-12 whereas college and university level math and science exams often have many more of these and the preparation methods taught in K-12 school may be inadequate.

Another sense of deliberate practice is “continuous improvement,” emphasizing continuous self-examination of failures. In this sense, the KIPP programs may well be examples of deliberate practice judging from the descriptions of KIPP. Indeed, one might expect this to be helpful in improving performance even if deliberate practice is an incomplete or even mostly wrong theory of expert performance.

A more difficult question is what effect the KIPP programs have on general and abstract reasoning skills which can be difficult to measure on standardized tests, in part because even their nature is poorly understood. If this effect is negative, as I suspect it would be if the KIPP methods are largely equivalent to rote memorization, then the widespread adoption of KIPP methods could have a disastrous effect on public education and the economy. It is likely that these general and abstract reasoning skills are used heavily in major inventions and discoveries, commonly referred to as “breakthroughs”, an extremely rare phenomenon often cited as a justification for education reforms, especially math and science education reforms.

Conclusion

The vast majority of people would like to see excellent schools that enable a vibrant growing economy and productive scientific and technological progress. Unionized teachers and education entrepreneurs alike, for the most part, probably agree on the goal of well educated, happy, successful students. The question for most people of good will is how to achieve this goal. Seemingly sophisticated scientific, mathematical, and statistical methods often fail to provide the clear answers that most would like and may obfuscate the issues, substituting intimidating mathematical formulas, obscure statistical jargon, and arcane computer programming for genuine understanding.

Many debates like the KIPP debate would be improved by the full disclosure of the raw data (individual tests and test scores in this case) as well as the precise computer programs used to analyze the data. This question has arisen in the context of global warming for example. With the Internet and the World Wide Web it is certainly possible to do so. The analyses can be performed using freely available high quality statistical tools such as R or the MATLAB compatible tools Octave or Scilab or a number of others. Conventions could be established for the programs and data such as requiring human readable (plain English) variable and functions names that reflect the meaning of the function or variable: e.g. StudentAttritionRate for the student attrition rate and so forth. Many people are intimidated by cryptic symbols and computer code. A point-and-click, drag-and-drop graphical user interface (GUI) front end to the mathematical scripting code would be helpful in making the data analysis more accessible and credible to ordinary citizens as well as policy makers. The wxMaxima front end to the Maxima computer algebra system is a limited example of a graphical front-end for a mathematical scripting language. With student performance data and medical records there are clearly issues regarding protecting the privacy of the students or medical patients. With full disclosure, anyone could examine and reproduce the entire data analysis, step by step.

With current statistical methods, there does not seem to be a very good, or at least a very good well known way to deal with the persistent issues of bias and measurement errors or the infinite regress that often results from these issues. Rather, these central problems are dealt with in an ad hoc fashion that varies greatly from field to field, perhaps necessarily. Disclosing raw data alone cannot fully address these problems. It is, at minimum, necessary to have detailed information on how the data was selected and measured in the disclosure (metadata in computer jargon). Existing statistical methods are probably inadequate to deal with some of these bias and measurement issues. In part for these reasons, it is difficult to reach a definitive conclusion regarding KIPP either as an education reform or as an exemplar of the theory of deliberate practice.

© 2011 John F. McGowan

About the Author

John F. McGowan, Ph.D. is a software developer, research scientist, and consultant. He works primarily in the area of complex algorithms that embody advanced mathematical and logical concepts, including speech recognition and video compression technologies. He has extensive experience developing software in C, C++, Visual Basic, Mathematica, MATLAB, and many other programming languages. He is probably best known for his AVI Overview, an Internet FAQ (Frequently Asked Questions) on the Microsoft AVI (Audio Video Interleave) file format. He has worked as a contractor at NASA Ames Research Center involved in the research and development of image and video processing algorithms and technology. He has published articles on the origin and evolution of life, the exploration of Mars (anticipating the discovery of methane on Mars), and cheap access to space. He has a Ph.D. in physics from the University of Illinois at Urbana-Champaign and a B.S. in physics from the California Institute of Technology (Caltech). He can be reached at [email protected].

References/Suggested Reading

Spin Cycle: How Research Is Used in Policy Debates: The Case of Charter Schools
Jeffrey R. Henig
Russel Sage Foundation Publications
(February 2008)

The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education
Dianne Ravitch
Basic Books
(March 2, 2010)

A Chance to Make History: What Works and What Doesn’t in Providing an Excellent Education for All
Wendy Kopp
Public Affairs
(January 25, 2011)

Crash Course: Imagining a Better Future For Public Education
Chris Whittle
Riverhead Hardcover
(September 8, 2005)

The Charter School Dust-up: Examining The Evidence On Enrollment And Achievement
Martin Carnoy (Author), Rebecca Jacobsen (Author), Lawrence Mishel (Author), Richard Rothstein (Author)
Teacher College Press & Economic Policy Institute
(April 30, 2005)

Work Hard. Be Nice.: How Two Inspired Teachers Created the Most Promising Schools in America
Jay Mathews
Algonquin Books; First Edition edition
(January 20, 2009)