Hard Facts and Statistical “Facts”

“Trust No One” — Fox Mulder’s Password

I recently had a conversation with some colleagues about Apple founder and CEO Steve Jobs death from pancreatic cancer in 2011 in which my colleagues repeated the notion that Jobs would have lived if he had not experimented with “alternative” medicine prior to his surgery for pancreatic cancer — which he described movingly in his famous commencement address to the Stanford Class of 2005. There were a number of news articles and Internet posts that claimed or implied this shortly after Jobs passed away, implying or even claiming that mainstream or “science-based” medicine could cure pancreatic cancer.

This is highly questionable at best. In his book Creativity Inc. about the Pixar animation studio, also headed by Jobs, Ed Catmull describes an October 2003 conversation with Jobs in which Jobs revealed that he had just been diagnosed with pancreatic cancer. Jobs passed away on October 5, 2011, almost exactly eight years later. Steve Jobs actually lived unusually long despite or even perhaps due to dabbling in alternative medicine.

There are two main forms of pancreatic cancer that develop from two different types of cells in the pancreas. The more common form of pancreatic cancer usually kills within about six months after diagnosis. Steve Jobs had the much rarer type of neuroendocrine pancreatic cancer which develops more slowly — like most other cancers. Only about seven percent of people diagnosed with all forms of pancreatic cancer survive at least five years. Steve Jobs lived at least eight years after his diagnosis.

Studies have reported five year survival rates from as low as twenty-four percent to as high as eighty-seven percent for patients diagnosed with the rare neuroendocrine type of pancreatic cancer that afflicted Steve Jobs. A large fraction of the seven percent of pancreatic cancer patients who survive five years have the rare form of pancreatic cancer.

On its NCI Web Site on Neuroendocrine Pancreatic Cancer, the National Cancer Institute (NCI) gives a five year survival rate of forty-two percent (42%) for the rare neuroendocrine pancreatic cancer. Close reading of the graphic on the NCI web site indicates a ten year survival rate of about thirty percent (30%) for neuroendocrine pancreatic cancer. The survival rate is still dropping noticeably at ten years. Thus surviving ten years should not be considered cured in the common usage sense.

Pancreatic Cancer Survival Rates (Source: NCI)

To be sure, Steve Jobs made use of orthodox mainstream medicine, had the primary tumor in his pancreas removed, had a liver transplant that probably extended his life by several months, and reportedly had chemotherapy. There is nothing in his case that demonstrates that his reported dabbling in diets or acupuncture deserves credit for his unusually long survival. Indeed he might have lived longer had he had surgery immediately after the diagnosis; he reportedly waited nine months before agreeing to surgery. It is unlikely that he would have been cured in the common usage meaning of cured: lived to old age to die of something else. It is also possible he lived unusually long due to his alleged experiments with alternative medicine. The studies cited above suggest that it is not that uncommon for patients with Jobs rare form of pancreatic cancer to live eight years.

Most cancers are a slow progressing disease, with several years between diagnosis and death, unlike the more common form of pancreatic cancer that usually kills in about six months. Because many cancer patients survive about five years, the widely computed and reported five-year “survival rates” are highly misleading and don’t correspond in a meaningful way to the common usage definition of “cured” or “surviving” an illness. For example, when one says “automobile accident survivor” one is typically talking about someone who has survived an accident, is still alive, and will die from some other cause in the future.

There are also several different “survival rates” and it is often not clear or difficult to determine which one is being reported. The naive, raw survival rate would be what percentage of patients diagnosed with cancer are still alive after five years or some other duration. This is probably what most readers would consider the “survival rate.” However, the raw survival rate includes deaths from other causes than the cancer.

The “relative survival rate” is defined as the ratio of the proportion of observed survivors in a cohort of cancer patients to the proportion of expected survivors in a comparable set of cancer free individuals. The “relative survival rate” requires performing a complex statistical analysis on the raw numbers. It makes the numbers look better and it’s accuracy depends on an accurate model of the survival rates for cancer free individuals. As we shall see, there are “childhood cancer survival rates,” “adult cancer survival rates,” “overall cancer survival rates,” and other variations which further complicate interpreting the statistics.

One needs to look at the survival rates out to at least ten years which are rarely reported. For example, if ninety percent of cancer patients survived to five years and nearly ninety percent were still alive after ten years, then there would be a good case that the cancer had been truly cured. On the other hand, if ninety percent survived to five years and only forty percent last ten years, it is likely the cancer was not cured and even more patients will still die after ten years.

Curiously, as of June 21, 2016, a Google search on the phrase “cancer cure rate” will bring up articles on the cancer survival rates although these are not cancer cure rates, instead of articles on the lack of actual cure rates in the scientific and medical literature or the distinction between the cancer survival rate in the scientific literature and the common usage meaning of “cure.” Peer-reviewed scientific and medical articles generally avoid the term “cure.”

The duration between diagnosis and death is highly variable on an individual basis. Some people with cancer last only a year, some two years, some three years, and very few past five years in the case of pancreatic cancer and a number of other types of cancer.

The current reigning theory of cancer, the oncogene theory attributes cancer to the accumulation of random mutations of genes in the cancer cells. It further claims that the cancer cells become genetically unstable with a much higher mutation rate than normal healthy cells. This increased mutation rate is sufficiently large that genetic sequencing has now reportedly shown that different cells in the same tumor in the same patient can have substantially different genetics (see here and here and here). This means that cancer is an inherently random “stochastic” process and the prognosis is inherently highly variable and unpredictable from patient to patient.

When mainstream cancer doctors and researchers make confident statements about effective cancer treatments or even “curing” cancer, they are almost invariably talking about statistical studies that show slight improvements, perhaps a few more months of life on average in data with high variability, compared to the current state of the art treatment.

These claims are usually impossible to confirm on an individual basis because of the high variability from individual to individual. In the same way that we cannot be certain of the contribution from acupuncture, surgery, chemotherapy, the liver transplant, other factors, or pure luck to Steve Jobs unusually long eight years of survival after his diagnosis. One needs to perform an expensive, complex, time consuming study of a large number of cancer patients to confirm the statistical results. Further, often a complex statistical analysis is required and the analyst might in the end only replicate the same mistake in the complex analysis made in the first study.

The main exception to this is a number of relatively rare cancers such as several forms of leukemia, non-Hodgkin’s lymphoma, and testicular cancer where mainstream medicine claims more dramatic although still statistical improvements from various forms of chemotherapy. These are mostly liquid forms of cancer, cancers of the blood, lymphatic fluids, and so on. In these one sometimes sees claims that, for example, about 80 percent of childhood leukemia patients with proper chemotherapy treatment will survive for ten years.

It is thought that liquid cancers are qualitatively different from the solid cancers that afflict most cancer patients and more responsive therefore to chemotherapy.

Childhood Cancer Five Year Survival Rates

There are differences between childhood cancers and leukemias which are extremely rare and adult cancers and leukemias. Here is a quote from the American Cancer Society web page “What are the differences between cancers in adult and children?” (accessed June 26, 2016)

There are some exceptions, but childhood cancers tend to respond better to treatments such as chemotherapy (also called chemo). Children’s bodies also tend to handle chemotherapy better than adults’ bodies do. But cancer treatments such as chemo and radiation therapy can cause long-term side effects, so children who have had cancer need careful follow-up for the rest of their lives.

Curiously, when I searched for “leukemia survival rate” on Google (on June 26, 2016), the top hit, prominently displayed — a Google “featured snippet,” was an article on the childhood leukemia five year survival rate, rather than the general rate including adults who make up the majority of leukemia victims.

Google Search for Leukemia Survival Rate

It is common to encounter the childhood leukemia survival rate in discussions of leukemia and leukemia treatment instead of the overall or adult leukemia survival rate.

It is worth emphasizing that these cancers are rare, comprising only 2-4 % of all cancers, and one does not encounter them frequently. The cyclist Lance Armstrong is a well-known high profile testicular cancer “survivor.” Whereas a dramatic advance in breast cancer treatment would be clearly visible to the general population due to the frequency of breast cancer and thus easily confirmed without relying on pharmaceutical industry sponsored studies, this is not the case with these rare cancers. One still encounters reports of deaths from these cancers. For example, the admissions director of the private school that I attended recently passed away from leukemia. Note that this was an adult leukemia case.

State of the art cancer treatments are usually extremely expensive, running up to $100,000 per year for many new purported cancer drugs. Pharmaceutical companies have an extremely strong financial reason to find a positive statistical effect, whether through deliberate fraud or unconscious bias.

What we are dealing with is a statistical “fact.” This is often a small effect that is detectable only as an average or trend, sometimes extracted by complex mathematical modeling, in a large sample of data with high variability in the individual samples. The purported effect is often smaller than the individual variation of the samples and thus impossible to confirm or deny based on our personal experience. These are the type of effects that can easily be produced through biased sampling, subtle and not-so-subtle errors in the statistical analysis, unconscious biases, and subtle and difficult to prove fraud. They can also be real.

Do Flu Vaccines Work?

I encountered another example of this earlier last year (2015). I ran into a number of highly educated technical friends and colleagues who were surprised and puzzled at getting sick with the common cold shortly after taking the much ballyhooed flu vaccine. In the United States the Centers for Disease Control along with many local pharmacies and doctors heavily promote getting an annual flu vaccine. Again medical scientists and doctors make confident statements about the importance of the flu vaccine. These are often interpreted by even highly educated technical people as meaning that the flu vaccine will prevent getting the flu.

In fact, in the fine print, what the authorities are claiming is that taking the flu vaccine will somewhat, slightly reduce the odds of getting a flu, a common cold. There is no guarantee that the vaccine will prevent any particular cold. Because of this variability, the patient cannot confirm the effectiveness of the flu vaccine. People average a couple of colds per year. Some people have more each year. Some people have less each year. You might get sick six times one year and never the next year.

Here are the conclusions of the Cochrane Collaboration analysis of 116 studies of the effectiveness of the flu vaccines:

The preventive effect of parenteral inactivated influenza vaccine on healthy adults is small: at least 40 people would need vaccination to avoid one ILI case (95% confidence interval (CI) 26 to 128) and 71 people would need vaccination to prevent one case of influenza (95% CI 64 to 80). Vaccination shows no appreciable effect on working days lost or hospitalisation.

Again, this is a statistical “fact,” a weak effect detectable on average in highly variable data. Yet, confident statements, the language of science, and aggressive advertising create the impression it is a hard fact, something you can count on. Get the flu vaccine and you won’t get sick!

Hard Facts

There are hard facts, assertions and beliefs that we can be very certain of. For example, I am confident beyond any doubt that I cannot walk magically through the wall of my room. This is something that I know from direct, repeated, fully reproducible experience. There is nothing statistical about it.

Nineteenth century, table-top, “little science” produced a body of knowledge that we can be confident of. We can be truly confident of Newtonian Mechanics and Maxwell’s Equations (Classical Electromagnetism) under everyday conditions. Students reproduce these results in high school and college physics labs every day. They can easily be tested if we so desire in a garage or kitchen at small cost. We buy and use machines ranging from electric fans to automobiles to iPhones that use these physical processes repeatably on command.

This reproducible, deterministic “hard” science and mathematics has worked its way into everyday life and gives us our exaggerated sense of the effectiveness and reliability of “science.”

Even in medicine and biology, which are inherently more variable than mass produced machines and laboratory equipment, there are hard facts we can be sure of. An example is a common problem with hearing in which the ear canals fill up with ear wax. I have had this happen twice. In this case, the problem can be seen with special microscopes for examining the ear used by doctors. It can be reliably fixed using a special machine to inject air and water into the ear. It can be explained with Newtonian Mechanics. The cure is also immediate and perfectly correlated in time with the treatment; hearing is restored immediately after the ear wax is cleaned out of the ear.

Statistical “Facts” are not Hard Facts

Statistical “facts” where a weak effect is extracted from highly variable data by averaging or extracting a trend, often using complex mathematical modeling and statistical analysis, and where the effect is often smaller than the natural individual variation of the samples are not hard facts. They can be true, but they can also be produced by a large variety of misleading and false causes. Statistical “facts” are all around us. They are pervasive in the research, development, and marketing of medical treatments and drugs, global warming/climate change, economics and politics, software engineering methodologies like Agile Programming, mortgage backed securities, laboratory parapsychology, purported research on alternative medicine such as acupuncture, and many other areas. They are often coupled to the language of hard science by proponents, yet there are substantive and qualitative differences. In many, many cases we cannot confirm or deny these statistical “facts” with our own experience or by performing explicit experiments or studies which are too complex, time consuming, and expensive for the vast majority of people. This is quite unlike the easily reproducible table-top “little science” we have inherited from the nineteenth and early twentieth century.

Statistical “facts” are soft facts. They should be viewed with caution and should not be elevated to the level of hard facts.

The World View Problem

In the abstract, many people, many technical people, certainly most readers of this blog are well aware of this. However, with statistical “facts” we encounter the enormous psychological and social power of our world view, our sense of personal identity, and our group or tribal self-identification. In specific terms, our world view is strongly influenced by or even identical with the world view of the social group or groups that we identify with — our family, ethnic group/tribe, political party, profession, nation, etc. — and we often will not accept evidence that contradicts that world view, even very strong evidence, and on the other hand will uncritically accept faulty evidence or arguments that supports our world view.

Even though we should know that statistical “facts” are suspect, if the statistical “fact” confirms our world view, we will often treat it as a hard fact, as though it was as certain as my certainty that I cannot walk magically through the wall of my room. We will often become emotional, angry, and indeed baffled that someone could challenge the statistical fact although such statistical facts are justifiably suspect.

We see this all the time with the many statistical “facts” that are proliferating in the modern world. With global warming/climate change, for example, liberals and leftists tend to uncritically embrace the statistical “fact” whereas conservatives tend to reject the same “fact” with never-ending critical analyses of sloppy methodologies, biased sampling and all the other difficult to rule out issues that make any statistical fact — whether we agree with it or not — suspect.

Statistical facts are soft facts. They should be viewed with caution and should not be elevated to the level of hard facts even if they confirm our world view. Nor should we discount them as false if they contradict our world view.

What To Do

With modern computers, sensors, and massive data collection that was technically infeasible just a few years ago, the volume of statistical “facts” is increasing dramatically and the use and abuse of statistics to generate these “facts” is expanding dramatically. It has outstripped the ability of most people to process these “facts,” and indeed probably exceeded the skills and knowledge of even experts in statistics and other relevant fields. Cynical marketers have developed sophisticated methods of packaging statistical “facts” to link them to our world views and thus greatly enhance their effectiveness is selling drugs, wars, and other products.

Statistical “facts” are not hard facts. We cannot and should not have the same level of confidence in them that we can for hard facts such as my confidence that I cannot walk magically through the hard unyielding wall of my room or that we have for some results of deterministic, easily reproducible, shoestring budget table-top science from the nineteenth and early twentieth century.

We need to separate out the truly hard facts, that we can be very confident of, from the softer facts which are plentiful and should be treated with caution. Statistical “facts” are soft facts. They are not hard facts.

Historically, scientists, engineers, and others have succeeded in elevating a soft fact to a truly hard fact only by isolating the underlying phenomenon, strengthening it, and demonstrating it reproducibly without significant statistical variation. In some cases, this is done as in a nuclear reactor or weapon by averaging over the statistical variations with a huge number of samples.

In the absence of a strong, reproducible, non-statistical effect, there are reforms that could reduce the probability of unconscious bias or deliberate fraud producing a false positive effect. The federal government could require that studies of the safety and effectiveness of a new medical treatment be conducted in a fully blinded way by a third party independent of the manufacturer who stands to profit from approval of the treatment, either a randomly selected contract lab or a randomly selected government lab staffed by civil servants.

In either case, the law would require that the patients, scientists and managers at the FDA responsible for the actual study have no idea what the treatment, which is generally a drug, is actually called, who makes it, and other identifying characteristics. The manufacturer would not directly fund or supervise the testing. The lab would simply know they are testing the safety and effectiveness of compound X321 from unknown manufacturer Y3B. The manufacturer would not select the third party laboratory or team and have no control over future selection of the laboratory or team for future evaluations.

Incidentally, there is nothing particularly new about proposals for independent testing of drugs and other medical treatments. They date back at least to hearings by Senator Estes Kefauver and other politicians in the 1950’s and 1960’s. Independent testing has been repeatedly proposed by medical doctors and other experts as well.

All data and meta-data used in the statistical analysis, suitably anonymized to protect the privacy of the test subjects, as well as all computer programs used to analyze the data should be made public so that anyone can quickly reproduce and review the entire analysis process to confirm the results.

Finally, one frequently encounters a revolving door between the government agencies and regulators and the medical and pharmaceutical firms, with government officials taking jobs as executives or scientists at the firms and vice versa. Certainly in the case of an in-house civil service laboratory that evaluates the safety and efficacy of medical treatments, it would be simple to provide a generous pension and benefits to attract good people but require that they never accept employment or money in any form from the regulated firms ever after, in particular those firms whose medical treatments they have evaluated for safety and effectiveness.