STEM Shortages, Purple Squirrels, and Leprechauns

This is the third in a series of articles that started with The Mathematics of the Ph.D. Glut. This article discusses claims of STEM worker shortages in private industry and how specific work-experience based skills are confused, intentionally or not, with general science and mathematics taught in schools and universities.

One often encounters claims of a shortage of STEM (Science, Technology, Engineering, and Mathematics) workers in the United States. This often seques into arguments for more STEM education, teaching, students, and specifically Ph.D.s in these fields. Indeed, the current Great Recession has been blamed on a “skills gap” in which employers have allegedly been unable to find appropriately skilled employees despite a desperate need. The “skills gap” often seems to mean a shortage of STEM workers, although it seems to have spilled over into everything from machinists to cotton candy machine operators.

These STEM shortage claims have been strongly challenged by a number of academics including Norman Matloff, Peter Cappelli, Ron Hira, Paula Stephan, Hal Salzman and a number of others. They have also been challenged by practicing engineers and their families — most famously by Jennifer Wedel, the wife of an out of work semiconductor engineer, who confronted President Obama about the claims.

Purple Squirrels

It is important to understand a subtlety of STEM worker shortage claims. When high tech companies and their lobbyists claim there is a shortage of skilled high technology workers, they usually use general language such as “there is a shortage of engineers,” “there is a shortage of programmers,” or “there is a shortage of technology talent.” However, when pressed about seemingly highly qualified, often older workers who cannot find jobs, they refer to both extensive and very narrowly defined specific skills that they claim they must have. Older often means over forty or even over thirty-five.

There is actually a term in the human resources business for such narrowly defined, extremely difficult or impossible to find job candidates: purple squirrels. This term has been popularized by Google recruiter Michael B. Junge in his recent book Purple Squirrel.

It has been common for over twenty years in the computer industry to encounter large numbers of extremely narrowly defined job descriptions.

For example, a job ad may say something like:

Required

3+ years work experience developing iPhone game apps using XCode, at least two apps in the Apple app store
5 years Objective-C (the programming language for the iPhone)
3 years experience with CoreAudio (a particular iPhone software compnent)
3 years Agile software development (Agile is a software development methodology)
etc. (it is not uncommon to see lists of twenty requirements like this for one job)

Since iPhones have only been in existence and hot for a few years, there are very few if any software engineers who can meet these qualifications. Keep in mind in 2008 (four years ago), Blackberry was the King of the smartphone world. The iPhone was the “iWhat?”

These lists of requirements are often so long and picky that there may not be a single engineer in the world, including genuine iPhone experts, who would match them.

When high technology companies refer to shortages or to battles for talent they are almost always referring to this sort of extremely rare or nonexistent “engineer” with extensive work experience in dozens of very specific, often new, tools. It is almost always expressed as years of experience using a specific tool, programming language, etc. rather than a more general knowledge of the activity — such as science or mathematics. It is often like demanding that a car driver have experience driving a very specific make and model of car; we only hire experienced Volvo drivers!

Yes, there is a shortage of purple squirrels, almost by definition, but it has little to do with science, technology, engineering, or mathematics skills or education as most people would define them. It is probably impossible for colleges to anticipate fads such as the iPhone displacing the Blackberry. Schools and colleges do not provide work experience. In fact, traditionally, schools and colleges have sought to provide general foundational skills such as reading, writing, basic arithmetic, critical thinking, and so on that will remain valuable for the life of the student — with good reason.

The fixation on purple squirrels means that STEM careers are precarious and often short-lived. If you are a purple squirrel and you can find the one or two employers interested in you, you may do well for a few years. But what then? What if a new fad or fashion comes along?

A Personal Experience

Back in the 1990’s the author worked for a startup company that wanted to get into the then new, suddenly hot world of the World Wide Web and the Internet. This startup had essentially no in-house expertise on the Web; actually, direct experience with the Web and Internet was quite limited at the time. The startup was considering streaming digital video over the web in 1995 — a technically challenging project to say the least.

The author introduced the startup to Alistair (name and some details changed to protect the privacy of the people). Alistair was a Ph.D. in physics from one of the top universities in England. As an experimental particle physicist at the Stanford Linear Accelerator Center, he had been heavily involved in the early days of the World Wide Web which started out as a tool used by the particle physics community.

Alistair actually had several years of paid professional experience at a government research laboratory working on the Web, something almost no one could legitimately claim in 1995. His work involved creating browsers, servers, and other software tools, not just or even mostly creating web pages or web sites, although he did some of that as well.

Alistair was an accomplished software developer and had done a number of consulting jobs for private industry on the side. Alistair also had good communication skills and, in my experience, did not suffer from the notorious arrogance of physicists. He was a genuine purple squirrel.

The startup turned up their collective noses at Alistair, hired a much less experienced consultant who admittedly dressed better than Alistair, and spent several weeks and several thousands dollars fumbling around without success.

How real is a purple squirrel hunt? It is not uncommon to see genuine purple squirrels like Alistair turned down in a purple squirrel hunt in the Silicon Valley.

Why did the startup turn down Alistair and go with a much less experienced consultant who failed? In all seriousness, I suspect that Alistair’s casual dress contributed. The winning consultant wore a suit and Alistair dressed — well — like many of the engineers at the startup.

It is likely that the startup executives did not realize that the particle physics community developed the World Wide Web. Even though the core technologies in the Silicon Valley can be traced time and time again back to government sponsored research programs, many Silicon Valley executives believe the myth that these technologies were developed in garages by teenagers or in sometimes non-existent corporate research labs. Gordon Crovitz’s notorious recent op-ed piece Gordon Crovitz: Who Really Invented the Internet? Contrary to legend, it wasn’t the federal government, and the Internet had nothing to do with maintaining communications during a war. is a prominent example of these faith-based beliefs.


The Market is my Shepherd; I shall not want.
The Market maketh me to lie down in green pastures:
The Market leadeth me beside the still waters.
The Market restoreth my soul:
The Market leadeth me in the paths of righteousness for His name’ sake.

Yea, though I walk through the valley of the shadow of death,
I will fear no evil: For the Market art with me;
Thy rod and thy staff, they comfort me.
The Market preparest a table before me in the presence of mine enemies;
The Market annointest my head with oil; My cup runneth over.

Surely goodness and mercy shall follow me all the days of my life,
and I will dwell in the House of the Market forever.

Market-Based Version of the 23rd Psalm

The startup executives were looking for a software developer with at least three years of paid professional industry experience working for the fantasy private companies that invented the World Wide Web!

Many (not all) hot new technologies are developed by relatively small groups of people working for government sponsored research programs, often in universities or government research labs. When the technologies suddenly become hot like the World Wide Web in the 1990s, of course, there is a “shortage” of people with direct experience. That is inevitable. In many cases, the few people with direct experience will not have industry experience, narrowly defined.

What does this “shortage” have to do with STEM education? Absolutely nothing.

Leprechauns

Laurent Bossavit’s recent book The Leprechauns of Software Engineering discusses several widely repeated and widely accepted shibboleths of software engineering, tracing them back to the few original studies, research papers, and, in some cases, unsubstantiated claims that they are based on. In every case that he discusses, the supporting evidence is very weak or nonexistent, hence “Leprechauns.”

Two chapters deal with the concept of the “10X programmer,” a widely-held belief in the computer industry. In its most extreme form, the “10X programmer” belief holds that there are superprogrammers who can write code ten times faster and better than the typical or average experienced programmer (NOT the worst programmer). This idea has been promoted by software engineering consultant Steve McConnell and Bossavit takes aim at McConnell’s supporting data in two chapters and a lengthy appendix detailing his hunt for the primary references and data on the 10X claim.

An obvious problem is that salaries for experienced software developers do not vary by a factor of ten for the vast majority of software developers including those with strong, impressive credentials. Rather the variation is more like a factor of two or three at most.

Bossavit found numerous problems with the 10X claims including the difficulty defining software productivity. In many cases, 10X referred to the difference between the best and worst programmers, in some cases studies of undergraduate computer science majors not professional experienced programmers. Some studies referred to differences in seeming productivity between different projects/teams, not broken down by individuals.

A widely reported claim of a 28:1 difference in a 1968 study referred to an apples to oranges comparison between one programmer coding a task in a high level language on a time sharing computer (like a modern computer) versus a different programmer coding the same task in low level machine language (zeros and ones) on a batch processing system (punch cards probably).

Although Bossavit found some evidence for the 10X claim in one detailed NASA study, there were significant issues in interpreting the NASA data that make even this one study open to serious debate. The study found much higher variations between programmers in smaller projects than longer projects which is difficult to understand; the study attributed this to less experienced programmers being assigned to the shorter/smaller projects. The bottom line is that the evidence behind the 10X belief is quite limited and subject to alternative interpretations.

That does not mean that companies and hiring managers don’t believe in 10X programmers (or scientists or mathematicians) and try to find them, repeatedly passing up seemingly highly qualified candidates. They may be hunting for non-existent or exceptionally rare “leprechauns” instead of the real 1X, 2X or at best 3X programmers, scientists, engineers, or mathematicians.

Could such exceptional people, if they exist at all, be due to STEM education? Perhaps, but such a large variation may have a genetic or unknown organic cause (eating lots of spinach?). In fact, flooding STEM fields with non-10X people, a logical consequence of hyping STEM education, may make it much harder to find these rare people even if they exist, much like SPAM making it difficult to spot serious e-mails from people that you do not know. Flooding STEM fields with non 10X people may even drive the rare 10X people (if they exist) out of the STEM fields altogether.

Should Kids Learn Math and Science?

Absolutely. In the modern world, essentially everyone needs to use basic arithmetic to buy food and other items, balance their checkbook, and perform other activities of daily living. We all use math every day. People need to understand raising numbers to powers to properly understand interest on savings accounts and loans. In the housing bubble, many people were bamboozled by complex mortgage terms and formulas. Algebra, trigonometry, and geometry — more advanced high school math — does actually turn up in various real world activities including building and architecture, mechanical design, navigation and other activities.

With the spread of computers into every home and device, the use of mathematics including quite advanced mathematics is becoming commonplace. Most people are now using sophisticated mathematical programs, often without realizing it: video compression such as YouTube and Skype, the GPS system which tells people where they are, and computer generated imagery, for example.

Complex mathematical models are increasingly being invoked in public policy such as the models underlying claims of “global warming.” In particular, a good understanding of probability and statistics at the college level is needed to understand and evaluate claims about expensive medical treatments such as chemotherapy for cancer. Should you rely on the claims of “experts” who often have conflicts of interest or do the calculations yourself?

However, while most kids should learn mathematics, should they get a Ph.D. in mathematics or pursue a career in mathematics? Most kids learn English, reading, and writing, but few become authors or newspaper writers, for example. Most kids play sports and many enjoy one or more sports, but few become professional athletes.

There are some jobs in the private sector that employ heavy mathematics, especially statistics. Some of these jobs pay quite well. However, they are few and far between, and often highly specialized. Probably less than one percent of present day software jobs are highly mathematical. The vast majority of software jobs utilize basic arithmetic heavily — essentially some form of bookkeeping — and little else. The competition for the rare highly mathematical jobs is fierce.

When private companies complain about shortages of STEM workers and specifically people with mathematical skills they are generally referring to difficulties finding specialists in a few narrow areas to fill very narrowly defined jobs: the purple squirrels.

Some companies such as some Wall Street investment firms are seeking or believe they are seeking truly exceptional scientists and mathematicians, e.g. the rare graduate student who has just received the Fields Medal for his dissertation on stochastic differential equations — the best of the best of the best. One can think of trading on Wall Street as a zero-sum game, a winner-take-all tournament in which even a tiny advantage can translate into vast returns. There is always a shortage of the best in the world. Only one athlete wins the gold medal at the Olympics in a given sport. Only one player can win the World Chess Championship.

Billionaire Land

STEM shortage claims can usually be traced back to blue ribbon panels, commissions, or private groups of corporate CEO’s and “power scientists,” politically active Nobel Laureates, major research university presidents, and similar academics. The current STEM mania can be traced back to the 2005 Rising Above the Gathering Storm report, chaired by former Lockheed CEO Norman Augustine.

Norman Augustine, for example, certainly seems sincere when he talks or writes about science and mathematics education.

Thirty, forty years ago, at least according to most media reports, there were only a handful of billionaires in the United States: Howard Hughes, Daniel Keith Ludwig, and perhaps one or two others. These men had fortunes that reportedly barely exceeded one-billion dollars. The number of billionaires has soared in the last forty years, as has the number of people with fortunes in the hundreds of millions of dollars.

These personal fortunes are so large, that most billionaires and centimillionaires probably have to live like ancient Roman Emperors, surrounded by bodyguards and handlers 24/7. Like Howard Hughes who may have become a prisoner of his mysterious personal security force, these folks live in a very different world even from the small rich with a few million dollars. Who in their right mind would disagree with a billionaire other than another billionaire? Not too many people.

When a billionaire CEO talks about a shortage of STEM “talent,” he or she may be sincere but they are often talking about something very different from common English usage. If, for example, one is having trouble hiring the next Nikola Tesla, a very unusual person, or even Thomas Edison, or at least one’s preconceived notion of what these people would look like in a modern resume, this is really very different from what a shortage of scientists or mathematicians means to most people, including students with strong or even exceptional scientific or mathematical talent.

Conclusion

STEM worker shortage claims in most cases refer to difficulty finding job candidates with extremely narrow, often lengthy job specifications, almost always several years of paid professional industry experience in specific tools or narrowly defined methodologies: purple squirrels. In some cases, the shortage claims probably refer to searches for truly exceptional candidates who may not exist: leprechauns. There is little relationship between these hiring difficulties and STEM education at the school or college level.

In most cases, companies can find job candidates by being less picky and not hunting for impossible or extremely difficult to find purple squirrels, especially in the current depressed economy.

Companies and hiring managers should also take a hard look at the actual evidence for 10X programmers or other varieties of supermen. Such people are certainly extremely rare and their existence is debatable. No one is going to find an actual leprechaun.

© 2012 John F. McGowan

About the Author

John F. McGowan, Ph.D. solves problems using mathematics and mathematical software, including developing video compression and speech recognition technologies. He has extensive experience developing software in C, C++, Visual Basic, Mathematica, MATLAB, and many other programming languages. He is probably best known for his AVI Overview, an Internet FAQ (Frequently Asked Questions) on the Microsoft AVI (Audio Video Interleave) file format. He has worked as a contractor at NASA Ames Research Center involved in the research and development of image and video processing algorithms and technology. He has published articles on the origin and evolution of life, the exploration of Mars (anticipating the discovery of methane on Mars), and cheap access to space. He has a Ph.D. in physics from the University of Illinois at Urbana-Champaign and a B.S. in physics from the California Institute of Technology (Caltech). He can be reached at [email protected].

References/Suggested Reading


Purple Squirrel
By Michael B. Junge


Why Good People Can’t Get Good Jobs: The Skills Gap and What Companies Can Do About It
By Peter Cappelli

The Leprechauns of Software Engineering
The Leprechauns of Software Engineering: How folklore turns into fact and what to do about it
By Laurent Bossavit


Antitrust
A very fun, very unrealistic movie with Tim Robbins, Ryan Phillipe, and Claire Forlani that depicts the classic mythology of teenagers in a garage developing breakthrough technologies in the Silicon Valley.

Hidden in Plain Sight: The Secret History of the Silicon Valley
Serial entrepreneur and high technology startup expert Steve Blank’s presentation on the role of the government, especially “members of the intelligence community,” in the Silicon Valley.

11 Comments

  1. Daniel Huarskon August 26, 2012
    • Reverse Engineer July 6, 2017
  2. John F. McGowan August 27, 2012
  3. Christian Marks August 28, 2012
    • zephyr September 4, 2012
  4. John F. McGowan August 28, 2012
  5. gander August 30, 2012
  6. John F. McGowan August 30, 2012
  7. gander August 30, 2012
  8. John F. McGowan November 25, 2012
  9. Reverse Engineer July 6, 2017

Leave a Reply