This article takes a look at the job prospects of freshly minted STEM (Science, Technology, Engineering, and Math) Ph.D.’s who pursue a job in private industry, which frequently means some form of software development in the “technology,” formerly known as “computer,” industry. On September 19, 2013, Chand John, a recent Ph.D. graduate from the top rated Stanford University computer science department, published an article “The Ph.D.-Industry Gap” in the Chronicle of Higher Education recounting his lengthy, difficult search for a job in industry. Judging from his account, his job hunt in no way resembled the claims made by his graduate program at Stanford or the many “talent war stories” found in the mass media about the alleged battle for programming talent in the Silicon Valley. The article spawned many comments on the Chronicle of Higher Education and Hacker News web sites. This article discusses the realities facing many freshly minted STEM Ph.D.’s seeking a job in private industry and a number of related issues that received scant mention either in Chand John’s article or the many comments.
Generally, I will argue that Ph.D.’s in practice are not sought after by most industry, including the “technology” industry. In fact, a STEM Ph.D., especially a recent one, will usually make it harder to find a job in industry and closes some doors entirely due a widespread prejudice against Ph.D.’s. However, there are special cases where a Ph.D. or a Ph.D. in a specific area will open some doors — including some doors one should not go through. Because a Ph.D. program usually takes at least five years — and often longer today — it is difficult or impossible to accurately predict if a specialized area will be “hot” when the degree is issued and the fresh Ph.D. needs to find a job.
The Plural of Anecdote may be Data
At least based on word of mouth, experiences such as Chand John’s lengthy search with many rejections are common amongst newly minted Ph.D.’s even from top rated programs, and even sometimes with allegedly “hot” skills. As an example, many moons ago, when I was finishing up my Ph.D., I knew a recent Ph.D. in EE (Electrical Engineering) who had worked in an area of biomedical engineering. Like Chand John, he had extensive programming skills but found Silicon Valley employers rather uninterested despite the perennial claims of a desperate shortage of programmers. His job search took over a year and eventually resulted in a good job at an industry sponsored research institute where his Ph.D. thesis research had a direct, specific application. Chand John also reportedly eventually found a job at a research institute more closely related to his Ph.D. research topic.
In the “technology” industry, Ph.D.’s and especially newly minted Ph.D.’s often report encountering strong prejudices against them which include: Even exceptionally difficult programming projects in a research setting are dismissed as not “real” in some often ill-defined way. A fear that Ph.D.’s may leave and found their own company. A general hostility that may disguise a fear that the Ph.D. is smarter or more skilled and poses a threat to the non-Ph.D. employees. This final prejudice is almost never stated :-).
Business Leaders Say They Want Ph.D.’s
It is common when recent Ph.D.’s complain that they encounter little interest in their degree and indeed active hostility in some cases, for commenters to express astonishment: “How could you think a Ph.D. makes you special or anyone in industry would want a Ph.D.?”
In fact, in certain contexts, business leaders and their allies claim or strongly imply they need and want Ph.D.’s. This often occurs in the context of the ubiquitous STEM shortage claims, lobbying for various visa programs such as the controversial H1-B visa, and lobbying for ostensible education reform. A particularly egregious example of this widespread practice is the editorial “Our Ph.D. Deficit” by former Lockheed Martin CEO Norman Augustine and former Stanford Linear Accelerator Center (SLAC) Director and Nobel prize winning physicist Burton Richter in the May 4, 2005 Wall Street Journal. Another specific example is the failed STAPLE (Stopping Trained in America PhDs from Leaving the Economy) Act to give new Ph.D.’s green cards (permanent resident status) that was introduced in the United States House of Representatives in 2009. Indeed, the graphic image of stapling a Green Card to a fresh Ph.D. diploma occurs frequently in debates about high-tech immigration policy to this day.
Many Businesses Exaggerate Their Research Activities
In certain contexts, especially political lobbying, many businesses exaggerate their research activities. The most extreme example of this is found in the United States pharmaceutical industry where research is routinely invoked to justify large and rising prices of medicines and huge profits and profit-margins, mostly funded by the public Medicare program. For example, the industry-funded Tufts Center for the Study of Drug Development (CSDD) has for decades put out widely quoted studies claiming astonishingly high costs for the research and development of drugs, currently $1.2 billion per drug in 2005 dollars. These estimates are based on secret industry data and questionable accounting that bears little resemblance to common household accounting.
The pharmaceutical industry is not alone. As oil and energy prices and profits have soared over the last decade and a half, the petroleum industry has sought to replicate the evident political success of the pharmaceutical industry. In 2008, as oil and gasoline prices soared, ExxonMobil launched an advertising and public relations campaign featuring profiles of photogenic scientists working on new energy technologies at the giant oil company, seeking to portray the company as a public spirited research lab staffed by lovable nerds. So too, British Petroleum (BP) has a “BP Beyond Petroleum” advertising and public relations campaign focussing on BP’s obviously token investments in alternative energy research. In fact, many studies show minimal investment in research and development by major oil companies. Remarkably, they seem to have few concerns about “Peak Oil” despite invoking the alleged rising cost of oil exploration and recovery to explain rising prices (and profits — how does that work?). The research that produced the reported breakthroughs in hydraulic fracturing, “fracking,” were funded by the US Department of Energy.
In the “technology” industry, it is common to find that the core technology was researched and developed at public expense, often at a university such as Stanford, with funding from government funding agencies such as DARPA (the Defense Advanced Research Projects Agency). Nonetheless, the high tech companies that license the technology, often from universities under very favorable terms due to the obscure Bayh-Dole Act, often strongly imply they did the original research and development. Their public relations and advertising activities often portray these firms as public spirited research labs full of lovable nerds with a sales and marketing department tacked on as an afterthought, which is grossly inaccurate even for the few companies such as Hewlett-Packard that do invest heavily in scientific research.
Successful research with perceived large benefits, “breakthroughs” such as penicillin, is very popular in the United States and probably around the world. It is also very long term (usually over five years), risky (high failure rate — often claimed to be eighty to ninety percent), and sometimes very expensive, e.g. the Manhattan Project at $20 billion in current dollars. Thus many businesses avoid research in practice, leaving the heavy lifting to government programs. But the popularity of successful research means that businesses that do little research, whose business model may be the commercialization of proven technology rather than true research or even something much farther removed from research, seek to wrap other agendas in the mantle of science. The major mass media in the United States, which receives millions, if not billions, of dollars in advertising revenues from these campaigns, rarely points out the factual problems with these claims in ostensibly objective news articles.
It is not surprising that many STEM students believe there is a strong industry appetite for freshly minted Ph.D.’s given these numerous advertising and public relations campaigns by many businesses.
The General and the Specific
It is impossible to understand the many claims and counter-claims about STEM jobs and careers without drawing a careful distinction between general claims such as “we can’t find Ph.D.’s” and specific claims such as “we can’t find Ph.D.’s in finance with a thesis on the application of stochastic differential equations to the valuation of collateralized debt obligations (CDO’s) in Sweden and at least three years of paid industry experience at the trading desk of a major European bank” :-). Businesses do seek Ph.D.’s with very specialized skills, usually with additional “industry” experience that freshly minted Ph.D.’s rarely possess. The vast majority of Ph.D.’s have not done research in the “hot” fields — meaning specialized sub-fields of broadly defined disciplines such as Computer Science or Physics. This is inherent to the high risk, which means “high failure rate,” of scientific research. Most research never succeeds; most research never becomes “hot.”
Businesses are generally interested in the research that produced not merely academic/scientific success but immediate business success. Anecdotally, only 10-20 percent of research is successful as defined by academics and scientists which often means peer-reviewed and widely cited by other academics and scientists. Many genuine scientific breakthroughs were not widely cited until decades or even centuries after publication :-). Industry is interested in the subset of successful research that may be immediately profitable, or at least investors can be convinced that it may be immediately profitable. So, for example, the recent discovery of the Higgs Boson at CERN has produced a big giant yawn from the business community, whereas the World Wide Web, originally a tool for sharing research articles at CERN, produced a frenzy in the business world.
Dennis Ritchie Couldn’t Code
One of the things that exasperated Chand John, and many other freshly minted Ph.D.’s over the years (the author included), is that his, by his account, extensive experience writing computer software, coding, starting at age eight and including, again by his account, a widely downloaded and presumably widely-used research program was frequently dismissed as not “real” coding experience by prospective employers. What does it mean to say someone knows how to code? A bright six year old can program a “Hello World” program or, indeed, more complex examples of software. When technology executives say they cannot find applicants who can “code,” what does this mean?
In practice, the industry definition of “coding” is remarkably difficult to clarify. It may really mean “coding” by people I happen to like, such as people like me who tell me what I want to hear, for example, how smart I am :-). This is not a problem only encountered by new college graduates and freshly minted Ph.D.’s. The history of the technology industry is littered with the failures of companies that for some reason decided that their successful, widely-used, even highly profitable flagship product was not “good code” and needed to be rewritten from scratch, which turned out to be very difficult, even impossible. The code empirically worked, was written by professional programmers (they were, in most cases, paid and paid well to write the code), and it even made lots of money, but, well, it was “not done right.” Joel Spolsky wrote a famous, widely read, widely ignored blog post on this striking phenomenon in the software industry: Things You Should Never Do.
Just because you are Dennis Ritchie and your resume says you invented the C programming language, wrote the first C compiler, and a good chunk of the original Unix operating system (and your resume is actually true), doesn’t mean you can “code”. After all, Dennis Ritchie was (ick) a “research programmer” at an airy-fairy research lab (Bell Labs). Not real coding at all. And besides, he probably never used Git (a popular version control system that is notoriously difficult to actually use) a day in his life.
Incidentally, the standard way to get around Spolsky’s article is to call “rewriting” code “refactoring” code which makes it all OK, in fact a mandated part of many versions of the Agile software development methodology.
Opening Doors, Including Some You Should Not Go Through
In general, a Ph.D. closes many doors due to the widespread prejudice against Ph.D.’s in industry. But, it is true that a Ph.D., especially in some narrow, specialized areas, can open some doors that would otherwise be closed. A Ph.D. can help getting a job at a genuine corporate research lab such as HP Labs. It can help getting a job at some aerospace and defense contractors or government agencies that pay competitive, non grad-school, wages. A STEM Ph.D. will often help convince employers that you have strong mathematical skills, even if you don’t, even though your coding experience, which may be extensive, may be dismissed as “not real.” At present, around one percent of software development jobs could be described as highly mathematical, going beyond basic arithmetic and bookkeeping. In my opinion, this is likely to increase as we learn to use advanced mathematics to exploit the enormous processing power of present day and future computers. Many of the current highly mathematical jobs are concentrated in probability and statistics, often so-called “data science” where Chand John also unsuccessfully sought work.
In his article, Chand John expressed some frustration with his unsuccessful, and apparently lengthy, interview process with Palantir, the murky start-up funded by PayPal founder Peter Thiel and In-Q-Tel, the venture capital arm of the Central Intelligence Agency (CIA). The Palantir, by the way, are the cursed seeing stones used by the diabolical Dark Lord Sauron in J.R.R. Tolkien’s Lord of the Rings trilogy :-).
“They are not all accounted for, the lost seeing stones. We do not know who else may be watching.”
—Gandalf to Saruman, in The Fellowship of the Ring
The government, especially “members of the intelligence community,” has some truly strange projects, the true nature of which is disguised by secrecy and perhaps active disinformation. Some of these projects employ STEM Ph.D.’s. and pay well. One should think long and hard before going through some of these doors if they should open.
There are also some companies and projects that are based on grossly unrealistic ideas about what mathematics or science can achieve, especially on the short time scale, often a business quarter or two, favored by private industry. These do seek and employ Ph.D.’s but often end badly, for example when the planned breakthrough doesn’t occur by the next quarter. These companies and projects often suffer from a magical picture of mathematics or science similar to that portrayed in much popular science and in the more realistic-seeming science fiction (think The Hunt for Red October instead of Iron Man).
Industry does express an interest in hiring STEM Ph.D.’s — loudly in certain contexts. In most cases, this interest appears to be feigned, primarily for political lobbying purposes. Whether this claimed interest is a deceit or a sincere delusion, most fresh STEM Ph.D.’s will find the Ph.D. more of a hindrance than a help when actually applying for a job in industry. Especially in the “technology” industry, in most cases they would have been better to stop at a bachelors or masters level. The Ph.D. often results in a lower salary than would have been possible if they got a job immediately after college or, in some cases, a master’s degree and worked for the same time period as their Ph.D.. Further, industry jobs for new college graduates in “technology” usually pay much better than grad school teaching assistant or research assistant stipends.
That said, a minority of employers are sincere in seeking STEM Ph.D.’s, either because they think they are smart or have other useful general characteristics, or, more often, because the Ph.D.’s have narrowly defined specific skills they seek. A Ph.D. may open some doors that would remain closed with only a B.S. or M.S.. On the other hand, a typical Ph.D. will close more doors than it opens due to a widespread prejudice against Ph.D.’s. Note that I don’t agree with this prejudice (after all I have a Ph.D.) but it is a part of the real world today.
© 2013 John F. McGowan
About the Author
John F. McGowan, Ph.D. solves problems using mathematics and mathematical software, including developing video compression and speech recognition technologies. He has extensive experience developing software in C, C++, Visual Basic, Mathematica, MATLAB, and many other programming languages. He is probably best known for his AVI Overview, an Internet FAQ (Frequently Asked Questions) on the Microsoft AVI (Audio Video Interleave) file format. He has worked as a contractor at NASA Ames Research Center involved in the research and development of image and video processing algorithms and technology. He has published articles on the origin and evolution of life, the exploration of Mars (anticipating the discovery of methane on Mars), and cheap access to space. He has a Ph.D. in physics from the University of Illinois at Urbana-Champaign and a B.S. in physics from the California Institute of Technology (Caltech). He can be reached at [email protected].
I received some comments questioning my evidence that industry and its allies either directly claim or strongly imply a desire for Ph.D.’s, and newly minted Ph.D.’s at that. Here is a small sub-sample of “staple a green card to a Ph.D.” news articles, op-eds, and related items.
Note the confusing blurring of categories in many of the articles. Are we talking about green cards for Ph.D.’s, for Masters degrees, “advanced graduate degrees” presumably encompassing both Ph.D.’s and Master degrees, or undergraduate STEM diplomas such as a B.S. in Physics?
Links from Google Search for “staple green card to phd” on Jan 7, 2014
which contains this quote:
Presumed Republican presidential nominee Mitt Romney also supports liberalization of immigration rules for tech specialists. “I’d staple a green card to the diploma of anybody who’s got a degree in math, science, a Master’s degree, PhD,” Romney said during a debate in November. (2012)
which contains this question and answer (from Mitt Romney):
I like legal immigration; let business determine visas
Q: In 2008, you said you favored allowing American companies to hire more skilled foreign workers. With unemployment at 9.1%, are you still for importing more foreign labor?
A: Well, of course not. We’re not looking to bring people in for jobs that can be done by Americans. But at the same time, we want to make sure that America welcomes the best and brightest in the world. If someone comes here and gets a PhD in physics, that’s the person I’d like to staple a green card to their diploma, rather than saying to them to go home. I want the best & brightest to be metered into the country based upon the needs of our employment sector & create jobs by bringing technology and innovation that comes from people around the world. I like legal immigration I’d have the number of visas that we give to people here that come here legally, determined in part by the needs of our employment community. But we have to secure our border and crack down on those that bring folks here and hire here illegally.
A Recent Op-Ed from realclearpolitics.com web site (not from Google search)
which contains this paragraph:
The way we currently treat highly skilled immigrants needs to be rethought too: We bring motivated, ambitious foreigners into the United States to earn graduate degrees, and then we send them out of the country after they’ve graduated, often against their wishes, just when they are at the point when they could contribute to the economy and create jobs. Any immigrant who earns a graduate degree in America in the physical sciences, engineering, high technology, or mathematics should be granted permanent residency. Highly skilled immigrants are job creators; one in four engineering and tech businesses founded between 1995 and 2005 had at least one immigrant founder. Reforming our immigration laws to admit more highly skilled immigrants is a great jobs program, and should be marketed and thought of as such.
Note: most readers would probably interpret “graduate degrees” as synonymous with Ph.D. though it could include Masters Degrees (for example).
Compete America: The Alliance for a Competitive Workforce
Some Ph.D. related quotes from the Compete America web site:
In many critical disciplines, particularly in the STEM — science, technology, engineering and mathematics — fields, a significant number of the post-graduate degrees at U.S. universities are awarded to foreign nationals. For example, according to the American Association of Engineering Societies, foreign-born students received 42 percent of U.S. engineering master’s and 53 percent of U.S. engineering Ph.D.s nationwide in the 2009-2010 academic year.
It is counterproductive for the United States to train foreign scientists and engineers and then send them home to compete against American businesses.
The U.S. visa system for highly educated professionals must be permanently fixed so that America can retain these talented minds for U.S. economic growth and job creation.
Despite the economic downturn and layoffs that have impacted many sectors of the U.S. economy, the brightest scientists, researchers, innovators and engineers in the world – whether American or foreign-born – will always be in demand and will always drive economic growth and job creation.
However, arbitrarily low visa quotas and massive backlogs in the system plague our employment- based visa process.
A study by the Kauffman Foundation estimated that more than 1,000,000 highly educated people are currently waiting for green cards, yet only 140,000 are available annually for them.
It’s time for Congress to fix the broken immigration system by enacting legislation that supports the growth of America’s innovation economy and provides U.S. employers, research institutions and universities access to the world’s top talent.
Compete America supports legislation that would provide a permanent fix to the broken visa system for highly educated foreign professionals – including:
(NOTE THE NEXT PARAGRAPH)
The end of arbitrary obstacles to employment and permanent residency for foreign-born master’s and Ph.D. graduates from U.S. universities
o A streamlined EB green card process
o A market-based H-1B visa cap
Permanently fixing the H-1B visa and EB green card programs is critical to our continued economic
strength, future job creation and ability to compete in the world marketplace.
America Competes represents industry:
The above quotes strongly imply, if not explicitly claim, a strong industry desire for Ph.D.’s., and indeed recent Ph.D.’s.
Note: I am not taking a position on immigration policy or the H1-B visa, I am only showing by direct quotes that business leaders and their allies claim or seem to claim a strong need for Ph.D.’s and indeed recent Ph.D.’s. in STEM fields.
I may find it hard to believe he can’t code, when he started the whole process by his research
These constant calls for more education just kill me, even in tech fields. where is the overwhelming evidence that in aggregate, our domestic workforce is already overeducated for the available jobs!
The BLS has found that employers ask for bachelors degrees twice as often as justified by the skills and knowledge required. Far too many BS/BA holders are underemployed. There is pretty solid evidence that for computer related fields, higher degrees confer little monetary or professional benefit. The oversupply of graduate degrees in STEM is well known. And why not, when there is a surfeit of BS holders there already?
Joel Spolsky may be right about not redoing your entire code base, but refactoring is vital in many cases. Legacy code is a curse that has crippled many software projects. Either because the code just can’t be extended the way management wants, or it is so hacked up you literally can’t figure out what is doing by looking at it. I’ve spent about 25% of my career reverse engineering godawful undocumented legacy code so that it can be fixed or modified. Like repairing a building without a blueprint, or a circuit board without a schematic. Sometimes it is better to start over. This is why I secretly envy “research programmers” like Dennis Ritchie, that got to do original designs . Far too much commercial programming is hack (kludge) oriented, needlessly reinventing the wheel, or trying to make legacy Model T software perform like a Mustang. Sometimes it just can’t be done without starting over.
Remember Y2K? It wasn’t because the original programmers were stupid or inept, as widely portrayed. But that computing resources were extremely limited and hugely expensive. But those programs should have been revamped, or even redone (Spolsky notwithstanding), well before the year 2000. I doubt any of the programmers expected their code to still be running 30 years later.