One tidbit that I found interesting was Sachs’ estimation of the required investment in research and development in sustainable technology in order to address the issues in climate change, water and food security, disease, et al. that the book covered. This required investment was set at 0.2% of GNP of the developed world. By his calculations, which were likely made in 2007, this amount is equal to 70 billion dollars. While his estimation methodology was unfortunately not clearly disclosed, lets run with it for the time being.

By comparison, the 2007 NSF budget was 5.9 G$ (source: NSF.gov), the NIH budget was 29 G$ (source: NIH.gov, and the Department of Defense research budget was 72 G$ (source: Defenselink). Exclusive of other smaller research programs, such as the Department of Energy research programs and NASA, this represents around 107 G$ in funded research. By comparison, the 2007 cost of the Iraq War (specifically excluding Afghanistan and other “War on Terror” expenditures) was 123 G$ (source: CBO)

The implication of these numbers is that it appears to be quite feasible to fund the required research and development in sustainable technology, perhaps even unilaterally. Further, investing that 70 G$ above and beyond current research funding would at least partially address the “green jobs” development that President-elect Obama has been advocating. While some portion of this money would go to academic grants, some non-trivial portion of the funding should be made available in a SBIR/STTR program. Additionally, some technology-driven small business development funds, something like an angel investment fund for sustainable technology, would encourage green job growth while meeting these sustainable technology R&D goals.

It also seems reasonable that such an initiative would incentivize growth in the science and engineering fields. Despite a lot of ado about the need to train more scientists and engineers, many technical fields are and have been producing a glut of students with advanced degrees (as Daniel Greenberg and various industry publications, such as Physics Today and C&E News, have pointed out.) It also goes without saying that once a technical professional transitions from science and engineering to business or law, they do not return – the disparity in pay scales is generally insurmountable, at least in my experience. Driving the demand for technical professionals with these R&D incentives could absorb at least part of this glut, preventing the loss of the most talented individuals from the technical fields.

Above all, the goal of this funding is worthwhile: many of the challenges facing the world have solutions that are either in whole or in part technological. While I am always skeptical of throwing money at problems, I find a world of difference between things like funding direct food aid to developing countries and funding research in drylands agriculture and permaculture in order to improve cropland yields while reversing soil degradation. The former is simply spreading the wealth while the latter so very clearly creating new wealth for the entire world. When these Millennium goals are met, political scientists and economists argue that conflicts over scarce resources in the developing world will dwindle. It seems reasonable , then, that the best investment in foreign aid and development should start here. Hopefully, President-elect Obama’s advisors will encourage him to champion this opportunity to make such an investment in sustainable technology.

The existence of technology will no more prevent religious fundamentalists of either the Muslim or Christian stripe from acting irrationally, anymore than the technology of the first Industrial Revolution caused an expansion of Enlightenment rationality as the 19th century positivists believed.

My chief complaint about this book is that there is no insight offered into the hygiene of the data used. This may seem to be a nitpicky sort of complaint, given that the book was written for the general audience, but since the book focused on case studies where hardcore statistical analysis yielded interesting and useful results, I think that an understanding of how you make sure your data is clean would be helpful. An example of this is the chapter on direct instruction (DI). I don’t question the studies or their data that show that DI is an effective method, since they have been almost certainly been peer reviewed by excellent statisticians. I do want to point out a couple of different types of errors that often exist in otherwise clean datasets. It turns out, as I will explain below, that one of these errors and the omission of any information about how the data used in this analysis were collected, is highly relevant to another very real-world type of problem – how your bosses decide whether or not you get a raise.

The first, and most common error in an experimental design is an uncontrolled variable. This happens when you have measurements that you assume to be varying as a function of your controlled variables, but in actuality are most highly correlated with a variable that you’re not controlling. This sounds like an easy thing to avoid, but in reality, systems are messy and you will not always design your experiment correctly the first time around. You will usually miss some things that affect your experiment, due to unfamiliarity if nothing else. An example of this kind of error would be measuring plant growth as a function of days of sunshine, but not controlling for the amount of water the plants receive. Most savvy experimental designs take this into account by limiting what can be inferred by the correlation – this is what people mean when they say that correlation doesn’t imply causation. They’re being careful about what they can and cannot infer from the data. Oddly, I don’t recall Ayres explicitly mentioning this issue in any of the case studies, but we can assume that any obvious errors in this regard would have been quickly picked out by the reviewers of the papers from which the case studies were drawn.

The second error, and the one that I think is actually most relevant is what I call proxy error. This is error or bias that enters when you are forced to measure a proxy for a property, rather than the property itself. This is not uncommon at all – most things that are interesting are also not directly measurable. A great example of this is measuring the quality of books. People have different tastes, and different opinions about what they value in a book. If its fiction, some people might prefer stories with gripping, twisty plots, while others might prefer interesting characters that can be identified with. Its important for the publishing industry to have an objective measure of how good a book is, so they can try to maximize their earnings by publishing what people want to read. There is thus an implicit assumption made that books that have high sales are good books. This is true, from the perspective of a business, make no mistake, but from the perspective of a book-lover this is clearly not the case, as anyone who has read The Da Vinci Code or the latest Laurell Hamilton novel can attest. What we see here is a proxy error, in that we believe that something which appeals to everyone must have high quality. By that same token, we might argue that McDonald’s is higher quality than your favorite local fine dining restaurant. In those terms, we can quickly see the absurdity.

In the case of the chapter on direct instruction in Super Crunchers, Ayres comes to the conclusion that DI has marked efficacy in both imparting core skills and improving creativity. What I want to point out is that there are serious issues with drawing that unequivocal conclusion on the basis of the “super crunching” that was done, at least according to what was reported in the book. The first issue is that there is no precise definition of creativity and thus it cannot be quantitatively measured. This means that in absence of a clear discussion about what was meant by creativity and how it was measured, I don’t think a conclusion can be drawn. Additionally, the lack of a precise definition means that you have to construct proxies that do have precise definitions and are measurable. Now you have two problems to solve: you have to construct a proxy and then measure it in a statistically reliable way.

For creativity specifically, I’m aware of several proxies. When we interview potential candidates for a research scientist/engineer position, creativity is one of the key traits we probe for. We therefore not only have to construct reliable, repeatable, and robust proxies for creativity, we have to construct them such that they can be measured within the constraints of our interview process. We might thus draw the following conclusion: “If a candidate is a good brainstormer and can use that modality effectively to solve problems placed before her, we believe that she will likely be a highly creative individual.” This comes through in the process as “so-and-so had some really creative ideas and response to my questions, so I ranked her highly on creativity.”

The researchers who design the educational testing that generated the data on DI went through a similar, though probably much more in-depth and rigorous, thought process when deciding how to measure creativity. Their design criteria were also necessarily different. Rather than being constrained to an arbitrary process like our recruiting process, their design was likely focused much more heavily on reliability and repeatability. We sacrifice some of those, knowing that our process may reject some qualified candidates as a result of that sacrifice, because the cost of doing scientifically reliable and repeatable measurement is too great for the payback on the margin.

But no matter how careful your design is, there are simply some cases where any proxy will not accurately reflect the property for which they are proxy. In the case of sociometric and psychometric testing, that error is itself unmeasurable and unbounded. You might be able gain some understanding of the error by constructing several different proxies and measuring them as well, but my sense is that this is rarely done in a single study. And in this particular case, the danger is that the children being tested had learned to do well on the tests rather than actually having learned to be creative. In other words, the children only appeared to be creative because they had altered their behaviors to meet the expectations of the researchers.

This subject is one that is highly relevant to the general audience, however, because of how often it affects peoples’ lives. It has become particularly interesting to me as we have developed better metrics for recruiting and rewarding researchers and managing our project portfolio. Setting metrics by which a group of PhD scientists and engineers are measured and rewarded is tricky, because of the richness in the strategies we’ll employ to maximize our benefit within the metrics given. It is therefore critical that the metrics are proxies for activities and outcomes that are truly valuable to the organization, since they will likely profoundly alter behaviors. Constructing these metrics is a difficult job and I believe that feedback on the metrics themselves is critical in getting it right. If people in general understood these connections and subtleties of constructing effective human metrics, then a lot of the pathos that exists within the business world around the subject would disappear. While this would be terrible for Scott Adams, since he’d lose a lot of material for Dilbert, I think we’d handle the loss just fine.

It’s also worth noting that my interest in this subject probably sensitized me to the proxy error that could be present in this data. Its possible, although I think unlikely, that the study accounted for this somehow. Its also very likely that I missed potential proxy errors in other chapters and that I’d probably not be competent to discuss them even if I did notice them. That doesn’t mean that they aren’t there, of course. Before anyone accuses me of picking on one part of the book unfairly, let me assure you that my criticisms stem only from my wish that he’d treated this subject in a little more depth, which is possibly an unreasonable request for a general audience book. Regardless, I think that if we’re teaching people to think statistically, as Ayres advocates in the book, we need to also teach them the first half of the problem, which is good experimental design, at the same time. After all, the statistics are no better than the data from which they are computed.