In my mind, this is a clear argument for open access scientific journals. While it would be possible to corrupt the process of peer review in an open access journal, it would be pretty difficult to hide the fact for very long.

I’m going to talk about how that love affair manifests, how you manage it, and how you move past it into a wonderful relationship that will hopefully last many years, bring in revenue, and perhaps even change the world.

What do I mean by “technology” here? It might mean the latest programming language hotness or it might mean the latest thing in biofuels or in carbon sequestration. It could mean the latest gadget for helping you build a Twitter-enabled flowerpot. Technology, as a functional definition, can be said to be the use of knowledge to interact with our environment – both our physical environment and our social environment. How can you not fall in love with something that is so fundamental?

The problem comes when we expect everything from some bit of technology. You find it and it looks cool. Novel. You can do things with it that you’ve never done before. So you start using it and soon you want to use it to do some other things too. Things that are important, but maybe things that the technology really wasn’t supposed to do. Like using the butt-end of a screwdriver to pound in a nail. Or trying to write a fast Fourier transform in pure Java. You can probably think of your own examples.

The main thing that I do at my job is technology development. We see the seeds of a new technology – sometimes they’re at the “little black speck in a paper envelope” stage. Sometimes they’re at the “tiny sprout” stage or even at the “seedling ready to transplant” stage. We either develop the technology ourselves or partner with the people who are currently developing it, when we think there is a good fit. We take it, grow it until its ready to bear fruit and then commercialize it. And sometimes, we see a technology that sets our hearts pounding. Our pupils dilate, our palms get sweaty – we fall in love. And that love is a beautiful thing, because it means that the technology is likely to be useful and that there is going to be an internal champion for it.

The problem comes a little ways into the relationship. Sure, your new love is a fun to take into the lab and gives you that sizzle when you’re down and dirty with it. You want to solve every problem with it. It’s a sexy beast! But, after a few months with it, your new love’s moved into your apartment and you’d really like for him to – ya know, pay some rent? Plus, he drips on the toilet seat occasionally and leaves his dirty dishes all over the place. But still, you can’t let him go because, well…. he’s just so good!

At some point, you have to make a decision, just like you do with every relationship. Maybe it was just a fling – you learned something from it and had a good time doing it, but it’s just not for you. Or maybe this one is worth keeping. Sure, he’s rough around the edges, but a little time and effort and he’ll be something to take home to Mama. Or at least to your C-office.

What makes the difference between the one you kick out of bed and the one you go steady with is how honest you are with yourself when you’re learning about the technology. If you don’t quickly learn what the limitations of the new technology are, your new love is bound to disappoint. And as thorough as you might be up front about the technical limitations, you’re going to have a second round of limitations that come up each time you try to take that technology to a new market. It’s just like taking that new girl you hooked up with a few months ago to an office party and watching her get hammered and puke into the ficus. Quickly, you learn that while she’s an excellent choice for the “wild night on the town” market, she’s perhaps not quite the one for the “impress your clients” market.

One way we keep track of our current thinking about a technology is with a chart like this:

This sort of bubble chart keeps track of where the technology falls in your portfolio in terms of strategic fit, development stage, and expected financial return. The size of the bubble indicates the financial return, typically either by “best N years” performance or “N years post launch” performance. The chart above shows some typical positions in a portfolio. You tend to want to have a decent scatter over the chart, but ultimately, if you’re a technology-driven company, you don’t want to have a lot of things sitting in the lower left hand corner. That means your putting a lot of resources into things that are farther out and will ultimately bring you no new revenue.

During the course of a love affair with a particular technology, you will sometimes find that its trajectory across the chart will look something like this:

This happens a lot more frequently than you might suspect. By the time people take a long hard look at what they’ve invited into their bed … lab, that is, they’ve realized that the technology just isn’t as shiny as it used to be. Back when things were good, you dumped a lot of resources into it. Lots of fancy dinners, flowers, jewelry. You put time and effort, maybe even a sizable fraction of your available development staff, into making this technology go. And for what?

Some of this is inevitable. You’re never going to know a priori what a technology’s domain of applicability is. And if you spend too much time trying to map that out oh-so-carefully before you start, you’ll never bring anything to market. Further, your initial estimates are just that. You’re going to overestimate its financial impact – except in those rare cases where you’ve underestimated it. You’re going to completely misjudge how close to commercialization the technology is. But ultimately, the goal is to move past the love affair into a stable relationship, where there is a clear path from technology to product.

The key to managing this transition is flexibility. Once you’ve made a decision to try the thing out, you must constantly be looking for the mismatches between your technology and the problems you’re looking to solve with it. At the same time, you need to be flexible enough to realize that while the new technology may not solve the particular problem you’d set in front of it, there may be a completely different problem that it is particularly adept at solving.

There’s a lot left unsaid here about technology development: commercialization strategies, product migration maps, technology adjacencies, market adjacencies. I’m not going to talk about those here. They’re important and you should be thinking about them, but they don’t help you deal with the emotions behind finding a new technology, rubbing the shiny off of it, and dealing with the commercialization endgame.

Do you have to be in a research and development group to follow this advice? Not at all. The advice holds for anyone who is prone to love affairs with technology. The specifics of methods and desired outcomes will change. Some people will try Ruby on Rails and ultimately find that sleek hottie too much for them, returning to the stable, quiet hippie in Django. Some folks will be looking for a greener energy source to put their activism into, take that one look at Clean Coal, realize that she’s just a two-bit whore and pass on to something that they know they can support while keeping their self-respect. At the end of the day, everyone will fall in love with a technology at least once. The trick is to be honest enough to know when its not the technology you’re seeking and flexible enough to let it solve the problems at which it is best.

While I’m very interested in scientific investigations of the traditional pharmacopia, such as what the Bent Creek Institute is doing here in town – i.e. lots of extractions and chromatography – I’m concerned that mainstream emphasis on unscientific treatments will lead to a lot more deaths like this one.

One tidbit that I found interesting was Sachs’ estimation of the required investment in research and development in sustainable technology in order to address the issues in climate change, water and food security, disease, et al. that the book covered. This required investment was set at 0.2% of GNP of the developed world. By his calculations, which were likely made in 2007, this amount is equal to 70 billion dollars. While his estimation methodology was unfortunately not clearly disclosed, lets run with it for the time being.

By comparison, the 2007 NSF budget was 5.9 G$ (source: NSF.gov), the NIH budget was 29 G$ (source: NIH.gov, and the Department of Defense research budget was 72 G$ (source: Defenselink). Exclusive of other smaller research programs, such as the Department of Energy research programs and NASA, this represents around 107 G$ in funded research. By comparison, the 2007 cost of the Iraq War (specifically excluding Afghanistan and other “War on Terror” expenditures) was 123 G$ (source: CBO)

The implication of these numbers is that it appears to be quite feasible to fund the required research and development in sustainable technology, perhaps even unilaterally. Further, investing that 70 G$ above and beyond current research funding would at least partially address the “green jobs” development that President-elect Obama has been advocating. While some portion of this money would go to academic grants, some non-trivial portion of the funding should be made available in a SBIR/STTR program. Additionally, some technology-driven small business development funds, something like an angel investment fund for sustainable technology, would encourage green job growth while meeting these sustainable technology R&D goals.

It also seems reasonable that such an initiative would incentivize growth in the science and engineering fields. Despite a lot of ado about the need to train more scientists and engineers, many technical fields are and have been producing a glut of students with advanced degrees (as Daniel Greenberg and various industry publications, such as Physics Today and C&E News, have pointed out.) It also goes without saying that once a technical professional transitions from science and engineering to business or law, they do not return – the disparity in pay scales is generally insurmountable, at least in my experience. Driving the demand for technical professionals with these R&D incentives could absorb at least part of this glut, preventing the loss of the most talented individuals from the technical fields.

Above all, the goal of this funding is worthwhile: many of the challenges facing the world have solutions that are either in whole or in part technological. While I am always skeptical of throwing money at problems, I find a world of difference between things like funding direct food aid to developing countries and funding research in drylands agriculture and permaculture in order to improve cropland yields while reversing soil degradation. The former is simply spreading the wealth while the latter so very clearly creating new wealth for the entire world. When these Millennium goals are met, political scientists and economists argue that conflicts over scarce resources in the developing world will dwindle. It seems reasonable , then, that the best investment in foreign aid and development should start here. Hopefully, President-elect Obama’s advisors will encourage him to champion this opportunity to make such an investment in sustainable technology.

This year, one of the most inspiring recipients of the MacArthur Fellowship is an agriculturalist named Will Allen. His non-profit, Growing Power, maintains an urban farm in Milwaukee, providing fresh vegetables to the residents of the distressed inner city there. Regular readers here will note that I have a strong interest in urban agriculture and small-lot permaculture, so it is especially rewarding to see the MacArthur Foundation take interest in the kind of project that Will Allen is leading.

The New York Times published a great article about a month back on Will Allen and Growing Power and MAKE magazine has the video of an interview with him.

Everybody needs to measure stuff. And whether you’re in a traditional business or a own a Web 2.0 startup or are just an average gal or guy, the need to measure things quickly and precisely has gotten a lot more intense in the past decade. You want to understand where your business is in the Long Tail or how “sticky” your website is or how much your coffee habit is costing you annually. And when I say precisely, I mean precisely in a, well, precise sense. To speak precisely, precise and accurate are not the same things. And this is the first thing to understand about measurement – a measurement is only valid when it is both sufficiently accurate and sufficiently precise.

Accuracy vs. precision

When you measure something accurately, your measurement gives you a number that is very close to the truth. You may not get the same number each time you make your measurement, but you know that its close to the actual value. When you measure something precisely, you’ll get close to the same result each time, but you may not be close to the actual answer. Ideally, we want our measurements to be both accurate and precise. In reality, most folks have a higher tolerance for lack of accuracy than they do for lack of precision. As long as the measurement is reasonably accurate, most people will settle for something that is off from the truth by a good bit so long as they get consistent answers from it. If you check your measuring cups in your kitchen drawers, you will find that they are pretty precise. Fill your 1/4 cup measure up 4 times and dump it in your 1 cup measure and it will fill it up exactly. (Or at least it did on each of the three sets of measures I had in my kitchen.) Yet, I have no idea – nor do I care – if the cups are calibrated properly. Do they deliver exactly 1 cup? If you care about that kind of accuracy, you’ll probably be using a graduated cylinder, not a plastic measuring cup. For most of us, the fact that the cups are precise is more important.

Does our tolerance for inaccuracy seem surprising? If you use Google Analytics to track your website stats, it shouldn’t be. Google can’t know, accurately, how many unique visitors actually visited your site. How can they? Even though they set a cookie to track your visitors, a lot of folks using Firefox will only accept cookies for that session, thus preventing Google from counting them over multiple visits. I do essentially the same thing with Omniweb. A lot of folks using IE will occasionally flush all of their cookies as a privacy measure. Each time the Google Analytics cookie for your site gets deleted, that user looks like a new user to Google. This means your unique visitor count is artificially high, as is your percentage of new users visiting your site. But, really, it doesn’t matter. You don’t care how accurate that number is, because whether you have 570 or 450 unique visitors per day isn’t as important as the trend. Is that number going up or down? Is it higher on Saturday mornings or weekday nights? As long as the measurement is precise, then those trends can be analyzed meaningfully.

Now we know what makes a measurement valid, and we understand that a large fraction of the time, we don’t need as much accuracy as we need precision. While I didn’t explicitly talk about it, it’s important to note that validity is predicated only upon sufficient accuracy and precision. Your car’s fuel gauge is neither terribly accurate nor terribly precise, but it represents a valid measurement because it gives you the data with sufficient accuracy and precision to keep you from running out of gas.

There are four other things to keep in mind about a measurement, which I’ll call the Four ‘R’s: Relevance, Range, Resolution, and Reproducibility.

Relevance

Relevance in measurement is both subtle and obvious: you want your measurement to give you the information you need. Obvious, right? If the answer is “you have a full tank of gas” or “1 cup of milk,” then relevance seems to be trivial. We don’t always ask such easy questions, though. A lot of the things we measure aren’t things we really care about, but are instead things we believe are highly correlated to the things we care about. If you choose a proxy to measure that isn’t very relevant to the answers you need, then you can accumulate a lot of data that is useless to you. This isn’t always an easy problem to solve.

One example of the difficulty of make a relevant measurement is SAT scores. The SAT is a measurement that we believe tells us something about how a student will perform in college. We can’t measure “successful in college” – at least not a priori, so ETS has made a test that some people think is an excellent proxy for measuring future success in college. Whether or not it is actually relevant is a matter of some debate currently.

Most college grads can name a buddy from their college days with excellent SAT scores who flunked out early on, due to boredom or irresponsibility. We might infer from this observation that discipline and work ethic is more highly correlated with college success than SAT score. If this were true, it would be a much more relevant measurement. To prove this hypothesis, though, we’d have to actually measure a student’s discipline. That’s a hard problem, one that doesn’t lend itself to easy measurement. Not to say that colleges don’t try – I recall having to include letters of recommendation for my college applications, which presumably were complimentary of my excellent academic discipline, stellar study habits, and good dental hygiene. Since, when I entered college, I only had the good dental hygiene part going for me, I expect that most colleges recognize that this measurement is neither precise nor accurate, despite its relevance to the answer we seek.

Range and Resolution

Range and resolution are the properties of measurement that are the easiest to explain and understand. Range is the distance between the highest and lowest value your measurement technique will register. Resolution is how many levels of distinction there are within your range. Your desk ruler has a range of 12″, or about 30.5 cm, and a resolution of 1/16″, or about 1 mm. If you want to measure out 6 yards of a geotextile sheet for your garden, you could do that with the ruler, but you’d certainly agree that a tape measure would be a better tool. With some measurements, though, making sure you have a method with sufficient range is a lot more critical. This is why your meat thermometer and your household medical thermometer are different instruments, with different means of measuring temperature. By the same token, the resolution of your meat thermometer is a lot lower. You don’t really need 0.1 F resolution on your meat thermometer, but when taking your temperature to determine if you have a fever, that resolution is important.

Reproducibility

A measurement’s reproducibility is different from its precision, even though the two are easily confused. One way to think of it is that if you measured something ten times, the variance in your numbers would be represent the precision of the measurement. If ten people measured it once, the variance in the measurement would represent its reproducibility. What this means is that if the measurement technique is easy to “do correctly,” then the measurement is reproducible. Conversely, it is possible to have a highly precise measurement that is not very reproducible, because of the difficulty in making the measurement. This is frequently the case with a lot of microscopical measurements, as variations in sample preparation and operator technique can affect the images before they’re measured. If you’re trying to determine whether or not a tissue biopsy contains cancerous cells by looking at a stained specimen, getting this right is critical.

Reproducibility is an issue with a lot of qualitative or semi-quantitative measurements. When I see in a recipe “fry onions until golden-brown,” I have to make a measurement with my eyes of the color of the onions in my pan as they are frying. I know from experience, though, that my idea of “golden-brown” means a lot more cooking than it does for many other people. The human visual system, it is safe to say, does not represent a very reproducible measurement. It works well enough, though, so I don’t expect that many recipes will include spectrophotometer data for accurate, precise and reproducible measurements of onion color anytime in the near future.

The key to making a measurement reproducible is to have a clear process for making the measurement. In my field, there are standards bodies that publish volumes and volumes of proper procedures for making measurements such as the stiffness of a plastic (ASTM D6272) or how well something resists burning. (ISO 11925) For the example I gave above about biopsied tissue, there are standard staining protocols, that ensure that the proper stains are used and at the appropriate levels. But you probably understand this concept instinctively – if you always level off a measuring spoon or hold a measuring cup up so that the liquid is at eye level, you’re following a standard procedure that will make your measurements reproducible.

Measuring things

So, what now? If you’ve gotten this far, you’re at least somewhat interested in the topic, since I am no J. K. Rowling. Thinking about measurement can be terribly addictive – at least, it is to me. Have you ever wondered how they measure inflation? While you’ve probably heard of the Consumer Price Index, the details of the measurement might surprise you. More practically, you might also be interested in combing your customer data for indications of whether you’re doing the right things in your business. You might want to measure your website traffic to determine whether your new web ad campaign is giving you a good return. If you own a restaurant, you may want to measure how effective your menu is. In any case, good measurement is the key to getting good answers in a lot of fields and understanding the mechanics of making a measurement is part of good measurement.

[I]f we’re going to avert ecological destruction, we need to to not only do things differently, we need to do different things.

What he’s saying here is something that I’ve pointed out to my colleagues in the innovation community: sustainability is not about making things with less stuff, or that last longer, or that aren’t toxic, or even that can be infinitely cradle-to-cradle recycled. Sustainability requires us to invent things that make it possible to live more sustainably. If the things, the stuff, that we have and use make it easier to live sustainable lives than to not do so, then we will live sustainably.

Its not an easy problem to solve, for the same reason that truly groundbreaking innovation is not easy. It is pretty straightforward to imagine a novel solution for a market that already exists. It is much harder to invent a new market. I think that the kinds of products that will help people live sustainably are products for a market that doesn’t exist yet. Our business strategists don’t know how to value them, so our market analysts can’t compute a return on investment, so no investment is made. And truthfully, our scientists and engineers don’t always have the global perspective necessary to understand what types of solutions are necessary.

The point of Steffen’s article was to underline the importance of community in making these changes in our systems. I think that it is also important to understand the systems themselves. As we grow in our understanding the network of interactions and dependencies in our economy and our society, this understanding will allow us to break out of unsustainable patterns and replace them with ones that are equally understood, but are sustainable to the best of our knowledge. And because we’ll be building from a base of understanding, we’ll be able to look at them in a rational fashion 40 years from now when we understand the ways in which the new patterns are not sustainable.

It may be that at first, these more-sustainable patterns will be obvious. Things that folks like Steffen have been telling us for years, like community gardening, reducing sprawl, and increasing bike transport. But as with everything else, the low-hanging fruits will be quickly exhausted. At that point, progress will only be made by deeper understanding. It will be interesting to see how the tools for gaining that understanding develop.

My chief complaint about this book is that there is no insight offered into the hygiene of the data used. This may seem to be a nitpicky sort of complaint, given that the book was written for the general audience, but since the book focused on case studies where hardcore statistical analysis yielded interesting and useful results, I think that an understanding of how you make sure your data is clean would be helpful. An example of this is the chapter on direct instruction (DI). I don’t question the studies or their data that show that DI is an effective method, since they have been almost certainly been peer reviewed by excellent statisticians. I do want to point out a couple of different types of errors that often exist in otherwise clean datasets. It turns out, as I will explain below, that one of these errors and the omission of any information about how the data used in this analysis were collected, is highly relevant to another very real-world type of problem – how your bosses decide whether or not you get a raise.

The first, and most common error in an experimental design is an uncontrolled variable. This happens when you have measurements that you assume to be varying as a function of your controlled variables, but in actuality are most highly correlated with a variable that you’re not controlling. This sounds like an easy thing to avoid, but in reality, systems are messy and you will not always design your experiment correctly the first time around. You will usually miss some things that affect your experiment, due to unfamiliarity if nothing else. An example of this kind of error would be measuring plant growth as a function of days of sunshine, but not controlling for the amount of water the plants receive. Most savvy experimental designs take this into account by limiting what can be inferred by the correlation – this is what people mean when they say that correlation doesn’t imply causation. They’re being careful about what they can and cannot infer from the data. Oddly, I don’t recall Ayres explicitly mentioning this issue in any of the case studies, but we can assume that any obvious errors in this regard would have been quickly picked out by the reviewers of the papers from which the case studies were drawn.

The second error, and the one that I think is actually most relevant is what I call proxy error. This is error or bias that enters when you are forced to measure a proxy for a property, rather than the property itself. This is not uncommon at all – most things that are interesting are also not directly measurable. A great example of this is measuring the quality of books. People have different tastes, and different opinions about what they value in a book. If its fiction, some people might prefer stories with gripping, twisty plots, while others might prefer interesting characters that can be identified with. Its important for the publishing industry to have an objective measure of how good a book is, so they can try to maximize their earnings by publishing what people want to read. There is thus an implicit assumption made that books that have high sales are good books. This is true, from the perspective of a business, make no mistake, but from the perspective of a book-lover this is clearly not the case, as anyone who has read The Da Vinci Code or the latest Laurell Hamilton novel can attest. What we see here is a proxy error, in that we believe that something which appeals to everyone must have high quality. By that same token, we might argue that McDonald’s is higher quality than your favorite local fine dining restaurant. In those terms, we can quickly see the absurdity.

In the case of the chapter on direct instruction in Super Crunchers, Ayres comes to the conclusion that DI has marked efficacy in both imparting core skills and improving creativity. What I want to point out is that there are serious issues with drawing that unequivocal conclusion on the basis of the “super crunching” that was done, at least according to what was reported in the book. The first issue is that there is no precise definition of creativity and thus it cannot be quantitatively measured. This means that in absence of a clear discussion about what was meant by creativity and how it was measured, I don’t think a conclusion can be drawn. Additionally, the lack of a precise definition means that you have to construct proxies that do have precise definitions and are measurable. Now you have two problems to solve: you have to construct a proxy and then measure it in a statistically reliable way.

For creativity specifically, I’m aware of several proxies. When we interview potential candidates for a research scientist/engineer position, creativity is one of the key traits we probe for. We therefore not only have to construct reliable, repeatable, and robust proxies for creativity, we have to construct them such that they can be measured within the constraints of our interview process. We might thus draw the following conclusion: “If a candidate is a good brainstormer and can use that modality effectively to solve problems placed before her, we believe that she will likely be a highly creative individual.” This comes through in the process as “so-and-so had some really creative ideas and response to my questions, so I ranked her highly on creativity.”

The researchers who design the educational testing that generated the data on DI went through a similar, though probably much more in-depth and rigorous, thought process when deciding how to measure creativity. Their design criteria were also necessarily different. Rather than being constrained to an arbitrary process like our recruiting process, their design was likely focused much more heavily on reliability and repeatability. We sacrifice some of those, knowing that our process may reject some qualified candidates as a result of that sacrifice, because the cost of doing scientifically reliable and repeatable measurement is too great for the payback on the margin.

But no matter how careful your design is, there are simply some cases where any proxy will not accurately reflect the property for which they are proxy. In the case of sociometric and psychometric testing, that error is itself unmeasurable and unbounded. You might be able gain some understanding of the error by constructing several different proxies and measuring them as well, but my sense is that this is rarely done in a single study. And in this particular case, the danger is that the children being tested had learned to do well on the tests rather than actually having learned to be creative. In other words, the children only appeared to be creative because they had altered their behaviors to meet the expectations of the researchers.

This subject is one that is highly relevant to the general audience, however, because of how often it affects peoples’ lives. It has become particularly interesting to me as we have developed better metrics for recruiting and rewarding researchers and managing our project portfolio. Setting metrics by which a group of PhD scientists and engineers are measured and rewarded is tricky, because of the richness in the strategies we’ll employ to maximize our benefit within the metrics given. It is therefore critical that the metrics are proxies for activities and outcomes that are truly valuable to the organization, since they will likely profoundly alter behaviors. Constructing these metrics is a difficult job and I believe that feedback on the metrics themselves is critical in getting it right. If people in general understood these connections and subtleties of constructing effective human metrics, then a lot of the pathos that exists within the business world around the subject would disappear. While this would be terrible for Scott Adams, since he’d lose a lot of material for Dilbert, I think we’d handle the loss just fine.

It’s also worth noting that my interest in this subject probably sensitized me to the proxy error that could be present in this data. Its possible, although I think unlikely, that the study accounted for this somehow. Its also very likely that I missed potential proxy errors in other chapters and that I’d probably not be competent to discuss them even if I did notice them. That doesn’t mean that they aren’t there, of course. Before anyone accuses me of picking on one part of the book unfairly, let me assure you that my criticisms stem only from my wish that he’d treated this subject in a little more depth, which is possibly an unreasonable request for a general audience book. Regardless, I think that if we’re teaching people to think statistically, as Ayres advocates in the book, we need to also teach them the first half of the problem, which is good experimental design, at the same time. After all, the statistics are no better than the data from which they are computed.