I’m going to talk about how that love affair manifests, how you manage it, and how you move past it into a wonderful relationship that will hopefully last many years, bring in revenue, and perhaps even change the world.

What do I mean by “technology” here? It might mean the latest programming language hotness or it might mean the latest thing in biofuels or in carbon sequestration. It could mean the latest gadget for helping you build a Twitter-enabled flowerpot. Technology, as a functional definition, can be said to be the use of knowledge to interact with our environment – both our physical environment and our social environment. How can you not fall in love with something that is so fundamental?

The problem comes when we expect everything from some bit of technology. You find it and it looks cool. Novel. You can do things with it that you’ve never done before. So you start using it and soon you want to use it to do some other things too. Things that are important, but maybe things that the technology really wasn’t supposed to do. Like using the butt-end of a screwdriver to pound in a nail. Or trying to write a fast Fourier transform in pure Java. You can probably think of your own examples.

The main thing that I do at my job is technology development. We see the seeds of a new technology – sometimes they’re at the “little black speck in a paper envelope” stage. Sometimes they’re at the “tiny sprout” stage or even at the “seedling ready to transplant” stage. We either develop the technology ourselves or partner with the people who are currently developing it, when we think there is a good fit. We take it, grow it until its ready to bear fruit and then commercialize it. And sometimes, we see a technology that sets our hearts pounding. Our pupils dilate, our palms get sweaty – we fall in love. And that love is a beautiful thing, because it means that the technology is likely to be useful and that there is going to be an internal champion for it.

The problem comes a little ways into the relationship. Sure, your new love is a fun to take into the lab and gives you that sizzle when you’re down and dirty with it. You want to solve every problem with it. It’s a sexy beast! But, after a few months with it, your new love’s moved into your apartment and you’d really like for him to – ya know, pay some rent? Plus, he drips on the toilet seat occasionally and leaves his dirty dishes all over the place. But still, you can’t let him go because, well…. he’s just so good!

At some point, you have to make a decision, just like you do with every relationship. Maybe it was just a fling – you learned something from it and had a good time doing it, but it’s just not for you. Or maybe this one is worth keeping. Sure, he’s rough around the edges, but a little time and effort and he’ll be something to take home to Mama. Or at least to your C-office.

What makes the difference between the one you kick out of bed and the one you go steady with is how honest you are with yourself when you’re learning about the technology. If you don’t quickly learn what the limitations of the new technology are, your new love is bound to disappoint. And as thorough as you might be up front about the technical limitations, you’re going to have a second round of limitations that come up each time you try to take that technology to a new market. It’s just like taking that new girl you hooked up with a few months ago to an office party and watching her get hammered and puke into the ficus. Quickly, you learn that while she’s an excellent choice for the “wild night on the town” market, she’s perhaps not quite the one for the “impress your clients” market.

One way we keep track of our current thinking about a technology is with a chart like this:

This sort of bubble chart keeps track of where the technology falls in your portfolio in terms of strategic fit, development stage, and expected financial return. The size of the bubble indicates the financial return, typically either by “best N years” performance or “N years post launch” performance. The chart above shows some typical positions in a portfolio. You tend to want to have a decent scatter over the chart, but ultimately, if you’re a technology-driven company, you don’t want to have a lot of things sitting in the lower left hand corner. That means your putting a lot of resources into things that are farther out and will ultimately bring you no new revenue.

During the course of a love affair with a particular technology, you will sometimes find that its trajectory across the chart will look something like this:

This happens a lot more frequently than you might suspect. By the time people take a long hard look at what they’ve invited into their bed … lab, that is, they’ve realized that the technology just isn’t as shiny as it used to be. Back when things were good, you dumped a lot of resources into it. Lots of fancy dinners, flowers, jewelry. You put time and effort, maybe even a sizable fraction of your available development staff, into making this technology go. And for what?

Some of this is inevitable. You’re never going to know a priori what a technology’s domain of applicability is. And if you spend too much time trying to map that out oh-so-carefully before you start, you’ll never bring anything to market. Further, your initial estimates are just that. You’re going to overestimate its financial impact – except in those rare cases where you’ve underestimated it. You’re going to completely misjudge how close to commercialization the technology is. But ultimately, the goal is to move past the love affair into a stable relationship, where there is a clear path from technology to product.

The key to managing this transition is flexibility. Once you’ve made a decision to try the thing out, you must constantly be looking for the mismatches between your technology and the problems you’re looking to solve with it. At the same time, you need to be flexible enough to realize that while the new technology may not solve the particular problem you’d set in front of it, there may be a completely different problem that it is particularly adept at solving.

There’s a lot left unsaid here about technology development: commercialization strategies, product migration maps, technology adjacencies, market adjacencies. I’m not going to talk about those here. They’re important and you should be thinking about them, but they don’t help you deal with the emotions behind finding a new technology, rubbing the shiny off of it, and dealing with the commercialization endgame.

Do you have to be in a research and development group to follow this advice? Not at all. The advice holds for anyone who is prone to love affairs with technology. The specifics of methods and desired outcomes will change. Some people will try Ruby on Rails and ultimately find that sleek hottie too much for them, returning to the stable, quiet hippie in Django. Some folks will be looking for a greener energy source to put their activism into, take that one look at Clean Coal, realize that she’s just a two-bit whore and pass on to something that they know they can support while keeping their self-respect. At the end of the day, everyone will fall in love with a technology at least once. The trick is to be honest enough to know when its not the technology you’re seeking and flexible enough to let it solve the problems at which it is best.

Everybody needs to measure stuff. And whether you’re in a traditional business or a own a Web 2.0 startup or are just an average gal or guy, the need to measure things quickly and precisely has gotten a lot more intense in the past decade. You want to understand where your business is in the Long Tail or how “sticky” your website is or how much your coffee habit is costing you annually. And when I say precisely, I mean precisely in a, well, precise sense. To speak precisely, precise and accurate are not the same things. And this is the first thing to understand about measurement – a measurement is only valid when it is both sufficiently accurate and sufficiently precise.

Accuracy vs. precision

When you measure something accurately, your measurement gives you a number that is very close to the truth. You may not get the same number each time you make your measurement, but you know that its close to the actual value. When you measure something precisely, you’ll get close to the same result each time, but you may not be close to the actual answer. Ideally, we want our measurements to be both accurate and precise. In reality, most folks have a higher tolerance for lack of accuracy than they do for lack of precision. As long as the measurement is reasonably accurate, most people will settle for something that is off from the truth by a good bit so long as they get consistent answers from it. If you check your measuring cups in your kitchen drawers, you will find that they are pretty precise. Fill your 1/4 cup measure up 4 times and dump it in your 1 cup measure and it will fill it up exactly. (Or at least it did on each of the three sets of measures I had in my kitchen.) Yet, I have no idea – nor do I care – if the cups are calibrated properly. Do they deliver exactly 1 cup? If you care about that kind of accuracy, you’ll probably be using a graduated cylinder, not a plastic measuring cup. For most of us, the fact that the cups are precise is more important.

Does our tolerance for inaccuracy seem surprising? If you use Google Analytics to track your website stats, it shouldn’t be. Google can’t know, accurately, how many unique visitors actually visited your site. How can they? Even though they set a cookie to track your visitors, a lot of folks using Firefox will only accept cookies for that session, thus preventing Google from counting them over multiple visits. I do essentially the same thing with Omniweb. A lot of folks using IE will occasionally flush all of their cookies as a privacy measure. Each time the Google Analytics cookie for your site gets deleted, that user looks like a new user to Google. This means your unique visitor count is artificially high, as is your percentage of new users visiting your site. But, really, it doesn’t matter. You don’t care how accurate that number is, because whether you have 570 or 450 unique visitors per day isn’t as important as the trend. Is that number going up or down? Is it higher on Saturday mornings or weekday nights? As long as the measurement is precise, then those trends can be analyzed meaningfully.

Now we know what makes a measurement valid, and we understand that a large fraction of the time, we don’t need as much accuracy as we need precision. While I didn’t explicitly talk about it, it’s important to note that validity is predicated only upon sufficient accuracy and precision. Your car’s fuel gauge is neither terribly accurate nor terribly precise, but it represents a valid measurement because it gives you the data with sufficient accuracy and precision to keep you from running out of gas.

There are four other things to keep in mind about a measurement, which I’ll call the Four ‘R’s: Relevance, Range, Resolution, and Reproducibility.

Relevance

Relevance in measurement is both subtle and obvious: you want your measurement to give you the information you need. Obvious, right? If the answer is “you have a full tank of gas” or “1 cup of milk,” then relevance seems to be trivial. We don’t always ask such easy questions, though. A lot of the things we measure aren’t things we really care about, but are instead things we believe are highly correlated to the things we care about. If you choose a proxy to measure that isn’t very relevant to the answers you need, then you can accumulate a lot of data that is useless to you. This isn’t always an easy problem to solve.

One example of the difficulty of make a relevant measurement is SAT scores. The SAT is a measurement that we believe tells us something about how a student will perform in college. We can’t measure “successful in college” – at least not a priori, so ETS has made a test that some people think is an excellent proxy for measuring future success in college. Whether or not it is actually relevant is a matter of some debate currently.

Most college grads can name a buddy from their college days with excellent SAT scores who flunked out early on, due to boredom or irresponsibility. We might infer from this observation that discipline and work ethic is more highly correlated with college success than SAT score. If this were true, it would be a much more relevant measurement. To prove this hypothesis, though, we’d have to actually measure a student’s discipline. That’s a hard problem, one that doesn’t lend itself to easy measurement. Not to say that colleges don’t try – I recall having to include letters of recommendation for my college applications, which presumably were complimentary of my excellent academic discipline, stellar study habits, and good dental hygiene. Since, when I entered college, I only had the good dental hygiene part going for me, I expect that most colleges recognize that this measurement is neither precise nor accurate, despite its relevance to the answer we seek.

Range and Resolution

Range and resolution are the properties of measurement that are the easiest to explain and understand. Range is the distance between the highest and lowest value your measurement technique will register. Resolution is how many levels of distinction there are within your range. Your desk ruler has a range of 12″, or about 30.5 cm, and a resolution of 1/16″, or about 1 mm. If you want to measure out 6 yards of a geotextile sheet for your garden, you could do that with the ruler, but you’d certainly agree that a tape measure would be a better tool. With some measurements, though, making sure you have a method with sufficient range is a lot more critical. This is why your meat thermometer and your household medical thermometer are different instruments, with different means of measuring temperature. By the same token, the resolution of your meat thermometer is a lot lower. You don’t really need 0.1 F resolution on your meat thermometer, but when taking your temperature to determine if you have a fever, that resolution is important.

Reproducibility

A measurement’s reproducibility is different from its precision, even though the two are easily confused. One way to think of it is that if you measured something ten times, the variance in your numbers would be represent the precision of the measurement. If ten people measured it once, the variance in the measurement would represent its reproducibility. What this means is that if the measurement technique is easy to “do correctly,” then the measurement is reproducible. Conversely, it is possible to have a highly precise measurement that is not very reproducible, because of the difficulty in making the measurement. This is frequently the case with a lot of microscopical measurements, as variations in sample preparation and operator technique can affect the images before they’re measured. If you’re trying to determine whether or not a tissue biopsy contains cancerous cells by looking at a stained specimen, getting this right is critical.

Reproducibility is an issue with a lot of qualitative or semi-quantitative measurements. When I see in a recipe “fry onions until golden-brown,” I have to make a measurement with my eyes of the color of the onions in my pan as they are frying. I know from experience, though, that my idea of “golden-brown” means a lot more cooking than it does for many other people. The human visual system, it is safe to say, does not represent a very reproducible measurement. It works well enough, though, so I don’t expect that many recipes will include spectrophotometer data for accurate, precise and reproducible measurements of onion color anytime in the near future.

The key to making a measurement reproducible is to have a clear process for making the measurement. In my field, there are standards bodies that publish volumes and volumes of proper procedures for making measurements such as the stiffness of a plastic (ASTM D6272) or how well something resists burning. (ISO 11925) For the example I gave above about biopsied tissue, there are standard staining protocols, that ensure that the proper stains are used and at the appropriate levels. But you probably understand this concept instinctively – if you always level off a measuring spoon or hold a measuring cup up so that the liquid is at eye level, you’re following a standard procedure that will make your measurements reproducible.

Measuring things

So, what now? If you’ve gotten this far, you’re at least somewhat interested in the topic, since I am no J. K. Rowling. Thinking about measurement can be terribly addictive – at least, it is to me. Have you ever wondered how they measure inflation? While you’ve probably heard of the Consumer Price Index, the details of the measurement might surprise you. More practically, you might also be interested in combing your customer data for indications of whether you’re doing the right things in your business. You might want to measure your website traffic to determine whether your new web ad campaign is giving you a good return. If you own a restaurant, you may want to measure how effective your menu is. In any case, good measurement is the key to getting good answers in a lot of fields and understanding the mechanics of making a measurement is part of good measurement.

We’re up and going now and I’m sitting in the lab working on a new model for the data we’re collecting while I watch the equipment run.