Do You Have the Right Beta Program?

The blogosphere and yours truly gave Microsoft a very hard time about Vista. My claim was that I truly don’t care if Vista is easier, nicer or has many new features as long as it is so much slower than Windows XP, its predecessor.

During the last month, Microsoft started releasing its first service pack for Vista: SP1. I downloaded the whole 600MB of it and installed it, and guess what? Windows Vista is working better and faster now. It is still not XP, but the operating system works well enough for me not to complain. I guess that the Vista ordeal cost Microsoft hundreds of millions in lost goodwill. So how come Microsoft was not smart enough to release the right Vista in the first place? I can think of few possible scenarios, read all of them: one can be relevant to the way you release products…

  • Wrong beta program—Windows Vista was tested ala Microsoft by thousands of people. How come these people did not revolt and tell Microsoft that it could not release Vista the way it did? My guess—Microsoft beta programs attract “bleeding edge” personalities who are so happy with anything new that they forgive faults and mistakes. Beta tests should include every type of customer: look at Geoffrey Moore’s Crossing the Chasm  customer bell curve and make sure you have a few of each type in your program. The real value will come from the middle of the curve, not from the edges.
  • We will be forgiven—The other option is that Microsoft got used to being forgiven for bad 1.0 releases and knew all along that the Vista release would be replaced by a better one. It is strange to see how fast all the OEM partners of Microsoft adopted Vista as the only option on new PCs. While we are all angry with Microsoft’s inconsiderate behavior, let’s not forget the computer manufacturers that recklessly offered a premature operating system to hundreds of thousands, and by that created negative user experience for their PCs.
  • Microsoft did not have a choice—After years of delays, Microsoft just could not wait any longer and cynically released Vista to buy time. History will judge if Microsoft did the right thing. I don’t think they did.

  • Microsoft did not stress test Vista—I can’t believe they didn’t, but I had to write it down as a reminder for other software companies: performance is part of your package. You cannot avoid testing performance before releasing a new product.

Now, enough ranting about Microsoft: here are the DOs and DON’Ts for software release:

  • Test your product using normal machines and ask your testers to perform real daily tasks so they can “feel the pain.”
  • Create a balanced beta program—don’t overpopulate it with the tech-enthusiastic but bring in the real customers: the early and late adopters.
  • You cannot make a first impression twice, so don’t release half-baked goods. You may not get the chance to serve them again…
Do You Have the Right Beta Program?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s