We often get e-mails that ask us about the criteria we use when reviewing a product. In some cases we are able to use empirical objective results. This would encompass reviews such as processors, graphics cards or hard drive performance. 
In performance-based testing and review, the comparison of benchmarks will often provide the majority of information for your result. In other words, if the objective is speed a simple speed test determines the winner and/or potentially best product.
The vast majority of the reviews and testing we conduct, however, are not that simple and objective. How well something works for a given task or in a given environment is often a subjective matter. Consider ergonomic mice. In testing them, we'll certainly look to see if they perform properly, accurately and have the buttons and controls the user is looking for. However when we get to that sticky "it's comfortable in your hand" section, you no longer can use empirical testing to get your answer. Simply put, it's going to boil down to someone saying "yes, this one feels best".
So in an effort to shed some light on that sticky, subjective area we have developed a set of criteria that we use to help illuminate those qualitative traits.
People in the real estate industry have a mantra for assigning value to a property. It lists three attributes.... and they all sound like "location". That works for real estate because of the overriding importance of that feature. For technology, and especially business or task oriented technology, we focus on three different attributes; productivity, convenience and sufficiency.
Is that what it's supposed to do..?????
I am a function over form kind of guy. Always have been, probably always will be. To me, the function of a device is its most important attribute. Form, and the aesthetics are just a bonus. A nice bonus mind you, but not as important as the actual function of the product in question.
You can see why Bradley Litwins Kinetic Sculpture might impede productivity.
In reviewing a piece of technology the first issue that we examine is whether or not this device will help make you more productive. This is a little beyond the basic issue of 'does it work?'. The real focus here is "does this new device actually improve your productivity and work output or enjoyment?"The second attribute we focus on his convenience. Is this technology convenient, simple and effective to use? Now it is true that the concept of convenience is very subjective and certainly relative, so we try to take the common man, common sense perspective to this more qualitative attribute.
The third attribute is sufficiency. Does this new technology do a sufficient job? Is it sufficient for its stated purpose?
First Real Estate...now a shovel??
To illustrate, we will look at the same three attributes of productivity, convenience and sufficiency in the framework of a common manual task and see what technology would get the best review.
Let's take a simple low-tech example, a shovel. The job at hand is to clear snow. It's something I had plenty of experience with this winter, as the Philadelphia area had a record 70+ inches of snow in one season. I shoveled enough snow this year to have a deep and abiding understanding and respect for what it takes to move snow from one place to another. So let's pretend we've just been sent some shovels to review for this task.
The first item is a short 9 inch wide garden shovel. While in fact this is a shovel, and could be used to remove snow, it would be sometime in July before I got finished. It would not improve my productivity in clearing snow; it would not be terribly convenient to try to remove the snow 9 inches at a time; and it certainly would not be sufficient, by any definition, for the job.
The second item is a standard snow shovel with a straight 
handle. While this item would certainly meet the criteria of productive and sufficient and allow me to clear the snow, 
it wouldn't necessarily be the best choice for convenience. I say this based on experience and the pain in my lower back.
The last item is a snow shovel that uses the ergonomic, bent shaft design. If you've ever seen one it looks like someone took the handle and bent it about 18°. This has the benefit of allowing you to load and lift the snow with much less stress on your lower back. This not only meets all three criteria but is assuredly the most convenient (convenience here being the alleviated workload on your body) and therefore more productive choice.
Connecting the Dots
Anytime we review a product we take an objective approach to the products purpose. We measure and test any and all attributes that are appropriate. We often compare an item to like products in the market space as an appropriate yardstick. We also, however, make careful note of our feelings and impressions as we go through the review process. This is because that is where that final "we like it... we don't like it" is going to come from.
Hopefully for those of you who have been curious, this has shed a little light. If not, at least you now have some guidelines for your review.