Delving Into the Review Process

[MylesBAstor]

Ian raised a really good question in the Kondo thread that deserves it own space. I think this applies to not only reviewers but consumer as well. How do people compare gear. Too often, I find audiophiles put something new in and never compare it to the piece of equipment that it replaced.

Tomorrow when I have more time I will respond too. Thank you Ian!

[Madfloyd]

Myles, when you live with a review component for 3 months, do you truly live with it or do you switch back and forth with your reference equipment now and then?

[fcrowder]

While I cannot speak for Myles, my practice when reviewing a product is to install it in my system and then listen to it for an extended period, finally changing back to the original component. It is generally the re-insertion of the known reference that is the most telling, but only after I have had the time to fix the sound of the new piece in my mind. I generally find quick A/B comparisons to be highly misleading.

[Madfloyd]

What if the new component isn't engaging you - or you outright dislike it? Do you forgo enjoying your system for the 3 months?

[mkuller]

No, you strongly ask to not have to review it and to have it sent back. No one wants to spend that amount of time with something that sounds bad. Fortunately, most components today sound pretty good.

[fcrowder]

Initially, I do try to optimize the sound of the new component. This can mean changing power cords, interconnects or sometimes speaker wire or tube rolling. If none of that helps, I generally remove the new piece from my system, but not before giving it a fair chance. At 66, life is to short to forego enjoying music for an extended period. I probably should say that I am very selective about those items which I agree to review. If they do not represent something that I might conceivably purchase then I do not accept the item for review.

[mkuller]

Optimizing the sound of the DUT is very important. I always felt it was my responsibility to listen to the component at it's best.

I recall one loudspeaker that sounded terrible just substituting it into my system. By the time I was finished I had a different preamp, amp and cables and it was one of the best sounding speakers I'd heard.

[Per Sundell]

I agree with mr fcrowder method above.
1. Insert the new product.
2. Listen for a given period while optimizing.
3. Re-insert the reference.
4. Come to some sort of conclusion.

If a component turns out to be missing any interesting features and fails to impress me I usually choose not to write about it at all.
Its better for all involved. A bad review doesn´t benifit me, importer/retailer/manufacturer nor reader.
(Unless something is totaly wrong with said product. Then consumer advice becomes a priority.)

[Johnny Vinyl]

How do reviews come to be?

Manufacturer/distributor request?
Reviewer request?
Editor assignments?
Other options?

[mkuller]

All of the above.

[Bill Hart]

Thoughts (was it Saturday Night Live, which I haven't watched since Belushi died, that had that meme about "Big Thoughts"?):

1. The idea of a "control" appeals to the "scientific" brain but how do we know that our "control" isn't skewed in some way? In other words, we always talk about synergy- that magical "whole is greater than the sum of parts" aspect to hi-fi--whether it is adjusting VTA, toe-in, bass (woofers), or the more elusive combination of cartridge or cable. Simply dropping another component into an existing system may not bring out the best in that component--whether because the existing system is "tweaked" to accommodate the usual configuration of components (the "control") or because the new component has different needs (for whatever reason). You may say that this way lies madness, but we know this stuff isn't plug and play (apart from basic electrical compatibility issues, gain, impedance, whatever).

2. To what extent do any "tweaks" described in item 1 above, represent a departure from the "equipment as sold" standard to which most reviewers like to adhere? I know people bitch about the price of, say, Stillpoints footers or fancy power cable, and complain that if G-D or the manufacturer intended that it be used with a particular isolation device or cable, it should be supplied by the manufacturer. On the other hand, we know most manufacturers don't want to build that extra cost into the pricing of their gear, so is it simply following whatever the manufacturer recommends? How far can a reviewer depart from those recommendations or should they? (e.g. "I know X said their amp works best with the X cable, but I tried the Y cable and thought it sounded better.")

3. Do you check output level with changes? Remember how, in the unscrupulous world of retail sales, the louder system usually sounded better? What does a component change do to the overall gain and output of the system? I know this is a factor in comparing different cuttings of records.

4. Is it really as simple as living with a component for a while,then switching back to the "control"? I usually like to think my first impressions are pretty good, but I'm always reluctant to judge a change based on immediate impressions without living with it for a while, to see if there are negatives lurking behind the immediate differences, often revealed over time, and a range of different source material. Does this apply when you switch back to the control? Why not?

5. Listening mindset and fatigue-- This gets a little 'voodoo'-y but I do think our state of mind plays a role in how we perceive things. Add to that, what I'll call the "analytical" mode of listening v the "listening for enjoyment" frame of mind. I find the "analytical" mode to be fatiguing in the sense that, after a while, I lose my ability to discern small differences (I'm not talking so much about fatigue in the sense of noise and harshness, but the loss of acuity). I find that "analytical" listening is often "listening too hard," i.e., like squinting to discern differences, rather than just letting it roll. (My most enjoyable listening is often when I'm a little wiped out, and my natural defenses are down, I'm just a little more inclined to let things soak in- perhaps a reason why people have a drink or whatever to relax and enjoy their tunes). Should there be a disclaimer on reviews, like the American Humane Association end-credit on films, that says, "The reviewer was of sound mind and not under the influence of any substances at the time of this review."

6. Long Term or Inherent Bias-- are reviewers really capable of being objective? I don't mean that they are taking money under the table or engaged in any scurrilous behavior, but my experience with listening and listeners, generally, says different people like different things. We presumably build our systems around these biases, so does it make sense to acknowledge that much at the outset? And how does it affect the review process if there is some natural preference of the reviewer to incline toward a particular attribute or sound? Second opinion?

Enough.
For now.

[mkuller]

Here, I'll give it a go:

1. I believe the component should be reviewed at its best and it is the reviewer's job to experiment with other system components to bring it out. The control is the sound of your system.

2. Tweaks include cables, footers, cords, etc. and the ones used should be mentioned in the review.

3. When switching I always use a digital multimeter and a 1kz test tone to set the level.

4. Long term listening - hours over days - is the only way to evaluate a component. Then you have to switch back and forth at your leisure. Your hearing is designed to recognize differences fairly quickly - it takes much longer to determine whether they are positive or negative.

5. With long term listening all of the the mood variables are neutralized because you will be in different moods. If you partake in mood altering substances once or twice during a review it may help put things in a different perspective which would need to be reality tested without them.

6. Everyone has their own inherent listening biases and these should be mentioned as such in the review. Reviewers strive to be very objective because the only things they have to recommend them is their perception, writing ability and integrity. Second opinions are the best practices but difficult to pull off logistically with deadlines.

[MylesBAstor]

Thanks Mike!

[Joe Pittman]

Ian raised a really good question in the Kondo thread that deserves it own space. I think this applies to not only reviewers but consumer as well. How do people compare gear. Too often, I find audiophiles put something new in and never compare it to the piece of equipment that it replaced.

You bring up a really important subject Myles. Not only is it good to know how people make comparisons, but what are their references and goals. How much gear have they heard, what is their experience? Is a persons opinion based on speculation or have they actually owned and used other equipment for long periods of time.

My primary objective in getting into R2R tapes was to have an absolute reference for improving my LP playback system and closing the GAP between tape and LP. Having a consistent reference is good for comparing cartridges, tables, and LP's, relative to my reference. It has also been handy to have multiple systems, but that is another subject.

We audiofools are each unique individuals and no two of us will ever agree upon every thing audio.

[MylesBAstor]

So let me backtrack for a moment from I what originally posted. Ian's question applies not just to reviewers but to audiophiles as well. There are too many instances of where audiophiles rush to judgement based on a quick listen or just because the new piece of equipment sounds different. So we go back to the old question of whether the DUT is better or simply sounds different.

Then we can ask what is the reviewers job? Is it 1) to describe the sound of the DUT as thoroughly as possible, paint a picture so the reader can reconstruct the sound in their mind's eye and draw their own conclusions or 2) to decide whether or not the new piece of gear is better or 3) some fusion of the aforementioned response? Neither answer is right, neither is wrong. Reviewers take both approaches and their are successful answers to either approach.

Myles, when you live with a review component for 3 months, do you truly live with it or do you switch back and forth with your reference equipment now and then?

Yes but depends. Like that answer?

It's very easy to swap out cartridges and compare on my table. Or arms if you can get two of the same cartridge. That's a major reason -- besides sound -- that I use the new VPI table with three arm. Or easily interchangeable arm wands. It's also not too hard to swap in and out digital front-ends like DACs or servers.

Where it gets more complicated is with bigger items like speakers or amplifiers. Kind of hard for one person to move a 300 pound amplifier around or finding room for two large monoblock amplifier in small to mid-size rooms. And we can argue whether switching in or out is meaningful because once you turn an amp on and off (or most electronics; DACs are an extreme example because I think they take at least 24 hrs after shutting off for the sound to start to stabilize.), they usually take 30-60 mins to sound right again. Or even a couple of hours in my experience say in the case of amps with interstage coupling transformers.

So my answer here is like Freds. In the case of something like a 300 pound speaker, I listen to the transducer or equipment for 2-3 months (most speakers or equipment aren't broken-in) and work on optimizing the sound (cables, supports, amplifiers if possible, etc). Large speakers are just so hard to do and will need a team to move around. So usually it's go back to reference system and if absolutely necessary switch the review speakers back in.

Cables that appear at first glance to be an easy thing to review aren't in reality that easy. That's because once you disturb the cable, it takes up to 24 hours for the sound to restabilize. One can hear it often as a loss of transparency or solidity of imaging.

Then you asked: What if the new component isn't engaging you - or you outright dislike it? Do you forgo enjoying your system for the 3 months?

Another good question and here's how I've (or magazines I've written for) have dealt with that sort of situation. But before getting into that, remember very few pieces that we get come broken-in. So sometimes it's just suck it up and hope the gear sounds better with time. Of course a big question is does that initial problem go away or just diminish in "intensity."

Back to the original question. The first thing I would do is contact the manufacturer to ask their advice and to check and ascertain that the equipment is working properly. Shipping is a crapshoot (I as well as most reviewers can tell you some horror stories) and things can and do happen in shipping. Whether it be Fedex, UPS or whatever.

Manufacturers obviously also have more experience with their gear than a reviewer so sometimes that contact goes a long way to straightening out a situation. For instance, there could just be an unpredictable interaction between equipment going on. Or maybe they might say try X or Y cables that we use at the factory. Or the caps take forever to burn in and they take 100-200 hours or more to break in. Or even speakers could take 600 hrs to sound their best (how long did it take you Magicos to break-in). And so on.

What if that didn't help? One thing that I did as a publisher was send the piece of equipment to one or two other reviewers on the staff to also hear and get their opinion on the review piece. Years ago had that issue with an MFA amplifier that reviewer A got in and didn't work with his Avalon speaker. He passed it onto me and it didn't work with my Maggies. So passed it onto someone else and didn't work there. Turned out that the amp was designed with an early Sound Labs estat and that was the only speaker the amplifier really worked with because the speaker's wild electrical swings and low sensitivity. That happened with the Berning EA2100 amplifier years ago too. Worked with one reviewer, sounded horrible on Maggies. Zero low end. Go figure.

If all else fails, then up to the reviewer and editors/publishers. As mentioned above, most reviewers are part time (there only full time reviewers that come to mind are Valin and Fremer) and just don't have the time to write negative reviews. That said, I have written some negative reviews in my lifetime and it wasn't fun and in one case, the manufacturer didn't take to it kindly. (Even though Scull in the parallel Stereophile wrote a similar review.)

[MylesBAstor]

You bring up a really important subject Myles. Not only is it good to know how people make comparisons, but what are their references and goals. How much gear have they heard, what is their experience? Is a persons opinion based on speculation or have they actually owned and used other equipment for long periods of time.

I think all of the aforementioned points are important, not the least of which is what qualities are important to them for the reproduction of music and what are their goals. All of us have certain things we are sensitive to and other areas that we an listen through or not as picky about. Some people it's dynamics. Others it's midrange. Yet others it might be soundstaging. And yet for others it might be quietness.

My primary objective in getting into R2R tapes was to have an absolute reference for improving my LP playback system and closing the GAP between tape and LP. Having a consistent reference is good for comparing cartridges, tables, and LP's, relative to my reference. It has also been handy to have multiple systems, but that is another subject.

We audiofools are each unique individuals and no two of us will ever agree upon every thing audio.

That was in part one of the reason why I too too the plunge into tape. And the sound.

[Dre_J]