Editorial Ethics Part 2: Where is the Trust?

Welcome to Lake Wobegon, where all the women are strong, all the men are good-looking, and all the children are above average. Garrison Keillor

Are CAD magazines looking through rose-colored glasses? Every product cannot be a "Highly Recommended" or "Readers Choice." When was the last time you saw a really bad review of a product in a CAD magazine?

Never. That is what Sean Dotson says, in response to Editorial Ethics, that appeared here. Sean says this is due to a fear of offending advertisers. Conspiracy?

Come to think of it, I don't recall seeing too many bad reviews lately either. And I see a lot of reviews. Our site, www.caddigest.com/subjects/reviews_sw/ lists thousands of reviews of CAD, CAM and CAE software culled from hundreds of sources. Does our industry have no clunkers? Or, as Mr. Fane suggests, do bad reviews simply not get printed?

Witness just April's reviews from Cadalyst magazine. Only one product does not get a "highly recommended" rating.

• April 17 - Dell UltraSharp 3007WFP LCD Cadalyst -- highly recommended
• April 10 - VizUp v2, Cadalyst --highly recommended
• April 7 - AutoCAD 2006 and AutoCAD LT 2006 Bible, Cadalyst --highly recommended
• April 6 - MaxonForm for ArchiCAD, Cadalyst -- no rating given
• April 1 - AutoCAD 2007 -- highly recommended
• April 1 - Adobe Acrobat 3D, Cadalyst --highly recommended

Granted, reviews are subjective. I have not reviewed these products so I can't truly say if the reviews are valid. But in the case of the Dell monitor, I can see how other reviewers (outside our industry) found it:

• CNET: the Dell UltraSharp 3007WFP gave it only a 7.6 out of 10 (see CNET review) using a weighted scale that accounted for setup, design, features and performance. In the $1,200 to $2,470 price category, the Apple Cinema HD Display scored higher, with a 7.9. But as far as I know, Apple doesn't advertise with CAD publications.
• ExtremeTech (owned by Ziff Davis) gave the Dell Computer an even more rigorous workout and found fault with "hotspotting" (really only noticeable fro DVD and high-def video playback), giving it an 8 out of 10 score. (see ExtremeTech review)
• PC Magazine gave the Dell unit and Editor's Choice award but only 4 out or 5 stars (see PCmag.com review)
• PC World gave it an 84 and a 'Very Good' rating (see PCW review)

The consensus from the computer press, based on the above non-scientific poll, would reveal that the Dell display is good, maybe even very good, but not "highly recommended." Cadalyst could easily argue that its rating is justified as faults found by the computer press that resulted in a lower rating (lack of DVD input, hot-spotting) may not be relevant to the CAD user and the Cadalyst reader.

Also, look at another review that appeared in Desktop Engineering, this one of Inventor 11 (see review). Throughout the review, the product is praised for all of its enhancements. However, in his summation, Al Dean states "Inventor is finally reaching the level of maturity that its position in the market place demands." Whoa! Finally? Is reaching? That means it's not quite there, huh? That is indeed damning by faint praise.

CNET, PC Magazine and PC World have been know to give bad reviews. Forget damning by faint praise, products have left their labs burning in flames. And what does this do for me, the reader? Well, for one thing, if respected experts are not afraid to say what they really feel, I don't have to read between the lines to make an informed buying decision. It's not necessarily that they condemn a lot of products, but they are courageous enough to point out faults where they exist. This, as CAD Insider blog readers point out, is something missing from our industry.

Comments

[al dean]

Roopinder,

Hmm. Just read your comments about the Inventor review that appeared in Desktop Engineering - and I'm not entirely sure what you're getting at.

Perhaps extracting snippets from reviews isn't a particularly wise idea. what we try to do is look at the content of each release÷ evaluate the benefits and inform. now, in these days of a mature market, much of that work is too assist existing users to understand what they're getting for their maintenance/subscription cash. So, yes I praised the new features in Inventor 11 - there's some good solid technology there that can help designers do their job more efficiciently.

Now, if you want to pick every review apart that appears in every magazine the that's your call but when the major CAD related web- sites simply run press releases verbatim with no editing, no filtering then I think you and the other parties out there should perhaps look a little closer to home before slinging the proverbial mud at those of us trying to disemminate information to readers and trying to add some value.

There's so little filtered content out there, it seems more than a little redunant..

[Bill Foster]

As the voice of CAD users the goal of these ratings should be a serious push to make the future of CAD a reality. Instead it seems that a good rating merely involves a pat on the back for fixing the mistakes of the last release.

[gary heath]

Organizations do determine an ethical climate, but isn’t it up to the individual to honor their own set of values, credibility, and honesty beyond all else?

Without honesty in a product review who really wins? Readership has to fall off (since less can be trusted of what is being read) and with that vendors will stop advertising because there isn’t a large enough readership. Vendors will also stop advertising if “glowing” – nothing but extremely positive reviews are constantly given to a competitive product that is advertised every month. A sense of the editorial being “bought” prevails.

Vendors are just as guilty of editorial ethical liberty. A competitor of ours has a habit of paying for “white papers” by vertical industry analyst and then turning around and extracting choice wording as quotes from those white papers as “analyst stated” product endorsements. Let’s call it creative marketing I guess, at least that is what one of the analyst suggested when then offered to write a white paper for us for the same purpose!

Nothing helps a vendor improve a product faster than a bad product review. It hits you in the gut and yes we don't like it! But short comings are usually rapidly addressed if there is serious intent to have a great quality product and be competitive in the market. The quality and goodness of the product will eventually be understood by the market, so users will know when an editorial review is way off the mark and when that happens it has to reflect badly on the honesty of the review and any repeat business for the vendor. In the end the best thing may be accuracy for the reader, blunt honesty for the vendor from an unbiased highly knowledgeable third party, and nothing put pure integrity in the editorial. Everyone actually wins.

[H. Edward Goldberg]

David:

You and Ralph are to be commended. I never worked with either of you at CADALYST, but I know that the team that writes for the magazine has the best interest of the reader in mind. Given that I only review what I think is excellent software in each genre, one can inply my preferences. If I think a software solution stands out, I say that. I can not speak for CADALYST, but I am very impressed by the Editor Sara Ferris, and respect her editorial judgement. I am also impressed by all the contributers. I think CADALYST is the best source of information on CAD, MCAD, and AEC on the market.

Ed Goldberg AIA, NCARB
HEGRA
http://www.hegra.org

[David Cohn]

I'm actually a bit surprised that Ralph didn't bring this up in his earlier response to your first article on this topic.

First, we need to recognize that the market has matured and there are probably fewer products today that are actually bad. That said, I quickly found a number of articles in CADCAMNet and EAReport over the past few years in which we warned users to beware of a particular aspect of a product, or actually said that we could not recommend it.

When Ralph and I worked together at CADalyst, there were many more start-ups and we saw quite a few products that were, shall we say, released prematurely.

Ralph and I used to debate the merits of a product, even doing the old Siskel and Ebert "Thumbs-Up/Thumbs-Down" routine. Our policy was always one in which if a manufacturer sent us a product to review and we actually took the time to review it, we would publish the review regardless of our conclusion: good, bad, or ugly. Our publisher always stood behind us.

There was at least one occasion when after reviewing a product, we concluded that the product was terrible, and we were prepared to publish a review that stated that potential customers should absolutely NOT spend their money on this product. I don't remember if the company was an advertiser or not, nor do I remember the product.

As was also our policy, we gave the manufacturer an opportunity to read a draft of the review to check it for technical accuracy. Needless to say, the manufacturer was extremely alarmed that such an unfavorable review would be published. We explained that since they had sent us the product and we had reviewed it, our decision stood. The manufacturer pulled the product from the market rather than have us publish such an unfavorable review.

[H. Edward Goldberg]

As the author of CADALYST's AEC column, and a practicing architect,as well as an Industry analyst; I only recommend the software that I like, and that I have either tried, or watched being used. As for me, I think all the software I recommend is excellent software, and what is best for one user, is not necessarly the best for another. The big productivity tool is familarity with the software. I am concerned, though by all the "BIM experts" that are coming out of the woodwork. How many of them are using the software in a BIM mode to produce real work.

Ed Goldberg AIA, NCARB
h.e.goldberg@verizon.net
http://www.hegra.org

[Jimmy Bergmark]

Good points. I don't consider myself as an editor or writer even though I have written quite many articles for CAD magazines, have a blog and so on. I've done it from my point of view and background and I've had at least a couple of articles that was not appreciated by the vendors and the editor got a lot of heat for it. I actually asked the vendors several weeks before handing in the article to comment on it in case I've missed something but they didn't do it until after it was published. Now I've written a review about AutoCAD 2007 for a Swedish CAD Magazine and I could not give one rating if looking at recommendations but I pointed out that AutoCAD 2007 could be worth upgrading to for some but not for all and the reasons why.

[ralphg]

As CADalyst magazine's first technical editor, I came up with the rating system:

* Not Recommended -- the product was severely flawed. Such as the HI pen plotter that spat out its paper, and the pen continued drawing on the grit roller.
* Limited Recommendation -- the product did work, but had a significant problem or was overpriced. Such as the Nth Graphics board that was way overpriced compared to software competitors.
* Recommended -- the default rating for most products.
* Highly Recommended -- rarely given, and only for an outstanding product, such as HP's DraftMaster plotter, that was significantly better than its competition.

It pains me to see CADalyst editors today abuse the system.