Gartner’s Magic Quadrant reports, for years, have been used by CSPs to help determine which vendors best fit the criteria they are looking at. Years ago there was a saying that nobody got fired for buying IBM, they even made a TV advertisement playing on it. The same could be said about the Gartner reports but they may be questioned now.

A recent article by Ed Finegold appearing in BillingViews questioning the rationale behind one of Gartner’s Magic Quadrant reports is causing quite a stir. Finegold steps out on a limb by asking how Gartner could come to some conclusions about business system vendors, well known to all of us, and be so wrong. That certainly takes some courage, but based on the responses the article has generated, many others have been wanting to ask the same questions but may have been afraid to.

The report, titled the “Magic Quadrant for Integrated Revenue and Customer Management (IRCM) for CSPs,” rates solution suites that provide billing, customer care, rating, charging, pricing, partner relationship management, policy management, mediation, self-service, analytics and other functions. That is a pretty comprehensive list for any suppliers and that alone, you would think, would restrict the list dramatically.

I can’t reprint any of the report here, even though I have read a copy, but I can support some of the observations made by Finegold in his article. For many suppliers included in the report there was also some confusion because they had been called by Gartner and provided information that was not included or taken into account in the report. One said, “we briefed Gartner three times and they said we had received credit for all the innovation we had done in our prime markets, and our huge scale deployments, but in practice they seem to have ignored this.”

For example, the report places NetCracker, Ericsson and AsiaInfo-Linkage, along with Redknee, in the niche player quadrant that Gartner explains is “because of a geographical shortfall, narrow focus (they may provide only a limited subset of IRCM core and adjacent functionality) or lack of financial strength (that is, they have not achieved financial viability compared with the Leaders), or they have not come as far as the Leaders in advancing their technologies or functionality.”

The first three, to my knowledge, are anything but niche players and offer a full suite of solutions as defined by IRCM. NetCracker is now part of the massive NEC group, Ericsson manages over 2 billion customers worldwide, and Asia-Info is probably the largest IRCM vendor by subscribers billed, in Asia, due to their massive presence in China. Redknee, having just acquired NSN’s BSS business is firmly in the big league now, but that may have been after the report was published.

As Finegold points out, not all the report is out of line, but the glaring inconsistencies put it into question. Maybe this particular report is an exception, maybe it was written by inexperienced analysts, maybe the IRCM bundle is just too broad to objectively address, whatever the case, it is not what we would expect from a firm like Gartner. You would also expect to see some detailed overview of how the results were arrived at. Whether analytical, statistical or physical, the evidence should be transparently provided as it is in many other MQ reports.

I’m sure that major purchasing decisions aren’t made solely on Gartner or any other commercially available analyst reports, but they do influence many decisions as pointed out earlier. For fear of sounding nepotistic, this is where the TM Forum’s own conformance certification program comes into its own. Used in conjunction with third party reports, CSPs can quickly determine the suitability of a solution that has been thoroughly evaluated and conforms to the requirements of the Frameworx that most CSPs utilize in their operations.

Heaven forbid, all this publicity for the Gartner IRCM report may even generate sales for it, especially from those keen to assess if all the criticism is justified.

First published at TM Forum as The Insider, 18 December, 2012