This is a long post. But it ends with some interesting data. I recently read Ben Edelman’s piece about the Sears-comScore debacle, where Sears (”Sears Holdings Community”) was partnering with comScore to track users’ online activities, encouraging an install in a way that many users might find to be deceptive. Ben is a well-known industry researcher focusing on adware/spyware and his overall take from this piece about comScore is that:

There’s no good reason why users should share information about their browsing, purchasing, and other online activities. So time and time again, ComScore and its partners resort to trickery (or worse) to get their software onto users’ PCs.

As a former market research analyst myself, since comScore’s data gets used quite a bit by online firms, venture capitalists, bankers, etc. I am interested in figuring out what types of biases the recruitment methods of a comScore might introduce into the data. We recently pointed out large discrepancies between their figures for older users of Facebook versus what the company itself is claiming about its US audience, for example.

I know from Ben’s site that one of the recruiting sites that comScore uses is RelevantKnowledge.com - on the website they inform the user that they are gathering information and using it to create reports that get quoted in various publications, they even go so far as to show examples (partial screenshots from 2004 data) of the kinds of reports that are created. comScore’s name doesn’t appear on the reports or anywhere on the site. The terms of use /privacy policy reference that the site is owned by TMRG, Inc. with an address of 11465 Sunset Hills Road, Reston VA which is comScore’s corporate address.

This is where it gets a bit more interesting. We looked at the data for Relevant Knowledge on Quantcast.com - and according to their figures this site had 397, 653 unique visitors in December 2007, with a large bias towards women. Helpfully, the affinities of the users also skew towards another TMRG site called OpinionSquare.com. It has a fairly large user base as well of 152,023 users in December 2007 (Quantcast).

On their “About” page, OpinionSquare disingenuously states:

The information that you contribute is used by comScore, Inc., a U.S.-based market research company that is a nationally-recognized authority on Internet and general economic trends, whose data are routinely cited by major media outlets such as the New York Times, the Wall Street Journal, and CNBC, and is extensively used by the largest Internet services companies and scores of Fortune 500 companies.

OpinionSquare.com is owned by TMRG, Inc. according to the website. TMRG, Inc.’s privacy policy references 11465 Sunset Hills Road, Reston VA which is of course comScore’s corporate address. By the way, OpinionSquare also biases heavily towards women (indexing at almost 150 according to Quantcast.com) and older users with age 55-64 being almost 25% higher than the web average for this demographic.

Once I went to www.tmrginc.com, TMRG’s “corporate site”, I found these and the other two sites they use and disclose which are Permission Research and MobileXpression.

Buried in the TMRG website’s FAQ is a mention of the connection with comScore, of course if someone might be a potential customer:

My company is interested in using TMRG to conduct market research. Who should I contact? TMRG is a service of comScore, Inc., a leading Internet ratings system that provides insight into consumer behavior and attitudes. For assistance with your market research needs, please complete comScore’s Information Request form [link removed] and someone will contact you shortly.

I wondered how these sites would show up in comScore’s own data because presumably comScore’s $50,000 a year (and up) service is going to be much better quality that some of the free resources out there. The answer is that they don’t show up at all.

Hmmm does that mean these users are systematically weighted out of the panel? They were generally older and more female (the kinds of people who take surveys, respond to ads run on gaming sites, fall victim to browser exploits that install software perhaps?). And if that is the case then what does it mean to how other people get weighted? Or more insidiously are these sites meant to be ranked but are just removed from the comScore rankings to avoid any perceived taint? Surely no matter how bad the data is that the other providers could be putting out there, you’re not telling me that none of these sites are going to show up in the top 20,000 web domains that comScore publishes, where the minimum site had just 11,000 combined work, home, university users?

Did I miss a buried entry in an FAQ somewhere, comScore?