Many opinions have been put forward about the ability of Klout, Peerindex and others to measure social influence accurately. Suffice to say, they’re controversial, few buy into them 100%, and those who do use them do so as one part of a broader spectrum of measures.
In their defence, the measurement providers talk of developing models and tweaking algorithms. Recently, after beingrumbled for purchasing twitter followers, PeerIndex CEO Azeem Azhar explained that he’d done so as an experiment to prove that follower numbers don’t affect your score. He explained that the boost in followers did change his score a little. They analysed it and then they changed their algorithm to eliminate the spike.
Changing the algorithm to make scores more accurate is, of course, sensible. The models are new, they need to be developed and the fact that PeerIndex, Klout and others are monitoring and improving them shows the investment in time that these companies are making to become better.
This changing of algorithms creates a problem: where does that leave tracking data?
If you do use Klout or PeerIndex to measure influence (at whatever level), then every time they change their algorithm, you’re technically starting from scratch because you can’t really, accurately compare the scores from before the change to the ones after the change.
This is a common problem in market research. Do you change your methodology to improve accuracy, or do you stick with your current process so you can retain the validity of your time-series data.
Realistically, most changes to the social influence algorithms won’t be so large that they’ll throw scores massively off, and comparisons to some extent will still be able to be made. However, that’s assuming you know when the algorithm has changed and how it’s changed. I’m not aware that it’s the policy of any of these firms to announce the date of an algorithm change and its likely affects.
Those who do use these scores for measuring and tracking social influence may well be watching a score rise or fall with no real change in their actual influence.