go-air.github.io

website

View the Project on GitHub

Ranking System Fallacies

I conjecture that almost all ranking systems, such as the original Google web page ranking system, and related search based rankings share a single global fallacy. In a nutshell, they all assume independence from the feedback arising from knowledge of the ranking method.

In an apparent effort to work around this, for example, Google and other search engines have put enormous effort into avoiding allowing people and organisations to spoof the ranking system. Now there are SEOs whose profession it is, essentially, to spoof the ranking system for the fee of their service and knowledge of how it all works, despite the fact that the popularity of the search engines was largely boosted by claims of fair and democratic ranking systems to begin with.

Similarly, GitHub crawlers today search for things like number of stars, issues, pull requests, dependency analysis, etc. As a result, I’ve observed a lot of activity which apparently is geared toward increasing these types of measures of projects, most of which would be better spent working on the product itself. Again, the knowledge of the ranking system creates a feedback which decredibilizes the ranking system to begin with.

The next time some service claims to “objectively” rank something you are interested in, I would encourage you to take a step back and look at how knowledge of that ranking system plays in the result. In all likelihood, many of those involved spend more time and energy gaming it than they do working on the thing that is being ranked. I think most of us would prefer they focus on their work.