Last year I mentioned to a client in passing that algorithmic results are evaluated and improved by the feedback from Search Quality Evaluators.
The client, a programming background CTO of a well-known brand, was horrified that I dare suggest that search engines have HUMANS doing anything vaguely related to improving search engine results. Despite politely mentioning that I had managed a team of Search Quality Evaluators he remained unconvinced and repeated algorithm in a semi chant as if to banish the notion of humans improving algorithmic processes….amusing since humans clearly create the rules for ranking.
So for those who remain unconvinced, here is a job posting for Google Search Quality Evaluators in India.
Search Quality Evaluator
Here is a LinkedIn to a chap who is a contract SQE in India: http://in.linkedin.com/pub/anil-yamarti/16/425/a6a
I may even find a team photo of my old Search Quality Group some time.
While the job title ‘Search Quality Evaluator’ can cover many types of task (12 distinct tasks in my old Ask.com team) the aim is generally the same, to try to avoid one-off changes (a VP’s restaurant venture not ranked in top 5) and to find patterns that will yield significant wide scale improvements and allow refinement to automated processes. Not to mention that hunting down Web Spam, hate, incite, illegal material etc is a full-time job.
Typically when an algo update is made, a team of SQE’s will run diagnostics and while much of the work is automated, the results still need evaluation and also human evaluation of a set % of pages means very well constructed spam doesn’t sneak by as well a vital human qualitative judgement comes into the mix.
True, it can be a very dull task at times, with huge pressure from engineering to review/approve a releases but equally when my team found new types of spam, it could be very rewarding to remove 350,000 pages from the index in a single hit or see a tweak to the ranking rules drop lower quality blogs in favour of slightly older but higher quality websites. Removing illegal material feels palpably like you’ve made the world a better place.
For the CTO who felt human evaluation of search results was not scalable he was correct but only as he probably understood the role
But it is not an issue of looking at every page, but looking at classes of data, patterns and designing new tests to identify quality issues such as freshness, adult material, radical material, spam etc.
Our critics other assertion was ‘No Google engineer would listen to a quality rater who isn’t an engineer himself’. On this point I suspect he is plain incorrect as the comment belies a fundamental belief that engineers cannot exist outside of teams writing the core code. Now while I cannot speak for Google, I know my Ask.com teams saw the relationship as co-dependant rather than subservient. In my core area of heuristic search I remember many times when a quality evaluator would draw my attention to needed improvements to patterns I created. I never refuted their comments purely on the basis that they didn’t create the search pattern or understand the vagaries of search architecture.
While Search Quality Evaluators don’t get much press, they are certainly an extremely valuable function in any search engine, every time your search results improve you probably have the Search Quality team at your favoured search engine to thank.
