Meet Google Quality Raters: The Indian Workers Evaluating Search Results

NEW DELHI — Much like everyone else, a large part of Sam’s day (last name withheld on request) involves poring over Google search results.  But where most of us use Google to sift through the internet’s infinite stores of information, Sam examines each result page through the prism of Google’s Search Quality Evaluator Guidelines, a 167 page document that helps professional evaluators assess the quality of the trillions of search results Google spits out each year.

“There are no timings, I work from home in Assam, and you don’t have to do it at a specific time or place,” Sam explained. “There is a Chrome extension that you have to install on your browser, and it tracks the tasks you are doing. Usually there aren’t too many tasks, but you can work between 30 to 40 hours a week.”

Sam is one of millions of people pulled into a vast, invisible human workforce that performs a kind of 21st century janitorial service on the interlocking infrastructure of artificial intelligence, automated programmes, and algorithms that continually shape the Internet as we know it. 

For nearly two decades from the early 1990s to the late 2010s, young Indian engineers were quickly snapped up by software companies for their ability to code quickly and cheaply. Eager to retain talent, companies in India mimicked the sprawling campuses of Silicon Valley’s tech giants. Now, as the Indian economy slows, young Indians find themselves adjusting to a new reality of working out of their bedrooms on laptops.

For the latest news and more, follow HuffPost India on Twitter , Facebook , and subscribe to our newsletter .

Sam exemplifies this trend. He studied for a B.Tech from Lovely Professional University , a private university in Punjab. He then joined an MNC in Gurgaon, quit, and wanted to join another—but fell sick during the training period and lost the job. Finding a new job after that wasn’t easy and Sam, in his own words, was “jobless, depressed, and suicidal.” 

Searching through the Internet for jobs, he came across a blog post that talked about working for Google indirectly. He quickly applied for a job. 

“It was a life saver for me, and the best work from home job I can think of,” Sam said. He’s now become an informal evangelist for the industry — Sam runs a blog called Outside That Cubicle about his work, and also offers to train people who are interested in applying for rater jobs.

The content on Sam’s blog reads like a code to crack the gig economy in a slow job market: “27 Survey Sites With No Minimum Payout Requirement”, “11 Best Survey Sites In Canada”, and “An authoritative guide on how to earn money online even if you have no idea!”

“You don’t have to work that much to make good money each month,” Sam said. “When you are working from home, you don’t have to deal with traffic, you don’t have to get ready to go to work. That alone gives you an extra two-three hours.”

Yet raters like Sam are not Google employees, but are freelance contractors working for third-party companies like Leapforce, Appen and Lionbridge. The job, like many new jobs in the gig economy, offer flexible hours and reasonable pay, but no benefits, insurance or guarantees. Some companies do require raters to work a minimum number of hours a week, but others don’t.

“The task availability varies according to countries, time of the day, or month of the year,” Sam said. “There were some months, where I had to literally run and grab the tasks before they were gone. There were some months when I felt so inundated by massive task volume that I had to literally walk away from my desk. It is not that I have to finish all of them, it was just a feeling of being drowned on it.” 

What does a search quality rater do?

Quality raters like Sam start their day by logging into work on the Internet and, depending on the task at hand, assess the quality of search results, shopping recommendations, YouTube results—anything that is algorithmically generated and could use human feedback. 

“A typical day starts with picking up a task to review search results. Like, results for when people search for ‘coffee shops near me’. So, you’re given a set of results,” Sam said, explaining that raters must log into an online tool that periodically sends them things to do.

Once the search results come in, raters must review them based on the information contained in the guidelines. 

“You rate how relevant or helpful those links are... You’re trying to see if needs were met, and quality of page. So, how helpful was it, and was it from a page that’s trustworthy, was it posted by experts?” 

Each decision Sam takes is informed by Google’s publicly available guidelines. 

“The guidelines are your Bible—as long as you stick to it closely, you’ll do great and make money,” said another Appen rater working on an assignment called Project Yukon for Google, who asked not to be named. “Other people are doing audits of what you’re reviewing, there is a monthly quality review, and it’s a very big deal. So you know, just because you don’t like something doesn’t mean you give it a bad rating, because someone else is going to be checking your work.”

To start off, evaluators must first take an online test that assesses their understanding of the guidelines. 

“The first 30 days we were limited to tasks for two hours a day, and then quality evaluation of the work we did,” Sam said. “After the first 30 days, you can work longer hours, but there are random checks of the quality of your searches, which have to be maintained.” 

The work itself is pretty straightforward, and the biggest common complaint most raters have, is that there aren’t enough tasks available for them to do. 

When Sam first started the job, most of his work was image and text related; but as Google has moved aggressively into voice-search to compete with Amazon’s Alexa, Sam said his work changed too. Text and image remain a staple, but he now has to evaluate other services too.

“The primary role that Search Quality raters play is to help us evaluate our algorithms. In addition, we use the data Search Quality raters produce to train our algorithms to better align with our guidelines,” Google said in a statement.

“Raters are trained against guidelines and are required to pass a test on the guidelines before they can begin rating. Additionally, we regularly include blind tests within the rating tasks as a way to ensure that raters continue to follow the guidelines,” Google said.

A big business opportunity

Last year, Google CEO Sundar Pichai said that the search engine served with over 3 trillion results in 2017, and that 15% of the queries were ones that had never been searched for before.

Thanks to that kind of scale, there is a fast growing market for jobs like Sam’s. Google isn’t the only company shopping around for cheap labour to cross-check their systems; artificial intelligence companies are increasingly looking for humans to train their systems by categorising information into easy-to-process buckets for their proprietary self-learning algorithms.

“In the first half of 2019, our revenue was up 60% year-over-year to $245.1 million,” a spokesperson for Appen told HuffPost India. “Today, we have over 600 full-time staff worldwide, and over one million skilled data annotators globally.” 

Appen told HuffPost India that it collects and labels the images, text, speech, audio, and video data that are used to build and continuously improve AI systems—a market that analysts expect to reach upwards of $1 billion by 2023.

“At any given time, about 20,000 to 30,000 of those annotators are working on data collection or labeling projects for our customers worldwide. As you know, the AI market is still in its early stages and growing rapidly, and as a result, so are we. India is no exception, as our industry is seeing increasing demand for Indian language and dialect expertise,” the Appen spokesperson added.

The job is pretty similar across companies—other workers that spoke to HuffPost India described similar programs and payments. One worker for LionBridge, who asked not to be named because they weren’t certain what was covered in their non-disclosure agreement, also described a similar Chrome extension that tracked their work, and allowed them to access a task dashboard.

“It’s a good job that pays really well,” he said. “I usually log in at night, and put in a couple of hours every day, including on weekends. It’s not a difficult job, you just pay attention to detail, and make sure that your quality ratings don’t dip. At first it can be difficult because you’ll spend a lot of time checking the guidelines and then checking the page and back again. But after a couple of months of this you learn the rules, and then you get much faster.”

“The only problem is when they change the guidelines. Like in August they just updated the guidelines. But you spend some time reading that, and it highlights the updates, so it’s very manageable,” he added.

The companies typically pay between $3—7 per hour, which can work out to up to Rs 50,000 per month, according to evaluators that spoke to HuffPost India. 

“We work in over 130 countries around the world, and pay does vary by region. We pay our curated Crowd above minimum wage in every market around the world in which we operate,” an Appen representative told HuffPost India.

“There is no cap on the number of hours an independent contractor can work; it depends on the amount of projects available. Also, independent contractors can work on multiple projects,” he added.

It’s a similar picture in China, reports Cyber Infrastructure : “In China, a new generation of low-wage employees is assembling the foundations of the future. Start-ups in smaller, more economical cities have sprung up to apply tags to China’s huge trove of pictures and reporting footage. If China is the Saudi Arabia of information, as one specialist states, these firms would be the refineries.”

Making sense of technology

For a lot of people who work as search raters, there’s a sense of being able to make sense of the incredibly confusing world of big tech through their jobs. Joydip Sadhu, a Kolkata based quality rater who has been working for LionBridge for the last two years while studying civil engineering at Jadavpur University talked about how the job has inspired him to study Web development.

The 22 year old just completed his graduation, and lives at home with his parents. Working at LionBridge, he was logging around 100 hours a month, which he would split according to his free time. “When I wanted to take a break, like I wanted to go on a vacation, so I did the extra work before that, and then went to Simla,” Sandhu told HuffPost India, “because it is up to me to decide how to organise my time, the company does not tell us what to do.”

A fan of the work-life balance that this job affords him, Sandhu told HuffPost India that he found LionBridge, while searching for work-from-home jobs on Google. “I did this while studying for my graduation for the last two years. I cannot disclose the exact rate, but I was making around Rs. 50,000 a month.”

Sandhu lives at home with his parents, and wanted a job that would allow him to contribute to the household, without leaving his mother alone in the day when his father went to his office. 

“My mother is my best friend, I like to work from home, I talk to her all day, it is a very nice way to work,” Sandhu said. “The Chrome extension tracks activity on the portal, and automatically tracks the time you spend working through Data Hub. The timer on the screen also shows you how much time you have left to complete a task.”

“Human intelligence can control artificial intelligence,” Sandhu added. “Google has improved so much, because of people like us.”