Americans rely on online reviews for selecting everything from an Airbnb room to an employer, but these reviews are often skewed. That’s because people whose opinions fall at the extreme ends of the spectrum tend to write reviews, according to the University of Pennsylvania’s Ioana Marinescu, University of Chicago Harris School of Public Policy postdoctoral scholar Nadav Klein, and Glassdoor’s Andrew Chamberlain and Morgan Smart.
But they suggest ways to elicit reviews from more people in the middle of the spectrum, which could moderate overall ratings and reduce bias.
The researchers analyzed data from Glassdoor, the largest employer-review website in the United States. Potential employees use the reviews to guide decisions about companies they may want to work for.
Researchers suggest platforms such as eBay should consider ranking search results so that products from high-quality sellers are prominently displayed.
Making a Bid for More-Accurate eBay Seller ProfilesGlassdoor allows visitors to see three pieces of content before asking them to leave an employer review (or some other information, such as their salary) in order to access more content. This incentive encourages reviewers who might not otherwise give their opinions, according to the researchers.
On the basis of more than 188,000 reviews between 2013 and 2016, the researchers find that opinions submitted without any incentive were more polarized than those from people writing in response to a minor nudge. Reviewers who offered up their opinions freely were 1 percentage point more likely to post a one-star review and 4 percentage points more likely to give five stars. Reviews provided in return for an incentive had a moderating effect on ratings.
The differences matter, the researchers write. Rankings of the most favorable employers differed significantly based on whether the reviews were voluntary or incentivized. For example, among incentivized reviewers, consulting was a more desirable industry than advertising and marketing. But the opposite was true based on voluntary reviews, which were skewed by the reviewers’ strong sentiments.
“By affecting the perception of employer desirability, the likely bias in voluntary online employer reviews has the potential to affect important life choices,” the researchers write.
Still, including incentivized reviews isn’t the same as polling all employees or taking a truly random sampling. So the team set up experiments using Amazon Mechanical Turk—an online platform that pays users approximately 20 cents for writing reviews, taking surveys, and providing other market-research feedback.
Participants in the study were asked to rate their employers. Some were allowed to decline and still receive payment, while others were told they needed to provide a review or forfeit the cash.
The resulting “forced” reviews closely aligned with Glassdoor’s incentivized reviews and were more moderate. Reviews from people who voluntarily gave their opinion when they could have declined and still received payment resulted in more negative ratings and ranked employers just over half a star lower than those provided in response to an incentive.
The researchers then examined what would motivate more people to leave reviews. While the analyses and experiments suggest that money works, the price has to be right. Offering MTurk users a 75 percent higher payment considerably increased participation. But a 25 percent raise didn’t.
Appealing to a sense of duty raised the number of reviews too. More participants left reviews when told their opinions would help others make informed decisions, but the increase was not as big as it was for the group offered a higher monetary reward, the researchers find.
Adequate incentives will lure in more reviewers and moderate overall ratings, and it doesn’t necessarily have to cost anything, the researchers find. But for the best results, it helps to sweeten the deal.
Ioana Marinescu, Nadav Klein, Andrew Chamberlain, and Morgan Smart, “Incentives Can Reduce Bias in Online Reviews,” Working paper, March 2018.
Conventional economic analysis of slavery overlooks the costs imposed upon the people who were enslaved.
Emancipation May Have Generated the Largest Economic Gains in US History
Economic experts consider what a nationwide ban on the app would mean for innovation and for the rest of the tech industry.
What Would Banning TikTok Mean for the US Economy?
Machine learning can help identify new hypotheses to test.
What’s the Next Big Question in the Social Sciences? Ask an AlgorithmYour Privacy
We want to demonstrate our commitment to your privacy. Please review Chicago Booth's privacy notice, which provides information explaining how and why we collect particular information when you visit our website.