Abstract: Crowdsourcing is the practice of engaging a
crowd or a group of people for a common task to be done.
It is an easy and efficient way for tasks to be done faster
with good accuracy and can be used in applications like
data collection, recommendation systems, social studies etc.
It also plays an important role in machine learning where
large amount of labeled data is needed to train the models.
In the process of labeling data sets, labelers with more qual-
ity should be selected to reduce noisy labels. A number of
offline and online approaches exist for this labeler selection
problem and some of them are discussed here.
Crowdsourcing is the process by which a work being
done by a group of people and the people get rewarded for
their work. The key idea behind crowdsourcing is to dis-
tribute a task(unlabelled data) to a large number of people
and aggregate the results obtained from them. A task can be
anything like labeling an image, rating a movie etc.
Main components of a crowdsourcing system is a task
master and some workers(labelers). Task master posts a task
and interested workers approach to do the task. The work-
ers, in turn will get paid by the task master. One famous
example for crowdsourcing system is Amazon Mechanical
Turk(AMT). It is a crowdsourcing internet marketplace for
work that requires human intelligence.
Crowdsourcing works based on the principle of more
heads are better than one. Because of the involvement of
more people with different skills and talents, good quality
results can be achieved for the tasks. Crowdsourcing plays
an important role in machine learning applications also. In
machine learning, huge amount of labeled data is needed to
train a model. Such huge amount of labeled data can be col-
lected via crowdsourcing.
The major problem concerned with crowdsourcing is the
quality of labeled data obtained from labelers. This is be-
cause some of the labelers assigned to a task may be showing
irresponsible behavior and some of them may be having low
degree of expertise. As a result, the obtained labels become
noisy and contain erroneous answers. Hence, the selection of
labelers should be done carefully so that the quality of labels
can be improved.
The problem of finding the best and trusted labelers is
called as ‘labeler selection problem’. Several techniques
have been proposed to solve the labeler selection problem.
The purpose of this paper is to survey a number of the more
promising of those techniques.
2 Literature Review
2.1 Who Moderates the Moderators? Crowdsourcing
Abuse Detection in User-Generated Content 1
User generated content(UGC) can be any form of posts,
comments, blogs posted by users in websites. These contents
may sometimes contain spam and abuse. Such abusive con-
tents should be recognised and eliminated from web pages.
This process is called as moderation.
To cope with large amount of contents to be moderated,
the authors suggest to use crowdsourced ratings for modera-
tion of UGC. That is, the viewers of websites will be allowed
to label the content as good or bad. By aggregating their rat-
ings, abusive contents can be detected and such contents can
be eliminated from the websites. But it is not necessary that
all the raters are honest and will give accurate ratings. So
trusted raters should be selected in order to obtain correct
The algorithm proposed in this paper works based on
the assumption that the identity of a single good or trusted
rater is known. It means that this trusted rater will rate the
content accurately almost all the time. Hence, by comparing
the labels obtained from other raters with that of the trusted
person, honest and good raters can be determined.
Limitation of this approach is that it is an offline algo-
rithm. The process of finding best raters is done first and the
newly arriving content is given to this best set of raters. But
it does not update the accuracy of raters based on each ar-
riving contents to be moderated. Hence, this approach is not
adaptive. Also, the elimination of bad raters is done as post
processing, i.e, after the rating is done by all raters. If most
of these labels are noisy, then time and resources has been
2.4 An Online Learning Approach to Improving the
Quality of Crowd-Sourcing 4
In this paper, the authors introduce an online learning
framework to solve the labeler selection problem whereby
the labeler quality is updates as tasks are assigned and per-
formed. It is thus adaptive to newly arrived tasks because,
the accuracy of labeler is updated on each task arrival. This
approach does not require any reference label set or ground-
truth for checking the correctness of the label. Instead of
using ground-truth information, they use weighted majority
rule for inferring the true label.
It consists of two steps namely exploration and exploita-
tion. A condition will be checked to determine whether ex-
ploration or exploitation is to be conducted. A set of tasks are
designated as testers and they are assigned repeatedly to each
labeler for estimating his labeling quality. The exploration
phase is entered if there is no enough number of testers or if
all the testers have not been tested enough number of times.
In the exploration phase, either an old tester task or the new
arrived task is given to the labelers. Weighted majority rule
is applied over the collected labels to infer the true label. The
accuracy of each labeler is the ratio of number of times his
label matches with the true label to the total number of tasks
assigned to him. It is updated again and again on each new
task arrival and the algorithm over time learns the best set of
Labelers who always conflict with others in their labels
are eliminated. Also, same task is given to same person to
check the consistency of his labels and inconsistent labelers
In the exploitation phase, the algorithm selects the best
set of labelers based on current quality estimates to label the
The limitation of this approach lies in the fact that it does
not consider the context of the arriving task and the quality
of labelers in different contexts. Each person will be having
knowledge in different domains. A person receiving a task
under a particular context in which he has less knowledge
can not give correct label even if he is having high accuracy
estimate. Due to this reason, there are chances for getting
labels having low quality.
Crowdsourcing is being used in a variety of applications
to get good quality results faster. Labeler selection should be
done carefully to obtain accurate output from crowdsourc-
ing. There are different offline and online approaches that
are used to select best set of labelers and thereby improving
the label quality. Some of these approaches were detailed in
moderators?: Crowdsourcing abuse detection in user-
generated content,” in Proc. 12th ACM Conf. Electron.
Commerce, New York, NY, USA, 2011, pp. 167176.
Efficient Crowdsourcing for Multi-class Labeling 2
In this paper, the authors try to increase the reliability of
labels by utilising the principle of redundancy. It means that
instead of giving one task to one person and trusting the label
given by him which could be incorrect, each task is given to
multiple workers. More accurate answers can be achieved
if more redundancy is introduced. Answer for a task is then
computed based on majority rule. That is, the label for which
majority of workers have been voted is selected as the correct
label. Based on these true labels, the accuracy of each labeler
The authors develop an algorithm for deciding which
tasks to be assigned to which workers, and estimating the an-
swers to the tasks from noisy answers collected from those
assigned workers. The algorithm is based on low-rank ap-
proximation of weighted adjacency matrix for a random reg-
ular bipartite graph, weighted according to the answers pro-
vided by the workers.
The disadvantage of this approach is that the task as-
signment is done in a one-shot fashion. It means that all
tasks are initially given to workers and once all answers are
collected, true labels are estimated. The algorithm does not
update the labeler accuracy on each task arrival and hence it
is non adaptive to newly arrived tasks. In that case, valuable
time and resources have been wasted if most of the labels
obtained are incorrect.
Here, an online learning approach is used for the task
assignment problem. That is, the tasks are given sequentially
and the accuracy of labeler is updated based on each task ar-
rival. The authors propose a method for determining which
task should be given to which person according to their ac-
curacy. It consists of two steps namely exploration and ex-
ploitation. This approach uses ground truth labels for infer-
ring correct labels. Ground truth labels are labels for some
tasks which are already known. In the exploration phase,
tasks with known ground truth labels are given to labelers
as one by one. By comparing the collected labels with these
ground truth labels, labeler accuracy is estimated and it is up-
dated based on each assigned task. The result of exploration
phase will be labelers with their corresponding accuracies.
During the exploitation phase, labelers with high accu-
racies are allowed to label further tasks. The limitation of
this approach is that it needs ground truth labels for some
tasks which might not be available always.