We hire a pool of Turkers for a large scale AI annotation task

Hello all,

We are Carnegie Mellon University researchers working on the intersection of Natural Language Processing, Computer Vision, and Machine Learning.

We have this new cool task where one person hides Waldo in panoramic images and provides a list of clues to find him, and another one is trying to find Waldo with those clues. Before we run a large scale study we would like to hire a pool of annotators.

So far we ran 2 pilot studies and thanks to Turkers we are able to improve it substantially. We fixed lots of bugs and compensate Turkers time for those bugs. Based on the average task completion time we keep readjusting the reward. For the large scale study, we plan to have a leaderboard for "the best hiders" and "the best finders". Top hiders and finders will receive extra bonuses.

HIT Information:
Requester: Carnegie Mellon 009
Title: Hide Waldo!
Reward: 2.25$
Expected HIT length:  10-15 minutes
Qualifications Required:  HIT Approval Rate > 90, Location is the US/UK

We are looking forward to your feedback! For further information please reply to this thread, DM us in this forum, or email us lti-crowdsourcing-009@andrew.cmu.edu

 

I found the hit but it also requires Masters so I can't do it


FYI
setting the qualifications something like 5,000 approved and HIT approval rate (%) is not less than 99 often gets better results than the master qual just remeber Amazon rounds down so make sure you include 99%. In other words if it is set for only >99% that is the same as requiring100%

 

Agreed. Masters is something that is limiting your potential pool of workers.

 

Just curious are you guys associated w/ the HCI lab at CMU?

 

A few folks got it just a week ago lol. Still limits the worker pool way too much to use for a large scale study IMO so your second half is probably correct but they do still replenish the pool at least a bit nowadays.

 

Thanks all! We are not pros like HCI people, new in the annotation stage, that's why we are making these rookie mistakes

The next batch will not have masters and we will readjust qualifications!

 

Oh, don't worry about removing masters, I've got it, and I'm all you need j/k

HIT is laid out well and pretty straightforward. Can't think of many improvements at the moment.

 

I'll be there once Jedi status is lifted.

 

Folks, we just published another batch of HITs, now we do not have masters requirement. We are looking forward to working with you and hear your feedback!

 

It looks like there's some wrong with the qualification.

http://prntscr.com/nh8hyw

 

what did you search for? I searched the requestor name and the hit title and it's not showing up for me

 

I just searched "mellon". Make sure you have "I'm qualified to work on" unchecked in your filter.

 

hmmm

 

Could you check again?

 

Was able to catch it now. Thank you.

 

still not showing up for me

 

I cannot find it and "I'm qualified to work on" unchecked in my filter.

 

@InfiniteChanges must have the magic touch this morning

 

I was able to find, catch, and complete the task... but man was that a pain on such a smaller work area.