Intelligent Control of Crowdsourcing

Crowd-sourcing labor markets (e.g., Amazon Mechanical Turk) are booming, because they enable rapid construction of complex workflows that seamlessly mix human computation with computer automation. Example applications range from photo tagging to audio-visual transcription and interlingual translation. Similarly, workflows on citizen science sites (e.g. GalaxyZoo) have allowed ordinary people to pool their effort and make interesting discoveries. Unfortunately, constructing a good workflow is difficult, because the quality of the work performed by humans is highly variable. Typically, a task designer will experiment with several alternative workflows to accomplish a task, varying the amount of redundant labor, until she devises a control strategy that delivers acceptable performance. Fortunately, this control challenge can often be formulated as an automated planning problem ripe for algorithms from the probabilistic planning and reinforcement learning literature. I describe our recent work on the decision-theoretic control of crowd sourcing and suggest open problems for future research. In particular, I discuss: – The use of partially-observable Markov decision Processes (POMDPs) to control voting on binary-choice questions and iterative improvement workflows. – Decision-theoretic methods that dynamically switch between alternative workflows in a way that improves on traditional (static) A-B testing. – A novel workflow for crowdsourcing the construction of a taxonomy – a challenging problem since it demands a global perspective of the input data when no one worker sees more than a tiny fraction. – Methods for optimizing the acquisition of labeled training data for use in machine learning applications; this an important special case, since data annotation is often crowd-sourced.

Speaker Bios

Daniel S. Weld is Thomas J. Cable / WRF Professor of Computer Science & Engineering and Entrepreneurial Faculty Fellow at the University of Washington. After formative education at Phillips Academy, he received bachelor’s degrees in both Computer Science and Biochemistry at Yale University in 1982. He landed a Ph.D. from the MIT Artificial Intelligence Lab in 1988, received a Presidential Young Investigator’s award in 1989, an Office of Naval Research Young Investigator’s award in 1990, was named AAAI Fellow in 1999 and deemed ACM Fellow in 2005. Dan was a founding editor for the Journal of AI Research, was area editor for the Journal of the ACM, guest editor for Computational Intelligence and Artificial Intelligence, and was Program Chair for AAAI-96. He has published two books and scads of technical papers.

Dan is an active entrepreneur with several patents and technology licenses. He co-founded Netbot Incorporated, creator of Jango Shopping Search (acquired by Excite), AdRelevance, a monitoring service for internet advertising (acquired by Nielsen NetRatings), and data integration company Nimble Technology (acquired by Actuate). Dan is a Venture Partner at the Madrona Venture Group and on the Scientific Advisory Boards of the Allen Institute for Artificial Intelligence, Context Relevant, Madrona, and Spare5.