Crowdsourcing has been shown to be effective at solving problems that are beyond the capabilities of current automated approaches. However, current crowdsourcing systems suffer from two main limitations: (i) tasks must first be repackaged for proper display to crowd workers, which generally requires substantial one-off programming effort and corresponding support infrastructure; and (ii) crowds generally participate asynchronously, without a tight feedback loop between workers and their task. This paper considers a new approach to crowd computing that surpasses both limitations by using existing graphical user interfaces and putting the crowd in control of the mouse and keyboard. We introduce Legion, a system that allows end users to easily capture existing GUIs and outsource them for collaborative, real-time control by the crowd.

Legion lets end users leverage crowdsourcing in ways previously not possible. Our original motivation was to provide a quick way of bootstrapping highly-robust, intelligent assistive robots. Such systems usually require significant (and costly) training to work automatically, are prone to errors, and so can often be controlled remotely by experts. We imagined a hybrid system in which robots could operate mostly automatically, but in which new tasks could be crowdsourced on demand for real-time control. Legion supports the flexible control of such existing remote-control interfaces.