Schedule for: 16w5140 - Models and Algorithms for Crowds and Networks

A welcome drink will be served at the hotel.

Auction theory governs the costly contributions of participants in crowd systems. For example, crowdsourcing contests and badge design have been modeled as all-pay auctions where payment equals effort and only some of the participants receive rewards. This talk will surveys recent results in the design of all-pay auctions for complex environments focusing on two recent results. The first result is on a data-driven approach for counterfactual estimation. It gives a method for estimating the performance of an all-pay auction from the equilibrium bids in another all-pay auction. The second result gives a design for nearly-optimal all-pay auctions in complex environments, such as the weighted set packing (a.k.a., single-minded combinatorial auction) environment.

If you were to design a crowdsourcing marketplace from scratch, how would you do it? Crowdsourcing marketplaces are computationally mediated, enabling a programmer to change everything from reputation feedback to worker collective action to task design. I will discuss best (and worst) practices in the human-centered design of crowdsourcing marketplaces. I base my reflections on my own experience leading 1,500 contributors from around the world in creating a new crowdsourcing marketplace.

A random walk is a natural way to explore a network. We will study the use of uniform random walks to estimate various properties such as the size of the network, average degree, number of triangles, etc. Less obvious random walks can also be designed to do other tasks such as uniformly generating a node or counting network motifs. However, our perspective is that one has to be careful in using random walks for applications.

Crowdsourcing makes it possible for systems to solve problems beyond the current reach of automation alone. However, traditional approaches to crowdsourcing have limited the scope of such systems to those that draw on human intelligence only through offline (non time-dependent) microtasks. This workshop will cover tools and techniques for quickly recruiting workers, routing them to tasks, and keeping them engaged for longer periods of time, so that tasks that require low-latency responses can benefit from human computation. Participants will learn the underlying theory, intuition behind approaches, and specific methods for integrating crowdsourcing into real-time systems in any domain.

YouTube competes with Hollywood as an entertainment channel, and also supplements Hollywood by acting as a distribution mechanism. Twitter has a similar relationship to news media, and Coursera to Universities. But there are no online alternatives for making democratic decisions at large scale as a society. In this talk, we will describe two algorithmic approaches towards large scale decision making that we are exploring: Knapsack Voting and Triadic Consensus. We will also describe our experience running crowdsourced democracy processes in the US, Canada, and Finland. Finally, we will outline several open algorithmic problems in this space.

This presentation shares some of the key findings from a long term, joint research project with Mary Gray that examines workers’ experiences in crowdsourcing-for-hire labor markets. One result from 19 months of ethnographic field work, in both India and the United States across 4 different crowdsourcing platforms, is that despite the designs of crowdsourcing sites to maximize efficiencies through atomized, autonomous workflows, the most active crowdworkers are not the independent workers they are assumed to be. Instead, workers collaborate extensively to address both technical and social needs left open by the platforms they work on. Specifically, crowdworkers collaborate with members of their networks to 1) manage the administrative overhead associated with crowdwork, 2) find lucrative tasks and reputable employers and 3) recreate the social connections and support often associated with brick-and-mortar work environments. We then build on and extend this discovery by mapping the entire communication network of workers on Amazon Mechanical Turk, a leading crowdsourcing platform. We execute a task in which over 10,000 workers from across the globe self-report their communication links to other workers, thereby mapping the communication network among workers. Our results suggest that while a large percentage of workers indeed appear to be independent, there is a rich network topology over the rest of the population. That is, there is a substantial communication network within the crowd.

Graphs naturally represent information in a wide range of disciplines, from social science to biology and transportation engineering. These graphs often span millions or even billions of nodes and interactions between them. Within this deluge of interconnected data, how can we extract useful knowledge, understand the underlying processes, and make interesting discoveries? This tutorial will focus on fundamental problems in large-scale graph mining including ranking, clustering and community detection, summarization, similarity, alignment, pattern discovery, and anomaly detection. It will also present challenges and open problems, and invite discussion of how some of these may be addressed by combining network science with the power of crowds.

The data management community has been historically (and perhaps myopically) focused on processing SQL queries on ever larger datasets : what does this have to do with crowdsourcing? This tutorial will convey how it helps to think of humans as relational data processors for the purpose of designing optimized crowd-powered algorithms and systems, in order to generate or process orders of magnitude more data at the same cost and same accuracy. The tutorial will also cover how the crowdsourcing work done by the data management community fits in with the other communities, its successes and failures, and open questions and potential opportunities for collaboration.

This talk will overview some of the successes in connecting Multi-armed Bandit (MAB) algorithm to crowdsourcing research. With MAB being a powerful tool for decision making and machine learning tasks, there are some salient challenges when we adopt this framework for crowdsourcing studies, including large exploration space, budget constraint, and lack of ground-truth. We will survey several recent results that help overcome the above hurdles. Time permitting, we will go through a couple of examples that use a bandit framework as a reputation system to provide long-term incentives.

Optional Group Outings (300MXN) Leave at 1:30pm SHARP -- back between 5:30-6:30.