User studies are important for many aspects of the design process and involve techniques ranging from informal surveys to rigorous laboratory studies. However, costs involved in engaging users often requires practitioners trade off between sample size, time requirements, monetary costs. Micro-task markets, such as Amazon's Mechanical Turk, offer a potential paradigm large number low Here we investigate utility micro-task market collecting user measurements, discuss considerations developing...

Paid crowd work offers remarkable opportunities for improving productivity, social mobility, and the global economy by engaging a geographically distributed workforce to complete complex tasks on demand at scale. But it is also possible that will fail achieve its potential, focusing assembly-line piecework. Can we foresee future workplace in which would want our children participate? This paper frames major challenges stand way of this goal. Drawing theory from organizational behavior...

Wikipedia's success is often attributed to the large numbers of contributors who improve accuracy, completeness and clarity articles while reducing bias. However, because coordination needed write an article collaboratively, adding costly. We examined how number editors in Wikipedia methods they use affect quality. distinguish between explicit coordination, which plan through communication, implicit a subset structure work by doing majority it. Adding more improved quality only when used...

Wikipedia, a wiki-based encyclopedia, has become one of the most successful experiments in collaborative knowledge building on Internet. As Wikipedia continues to grow, potential for conflict and need coordination increase as well. This article examines growth such non-direct work describes development tools characterize costs Wikipedia. The results may inform design new systems.

Micro-task markets such as Amazon's Mechanical Turk represent a new paradigm for accomplishing work, in which employers can tap into large population of workers around the globe to accomplish tasks fraction time and money more traditional methods. However, typically support only simple, independent tasks, labeling an image or judging relevance search result. Here we present general purpose framework micro-task that provides scaffolding complex human computation require coordination among...

This paper examines the location traces of 489 users a sharing social network for relationships between users' mobility patterns and structural properties their underlying network. We introduce novel set location-based features analyzing context geographic region, including entropy, which measures diversity unique visitors location. Using these features, we provide model predicting friendship two by trails. Our achieves significant gains over simpler models based only on direct co-location...

Cognitive neuroscience aims to map mental processes onto brain function, which begs the question of what "mental processes" exist and how they relate tasks that are used manipulate measure them. This topic has been addressed informally in prior work, but we propose cumulative progress cognitive requires a more systematic approach representing entities being mapped function processes. We describe new open collaborative project provide knowledge base for neuroscience, called Atlas (accessible...

Micro-task markets such as Amazon's Mechanical Turk represent a new paradigm for accomplishing work, in which employers can tap into large population of workers around the globe to accomplish tasks fraction time and money more traditional methods. However, have been primarily used simple, independent tasks, labeling an image or judging relevance search result. Here we present general purpose framework complex interdependent using micro-task markets. We describe our framework, web-based...

Crowdsourced labor markets represent a powerful new paradigm for accomplishing work. Understanding the motivating factors that lead to high quality work could have significant benefits. However, researchers so far found such as increased monetary reward generally increase workers’ willingness accept task or speed at which is completed, but do not improve of We hypothesize intrinsic motivation – framing helping others may succeed in improving output where extrinsic motivators pay not. In this...

Sensemaking in unfamiliar domains can be challenging, demanding considerable user effort to compare different options with respect various criteria. Prior research and our formative study found that people would benefit from reading an overview of information space upfront, including the criteria others previously useful. However, existing sensemaking tools struggle "cold-start" problem — it not only requires significant input previous users generate share these overviews, but such overviews...

Reverts are important to maintaining the quality of Wikipedia. They fix mistakes, repair vandalism, and help enforce policy. However, reverts can also be damaging, especially aspiring editor whose work they destroy. In this research we analyze 400,000 Wikipedia revisions understand effect that had on editors. We seek extent which demotivate users, reducing workforce contributors, versus users improve as encyclopedia Overall find powerfully demotivating, but their net influence is more done...

Extracting useful knowledge from large network datasets has become a fundamental challenge in many domains, scientific literature to social networks and the web. We introduce Apolo, system that uses mixed-initiative approach - combining visualization, rich user interaction machine learning guide incrementally interactively explore data make sense of it. Apolo engages bottom-up sensemaking gradually build up an understanding over time by starting small, rather than big drilling down. also...

Online production groups have the potential to transform way that knowledge is produced and disseminated. One of most widely used forms online wiki, which has been in domains ranging from science education enterprise. We examined development interactions between coordination conflict a sample 6811 wiki groups. investigated influence four mechanisms: intra-article communication, inter-user concentration workgroup structure, policy procedures. also growth conflict, finding density users an...

Detecting and correcting low quality submissions in crowdsourcing tasks is an important challenge. Prior work has primarily focused on worker outcomes or reputation, using approaches such as agreement across workers with a gold standard to evaluate quality. We propose alternative complementary technique that focuses the way rather than products they produce. Our captures behavioral traces from online crowd uses them predict outcome measures quality, errors, likelihood of cheating....

While many organizations turn to human computation labor markets for jobs with black-or-white solutions, there is vast potential in asking these workers original thought and innovation.

Traditional research on leadership in online communities has consistently focused the small set of people occupying roles. In this paper, we use a model shared leadership, which posits that behaviors come from members at all levels, not simply high-level positions. Although every member can exhibit some behavior, different types behavior performed by leaders may be equally effective. This paper investigates how distinct (transactional, aversive, directive and person-focused) legitimacy who...

Crowdsourced labor markets represent a powerful new paradigm for accomplishing work. Understanding the motivating factors that lead to high quality work could have significant benefits. However, researchers so far found such as increased monetary reward generally increase workers’ willingness accept task or speed at which is completed, but do not improve of We hypothesize intrinsic motivation – framing helping others may succeed in improving output where extrinsic motivators pay not. In this...

Consumers conducting comparison shopping, researchers making sense of competitive space, and developers looking for code snippets online all face the challenge capturing information they find later use without interrupting their current flow. In addition, during many learning exploration tasks, people need to externalize mental context, such as estimating how urgent a topic is follow up on, or rating piece evidence "pro" "con," which helps scaffold subsequent deeper exploration. However,...

Wikis are collaborative systems in which virtually anyone can edit anything. Although wikis have become highly popular many domains, their mutable nature often leads them to be distrusted as a reliable source of information. Here we describe social dynamic analysis tool called WikiDashboard aims improve transparency and accountability on Wikipedia articles. Early reactions from users suggest that the increased afforded by interpretation, communication, trustworthiness

Wikipedia is an online encyclopedia which has undergone tremendous growth. However, this same growth made it difficult to characterize its content and coverage. In paper we develop measures map using socially annotated, hierarchical category structure. We introduce a mapping technique that takes advantage of socially-annotated categories while dealing with the inconsistencies noise inherent in distributed way they are generated. The demonstrated through two applications: distribution topics...

Wikipedia has become one of the most important information resources on Web by promoting peer collaboration and enabling virtually anyone to edit anything. However, this mutability also leads many distrust it as a reliable source information. Although there have been attempts at developing metrics help users judge trustworthiness content, is unknown how much impact such measures can system that perceived inherently unstable. Here we examine whether visualization exposes hidden article...

The success of Wikipedia has demonstrated the power peer production in knowledge building. However, unlike many other examples collective intelligence, tasks can be deeply interdependent and may incur high coordination costs among editors. Increasing number editors increases resources available to system, but it also raises coordination. This suggests that dependencies determine whether they benefit from increasing involved. Specifically, we hypothesize adding low-coordination have negative...

Wikipedia is a highly successful example of what mass collaboration in an informal peer review system can accomplish. In this paper, we examine the role that quality contributions, experience contributors and ownership content play decisions over which contributions become part ones are rejected by community. We introduce justify versatile metric for automatically measuring contribution. find little evidence helps avoid rejection. fact, as they gain experience, even more likely to have their...

Crowdsourcing has become a powerful paradigm for accomplishing work quickly and at scale, but involves significant challenges in quality control. Researchers have developed algorithmic control approaches based on either worker outputs (such as gold standards or agreement) behavior task fingerprinting), each approach serious limitations, especially complex creative work. Human evaluation addresses these limitations does not scale well with increasing numbers of workers. We present CrowdScape,...

Though toolkits exist to create complex crowdsourced workflows, there is limited support for management of those workflows. Managing crowd workers and tasks requires significant iteration experimentation on task instructions, rewards, flows. We present CrowdWeaver, a system visually manage work. The supports the creation reuse crowdsourcing computational into integrated flows, manages flow data between tasks, allows tracking notification progress, with real-time modification. describe...