Experience froman operational Map-Reduce cluster reveals that outliers significantly prolong job completion. The causes for include run-time contention processor, memory and other resources, disk failures, varying bandwidth congestion along network paths and, imbalance in task workload. We present Mantri, a system monitors tasks culls using cause- resource-aware techniques. Mantri's strategies restarting outliers, network-aware placement of protecting outputs valuable tasks. Using real-time...

Video analytics will drive a wide range of applications with great potential to impact society. A geographically distributed architecture public clouds and edges that extend down the cameras is only feasible approach meeting strict real-time requirements large-scale live video analytics.

Tasks in modern data parallel clusters have highly diverse resource requirements, along CPU, memory, disk and network. Any of these resources may become bottlenecks hence, the likelihood wasting due to fragmentation is now larger. Today's schedulers do not explicitly reduce fragmentation. Worse, since they only allocate cores that ignore (disk network) can be over-allocated leading interference, failures hogging or memory could been used by other tasks. We present Tetris, a cluster scheduler...

Applying deep convolutional neural networks (NN) to video data at scale poses a substantial systems challenge, as improving inference accuracy often requires prohibitive cost in computational resources. While it is promising balance resource and by selecting suitable NN configuration (e.g., the resolution frame rate of input video), one must also address significant dynamics configuration's impact on analytics accuracy. We present Chameleon, controller that dynamically picks best...

To improve data availability and resilience MapReduce frameworks use file systems that replicate uniformly. However, analysis of job logs from a large production cluster shows wide disparity in popularity. Machines racks storing popular content become bottlenecks; thereby increasing the completion times jobs accessing this even when there are machines with spare cycles cluster. address problem, we present Scarlett, system replicates blocks based on their By accurately predicting popularity...

Organizations deploy a hierarchy of clusters - cameras, private clusters, public clouds for analyzing live video feeds from their cameras. Video analytics queries have many implementation options which impact resource demands and accuracy outputs. Our objective is to select the "query plan" implementations (and knobs) place it across merge common components maximize average query accuracy. This challenging task, because we consider multi-resource (network compute) constraints in hierarchical...

Low latency analytics on geographically distributed datasets (across datacenters, edge clusters) is an upcoming and increasingly important challenge. The dominant approach of aggregating all the data to a single datacenter significantly inflates timeliness analytics. At same time, running queries over geo-distributed inputs using current intra-DC frameworks also leads high query response times because these cannot cope with relatively low variable capacity WAN links. We present Iridium,...

Tasks in modern data parallel clusters have highly diverse resource requirements, along CPU, memory, disk and network. Any of these resources may become bottlenecks hence, the likelihood wasting due to fragmentation is now larger. Today's schedulers do not explicitly reduce fragmentation. Worse, since they only allocate cores that ignore (disk network) can be over-allocated leading interference, failures hogging or memory could been used by other tasks. We present Tetris, a cluster scheduler...

Low latency analytics on geographically distributed datasets (across datacenters, edge clusters) is an upcoming and increasingly important challenge. The dominant approach of aggregating all the data to a single datacenter significantly inflates timeliness analytics. At same time, running queries over geo-distributed inputs using current intra-DC frameworks also leads high query response times because these cannot cope with relatively low variable capacity WAN links. We present Iridium,...

Mobile devices are increasingly equipped with multiple network interfaces complementary characteristics. In particular, the Wi-Fi interface has high throughput and transfer power efficiency, but its idle consumption is prohibitive. this paper we present, Blue-Fi, a sytem that predicts availability of connectivity by using combination bluetooth contact-patterns cell-tower information. This allows device to intelligently switch on only when there available, thus avoiding long periods in state...

In big data analytics, timely results, even if based on only part of the data, are often good enough. For this reason, approximation jobs, which have deadline or error bounds and require a subset their tasks to complete, projected dominate workloads. Straggler an important hurdle when designing approximate analytic frameworks, widely adopted approach deal with them is speculative execution. paper, we present GRASS, carefully uses speculation mitigate impact stragglers in jobs. GRASS's design...

Given the well-known tradeoffs between fairness, performance, and efficiency, modern cluster schedulers often prefer instantaneous fairness as their primary objective to ensure performance isolation users groups. However, instantaneous, short-term convergence does not result in noticeable long-term benefits. Instead, we propose an altruistic, approach, CARBYNE, where jobs yield fractions of allocated resources without impacting own completion times. We show that leftover collected via...

As clusters continue to grow in size and complexity, providing scalable predictable performance is an increasingly important challenge. A crucial roadblock achieving stragglers, i.e., tasks that take significantly longer than expected run. At this point, speculative execution has been widely adopted mitigate the impact of stragglers. However, speculation mechanisms are designed operated independently job scheduling when, fact, a copy task direct on resources available for other jobs. In...

Straggler tasks continue to be a major hurdle in achieving faster completion of data intensive applications running on modern data-processing frameworks. Existing straggler mitigation techniques are inefficient due their reactive and replicative nature -- they rely wait-speculate-re-execute mechanism, thus leading delayed detection resource utilization. proactive also over-utilize resources replication. modeling-based approaches hard for production-level adoption modeling errors. We present...

Large volumes of video are continuously recorded by cameras deployed for traffic control and surveillance with the goal answering after fact queries such as: identify frames objects certain classes (cars, bags) from many days video. Current systems processing on large datasets incur either high cost at ingest time or latency query time. We present Focus, a system providing both low-cost low-latency querying datasets. Focus' architecture flexibly effectively divides work between At (on live...

Interactive real-time streaming applications such as audio-video conferencing, online gaming and app streaming, place stringent requirements on the network in terms of delay, jitter, packet loss. Many these inherently involve client-to-client communication, which is particularly challenging since performance need to be met while traversing public wide-area (WAN). This different from typical situation cloud-to-client where WAN can often bypassed by moving a communication end-point cloud...

Recent work has made the case for geo-distributed analytics, where data collected and stored at multiple datacenters edge sites world-wide is analyzed in situ to drive operational management decisions. A key issue such systems ensuring low response times analytics queries issued against data. central determinant of time query execution plan (QEP). Current optimizers do not consider network when deriving QEPs, which a drawback as are connected via WAN links with heterogeneous modest...

Mobile devices are increasingly equipped with multiple network interfaces: Wireless Local Area Network (WLAN) interfaces for local connectivity and Wide (WWAN) wide-area connectivity. The WWAN typically provides much wider coverage but lower speeds than the WLAN. To address this dichotomy, we present COMBINE, a system collaborative downloading wherein that within WLAN range pool together their links, significantly increasing effective speed available to them.

Providing timely results in the face of rapid growth data volumes has become important for analytical frameworks. For this reason, frameworks increasingly operate on only a subset input data. A key property such sampling is that combinatorially many subsets are present. We present KMN, system leverages these choices to perform data-aware scheduling, i.e., minimize time taken by tasks read their inputs, DAG tasks. KMN not uses co-locate with but also percolates combinatorial downstream...