The study of algorithms to automatically answer visual questions currently is motivated by question answering (VQA) datasets constructed in artificial VQA settings. We propose VizWiz, the first goal-oriented dataset arising from a natural setting. VizWiz consists over 31,000 originating blind people who each took picture using mobile phone and recorded spoken about it, together with 10 crowdsourced answers per question. differs many existing because (1) images are captured photographers so...

Common appliances have shifted toward flat interface panels, making them inaccessible to blind people. Although people can label with Braille stickers, doing so generally requires sighted assistance identify the original functions and apply labels. We introduce Facade - a crowdsourced fabrication pipeline help independently make physical interfaces accessible by adding 3D printed augmentation of tactile buttons overlaying panel. users capture photo appliance readily available fiducial marker...

AI technologies have the potential to dramatically impact lives of people with disabilities (PWD). Indeed, improving PWD is a motivator for many state-of-the-art systems, such as automated speech recognition tools that can caption videos who are deaf and hard hearing, or language prediction algorithms augment communication cognitive disabilities. However, widely deployed systems may not work properly PWD, worse, actively discriminate against them. These considerations regarding fairness in...

We introduce the first visual privacy dataset originating from people who are blind in order to better understand their disclosures and encourage development of algorithms that can assist preventing unintended disclosures. It includes 8,862 regions showing private content across 5,537 images taken by people. Of these, 1,403 paired with questions 62\% those directly ask about content. Experiments demonstrate utility this data for predicting whether an image shows information a question asks...

Content creators are instructed to write textual descriptions of visual content make it accessible; yet existing guidelines lack specifics on how about people's appearance, particularly while remaining mindful consequences (mis)representation. In this paper, we report interviews with screen reader users who were also Black, Indigenous, People Color, Non-binary, and/or Transgender their current image description practices and preferences, experiences negotiating theirs others' appearances...

Wearable and contextually aware technologies have great applicability in task guidance systems. Order picking is the of collecting items from inventory a warehouse sorting them for distribution; this process accounts about 60% total operational costs these warehouses. Current practice industry includes paper pick lists pick-by-light We evaluated order assisted by four approaches: head-up display (HUD); cart-mounted (CMD); pick-by-light; list. report accuracy, error types, time, subjective...

The world is full of physical interfaces that are inaccessible to blind people, from microwaves and information kiosks thermostats checkout terminals. Blind people cannot independently use such devices without at least first learning their layout, usually only after labeling them with sighted assistance. We introduce VizLens - an accessible mobile application supporting backend can robustly interactively help nearly any interface they encounter. users capture a photo send it multiple crowd...

The physical constraints of smartwatches limit the range and complexity tasks that can be completed. Despite interface improvements on smartwatches, promise enabling productive work remains largely unrealized. This paper presents WearWrite, a system enables users to write documents from their by leveraging crowd help translate ideas into text. WearWrite dictate tasks, respond questions, receive notifications major edits watch. Using dynamic task queue, receives issued watch user generic...

Disaggregated evaluations of AI systems, in which system performance is assessed and reported separately for different groups people, are conceptually simple. However, their design involves a variety choices. Some these choices influence the results that will be obtained, thus conclusions can drawn; others impacts---both beneficial harmful---that disaggregated evaluation have on including people whose data used to conduct evaluation. We argue deeper understanding enable researchers...

Blind users rely on alternative text (alt-text) to understand an image; however, alt-text is often missing. AI-generated captions are a more scalable alternative, but they miss crucial details or completely incorrect, which may still falsely trust. In this work, we sought determine how additional information could help better judge the correctness of captions. We developed ImageExplorer, touch-based multi-layered image exploration system that allows explore spatial layout and hierarchies...

Existing assistive technologies (AT) often fail to support the unique needs of blind and visually impaired (BVI) people. Thus, BVI people have become domain experts in customizing 'hacking' AT, creatively suiting their needs. We aim understand this behavior depth, how envision creating future DIY personalized AT. conducted a multi-part qualitative study with 12 participants: an interview on uses two-week diary log use cases, scenario-based design session imagine technologies. found that...

The COVID-19 pandemic has forced institutions to rapidly alter their behavior, which typically disproportionate negative effects on people with disabilities as accessibility is overlooked. To investigate these issues, we analyzed Twitter data examine problems surfaced by the crisis. We identified three key domains at intersection of and technology: (i) allocation product delivery services, (ii) transition remote education, (iii) dissemination public health information. found that essential...

Immersive authoring tools allow users to intuitively create and manipulate 3D scenes while immersed in Virtual Reality (VR). Collaboratively designing these is a creative process that involves numerous edits, explorations of design alternatives, frequent communication with collaborators. Version Control Systems (VCSs) help achieve this by keeping track the version history creating shared hub for communication. However, most VCSs are unsuitable managing VR content because their underlying...

Experiments suggest that using head-up displays like Google Glass to support parts picking for distribution results in fewer errors than current processes. Making opaque instead of transparent further improves selection efficiency. The Web extra at http://youtu.be/yUZFaCP6rP4 is a video demonstrating order assisted by display (HUD) faster, has errors, requires less workload, and preferred pick-by-light, paper pick list, cart-mounted (CMD). second http://youtu.be/RApBJ0U3XpI with an three...

The growing accessibility of 3D printing to everyday users has led the rapid adoption, sharing models on sites such as Thingiverse.com, and visions a future in which customization is norm can solve variety real-world problems. However, practice, creating difficult many end simply print created by others. In this paper, we explore specific area model design that challenge for users' measurement. When must conform real world goal once printed, it important precisely specified. We demonstrate...

We introduce Blocks, a mobile application that enables people to co-create AR structures persist in the physical environment. Using end users can collaborate synchronously or asynchronously, whether they are colocated remote. Additionally, be tied location accessed from anywhere. evaluated how used Blocks through series of lab and field deployment studies with over 160 participants, explored interplay between two collaborative dimensions: space time. found participants preferred creating...

Augmented Reality (AR) technology creates new immersive experiences in entertainment, games, education, retail, and social media. AR content is often primarily visual it challenging to enable access non-visually due the mix of virtual real-world content. In this paper, we identify common constituent tasks by analyzing existing mobile applications for iOS, characterize design space that require accessible alternatives. For each major task categories, create prototype alternatives evaluate a...

Smartphone overuse poses risks to people's physical and mental health. However, current intervention techniques mainly focus on explicitly changing screen content (i.e., output) often fail persistently reduce smartphone due being over-restrictive or over-flexible. We present the design implementation of InteractOut, a suite implicit input manipulation that leverage interaction proxies weakly inhibit natural execution common user gestures mobile devices. space for manipulations demonstrate 8...

User identification and differentiation have implications in many application domains, including security, personalization, co-located multiuser systems. In response, dozens of approaches been developed, from fingerprint retinal scans, to hand gestures RFID tags. this work, we propose CapAuth, a technique that uses existing, low-level touchscreen data, combined with machine learning classifiers, provide real-time authentication even users. As proof-of-concept, ran our software on an...

Blind people frequently encounter inaccessible dynamic touchscreens in their everyday lives that are difficult, frustrating, and often impossible to use independently. Touchscreens the only way control everything from coffee machines payment terminals, subway ticket in-flight entertainment systems. Interacting with is difficult non-visually because visual user interfaces change, interactions occur over multiple different screens, it easy accidentally trigger interface actions while exploring...

AI technologies have the potential to dramatically impact lives of people with disabilities (PWD). Indeed, improving PWD is a motivator for many state-of-the-art systems, such as automated speech recognition tools that can caption videos who are deaf and hard hearing, or language prediction algorithms augment communication cognitive disabilities. However, widely deployed systems may not work properly PWD, worse, actively discriminate against them. These considerations regarding fairness in...

Sensing technologies that implicitly and explicitly mediate digital experiences are an increasingly pervasive part of daily living; it is vital to ensure these work appropriately for people with physical disabilities. We conducted on online survey 40 adults disabilities, gathering open-ended descriptions about respondents' a variety sensing systems, including motion sensors, biometric speech input, as well touch gesture systems. present findings regarding the many challenges status quo...

Because smartwatches are worn on the wrist, they do not require users to hold device, leaving at least one hand free engage in other activities. Unfortunately, this benefit is thwarted by typical interaction model of smartwatches; for interactions beyond glancing information or using speech, must utilize their manipulate a touchscreen and/or hardware buttons. In order enable no-touch, wrist-only smartwatch so that can, example, cup coffee while controlling we explore two tilt-based...

A large number of Internet-of-Things (IoT) devices will soon populate our physical environments. Yet, IoT devices' reliance on mobile applications and voice-only assistants as the primary interface limits their scalability expressiveness. Building off classic 'Put-That-There' system, we contribute an exploration design space voice + gesture interaction with spatially-distributed devices. Our decomposes users' commands into two components—selection interaction. We articulate how permutations...

Collaborative document editing tools are widely used in professional and academic workplaces. While these provide basic accessibility support, it is challenging for blind users to gain collaboration awareness that sighted people can easily obtain using visual cues (e.g., who where what). Through a series of co-design sessions with coauthor, we identified the current practices challenges collaborative editing, iteratively designed CollabAlly, system makes accessible users. CollabAlly extracts...