iPhones can now automatically recognize and label buttons and UI features for blind users

Apple has always gone out of its way to build features for users with disabilities, and VoiceOver on iOS is an invaluable tool for anyone with a vision impairment — assuming every element of the interface has been manually labeled. But the company just unveiled a brand new feature that uses machine learning to identify and label every button, slider and tab automatically.

Screen Recognition, available now in iOS 14, is a computer vision system that has been trained on thousands of images of apps in use, learning what a button looks like, what icons mean and so on. Such systems are very flexible — depending on the data you give them, they can become expert at spotting cats, facial expressions or, as in this case, the different parts of a user interface.

The result is that in any app now, users can invoke the feature and a fraction of a second later every item on screen will be labeled. And by “every,” they mean every — after all, screen readers need to be aware of every thing that a sighted user would see and be able to interact with, from images (which iOS has been able to create one-sentence summaries of for some time) to common icons (home, back) and context-specific ones like “…” menus that appear just about everywhere.

The idea is not to make manual labeling obsolete — developers know best how to label their own apps, but updates, changing standards and challenging situations (in-game interfaces, for instance) can lead to things not being as accessible as they could be.

Fable aims to make disability-inclusive design as simple as a service

I chatted with Chris Fleizach from Apple’s iOS accessibility engineering team, and Jeff Bigham from the AI/ML accessibility team, about the origin of this extremely helpful new feature. (It’s described in a paper due to be presented next year.)

Image Credits: Apple

“We looked for areas where we can make inroads on accessibility, like image descriptions,” said Fleizach. “In iOS 13 we labeled icons automatically — Screen Recognition takes it another step forward. We can look at the pixels on screen and identify the hierarchy of objects you can interact with, and all of this happens on device within tenths of a second.”

The idea is not a new one, exactly; Bigham mentioned a screen reader, Outspoken, which years ago attempted to use pixel-level data to identify UI elements. But while that system needed precise matches, the fuzzy logic of machine learning systems and the speed of iPhones’ built-in AI accelerators means that Screen Recognition is much more flexible and powerful.

It wouldn’t have been possible just a couple of years ago — the state of machine learning and the lack of a dedicated unit for executing it meant that something like this would have been extremely taxing on the system, taking much longer and probably draining the battery all the while.

But once this kind of system seemed possible, the team got to work prototyping it with the help of their dedicated accessibility staff and testing community.

“VoiceOver has been the standard-bearer for vision accessibility for so long. If you look at the steps in development for Screen Recognition, it was grounded in collaboration across teams — Accessibility throughout, our partners in data collection and annotation, AI/ML, and, of course, design. We did this to make sure that our machine learning development continued to push toward an excellent user experience,” said Bigham.

It was done by taking thousands of screenshots of popular apps and games, then manually labeling them as one of several standard UI elements. This labeled data was fed to the machine learning system, which soon became proficient at picking out those same elements on its own.

It’s not as simple as it sounds — as humans, we’ve gotten quite good at understanding the intention of a particular graphic or bit of text, and so often we can navigate even abstract or creatively designed interfaces. It’s not nearly as clear to a machine learning model, and the team had to work with it to create a complex set of rules and hierarchies that ensure the resulting screen reader interpretation makes sense.

The new capability should help make millions of apps more accessible, or just accessible at all, to users with vision impairments. You can turn it on by going to Accessibility settings, then VoiceOver, then VoiceOver Recognition, where you can turn on and off image, screen and text recognition.

It would not be trivial to bring Screen Recognition over to other platforms, like the Mac, so don’t get your hopes up for that just yet. But the principle is sound, though the model itself is not generalizable to desktop apps, which are very different from mobile ones. Perhaps others will take on that task; the prospect of AI-driven accessibility features is only just beginning to be realized.

from iPhone – TechCrunch https://ift.tt/3lIOp6w

Android’s winter update adds new features to Gboard, Maps, Books, Nearby Share and more

Google announced this morning Android phones will receive an update this winter that will bring some half-dozen new features to devices, including improvements to apps like Gboard, Google Play Books, Voice Access, Google Maps, Android Auto, and Nearby Share. The release is the latest in a series of update bundles that now allow Android devices to receive new features outside of the usual annual update cycle.

The bundles may not deliver Android’s latest flagship features, but they offer steady improvements on a more frequent basis.

One of the more fun bits in the winter update will include a change to “Emoji Kitchen,” the feature in the Gboard keyboard app that lets users combine their favorite emoji to create new ones that can be shared as customized stickers. To date, users have remixed emoji over 3 billion times since the feature launched earlier this year, Google says. Now, the option is being expanded. Instead of offering hundreds of design combinations, it will offer over 14,000. You’ll also be able to tap two emoji to see suggested combinations or double tap on one emoji to see other suggestions.

Image Credits: Google

This updated feature had been live in the Gboard beta app, but will now roll out to Android 6.0 and above devices in the weeks ahead.

Another update will expand audiobook availability on Google Play Books. Now, Google will auto-generate narrations for books that don’t offer an audio version. The company says it worked with publishers in the U.S. and U.K. to add these auto-narrated books to Google Play Books. The feature is in beta but will roll out to all publishers in early 2021.

An accessibility feature that lets people use and navigate their phone with voice commands, Voice Access, will also be improved. The feature will soon leverage machine learning to understand interface labels on devices. This will allow users to refer to things like the “back” and “more” buttons, and many others by name when they are speaking.

The new version of Voice Access, now in beta, will be available to all devices worldwide running Android 6.0 or higher.

An update for Google Maps will add a new feature to one of people’s most-used apps.

In a new (perhaps Waze-inspired) “Go Tab,” users will be able to more quickly navigate to frequently visited places — like a school or grocery store, for example — with a tap. The app will allow users to see directions, live traffic trends, disruptions on the route, and gives an accurate ETA, without having to type in the actual address. Favorite places — or in the case of public transit users, specific routes — can be pinned in the Go Tab for easy access. Transit users will be able to see things like accurate departure and arrival times, alerts from the local transit agency, and an up-to-date ETA.

Image Credits: Google

One potentially helpful use case for this new feature would be to pin both a transit route and driving route to the same destination, then compare their respective ETAs to pick the faster option.

This feature is coming to both Google Maps on Android as well as iOS in the weeks ahead.

Android Auto will expand to more countries over the next few months. Google initially said it would reach 36 countries, but then updated the announcement language as the timing of the rollout was pushed back. The company now isn’t saying how many countries will gain access in the months to follow or which ones, so you’ll need stay tuned for news on that front.

Image Credits: Google

The final change is to Nearby Share, the proximity-based sharing feature that lets users share things like links, files, photos and and more even when they don’t have a cellular or Wi-Fi connection available. The feature, which is largely designed with emerging markets in mind, will now allow users to share apps from Google Play with people around them, too.

To do so, you’ll access a new “Share Apps” menu in “Manage Apps & Games” in the Google Play app. This feature will roll out in the weeks ahead.

Some of these features will begin rolling out today, so you may receive them earlier than a timeframe of several “weeks,” but the progress of each update will vary.

from Android – TechCrunch https://ift.tt/2Jnppof
via IFTTT

iPhones can now automatically recognize and label buttons and UI features for blind users

Apple has always gone out of its way to build features for users with disabilities, and Voiceover on iOS is an invaluable tool for anyone with a vision impairment — assuming every element of the interface has been manually labeled. But the company just unveiled a brand new feature that uses machine learning to identify and label every button, slider, and tab automatically.

Screen Recognition, available now in iOS 14, is a computer vision system that has been trained on thousands of images of apps in use, learning what a button looks like, what icons mean, and so on. Such systems are very flexible — depending on the data you give them, they can become expert at spotting cats, facial expressions, or as in this case the different parts of a user interface.

The result is that in any app now, users can invoke the feature and a fraction of a second later every item on screen will be labeled. And by “every,” they mean every — after all, screen readers need to be aware of every thing that a sighted user would see and be able to interact with, from images (which iOS has been able to create one-sentence summaries of for some time) to common icons (home, back) and context-specific ones like “…” menus that appear just about everywhere.

The idea is not to make manual labeling obsolete — developers know best how to label their own apps, but updates, changing standards, and challenging situations (in-game interfaces, for instance) can lead to things not being as accessible as they could be.

Fable aims to make disability-inclusive design as simple as a service

I chatted with Chris Fleizach from Apple’s iOS accessibility engineering team, and Jeff Bigham from the AI/ML accessibility team, about the origin of this extremely helpful new feature. (It’s described in a paper due to be presented next year.)

“We looked for areas where we can make inroads on accessibility, like image descriptions,” said Fleizach. “In iOS 13 we labeled icons automatically – Screen Recognition takes it another step forward. We can look at the pixels on screen and identify the hierarchy of objects you can interact with, and all of this happens on device within tenths of a second.”

The idea is not a new one, exactly; Bigham mentioned a screen reader, Outspoken, which years ago attempted to use pixel-level data to identify UI elements. But while that system needed precise matches, the fuzzy logic of machine learning systems and the speed of iPhones’ built-in AI accelerators means that Screen Recognition is much more flexible and powerful.

It wouldn’t have been possibly just a couple years ago — the state of machine learning and the lack of a dedicated unit for executing it meant that something like this would have been extremely taxing on the system, taking much longer and probably draining the battery all the while.

But once this kind of system seemed possible, the team got to work prototyping it with the help of their dedicated accessibility staff and testing community.

“VoiceOver has been the standard bearer for vision accessibility for so long. If you look at the steps in development for Screen Recognition, it was grounded in collaboration across teams — Accessibility throughout, our partners in data collection and annotation, AI/ML, and, of course, design. We did this to make sure that our machine learning development continued to push toward an excellent user experience,” said Bigham.

It was done by taking thousands of screenshots of popular apps and games, then manually labeling them as one of several standard UI elements. This labeled data was fed to the machine learning system, which soon became proficient at picking out those same elements on its own.

It’s not as simple as it sounds — as humans, we’ve gotten quite good at understanding the intention of a particular graphic or bit of text, and so often we can navigate even abstract or creatively designed interfaces. It’s not nearly as clear to a machine learning model, and the team had to work with it to create a complex set of rules and hierarchies that ensure the resulting screen reader interpretation makes sense.

The new capability should help make millions of apps more accessible, or just accessible at all, to users with vision impairments. You can turn it on by going to Accessibility settings, then VoiceOver, then VoiceOver Recognition, where you can turn on and off image, screen, and text recognition.

It would not be trivial to bring Screen Recognition over to other platforms, like the Mac, so don’t get your hopes up for that just yet. But the principle is sound, though the model itself is not generalizable to desktop apps, which are very different from mobile ones. Perhaps others will take on that task; the prospect of AI-driven accessibility features is only just beginning to be realized.

from Apple – TechCrunch https://ift.tt/3lIOp6w

Sight Tech Global day 2 is live! Hear from Apple, Waymo, Microsoft, Sara Hendren and Haben Girma

Day 2 for the virtual event Sight Tech Global is streaming on TechCrunch from 8 a.m. PST to 12:30. The event looks at how AI-based technologies are rapidly changing the field of accessibility, especially for blind people and those with low vision. Today’s programming includes top accessibility product and technology leaders from Apple, Waymo, Microsoft and Google, plus sessions featuring disability rights lawyer Haben Girma and author and designer Sara Hendren. Check out the event’s full agenda.

The Sight Tech Global project aims to showcase the remarkable community of technologists working on accessibility-related products and platforms. It is a project of the nonprofit Vista Center for the Blind and Visually Impaired, which is based in Silicon Valley.

This year’s event sponsors include: Waymo, Verizon Media, TechCrunch, Ford, Vispero, Salesforce, Mojo Vision, iSenpai, Facebook, Ability Central, Google, Microsoft, Wells Fargo, Amazon, Eyedaptic, Verizon 5G, Humanware, APH, and accessiBe. Our production partners: Cohere Studio (design),  Sunol Media Group (video production), Fable (accessibility crowd testing), Clarity Media (speaker prep), Be My Eyes (customer service), 3Play and Vitac  (captioning).

from Apple – TechCrunch https://ift.tt/39LvOEa

Google now lets anyone contribute to Street View using AR and an app

An update to Google’s Street View app on Android will now let anyone contribute their photos to help enhance Google Maps, the company announced this morning. Using a “connected photos” tool in the new version of the Street View app, users are able to record a series of images as they move down the street or a path. The feature requires an ARCore-compatible Android device, and for the time being, will only support image capture and upload in select geographic regions.

ARCore is Google’s platform for building augmented reality experiences. It works by allowing the phone to sense its environment, including the size and location of all types of surfaces, the position of the phone in relation to the world around it, and the lighting conditions of the environment. This is supported on a variety of Android devices running Android 7.0 (Nougat) or higher.

Meanwhile, Google’s Street View app has been around for half a decade. Initially, it was designed to allow users to share their own panoramic photos to improve the Google Maps experience. But as phones have evolved, so has the app.

The updated version of the Street View app allows users to capture images using ARCore — the same AR technology Google users for its own Live View orientation experiences in Maps, which helps phones “see” various landmarks to help users get their bearings.

After the images are published in the Street View app, Google will then automatically rotate, position and create a series of connected photos using those images, and put them in the correct place on Google Maps so others can see them.

It will also use the same privacy controls on these contributed photos as are offered on its own Street View images (the ones it captured by driving the Street View car around). This include blurring people’s faces and license plates, and allowing users to report imagery and other content for review, if needed.

Image Credits: Google

The new system of connected photos won’t be as polished as Google’s own Street View images, but it does make the ability to publish to Street View more accessible. Now, the image capturing process no longer requires a 360-degree camera or other equipment mounted to a top of car, for example. And that means users who live in more remote regions will be able to contribute to Street View, without needing anything more than a supported Android phone and internet connection.

Google says it will still default to showing its own Street View imagery when it’s available, which will be indicated with a solid blue line. But in the case where there’s no Street View option, the contributed connected photos will appear in the Street View layer as a dotted blue line instead.

Image Credits: Google

The company will also use the data in the photos to update Google Maps with the names and addresses of businesses that aren’t already in the system, including their posted hours, if that’s visible on a store sign, for instance.

During early tests, users captured photo using this technology in Nigeria, Japan and Brazil.

Today, Google says it’s officially launching the connected photos feature in beta in the Street View app. During this public beta period, users will be able to try the feature in Toronto, Canada, New York, NY and Austin, TX, along with Nigeria, Indonesia and Costa Rica. More regions will be supported in the future as the test progresses, Google says.

from Android – TechCrunch https://ift.tt/3mDSD0b
via IFTTT

Daily Crunch: Apple announces its best apps of 2020

Apple releases its annual best apps list, a self-driving truck startup raises $350 million and the BioNTech/Pfizer COVID-19 vaccine gets emergency approval in the United Kingdom. This is your Daily Crunch for December 2, 2020.

The big story: Apple announces its best apps of 2020

There were different winners — all selected by App Store editors — for different devices. Home workout app Wakeout! was named the iPhone App of the Year, Disney+ was the Apple TV App of the Year and the productivity app Fantastical was the Mac App of the Year. As for the iPad App of the Year, it went to perhaps the most obvious choice: Zoom.

As far as user popularity goes, Apple said that Zoom was the biggest free iPhone app, followed by TikTok and Disney+ (which must qualify as free on a technicality), while the most popular free iPhone game was Among Us.

The tech giants

Loon’s stratospheric balloons are now teaching themselves to fly better thanks to Google AI — Alphabet’s Loon has been using algorithmic processes to optimize the flight of its stratospheric balloons for years, but the company is now deploying a new navigation system.

Apple’s MagSafe Duo charger is now available — The MagSafe Duo appeared yesterday on Apple’s own store and has delivery estimates as soon as this week.

Google says its News Showcase will add free access to paywalled stories — So far, Google News Showcase has launched in countries including Germany, Brazil, Argentina, Canada, France, U.K. and Australia.

Startups, funding and venture capital

Self-driving trucks startup TuSimple raises $350M from US rail, retail and freight giants — TuSimple was one of the first autonomous trucking startups to emerge in what has become a small-yet-bustling industry.

Virta Health’s behavioral diabetes treatment service is now worth over $1B — Virta aims to reverse the presence of type 2 diabetes and other chronic metabolic conditions by changing a user’s diet and exercise.

Space Perspective raises $7M for its plan to ferry tourists to the edge of space — Spaceship Neptune is designed to carry up to eight passengers on a six-hour journey that will include two hours spent at the upper edge of Earth’s atmosphere.

Advice and analysis from Extra Crunch

From surviving to thriving as a hardware startup — Six strategies from Minut CEO Nils Mattisson.

A roundup of recent unicorn news — So much for a December news slowdown.

(Extra Crunch is our membership program, which aims to democratize information about startups. You can sign up here.)

Everything else

The U.K. approves the BioNTech/Pfizer COVID-19 vaccine for emergency use — The U.K. is the first country to approve the vaccine for widespread use.

Discovery will launch its own streaming service on January 4 — Discovery is the latest media company to launch a standalone streaming service, and the latest to adopt the simple naming strategy of just adding a plus sign.

Gift Guide: The best books for 2020 recommended by VCs and TechCrunch writers (Part 1) — Includes lots of good books for tech and business readers, plus my recommendation for the non-new, non-tech, yet extremely good novel “Jonathan Strange & Mr. Norrell.”

The Daily Crunch is TechCrunch’s roundup of our biggest and most important stories. If you’d like to get this delivered to your inbox every day at around 3pm Pacific, you can subscribe here.

from Apple – TechCrunch https://ift.tt/2JIZPJO

Looking Glass’s next product is a holographic digital photo frame

Looking Glass’s technology is extremely cool, but has, thus far, been prohibitively expensive, ranging from $600 for its 8.9-inch product to $6,000 for the 15-inch model — and an undisclosed sum for its 32-inch 8K display. Portrait might not be the most technically ambitious or versatile product the Brooklyn-based startup has produced, but it’s easily the most accessible.

At $349, it’s still not cheap (though it’s $199 if you get in early via Kickstarter), but it has to be one of the most affordable holographic displays on the market. Along with the pricing, the company is taking a more populist approach to functionality, as well, offering up what is essentially a very advanced digital photo frame. The system offers up to 100 different perspectives on a 3D image, which, unlike products like Sony’s new 3D display, can be viewed by multiple people at once.

Image Credits: Looking Glass

The system can also be run without being tethered to a computer. Its standalone mode relies on a built-in computer to deliver a 60 FPS holographic image. Those photos, meanwhile, can be captured with an iPhone and edited into a 3D image using the included HoloPlay Studio software. Holographic videos can also be captured with Azure Kinect and Intel RealSense cameras.

Looking Glass is targeting enterprises and eventually gamers with 8K holographic display

“It’s the first step toward holographic video calls,” the company notes, perhaps tipping its hand a bit about future plans.

“Ever since I was a little kid, I dreamed of the moment that I’d be able to have a holographic display of my own,” CEO Shawn Frayne says in the release. “I imagined what it would be like to send someone a holographic birthday message, or to say hello as a hologram to my great-great-great granddaughter. Looking Glass Portrait, the culmination of six years of work by our Brooklyn and Hong Kong based team, makes those dreams real for more people than ever before.”

The Portrait goes up on Kickstarter today, and will be available to early backers for $199.

Looking Glass starts shipping its 8K holographic display

from iPhone – TechCrunch https://ift.tt/3mxwfps