Tag: Seeing AI

Seeing AI Advances
Seeing AI Advances & Updates

Since it’s initial launch in 2017, the Seeing AI app from Microsoft is the gift that keeps giving to visually impaired people. We embraced the intelligent camera for it’s ability to translate text, images, objects and environments into spoken words. Even as it inspired many competitors, Seeing AI has maintained it’s prominent position. As the software continued to improve, the app’s interface did not change much…until now.

The Seeing AI interface has been consolidated into 3 main tabs labeled: Read, Describe and More. The idea was to streamline what was previously a dedicated “channel” for each feature. The result does not seem simplified, at first, and it takes some getting used to. Here’s how the reshuffled tabs lay out.

Seeing AI graphic image with app icon overlay. Image credit: Microsoft

READ Tab

The first tab combines three channels, from previous versions, for reading Short Text, Documents and Handwriting. Point the camera at any text, or handwritten words, and it will read instantly. Taking a picture of the text may improve accuracy and will allow you to ask for more information.

DESCRIBE Tab

In the second tab, take a photo of an environment and it will be described to you in detail, including recognition of objects and people. This feature offers option to have photos in your library described, and to ask for more information or explore image by touch.

MORE Tab

The third tab is home to all the remaining channels, seven in all, each standing on it’s own.
- PRODUCT – Reads barcodes and accessible QR codes. Audio guidance helps you find the code, then scans it. In the grocery store, for example, the code might identify the brand and flavor on the ice cream container, or the fat content of the milk, or if the bread is wheat or white.
- PERSON – Use the camera to scan and tell you about the people nearby. Facial descriptions include an estimate of age and expression. Set face recognition to announce the name of someone you know when they come into view.
- CURRENCY – Select the paper currency you wish to identify (choose from 17 currency options) and hold the camera over a single note to hear the value.
- WORLD — An experimental combination of artificial intelligence and augmented reality helps you navigate unfamiliar indoor spaces in 3D guided by spatial audio using headphones. Caution is advised and feedback is appreciated.
- FIND MY THINGS — Teach Seeing AI to recognize the items you often find yourself searching for, like keys, wallet, AirPods, sunglasses, etc. Follow the prompts to take videos of the item and save it to your list. Once set up, the camera can scan the environment to help you locate the selected item through audible beeps that quicken as you get closer to it.
- COLOR – Point the camera at an item and it will identify the color.
- LIGHT – For users with no light perception, this channel indicates intensity and direction of light through audible tone and pitch changes.
The Seeing AI app is available for iOS on the App Store, and for Android on Google Play.

Get quick help at the upper right corner on every channel.

Feedback helps improve the app, so do share your thoughts through the app menu or by email to [email protected].

Post originally published on Jan 19, 2018 and most recently updated Apr 16, 2025
April 16, 2025
Try a New Approach to Reading Restaurant Menus
Man reading restaurant menu options to his
dinner companion

One of the most frequently asked questions from people living with low vision is: How do you read the restaurant menu? There are many good options for accomplishing this task depending on the degree of visual impairment.

This is all about making adjustments, and accepting that picking up a menu and reading it the way you once did, may not be something you can reenact. Keep in mind, the goal is to select food you will enjoy, not to read every dish on the menu.

Here are some of the ways we can successfully overcome the challenges of menu reading and return to savoring the culinary and social experience.
- Flashlights & Magnifiers – It is not at all unusual to see people in dimly lit restaurants pull out their mobile phone flashlight and point it at the menu. The same goes for magnifiers, sometimes on the phone, sometimes on their own with the light built in. So no need to feel weird — everybody’s doing it.
- Online Menu – Making a menu selection in advance will allow you to relax. Google search “menu” + the restaurant name to review choices. (With iPhone use Speak Screen or VoiceOver to read the menu aloud, on larger screens increase the text size.)
- Ask – It is perfectly fine to ask a companion to read some sections from the menu to you, but try to ask for specific categories like, “What are the salads or soups?” Also, ask if there is a large print menu.
- Specials – Asking the wait staff to read you the menu is not realistic, but do ask for the specials of the day or for their recommendations in specific categories.
- Seeing AI – The popular iPhone app from Microsoft has Short Text and Document channels that can be useful for menu reading. A recent update to the Document channel (version 5.2) takes a picture of the menu and reads it back or allows you to ask questions like, “What are the seafood options?” Seeing AI Gets Upgrade
- The QR Code Menu — When it’s available, use your mobile phone to scan the code and the menu is instantly available on your screen. Make it a talking menu using accessibility speech features like VoiceOver and Speak Screen in iOS, or TalkBack and Select to Speak in Android. Listen to the menu options privately with your ear buds.
In essence, finding a new approach to reading menus will get you back to enjoying the food, the company, the conversation, and the atmosphere. It’s even okay to just say, “I’ll have what she’s having.”

This article was originally published January 24, 2018 and last updated Oct 20, 2023.
October 20, 2023
Google’s Lookout 2.0
Not very long ago, news about developing technologies for people visually impaired or blind was not widely reported, if reported at all. It was also unusual for this kind of news to be attached to the biggest technology companies on the planet. All that has definitely changed! In 2018, Google announced an Android app in development called ‘Lookout,’ its purpose to help people with vision loss, identify objects around them and read text. The news was quickly shared by many news outlets, noteworthy because while Google has been involved in the creation of accessible Android applications for quite some time, they had not previously made big pronouncements about it.

The accessibility app was first launched in 2019 and has recently been updated. Lookout is designed to deliver spoken notifications with minimal interaction. So it does not create a juggling act for the user, it is recommended that the smartphone be carried in a shirt pocket or on a lanyard, camera facing out.

Powered by artificial intelligence (AI), it utilizes camera vision to help people, who are visually impaired, get the information they need without having to ask others for help. The enhanced access is delivered now in these 5 modes:
- Explore – identifies objects in your environment
- Shopping – reads barcodes and currency
- Quick Read – for signs, labels and mail
The newest features in 2.0 are:
- Food labels – identification of packaged foods
- Document scan – captures and reads full detail
Lookout 2.0 is now available on Google Play for devices running Android 6.0 Marshmallow or later with 2GB of RAM available. This is a free app, it does not require an internet connection and uses machine learning to identify items of importance and keep delivering the information people are interested in. In addition to English, the app is now localized for French, German and Italian.

Get more information about using Lookout and share your feedback by contacting the Google Disability Support Team online or on the BeMYEyes app.

Lookout is not currently available for Apple iOS devices, but a similar app called Seeing AI is.

This article was previously published Jul 26, 2019 and updated Mar 26, 2021.
March 26, 2021
Letter from the Editor

Dorrie Rush

Dorrie Rush, Chief Content Editor of OEPatients.org

2019 Is A Year of Possibilities for People with Vision Loss

It is true, I do love a fresh new year brimming with promise. My optimistic outlook continues to center on the factor most impacting life with vision loss today, and that is accessible and inclusive technology. In terms of progress, last year was no slouch either, and it set the pace for more good things to come!

The momentum in everyday accessible technology has been building for more than a decade now. It started with Apple’s delivery of the iPhone in 2009, accessible out-of-the-box for people with low or no vision. All products that followed from Apple included accessibility by design. The paradigm was shifting. These developments significantly changed the direction of my life, I no longer felt technology was leaving me behind.

Last year we were the beneficiaries of a few big surprises coming from somewhat unexpected places.

Microsoft created two new important accessibility apps. Seeing AI is a smart camera with 9 apps in 1, and Soundscape is a “3D map with sound.” This delightful surprise extends even further, both apps are free and they were built exclusively for Apple’s iPhone. A magnanimous gesture by Microsoft, perhaps more will be revealed about this in the coming year.

I enjoyed every one of our accessibility events last year and delighted in the lively, sophisticated and always curious audiences. I’d have to say the most fun was our session on smart speakers. They are entertaining and incredibly useful in daily life. Smart speakers are being adopted at a faster rate than smartphones were, and they are especially popular with people visually impaired or blind. This is a category we will continue to pay close attention to.

At another event we applauded the proliferation of audio description, now available on demand for Broadway shows and at the movies. I love the voice that speaks quietly in my ear and fills in the details, I might have otherwise missed.

So, it’s pretty obvious why we should be enthusiastic about the coming year for inclusivity. Apple doesn’t like to say much about their developing technology, but there are reports and rumors that lead us to believe we may be seeing some smart glasses and even clothing designed for accessibility. Actually, I would not be surprised to find many variations of smart glasses showing up, and would expect they’ll begin shrinking to a more normal, wearable size, from the rather heavy visor like devices we saw last year.

We can look forward to new developments in navigating airports and shopping centers.

There is Aira to watch, a service that teams technology with the eyes of a live agent for reading, identification, travel, etc. IBM says it is close to making the NavCog, indoor voice activated navigation app, available to the public. Toyota has been working on Project BLAID, an indoor navigation device, for some time, maybe this is the year we’ll try it. Google’s app “Lookout” for object recognition, is anticipated any day now.

AI (Artificial Intelligence) is powering much of the progress. Our digital assistants and smart speakers will mature and become more intuitive and dependable.

As I write this letter, Google, Alexa, and Siri are all sitting on my desk, ready to assist whenever I ask. They’re excellent researchers, spellers, fact checkers, appointment schedulers, alarm setters, time tellers, and they have instant access to massive amounts of music. Like all assistants, these 3 were not created equal, they all possess their own individual strengths and weaknesses. We’ll definitely talk more about that as the year moves forward.

One thing is for sure, we have only just begun to tap the potential of what technology can do to empower people living and working with vision loss. Stay with us, as we report on an exciting year, packed with possibility!

January 7, 2019
OE Patients: Best Of 2018 Articles

This year, we created a roundup of our most popular articles on OE Patients. Our best-of list for 2018 covers technology, health, and tips for making adjustments — from the secret to mastering iPhone’s VoiceOver to accessibility at arts museums.

We hope you’ll be inspired to develop a new skill or create a good habit for the fresh new year!

Apple’s iPhone

10 Things To Know About iPhone Accessibility For Vision

The iPhone is quite possibly the most adopted device, ever, for people with vision loss. This is because it is accessible out-of-the-box, even if you can’t see the screen at all.

Discover the top reasons for using the iPhone.

Let iPhone’s VoiceOver Do The Reading

Apple’s built-in screen reader was a game-changer for those living with vision loss. Our accessibility expert, Dorrie Rush, writes about the secret to mastering iPhone’s VoiceOver gestures — practice, practice, practice.

Learn how to master VoiceOver.

Microsoft

“Seeing AI” Is A Gift from Microsoft

We love Microsoft’s Seeing AI, a free, intelligent camera app that translates images into words for people with visual impairments and blindness. Here are four features we especially like.

Learn more about Seeing AI.

Microsoft Soundscape 2018, A Review of What It Can Do

Our accessibility expert Dorrie Rush appreciates the enhanced narration and awareness provided by Microsoft Soundscape, a navigation tool for people with vision loss.

Read the full review of Soundscape.

Apps and Technology

App Happy

Discover a collection of wonderful apps that empower people with visual impairments in ways we could only once dream.

Click for our recommended apps.

Electronic Glasses Event Recap

In October, OE Patients held an accessibility event on wearable devices for audiences with vision loss. We took a closer look into head mounted products including: OrCam, IrisVision, Jordy and more.

Explore featured electronic glasses.

GoGo Grandparent Gets Uber Without An App

For older adults who need to get an Uber or Lyft without an app or smartphone, GoGoGrandparent is perfect.

Find out more about GoGo Grandparent.

Making Adjustments and Health

Doctor, Am I Going Blind?

Every eye condition is different. Read our article by ophthalmologist Dr. Yale Fisher, to understand how vision loss caused by macular degeneration, diabetic retinopathy and glaucoma rarely results in blindness.

Learn more about vision across eye conditions.

What Is A Cataract?

Cataract is the leading cause of reversible blindness and vision loss in the world. Read more by Dr. Amilia Schrier, Director of Education at Manhattan Eye, Ear and Throat Hospital.

Learn more about cataracts.

Museums Making Art More Accessible

From Florence to New York City, museums everywhere are making it easier for visitors who are visually impaired or blind to experience art.

Find out how museums are opening up art.

December 7, 2018
Event Recap: App Happy
Thanks to all who attended our accessibility session on Apps, coming out despite the Arctic blast that hit with a relentless mix of icy rain and snow. The discussion was lively and always interesting and informative, exactly as we hoped it would be. People sharing their experience and learning from each other.

We acknowledged, upfront, that the topic of apps is massive and could be discussed for days. Apps that help us compensate for vision loss are available in many categories. We have lots of choices, and selecting the apps we integrate into daily life is a very individual process.

It was also delightful to note, that accessibility for people with vision loss is a priority, emphasized by tech leaders like Apple, Microsoft, Google, and Amazon. Inclusive technology is shifting our expectations and bridging the gaps. Although it’s not perfect…it’s really good.

Finding the apps that work for you will make life a little easier.

The recap that follows is a look at some specific apps and new developments we talked about at our “App Happy” session. It’s not only the apps with the bells and whistles that can change your life. Access to the basic built-in iPhone apps like Contacts, Camera, Messages, News, Clock, and Maps, can make a huge difference in your everyday.

Here are the apps that kicked off our “App Happy” discussion:

Seeing AI App (iOS Only)

This app from Microsoft has been around for just over a year and has already expanded its channels and improved its performance noticeably. Built for iOS exclusively, but rumored to be adding Android soon. Read more about Seeing AI in our article here.

9 Channels:
- Short Text – Reads instantly
- Document – Aligns and reads full page
- Product – Barcode reader
- Person – Facial recognition; fun but not sure how functional
- Currency Reader – US and Canadian
- Scene Reader – take a photo and it describes the scene and reads text
- Color Reader – identifies color
- Handwriting Reader – much improved for handwritten print or cursive
- Light Indicator – changes tone to the direction of light
Soundscape App (iOS Only)

A “map with 3D sound” provides information about your environment. Also from Microsoft, exclusively for iOS (so far). Read more about Microsoft Soundscape in our article here.

4 Channels:
- My Location – figure out where you are, direction facing, points of interest
- Nearby Marker – locations you have marked
- Around Me – what’s in the 4 surrounding quadrants
- Ahead of Me – street names, buildings, businesses coming up
Aira App

A new assisted navigation service provided by trained agents viewing your location by live video.
- Log in as a guest to try in fee-free locations like Walgreens, AT&T, Wegmans and a growing list of airports and businesses
- Also available for a monthly subscription starting at $89. for 100 minutes
Be My Eyes App

This app provides live video access to a network of 1 million volunteers helping 80 thousand people with vision loss in 180 countries.
- Average wait for a volunteer: 30 seconds
- Get help with appliances, electronics, instructions, identification, etc.
- Not recommended to share private information
- Newly partnered with Microsoft Disability Desk
GalaPro App

The app making New York’s Broadway shows accessible to audience members with vision and hearing loss, from the privacy of their own mobile phone. Read more about GalaPro in our article here.
- Audio descriptions syncs with live performance
- An accessibility kiosk in every theater helps get you started
- Also, offers closed captioning and language translation
More App Happy audience favorites:

Amazon App
- Now with access to live Disability Hotline 888-283-1678 for shopping assistance.
Alexa App
- For Book Reading, Shopping, Calendar, Music, Recipes, Games, Translation and more.
Amazon Prime Video App
- Watch Prime Video on numerous devices with audio description.
Apple News App
- The standard iOS app that provides access to a large selection of curated news from major outlets including New York Times, Wall Street Journal, Washington Post, Vanity Fair, Bloomberg, CNN, Food Network and more.
NFB Newsline App (iOS Only)
- The National Federation of the Blind now offers an app for iOS that extends their newspaper reading service to over 300 publications, for eligible users.
Shazam App
- An entertaining app that identifies music, movies, TV, and ads, after hearing just a short snippet. Apple recently bought the technology and now offers it on iOS, Mac OS, TV OS, and Watch OS.
Drop us a note by email, or post a comment on our Facebook.
November 16, 2018
AAO Video: Technologies for Patients with Low Vision

In a recent video from the American Academy of Ophthalmology, Drs. Mary Lou Jackson and Mark Bona take a brief look at an interesting selection of low vision technology. Here is the list of products they mention with our summary and comment. The video is linked at the end of this post.

Handheld Victor Reader Stream

Victor Reader Stream: $369

This was the go-to digital reading device for people with low vision or blindness in the pre-iPhone era. Still recommended by low vision specialists for it’s simple and tactile interface.

Seeing AI app reading an address on an envelope.

Seeing AI App: FREE

This very new and very popular iPhone app from Microsoft is available on the App Store. It reads text instantly and on the go, identifies products, colors, currency and more. We talked about the first release and the upgrade on OE Patients posts titled:

Seeing AI is a Gift from Microsoft

Seeing AI Gets an Upgrade

Portable video magnifier used on newspaper text.

Portable Video Magnifier: $200 – $1,000

We agree video magnifiers offer a powerful boost to a wide range of low vision patients. Read what we say on OE Patients in the post titled: Magnifiers Are A Must.

A man using eSight glasses to read a document.

eSight: $5,950

A head-mounted video magnification device. Here’s what we said about it in OE Patients post titled: eSight Up Close

Jordy, optical viewing device.

Jordy: $3,620

A head-mounted video magnifier recently reintroduced as a lower priced competitor to eSight. Jordy also converts to a desktop magnifier with the purchase of an optional dock.

Aira: $89 – $329 per month

Aira (pronounced I-ra) is a new service, for a monthly fee offers access to a sighted agent, summoned by a tap on the Aira smart glasses and connected by a live video feed. The agent will provide on the spot detailed guidance when needed. This is an innovative concept and we are very interested to see how people with low vision take to it.

Click here for the AAO video.

March 23, 2018
Innovation Drives Inclusion

It’s always good to step outside ourselves and understand what others are experiencing. While we focus on overcoming the adversity of vision loss, our colleagues, friends, and neighbors are busy meeting their own challenges.

Recently, Microsoft CEO, Satay Nadella, shared the very personal experience that inspires his commitment to making accessibility and inclusive design a priority. He notes that this year’s National Disability Employment Awareness Month’s theme was dubbed, “Inclusion Drives Innovation,” a concept Microsoft totally supports.

The flip side is equally true…innovation drives inclusion. Either way, we’ll take more of it, please! The scope of work and it’s sense of inclusion is conveyed in the Microsoft Accessibility Blog’s subtext, “Making technology easier to see, hear, and use!”

Here is an excerpt from Satay’s blog post:

“Becoming a father of a son with special needs was the turning point in my life that has shaped who I am today. It has helped me better understand the journey of people with disabilities. It has shaped my personal passion for and philosophy of connecting new ideas to empathy for others. And it is why I am deeply committed to pushing the bounds on what love and compassion combined with human ingenuity and passion to have impact can accomplish with my colleagues at Microsoft.

“It is fitting that the theme of this year’s U.S. National Disability Employment Awareness Month — recognized every October — is “Inclusion Drives Innovation.” We could not agree more. At Microsoft, we are making accessibility a top priority in our product development efforts, from core features in Windows 10 like Narrator to pushing the boundaries of what’s possible through innovations like Hearing AI created by Azure engineer Swetha Machanavajhala. In addition, we know that having an employee base that is representative of all backgrounds and abilities is one of our greatest strengths. This is why we continue to be passionate about hiring people with disabilities through initiatives like the Autism Hiring Program and Supported Employment Program, which help us advance our mission to empower every person on the planet — including more than 1 billion people who have disabilities. I am an optimist who firmly believes that at its most powerful, technology can contribute to incredible human capability and insight. I also believe it can help people find immense joy and a deeper connection to the world around them by enabling them to realize their professional and personal potentials. Some of my favorite examples of technology in action are where people have come together from across Microsoft, united by a shared passion to hack and build solutions that have the potential to benefit millions.

“Recently, one of my colleagues, Angela Mills, shared her personal story with me. Angela is visually impaired — having lost her central vision when she was a child, today she has only her peripheral vision. Over the years, she has used various technologies to assist her daily activities, but she calls the early release of the new Seeing AI app “life-changing.” Before, when she went into a cafeteria for lunch, she would have to find a cafeteria worker and ask for help in picking out a salad. Now, she can go to the salad area and use the app on her phone to read the labels of the salads before choosing one by herself. She can walk confidently into a conference room, knowing for sure she’s in the right place with the right people; and when she’s cooking at home, she doesn’t have to rely on her husband to read out the labels on the spice jars. It’s incredible to see how this idea that was sparked and developed at our annual hackathon by Saqib Shaikh, Anirudh Koul, and team, is now a free application that enables Angela and many others to feel more included and connected to the world around them, and we can’t wait to bring it to more folks around the world in weeks and months to come.” S

To read the full article on Microsoft’s inclusive design projects for people with Autism, ALS, Learning Issues and Visual Impairments, and to access audio described videos on these subjects…Click Here.

October 26, 2017