A Birder in the Hand: Mobile Phone App Developed by Caltech Researchers Can Recognize Birds From Photos

The Merlin Bird Photo ID mobile app has been launched and, thanks to machine-learning technology, can identify hundreds of North American species it “sees” in photos.

The app was developed by Caltech and Cornell Tech computer-vision researchers in partnership with the Cornell Lab of Ornithology and bird enthusiasts. Once it is downloaded on a mobile device, Merlin Bird Photo ID can go anywhere bird watchers go—even places without cell service or Wi-Fi.

“When you open the Merlin Bird Photo ID app, you’re asked if you want to take a picture with your smartphone or pull in an image from your digital camera,” says Merlin project leader Jessie Barry at the Cornell Lab. “You zoom in on the bird, confirm the date and location, and Merlin will show you the top choices for a match from among the 650 North American species it knows.”

Caltech and Cornell Tech computer scientists trained Merlin to recognize birds by showing it nearly 1 million photos that were collected and annotated by birders and volunteers mobilized by the Cornell Lab. Annotations include the species of the bird as well as key points identifying physical elements such as wings, beaks, and claws. When a user of the app presents it with a bird to identify, Merlin selects species that closely match those characteristics. Like any good birder, the system considers species that would be found at that specific time of year and in that location using information from eBird, an online bird resource that collects and records an average of 7 million bird observation records each month from around the world.

“In building Merlin Bird Photo ID we were especially concerned with the quality and the organization of the data,” says Serge Belongie (BS ’95), professor of computer science at Cornell Tech. Together with Pietro Perona of Caltech, he is the cofounder of Visipedia, the Google-funded umbrella project that is using advances in machine learning and computer vision to help classify objects in photographs. “Ultimately we want to create an open platform that any community can use to make a visual classification tool for butterflies, frogs, plants, or whatever they need,” Belongie says.

“This app is the culmination of seven years of our students’ hard work and is propelled by the tremendous progress that computer-vision and machine-learning scientists are making around the world,” says Perona, the Allen E. Puckett Professor of Electrical Engineering in Caltech’s Division of Engineering and Applied Science. “A machine that recognizes objects in images, like humans do, was a distant dream when I was a graduate student and now it’s finally happening.”

How good is Merlin Bird Photo ID? “Accuracy is around 90 percent if the user’s photo is of good quality. Submit a fuzzy image or one in which the bird is small or partially covered by leaves and the odds of getting an accurate match go down,” says Steve Branson, senior postdoctoral scholar in electrical engineering at Caltech.

Despite the high-tech advances, humans are still an important part of the process. “We need eBird data from bird watchers along with experts who can label the photos used to train Merlin,” says Caltech graduate student Grant Van Horn. “You need teachers to teach the machine what it needs to do. Our system combines the expertise of hundreds of birders and ornithologists.” Van Horn and Branson are both part of the Visipedia team, and developed the algorithms that allow Merlin to learn to recognize the birds.

There is more to come. Just around the corner is a Merlin Bird Photo ID release in Spanish for birds in Mexico. Down the road, the Merlin team will produce versions for South America, Europe, Asia, Africa, and Australia—eventually expanding to cover the globe.

“The wonderful thing about this project is the collaboration with the Visipedia team,” says Cornell’s Barry. “We have a product that really works because it’s supported by fantastic research and is great for the birding community because it’s built for birders by birders.”

The updated Merlin Bird Photo ID may be downloaded free for iOS or Android systems from the Apple iTunes and Google Play app stores. It is paired with the original Merlin app, released in 2014, which uses a few short questions to help users narrow down the identity of a new bird.

Merlin Bird Photo ID is powered by Visipedia with support from Google, the Jacobs Technion-Cornell Institute, and the National Science Foundation.