Less Toy, More Workhorse: Drones Get Functional

This story was originally published by Knowable Magazine.

You hear a mechanical buzz. You look up, and there it is, hovering in the sky—four whirring rotors. Over your shoulder, you catch sight of someone tracking the flight and manipulating controls.

Drones, little flying vehicles with varying states of autonomy, have arrived. Some survey crops from above. Some film dazzling videos. Some just annoy the neighbors. In Virginia, drones have delivered library books. And NASA launched the Mars rover, Perseverance, on July 30, 2020, which features a helicopter named Ingenuity that will have to be somewhat autonomous because of the communication delay between Earth and Mars.

Drones have come far in recent years, says roboticist Sarah Tang, who studied drones and unmanned aerial vehicles (UAVs)—a more comprehensive term for autonomous flyers—for her Ph.D. work at the University of Pennsylvania. But though the machinery and software have both seen strides, work remains before drones will be as useful as many hope, as Tang and her former doctoral adviser Vijay Kumar describe in the Annual Review of Control, Robotics and Autonomous Systems.

Knowable Magazine asked Tang, now a software engineer working on self-driving cars at Nuro in Mountain View, California, about the promise and the challenges. This conversation has been edited for length and clarity.

Knowable Magazine: What got you interested in studying robotics and drones?

Sarah Tang: At Princeton, in my junior year, I did an independent project on coordinating underwater robots to synchronize their movements. I got really interested in teams of robots working together and how they can collaborate, communicate, and coordinate.

I began working with drones at UPenn. They have this great capability for very agile maneuvers. And that was a really interesting problem from a planning and control standpoint: On the one hand, you have this vehicle that’s very hard to control and stabilize, and on the other, you have to plan very quickly to move past obstacles on the fly—you have to get it to do all these acrobatic things. I thought those two things together were really cool.

KM: What’s the difference between a drone and an unmanned aerial vehicle?

ST: Drone refers to the platform itself, a flying thing. UAV refers to any flying platform that is not piloted by a person—there’s software controlling it.

In this video from Alphabet Inc.’s Wing project, a librarian working for the Montgomery County Public School in Virginia discusses using drones to deliver books to students. Along with library deliveries, the drones also deliver coffee, FedEx packages, prescriptions, and more in Montgomery County. Wing is one of the first high-profile deployments of drone delivery. According to the company’s website, Virginia is the only place in the United States where it is operating. Wing also has operations in Canberra, Australia; and Helsinki, Finland. Companies such as Amazon, Postmates, and 7-Eleven also have dabbled in drone delivery, with bigger plans for the future. Credit: WING

KM: What are some of the most common misconceptions about these vehicles?

ST: Mostly they’re about what the drones are capable of doing on their own. Part of that comes from the research videos: People show robots doing these intricate, acrobatic things. But what viewers might not realize is that the robots were dependent on external systems. They had markers on them. Think motion-capture technology, like what was done with the character Gollum in The Lord of the Rings: Actor Andy Serkis wore a suit with balls all over it—markers—to guide the animation. We put those same things on our robots.

Based on the markers, an external camera system detects, with centimeter accuracy, where each robot is spatially, how fast it is moving, and its orientation. That is very, very far from a fully autonomous system that is using only its own sensors to map and understand the world and where it is.

A lot of the current work is thinking about fully autonomous vehicles that can do things like food deliveries or surveillance. There’s still a lot of work to be done there, in terms of getting the autonomy onboard so the vehicles can navigate without external help.

KM: What are the some of the sensors that go into a more successful unmanned aerial vehicle?

ST: There are two big fields. If you have a robot that’s designed for something like mine inspection, the robot doesn’t have to be quite as acrobatic, so it can carry more sensors. It will usually have a LIDAR system [a laser scanning system that helps build 3D maps of the world] and cameras. Something like a video drone, which requires less precision in its flight path, will typically have only a camera, which will keep it light and agile.

KM: How are people are using drones and/or UAVs?

ST: Right now, most of it is just in the hobbyist space. Some companies, like DJI and Skydio are huge for people who are into videography. This could extend to filmmaking and other video projects.

On the commercial side, there are quite a few applications that are being explored. Insurance companies use them for roof inspection, bridge inspection, or cell-tower inspections. Things that would be very expensive if you needed to get a human to climb into those spaces—places like mines and tunnels.

Another application is reconstruction of buildings: Real-estate companies use drones to fly around and make 3D models of houses and buildings. In agriculture, there’s this idea of using drones to fly over and inspect crops. Entertainment is another big one, with light shows like the 2018 Winter Olympics opening ceremonies and the Lady Gaga Super Bowl halftime show using drones the year before. And of course, there’s drone delivery.

There are different designs depending on the use—the helicopter-like rotor ones and fixed-wing robots. In the research space, there are also bio-inspired robots with flexible wings that fly like birds or look like dragonflies.

KM: What drives the bio-inspired research? Is it about trying to figure out things like how insects fly, or is it oriented toward finding new ways to fly?

ST: I think one of the big motivations is trying to figure out how to make these robots smaller. Researchers look at insects and the way that they fly to try to figure out what form their drones can take so that they can have all the sensors and motors they need but be as small as possible.

The RoboBee from Harvard’s Wyss Institute is one of a handful of animal-inspired drones. Potential uses include surveillance, search and rescue missions, monitoring the weather and other environmental conditions, and to help pollinate crops.

KM: What is the advantage of being small?

ST: One idea is that you could use a “swarm” of drones to do a task, like a coordinated light show or a search-and-rescue operation. And when you have more drones, obviously you need to scale down the size of them. On the research side, it is mostly driven by the possibility of creating cooperative teams. Companies like Intel are working on swarms of larger drones for entertainment.

KM: What has driven the growth of aerial vehicles?

ST: There was a huge push in the research community to use them as a platform for robotics applications, because they were being mass-produced for recreational uses, and became cheap. You can buy them off the shelf, set up the hardware and run experiments with them.

Later, the research field started to see the autonomous capabilities they could have. This led to startup companies that proposed offering services, with funding from venture capitalists and other sources.

Another big push came from DARPA [Defense Advanced Research Projects Agency, the U.S. Department of Defense’s technology research agency]. Historically, DARPA’s been interested in robotics and autonomy, and its Fast Lightweight Autonomy program promotes development of autonomous vehicles using limited sensors and batteries. They had to be built with commercial parts and there was a weight limit as well. They had to be able to operate autonomously with no GPS, no pre-built maps, and no pilot intervention.

For the task, it would be given the position of a red barrel. The drone would know what the barrel looked like beforehand. It would have to fly to that position, identify the red barrel, and then fly back. But along the way, it would have to do all sorts of things: It might start outdoors and have to fly through a window to get indoors, then go down the stairs of the building and come back out a door to fly back to its starting position.

The idea was that the robot would have no idea what could be between itself and this red barrel’s location, and would have to be able to sense and map and make decisions in real time to navigate these previously unknown environments. There was also an emphasis placed on speed: They really wanted to push the speed at which these robots would be able to fly.

The Department of Defense is interested in surveillance and mapping applications, and previously had a humanoid disaster-response challenge and an urban driverless vehicle challenge that both involved autonomous systems. The funding that it provided really pushed the research forward.

A drone participates in DARPA’s Fast Lightweight Autonomy program, which is intended to speed up drone development. DARPA (the Defense Advanced Research Projects Agency) is interested in military, surveillance, and search-and-rescue applications. Credit: DARPA TV

KM: What kind of dead ends have there been?

ST: There was a period of disillusionment after all the startup funding. There were some pretty high-profile duds. For example, a company called Torquing Group got $3 million from a Kickstarter campaign for a drone called Zano and it never delivered a product. A company called Lily Robotics also folded before delivering a product; it did a crowdfunding campaign for a drone that would follow you but had to close before it was able to ship the product. GoPro made a drone called Karma. Users reported that it had an issue where it would just randomly fall out of the sky. GoPro ended up recalling the product and stopping production altogether.

People began to realize that this problem is harder than we initially thought. It’s not something that a couple million dollars and implementing ideas from research papers can solve. Being able to solve the manufacturing problem, solve the robotics and autonomy problem, and then put it all together into a product that you deliver to a customer is really challenging and difficult. Funders are a bit more selective now.

KM: What does it take to keep a UAV stabilized in flight?

ST: If you think about a drone with four rotors—a quadrotor—this is not something that can stabilize itself. If you buy one off the shelf, there’s software running on board to stabilize it for you, and then you control where it goes. The ability to hover in place is complex. The robot needs to know its orientation and its speed. It also needs to control the rate at which each of its motors spin to stay in balance. That all requires software.

A drone must make decisions in order to safely navigate its environment. On the left is an example of an outdoor environment a drone might need to fly through; on the right is an indoor environment. Blue lines represent suitable flight paths from the drone’s starting position to the intended destinations (visible as red points on the right). Sensors, including cameras, on the drone provide input that helps the software autonomously navigate the environment while staying stable in flight. Early navigation successes have helped with the development of ways to further interact with an environment, such as grasping objects.

A drone must make decisions in order to safely navigate its environment. On the left is an example of an outdoor environment a drone might need to fly through; on the right is an indoor environment. Blue lines represent suitable flight paths from the drone's starting position to the intended destinations (visible as red points on the right). Sensors, including cameras, on the drone provide input that helps the software autonomously navigate the environment while staying stable in flight. Early navigation successes have helped with the development of ways to further interact with an environment, such as grasping objects.

KM: Your review paper covers a lot of the software needs of drone systems. One of the topics covered is artificial intelligence systems, including Kalman filtering. Can you explain what this is and how it connects to UAV systems?

ST: A Kalman filter is a mathematical model that provides a way for a robot to fuse information that comes from its sensors with its internal model of how it expects to be moving. One often-used example is to picture a robot that is moving down a hallway. It knows how many doors this hallway has, and it knows a door is every 2 meters.

If it measures velocity and time, it can estimate distance. That’s one mode of information. And the other mode is, “Do you see a door or not?” Say the robot thinks that it’s gone 2.5 meters, and then suddenly it sees a door. It expected the first door to be at 2 meters, so it has to take these two pieces of information and resolve them to come up with a belief of where it is in the world.

The Kalman filter allows robots to combine information over time from multiple sensors to understand where they are in the world. Most important, it allows us to model noise in the robot’s measurements and understand how certain or uncertain the robot should be.

KM: You also talk about iterative learning control in your article. What is that?

ST: One of the coolest things that’s happening is the possibility of using machine learning and deep learning to better control robots. For example, you don’t have an exact model of how your motors will respond to a command or disturbances from the wind or other things around you. In iterative learning, as you keep performing the task, you use that information to improve your model of the world.

Let’s say I want my drone to fly in a certain shape, like a circle. If I feed the robot a sequence of input, like rotor commands, then it’s going to follow the circle. I observe what the robot actually does and what I wanted it to do. Based on its errors, I calculate the corrections to apply to the input to improve performance the next time.

KM: How have advancements in machine learning affected UAVs?

ST: It’s helped with understanding the sensor data that the UAV is getting. The robot might have a suite of six or eight cameras, and it’s getting all this imagery data. Machine learning can help us process that into information about the world that the robot can use, such as how fast it’s moving. That’s one area.

The other is controls. You can use machine learning to improve how well your robot can fly while compensating for things like wind that are hard to model mathematically.

KM: What do you see happening for UAVs in the next few years?

ST: On the research side, there’s more to be done in terms of fully autonomous operation and demonstrating that that can be done robustly. For example, Skydio has this drone that can track and follow you while you’re driving or mountain biking or riding a horse. That’s been a huge advancement.

Another is working with regulators to prove that these things are safe to fly autonomously outside of line-of-sight and over cities and populations. We also need to be able to make that jump from testing within line-of-sight to having some sort of pilot program where a company says, “OK, we’re going to start doing deliveries with these,” and then launch a service that makes money. I think that’s really exciting for the future.

KM: What have the more successful drone researchers or drone startups done to excel in this space?

ST: This is a personal opinion, but I think just having some experience in robotics and having the perspective that there are challenges, that things take a while. That’s the differentiating factor, especially for companies. Don’t overpromise. Don’t say, “Give me $100 and I’ll deliver a drone to you in four months that I haven’t even started developing yet.” Take things one step at a time and do this systematically.

KM: For drones, what holds the most promise right now?

ST: I think inspection is what resonates the most: using robots to inspect under bridges or on top of roofs or in other hazardous places.

I’m a little bit wary about personal robotics. If you take a robot and you put it in consumers’ hands and say, “You can interact with this robot however you want,” the needs for how well this robot has to function and the situations that this robot has to understand and handle are just endless.

But if you have a defined operational area, you can say, “This is my task and I need to be very robust and very reliable within this task.” This narrows down your problem to something manageable. You can focus on the autonomy features that you need to get that working very, very well.

This article originally appeared in Knowable Magazine on July 23, 2020. Knowable Magazine is an independent journalistic endeavor from Annual Reviews, a nonprofit publisher dedicated to synthesizing and integrating knowledge for the progress of science and the benefit of society. Sign up for Knowable Magazine’s newsletter.

About The Authors