Robot Operating Systems

Presented by Sravan Balaji

Hello everyone, my name is Sravan Balaji. Today I'll be giving a presentation on robot operating systems which I hope will provide a light introduction to some fundamental concepts in autonomous robotics.

We'll start with a quick overview of what robotics is, some examples of autonomous robotic systems out in the real world, how robots work at a high level, and how Linux is used to develop and deploy these autonomous systems - mainly through ROS, the robot operating system.

I included this picture from CNN of a robot interacting with an Emperor penguin colony in Antarctica which I thought nicely tied together today's topics of Robotics and Linux.

Disclaimers

Any opinions expressed here are my own and do not necessarily represent the positions or policies of my current or former employers

Information presented here is for educational purposes and does not replace independent professional judgment

While I am not currently, nor have I ever been, a government employee, I feel it is prudent to provide this quick disclaimer. I'm giving this presentation in my personal capacity. Any opinions expressed are not necessarily representative of the positions or policies of any of my current or former employers.

Additionally, the information I'll be sharing is intended for educational purposes only.

Robotics

"Embodied Intelligence that interacts with the physical and human environment"

For those who are unfamiliar, let's briefly touch on what the field of robotics is through a couple descriptions I found online.

The first, from my Alma mater's website, describes robots as "embodied intelligence that interacts with the physical and human environment."

Robotics

"A machine capable of carrying out a complex series of actions automatically"

The second, from Wikipedia, describes robots as "a machine capable of carrying out a complex series of actions automatically."

Robotics

Hardware

+

Software

+

Intelligent Decision Making

I like to describe robots as hardware + software + some level of intelligent decision making.

This helps differentiate robots from other electro-mechanical systems like a toaster or an electric motor.

In my mind, it's really the autonomy software that makes robots special - the ability to interact with and react to changes in the environment.

Types of Robots

So, what are some real world examples of robots that people may have seen or heard of?

Drones

Drones, like the quadcopter shown here, have gained a lot of popularity in recent years among tech enthusiasts because they have dramatically come down in price, have decent battery life, are fun to fly, and allow you to take amazing aerial photography that just isn't otherwise possible.

It's a little more difficult to get into this hobby in Northern Virginia due to all the restricted airspace around the capital, but if you're willing to drive a bit farther out, you can have a fun time flying drones around.

The Washington Post

Autonomous Vehicles

Another example you might be familiar with is autonomous vehicles or self-driving cars.

They use an array of different sensors and computational algorithms to map out their environment, avoid obstacles, and transport passengers to their destination.

Waymo, a subsidiary of Alphabet, operates self-driving rideshares in Phoenix, San Francisco, and Los Angeles at the moment.

Industrial

Robots are also widely used in the manufacturing industry because of the repetitive nature of assembly line tasks. However, you can imaging extending the idea of manipulation arms like the ones shown here to mobile systems that could operate in a residential environment and perform tasks like cooking or cleaning autonomously.

Humanoid

Humanoid robots have also been showing up in the news doing amazing thinks like parkour or dance routines. One of the main selling points of these types of robots is that they can navigate and interact with environments designed for humans, unlike wheeled or aerial drones. This would avoid the need to redesign existing environments to work around the limitations of other robot designs.

Boston Dynamics

Underwater

This one was a little more obscure to me through most of my college and professional career, but is now the domain that I work in.

Unmanned Underwater Vehicles (UUVs) are yet another example of robots. These are widely used in industries like oil & gas because they allow you to more easily inspect pipelines and other infrastructure underwater. These are also pretty widely used in the defense industry, which is where I was first introduced to them.

You can imagine that operating a fully crewed ship to perform underwater inspection can be dangerous, time consuming, and expensive. So there is a lot of interest in introducing more autonomous, or at the very least remotely operated, underwater vehicles.

One of the interesting things I learned when I first started working in this domain is how much harder the problem is compared to other domains like ground or air. In those domains, you have near constant access to GPS and other infrastructure that can provide up-to-date and accurate location information. You can also get pretty decent communication with the vehicle at all times to understand what it is doing.

In the underwater domain, you basically just drop it in the water, wait a while, and hope it returns to the surface safely to see what happened. GPS signal can't penetrate through water and you have no way of communicating with the vehicle unless you attach a fiber optic cable to it which comes with its own disadvantages.

Metron & Cellula Robotics

And Many More!

Space
Consumer
Animatronics
Exoskeletons
Medical
etc.

These are just a few, but there are other domains and industry where robots are used like space, consumer (for example, the roomba vacuum), animatronics, exoskeletons, medical (like the surgery robots), and countless others.

Embodied Intelligence

Now, I would like to transition to talking a bit about how robots work by introducing a useful framework or mental model.

Sense-Plan-Act Paradigm

Sensing of the environment, external agents, and internal body information to determine state information
Reasoning with that information to make decisions for guidance, control, and localization
Acting upon the body and environment to produce motion or other outputs that enable the robot to locomote or interact with the environment

It's helpful to think about how robots work in the context of how humans perform various tasks.

We use our senses to gain information about our environment, other people, and ourselves.

Then we use that information to make decisions about how to move our body, figure out where we are, and figure out how to get to where we want to go.

Finally, we'll put our decision into motion by actually moving our body and making small adjustments as necessary to make progress towards our goal.

This process is continuously repeated as the environment and our position within it evolves.

Mars Rover

Let's use the sense-plan-act paradigm to think about how a Mars rover works.

The Mars rover is a remote-controlled vehicle designed to explore the surface of Mars. Due to the communication latency, newer iterations like Perseverance increasingly leverage autonomous path planning so operators have less to do and can allow the rover to safely move at higher speeds.

The rover starts by collecting data about the environment using stereo vision cameras and other sensors.

It can use the images captured from these cameras to build a map of the surface of mars to identify obstacles or rough terrain.

An operator can provide a general area to go investigate (shown as the blue circle in the video), then the rover will plan a path which avoids obstacles to get to that region (shown as the black and blue lines extending out from the rover).

After figuring out the path, it will need to control its motors so it can actually follow the path it planned out.

Sensing

Now let's explore each of these areas in a little more detail, starting with sensing.

Cameras

One of the most popular sensors for various applications are cameras. There are specialized cameras like the Intel RealSense here that can capture RGB images as well as depth, allowing autonomous systems to get a 3D view of the world.

The Intel RealSense D415 shown on the left uses stereo cameras, which can be processed into the point cloud shown on the right.

Intel RealSense

LiDAR

Another popular sensor, especially in autonomous vehicles, is LiDAR - Laser imaging, Detection, and Ranging.

This works by shooting a beam of light out and measuring how long it took for the light to reflect off the environment and bounce back to the sensor.

Using the speed of light and the travel time, you can measure the distance of obstacles in the environment and build up a map.

(Left) Wikipedia | (Right) Luminar

Inertial Measurement Unit

So far, we've mostly been talking about sensing the environment, but how does the robot know what direction its facing or how fast it is moving?

An Inertial Measurement Unit (IMU) contains an accelerometer which measures acceleration and a gyroscope which measures angular velocity.

These are typically used in Inertial Navigation Systems (INS) along with GPS to calculate orientation, angular rates, linear velocity, and position.

The one shown here is used in spacecraft, which require very accurate information about where they are, so the IMU ends up being a lot larger and more expensive than the kinds used for other platforms.

Reasoning

Let's move on to the second area now, reasoning. Once you have raw data from your sensors, what do you do with it and how do you make decisions about what actions to take?

Simultaneous Localization and Mapping (SLAM)

Imagine you are blindfolded and dropped into an unknown location and need to find your way out. You would probably start by moving around and building a mental map of your environment while keeping track of key landmarks so you can figure out where you are. This is basically how algorithms like Simultaneous Localization and Mapping (SLAM) work. The robot uses sensor readings to build a map based on what it can currently see. Then, it moves around and updates its estimate of where it is in the map. This is called localization. This process is continuously repeated to produce results like those shown in the figure on the left.

One of them most commonly used localization techniques is the particle filter. This is a type of Kalman Filter that represents probability distributions as a set of particle estimates rather than mathematical distributions like a Gaussian. The figure on the right shows how the particle distribution is initially evenly spread across the entire map, meaning the robot has no idea where it is. As it moves around and collects data, it is able to refine its estimate by comparing the sensor reading it actually measures to what the sensor reading theoretically would have been at each of the estimate locations. The particles that best match the sensor readings get a higher likelihood which allows the distribution to "collapse" towards where the robot actually is.

(Left) Teach Kids Robotics | (Right) Udacity

Path Planning

After building a map and figuring out where you are, you probably want to get towards a goal location. There are various path planning techniques that can find a path. Some have guarantees for finding the most optimal path in terms of distance or other metrics, but may take longer to run than techniques involving randomness.

The figure on the left shows a potential field where warmer colors indicate more dangerous areas and cooler colors indicate safer areas. Path A might be shorter, but not as safe as path B.

The figure on the right shows the rapidly exploring random tree (RRT*) algorithm which, as the name implies, uses randomness to explore the map and with enough iterations converges towards the most optimal path.

(Left) Reality Bytes | (Right) Akshay Kumar

Computer Vision

Maps and paths are great, but maybe you want to react differently to different types of obstacles. This is where object detection and classification through computer vision techniques comes in handy. Convolutional Neural Networks, a type of machine learning algorithm, are used extensively to help autonomous vehicles detect other vehicles on the road, pedestrians, trees, speed limit signs, traffic lights, etc. You can use this information to better plan routes if you know that, for example, a tree is not going to move over time, but a pedestrian might.

Acting

Finally, let's talk about acting. You've come up with a path you would like to follow to get to your goal based on sensor readings from the environment, but how do you actually move yourself where you want to go?

Motion Planning

Path planning is like what Google Maps gives you. You know the general route, but you as the driver need to control the vehicle. Motion planning involves planning paths around dynamic obstacles in your local environment. The figure shown here illustrates how an autonomous vehicle might need to switch lanes and plan around the motion of other vehicles to get towards its goal. It needs to take into account the constraints of the vehicle itself since the car cannot just teleport to wherever it wants to go. It can only control its speed and steering angle, so the types of paths it can follow (shown by the white splines in the figure) are limited.

Controls

TODO - talk about real-world examples, motor controls, etc.

Machine Learning

TODO - reinforcement learning like AWS racer, etc.

ROS - Robot Operating System

Now that you hopefully have some understanding of what is involved in robotics, we can talk a bit about how you actually implement the algorithms we just went over.

As I mentioned at the start of this presentation, robotics is a combination of hardware and software. It involves mechanical, electrical, and computer hardware being controlled by software.

The Robot Operating System (ROS) is a free and open source project built on top of Ubuntu that provides a set of software libraries and tools to make building advanced robots easier.

ROS

TODO - talk about topics/messages for communicating between services, message recording / playback, visualization / teleoperation tools, etc.

ROS 1 vs 2

TODO - discuss differences between ROS 1 and 2 in terms of overall design

Live Demo

Show live demo of MITRE Fast-RRT* and/or RRRobot - Recycling Robot projects. Or just make a recording and show that.

Thank You!

Website: sravanbalaji.com
GitHub: balajsra
LinkedIn: sravan-balaji

Robot Operating Systems Presented by Sravan Balaji CNN

Slides
Close