AirSimSensor Plugin Setup¶

SCRIMMAGE’s AirSimSensor Plugin can be used to generate photo-realistic camera images and realistic LIDAR/ IMU data during a multiple drone simulation created using the SCRIMMAGE simulation architecture. SCRIMMAGE provides a sensor plugin that communicates over AirSim’s RPC protocol to set the position and orientation of an entity in AirSim and retrieve the camera images, LIDAR and IMU data generated by AirSim. This sensor data can be used along with the SCRIMMAGE simulation architecture to design computer vision and navigation algorithms for autonomous aerial systems. The quad-airsim-ex1.xml mission file provides an example for configuring SCRIMMAGE to use the AirSimSensor plugin with a SCRIMMAGE entity. The SCRIMMAGE ROSAirSim plugin gives an example of publishing image and LIDAR data to ROS from SCRIMMAGE messages.

System Setup¶

In order to run SCRIMMAGE’s AirSimSensor Plugin there are 2 setup options: Linux Only or Linux/Windows setup. Both options require a Linux machine because SCRIMMAGE only runs on a Linux operating system (Ubuntu 18.04 Recommended). First we will setup SCRIMMAGE/AirSim on a Linux machine, afterward we will setup Unreal/AirSim on either the same(or different) Linux machine or on a Windows machine.

Some may find the 2 machine setup, such as Linux/ Windows, useful because two machines can handle the processing load for larger simulations better than one machine. Furthermore, you could also achieve the Linux Only setup using 2 Linux machines by installing SCRIMMAGE/AirSim on one Linux machine and Unreal/AirSim on another Linux machine. All 2 machine options require the machines are on the same local network.

Again, the Linux Only setup can be achieved using one Linux machine. The Linux/Windows setup could also be achieved by running a Linux virtual machine on a Windows machine, therefore only requiring one Windows machine (try at own discretion).

There is no difference in features between the Linux Only or Linux/Windows setups, however Unreal Engine being originally Windows based may have some gaps in support for Linux. For more information see AirSim’s build_linux.md documentation: https://github.com/microsoft/AirSim/blob/master/docs/build_linux.md

Note

You will notice we clone and build AirSim twice in this tutorial. Unreal Engine uses libc++ while SCRIMMAGE uses libstdc++. You cannot easily link binaries that are built from one library with binaries that were built with the other library. The simplest way of dealing with this issue is to build the AirSim library in two different directories.

1. Setup SCRIMMAGE/AirSim on your Linux machine¶

Ubuntu 18.04 Recommended

Clone SCRIMMAGE and install dependencies

Some of the following directions were taken from SCRIMMAGE’s README.md documentation, but they are placed here for completeness.

# go to folder where you clone GitHub projects
git clone https://github.com/gtri/scrimmage.git

# Install SCRIMMAGE dependencies
cd scrimmage
sudo ./setup/install-binaries.sh -e 0 -p 3
sudo add-apt-repository ppa:kevin-demarco/scrimmage
sudo apt-get update
sudo apt-get install scrimmage-dependencies scrimmage-jsbsim
echo "source ~/.scrimmage/setup.bash" >> ~/.bashrc
source ~/.scrimmage/setup.bash

Build the libstdc++ version of AirSim (which will be loaded into SCRIMMAGE)

# go to folder where you clone GitHub projects
git clone https://github.com/Microsoft/AirSim.git AirSim-gcc
cd AirSim-gcc
git checkout 9e3aed1b020b46f9fb975b4ab343b06364bdd544
git apply /path/to/scrimmage/3rd-party/patches/airsim_build_updates.patch
./setup.sh --gcc
./build.sh --gcc

Build SCRIMMAGE’s AirSimSensor Plugin

Go to your SCRIMMAGE build directory and provide the location of the AirSim libraries (gcc version) to cmake to build:
cd /path/to/scrimmage rm -rf build && mkdir build && cd build cmake .. -DAIRSIM_ROOT_SEARCH=/path/to/AirSim-gcc make
Ensure that the AirSimSensor_plugin target built successfully.

2. Setup Unreal/AirSim on Linux OR Windows¶

Make sure you are registered with Epic Games. This is required to get source code access for Unreal engine.

Visit https://github.com/EpicGames/Signup for instructions.
Unreal/AirSim Setup (this may take a while!). Note: We only support Unreal Engine 4.24 and newer.
Note

Below you will either follow the Linux Unreal/AirSim Setup or the Windows Unreal/AirSim Setup, but not both. See beginning of System Setup for more details.

Linux Unreal/AirSim Setup

Some of the following directions were taken from AirSim’s build_linux.md documentation, but they are placed here for completeness.
######################################### # Clone and Build Unreal Engine for Linux ######################################### # go to folder where you clone GitHub projects git clone -b 4.24 https://github.com/EpicGames/UnrealEngine.git cd UnrealEngine ./Setup.sh ./GenerateProjectFiles.sh make ####################################################################### # Build the libc++ version of AirSim (which will be loaded into Unreal) ####################################################################### # go to folder where you clone GitHub projects git clone https://github.com/Microsoft/AirSim.git AirSim-clang cd AirSim-clang git checkout 9e3aed1b020b46f9fb975b4ab343b06364bdd544 ./setup.sh ./build.sh
Windows Unreal/AirSim Setup

Some of the following directions were taken from AirSim’s build_windows.md documentation, but they are placed here for completeness.
- Install Unreal Engine
[Download](https://www.unrealengine.com/download) the Epic Games Launcher. While the Unreal Engine is open source and free to download, registration is still required.

Run the Epic Games Launcher, open the Library tab on the left pane.

Click on the Add Versions which should show the option to download Unreal 4.24.

If you have multiple versions of Unreal installed then make sure 4.24 is set to current by clicking down arrow next to the Launch button for the version.
- Build AirSim
1. Install Visual Studio 2019. Make sure to select Desktop Development with C++ and Windows 10 SDK 10.0.18362 (should be selected by default) while installing VS 2019.

Start the Windows Program Developer Command Prompt for VS 2019.

# Clone the AirSim repo into or close to the *c:/ directory*, else you will receive errors during installation for filenames being too long git clone https://github.com/Microsoft/AirSim.git # Go to the AirSim directory and checkout the same commit used when setting up SCRIMMAGE/AirSim on Linux cd AirSim && git checkout 9e3aed1b020b46f9fb975b4ab343b06364bdd544 # Run build.cmd from the command line. This will create plugin bits in the Unreal\Plugins folder that can be dropped into any Unreal project. build.cmd
Open the Blocks Environment in Unreal Engine
Go the Linux or Windows machine that you setup Unreal/AirSim. Some of the following directions were taken from AirSim’s unreal_blocks.md documentation, but they are placed here for completeness.
Open Blocks in Linux Unreal Engine
Go to the UnrealEngine repository that you cloned and run the UE4Editor binary that was built.

cd /path/to/UnrealEngine ./Engine/Binaries/Linux/UE4Editor

Use the UE4Editor to open the Blocks project (Blocks.uproject) in the /path/to/AirSim-plugin/Unreal/Environments/Blocks directory. You will probably be prompted about copying the project for updating the project version. Look under “more details” and select convert the project “in place.”

When you hit the “Play” button in the project, choose quadcopter, and you should see a quadrotor appear on the screen and it may start flying around.

If you receive an error due to Unreal requiring your GPU to have Vulkan Drivers:

First install the Latest Tested, Proprietary NVIDIA Driver using this tutorial https://www.cyberciti.biz/faq/ubuntu-linux-install-nvidia-driver-latest-proprietary-driver/

Install Vulcan:

sudo add-apt-repository ppa:graphics-drivers/ppa sudo apt upgrade sudo apt install vulkan-utils
Open Blocks in Windows Unreal Engine
Navigate to folder AirSim\Unreal\Environments\Blocks and run update_from_git.bat.

Double click on generated .sln file to open in Visual Studio 2019.

Make sure Blocks project is the startup project, build configuration is set to DebugGame_Editor and Win64. Hit F5 to run.

Press the Play button in Unreal Editor, choose quadcopter, and you should see a quadrotor appear on the screen and it may start flying around.

3. Run the Default AirSimSensor Simulation¶

Configure AirSim Settings for SCRIMAMGE

Unreal/AirSim reads a json file located at ~/Documents/AirSim/settings.json to configure itself on the machine where Unreal/AirSim is running. SCRIMMAGE AirSimSensor provides an example json file that corresponds with the default AirSimSensor simulation and AirSimSensor.xml settings. After running AirSim to open the Blocks environment, it will create an empty json file. Let’s remove it and copy the one provided by SCRIMMAGE AirSimSensor to the ~/Documents/AirSim/ directory.
```
### If Unreal/AirSim is running on Linux:
rm ~/Documents/AirSim/settings.json
cd ~/Documents/AirSim
cp /path/to/scrimmage/include/scrimmage/plugins/sensor/AirSimSensor/settings.json .

### If Unreal/AirSim is running on Windows:
# Replace ~/Documents/AirSim/settings.json with file located here:
# https://github.com/gtri/scrimmage/blob/master/include/scrimmage/plugins/sensor/AirSimSensor/settings.json
```

Run the Default SCIMMAGE AirSimSensor Simulation

Find the IP of the machine that you are running Unreal/AirSim. If Unreal/AirSim is running on the same Linux machine you installed SCRIMMAGE use the Local Host IP (usually 127.0.0.1). Place the Unreal/AirSim IP in the quad-airsim-ex1.xml mission file in scrimmage on your Linux machine. Run the mission in scrimmage, watch AirSim connect, and hit ‘b’ to start the simulation.

# Open "quad-airsim-ex1.xml" in /scrimmage/missions directory using your favorite text editor
nano /path/to/scrimmage/missions/quad-airsim-ex1.xml
# change "airsim_ip" to your Unreal/AirSim IP
<sensor airsim_ip="YOUR_IP">AirSimSensor</sensor>
# Save and run the mission
scrimmage ./quad-airsim-ex1.xml
# Should connect successfully
# hit 'b' key to start simulation
# You should see 4 image streams appear in separate windows from the scrimmage window.
# for directions on how to control scrimmage see scrimmage github:
# (https://github.com/gtri/scrimmage)

4. Setup Photo-Realistic Environments¶

AirSim offers photo-realistic environments that can be played(not edited) for every release version called “Assets”. They can be found here: (https://github.com/Microsoft/AirSim/releases). Below is a picture of the LandscapeMountains Environment.

Windows Setup
If you are running Unreal/AirSim on a Windows machine, download one of the Asset ZIP files under v1.2.2-Windows and place in the directory: c:/path/to/AirSim/Unreal/Environments/. Assets for newer versions of AirSim will not work with older versions of AirSim installed on your Windows machine. Use 7-Zip to extract by selecting “Extract to <ASSET_NAME>". Inside the folder you will find a <ASSET_NAME>.exe application. Double click the application file to start the environment, choose “no” for quadcopter. The environment will start in full-screen. Now start the scrimmage mission on your Linux machine using “scrimmage ./missions/quad-airsim-ex1.xml”. The scrimmage simulation will connect to AirSim and control the car/ quadcopter in the new Asset environment. To end the simulation you will need to enter the Task Manager by pressing CTRL+ALT+DELETE, select the Unreal process, and press “End Task”.

Linux Setup
If you are running Unreal/AirSim on a Linux machine, download one of the Asset ZIP files under v1.3.1-Linux and place in the directory: ~/path/to/AirSim-clang/Unreal/Environments/. Extract the zip file, and run as shown below:
### Extract and run:
unzip LandscapeMountains.zip
./LandscapeMountains/LandscapeMountains.sh -windowed
# If the window captures your mouse use ALT+Tab to escape
For more information on developing with Unreal Engine in Linux visit: Epic Games

Configure AirSimSensor Settings¶

Vehicle Settings¶

Multiple Vehicles
The default SCRIMMAGE AirSimSensor mission file, quad-airsim-ex1.xml, and the settings.json file given in this tutorial are already setup to use 2 quadcopters. You will notice there are 2 entity tags stated in the quad-airsim-ex1.xml mission file. You can add as many Entities/quadcopters as you need, however if you are saving images from each quadcopter you will eventually see a lag in the simulation as you add more and more quadcopters. You can add an additional quadcopter by adding a new entity tag with count=1 in the quad-airsim-ex1.xml mission file and adding a new vehicle to the Documents/AirSim/settings.json file on the machine running Unreal/AirSim. please use unique, lowercase names for naming each vehicle.

Each new Entity you add must contain a “<sensor>AirSimSensor</sensor>” tag (and if you are using ROS an “<autonomy>ROSAirSim</autonomy>” tag). The vehicle_name variables in the quad-airsim-ex1.xml mission file under the “<sensor>AirSimSensor</sensor>” tags MUST match both the VehicleName variables used in the AirSimSensor.xml configuration file and the user chosen names of vehicles in the Documents/AirSim/settings.json file on the machine running Unreal/AirSim under “Vehicles{}”, by default these are “robot1” and “robot2”. Failure to use matching names/ settings will result in an RPC error.

Vehicle Pose
To set custom vehicle pose relative to Latitude/Longitude you must specify the vehicle pose in ENU frame coordinates in the quad-airsim-ex1.xml mission file and the same vehicle coordinates in NED frame in the settings.json file. For the scrimmage and AirSim simulations to match up you must do this conversion correctly.

To Transform between ENU and NED coordinate frames -> switch x & y, negate z. A great tool for checking conversions is: https://www.andre-gaschler.com/rotationconverter/

Warning

For vehicles please leave Z=0.0 in the settings.json file or you will experience a simulation viewing error. Only set Z, initial altitude, in the quad-airsim-ex1.xml mission file.

Image Settings¶

AirSimSensor.xml
Open the configuration file for the AirSimSensor SCRIMMAGE plugin, which is located at: /path/to/scrimmage/include/scrimmage/plugins/sensor/AirSimSensor/AirSimSensor.xml.

Warning

The settings in the AirSimSensor.xml must match the corresponding settings in the Unreal/AirSim ~/Documents/AirSim/settings.json file or the SCRIMMAGE simulation and the Unreal/AirSim simulation will not match up and you will receive an RPC error on start-up.

Multiple simulated cameras can be configured through the camera_config example tag. This tag takes a list of camera configurations, where each camera configuration is of the form:

[VehicleName CameraName ImageType Width Height FOV]

The following shows example configurations for simulating multiple camera sensors per vehicle:
<camera_config>
  [robot1 front_center Scene 256 144 90]
  [robot1 front_center DepthPlanner 256 144 90]
  [robot1 front_center DepthPerspective 256 144 90]
  [robot1 front_center DepthVis 256 144 90]

  [robot2 front_right DisparityNormalized 256 144 90]
  [robot2 front_left Segmentation 256 144 90]
  [robot2 bottom_center SurfaceNormals 256 144 90]
  [robot2 back_center Infrared 256 144 90]
</camera_config>
The VehicleName corresponds to the user chosen vehicle name set in settings.json file under “Vehicles{}”, please use unique, lowercase names for naming each vehicle. The CameraName corresponds to the user chosen camera name set in settings.json file under “Cameras{}” for each vehicle, please use unique, lowercase names when naming cameras and sensors. Camera Settings, such as camera location on the vehicle body can be changed in the settings.json file on the on the machine running Unreal/Airsim located in /Documents/AirSim. Camera pose relative to the vehicle body must be specifed in NED frame in the settings.json file. All variables below MUST also match settings in the settings.json file. The ImageType is a type defined by AirSim that specifies the type of camera (optical, depth, segmentation, etc.). Each Camera can have multiple ImageTypes requested from it, an example of this setup is shown above for robot1. The Width and Height variables correspond to the pixel dimensions of the images we want to receive in scrimmage from AirSim. The FOV variable corresponds to the camera field of view (integer) in degrees.To learn more about the options available in AirSim see: https://github.com/microsoft/AirSim/blob/master/docs/image_apis.md
quad-airsim-ex1.xml
Open the mission file for the AirSimSensor SCRIMMAGE plugin, which is located at: /path/to/scrimmage/missions/quad-airsim-ex1.xml. Read the notes under the commented tag <-- AirSimSensor Plugin -->. Set get_image_data="true" to request image data from Unreal/ AirSim. If requesting data from AirSim, set save_airsim_data="true" to save all images requested and a CSV of vehicle quaternion pose values (ENU, World frame) to scrimmage logs (~/.scrimmage/logs) on Linux. The image_acquisition_period determines the number of times image data is requested from AirSim per second.

LIDAR Settings¶

AirSimSensor.xml
Open the configuration file for the AirSimSensor SCRIMMAGE plugin, which is located at: /path/to/scrimmage/include/scrimmage/plugins/sensor/AirSimSensor/AirSimSensor.xml.

Warning

The settings in the AirSimSensor.xml must match the corresponding settings in the Unreal/AirSim ~/Documents/AirSim/settings.json file or the SCRIMMAGE simulation and the Unreal/AirSim simulation will not match up and you will receive an RPC error on start-up.

Multiple simulated LIDARs can be configured under the lidar_config tag in AirSimSensor.xml. This tag takes a list of LIDAR configurations, where each LIDAR configuration is of the form:

[VehicleName LidarName]

The following shows example configurations for simulating multiple LIDAR sensors per vehicle:
<lidar_config>
  [robot1 lidar1]
  [robot1 lidar2]

  [robot2 lidar1]
  [robot2 lidar2]
</lidar_config>
The VehicleName corresponds to the user chosen vehicle name set in settings.json file under “Vehicles{}”, please use unique, lowercase names for naming each vehicle. The LidarName corresponds to the user chosen LIDAR name set in settings.json file under “Sensors{}” for each vehicle, please use unique, lowercase names when naming cameras and sensors. LIDAR settings, such as LIDAR location on the vehicle body, can be changed in the settings.json file on the on the machine running Unreal/Airsim located in /Documents/AirSim. See (https://github.com/microsoft/AirSim/blob/master/docs/lidar.md) for more details. LIDAR pose relative to the vehicle body must be specifed in NED frame in the settings.json file. Lidar variable DrawDebugPoints in the settings.json file will show the LIDAR pointcloud in the simulation as seen in the image below, however it will also appear in images saved to the local machine so by default it is set to false.
quad-airsim-ex1.xml
Open the mission file for the AirSimSensor SCRIMMAGE plugin, which is located at: /path/to/scrimmage/missions/quad-airsim-ex1.xml. Read the notes under the commented tag <-- AirSimSensor Plugin -->. Set get_lidar_data="true" to request LIDAR data from Unreal/ AirSim. The lidar_acquisition_period determines the number of times image data is requested from AirSim per second.

IMU Settings¶

AirSimSensor.xml
Open the configuration file for the AirSimSensor SCRIMMAGE plugin, which is located at: /path/to/scrimmage/include/scrimmage/plugins/sensor/AirSimSensor/AirSimSensor.xml.

Warning

The settings in the AirSimSensor.xml must match the corresponding settings in the Unreal/AirSim ~/Documents/AirSim/settings.json file or the SCRIMMAGE simulation and the Unreal/AirSim simulation will not match up and you will receive an RPC error on start-up.

Multiple simulated IMUs can be configured under the imu_config tag in AirSimSensor.xml. This tag takes a list of IMU configurations, where each IMU configuration is of the form:

[VehicleName ImuName]

The following shows example configurations for simulating multiple imu sensors per vehicle:
<imu_config>
  [robot1 imu1]
  [robot1 imu2]

  [robot2 imu1]
  [robot2 imu2]
</imu_config>
The VehicleName corresponds to the user chosen vehicle name set in settings.json file under “Vehicles{}”, please use unique, lowercase names for naming each vehicle. The ImuName corresponds to the user chosen IMU name set in settings.json file under “Sensors{}” for each vehicle, please use unique, lowercase names when naming cameras and sensors. IMU settings, such as IMU location on the vehicle body, can be changed in the settings.json file on the machine running Unreal/Airsim located in /Documents/AirSim. See (https://github.com/microsoft/AirSim/blob/master/docs/sensors.md) for more details. IMU sensor pose relative to the vehicle body must be specifed in NED frame in the settings.json file.
quad-airsim-ex1.xml
Open the mission file for the AirSimSensor SCRIMMAGE plugin, which is located at: /path/to/scrimmage/missions/quad-airsim-ex1.xml. Read the notes under the commented tag <-- AirSimSensor Plugin -->. Set get_imu_data="true" to request IMU data from Unreal/ AirSim. The imu_acquisition_period determines the number of times imu data is requested from AirSim per second.

Publish Data to ROS¶

First ensure that you have ROS installed on your Linux machine running SCRIMMAGE/AirSim. To publish AirSim data to ROS you must build scrimmage with -DBUILD_ROS_PLUGINS=ON, example below. Open the mission file for the AirSimSensor SCRIMMAGE plugin, which is located at: /path/to/scrimmage/missions/quad-airsim-ex1.xml.
ROSAirSim
Uncomment the “<autonomy>ROSAirSim</autonomy>” tag in the scrimmage mission file quad-airsim-ex1.xml for each entity. Please set the vehicle_name to te same setting as the vehicle_name in the AirSimSensor tag in the same entity. Setting show_camera_images="true will display images from each camera type in OpenCV windows. By default images are shown using the “<autonomy>Straight</autonomy>” tag in quad-airsim-ex1.xml, the show_camera_images setting should only be set to “true” in one plugin tag per entity.

If you would like to publish sensor data to ROS be sure to have the corresponding get_(sensor)_data variable set to true in the AirSimSensor tag in the same entity in order to receive the data from AirSim. Within the ROSAirSim tag:

pub_image_data="true" publishes images to rostopic “/<VehicleName>/camera/<CameraName>/<ImageType>/raw”

pub_lidar_data="true" publishes LIDAR data to rostopic “/<VehicleName>/<LidarName>”

pub_imu_data="true" publishes IMU data to rostopic “/<VehicleName>/<ImuName>”

There are also some ROS configuration options. Set ros_python="true" to specify for ROSAirSim to publish image messages that are compatible with the ROS Python API. Setting ros_python="false" will specify for ROSAirSim to publish images using the Image Transports ROS message type which is compatible with other ROS packages such as OpenVSLAM’s ROS package. Set ros_cartographer="true" for ROS AirSim to organize it’s Transform Tree (TF2) structure in a way that is compatible with ROS Cartographer.

SCRIMMAGE ROS Data Plugins
In the quad-airsim-ex1.xml mission file you will also find some SCRIMMAGE internal ROS plugins. These publish data generated by SCRIMMAGE to ROS. Unlike the AirSim plugins you can not add multiple of these sensors to an entity or specify their position on the vehicle (coming soon).

Uncomment <sensor>ROSAltimeter</sensor> to publish simulated Altimeter data to ROS.

Uncomment <sensor>ROSCompass</sensor> to publish simulated Compass data to ROS.

Uncomment <sensor>ROSIMUSensor</sensor> to publish simulated IMU data to ROS.

Build and Run with ROS
# Go into Scrimmage
cd /path/to/scrimmage/
# Delete build directory
rm -rf build/ && mkdir build/ && cd build/

# Config CMake to build SCRIMMAGE ROS Plugins
cmake .. -DAIRSIM_ROOT_SEARCH=/path/to/AirSim-gcc/ -DROS_VERSION=melodic -DBUILD_ROS_PLUGINS=ON
# Build
make -j7

# Open a second Terminal window and start ROS
roscore

# Run from original Terminal window
cd .. && scrimmage ./missions/quad-airsim-ex1.xml

# Press 'b' on your keyboard to unpause the simulation.
# More about SCRIMMAGE controls here: https://github.com/gtri/scrimmage#gui-commands
View in ROS RVIZ
# Start RVIZ in a third Terminal window to visualize Transform Trees and LIDAR data
rviz ./scrimmage/include/scrimmage/plugins/autonomy/ROSAirSim/rviz.rviz