AirSimSensor Plugin Setup¶
SCRIMMAGE provides a sensor plugin that communicates over AirSim’s RPC protocol
to set the position and orientation of an entity in AirSim and retrieve the
camera images generated by AirSim. The
quad-airsim-ex1.xml mission file
provides an example for configuring SCRIMMAGE to use the AirSimSensor plugin
with a SCRIMMAGE entity. An example of extracting the image data from the
Sensor is provided in the Straight Autonomy plugin in the Straight.cpp file.
Linux System Setup¶
In order to run SCRIMMAGE and AirSim / Unreal on Linux, we will have to build Unreal and AirSim from source. Some of the following directions were taken from AirSim’s build_linux.md documentation, but they are placed here for posterity.
The Unreal Engine uses libc++ while SCRIMMAGE uses libstdc++. You cannot easily link binaries that are built from one library with binaries that were built with the other library. The simplest way of dealing with this issue is to build the AirSim libraries/plugins in two different directories.
Make sure you are registered with Epic Games. This is required to get source code access for Unreal engine.
Clone Unreal in your favorite folder and build it (this may take a while!). Note: We only support Unreal 4.16 and newer.
# go to folder where you clone GitHub projects git clone -b 4.17 https://github.com/EpicGames/UnrealEngine.git cd UnrealEngine # few times Epic folks broke the build so we will get commit that works git checkout af96417313a908b20621a443175ba91683c238c8 ./Setup.sh ./GenerateProjectFiles.sh make
Build the libstdc++ version of AirSim (which will be loaded into SCRIMMAGE)
# go to folder where you clone GitHub projects git clone https://github.com/Microsoft/AirSim.git cd AirSim git checkout v1.1.8 git apply /path/to/scrimmage/3rd-party/patches/airsim_build_updates.patch ./setup.sh cd cmake chmod +x gcc_build.sh ./gcc_build.sh
Build the libc++ version of AirSim (which will be loaded into Unreal)
# go to folder where you clone GitHub projects git clone https://github.com/Microsoft/AirSim.git AirSim-plugin cd AirSim-plugin git checkout v1.1.8 ./setup.sh ./build.sh
Build SCRIMMAGE’s AirSimSensor Plugin
Make sure you have OpenCV installed on your system and that you have successfully built the AirSim libraries (using libstdc++) in step 3. Go to your scrimmage build directory and provide the location of the AirSim libraries (libstdc++ version) to cmake.
cd /path/to/scrimmage/build cmake .. -DAIRSIM_ROOT_SEARCH=/path/to/AirSim make
Ensure that the
AirSimSensor_plugintarget built successfully.
Open the Blocks Environment in Unreal
Go to the UnrealEngine repository that you cloned and run the UE4Editor binary that was built.
cd /path/to/UnrealEngine ./Engine/Binaries/Linux/UE4Editor
Use the UE4Editor to open the Blocks project (Blocks.uproject) in the /path/to/AirSim-plugin/Unreal/Environments/Blocks directory. You will probably be prompted about updating the project version, say “Yes” to update. Now we have to sync the AirSim plugin with this updated project version. Close the project and UE4Editor and open a terminal.
cd /path/to/AirSim-plugin rsync -t -r Unreal/Plugins ./Unreal/Environments/Blocks\ Blocks 4.17
Run the UE4Editor binary again and open the new Blocks 4.17 project. When you hit the “Play” button in the project, you should see a quadrotor appear on the screen and it may start flying around.
AirSim reads a json file located at ~/Documents/AirSim/settings.json to configure itself. SCRIMMAGE provides an example json file. After running AirSim one time, it will create the json file. Let’s remove it and link to the one provided by SCRIMMAGE.
rm ~/Documents/AirSim/settings.json cd ~/Documents/AirSim ln -s /path/to/scrimmage/include/scrimmage/plugins/sensor/AirSimSensor/settings.json .
Open the configuration file for the AirSimSensor SCRIMMAGE plugin, which is located at: /path/to/scrimmage/include/scrimmage/plugins/sensor/AirSimSensor/AirSimSensor.xml. Note that the
airsim_portXML tags can be used to connect to an AirSim instance on either the local computer or a remote networked computer.
airsim_port(from AirSimSensor.xml) and
ApiServerPort(from settings.json) should match. Multiple simulated cameras can be configured through the
camera_configexample tag. This tag takes a list of camera configurations, where each camera configuration is of the form:
[CameraName ImageType CameraNumber Width Height]. The following example configures multiple simulated camera sensors:
<camera_config> [SceneForward Scene 0 256 144] [SceneDownward Scene 3 256 144] [DepthPlanner DepthPlanner 0 256 144] [DepthPerspective DepthPerspective 0 256 144] [DepthViz DepthVis 0 256 144] [DisparityNormalized DisparityNormalized 0 256 144] [Segmentation Segmentation 0 256 144] [SurfaceNormals SurfaceNormals 0 256 144] </camera_config>
CameraNameis an arbitrary string that can be used by SCRIMMAGE plugins to identify camera images. The ImageType is a type defined by AirSim that specifies the type of camera (optical, depth, segmentation, etc.). The camera number refers to different camera translations and orientations on the Unreal actor instance. 0 looks forward, 3 looks down, etc.
Run a simulation with AirSim Camera Images
After configuring settings.json and AirSimSensor.xml, you can run a simulation. Use the UE4Editor to open the updated Blocks environment and press “Play” to start the AirSim instance. Now, you can start the SCRIMMAGE instance, which will move the AirSim actor, request images, and display the images.
You should see camera images open in OpenCV windows when the SCRIMMAGE simulation begins (hit ‘b’ to unpause, if necessary).
Windows / Linux Setup¶
You can run the Unreal Engine on a Windows computer and SCRIMMAGE on a Linux computer. You will need to build the libstdc++ version of AirSim on your Linux computer (see above: “Build the libstdc++ version of AirSim”) and build the AirSim SCRIMMAGE plugin (see above “Build SCRIMMAGE’s AirSimSensor Plugin”) on your Linux computer.
See AirSim’s documentation for either downloading the binaries for Windows or building it on Windows. Make sure you that you use the same version of AirSim across your systems. For example, at the time that this tutorial was written, we used AirSim version v1.1.8.