AirSimSensor Plugin Setup

SCRIMMAGE provides a sensor plugin that communicates over AirSim’s RPC protocol to set the position and orientation of an entity in AirSim and retrieve the camera images generated by AirSim. The quad-airsim-ex1.xml mission file provides an example for configuring SCRIMMAGE to use the AirSimSensor plugin with a SCRIMMAGE entity. An example of extracting the image data from the Sensor is provided in the Straight Autonomy plugin in the Straight.cpp file.

Linux System Setup

In order to run SCRIMMAGE and AirSim / Unreal on Linux, we will have to build Unreal and AirSim from source. Some of the following directions were taken from AirSim’s build_linux.md documentation, but they are placed here for posterity.

Note

The Unreal Engine uses libc++ while SCRIMMAGE uses libstdc++. You cannot easily link binaries that are built from one library with binaries that were built with the other library. The simplest way of dealing with this issue is to build the AirSim libraries/plugins in two different directories.

  1. Make sure you are registered with Epic Games. This is required to get source code access for Unreal engine.

  2. Clone Unreal in your favorite folder and build it (this may take a while!). Note: We only support Unreal 4.16 and newer.

    # go to folder where you clone GitHub projects
    git clone -b 4.17 https://github.com/EpicGames/UnrealEngine.git
    cd UnrealEngine
    # few times Epic folks broke the build so we will get commit that works
    git checkout af96417313a908b20621a443175ba91683c238c8
    ./Setup.sh
    ./GenerateProjectFiles.sh
    make
    
  3. Build the libstdc++ version of AirSim (which will be loaded into SCRIMMAGE)

    # go to folder where you clone GitHub projects
    git clone https://github.com/Microsoft/AirSim.git
    cd AirSim
    git checkout v1.1.8
    git apply /path/to/scrimmage/3rd-party/patches/airsim_build_updates.patch
    ./setup.sh
    cd cmake
    chmod +x gcc_build.sh
    ./gcc_build.sh
    
  4. Build the libc++ version of AirSim (which will be loaded into Unreal)

    # go to folder where you clone GitHub projects
    git clone https://github.com/Microsoft/AirSim.git AirSim-plugin
    cd AirSim-plugin
    git checkout v1.1.8
    ./setup.sh
    ./build.sh
    
  5. Build SCRIMMAGE’s AirSimSensor Plugin

    Make sure you have OpenCV installed on your system and that you have successfully built the AirSim libraries (using libstdc++) in step 3. Go to your scrimmage build directory and provide the location of the AirSim libraries (libstdc++ version) to cmake.

    cd /path/to/scrimmage/build
    cmake .. -DAIRSIM_ROOT_SEARCH=/path/to/AirSim
    make
    

    Ensure that the AirSimSensor_plugin target built successfully.

  6. Open the Blocks Environment in Unreal

    Go to the UnrealEngine repository that you cloned and run the UE4Editor binary that was built.

    cd /path/to/UnrealEngine
    ./Engine/Binaries/Linux/UE4Editor
    

    Use the UE4Editor to open the Blocks project (Blocks.uproject) in the /path/to/AirSim-plugin/Unreal/Environments/Blocks directory. You will probably be prompted about updating the project version, say “Yes” to update. Now we have to sync the AirSim plugin with this updated project version. Close the project and UE4Editor and open a terminal.

    cd /path/to/AirSim-plugin
    rsync -t -r Unreal/Plugins ./Unreal/Environments/Blocks\ Blocks 4.17
    

    Run the UE4Editor binary again and open the new Blocks 4.17 project. When you hit the “Play” button in the project, you should see a quadrotor appear on the screen and it may start flying around.

  7. Configure Settings

    AirSim reads a json file located at ~/Documents/AirSim/settings.json to configure itself. SCRIMMAGE provides an example json file. After running AirSim one time, it will create the json file. Let’s remove it and link to the one provided by SCRIMMAGE.

    rm ~/Documents/AirSim/settings.json
    cd ~/Documents/AirSim
    ln -s /path/to/scrimmage/include/scrimmage/plugins/sensor/AirSimSensor/settings.json .
    

    Open the configuration file for the AirSimSensor SCRIMMAGE plugin, which is located at: /path/to/scrimmage/include/scrimmage/plugins/sensor/AirSimSensor/AirSimSensor.xml. Note that the airsim_ip and airsim_port XML tags can be used to connect to an AirSim instance on either the local computer or a remote networked computer. airsim_port (from AirSimSensor.xml) and ApiServerPort (from settings.json) should match. Multiple simulated cameras can be configured through the camera_config example tag. This tag takes a list of camera configurations, where each camera configuration is of the form: [CameraName ImageType CameraNumber Width Height]. The following example configures multiple simulated camera sensors:

    <camera_config>
      [SceneForward Scene 0 256 144]
      [SceneDownward Scene 3 256 144]
      [DepthPlanner DepthPlanner 0 256 144]
      [DepthPerspective DepthPerspective 0 256 144]
      [DepthViz DepthVis 0 256 144]
      [DisparityNormalized DisparityNormalized 0 256 144]
      [Segmentation Segmentation 0 256 144]
      [SurfaceNormals SurfaceNormals 0 256 144]
    </camera_config>
    

    The CameraName is an arbitrary string that can be used by SCRIMMAGE plugins to identify camera images. The ImageType is a type defined by AirSim that specifies the type of camera (optical, depth, segmentation, etc.). The camera number refers to different camera translations and orientations on the Unreal actor instance. 0 looks forward, 3 looks down, etc.

  8. Run a simulation with AirSim Camera Images

    After configuring settings.json and AirSimSensor.xml, you can run a simulation. Use the UE4Editor to open the updated Blocks environment and press “Play” to start the AirSim instance. Now, you can start the SCRIMMAGE instance, which will move the AirSim actor, request images, and display the images.

    scrimmage /path/to/scrimmage/missions/quad-airsim-ex1.xml
    

    You should see camera images open in OpenCV windows when the SCRIMMAGE simulation begins (hit ‘b’ to unpause, if necessary).

Windows / Linux Setup

You can run the Unreal Engine on a Windows computer and SCRIMMAGE on a Linux computer. You will need to build the libstdc++ version of AirSim on your Linux computer (see above: “Build the libstdc++ version of AirSim”) and build the AirSim SCRIMMAGE plugin (see above “Build SCRIMMAGE’s AirSimSensor Plugin”) on your Linux computer.

See AirSim’s documentation for either downloading the binaries for Windows or building it on Windows. Make sure you that you use the same version of AirSim across your systems. For example, at the time that this tutorial was written, we used AirSim version v1.1.8.