Low FPS and AirSim as a viable RL training simulator

[HarrySoteriou]

I have searched through all issues both open and closed that highlight low FPS of simGetImages such as #3333, #3070, #2891 and #3018.

Question

What's your question?

How close are you to resolving the msgpack bottleneck and what are the most up to date tricks that can significantly improve FPS?

I think I saw something about compiling with specific binaries but I had a hard time compiling AirSim before so I don't want to risk it if it's not worth it. If at least 30fps are not attainable then AirSim is not a viable RL simulator.

Include context on what you are trying to achieve

I am training an RL script that takes as input low resolution RGB images (256x256x3) and then takes action. I started with 1-2FPS and started trying everything I could to increase my FPS because training 1 million timesteps took a whooping 5d7h and I can't run experiments that take so long.

I use the following function to make image calls but I have to ensure I am not getting an empty string binary or else training stops (hence the while loop and the .shape==0 check)

def _get_obs(self, responses=None):
    while responses is None:
      responses = self.image_client.simGetImages(
          [airsim.ImageRequest(self.camera_name, self.image_type, False, False)])
      img1d = np.fromstring(responses[0].image_data_uint8, dtype=np.uint8)
      if img1d.shape == 0:
          responses = None
      try:
          image = np.reshape(img1d, self.image_shape)
      except ValueError as e:
          print(e)
          responses = None
    return image

Context details

Specs: Windows 10.0.19044
UE: 4.27.2
AirSim: 1.6
Graphics Card: NVIDIA GeForce GTX 1080 Ti
RAM: 4GB
Python: 3.9.10
msgpack-python 0.5.6
msgpack-rpc-python 0.4.1

{
  "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
  "SettingsVersion": 1.2,
  "SimMode": "Multirotor",
  "CameraDefaults": {
    "CaptureSettings": [
      {
        "ImageType": 0,
        "Width": 256,
        "Height": 256,
        "FOV_Degrees": 84
      }
    ]
  },
    "Gimbal": {
      "Stabilization": 0.6,
      "Pitch": -90, "Roll": 0
    },
  "Recording": {
    "RecordOnMove": true,
    "RecordInterval": 0.05,
    "Folder": "D:\\Documents\\AirSim\\Recordings",
    "Enabled": false,
    "Cameras": [
        { "CameraName": "3", "ImageType": 0, "PixelsAsFloat": false,  "VehicleName": "SimpleFlight", "Compress": true }
    ]
  },
    "Vehicles": {
      "SimpleFlight": {
        "VehicleType": "SimpleFlight",
        "X": 2, "Y": 0, "Z": 0.0,
        "Pitch": -15.0, "Roll": 0.0,"Yaw": 0.0,
        "Sensors": {
          "Distance": {
            "SensorType": 5,
            "Enabled" : true,
            "MinDistance": 0.05,
            "MaxDistance": 80,
            "X": 0, "Y": 0, "Z": -1,
            "Pitch": -90, "Roll": 0, "Yaw": 0, 
            "DrawDebugPoints": false
        }
      }
    }
  }
}

Include details of what you already did to find answers

I have tried the following:

0. Request uncompressed images (you can see the code I used above)
1. Using separate image and drone clients -> No improvement
2. Setting "ViewMode" :"NoDisplay" -> No improvements
3. Decreasing duration of step_size during movement -> x2.5 improvement (from 0.25 to 0.1)
4. removing .join() from self.client.moveByVelocityAsync(vx, vy, vz, duration=0.1).join() --> 17fps
5. increase wallclockspeed from x1 to x3 ->19fps
6. Replaced simGetImages with simGetImage -> No improvements

Steps 4 and 5 cause instabilities (when evaluating a trained example that used asynchronous steps or increased wallclockspeed, it's behaviour was worse than random. An example trained for 1million time steps without using 4 and 5 converged to the expected behaviour). With everything I tried I am running training with 5-7 FPS.

[misakabribri]

Here are testing results on my machine:

Devices: Intel i7-9700K+NVIDIA GeForce 3070
Image size: 1270*720
Average rendering time: 40ms

I also need an approach to increase the FPS to 30.

[mgrova]

Hi Misaka (@misakabribri),
I am also trying to get a valid FPS for visual odometry tests (30 fps would be perfect), but I can't get more than 15 fps approx.

Have you made any progress in this time?

Best regards and thanks in advance!

[misakabribri]

Hi Marco Montes Grova (@mgrova)
Maybe you can package your UE4 project as a released app and then try again. By my observation, if the UE4 project's window is not active, it will also cause low fps. Also, I use multi threads in C++ code to reduce total time of the image capture loop.

[mgrova]

Hello Misaka (@misakabribri)!

I'm working with the binary of my own enviroment with "NoDisplay" as "ViewMode" in settings.json file. In addition, I use the C++ wrapper and ROS to grab the images. I'm going to try to generate the binaries in release mode, thanks!

Using multithreathing have you achieved 30 fps when grabbing the images in C++ in real-time?

[misakabribri]

It depens on the size of images you grabbing, I think. When I using multi-threads, I can achieve average 25fps(1280*720) on my workstation with GTX 3070. Besides, avoid using imwrite() and imshow() in the image-capturing thread, since these functions cost lots of time.

…

________________________________ 发件人: Marco Montes Grova ***@***.***> 发送时间: 2022年9月21日 5:57 收件人: microsoft/AirSim ***@***.***> 抄送: Misaka ***@***.***>; Mention ***@***.***> 主题: Re: [microsoft/AirSim] Low FPS and AirSim as a viable RL training simulator (Issue #4576) Hello @misakabribri<https://github.com/misakabribri>! I'm working with the binary of my own enviroment with "NoDisplay" as "ViewMode" in settings.json file. In addition, I use the C++ wrapper and ROS to grab the images. I'm going to try to generate the binaries in release mode, thanks! Using multithreathing have you achieved 30 fps when grabbing the images in C++ in real-time? ― Reply to this email directly, view it on GitHub<#4576 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AL2FJ5WJ7UG2PWQSFRY2ABDV7KPTLANCNFSM5Y6HQVKA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[mgrova]

Im taking the images (one depth and one scene) at 640x480. Simulating only these two cameras and using the ROS1 wrapper, I don't get more than 8-10 FPS in my workstarion with RTX 2080Ti.

Do you have any repository or code with which you can get that frequency when obtaining the images?

Sorry for the inconvenience and thank you very much.

[misakabribri]

I used AirSim's image capture APIs directly rather than ROS wrappers. I modified the C++ project named "Example" from AirSim default source code, and added a TCP thread to send images. Usually, it won't take too much time to get 640x480 images (less than 20ms) by capture APIs. I also used OpenCV APIs to incode images before transmitting them via TCP. In my opinion, you can record the total time that ROS nodes spend gathering images, for determing which places need further optimization.

…

________________________________ 发件人: Marco Montes Grova ***@***.***> 发送时间: 2022年9月22日 6:40 收件人: microsoft/AirSim ***@***.***> 抄送: Misaka ***@***.***>; Mention ***@***.***> 主题: Re: [microsoft/AirSim] Low FPS and AirSim as a viable RL training simulator (Issue #4576) Im taking the images (one depth and one scene) at 640x480. Simulating only these two cameras and using the ROS1 wrapper, I don't get more than 8-10 FPS in my workstarion with RTX 2080Ti. Do you have any repository or code with which you can get that frequency when obtaining the images? Sorry for the inconvenience and thank you very much. ― Reply to this email directly, view it on GitHub<#4576 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AL2FJ5S3QNLTJ7CSW27FFCTV7P5MBANCNFSM5Y6HQVKA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[jonyMarino]

Project AirSim has topics for images, they ran about 10 times faster than AirSim v1. Pixel Streaming is also enabled.
HarrySoteriou Marco Montes Grova (@mgrova) Misaka (@misakabribri) Matt Todd (@mtodd)