Render Engine Benchmarks

The benchmark consists of a scene with one or more spheres and one or more cameras above the spheres, looking down at the spheres.

If there are multiple spheres, they are positioned in a regular grid placed at a uniform height above the ground plane. Increasing the number of spheres provides an approximate measure of how the renderer performs with increased scene complexity.

If there are multiple cameras, they are all at the same position, looking in the same direction, with the same intrinsic properties. In other words, each should produce the same output image. This provides a measure of the scalability as a simulation includes an increasing number of cameras.

The output image can be configured to an arbitrary size; larger images take more rendering time.

Warm-starting RenderEngine Implementations

Some RenderEngine implementations may defer some of the initialization work until the rendering API is actually invoked (e.g., RenderEngineVtk). In these cases, the first rendering will be misleadingly expensive. The benchmark "warm starts" each render engine under evaluation by performing two renderings outside of the timed loop. This allows the benchmark to better report the expected results in the engine's "steady state".

When comparing these benchmark results with observed performance in applications, remember that the first and possibly second renderings may deceptively impact any analysis of average render performance.

Benchmarks

The benchmarks have been configured to examine the following characteristics of the included RenderEngine implementations:

• Scalability w.r.t. scene complexity
  • For a fixed image size, we render a sequence of images with varying numbers of spheres.
• Scalability w.r.t. number of cameras
  • For a fixed image size, render multiple cameras at the simplest and most complex scene complexities.
• Scalability w.r.t. image size
  • With a single camera, we render multiple output images of varying sizes with both simple and complex scenes.

We examine those same properties for all three image types: color, depth, and label.

Running the benchmark

The benchmark can be executed as:

bazel run //geometry/benchmarking:render_experiment -- \
   --output_dir=trial1

The output will be something akin to the following (although not identical; lines have been removed for brevity):

Run on (12 X 4400 MHz CPU s)
CPU Caches:
  L1 Data 32K (x6)
  L1 Instruction 32K (x6)
  L2 Unified 256K (x6)
  L3 Unified 12288K (x1)
Load Average: 9.75, 4.65, 3.26
------------------------------------------------------------------------------------  // NOLINT(*)
Benchmark                                          Time             CPU   Iterations  // NOLINT(*)
------------------------------------------------------------------------------------  // NOLINT(*)
RenderBenchmark/VtkColor/1/1/640/480           0.641 ms        0.641 ms         1026  // NOLINT(*)
RenderBenchmark/VtkColor/12/1/640/480          0.843 ms        0.843 ms          882  // NOLINT(*)
RenderBenchmark/VtkColor/120/1/640/480          2.58 ms         2.58 ms          280  // NOLINT(*)
RenderBenchmark/VtkColor/240/1/640/480          5.30 ms         5.30 ms          100  // NOLINT(*)
RenderBenchmark/VtkColor/480/1/640/480          9.10 ms         9.10 ms           79  // NOLINT(*)
RenderBenchmark/VtkColor/1200/1/640/480         23.4 ms         23.4 ms           30  // NOLINT(*)
RenderBenchmark/VtkColor/1/10/640/480           6.36 ms         6.36 ms          117  // NOLINT(*)
RenderBenchmark/VtkColor/1200/10/640/480         179 ms          179 ms            4  // NOLINT(*)
RenderBenchmark/VtkColor/1/1/320/240           0.267 ms        0.267 ms         2318  // NOLINT(*)
RenderBenchmark/VtkColor/1/1/1280/960           2.39 ms         2.39 ms          317  // NOLINT(*)
RenderBenchmark/VtkColor/1/1/2560/1920          11.5 ms         11.5 ms           65  // NOLINT(*)
RenderBenchmark/VtkColor/1200/1/320/240         24.1 ms         24.0 ms           30  // NOLINT(*)
RenderBenchmark/VtkColor/1200/1/1280/960        27.0 ms         26.9 ms           26  // NOLINT(*)
RenderBenchmark/VtkColor/1200/1/2560/1920       39.8 ms         39.7 ms           19  // NOLINT(*)
RenderBenchmark/GlColor/1/1/640/480            0.523 ms        0.480 ms         1382  // NOLINT(*)
RenderBenchmark/GlColor/12/1/640/480           0.553 ms        0.511 ms         1236  // NOLINT(*)
RenderBenchmark/GlColor/120/1/640/480          0.833 ms        0.790 ms          884  // NOLINT(*)
RenderBenchmark/GlColor/240/1/640/480           1.20 ms         1.15 ms          591  // NOLINT(*)
RenderBenchmark/GlColor/480/1/640/480           1.87 ms         1.83 ms          384  // NOLINT(*)
RenderBenchmark/GlColor/1200/1/640/480          3.76 ms         3.72 ms          188  // NOLINT(*)
RenderBenchmark/GlColor/1/10/640/480            4.98 ms         4.56 ms          137  // NOLINT(*)
RenderBenchmark/GlColor/1200/10/640/480         37.1 ms         36.6 ms           19  // NOLINT(*)
RenderBenchmark/GlColor/1/1/320/240            0.151 ms        0.147 ms         4830  // NOLINT(*)
RenderBenchmark/GlColor/1/1/1280/960            1.93 ms         1.88 ms          335  // NOLINT(*)
RenderBenchmark/GlColor/1/1/2560/1920           9.14 ms         9.08 ms           72  // NOLINT(*)
RenderBenchmark/GlColor/1200/1/320/240          3.46 ms         3.42 ms          199  // NOLINT(*)
RenderBenchmark/GlColor/1200/1/1280/960         5.29 ms         5.25 ms          127  // NOLINT(*)
RenderBenchmark/GlColor/1200/1/2560/1920        12.5 ms         12.5 ms           57  // NOLINT(*)
RenderBenchmark/VtkDepth/1/1/640/480            1.89 ms         1.89 ms          391  // NOLINT(*)
RenderBenchmark/GlDepth/1/1/640/480            0.486 ms        0.486 ms         1578  // NOLINT(*)
RenderBenchmark/VtkLabel/1/1/640/480            1.35 ms         1.35 ms          513  // NOLINT(*)
RenderBenchmark/GlLabel/1/1/640/480             1.34 ms         1.30 ms          521  // NOLINT(*)

Additional configuration is possible via the following flags:

• save_image_path: Enables saving the rendered images in the given location. Defaults to no saving.
• show_window: Whether to display the rendered images. Defaults to false.

For example:

bazel run //geometry/benchmarking:render_experiment -- \
   --output_dir=trial2 -- -save_image_path="/tmp" -show_window=true

Interpreting the benchmark

Each line in the table represents a particular configuration of the benchmark parameters of the form:

RenderBenchmark/TestName/sphere_count/camera_count/image_width/image_height

• TestName: One of
  • VtkColor: Renders the color image from RenderEngineVtk.
  • VtkDepth: Renders the depth image from RenderEngineVtk.
  • VtkLabel: Renders the label image from RenderEngineVtk.
  • GlColor: Renders the color image from RenderEngineGl.
  • GlDepth: Renders the depth image from RenderEngineGl.
  • GlLabel: Renders the label image from RenderEngineGl.
• sphere_count: The total number of spheres.
• camera_count: Simply the number of independent cameras being rendered. The cameras are all co-located (same position, same view direction) so they each render the same image.
• image width and image_height: The dimensions of the output image in pixels.

The Time and CPU columns are measures of the average time it took to create a single frame for all the specified cameras. The Iterations indicates how often the action was performed to compute the average value. For more information see the google benchmark documentation.

Now we can analyze the example output and draw some example inferences (not a complete set of valid inferences):

• Frame cost scales linearly with the number of cameras.
  • For RenderEngineVtk a 10X increase in number of cameras leads to a 10X increase in time for both color and depth, but a (roughly) 20X increase for label.
• RenderEngineGl scales better with respect to scene complexity than RenderEngineVtk.