Quickstart

evalio is available on PyPi, so simply install via your favorite python package manager,

uv add evalio      # uv
pip install evalio # pip

evalio can be used both as a python library and as a CLI for both datasets and pipelines.

Datasets

Once evalio is installed, datasets can be listed and downloaded via the CLI interface. For example, to list all datasets and then download a sequence from the hilti-2022 dataset,

evalio ls datasets
evalio download hilti_2022/basement_2

evalio downloads data to the path given by -D, EVALIO_DATA environment variable, or if both are unset to the local folder ./evalio_data. All the trajectories in a dataset can also be downloaded by using the wildcard hilti_2022/*, making sure to escape the asterisk as needed.

Tip

evalio also comes with autocomplete, which makes typing the long dataset and pipeline names much easier. To install, do one of the following,

eval "$(evalio --show-completion)" # install for the current session
evalio --install-completion        # install for all future sessions

Note

Many datasets use gdown to download datasets from google drive. Unfortunately, this can occasionally be finicky due to google's download limits, however downloading cookies from your browser can often help.

Once downloaded, a trajectory can then be easily used in python,

from evalio import datasets as ds

# for all data
for mm in ds.Hilti2022.basement_2:
    print(mm)

# for lidars
for scan in ds.Hilti2022.basement_2.lidar():
    print(scan)

# for imu
for imu in ds.Hilti2022.basement_2.imu():
    print(imu)

For example, you can easily get a single scan to plot a bird-eye view,

import matplotlib.pyplot as plt
import numpy as np

# get the 10th scan
scan = ds.Hilti2022.basement_2.get_one_lidar(10)
# always in row-major order, with stamp at start of scan
x = np.array([p.x for p in scan.points])
y = np.array([p.y for p in scan.points])
z = np.array([p.z for p in scan.points])
plt.scatter(x, y, c=z, s=1)
plt.axis('equal')
plt.show()

evalio also comes with a built wrapper for converting to rerun types,

import rerun as rr
from evalio.rerun import convert

rr.init("evalio")
rr.connect_tcp()
for scan in ds.Hilti2022.basement_2.lidar():
    rr.set_time("timeline", timestamp=scan.stamp.to_sec())
    rr.log("lidar", convert(scan, color=[255, 0, 255]))

Note

To run the rerun visualization, rerun must be installed. This can be done by installing rerun-sdk or evalio[vis] from PyPi.

Once installed, rerun must be spawned from the CLI simply by running rerun in a terminal. This will start the rerun viewer, which can then be used to visualize the data logged by evalio.

We recommend checking out the API reference for more information on how to interact with datasets, and the example for an example of how to create your own dataset.

Pipelines

The other half of evalio is the pipelines that can be run on various datasets. All pipelines and their parameters can be shown via,

evalio ls pipelines

For example, to run KissICP on a dataset,

evalio run -o results -d hilti_2022/basement_2 -p kiss

This will run the pipeline on the dataset and save the results to the results folder. The results can then be used to compute statistics on the trajectory,

evalio stats results

Note

KissICP does poorly by default on hilti_2022/basement_2, due to the close range and large default voxel size. You can visualize this by adding -s ms to the run command to visualize the map and scan in rerun.

More complex experiments can be run, including varying pipeline parameters, via specifying a config file,

# If not specified, defaults to ./evalio_results/config_file_name
output_dir: ./results/

datasets:
  # Run on all of newer college trajectories
  - hilti_2022/*
  # Run on first 1000 scans of multi campus
  - name: multi_campus/ntu_day_01
    length: 1000

pipelines:
  # Run vanilla kiss with default parameters
  - kiss
  # Tweak kiss parameters
  - name: kiss_tweaked
    pipeline: kiss
    deskew: true
    # Perform a parameter sweep over other parameters
    sweep:
      voxel_size: [0.1, 0.5, 1.0]
      # Can also eval python code to create sweep (including numpy functionality)
      initial_threshold: linspace(0.5, 2.0, 4)

This can then be run via

evalio run -c config.yml

Additionally, the run command supports visualization via rerun as well. -v will do a simple visualization of the ground truth trajectory and odometry, while -s can be used to enable additional visualizations,

evalio run -o results -d hilti_2022/basement_2 -p kiss -s msif

where m -> map, s -> scan, i -> intensity image, and f -> extracted features can all be selectively visualized in rerun.

Note

As mentioned above, rerun must be installed and launched to visualize the results.

That's about the gist of it! Try playing around the CLI interface to see what else is possible. Feel free to open an issue if you have any questions, suggestions, or problems.

Additionally, we recommend checking out the examples section for specific use cases for evalio.