evalio.datasets
For more information about the datasets included in evalio, see the included datasets section.
Bases: StrEnum
The base class for all datasets.
This class provides an interface for loading datasets, including loading parameters and iterating over measurements. All datasets are string enums, where each enum member represents a trajectory sequence in the dataset.
property
The folder in the global dataset directory where this dataset is stored.
Returns:
-
Path(Path) –Path to the dataset folder.
property
Full name of the dataset, including the dataset name and sequence name.
Example: "dataset_name/sequence_name"
Returns:
-
str(str) –Full name of the dataset.
property
Name of the sequence, in snake case.
Returns:
-
str(str) –Name of the sequence.
Main interface for iterating over measurements of all types.
Returns:
-
Iterator[Measurement]–Iterator[Measurement]: Iterator of all measurements (IMU and Lidar).
Return the number of lidar scans.
If quick_len is available, it will be used. Otherwise, it will load the entire dataset to get the length.
Returns:
-
int(int) –Number of lidar scans.
data_iter() -> DatasetIterator
Provides an iterator over the dataset's measurements.
Returns:
-
DatasetIterator(DatasetIterator) –An iterator that yields measurements from the dataset.
classmethod
Name of the dataset, in snake case.
This is the name that will be used when parsing directly from a string. Currently is automatically generated from the class name, but can be overridden.
Returns:
-
str(str) –description
Method to download the dataset.
Completely optional to implement, although most datasets do.
Raises:
-
NotImplementedError–If not implemented.
Environment where the dataset was collected.
Returns:
-
str(str) –Environment where the dataset was collected.
Return list of files required to run this dataset.
If a returned type is a Path, it will be checked as is. If it is a string, it will be prepended with folder.
Returns:
-
Sequence[str | Path]–list[str]: description
get_one_imu(idx: int = 0) -> ImuMeasurement
Get a single IMU measurement.
Note, this can be expensive to compute, as it will iterate over the entire dataset until it finds the measurement.
Parameters:
-
idx(int, default:0) –Index of measurement to get. Defaults to 0.
Returns:
-
ImuMeasurement(ImuMeasurement) –The IMU measurement at the given index.
get_one_lidar(idx: int = 0) -> LidarMeasurement
Get a single Lidar measurement.
Note, this can be expensive to compute, as it will iterate over the entire dataset until it finds the measurement.
Parameters:
-
idx(int, default:0) –Index of measurement to get. Defaults to 0.
Returns:
-
LidarMeasurement(LidarMeasurement) –The Lidar measurement at the given index.
ground_truth() -> Trajectory
Get the ground truth trajectory in the IMU frame, rather than the ground truth frame as returned in ground_truth_raw.
Returns:
-
Trajectory(Trajectory) –The ground truth trajectory in the IMU frame.
ground_truth_raw() -> Trajectory
Retrieves the raw ground truth trajectory, as represented in the ground truth frame.
Returns:
-
Trajectory(Trajectory) –The raw ground truth trajectory data.
imu() -> Iterable[ImuMeasurement]
Iterate over just IMU measurements.
Returns:
-
Iterable[ImuMeasurement]–Iterable[ImuMeasurement]: Iterator of IMU measurements.
imu_T_gt() -> SE3
Retrieves the transformation from IMU to ground truth frame.
Returns:
-
SE3(SE3) –Transformation from IMU to ground truth frame.
imu_T_lidar() -> SE3
Returns the transformation from IMU to Lidar frame.
Returns:
-
SE3(SE3) –Transformation from IMU to Lidar frame.
imu_params() -> ImuParams
Verify if the dataset is downloaded.
Returns:
-
bool(bool) –True if the dataset is downloaded, False otherwise.
lidar() -> Iterable[LidarMeasurement]
Iterate over just Lidar measurements.
Returns:
-
Iterable[LidarMeasurement]–Iterable[LidarMeasurement]: Iterator of Lidar measurements.
lidar_params() -> LidarParams
Specifies the parameters of the Lidar.
Returns:
-
LidarParams(LidarParams) –Parameters of the Lidar.
Hardcoded number of lidar scans in the dataset, rather than computing by loading all the data (slow).
Returns:
-
Optional[int]–Optional[int]: Number of lidar scans in the dataset. None if not available.
classmethod
sequences() -> list[Dataset]
All sequences in the dataset.
Returns:
-
list[Dataset]–list[Dataset]: List of all sequences in the dataset.
Shows the size of the dataset on disk, in GB.
Returns:
-
Optional[float]–Optional[float]: Size of the dataset on disk, in GB. None if the dataset is not downloaded.
staticmethod
Webpage with the dataset information.
Returns:
-
str(str) –URL of the dataset webpage.
Bases: Iterable[Measurement]
This is the base class for iterating over datasets.
This class is the main interface used to iterate over the dataset's measurements. It provides an interface for iterating over IMU and Lidar measurements, as well as all measurements interleaved. This allows for standardizing access to loading data, while allowing for loading parameters in Dataset without having to load the data. Generally, will be created by the Dataset.data_iter method.
Main interface for iterating over all measurements.
Yields:
-
Measurement–Iterator[Measurement]: Iterator of all measurements (IMU and Lidar).
imu_iter() -> Iterator[ImuMeasurement]
Main interface for iterating over IMU measurements.
Yields:
-
ImuMeasurement–Iterator[ImuMeasurement]: Iterator of IMU measurements.
lidar_iter() -> Iterator[LidarMeasurement]
Main interface for iterating over Lidar measurements.
Yields:
-
LidarMeasurement–Iterator[LidarMeasurement]: Iterator of Lidar measurements.
Bases: DatasetIterator
An iterator for loading from rosbag files.
This is a wrapper around the rosbags library, with some niceties for converting ros PointCloud2 messages to a standardized format. Has identical methods to DatasetIterator.
__init__(
path: Path,
lidar_topic: str,
imu_topic: str,
lidar_params: LidarParams,
is_mcap: bool = False,
lidar_format: Optional[LidarFormatParams] = None,
custom_col_func: Optional[
Callable[[LidarMeasurement], None]
] = None,
)
Parameters:
-
path(Path) –Location of rosbag file(s). If a directory is passed, all .bag files in the directory will be loaded.
-
lidar_topic(str) –Name of lidar topic.
-
imu_topic(str) –Name of imu topic.
-
lidar_params(LidarParams) –Lidar parameters, can be gotten from lidar_params.
-
is_mcap(bool, default:False) –If an mcap file, will not try to glob over all rosbags. Defaults to False.
-
lidar_format(Optional[LidarFormatParams], default:None) –Various parameters for how lidar data is stored. If not specified, most will try to be inferred. We strongly recommend setting this to ensure data is standardized properly. Defaults to None.
-
custom_col_func(Optional[Callable[[LidarMeasurement], None]], default:None) –Function to put the point cloud in row major format. Will generally not be needed, except for strange default orderings. Defaults to None.
Raises:
-
FileNotFoundError–If the path is a directory and no .bag files are found.
-
ValueError–If the lidar or imu topic is not found in the bag file.
Bases: DatasetIterator
An iterator for loading from python iterables.
Interleaves imu and lidar iterables. Allows for arbitrary data to be loaded and presented in a consistent manner for the base Dataset class. Has identical methods to DatasetIterator.
__init__(
iter_lidar: Iterator[LidarMeasurement],
iter_imu: Iterator[ImuMeasurement],
num_lidar: int,
)
Parameters:
-
iter_lidar(Iterator[LidarMeasurement]) –An iterator over LidarMeasurement
-
iter_imu(Iterator[ImuMeasurement]) –An iterator over ImuMeasurement
-
num_lidar(int) –The number of lidar measurements
from evalio.datasets.loaders import RawDataIter
from evalio.types import ImuMeasurement, LidarMeasurement, Stamp
import numpy as np
# Create some fake data
imu_iter = (
ImuMeasurement(Stamp.from_sec(i), np.zeros(3), np.zeros(3))
for i in range(10)
)
lidar_iter = (LidarMeasurement(Stamp.from_sec(i + 0.1)) for i in range(10))
# Create the iterator
rawdata = RawDataIter(imu_iter, lidar_iter, 10)
Get the global data directory. This will be used to store the downloaded data.
Returns:
-
Path(Path) –Directory where datasets are stored.
Bases: Dataset
Dataset taken from a botanical garden, specifically for testing unstructured environments. Ground truth is gathered using a survey grade lidar.
Note, there is no automatic downloader for this dataset due to being uploaded on onedrive. Data can be downloaded manually and placed in the correct folder in EVALIO_DATA.
Additionally, we only include the public datasets here; more are potentially available upon request.
Bases: Dataset
Dataset taken in purposely degenerate locations such as a field, intersections, tunnels, and runways. All data comes directly from the Ouster unit.
Note, this dataset does not have ground truth orientation, only ground truth positional values taken from a Leica MS60 Prism.
Bases: Dataset
Self-driving car dataset taken in urban environments. Ground truth is generated using filtering of an RTK-GNSS system.
The vehicle had multiple lidar sensors mounted; we utilize the high resolution Ouster at the top of the vehicle.
Bases: Dataset
Sequences with ground truth taken from the Hilti 2022 SLAM Challenge, mostly taken from indoors.
Bases: Dataset
Data taken from a variety of campus (KTH, NTU, TUHH) in Asia and Europe at different seasons, at day and night, and with an ATV and handheld platform.
Ground truth was measured using a continuous optimization of lidar scans matched against a laser scanner map.
Bases: Dataset
Dataset taken from outdoor Oxford Campus. Ground truth is generated using ICP matching against a laser scanner.
Note, there have been some reports that the laser scanner and data were collected months apart, which may have caused some inaccuracies in the ground truth data.
There are two IMUs on the handheld device, but the realsense IMU is not time-synced with the lidar data. Therefore, we utilize the Ouster IMU data instead.
Bases: Dataset
Dataset outdoors on oxford campus with a handheld device consisting of an alphasense core and a Ouster lidar. Ground truth is captured ICP matching against a laser scanner map.
Note there are two IMUs present; we utilize the Ouster IMU (ICM-20948)) instead of the alphasense one (Bosch BMI085). We expect the Ouster IMU to have more accurate extrinsics and the specs between the two IMUs are fairly similar.
Bases: Dataset
Dataset taken both indoors and outdoors on the Oxford campus.
Note, we skip over a number of trajectories due to missing ground truth data.
Additionally, some of the ground truth has poses within a few milliseconds of each other - we skip over any ground truth values within 10 milliseconds of each other.