Export as HDF5

Download hands back the raw .capture package: a folder of CSVs, JSON and video. Export as HDF5 writes the same data as one .h5 file with typed datasets, a shared time axis and the media embedded, for notebooks, robotics pipelines and anyone who would rather not parse CSVs.

Exporting

Right-click a completed capture in the Captures table or tile grid and choose Export as HDF5. A worker writes the file, so the toast follows the job (waiting for a worker, writing, uploading) and the download starts when it is done. A capture whose export is still current hands back the same file without a new job; editing the transcript makes the last export stale, and the next ask writes a fresh one. Picture captures have no package and do not offer the export.

The conventions

The group tree is built from three published conventions, each one checkable with a tool you already have, rather than a layout of our own.

HDF Group specifications (file level)

Dimension Scales. Every axis of every gridded dataset carries a scale: time for the sample axis, pose_component (x y z qx qy qz qw), joint, tracked_object, state_component. This is the mechanism netCDF-4 is built on, so xarray, netCDF4 and MATLAB read the axes without custom code.
Image and Palette Specification 1.2 on image datasets (CLASS="IMAGE", IMAGE_VERSION="1.2", IMAGE_SUBCLASS="IMAGE_TRUECOLOR", INTERLACE_MODE="INTERLACE_PIXEL"), so HDFView renders them.
Opaque datatypes with a MIME tag for embedded container bytes: the camera .mov files are H5T_OPAQUE tagged video/quicktime, the microphone .wav is tagged audio/wav. Readers know what they are holding without a sidecar.
Chunked storage with gzip, shuffle and Fletcher-32 checksums on anything that is not tiny; one self-contained file with no external links; strings as variable-length UTF-8.

CF-1.11 attributes (metadata level)

Root: Conventions="CF-1.11, HolosCapture-1.0", title, institution, source (device and app), history (exporter version and time), references (this page), capture_id, organization_id, modality, capture_start.
Every numeric dataset: units (m, s, ns, 1), long_name, and _FillValue (NaN) where a stream can be absent.
time: float64 with units="seconds since <capture start, ISO-8601 UTC>" and calendar="proleptic_gregorian", so xarray decodes it to datetimes. timestamp_ns beside it is int64 nanoseconds since the Unix epoch, the ROS 2 convention.
The coordinate frame is stated once on the demo group and never left implicit: frame="arkit_world", handedness="right", up_axis="Y", quaternion_order="xyzw", length_unit="m".

robomimic dataset layout (domain level)

robomimic’s HDF5 format is the de facto shape for learning-from-demonstration data (robosuite, MimicGen, LIBERO and ManiSkill exporters all read or write it). One capture is one demo:

/data with attrs total (sample count) and env_args (JSON: env_name="holos_capture", type=99 meaning no simulator behind this, and the capture’s metadata under env_kwargs).
/data/demo_0 with attrs num_samples and the frame conventions above, holding obs/*, actions, states, time and timestamp_ns.
/mask/all listing demo_0, so filter_by_attribute="all" works.

Everything Holos-specific lives outside /data, under /holos and /media, so a stock robomimic loader never sees it.

What is inside

/pose_component, /joint, /tracked_object      shared coordinate scales (string arrays)
/data                                         attrs: total, env_args
/data/demo_0                                  attrs: num_samples, frame, handedness, up_axis, quaternion_order, length_unit, master_clock, resampling
/data/demo_0/time                             float64 [T]        seconds since capture start (dimension scale)
/data/demo_0/timestamp_ns                     int64   [T]        epoch nanoseconds
/data/demo_0/obs/timestamp                    float64 [T]        same values as time, as an obs key
/data/demo_0/obs/device_pose                  float64 [T, 7]     headset pose, ARKit world frame
/data/demo_0/obs/hand_left_joints_world       float64 [T, J, 7]  J = 26 ARKit HandSkeleton joints (names on the joint scale)
/data/demo_0/obs/hand_right_joints_world      float64 [T, J, 7]
/data/demo_0/obs/hand_left_joints_local       float64 [T, J, 7]  relative to the wrist anchor
/data/demo_0/obs/hand_right_joints_local      float64 [T, J, 7]
/data/demo_0/obs/object_pose                  float64 [T, K, 7]  tracked object anchors (ids and names on the tracked_object scale)
/data/demo_0/obs/hand_{left,right}_landmarks_3d   float64 [T, 21, 3]  iPhone: Vision landmarks in metres (names on the landmark scale)
/data/demo_0/obs/hand_{left,right}_landmarks_confidence float64 [T, 21]  iPhone: per-landmark confidence, 0 = not seen
/data/demo_0/obs/imu_accel, imu_gyro          float64 [T, 3]     iPhone or derived: user acceleration (m s-2), rotation rate (rad s-1), device frame
/data/demo_0/actions                          float64 [T, 7]     dominant-hand wrist delta (see below)
/data/demo_0/states                           float64 [T, S]     device pose + wrist poses, layout on the state_component scale
/holos/raw/device_pose/{time, timestamp_ns, pose}                every sample, on its own clock
/holos/raw/hand_pose_world/{left,right}/{time, timestamp_ns, joints}
/holos/raw/hand_pose_local/{left,right}/{time, timestamp_ns, joints}
/holos/raw/object_pose/<anchor id>/{time, timestamp_ns, pose}
/holos/raw/imu/{time, timestamp_ns, accel, gyro}              iPhone / derived: m s-2 and rad s-1, device frame; attr source
/holos/raw/hand_landmarks_3d/{left,right}/{time, timestamp_ns, points, confidence, hand_confidence}   iPhone: Vision's 21 landmarks in metres
/holos/raw/hand_landmarks_2d/{left,right}/{time, timestamp_ns, points, confidence, hand_confidence}   iPhone: normalized image u, v
/holos/files/<path>                           opaque bytes, MIME-tagged  any package file nothing above types
/holos/metadata/session                       JSON string         metadata.json verbatim
/holos/metadata/stereo_calibration            JSON string
/holos/metadata/thumbnail                     uint8 [H, W, 3]     Image Spec 1.2
/holos/transcript/segments                    compound (start_s, end_s, text, is_final)
/holos/transcript/tokens                      compound (segment, start_s, end_s, text)
/holos/transcript/full_text                   string
/holos/events/procedure_steps                 compound (index, step_id, title, t_start, t_end, start_sample, end_sample)
/holos/events/procedure_steps_json            JSON string         events/procedure_steps.json verbatim
/holos/vlm/output                             compound (t_mono, t_wall, prompt, output)
/media/<camera>                               opaque bytes, video/quicktime; attrs codec, width, height, fps, duration_s. camera_left, camera_right and camera_sbs on Vision Pro; camera_back on iPhone
/media/audio                                  opaque bytes, audio/wav; attrs sample_rate_hz, channels

Only what the package holds is written. Sources the capture lacks are listed in the job’s notes as skipped. A file the exporter has no typed home for is embedded verbatim under /holos/files/<path> with its MIME type and listed in the notes as an extra, so a new track is never dropped silently.

Vision Pro and iPhone packages

A Vision Pro capture carries ARKit’s 26-joint hand skeleton (hand_*_joints_*) and stereo video. An iPhone capture carries Vision’s 21 hand landmarks instead (hand_*_landmarks_3d in metres and _2d in image space, each landmark with a confidence, 0 where Vision did not see it), a Core Motion IMU track, one back-camera video, and a tracking_state column on the device pose. Both export into the same tree; the demo group’s master_clock says which hand track set the clock, and source on the root names the device.

One clock for the demo, every clock under raw

The device records streams at different rates: the head pose at about 60 Hz, each hand at about 90 Hz on Vision Pro and about 15 Hz on iPhone, the IMU at 100 Hz, object anchors when they move. robomimic assumes one T, so /data/demo_0 samples everything on the dominant hand’s clock (the right hand when it was tracked, else the left, ARKit’s skeleton before Vision’s landmarks, else the device) by zero-order hold: each sample carries the latest value of every other stream, and NaN where a stream has nothing yet or its newest value is older than 0.25 s. The demo group’s master_clock and resampling attrs say exactly this. Nothing is thrown away: /holos/raw keeps every stream at its own rate with its own time scale.

What `actions` means for a human demonstration

robomimic loaders require the key, and a person has no commanded action. actions[t] is the dominant hand’s wrist pose delta: the position difference to the next sample and the relative rotation q[t+1] * conj(q[t]) (xyzw), with an identity delta on the last row. On an iPhone capture, where Vision’s wrist landmark carries no rotation, it is the [T, 3] position delta alone. The dataset’s derivation and source_hand attrs state it, so nobody has to guess.

Procedure steps as sample ranges

When the capture was recorded against a procedure, /holos/events/procedure_steps carries each step visit with t_start and t_end in capture seconds and start_sample and end_sample as indices into /data/demo_0 (end exclusive), so a step is a slice: obs[start:end].

Reading it

import h5py, json

with h5py.File("capture.h5") as f:
    print(f.attrs["Conventions"], f.attrs["title"])
    demo = f["data/demo_0"]
    t = demo["time"][...]                                   # seconds since capture start
    right = demo["obs/hand_right_joints_world"][...]        # [T, 26, 7]
    joints = [j.decode() for j in f["joint"][...]]
    wrist = right[:, joints.index("forearmWrist"), :3]      # [T, 3] metres, ARKit world
    env = json.loads(f["data"].attrs["env_args"])
    steps = f["holos/events/procedure_steps"][...]
    for s in steps:
        print(s["title"].decode(), t[s["start_sample"]], "to", t[s["end_sample"] - 1])

import xarray as xr

obs = xr.open_dataset("capture.h5", group="data/demo_0/obs", engine="h5netcdf")
obs["hand_right_joints_world"].sel(joint="forearmWrist", pose_component=["x", "y", "z"]).plot.line(x="time")

raw = xr.open_dataset("capture.h5", group="holos/raw/device_pose", engine="h5netcdf")
print(raw["pose"].sizes)   # {'time': 5486, 'pose_component': 7}

from robomimic.utils.dataset import SequenceDataset
import robomimic.utils.file_utils as FileUtils

meta = FileUtils.get_env_metadata_from_dataset("capture.h5")    # env_name "holos_capture"
ds = SequenceDataset(
    hdf5_path="capture.h5",
    obs_keys=["device_pose", "hand_right_joints_world"],
    dataset_keys=["actions"],
    seq_length=10,
    filter_by_attribute="all",
)
sample = ds[0]

import h5py, numpy as np

with h5py.File("capture.h5") as f:
    ds = f["media/camera_left"]
    print(ds.id.get_type().get_tag())            # b'video/quicktime'
    raw = np.empty(ds.shape, dtype="V1")
    ds.id.read(h5py.h5s.ALL, h5py.h5s.ALL, raw, mtype=ds.id.get_type())
    open("camera_left.mov", "wb").write(raw.tobytes())

Decoded frames

The export embeds the camera files as bytes rather than decoding them: a ten-minute stereo capture decoded to uint8 arrays is tens of gigabytes. Asking the API with decodeFrames: true adds /holos/raw/video/<camera>/frames (uint8 [F, H, W, 3], Image Spec 1.2, one frame per chunk, with its own time scale) and /data/demo_0/obs/<camera>_frame_index (int64 [T], the frame nearest each demo sample), so a loader can gather frames per sample without the file repeating them.

The convention version

The root attrs carry holos_convention_version="HolosCapture-1.0". On top of robomimic’s layout, 1.0 adds:

the shared coordinate scales at the root (pose_component, joint, tracked_object, axis_xyz, landmark, image_uv) and the per-demo state_component and action_component scales
timestamp_ns, time as a CF dimension scale, and the frame attrs on the demo group
/holos (metadata, raw streams, transcript, events, VLM) and /media
actions as the wrist delta, declared in derivation

A change to any of this is a new version here, not a surprise in a file.

API

Endpoint	What it does
`POST /captures/{captureId}/export-hdf5`	Queue an export (202), return the open one (200), or return a current finished export with a presigned download (200 `ready`). Body: `force` to re-export regardless, `decodeFrames` to add frame arrays.
`GET /captures/{captureId}/export-hdf5`	`status` (`none`, `queued`, `in_progress`, `ready`, `failed`), `progress`, `artifact` (`filename`, `size`, `downloadUrl`, `resolvedAt`, `current`) and `error`.

The file lives under the capture’s exports/ prefix and is removed with the capture.

Overview

Manage

Exporting

The conventions

What is inside

Vision Pro and iPhone packages

One clock for the demo, every clock under raw

What `actions` means for a human demonstration

Procedure steps as sample ranges

Reading it

Decoded frames

The convention version

API

​Exporting

​The conventions

​What is inside

​Vision Pro and iPhone packages

​One clock for the demo, every clock under raw

​What actions means for a human demonstration

​Procedure steps as sample ranges

​Reading it

​Decoded frames

​The convention version

​API

Exporting

The conventions

What is inside

Vision Pro and iPhone packages

One clock for the demo, every clock under raw

What `actions` means for a human demonstration

Procedure steps as sample ranges

Reading it

Decoded frames

The convention version

API