Video Capture

Video capture uses the Device data plane. USB carries control and pixels on one cable; a wireless session uses BLE for control and WiFi/UDP for pixels.

Capture loop

open() configures the session. The first grab() starts the live data plane and waits for a frame. retrieve_image() copies that frame into an owned Mat.

Device dev;
InitParameters init;
init.resolution  = RESOLUTION::AUTO;
init.fps         = 30;
init.compression = COMPRESSION_MODE::H265;
if (dev.open(init) != ERROR_CODE::SUCCESS) return 1;

Mat image;
while (true) {
    ERROR_CODE ec = dev.grab();
    if (ec == ERROR_CODE::GRAB_TIMEOUT)    continue;  // no frame yet, keep looping
    if (ec == ERROR_CODE::CORRUPTED_FRAME) continue;  // lossy frame dropped
    if (ec != ERROR_CODE::SUCCESS)         break;     // transport fault

    ERROR_CODE image_ec = dev.retrieve_image(image, VIEW::RGBA);
    if (image_ec == ERROR_CODE::CORRUPTED_FRAME) continue;
    if (image_ec != ERROR_CODE::SUCCESS)         break;

    // image.getPtr()       owned pixel buffer
    // image.getTimestamp() capture time, device clock
    // image.getFrameId()   monotonic counter
}
dev.close();

grab() returns GRAB_TIMEOUT when no frame arrived within InitParameters::grab_timeout_ms (default 1000 ms). It is non-fatal, so keep looping. CORRUPTED_FRAME is also non-fatal when partial delivery is enabled. END_OF_BUFFER is the normal exit for MCAP replay. Other non-SUCCESS results should end or recover the session. retrieve_image() can also return CORRUPTED_FRAME. Skip that image and continue to the next grab().

Select a capture mode

The public capability menu contains only modes enabled on the connected M1:

DeviceInformation info = dev.get_device_information();
for (const SupportedMode& mode : info.capabilities.modes) {
    std::printf("%dx%d @ %d\n",
                mode.resolution.width,
                mode.resolution.height,
                mode.fps);
}

Each mode also reports binning, either "none" or "2x2", which is how it derives from the full sensor. Set the session mode in InitParameters before open(). To change the device’s persistent boot configuration instead, stop live work and call set_configuration() while the device is IDLE. Frames arrive as raw fisheye unless the device is set to rectify them itself. See Calibration.

Views

retrieve_image() converts to the requested VIEW:

VIEW	Layout	Typical use
`NV12`	planar Y + interleaved UV	native wire format
`RGBA` / `BGRA`	packed 4-channel	rendering, GPU upload
`RGB` / `BGR`	packed 3-channel	OpenCV (`BGR` is its native order)
`GRAY`	single channel	feature tracking

Decoding and pixel-format conversion both need FFmpeg. Built without it, the only working combination is COMPRESSION_MODE::RAW retrieved as VIEW::NV12; everything else returns UNSUPPORTED_COMPRESSION, including NV12 itself when the session uses an encoded codec, which is the default. The standard ./build.sh --deps install includes FFmpeg. The bundled <ef/OpenCV.hpp> header provides a zero-copy OpenCV view:

#include <ef/OpenCV.hpp>

dev.retrieve_image(image, VIEW::BGR);   // BGR = OpenCV's native channel order
cv::Mat frame = ef::toCvMat(image);     // zero-copy view, no conversion

The cv::Mat shares the ef::Mat buffer. Call .clone() when OpenCV must own the pixels beyond the next retrieval.

Compression modes

Set once in InitParameters::compression:

Mode	Wire format	Notes
`RAW`	uncompressed NV12	no decode needed, but USB only
`H264` / `H265`	encoded	default `H265`; decode needs FFmpeg
`H264_HQ` / `H265_HQ`	near-lossless	high fixed-quality tier

Raw NV12 requires approximately 830 Mbit/s at 1200p30, which exceeds the capacity of the WiFi link, so requesting RAW with udp_host set is rejected at open() with INSUFFICIENT_WIFI_BANDWIDTH. Use an encoded codec over WiFi/UDP. Raw capture is supported on the wired connection, which provides sufficient bandwidth. The device validates the resolution/fps/codec tuple against its enabled capability menu at open(); a bad combination fails with INVALID_RESOLUTION, INVALID_FPS, or UNSUPPORTED_COMPRESSION. The menu itself is in get_device_information().capabilities.

Flip

A camera mounted upside-down can be corrected host-side with InitParameters::flip_mode:

OFF (default) and ON are fixed
AUTO latches once from the first suitable IMU gravity sample

The flip applies to retrieved images only, never to device-local recordings. Image-only consumers are supported: retrieve_image() peeks at the latest acceleration sample without draining the IMU queue.

Timestamps

Mat::getTimestamp() is the capture time on the device clock, which open() aligns to the host clock (sync_time()). dev.get_timestamp(TIME_REFERENCE::CURRENT) returns host wall time for latency measurements.

Stop capture

close() stops this handle’s live data plane and releases the control transport. It also finalizes an active host-file recording. It does not stop a device-local recording.

Getting Started

Device

Command-Line Tools

Capture loop

Select a capture mode

Views

Compression modes

Flip

Timestamps

Stop capture

​Capture loop

​Select a capture mode

​Views

​Compression modes

​Flip

​Timestamps

​Stop capture

Capture loop

Select a capture mode

Views

Compression modes

Flip

Timestamps

Stop capture