> ## Documentation Index
> Fetch the complete documentation index at: https://docs.efference.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Video Capture

> The open / grab / retrieve loop, pixel formats, and compression modes

Video capture uses the `Device` data plane. USB carries control and pixels on
one cable; a wireless session uses BLE for control and WiFi/UDP for pixels.

## Capture loop

`open()` configures the session. The first `grab()` starts the live data plane
and waits for a frame. `retrieve_image()` copies that frame into an owned
`Mat`.

```cpp theme={null}
Device dev;
InitParameters init;
init.resolution  = RESOLUTION::AUTO;
init.fps         = 30;
init.compression = COMPRESSION_MODE::H265;
if (dev.open(init) != ERROR_CODE::SUCCESS) return 1;

Mat image;
while (true) {
    ERROR_CODE ec = dev.grab();
    if (ec == ERROR_CODE::GRAB_TIMEOUT)    continue;  // no frame yet, keep looping
    if (ec == ERROR_CODE::CORRUPTED_FRAME) continue;  // lossy frame dropped
    if (ec != ERROR_CODE::SUCCESS)         break;     // transport fault

    ERROR_CODE image_ec = dev.retrieve_image(image, VIEW::RGBA);
    if (image_ec == ERROR_CODE::CORRUPTED_FRAME) continue;
    if (image_ec != ERROR_CODE::SUCCESS)         break;

    // image.getPtr()       owned pixel buffer
    // image.getTimestamp() capture time, device clock
    // image.getFrameId()   monotonic counter
}
dev.close();
```

`grab()` returns `GRAB_TIMEOUT` when no frame arrived within
`InitParameters::grab_timeout_ms` (default 1000 ms). It is non-fatal, so keep looping.
`CORRUPTED_FRAME` is also non-fatal when partial delivery is enabled.
`END_OF_BUFFER` is the normal exit for MCAP replay. Other non-`SUCCESS`
results should end or recover the session.

`retrieve_image()` can also return `CORRUPTED_FRAME`. Skip that image and
continue to the next `grab()`.

## Select a capture mode

The public capability menu contains only modes enabled on the connected M1:

```cpp theme={null}
DeviceInformation info = dev.get_device_information();
for (const SupportedMode& mode : info.capabilities.modes) {
    std::printf("%dx%d @ %d\n",
                mode.resolution.width,
                mode.resolution.height,
                mode.fps);
}
```

Each mode also reports `binning`, either `"none"` or `"2x2"`, which is how it
derives from the full sensor.

Set the session mode in `InitParameters` before `open()`. To change the
device's persistent boot configuration instead, stop live work and call
`set_configuration()` while the device is `IDLE`.

Frames arrive as raw fisheye unless the device is set to rectify them itself.
See [Calibration](/device/calibration#on-device-rectification).

## Views

`retrieve_image()` converts to the requested `VIEW`:

| VIEW            | Layout                    | Typical use                        |
| --------------- | ------------------------- | ---------------------------------- |
| `NV12`          | planar Y + interleaved UV | native wire format                 |
| `RGBA` / `BGRA` | packed 4-channel          | rendering, GPU upload              |
| `RGB` / `BGR`   | packed 3-channel          | OpenCV (`BGR` is its native order) |
| `GRAY`          | single channel            | feature tracking                   |

Decoding and pixel-format conversion both need FFmpeg. Built without it, the
only working combination is `COMPRESSION_MODE::RAW` retrieved as `VIEW::NV12`;
everything else returns `UNSUPPORTED_COMPRESSION`, including `NV12` itself
when the session uses an encoded codec, which is the default. The standard
`./build.sh --deps` install includes FFmpeg.

The bundled `<ef/OpenCV.hpp>` header provides a zero-copy OpenCV view:

```cpp theme={null}
#include <ef/OpenCV.hpp>

dev.retrieve_image(image, VIEW::BGR);   // BGR = OpenCV's native channel order
cv::Mat frame = ef::toCvMat(image);     // zero-copy view, no conversion
```

The `cv::Mat` shares the `ef::Mat` buffer. Call `.clone()` when OpenCV must own
the pixels beyond the next retrieval.

## Compression modes

Set once in `InitParameters::compression`:

| Mode                  | Wire format       | Notes                               |
| --------------------- | ----------------- | ----------------------------------- |
| `RAW`                 | uncompressed NV12 | no decode needed, but USB only      |
| `H264` / `H265`       | encoded           | default `H265`; decode needs FFmpeg |
| `H264_HQ` / `H265_HQ` | near-lossless     | high fixed-quality tier             |

Raw NV12 requires approximately 830 Mbit/s at 1200p30, which exceeds the
capacity of the WiFi link, so requesting `RAW` with `udp_host` set is rejected at
`open()` with `INSUFFICIENT_WIFI_BANDWIDTH`. Use an encoded codec over WiFi/UDP.
Raw capture is supported on the wired connection, which provides sufficient
bandwidth.

The device validates the resolution/fps/codec tuple against its enabled
capability menu at `open()`; a bad combination fails with
`INVALID_RESOLUTION`, `INVALID_FPS`, or `UNSUPPORTED_COMPRESSION`. The menu
itself is in `get_device_information().capabilities`.

## Flip

A camera mounted upside-down can be corrected host-side with
`InitParameters::flip_mode`:

* `OFF` (default) and `ON` are fixed
* `AUTO` latches once from the first suitable IMU gravity sample

The flip applies to retrieved images only, never to device-local recordings.
Image-only consumers are supported: `retrieve_image()` peeks at the latest
acceleration sample without draining the IMU queue.

## Timestamps

`Mat::getTimestamp()` is the capture time on the **device clock**, which `open()`
aligns to the host clock (`sync_time()`). `dev.get_timestamp(TIME_REFERENCE::CURRENT)`
returns host wall time for latency measurements.

## Stop capture

`close()` stops this handle's live data plane and releases the control
transport. It also finalizes an active host-file recording. It does not stop a
device-local recording.