Moto Mods Firmware: Camera-Ext Protocol

Overview

The Camera-Ext protocol defines the interface for controlling a camera device in a Moto Mod. The protocol carries control signaling data to operate the camera in the Mod while the actual image data is sent over CSI-2 bus via Motorola High-Speed Bridge (MHB).

The Camera protocol interface provides the following functions:

  • The list of formats and resolutions supported by the camera device.
  • Power management of the camera device.
  • Streaming control of the camera device.
  • Sub-function control on the camera device.
  • State and Error reporting from the camera device to the core.

The Camera protocol is defined as “CAMERA_EXT” under Greybus protocol suite. There is a convenient framework called “mhb_camera” on which your camera driver can base off to fulfill the characteristic and behavior of the own camera device.


Hardware Manifest

0xFD is the identifier of the CAMERA_EXT protocol

The below is the example of the manifest lines contains the CAMERA_EXT protocol:

; CAMERA_EXT on CPort XX
[cport-descriptor XX]
bundle = YY
protocol = 0xed

; Camera Bundle YY as Camera Class
[bundle-descriptor YY]
class = 0x0d

Implementation

Files

nuttx/drivers/greybus/camera_ext-gb.c
      Defines CAMERA_EXT Greybus operations.
nuttx/include/nuttx/device_mhb_cam.h
      Defines the device driver operations to be implemented by MHB Camera driver.
nuttx/include/nuttx/camera/camera_ext.h
      Defines the structures to carry the list of formats and resolutions supported by the camera driver.
nuttx/include/nuttx/camera/v4l2_camera_ext_ctrls.h
      Defines the list of camera sub-function controls which can be used by the camera drivers.
nuttx/include/nuttx/camera/camera_ext_meta.h
      Defines the structures and methods of meta status information which can be sent from the camera driver to the core.
nuttx/include/nuttx/mhb/mhb_csi_camera.h
      Defines utility function to read/write the data from/to I2C bus with different data widths.

Protocol Version

The file "nuttx/drivers/greybux/camera_ext-gb.c" defines the protocol version used by CAMERA_EXT. Currently, only version 1.0 is available. An appropriate version should be selected for your driver in the future.

/* Version of the Greybus camera protocol we support */
#define GB_CAMERA_EXT_VERSION_MAJOR 0x01
#define GB_CAMERA_EXT_VERSION_MINOR 0x00

Exposing Camera Inputs, Formats, Resolutions and Frame Rates

The formats and the resolutions of the camera are defined in hierarchical structure like below. Each camera driver should define one set of its “format database”. It will then be passed up to the Core through CAMERA_EXT protocol. The Core software will select one set of ”Format”-”FrameSize”-”FrameInterval” depending on the setting of the camera application running in the Core.

--> Input[0] +--> Format[0] +--> FrameSize[0] +--> FrameInterval[0]
             |              |                 +--> FrameInterval[1]
             |              +--> FrameSize[1] +--> FrameInterval[0]
             |                                +--> FrameInterval[1]
             |                                +--> FrameInterval[2]
             |
             +--> Format[1] +--> FrameSize[0] +--> FrameInterval[0]
             |              +--> FrameSize[1] +--> FrameInterval[0]
             |
             +--> Format[3] +--> FrameSize[0] +--> FrameInterval[0]
                            |                 +--> FrameInterval[1]
                            +--> FrameSize[1] +--> FrameInterval[0]

The input is defined as a different types of stream supported by the camera device. It is expected there is only one Input node defined in the current MDK release.

struct camera_ext_input_node {
    const char *name;
    uint32_t type;
    uint32_t status;
    uint32_t capabilities;

    int num_formats;
    struct camera_ext_format_node const *format_nodes;
};

// used in capabilities field as a mask
#define CAMERA_EXT_STREAM_CAP_PREVIEW   (1 << 0)
#define CAMERA_EXT_STREAM_CAP_VIDEO     (1 << 1)
#define CAMERA_EXT_STREAM_CAP_SNAPSHOT  (1 << 2)

Each Input_node contains a list of format_node. Current supported formats in the Core are YUV422 (UYVY) 8 bits format and RAW10 bits formats.

//format for a specific input
struct camera_ext_format_node {
    const char *name;
    uint32_t fourcc;
    uint32_t depth;
    int num_frmsizes;
    struct camera_ext_frmsize_node const *frmsize_nodes;
};

// Supported Format
#define V4L2_PIX_FMT_UYVY    v4l2_fourcc('U', 'Y', 'V', 'Y')
#define V4L2_PIX_FMT_SBGGR10 v4l2_fourcc('B', 'G', '1', '0')
#define V4L2_PIX_FMT_SGBRG10 v4l2_fourcc('G', 'B', '1', '0')
#define V4L2_PIX_FMT_SGRBG10 v4l2_fourcc('B', 'A', '1', '0')
#define V4L2_PIX_FMT_SRGGB10 v4l2_fourcc('R', 'G', '1', '0')

The frame sizes under each format are defined in frmsize_node below. Note the maximum frame width are 3062 pixel (for YUV422) or 4912 pixel (for RAW10) due to current HW limitation.

/* frame size  a specific format */
struct camera_ext_frmsize_node {
    uint32_t width;for
    uint32_t height;
    int num_frmivals;
    struct camera_ext_frmival_node const *frmival_nodes;
};

The frame rate is defined in frmival_node. The actual frame interval will be calculated by numerator/denominator.

/* Frame interval for a specific frame size and format.
 * It's a leaf node and may store some user data for the format.
 */
struct camera_ext_frmival_node {
    uint32_t numerator;
    uint32_t denominator;
    const void *user_data;
};

Note the “user_data” field is a pointer to the data structure specific to the format/resolution/fps tuple in the format hierarchy.

Finally, the input hierarchy is maintained in a format_db structure which act as a root of the supported formats in the camera.

/* root node of the supported formats/resolutions */
struct camera_ext_format_db {
    const int num_inputs;
    struct camera_ext_input_node const *input_nodes;
};

Once format_db is defined in your driver, it must be registered through the function call below so mhb_camera framework knows the list of format to work with.

void camera_ext_register_format_db(const struct camera_ext_format_db *db);

Declare DEVICE_TYPE_MHB_CAMERA_HW

Your driver needs to define one instance of DEVICE_TYPE_MHB_CAMERA_HW. The driver must implement the following device operations so that the MHB Camera Framework can interact with your camera system.

struct device_mhb_camera_dev_type_ops {
    int (*soc_enable)(struct device *dev, uint8_t bootmode);
    int (*soc_disable)(struct device *dev);
    int (*stream_configure)(struct device *dev);
    int (*stream_enable)(struct device *dev);
    int (*stream_disable)(struct device *dev);
    int (*stream_reset)(struct device *dev);
    int (*lens_retract)(struct device *dev);
    int (*lens_extend)(struct device *dev);
    int (*power_limit)(struct device *dev, uint8_t enable);
    int (*get_csi_config)(struct device *dev, void *config);
    int (*get_fw_version)(struct device *dev, uint32_t *fw_ver);
    int (*set_err_callback)(struct device *dev, mhb_cam_cb callback);
};


int (*soc_enable)(struct device *dev, uint8_t bootmode);

This operation is invoked when the camera application opens up the Camera through Android Framework API. It is necessary the driver powers up all the components in the camera device so the subsequent “stream_enable” request can be handled immediately.

The “bootmode” parameter is the hint whether “stream_enable()” would follow after soc_enable() call in case the driver wants to speed up the time to enable its streaming pipeline.


int (*soc_disable)(struct device *dev);

This operation is invoked when the camera application closed the Camera through Android Framework API. The driver should power down the camera device and clean up all resources being used.


int (*stream_configure)(struct device *dev);

This operation is invoked when the Camera application on the Core started to capture images from the Mod (i.e. preview, video recording etc). This is the opportunity to peek in the format and resolution selected by the Core and setting up the camera device according to the selected configurations.


int (*stream_enable)(struct device *dev);

This operation is invoked when the Camera application on the Core started to capture images from the Mod (i.e. preview, video recording etc). This method is meant for the camera device actually start to generate the image traffic over CSI-2 bus.


int (*stream_disable)(struct device *dev);

This operation is invoked when the Camera application on the Core stopped capturing images from the Mod. The driver should stop the camera device to generate the traffic on CSI-2 bus.


int (*stream_reset)(struct device *dev);

This is the optional operation and it is invoked in case of image streaming needs to be re-enabled due to an issue detected in the MHB Camera Framework. It’s up to each camera device to decide whether anything needs to be restarted when stream is re-enabled.


int (*lens_retract)(struct device *dev);

This is the optional operation and only applicable if the camera has optical zoom lens module. It is invoked after stream_disable(), implying the safe timing to retract zoom lens.


int (*lens_extend)(struct device *dev);

This is the optional operation and only applicable if the camera has optical zoom lens module. It is invoked prior to stream_enable(), implying the best timing to extend zoom lens.


int (*power_limit)(struct device *dev, uint8_t enable);

This is the optional operation. It is invoked when the Core is experiencing low battery and requires the camera device not to perform any high current drain operations. The “enable” argument is set to 1 if the Core is in low battery, 0 if the Core is no longer in such state.


int (*get_csi_config)(struct device *dev, void *config);

This API is for the MHB Camera Framework to retrieve the CSI configuration for your camera system. The “config” argument points to the struct mhb_cdsi_config described later.


int (*get_fw_version)(struct device *dev, uint32_t *fw_ver);

This is the optional operation. The MHB Camera Framework may query your driver’s version string for the debugging purpose.


int (*set_err_callback)(struct device *dev, mhb_cam_cb callback);

This is the optional operation. The MHB Camera Framework will notify the callback to be invoked in case error being detected in your camera system. The callback is useful if an error condition needs to be notified asynchronously to the MHB Camera framework.


Starting Stream with CSI-2/MHB

When the Camera application started the preview or video recording on the Core, CSI-2 transport needs to be configured on both Mod and Core side so that image frames can be transmitted with agreed-on configuration. The MHB Camera Framework abstracts most of procedure for your driver. Your driver needs to follow the following 3 steps.

camera-ext-diagram-2.png

Step 1: Identify the selected format/resolution and frame interval

By the time the stream is started on the Mod, the Camera application must have been selected the format, resolution and frame rate to use for the session. The “stream_configure()” callback will be, then, invoked for your driver to setup your camera system. The following utility API in nuttx/camera/camera_ext.h can be used to identify the selected configuration.

struct camera_ext_format_user_config *camera_ext_get_user_config(void);

This API returns the last selected configuration by the Camera application. The returned value should be used in conjunction with the APIs below.


struct camera_ext_format_node const *get_current_format_node(
    struct camera_ext_format_db const *db,
    struct camera_ext_format_user_config const *cfg);

This API returns the selected Format node from the “cfg” returned from camera_ext_get_user_config().


struct camera_ext_frmsize_node const *get_current_frmsize_node(
    struct camera_ext_format_db const *db,
    struct camera_ext_format_user_config const *cfg);

This API returns the selected FrmSize node from the “cfg” returned from camera_ext_get_user_config().


struct camera_ext_frmival_node const *get_current_frmival_node(
    struct camera_ext_format_db const *db,
    struct camera_ext_format_user_config const *cfg);

This API returns the selected Frmival node from the “cfg” returned from camera_ext_get_user_config().

Step 2: Configure sensor module

Once the format/frame size/frame rate tuple is known, it’s time to configure your camera sensor according to the configuration. Typically this process involves I2C transactions with the sensor module, but it may vary depending on the camera system on the Mod. Note “camera_ext_frmival_node” has a user data pointer convenient to maintain (I2C) command set specific to each configuration.

After the sensor module is configured in stream_configure(), stream_enable() callback is the place to kick off your sensor module to generate the data over its CSI output.

Step 3: Return CDSI configuration to MHB Camera Framework

When MHB Camera Framework needs to configure CSI bus over MHB, get_csi_config() will be invoked. The following CSI configuration structure should be fulfilled and returned to the callback. Although the structure is large, many fields are auto-calculated or have valid defaults (as seen in the comments for each). The required fields for your specific sensor are:

tx_num_lanes
rx_bits_per_lane
tx_bits_per_lane
framerate
width
height
bpp

struct mhb_cdsi_config {
        uint8_t direction;               /* Set to 0 */
        uint8_t mode;                    /* Set to 1 */

        uint8_t rx_num_lanes;            /* Set to 4 */
        uint8_t tx_num_lanes;            /* 1 to 4 */
        uint32_t rx_bits_per_lane;       /* bits-per-lane */
        uint32_t tx_bits_per_lane;       /* bits-per-lane */

        uint32_t hs_rx_timeout;          /* Set to 0xffffffff */

        uint32_t pll_frs;                /* Set to 0 (auto calculate) */
        uint32_t pll_prd;                /* Set to 0 (auto calculate) */
        uint32_t pll_fbd;                /* Set to 0 (auto calculate) */

        uint32_t framerate;              /* frames-per-second */

        uint32_t width;                  /* pixels */
        uint32_t height;                 /* pixels */
        uint16_t physical_width;         /* Set to 0 (do not care) */
        uint16_t physical_height;        /* Set to 0 (do not care) */

        uint32_t bpp;                    /* bits-per-pixel */

        uint32_t vss_control_payload;    /* Set to 0 (do not care) */
        uint8_t bta_enabled;             /* Set to 0 (do not care) */
        uint8_t continuous_clock;        /* Set to 0 */
        uint8_t blank_packet_enabled;    /* Set to 0 */
        uint8_t video_mode;              /* Set to 0 (do not care) */
        uint8_t color_bar_enabled;       /* Set to 0 (do not care) */
        uint8_t keep_alive;              /* Set to 0 (do not care) */

        uint8_t t_clk_pre;               /* Set to 0 (use default) */
        uint8_t t_clk_post;              /* Set to 0 (use default) */

        uint8_t horizontal_front_porch;  /* Set to 0 (use default) */
        uint8_t horizontal_back_porch;   /* Set to 0 (use default) */
        uint8_t horizontal_pulse_width;  /* Set to 0 (use default) */
        uint8_t horizontal_sync_skew;    /* Set to 0 (use default) */
        uint8_t horizontal_left_border;  /* Set to 0 (use default) */
        uint8_t horizontal_right_border; /* Set to 0 (use default) */

        uint8_t vertical_front_porch;    /* Set to 0 (use default) */
        uint8_t vertical_back_porch;     /* Set to 0 (use default) */
        uint8_t vertical_pulse_width;    /* Set to 0 (use default) */
        uint8_t vertical_top_border;     /* Set to 0 (use default) */
        uint8_t vertical_bottom_border;  /* Set to 0 (use default) */

        uint8_t vsync_mode;              /* Set to 0 */
        uint8_t eot_mode;                /* Set to 0 */
        uint8_t traffic_mode;            /* Set to 0 */

        uint8_t reserved[6];
};

Implement Camera Controls

The Camera Controls framework is used by the Camera application on the core to 1) query the capability of the camera system and 2) control the behavior of the camera while a session is running. The Camera Control needs to be defined in your driver according to your camera system. And your Camera Controls needs to be registered to the Camera Controls framework.

Defining and registering Camera Controls

The list of Camera Controls for your driver should be defined using the structure below.

struct camera_ext_ctrl_db {
    uint32_t num_ctrls;
    const struct camera_ext_ctrl_cfg **ctrls;
};

The “struct camera_ext_ctrl_cfg” will be explained later. Once the list is declared, the “ctrl_db” must be registered to CAMERA_EXT protocol by the API below.

void camera_ext_register_control_db(struct camera_ext_ctrl_db *ctrl_db);

The registered control list will be read by the Core when your Mod is attached to the Core.

Camera Control details

The “struct camera_ext_ctrl_cfg” represents single control. The structure is defined in nuttx/include/nuttx/camera/camera_ext.h as following.

struct camera_ext_ctrl_cfg {
    uint32_t id;
    uint32_t flags;
    union {
        int64_t min;
        float min_f;
    };
    union {
        int64_t max;
        float max_f;
    };
    uint64_t step;
    uint64_t menu_skip_mask;
    uint32_t array_size;
    union {
        const uint32_t *dims;
        const int64_t *menu_int;
        const float *menu_float;
    };

    struct camera_ext_ctrl_val_cfg val_cfg;
    const camera_ext_ctrl_val_t def;
    int (*get_volatile_ctrl)(struct device *dev,
            const struct camera_ext_ctrl_cfg *self,
            camera_ext_ctrl_val_t *val);
    int (*set_ctrl)(struct device *dev,
            const struct camera_ext_ctrl_cfg *self,
            const camera_ext_ctrl_val_t *val);
    int (*try_ctrl)(struct device *dev,
            const struct camera_ext_ctrl_cfg *self,
            const camera_ext_ctrl_val_t *val);
};
"id"

Identifier for the specific control defined under the structure. Need to be set to one of CAM_EXT_CID values assigned in v4l2_camera_ext_ctrls.h.

The CID will dictates the type of the Control. It is predefined on both Core and Mod. The following control types are used currently.

Integer
Integer Menu
Float
Float Menu
Double
Boolean
String
"flags"

Indicates how data in this structure needs to be processed at the Core. It is the mask of following values.

#define CAMERA_EXT_CTRL_FLAG_NEED_MIN           0x00010000
#define CAMERA_EXT_CTRL_FLAG_NEED_MAX           0x00020000
#define CAMERA_EXT_CTRL_FLAG_NEED_STEP          0x00040000
#define CAMERA_EXT_CTRL_FLAG_NEED_DEF           0x00080000
#define CAMERA_EXT_CTRL_FLAG_NEED_DIMS          0x00100000
#define CAMERA_EXT_CTRL_FLAG_NEED_MENU_MASK     0x00200000
#define CAMERA_EXT_CTRL_FLAG_NEED_MENU_INT      0x00400000
#define CAMERA_EXT_CTRL_FLAG_NEED_MENU_FLOAT    0x00800000

Each mask value specify whether min, max, step, def, dims, menu_mask, menu_int and menu_float field must be read or not.

"min", "min_f", "max", "max_f"

Used for Integer/Float type with range. Specifies min and max value of the Control.

"step"

Used for integer type with range plus predefined steps.

“menu_skip_mask”

Used for Integer Menu. If Menu items are predefined, this mask can be used to nullify particular set of items not supported on your camera.

“array_size”

Used for Integer, Float and Double type. Indicate the Control has the array and its array size.

“dims”

Used for Integer, Float and Double type. If the Control value set has a specific dimension. (i.e. 2x2 etc.)

“menu_int, menu_float”

Used for Integer Menu or Integer Float type. Defines custom list of menu items for the Control. The Core then can select one of the item to use.

“val_cfg”

Used to hint size of memory space necessary to store all values defined in the Control.

struct camera_ext_ctrl_val_cfg {
    uint32_t nr_of_elem;
    uint32_t elem_type;
};

Elem_type (Integer, Float, Boolean, Double, String) would dictates size of single element. The memory size will be estimated by “size for elem_type” * nr_of_elem.

“def”

Specify the default value of the Control if needed. It is a union of different data types.

“get_volatile_ctrl”

Function pointer to support getting the value of the Control. Normally, the Control values are cached in the Core. Only specify this function if the cached value is not guaranteed to be up to date.

“set_ctrl”

Function pointer to support setting a value to the Control. The function would be good place to propagate the new configuration to your camera system.

“try_ctrl”

Function pointer to support validating the value for the Control. The function needs to check if the passed-in value can be supported or not.

Available Camera Controls

This link provides the list of Camera Control currently available for your driver. The file “nuttx/include/nuttx/camera/v4l2_camera_ext_ctrls.h” defines all the controls for the CAMERA_EXT class, but only subset of them are used at this moment due to HAL v1 limitation at the Core.


Runtime information feedback via Camera Metadata

While the session is running, it is often necessary to send the status of the camera to the Core to achieve a certain operations. Mods Camera Metadata is used for accommodating such state information. The data structure is defined in this link.

Mods Camera Metadata content can be sent over either Greybus or CSI. The following Camera Control must be defined in your driver to specify which metadata path would be in use.

/* integer */
#define CAM_EXT_CID_MOD_META_DATA_PATH

    CAM_EXT_CID_MOD_META_DATA_PATH_CSI = 1
    CAM_EXT_CID_MOD_META_DATA_PATH_GB = 2

The Camera Control below also needs to be defined to hint the size of metadata to the Moto Z.

/* integer. For GB path, it's max packet size. For CSI path, it's line num */
#define CAM_EXT_CID_MOD_META_DATA_SIZE

1. Sending Metadata over Greybus

The file “nuttx/include/nuttx/camera/camera_ext_meta.h” defines API for your driver to set and send the metadata. The below is the usage of key APIs.

int init_metadata_task(void);
int start_metadata_task(void);
void stop_metadata_task(void);

The Moto Z expects the metadata sent up periodically even if the content is empty. The above APIs is the helper function to init, start and stop periodic sending of metadata frames up over Greybus. Once your driver is probed, “init_meda_task()” must be called. “start_metadata_task()” should be invoked when stream is enabled. “stop_metadata_task()” should be called on stream disable.

void set_autofocus_metadata(cam_metadata_autofocus_request_result_e result,
        cam_metadata_autofocus_status_e status);
void set_zoomchange_metadata(cam_metadata_zoom_activity_e activity,
        uint8_t zoom_numerator, uint8_t zoom_denominator);
....

void send_metadata_oneshot(void);

When one of the metadata got ready, your driver needs to set information by calling the corresponding “set_xxxxx_metadata()” method. Once all the metadata is set, use “send_metadata_oneshot()” to trigger the content sent over.

2. Sending Metadata over CSI

As stated in the metadata specification, the content can be sent as part of CSI image frame as the extra lines after the image lines. How to embed metadata into each frame is up to your implementation.


Error Handling

If your driver encountered an error in context of device_mhb_camera_dev_type_ops callback, non-zero value should be returned. A corresponding error code is returned as CAMERA_EXT protocol response and the Camera application will be interrupted.

If your driver encountered an error in non-driver callback context, the “mbb_cam_cb” set through the callback below should be invoked.

int (*set_err_callback)(struct device *dev, mhb_cam_cb callback);

typedef void (*mhb_cam_cb)(int reason);

“0” should be set as a reason argument which indicates fatal error so that you can stop running Camera application on the Core.


Support of JPEG capture via USB

If your Mod is aimed for the fine photography, you would want to create images captured in JPEG in the Mod. This, however, requires additional setups in your Mod and the camera firmware. Camera Firmware supports the mode higher resolution picture can be sent over USB interface by using “USB Video Class (a.k.a. UVC)”. This section explain how UVC can be enabled with Camera Firmware.

1. USB-Ext addition to the manifest

USB-EXT (0xEC) protocol needs to be defined in your manifest. It should be under the same bundle as your CAMERA_EXT protocol.

; USB-EXT on CPort XX
[cport-descriptor XX]
bundle = YY
protocol = 0xec

2. Implement/Configure UVC stack

UVC needs to be implemented on your USB device. The UVC device needs to expose one Bulk-IN Endpoint for the captured image transfer. Please refer to USB Device Class Specifications for detailed implementation of the UVC.

Snapshot resolution used in JPEG file has to be configured in UVC implementation. The Moto Z will read out JPEG resolutions separately from Greybus interface.

3. Declare UVC snapshot capability in Camera Firmware

The firmware needs to declare the following Camera Control in its Control DB.

/* int[4] ready only: [has_uvc, vid, pid]  */
#define CAM_EXT_CID_MOD_CAPS_UVC_SNAPSHOT

int[4] should be set to 1, indicating JPEG is sent over UVC. int[1] and int[4] should be set to VID and PID of your USB configuration descriptor. This is necessary for the Moto Z to distinguish USB device to focus.

4. Control UVC device enumeration from Camera Firmware

Whenever your USB device is ready, notify that USB-EXT device is “attached” to the core. The detail is described in Explore > Firmware > USB-EXT. Note it is required UVC enumeration happens at attach time once so that the Moto Z can read out its supported format and resolutions.

5. Handling Capture Request from the Moto Z

Once a picture has to be taken, the Moto Z will be setting the following Camera Control to notify the Mod.

#define CAM_EXT_CID_CAPTURE

Once this Control is set, the Moto Z will be waiting for the image sent over UVC interface. Your firmware needs to interact UVC stack so that the created JPEG file is transferred over promptly.

For detail of Camera Controls for Capture and JPEG operations, please refer to Camera Controls page.