Pickit socket interface
Pickit offers a low-level interface based on a TCP/IP socket communication which can be used to connect your robot (or PLC) with Pickit. Using the socket interface, the robot can for instance trigger detections, retrieve object poses and change the Pickit configuration.
The following article describes the technical details of the socket interface. This is relevant if you are planning to interface a new robot brand with Pickit. It is structured as follows:
Note
Before starting, make sure that the robot integration does not already exist and confirm with support@pickit3d.com that the integration is not under development.
Overview
The robot and Pickit exchange information in a request-response pattern, where the robot acts as the client and Pickit as the server. The robot sends a request, such as e.g. an object detection, to Pickit, which replies with a corresponding response. All requests are implemented in a purely synchronous fashion, and it is up to the client to ensure that no new requests are sent before having received a response for the previous request.
Next to this request-response exchange, Pickit also expects to receive periodic robot flange pose updates from the client. Note that the robot pose update is the only request for which the Pickit system does not reply with a matching response message.
Connection details
Type |
TCP/IP socket |
Port |
5001 (TCP) |
Byte order |
Network order (big endian) |
Once a Pickit system is started, it listens to TCP port 5001 and waits until the robot initiates
a connection. This is done on the robot side by opening a TCP socket targeting the IP address of the Pickit system
and the given port.
The IP address of the Pickit system can be found and changed in the Pickit network settings.
The default IP for the Ethernet port labeled SWITCH or ROBOT is 169.254.5.180.
Protocol
Request and response messages that are exchanged between a robot and Pickit have a fixed size. A fixed-size message protocol has the advantage that it is easy to implement on the robot side, even with limited programming features.
Note
Request and response messages have a fixed size and do not have a begin and/or end character. While the TCP/IP protocol prevents data loss, the robot client implementation is responsible for keeping track of the boundary between messages by counting the number of sent/received bytes and comparing with the expected message size.
Request and response messages consist of a number of fields, each represented by an int32 (4 bytes). Floating-point data, such
as distances and angles, are multiplied by a constant factor MULT and truncated, before being sent as an int32. The receiving side
then decodes this field by dividing the received value by MULT again. This constant depends on the resolution of the chosen
protocol version: for protocol version 11 the constant is MULT=10000, resulting in a resolution of 0.1mm,
while for the protocol version 12 the value is MULT=1000000 resulting in a resolution of 0.001mm. Negative numbers are encoded using
two’s complement.
Request message
The request message that is sent from the robot to Pickit is 48 bytes long and consists of the following fields:
Field |
Type |
Length |
Description |
|---|---|---|---|
position |
int32[3] |
12 bytes |
Robot flange position (XYZ, in meters) expressed in the right-handed robot base frame. Each field has to be multiplied by the MULT factor. |
orientation |
int32[4] |
16 bytes |
Robot flange orientation expressed in the right-handed robot base frame. The orientation encoding and units depend on the chosen orientation convention. Each field has to be multiplied by the MULT factor. |
command |
int32 |
4 bytes |
One of the possible request commands. |
payload |
int32[2] |
8 bytes |
Optional payload fields. |
meta |
int32[2] |
8 bytes |
Orientation convention and protocol version. See the detailed meta field explanation. |
Except for the optional payload fields, all fields are mandatory, and have to be set to sane values for every request.
The payload fields are only required for certain commands, and are otherwise not used by Pickit.
The command field indicates which operation Pickit should execute. Possible commands and their
corresponding response messages are explained in more detail below.
Response message
Except for the pose-update request, all requests are answered with a 64 byte long response message
with the following structure:
Field |
Type |
Length |
Description |
|---|---|---|---|
position |
int32[3] |
12 bytes |
Object position or pick point offset translation (XYZ, in meters), depending on the response status. See also the detailed command explanation. Each value has to be divided by MULT. |
orientation |
int32[4] |
16 bytes |
Object orientation or pick point offset rotation, depending on the response status. See also the detailed command explanation. Encoding and units depend on the chosen orientation convention and have to be divided by MULT. |
payload |
int32[6] |
24 bytes |
Optional payload fields. |
status |
int32 |
4 bytes |
One of the defined status values. |
meta |
int32[2] |
8 bytes |
Orientation convention and protocol version. See the detailed meta field explanation. |
Not every command response conveys pose information or additional payload. The status field determines if (and how)
position, orientation and payload fields have to be interpreted. In the following sections, the individual
commands and their corresponding responses are explained in more detail.
Available commands
The available Pickit commands are introduced in this section, grouped by category.
System commands
RC_PICKIT_NO_COMMAND (Robot pose update)
Send the current robot flange pose to Pickit. This information is used by Pickit to determine if the robot is still connected, as well as to update the 3D views in the Pickit web interface.
Robot pose updates should be sent periodically to Pickit, typically in the range of 10 messages per second. This command does not trigger a response, so it can be sent also while waiting for a response for a different command.
RC_PICKIT_CHECK_MODE
Check the current mode of Pickit. Pickit can be in the following modes: ROBOT or IDLE.
Response
Field |
Value |
Description |
|---|---|---|
status |
Pickit is in robot mode and able to send object poses to the robot. After booting, the Pickit system is in robot mode, which can be disabled and re-enabled via the web interface. |
|
When not in robot mode, Pickit is in idle mode, and can be configured via the web interface. |
RC_PICKIT_SHUTDOWN_SYSTEM
Cleanly shut down the Pickit processor.
Response
Field |
Value |
Description |
|---|---|---|
status |
Shutdown request was accepted. |
|
Could not shutdown the processor. |
Detection commands
RC_PICKIT_CAPTURE_IMAGE
Capture a camera image to be used by a following RC_PICKIT_PROCESS_IMAGE request. This command waits until the image capture is finished. Immediately afterwards, the image processing can be done at the same time as the robot moves.
Note
This is the recommended command (followed RC_PICKIT_PROCESS_IMAGE) for robot-mounted cameras, where the robot has to stand still during image capture. For fixed cameras, prefer RC_PICKIT_LOOK_FOR_OBJECTS.
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully captured camera image. |
|
Failed to capture camera image. |
RC_PICKIT_PROCESS_IMAGE
Trigger an object detection on the camera image that was previously captured via RC_PICKIT_CAPTURE_IMAGE (or RC_PICKIT_LOOK_FOR_OBJECTS).
Response
Due to the fixed-size structure of the response message, only one object can be transmitted at a time. If more than one object has been detected, the information of the first object is communicated in the response message of a detection request, and the remaining objects can be obtained using the RC_PICKIT_NEXT_OBJECT request, one at a time.
Field |
Value / Description |
|
|---|---|---|
position |
Object position (XYZ, in meters) expressed in the robot base frame. Each field has to be divided by MULT. |
|
orientation |
Object orientation expressed in the robot base frame. The orientation encoding and units depend on the chosen orientation convention. Each field has to be divided by MULT. |
|
payload[0] |
The duration (in seconds) elapsed between the capturing of the camera image and the moment the object information is sent to the robot. This value has to be divided by MULT. |
|
payload[1] |
|
|
payload[2] |
Object length (SQUARE, RECTANGLE, ELLIPSE, CYLINDER, POINTCLOUD, BLOB) or diameter (CIRCLE, SPHERE) in meters. Needs to be divided by MULT. |
|
payload[3] |
Object width (RECTANGLE, ELLIPSE, POINTCLOUD, BLOB) or diameter (CYLINDER) in meters. Needs to be divided by MULT. |
|
payload[4] |
Object height (POINTCLOUD, BLOB) in meters. Needs to be divided by MULT. |
|
payload[5] |
If this field is non-zero, it contains the number of remaining objects that can be retrieved via consecutive RC_PICKIT_NEXT_OBJECT requests. |
|
status |
At least one object has been detected. |
|
No objects have been detected, but the ROI is not empty. |
||
No camera image has been captured. |
||
An empty ROI has been detected. |
||
A license module is missing to run the detection. |
||
Note
The object payloads (pose, size, …) should only be used if the status is PICKIT_OBJECT_FOUND. Otherwise, their value is undefined.
Attention
Object poses communicated by Pickit, via the position and orientation fields, have their Z-axis pointing upwards.
Depending on the orientation of your robot’s flange frame, it might be necessary
to flip the received poses by 180 degrees around the X-axis, such that the tool correctly aligns with the object at the moment of picking.
RC_PICKIT_LOOK_FOR_OBJECTS
Request Pickit to find objects in the current scene. This command performs an image capture and image processing in a single request.
Note
This is the recommended command for fixed cameras, where the robot does not have to stand still during image capture. For robot-mounted cameras, you should prefer RC_PICKIT_CAPTURE_IMAGE followed by RC_PICKIT_PROCESS_IMAGE.
Response
See response message of RC_PICKIT_PROCESS_IMAGE.
RC_PICKIT_NEXT_OBJECT
Request to return the next detected (valid and pickable) object in the objects table. Use this command when a single object detection run yields multiple detected objects. Note that the RC_PICKIT_LOOK_FOR_OBJECTS (or RC_PICKIT_PROCESS_IMAGE) command already returns the first object, if at least one was detected.
Response
See response message of RC_PICKIT_PROCESS_IMAGE.
If RC_PICKIT_NEXT_OBJECT is called after the last detected object has been sent to the robot, Pickit replies with a PICKIT_NO_OBJECTS status.
RC_PICKIT_LOOK_FOR_OBJECTS_WITH_RETRIES
Request Pickit to find objects in the current scene with retries. This command is similar to RC_PICKIT_LOOK_FOR_OBJECTS, but when no objects are found (but the Region of Interest (ROI) is not empty), Pickit will retry up to n times to find objects before giving up.
Request
Field |
Description |
|---|---|
payload[0] |
Maximum number of detection retries. |
Response
See response message of RC_PICKIT_PROCESS_IMAGE.
RC_PICKIT_GET_PICK_POINT_DATA
With multiple or flexible pick points, the robot needs to know how an object is being picked in order to drop it off at a fixed position. This information can be retrieved via RC_PICKIT_GET_PICK_POINT_DATA, which requests the pick point ID and pick point offset of the last requested object.
In the simplest case, the pick point ID is sufficient to know how an object was picked and how it should be dropped off. For applications with multiple pick points and/or pick points with flexible orientations, it is advised to make use of the pick point offset as well. Using the pick point offset, you don’t need to define a drop off point for every pick point. Instead, you only have to define a drop off point for every pick point reference, and apply the inverse pick point offset to this point.
Note that, due to the flipping of object poses, it is necessary to correct the communicated pick point offset on the robot side.
This corrected offset is computed by pick_offset_to_apply = Rx × inv(offset_from_pickit) × Rx, where Rx denotes a rotation of 180 degrees
around the X-axis. The pick point offset fpp_T_ppr is the transform between the final pick point and the pick point’s
reference.
Response
Field |
Value / Description |
|
|---|---|---|
position |
Pick point offset position (XYZ, in meters) expressed in the final pick point frame. Values have to be divided by MULT. |
|
orientation |
Pick point offset orientation expressed in the final pick point frame. The encoding and units depend on the chosen orientation convention. Values have to be divided by MULT. |
|
payload[0] |
ID of the selected pick point’s reference pick point. |
|
payload[1] |
ID of the pick point that was selected for the given object. |
|
status |
Successfully retrieved pick point data. |
|
Failed to retrieve pick point data. |
||
Configuration commands
These commands are used to change the active product and setup configuration.
RC_PICKIT_CONFIGURE
Request Pickit to load a specific setup and product configuration. Each setup and product configuration have a unique ID assigned, which is shown in the web interface, next to the configuration name.
Request
Field |
Description |
|---|---|
payload[0] |
ID of the setup configuration. |
payload[1] |
ID of the product configuration. |
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully loaded the specified configurations. |
|
Failed to load the specified configurations. |
RC_PICKIT_SET_CYLINDER_DIM
Request Pickit to set the cylinder dimensions when using the Teach cylinder model. For the command to succeed, there can be only one Teach model, it must be of type cylinder, and it must be enabled. Notice that, after calling this command with different dimensions than the ones of the current cylinder model, the product file will have unsaved changes. If you wish you can save them by calling the command RC_SAVE_ACTIVE_PRODUCT.
Request
Field |
Description |
|---|---|
payload[0] |
Length of the cylinder (in meters). The value has to be multiplied by MULT. |
payload[1] |
Diameter of the cylinder (in meters). The value has to be multiplied by MULT. |
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully set the cylinder dimensions. |
|
Failed to set the cylinder dimensions. |
RC_SAVE_ACTIVE_SETUP
Request Pickit to save the currently loaded setup.
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully saved the active setup. |
|
Failed to save the active setup. |
RC_SAVE_ACTIVE_PRODUCT
Request Pickit to save the currently loaded product.
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully saved the active product. |
|
Failed to save the active product. |
RC_PICKIT_BUILD_BACKGROUND
Request Pickit to capture the current scene as background for background removal.
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully built background scene. |
|
Failed to built background scene. |
Calibration commands
These commands are used to perform the robot-camera calibration.
RC_PICKIT_CONFIGURE_CALIB
Set the robot-camera calibration configuration.
Note
This command is intended for robot programs that do the entire calibration process from the robot program, and don’t use the Pickit web interface. The typical usage consists of the following sequence of commands:
Call RC_PICKIT_FIND_CALIB_PLATE zero or more times:
Zero times for
METHOD_MANUALOne time for
METHOD_SINGLE_POSE(learn more)Multiple times for
METHOD_MULTI_POSE(learn more)For
METHOD_DEFAULT, either one or multiple times, depending of what has been configured in the web interface.
Call RC_PICKIT_COMPUTE_CALIB (but not for
METHOD_MANUAL)
Request
Field |
Value / Description |
|---|---|
payload[0] |
Calibration method. Possible values are: METHOD_SINGLE_POSE = 0
METHOD_MULTI_POSE = 1
METHOD_MANUAL = 2
METHOD_DEFAULT = 3
|
payload[1] |
Camera mount. Possible values are: CAMERA_MOUNT_FIXED = 0
CAMERA_MOUNT_ON_ROBOT = 1
For calibration method |
position |
Position (XYZ, in meters). Each field has to be multiplied by MULT. The value has a different meaning depending on the selected calibration method:
|
orientation |
The value has a different meaning depending on the selected calibration method. See above for details. Its encoding and units depend on the chosen orientation convention. Each field has to be multiplied by MULT. |
Note
The combination of METHOD_SINGLE_POSE and CAMERA_MOUNT_ROBOT is not allowed.
Note
If you have a 4 DOF robot, you should perform an initial calibration using the calibration wizard of the Pickit web interface.
There you can set the robot type to be 4 DOF, as well as the required distance offset.
From this point on, calling the RC_PICKIT_CONFIGURE_CALIB command will work for a 4 DOF robot.
Note
The calibration method METHOD_DEFAULT requests to use the parameters specified in the user-interface. If you select this method,
any other parameter specified in the socket message of the calibration configuration will be overruled by the user-interface settings.
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully configured robot-camera calibration. |
|
Failed to configure robot-camera calibration. |
RC_PICKIT_FIND_CALIB_PLATE
Trigger Pickit to localize the calibration plate.
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully localized calibration plate. |
|
Failed to localize calibration plate. |
RC_PICKIT_COMPUTE_CALIB
Trigger the computation of the robot-camera calibration.
Note
This command is intended for robot programs that do the entire calibration process from the robot program, and don’t use the Pickit web interface. See the documentation of the RC_PICKIT_CONFIGURE_CALIB command to learn more.
Response
The robot-camera calibration transform robot_T_camera (camera in robot base coordinates) is such that robot corresponds to the robot base (for camera fixed), or the robot flange (for camera on robot); and camera is the camera’s optical frame.
The error frame used in the response corresponds to the maximum difference observed when using the computed calibration to close a kinematic loop at the detected calibration plate locations.
Field |
Value / Description |
|
|---|---|---|
position |
|
|
orientation |
|
|
payload[0] |
|
|
payload[1] |
|
|
status |
Successfully computed the robot-camera calibration. |
|
Failed to compute the robot-camera calibration. |
||
RC_PICKIT_VALIDATE_CALIB
Validate the current robot-camera calibration.
Calibration validation requires the placement of the calibration plate as described in the validation procedure.
The error frame used in the response corresponds to the difference observed when using the current calibration to close a kinematic loop at the detected calibration plate location.
Response
Field |
Value / Description |
|
|---|---|---|
position |
Error frame vector (XYZ, in meters), in robot base frame coordinates. Each field has to be divided by MULT. |
|
orientation |
Error frame orientation, in robot base frame coordinates. The orientation encoding and units depend on the chosen orientation convention. Each field has to be divided by MULT. |
|
payload[0] |
|
|
payload[1] |
|
|
status |
The validate command is successful. |
|
The validate command failed. |
||
Monitoring commands
This command is used to save data about a running application. This is specially useful when ramping up in production, as the stored data allows detecting eventual issues in the application.
RC_PICKIT_SAVE_SNAPSHOT
Request Pickit to save a snapshot with the last captured scene and the current configuration.
Snapshots will be saved in the robot subfolder, which can be accessed from the web interface.
If you wish to distinguish snapshots of different situations (e.g. mispicks and no detected objects), it is possible to optionally specify a subfolder inside the robot folder.
Request
Field |
Description |
|---|---|
payload[0] |
Subfolder (inside |
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully saved a snapshot. |
|
Failed to save a snapshot. |
Constants
RC_PICKIT_NO_COMMAND = -1
RC_PICKIT_CHECK_MODE = 0
RC_PICKIT_SHUTDOWN_SYSTEM = 2
RC_PICKIT_FIND_CALIB_PLATE = 10
RC_PICKIT_CONFIGURE_CALIB = 11
RC_PICKIT_COMPUTE_CALIB = 12
RC_PICKIT_VALIDATE_CALIB = 13
RC_PICKIT_LOOK_FOR_OBJECTS = 20
RC_PICKIT_LOOK_FOR_OBJECTS_WITH_RETRIES = 21
RC_PICKIT_CAPTURE_IMAGE = 22
RC_PICKIT_PROCESS_IMAGE = 23
RC_PICKIT_NEXT_OBJECT = 30
RC_PICKIT_CONFIGURE = 40
RC_PICKIT_SET_CYLINDER_DIM = 41
RC_SAVE_ACTIVE_SETUP = 42
RC_SAVE_ACTIVE_PRODUCT = 43
RC_PICKIT_SAVE_SCENE = 50
RC_PICKIT_BUILD_BACKGROUND = 60
RC_PICKIT_GET_PICK_POINT_DATA = 70
PICKIT_UNKNOWN_COMMAND = -99
PICKIT_ROBOT_MODE = 0
PICKIT_IDLE_MODE = 1
PICKIT_SHUTDOWN_REQUEST_ACCEPTED = 5
PICKIT_SHUTDOWN_REQUEST_REJECTED = 6
PICKIT_FIND_CALIB_PLATE_OK = 10
PICKIT_FIND_CALIB_PLATE_FAILED = 11
PICKIT_CONFIGURE_CALIB_OK = 12
PICKIT_CONFIGURE_CALIB_FAILED = 13
PICKIT_COMPUTE_CALIB_OK = 14
PICKIT_COMPUTE_CALIB_FAILED = 15
PICKIT_VALIDATE_CALIB_OK = 16
PICKIT_VALIDATE_CALIB_FAILED = 17
PICKIT_OBJECT_FOUND = 20
PICKIT_NO_OBJECTS = 21
PICKIT_NO_IMAGE_CAPTURED = 22
PICKIT_EMPTY_ROI = 23
PICKIT_IMAGE_CAPTURED = 26
PICKIT_INVALID_LICENSE = 27
PICKIT_CONFIG_OK = 40
PICKIT_CONFIG_FAILED = 41
PICKIT_SAVE_SNAPSHOT_OK = 50
PICKIT_SAVE_SNAPSHOT_FAILED = 51
PICKIT_BUILD_BKG_CLOUD_OK = 60
PICKIT_BUILD_BKG_CLOUD_FAILED = 61
PICKIT_GET_PICK_POINT_DATA_OK = 70
PICKIT_GET_PICK_POINT_DATA_FAILED = 71
PICKIT_TYPE_SQUARE = 21
PICKIT_TYPE_RECTANGLE = 22
PICKIT_TYPE_CIRCLE = 23
PICKIT_TYPE_ELLIPSE = 24
PICKIT_TYPE_CYLINDER = 32
PICKIT_TYPE_SPHERE = 33
PICKIT_YTPE_POINTCLOUD = 35
PICKIT_TYPE_BLOB = 50
Message metadata
To guarantee correct interpretation of the data on both the robot and the Pickit side, the following metadata is always sent along with both request and response messages:
Field |
Value / Description |
|---|---|
orientation convention |
Convention that is being used to encode object or robot flange orientations. The following conventions are supported:
|
protocol version |
The version of the robot-Pickit communication:
|
If your robot does not adhere to any of the above orientation conventions, it is recommended to use the GENERIC (quaternions) convention. The robot-side interface would then take the responsibility of converting back and forth between quaternions and the representation used by the robot.
Communication flow
An example communication flow is as follows:
The robot checks the Pickit mode using RC_PICKIT_CHECK_MODE.
After confirming that Pickit is in robot mode, the robot initiates a background thread that periodically sends robot pose updates (RC_PICKIT_NO_COMMAND) to Pickit.
To configure Pickit for the given product and workspace, the robot loads the desired product and setup configurations via RC_PICKIT_CONFIGURE.
The robot requests an object detection with RC_PICKIT_LOOK_FOR_OBJECTS. For this example, Pickit responds that two objects were found, of which the first object is part of the response message.
The robot requests specific pick point data for the first object with RC_PICKIT_GET_PICK_POINT_DATA.
After the first object has been picked, the robot requests the second and final object with RC_PICKIT_NEXT_OBJECT. This is again followed by RC_PICKIT_GET_PICK_POINT_DATA, to retrieve the pick point data of the last requested object.
Example communication flow between robot and Pickit.