Warning
You are reading the documentation for an older Pickit release (2.3). Documentation for the latest release (3.4) can be found here.
Pickit socket interface
Pickit offers a low-level interface based on a TCP/IP socket communication which can be used to connect your robot (or PLC) with Pickit. Using the socket interface, the robot can for instance trigger detections, retrieve object poses and change the Pickit configuration.
The following article describes the technical details of the socket interface. This is relevant if you are planning to interface a new robot brand with Pickit.
Note
Before starting, make sure that the robot integration does not already exist and confirm with support@pickit3d.com that the integration is not under development.
Overview
The robot and Pickit exchange information in a request-response pattern, where the robot acts as the client and Pickit as the server. The robot sends a request, such as e.g. an object detection, to Pickit, which replies with a corresponding response. All requests are implemented in a purely synchronous fashion, and it is up to the client to ensure that no new requests are sent before having received a response for the previous request.
Next to this request-response exchange, Pickit also expects to receive periodic robot flange pose updates from the client. Note that the robot pose update is the only request for which the Pickit system does not reply with a matching response message.
Connection details
Type |
TCP/IP socket |
Port |
5001 (TCP) |
Byte order |
Network order (big endian) |
Once a Pickit system is started, it listens to TCP port 5001 and waits until the robot initiates
a connection. This is done on the robot side by opening a TCP socket targeting the IP address of the Pickit system
and the given port.
The IP address of the Pickit system can be found and changed in the Pickit network settings.
The default IP for the Ethernet port labeled ROBOT is 169.254.5.180.
Protocol
Request and response messages that are exchanged between a robot and Pickit have a fixed size. A fixed-size message protocol has the advantage that it is easy to implement on the robot side, even with limited programming features.
Note
Request and response messages have a fixed size and do not have a begin and/or end character. While the TCP/IP protocol prevents data loss, the robot client implementation is responsible for keeping track of the boundary between messages by counting the number of sent/received bytes and comparing with the expected message size.
Request and response messages consist of a number of fields, each represented by an int32 (4 bytes). Floating-point data, such
as distances and angles, are multiplied by a constant factor MULT = 10000 and truncated, before being sent as an int32. The receiving side
then decodes this field by dividing the received value by MULT again. Negative numbers are encoded using
two’s complement.
Request message
The request message that is sent from the robot to Pickit is 48 bytes long and consists of the following fields:
Field |
Type |
Length |
Description |
|---|---|---|---|
position |
int32[3] |
12 bytes |
Robot flange position (XYZ, in meters) expressed in the right-handed robot base frame. Each field has to be multiplied by the MULT factor. |
orientation |
int32[4] |
16 bytes |
Robot flange orientation expressed in the right-handed robot base frame. The orientation encoding and units depend on the chosen orientation convention. Each field has to be multiplied by the MULT factor. |
command |
int32 |
4 bytes |
One of the possible request commands. |
payload |
int32[2] |
8 bytes |
Optional payload fields. |
meta |
int32[2] |
8 bytes |
Orientation convention and protocol version. See the detailed meta field explanation. |
Except for the optional payload fields, all fields are mandatory, and have to be set to sane values for every request.
The payload fields are only required for certain commands, and are otherwise not used by Pickit.
The command field indicates which operation Pickit should execute. Possible commands and their
corresponding response messages are explained in more detail below.
Response message
Except for the pose-update request, all requests are answered with a 64 byte long response message
with the following structure:
Field |
Type |
Length |
Description |
|---|---|---|---|
position |
int32[3] |
12 bytes |
Object position or pick point offset translation (XYZ, in meters), depending on the response status. See also the detailed command explanation. Each value has to be divided by MULT. |
orientation |
int32[4] |
16 bytes |
Object orientation or pick point offset rotation, depending on the response status. See also the detailed command explanation. Encoding and units depend on the chosen orientation convention and have to be divided by MULT. |
payload |
int32[6] |
24 bytes |
Optional payload fields. |
status |
int32 |
4 bytes |
One of the defined status values. |
meta |
int32[2] |
8 bytes |
Orientation convention and protocol version. See the detailed meta field explanation. |
Not every command response conveys pose information or additional payload. The status field determines if (and how)
position, orientation and payload fields have to be interpreted. In the following sections, the individual
commands and their corresponding responses are explained in more detail.
Available commands
RC_PICKIT_NO_COMMAND (Robot pose update)
Send the current robot flange pose to Pickit. This information is used by Pickit to determine if the robot is still connected, as well as to update the 3D views in the Pickit web interface.
Robot pose updates should be sent periodically to Pickit, typically in the range of 10 messages per second. This command does not trigger a response, so it can be sent also while waiting for a response for a different command.
RC_PICKIT_CHECK_MODE
Check the current mode of Pickit. Pickit can be in the following modes: ROBOT, CALIBRATION or IDLE.
Response
Field |
Value |
Description |
|---|---|---|
status |
Pickit is in robot mode and able to send object poses to the robot. After booting, the Pickit system is in robot mode, which can be disabled and re-enabled via the web interface. |
|
Pickit is in calibration mode and able to localize the calibration plate. The system is in calibration mode while the calibration wizard in the web interface is open. |
||
When not in robot or calibration mode, Pickit is in idle mode, and can be configured via the web interface. |
RC_PICKIT_FIND_CALIB_PLATE
Trigger Pickit to localize the calibration plate. If sufficient calibration poses have been collected, Pickit will additionally trigger the computation of the robot-camera calibration. Note that Pickit has to be in calibration mode when this command is sent.
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully localized calibration plate. |
|
Failed to localize calibration plate. |
RC_PICKIT_LOOK_FOR_OBJECTS
Request Pickit to find objects in the current scene. This command performs an image capture and image processing in a single request.
Response
Due to the fixed-size structure of the response message, only one object can be transmitted at a time. If more than one object has been detected, the information of the first object is communicated in the response message of a detection request, and the remaining objects can be obtained using the RC_PICKIT_NEXT_OBJECT request, one at a time.
Field |
Value / Description |
|
|---|---|---|
position |
Object position (XYZ, in meters) expressed in the robot base frame. Each field has to be divided by MULT. |
|
orientation |
Object orientation expressed in the robot base frame. The orientation encoding and units depend on the chosen orientation convention. Each field has to be divided by MULT. |
|
payload[0] |
The duration (in seconds) elapsed between the capturing of the camera image and the moment the object information is sent to the robot. This value has to be divided by MULT. |
|
payload[1] |
|
|
payload[2] |
Object length (SQUARE, RECTANGLE, ELLIPSE, CYLINDER, POINTCLOUD, BLOB) or diameter (CIRCLE, SPHERE) in meters. Needs to be divided by MULT. |
|
payload[3] |
Object width (RECTANGLE, ELLIPSE, POINTCLOUD, BLOB) or diameter (CYLINDER) in meters. Needs to be divided by MULT. |
|
payload[4] |
Object height (POINTCLOUD, BLOB) in meters. Needs to be divided by MULT. |
|
payload[5] |
If this field is non-zero, it contains the number of remaining objects that can be retrieved via consecutive RC_PICKIT_NEXT_OBJECT requests. |
|
status |
At least one object has been detected. |
|
No objects have been detected, but the ROI is not empty. |
||
No camera image has been captured. |
||
An empty ROI has been detected. |
||
Attention
Object poses communicated by Pickit, via the position and orientation fields,
have their Z-axis pointing upwards. Depending on the orientation of your robot’s flange frame, it might be necessary
to flip the received poses by 180 degrees around the X-axis, such that the tool correctly aligns with the object at the moment of picking.
RC_PICKIT_LOOK_FOR_OBJECTS_WITH_RETRIES
Request Pickit to find objects in the current scene with retries. This command is similar to RC_PICKIT_LOOK_FOR_OBJECTS, but when no objects are found (but the Region of Interest (ROI) is not empty), Pickit will retry up to n times to find objects before giving up.
Request:
Field |
Value / Description |
|---|---|
payload[0] |
Maximum number of detection retries. |
Response
See response message of RC_PICKIT_LOOK_FOR_OBJECTS.
RC_PICKIT_NEXT_OBJECT
Request to return the next detected (valid and pickable) object in the detection grid. Use this command when a single object detection run yields multiple detected objects. Note that the RC_PICKIT_LOOK_FOR_OBJECTS (or RC_PICKIT_PROCESS_IMAGE) command already returns the first object, if at least one was detected.
Response
See response message of RC_PICKIT_LOOK_FOR_OBJECTS.
If RC_PICKIT_NEXT_OBJECT is called after the last detected object has been sent to the robot, Pickit replies with a PICKIT_NO_OBJECTS status.
RC_PICKIT_CAPTURE_IMAGE
Trigger Pickit to capture a camera image to be used by a following RC_PICKIT_PROCESS_IMAGE request. This command allows to wait until image capture is done and afterwards parallelize image processing with robot motion. This is especially relevant for robot-mounted cameras where the robot has to stand still during image capture.
However, in most cases, it is sufficient and more convenient to call RC_PICKIT_LOOK_FOR_OBJECTS, which performs image capture and processing in a single request.
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully captured camera image. |
|
Failed to capture camera image. |
RC_PICKIT_PROCESS_IMAGE
Trigger an object detection on the camera image that was previously captured via RC_PICKIT_CAPTURE_IMAGE (or RC_PICKIT_LOOK_FOR_OBJECTS).
Response
See response message of RC_PICKIT_LOOK_FOR_OBJECTS.
RC_PICKIT_CONFIGURE
Request Pickit to load a specific setup and product configuration. Each setup and product configuration have a unique ID assigned, which is shown in the web interface, next to the configuration name.
Request:
Field |
Value / Description |
|---|---|
payload[0] |
ID of the setup configuration. |
payload[1] |
ID of the product configuration. |
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully loaded the specified configurations. |
|
Failed to load the specified configurations. |
RC_PICKIT_SAVE_SNAPSHOT
Request Pickit to save a snapshot with the last captured scene and the current configuration.
Snapshots will be saved in the robot subfolder, which can be accessed from the web interface.
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully saved a snapshot. |
|
Failed to save a snapshot. |
RC_PICKIT_BUILD_BACKGROUND
Request Pickit to capture the current scene as background for background removal.
Response
Field |
Value |
Description |
|---|---|---|
status |
Successfully built background scene. |
|
Failed to built background scene. |
RC_PICKIT_GET_PICK_POINT_DATA
With multiple or flexible pick points, the robot needs to know how an object is being picked in order to drop it off at a fixed position. This information can be retrieved via RC_PICKIT_GET_PICK_POINT_DATA, which requests the pick point ID and pick point offset of the last requested object.
In the simplest case, the pick point ID is sufficient to know how an object was picked and how it should be dropped off. For applications with multiple pick points and/or pick points with flexible orientations, it is advised to make use of the pick point offset as well. Using the pick point offset, you don’t need to define a drop off point for every pick point. Instead, you only have to define a drop off point for every pick point reference, and apply the inverse pick point offset to this point.
Note that, due to the flipping of object poses, it is necessary to correct the communicated pick point offset on the robot side.
This corrected offset is computed by pick_offset_to_apply = Rx × inv(offset_from_pickit) × Rx, where Rx denotes a rotation of 180 degrees
around the X-axis. The pick point offset fpp_T_ppr is the transform between the final pick point and the pick point’s
reference.
Response
Field |
Value / Description |
|
|---|---|---|
position |
Pick point offset position (XYZ, in meters) expressed in the final pick point frame. Values have to be divided by MULT. |
|
orientation |
Pick point offset orientation expressed in the final pick point frame. The encoding and units depend on the chosen orientation convention. Values have to be divided by MULT. |
|
payload[0] |
ID of the selected pick point’s reference pick point. |
|
payload[1] |
ID of the pick point that was selected for the given object. |
|
status |
Successfully retrieved pick point data. |
|
Failed to retrieve pick point data. |
||
Constants
RC_PICKIT_NO_COMMAND = -1
RC_PICKIT_CHECK_MODE = 0
RC_PICKIT_FIND_CALIB_PLATE = 10
RC_PICKIT_LOOK_FOR_OBJECTS = 20
RC_PICKIT_LOOK_FOR_OBJECTS_WITH_RETRIES = 21
RC_PICKIT_CAPTURE_IMAGE = 22
RC_PICKIT_PROCESS_IMAGE = 23
RC_PICKIT_NEXT_OBJECT = 30
RC_PICKIT_CONFIGURE = 40
RC_PICKIT_SAVE_SCENE = 50
RC_PICKIT_BUILD_BACKGROUND = 60
RC_PICKIT_GET_PICK_POINT_DATA = 70
PICKIT_UNKNOWN_COMMAND = -99
PICKIT_ROBOT_MODE = 0
PICKIT_IDLE_MODE = 1
PICKIT_CALIBRATION_MODE = 2
PICKIT_FIND_CALIB_PLATE_OK = 10
PICKIT_FIND_CALIB_PLATE_FAILED = 11
PICKIT_OBJECT_FOUND = 20
PICKIT_NO_OBJECTS = 21
PICKIT_NO_IMAGE_CAPTURED = 22
PICKIT_EMPTY_ROI = 23
PICKIT_IMAGE_CAPTURED = 26
PICKIT_CONFIG_OK = 40
PICKIT_CONFIG_FAILED = 41
PICKIT_SAVE_SNAPSHOT_OK = 50
PICKIT_SAVE_SNAPSHOT_FAILED = 51
PICKIT_BUILD_BKG_CLOUD_OK = 60
PICKIT_BUILD_BKG_CLOUD_FAILED = 61
PICKIT_GET_PICK_POINT_DATA_OK = 70
PICKIT_GET_PICK_POINT_DATA_FAILED = 71
PICKIT_TYPE_SQUARE = 21
PICKIT_TYPE_RECTANGLE = 22
PICKIT_TYPE_CIRCLE = 23
PICKIT_TYPE_ELLIPSE = 24
PICKIT_TYPE_CYLINDER = 32
PICKIT_TYPE_SPHERE = 33
PICKIT_YTPE_POINTCLOUD = 35
PICKIT_TYPE_BLOB = 50
Message metadata
To guarantee correct interpretation of the data on both the robot and the Pickit side, the following metadata is always sent along with both request and response messages:
Field |
Value / Description |
|---|---|
orientation convention |
Convention that is being used to encode object or robot flange orientations. The following conventions are supported:
|
protocol version |
The version of the robot-Pickit communication. The current version number is |
If your robot does not adhere to any of the above orientation conventions, it is recommended to use the GENERIC (quaternions) convention. The robot-side interface would then take the responsibility of converting back and forth between quaternions and the representation used by the robot.
Communication flow
An example communication flow is as follows:
The robot checks the Pickit mode using RC_PICKIT_CHECK_MODE.
After confirming that Pickit is in robot mode, the robot initiates a background thread that periodically sends robot pose updates (RC_PICKIT_NO_COMMAND) to Pickit.
To configure Pickit for the given product and workspace, the robot loads the desired product and setup configurations via RC_PICKIT_CONFIGURE.
The robot requests an object detection with RC_PICKIT_LOOK_FOR_OBJECTS. For this example, Pickit responds that two objects were found, of which the first object is part of the response message.
The robot requests specific pick point data for the first object with RC_PICKIT_GET_PICK_POINT_DATA.
After the first object has been picked, the robot requests the second and final object with RC_PICKIT_NEXT_OBJECT. This is again followed by RC_PICKIT_GET_PICK_POINT_DATA, to retrieve the pick point data of the last requested object.
Example communication flow between robot and Pickit.