Warning

You are reading the documentation for an older Pickit release (2.2). Documentation for the latest release (3.4) can be found here.

Pickit socket interface

Pickit offers a low-level interface based on a TCP/IP socket communication which can be used to connect your robot (or PLC) with Pickit. Using the socket interface, the robot can for instance trigger detections, retrieve object poses and change the Pickit configuration.

The following article describes the technical details of the socket interface. This is relevant if you are planning to interface a new robot brand with Pickit.

Note

Before starting, make sure that the robot integration does not already exist and confirm with support@pickit3d.com that the integration is not under development.

Overview

The robot and Pickit exchange information in a request-response pattern, where the robot acts as the client and Pickit as the server. The robot sends a request, such as e.g. an object detection, to Pickit, which replies with a corresponding response. All requests are implemented in a purely synchronous fashion, and it is up to the client to ensure that no new requests are sent before having received a response for the previous request.

Next to this request-response exchange, Pickit also expects to receive periodic robot flange pose updates from the client. Note that the robot pose update is the only request for which the Pickit system does not reply with a matching response message.

Connection details

Type	TCP/IP socket
Port	5001 (TCP)
Byte order	Network order (big endian)

Once a Pickit system is started, it listens to TCP port 5001 and waits until the robot initiates a connection. This is done on the robot side by opening a TCP socket targeting the IP address of the Pickit system and the given port.

The IP address of the Pickit system can be found and changed in the Pickit network settings. The default IP for the Ethernet port labeled ROBOT is 169.254.5.180.

Protocol

Request and response messages that are exchanged between a robot and Pickit have a fixed size. A fixed-size message protocol has the advantage that it is easy to implement on the robot side, even with limited programming features.

Note

Request and response messages have a fixed size and do not have a begin and/or end character. While the TCP/IP protocol prevents data loss, the robot client implementation is responsible for keeping track of the boundary between messages by counting the number of sent/received bytes and comparing with the expected message size.

Request and response messages consist of a number of fields, each represented by an int32 (4 bytes). Floating-point data, such as distances and angles, are multiplied by a constant factor MULT = 10000 and truncated, before being sent as an int32. The receiving side then decodes this field by dividing the received value by MULT again. Negative numbers are encoded using two’s complement.

Request message

The request message that is sent from the robot to Pickit is 48 bytes long and consists of the following fields:

Request message structure
Field	Type	Length	Description
position	int32[3]	12 bytes	Robot flange position (XYZ, in meters) expressed in the right-handed robot base frame. Each field has to be multiplied by the MULT factor.
orientation	int32[4]	16 bytes	Robot flange orientation expressed in the right-handed robot base frame. The orientation encoding and units depend on the chosen orientation convention. Each field has to be multiplied by the MULT factor.
command	int32	4 bytes	One of the possible request commands.
payload	int32[2]	8 bytes	Optional payload fields.
meta	int32[2]	8 bytes	Orientation convention and protocol version. See the detailed meta field explanation.

Except for the optional payload fields, all fields are mandatory, and have to be set to sane values for every request. The payload fields are only required for certain commands, and are otherwise not used by Pickit.

The command field indicates which operation Pickit should execute. Possible commands and their corresponding response messages are explained in more detail below.

Response message

Except for the pose-update request, all requests are answered with a 64 byte long response message with the following structure:

Response message structure
Field	Type	Length	Description
position	int32[3]	12 bytes	Object position or pick point offset translation (XYZ, in meters), depending on the response status. See also the detailed command explanation. Each value has to be divided by MULT.
orientation	int32[4]	16 bytes	Object orientation or pick point offset rotation, depending on the response status. See also the detailed command explanation. Encoding and units depend on the chosen orientation convention and have to be divided by MULT.
payload	int32[6]	24 bytes	Optional payload fields.
status	int32	4 bytes	One of the defined status values.
meta	int32[2]	8 bytes	Orientation convention and protocol version. See the detailed meta field explanation.

Not every command response conveys pose information or additional payload. The status field determines if (and how) position, orientation and payload fields have to be interpreted. In the following sections, the individual commands and their corresponding responses are explained in more detail.

Available commands

RC_PICKIT_NO_COMMAND (Robot pose update)

Send the current robot flange pose to Pickit. This information is used by Pickit to determine if the robot is still connected, as well as to update the 3D views in the Pickit web interface.

Robot pose updates should be sent periodically to Pickit, typically in the range of 10 messages per second. This command does not trigger a response, so it can be sent also while waiting for a response for a different command.

RC_PICKIT_CHECK_MODE

Check the current mode of Pickit. Pickit can be in the following modes: ROBOT, CALIBRATION or IDLE.

Response

Field	Value	Description
status	PICKIT_ROBOT_MODE	Pickit is in robot mode and able to send object poses to the robot. After booting, the Pickit system is in robot mode, which can be disabled and re-enabled via the web interface.
	PICKIT_CALIBRATION_MODE	Pickit is in calibration mode and able to localize the calibration plate. The system is in calibration mode while the calibration wizard in the web interface is open.
	PICKIT_IDLE_MODE	When not in robot or calibration mode, Pickit is in idle mode, and can be configured via the web interface.

RC_PICKIT_FIND_CALIB_PLATE

Trigger Pickit to localize the calibration plate. If sufficient calibration poses have been collected, Pickit will additionally trigger the computation of the robot-camera calibration. Note that Pickit has to be in calibration mode when this command is sent.

Response

Field	Value	Description
status	PICKIT_FIND_CALIB_PLATE_OK	Successfully localized calibration plate.
status	PICKIT_FIND_CALIB_PLATE_FAILED	Failed to localize calibration plate.

RC_PICKIT_LOOK_FOR_OBJECTS

Request Pickit to find objects in the current scene. This command performs an image capture and image processing in a single request.

Response

Due to the fixed-size structure of the response message, only one object can be transmitted at a time. If more than one object has been detected, the information of the first object is communicated in the response message of a detection request, and the remaining objects can be obtained using the RC_PICKIT_NEXT_OBJECT request, one at a time.

Field	Value / Description
position	Object position (XYZ, in meters) expressed in the robot base frame. Each field has to be divided by MULT.
orientation	Object orientation expressed in the robot base frame. The orientation encoding and units depend on the chosen orientation convention. Each field has to be divided by MULT.
payload[0]	The duration (in seconds) elapsed between the capturing of the camera image and the moment the object information is sent to the robot. This value has to be divided by MULT.
payload[1]	For a Teach detection, this field contains the model ID of the current object. For a Flex/Pattern detection, this field contains the object type.
payload[2]	Object length (SQUARE, RECTANGLE, ELLIPSE, CYLINDER, POINTCLOUD, BLOB) or diameter (CIRCLE, SPHERE) in meters. Needs to be divided by MULT.
payload[3]	Object width (RECTANGLE, ELLIPSE, POINTCLOUD, BLOB) or diameter (CYLINDER) in meters. Needs to be divided by MULT.
payload[4]	Object height (POINTCLOUD, BLOB) in meters. Needs to be divided by MULT.
payload[5]	If this field is non-zero, it contains the number of remaining objects that can be retrieved via consecutive RC_PICKIT_NEXT_OBJECT requests.
status	PICKIT_OBJECT_FOUND	At least one object has been detected.
	PICKIT_NO_OBJECTS	No objects have been detected, but the ROI is not empty.
	PICKIT_NO_IMAGE_CAPTURED	No camera image has been captured.
	PICKIT_EMPTY_ROI	An empty ROI has been detected.

Attention

Object poses communicated by Pickit, via the position and orientation fields, have their Z-axis pointing upwards. Depending on the orientation of your robot’s flange frame, it might be necessary to flip the received poses by 180 degrees around the X-axis, such that the tool correctly aligns with the object at the moment of picking.

RC_PICKIT_LOOK_FOR_OBJECTS_WITH_RETRIES

Request Pickit to find objects in the current scene with retries. This command is similar to RC_PICKIT_LOOK_FOR_OBJECTS, but when no objects are found (but the Region of Interest (ROI) is not empty), Pickit will retry up to n times to find objects before giving up.

Request:

Field	Value / Description
payload[0]	Maximum number of detection retries.

Response

See response message of RC_PICKIT_LOOK_FOR_OBJECTS.

RC_PICKIT_NEXT_OBJECT

Request to return the next detected (valid and pickable) object in the detection grid. Use this command when a single object detection run yields multiple detected objects. Note that the RC_PICKIT_LOOK_FOR_OBJECTS (or RC_PICKIT_PROCESS_IMAGE) command already returns the first object, if at least one was detected.

Response

See response message of RC_PICKIT_LOOK_FOR_OBJECTS.

If RC_PICKIT_NEXT_OBJECT is called after the last detected object has been sent to the robot, Pickit replies with a PICKIT_NO_OBJECTS status.

RC_PICKIT_CAPTURE_IMAGE

Trigger Pickit to capture a camera image to be used by a following RC_PICKIT_PROCESS_IMAGE request. This command allows to wait until image capture is done and afterwards parallelize image processing with robot motion. This is especially relevant for robot-mounted cameras where the robot has to stand still during image capture.

However, in most cases, it is sufficient and more convenient to call RC_PICKIT_LOOK_FOR_OBJECTS, which performs image capture and processing in a single request.

Response

Field	Value	Description
status	PICKIT_IMAGE_CAPTURED	Successfully captured camera image.
status	PICKIT_NO_IMAGE_CAPTURED	Failed to capture camera image.

RC_PICKIT_PROCESS_IMAGE

Trigger an object detection on the camera image that was previously captured via RC_PICKIT_CAPTURE_IMAGE (or RC_PICKIT_LOOK_FOR_OBJECTS).

Response

See response message of RC_PICKIT_LOOK_FOR_OBJECTS.

RC_PICKIT_CONFIGURE

Request Pickit to load a specific setup and product configuration. Each setup and product configuration have a unique ID assigned, which is shown in the web interface, next to the configuration name.

Request:

Field	Value / Description
payload[0]	ID of the setup configuration.
payload[1]	ID of the product configuration.

Response

Field	Value	Description
status	PICKIT_CONFIG_OK	Successfully loaded the specified configurations.
status	PICKIT_CONFIG_FAILED	Failed to load the specified configurations.

RC_PICKIT_SAVE_SNAPSHOT

Request Pickit to save a snapshot with the last captured scene and the current configuration. Snapshots will be saved in the robot subfolder, which can be accessed from the web interface.

Response

Field	Value	Description
status	PICKIT_SAVE_SNAPSHOT_OK	Successfully saved a snapshot.
status	PICKIT_SAVE_SNAPSHOT_FAILED	Failed to save a snapshot.

RC_PICKIT_BUILD_BACKGROUND

Request Pickit to capture the current scene as background for background removal.

Response

Field	Value	Description
status	PICKIT_BUILD_BKG_CLOUD_OK	Successfully built background scene.
status	PICKIT_BUILD_BKG_CLOUD_FAILED	Failed to built background scene.

RC_PICKIT_GET_PICK_POINT_DATA

With multiple or flexible pick points, the robot needs to know how an object is being picked in order to drop it off at a fixed position. This information can be retrieved via RC_PICKIT_GET_PICK_POINT_DATA, which requests the pick point ID and pick point offset of the last requested object.

In the simplest case, the pick point ID is sufficient to know how an object was picked and how it should be dropped off. For applications with multiple pick points and/or pick points with flexible orientations, it is advised to make use of the pick point offset as well. Using the pick point offset, you don’t need to define a drop off point for every pick point. Instead, you only have to define a drop off point for every pick point reference, and apply the inverse pick point offset to this point.

Note that, due to the flipping of object poses, it is necessary to correct the communicated pick point offset on the robot side. This corrected offset is computed by pick_offset_to_apply = Rx × inv(offset_from_pickit) × Rx, where Rx denotes a rotation of 180 degrees around the X-axis. The pick point offset fpp_T_ppr is the transform between the final pick point and the pick point’s reference.

Response

Field	Value / Description
position	Pick point offset position (XYZ, in meters) expressed in the final pick point frame. Values have to be divided by MULT.
orientation	Pick point offset orientation expressed in the final pick point frame. The encoding and units depend on the chosen orientation convention. Values have to be divided by MULT.
payload[0]	ID of the selected pick point’s reference pick point.
payload[1]	ID of the pick point that was selected for the given object.
status	PICKIT_GET_PICK_POINT_DATA_OK	Successfully retrieved pick point data.
status	PICKIT_GET_PICK_POINT_DATA_FAILED	Failed to retrieve pick point data.

Constants

Request command constants

RC_PICKIT_NO_COMMAND              = -1
RC_PICKIT_CHECK_MODE              = 0
RC_PICKIT_FIND_CALIB_PLATE        = 10
RC_PICKIT_LOOK_FOR_OBJECTS        = 20
RC_PICKIT_CAPTURE_IMAGE           = 22
RC_PICKIT_PROCESS_IMAGE           = 23
RC_PICKIT_NEXT_OBJECT             = 30
RC_PICKIT_CONFIGURE               = 40
RC_PICKIT_SAVE_SCENE              = 50
RC_PICKIT_BUILD_BACKGROUND        = 60
RC_PICKIT_GET_PICK_POINT_DATA     = 70

Response status constants

PICKIT_UNKNOWN_COMMAND            = -99
PICKIT_ROBOT_MODE                 =   0
PICKIT_IDLE_MODE                  =   1
PICKIT_CALIBRATION_MODE           =   2
PICKIT_FIND_CALIB_PLATE_OK        =  10
PICKIT_FIND_CALIB_PLATE_FAILED    =  11
PICKIT_OBJECT_FOUND               =  20
PICKIT_NO_OBJECTS                 =  21
PICKIT_NO_IMAGE_CAPTURED          =  22
PICKIT_EMPTY_ROI                  =  23
PICKIT_IMAGE_CAPTURED             =  26
PICKIT_CONFIG_OK                  =  40
PICKIT_CONFIG_FAILED              =  41
PICKIT_SAVE_SNAPSHOT_OK           =  50
PICKIT_SAVE_SNAPSHOT_FAILED       =  51
PICKIT_BUILD_BKG_CLOUD_OK         =  60
PICKIT_BUILD_BKG_CLOUD_FAILED     =  61
PICKIT_GET_PICK_POINT_DATA_OK     =  70
PICKIT_GET_PICK_POINT_DATA_FAILED =  71

Object type constants

PICKIT_TYPE_SQUARE                =  21
PICKIT_TYPE_RECTANGLE             =  22
PICKIT_TYPE_CIRCLE                =  23
PICKIT_TYPE_ELLIPSE               =  24
PICKIT_TYPE_CYLINDER              =  32
PICKIT_TYPE_SPHERE                =  33
PICKIT_YTPE_POINTCLOUD            =  35
PICKIT_TYPE_BLOB                  =  50

Message metadata

To guarantee correct interpretation of the data on both the robot and the Pickit side, the following metadata is always sent along with both request and response messages:

Metadata message
Field	Value / Description
orientation convention	Convention that is being used to encode object or robot flange orientations. The following conventions are supported: Angle-axis (3D vector consisting of the unit axis multiplied by the angle in radians) → UNIVERSAL ROBOTS Quaternions (w,x,y,z) → GENERIC, ABB Euler Angles (x-y’-z”, in degrees) → STÄUBLI Fixed Angles (x-y-z, in degrees) → FANUC, NACHI, OMRON TM, YASKAWA Euler Angles (z-y’-x”, in degrees) → HANWHA, KUKA Euler Angles (z-y’-z”, in degrees) → COMAU, DOOSAN, OMRON
protocol version	The version of the robot-Pickit communication. The current version number is `11` and is not expected to change in the near future.

If your robot does not adhere to any of the above orientation conventions, it is recommended to use the GENERIC (quaternions) convention. The robot-side interface would then take the responsibility of converting back and forth between quaternions and the representation used by the robot.

Communication flow

An example communication flow is as follows:

The robot checks the Pickit mode using RC_PICKIT_CHECK_MODE.
After confirming that Pickit is in robot mode, the robot initiates a background thread that periodically sends robot pose updates (RC_PICKIT_NO_COMMAND) to Pickit.
To configure Pickit for the given product and workspace, the robot loads the desired product and setup configurations via RC_PICKIT_CONFIGURE.
The robot requests an object detection with RC_PICKIT_LOOK_FOR_OBJECTS. For this example, Pickit responds that two objects were found, of which the first object is part of the response message.
The robot requests specific pick point data for the first object with RC_PICKIT_GET_PICK_POINT_DATA.
After the first object has been picked, the robot requests the second and final object with RC_PICKIT_NEXT_OBJECT. This is again followed by RC_PICKIT_GET_PICK_POINT_DATA, to retrieve the pick point data of the last requested object.

Socket communication flow — Example communication flow between robot and Pickit.