Pickit socket interface

Pickit offers a low-level interface based on a TCP/IP socket communication which can be used to connect your robot (or PLC) with Pickit. Using the socket interface, the robot can for instance trigger detections, retrieve object poses and change the Pickit configuration.

The following article describes the technical details of the socket interface. This is relevant if you are planning to interface a new robot brand with Pickit. It is structured as follows:

Note

Before starting, make sure that the robot integration does not already exist and confirm with support@pickit3d.com that the integration is not under development.

Overview

The robot and Pickit exchange information in a request-response pattern, where the robot acts as the client and Pickit as the server. The robot sends a request, such as e.g. an object detection, to Pickit, which replies with a corresponding response. All requests are implemented in a purely synchronous fashion, and it is up to the client to ensure that no new requests are sent before having received a response for the previous request.

Next to this request-response exchange, Pickit also expects to receive periodic robot flange pose updates from the client. Note that the robot pose update is the only request for which the Pickit system does not reply with a matching response message.

Connection details

Type	TCP/IP socket
Port	5001 (TCP)
Byte order	Network order (big endian)

Once a Pickit system is started, it listens to TCP port 5001 and waits until the robot initiates a connection. This is done on the robot side by opening a TCP socket targeting the IP address of the Pickit system and the given port.

The IP address of the Pickit system can be found and changed in the Pickit network settings. The default IP for the Ethernet port labeled ROBOT is 169.254.5.180.

Protocol

Request and response messages that are exchanged between a robot and Pickit have a fixed size. A fixed-size message protocol has the advantage that it is easy to implement on the robot side, even with limited programming features.

Note

Request and response messages have a fixed size and do not have a begin and/or end character. While the TCP/IP protocol prevents data loss, the robot client implementation is responsible for keeping track of the boundary between messages by counting the number of sent/received bytes and comparing with the expected message size.

Request and response messages consist of a number of fields, each represented by an int32 (4 bytes). Floating-point data, such as distances and angles, are multiplied by a constant factor MULT = 10000 and truncated, before being sent as an int32. The receiving side then decodes this field by dividing the received value by MULT again. Negative numbers are encoded using two’s complement.

Request message

The request message that is sent from the robot to Pickit is 48 bytes long and consists of the following fields:

Request message structure

Field	Type	Length	Description
position	int32[3]	12 bytes	Robot flange position (XYZ, in meters) expressed in the right-handed robot base frame. Each field has to be multiplied by the MULT factor.
orientation	int32[4]	16 bytes	Robot flange orientation expressed in the right-handed robot base frame. The orientation encoding and units depend on the chosen orientation convention. Each field has to be multiplied by the MULT factor.
command	int32	4 bytes	One of the possible request commands.
payload	int32[2]	8 bytes	Optional payload fields.
meta	int32[2]	8 bytes	Orientation convention and protocol version. See the detailed meta field explanation.

Except for the optional payload fields, all fields are mandatory, and have to be set to sane values for every request. The payload fields are only required for certain commands, and are otherwise not used by Pickit.

The command field indicates which operation Pickit should execute. Possible commands and their corresponding response messages are explained in more detail below.

Response message

Except for the pose-update request, all requests are answered with a 64 byte long response message with the following structure:

Response message structure

Field	Type	Length	Description
position	int32[3]	12 bytes	Object position or pick point offset translation (XYZ, in meters), depending on the response status. See also the detailed command explanation. Each value has to be divided by MULT.
orientation	int32[4]	16 bytes	Object orientation or pick point offset rotation, depending on the response status. See also the detailed command explanation. Encoding and units depend on the chosen orientation convention and have to be divided by MULT.
payload	int32[6]	24 bytes	Optional payload fields.
status	int32	4 bytes	One of the defined status values.
meta	int32[2]	8 bytes	Orientation convention and protocol version. See the detailed meta field explanation.

Not every command response conveys pose information or additional payload. The status field determines if (and how) position, orientation and payload fields have to be interpreted. In the following sections, the individual commands and their corresponding responses are explained in more detail.

Available commands

The available Pickit commands are introduced in this section, grouped by category.

System commands

RC_PICKIT_NO_COMMAND (Robot pose update)

Send the current robot flange pose to Pickit. This information is used by Pickit to determine if the robot is still connected, as well as to update the 3D views in the Pickit web interface.

Robot pose updates should be sent periodically to Pickit, typically in the range of 10 messages per second. This command does not trigger a response, so it can be sent also while waiting for a response for a different command.

RC_PICKIT_CHECK_MODE

Check the current mode of Pickit. Pickit can be in the following modes: ROBOT or IDLE.

Response

Field	Value	Description
status	PICKIT_ROBOT_MODE	Pickit is in robot mode and able to send object poses to the robot. After booting, the Pickit system is in robot mode, which can be disabled and re-enabled via the web interface.
	PICKIT_IDLE_MODE	When not in robot mode, Pickit is in idle mode, and can be configured via the web interface.

RC_PICKIT_SHUTDOWN_SYSTEM

Cleanly shut down the Pickit processor.

Response

Field	Value	Description
status	PICKIT_SHUTDOWN_REQUEST_ACCEPTED	Shutdown request was accepted.
	PICKIT_SHUTDOWN_REQUEST_REJECTED	Could not shutdown the processor.

Detection commands

RC_PICKIT_CAPTURE_IMAGE

Capture a camera image to be used by a following RC_PICKIT_PROCESS_IMAGE request. This command waits until the image capture is finished. Immediately afterwards, the image processing can be done at the same time as the robot moves.

Note

This is the recommended command (followed RC_PICKIT_PROCESS_IMAGE) for robot-mounted cameras, where the robot has to stand still during image capture. For fixed cameras, prefer RC_PICKIT_LOOK_FOR_OBJECTS.

Response

Field	Value	Description
status	PICKIT_IMAGE_CAPTURED	Successfully captured camera image.
	PICKIT_NO_IMAGE_CAPTURED	Failed to capture camera image.

RC_PICKIT_PROCESS_IMAGE

Trigger an object detection on the camera image that was previously captured via RC_PICKIT_CAPTURE_IMAGE (or RC_PICKIT_LOOK_FOR_OBJECTS).

Response

Due to the fixed-size structure of the response message, only one object can be transmitted at a time. If more than one object has been detected, the information of the first object is communicated in the response message of a detection request, and the remaining objects can be obtained using the RC_PICKIT_NEXT_OBJECT request, one at a time.

Field	Value / Description
position	Object position (XYZ, in meters) expressed in the robot base frame. Each field has to be divided by MULT.
orientation	Object orientation expressed in the robot base frame. The orientation encoding and units depend on the chosen orientation convention. Each field has to be divided by MULT.
payload[0]	The duration (in seconds) elapsed between the capturing of the camera image and the moment the object information is sent to the robot. This value has to be divided by MULT.
payload[1]	For a Teach detection, this field contains the model ID of the current object. For a Flex/Pattern detection, this field contains the object type.
payload[2]	Object length (SQUARE, RECTANGLE, ELLIPSE, CYLINDER, POINTCLOUD, BLOB) or diameter (CIRCLE, SPHERE) in meters. Needs to be divided by MULT.
payload[3]	Object width (RECTANGLE, ELLIPSE, POINTCLOUD, BLOB) or diameter (CYLINDER) in meters. Needs to be divided by MULT.
payload[4]	Object height (POINTCLOUD, BLOB) in meters. Needs to be divided by MULT.
payload[5]	If this field is non-zero, it contains the number of remaining objects that can be retrieved via consecutive RC_PICKIT_NEXT_OBJECT requests.
status	PICKIT_OBJECT_FOUND	At least one object has been detected.
	PICKIT_NO_OBJECTS	No objects have been detected, but the ROI is not empty.
	PICKIT_NO_IMAGE_CAPTURED	No camera image has been captured.
	PICKIT_EMPTY_ROI	An empty ROI has been detected.
	INVALID_LICENSE	A license module is missing to run the detection.

Note

The object payloads (pose, size, …) should only be used if the status is PICKIT_OBJECT_FOUND. Otherwise, their value is undefined.

Attention

Object poses communicated by Pickit, via the position and orientation fields, have their Z-axis pointing upwards. Depending on the orientation of your robot’s flange frame, it might be necessary to flip the received poses by 180 degrees around the X-axis, such that the tool correctly aligns with the object at the moment of picking.

RC_PICKIT_LOOK_FOR_OBJECTS

Request Pickit to find objects in the current scene. This command performs an image capture and image processing in a single request.

Note

This is the recommended command for fixed cameras, where the robot does not have to stand still during image capture. For robot-mounted cameras, you should prefer RC_PICKIT_CAPTURE_IMAGE followed by RC_PICKIT_PROCESS_IMAGE.

Response

See response message of RC_PICKIT_PROCESS_IMAGE.

RC_PICKIT_NEXT_OBJECT

Request to return the next detected (valid and pickable) object in the objects table. Use this command when a single object detection run yields multiple detected objects. Note that the RC_PICKIT_LOOK_FOR_OBJECTS (or RC_PICKIT_PROCESS_IMAGE) command already returns the first object, if at least one was detected.

Response

See response message of RC_PICKIT_PROCESS_IMAGE.

If RC_PICKIT_NEXT_OBJECT is called after the last detected object has been sent to the robot, Pickit replies with a PICKIT_NO_OBJECTS status.

RC_PICKIT_LOOK_FOR_OBJECTS_WITH_RETRIES

Request Pickit to find objects in the current scene with retries. This command is similar to RC_PICKIT_LOOK_FOR_OBJECTS, but when no objects are found (but the Region of Interest (ROI) is not empty), Pickit will retry up to n times to find objects before giving up.

Request

Field	Description
payload[0]	Maximum number of detection retries.

Response

See response message of RC_PICKIT_PROCESS_IMAGE.

RC_PICKIT_GET_PICK_POINT_DATA

With multiple or flexible pick points, the robot needs to know how an object is being picked in order to drop it off at a fixed position. This information can be retrieved via RC_PICKIT_GET_PICK_POINT_DATA, which requests the pick point ID and pick point offset of the last requested object.

In the simplest case, the pick point ID is sufficient to know how an object was picked and how it should be dropped off. For applications with multiple pick points and/or pick points with flexible orientations, it is advised to make use of the pick point offset as well. Using the pick point offset, you don’t need to define a drop off point for every pick point. Instead, you only have to define a drop off point for every pick point reference, and apply the inverse pick point offset to this point.

Note that, due to the flipping of object poses, it is necessary to correct the communicated pick point offset on the robot side. This corrected offset is computed by pick_offset_to_apply = Rx × inv(offset_from_pickit) × Rx, where Rx denotes a rotation of 180 degrees around the X-axis. The pick point offset fpp_T_ppr is the transform between the final pick point and the pick point’s reference.

Response

Field	Value / Description
position	Pick point offset position (XYZ, in meters) expressed in the final pick point frame. Values have to be divided by MULT.
orientation	Pick point offset orientation expressed in the final pick point frame. The encoding and units depend on the chosen orientation convention. Values have to be divided by MULT.
payload[0]	ID of the selected pick point’s reference pick point.
payload[1]	ID of the pick point that was selected for the given object.
status	PICKIT_GET_PICK_POINT_DATA_OK	Successfully retrieved pick point data.
	PICKIT_GET_PICK_POINT_DATA_FAILED	Failed to retrieve pick point data.

Configuration commands

These commands are used to change the active product and setup configuration.

RC_PICKIT_CONFIGURE

Request Pickit to load a specific setup and product configuration. Each setup and product configuration have a unique ID assigned, which is shown in the web interface, next to the configuration name.

Request

Field	Description
payload[0]	ID of the setup configuration.
payload[1]	ID of the product configuration.

Response

Field	Value	Description
status	PICKIT_CONFIG_OK	Successfully loaded the specified configurations.
	PICKIT_CONFIG_FAILED	Failed to load the specified configurations.

RC_PICKIT_SET_CYLINDER_DIM

Request Pickit to set the cylinder dimensions when using the Teach cylinder model. For the command to succeed, there can be only one Teach model, it must be of type cylinder, and it must be enabled. Notice that, after calling this command with different dimensions than the ones of the current cylinder model, the product file will have unsaved changes. If you wish you can save them by calling the command RC_SAVE_ACTIVE_PRODUCT.

Request

Field	Description
payload[0]	Length of the cylinder (in meters). The value has to be multiplied by MULT.
payload[1]	Diameter of the cylinder (in meters). The value has to be multiplied by MULT.

Response

Field	Value	Description
status	PICKIT_CONFIG_OK	Successfully set the cylinder dimensions.
	PICKIT_CONFIG_FAILED	Failed to set the cylinder dimensions.

RC_SAVE_ACTIVE_SETUP

Request Pickit to save the currently loaded setup.

Response

Field	Value	Description
status	PICKIT_CONFIG_OK	Successfully saved the active setup.
	PICKIT_CONFIG_FAILED	Failed to save the active setup.

RC_SAVE_ACTIVE_PRODUCT

Request Pickit to save the currently loaded product.

Response

Field	Value	Description
status	PICKIT_CONFIG_OK	Successfully saved the active product.
	PICKIT_CONFIG_FAILED	Failed to save the active product.

RC_PICKIT_BUILD_BACKGROUND

Request Pickit to capture the current scene as background for background removal.

Response

Field	Value	Description
status	PICKIT_BUILD_BKG_CLOUD_OK	Successfully built background scene.
	PICKIT_BUILD_BKG_CLOUD_FAILED	Failed to built background scene.

Calibration commands

These commands are used to perform the robot-camera calibration.

RC_PICKIT_CONFIGURE_CALIB

Set the robot-camera calibration configuration.

Note

This command is intended for robot programs that do the entire calibration process from the robot program, and don’t use the Pickit web interface. The typical usage consists of the following sequence of commands:

• Call RC_PICKIT_CONFIGURE_CALIB

• Call RC_PICKIT_FIND_CALIB_PLATE zero or more times:

  • Zero times for METHOD_MANUAL

  • One time for METHOD_SINGLE_POSE (learn more)

  • Five times for METHOD_MULTI_POSE (learn more)

  • For method METHOD_DEFAULT, either one time or five times, depending of what has been configured in the user-interface.

• Call RC_PICKIT_COMPUTE_CALIB (but not for METHOD_MANUAL)

Request

Field	Value / Description
payload[0]	Calibration method. Possible values are: METHOD_SINGLE_POSE = 0 METHOD_MULTI_POSE = 1 METHOD_MANUAL = 2 METHOD_DEFAULT = 3
payload[1]	Camera mount. Possible values are: CAMERA_MOUNT_FIXED = 0 CAMERA_MOUNT_ON_ROBOT = 1 For calibration method METHOD_DEFAULT, this field is ignored.
position	Position (XYZ, in meters). Each field has to be multiplied by MULT. The value has a different meaning depending on the selected calibration method: METHOD_SINGLE_POSE: This is the helper transformation. METHOD_MULTI_POSE: Not required, do not specify. METHOD_MANUAL This is the full calibration transform. The pose shall be either expressed in the robot base frame (for CAMERA_MOUNT_FIXED), or the robot flange frame (for CAMERA_MOUNT_ON_ROBOT). METHOD_DEFAULT: Not required. do not specify.
orientation	The value has a different meaning depending on the selected calibration method. See above for details. Its encoding and units depend on the chosen orientation convention. Each field has to be multiplied by MULT.

Note

The combination of METHOD_SINGLE_POSE and CAMERA_MOUNT_ROBOT is not allowed.

Note

If you have a 4 DOF robot, you should perform an initial calibration using the calibration wizard of the Pickit web interface. There you can set the robot type to be 4 DOF, as well as the required distance offset. From this point on, calling the RC_PICKIT_CONFIGURE_CALIB command will work for a 4 DOF robot.

Note

The calibration method METHOD_DEFAULT requests to use the parameters specified in the user-interface. If you select this method, any other parameter specified in the socket message of the calibration configuration will be overruled by the user-interface settings.

Response

Field	Value	Description
status	PICKIT_CONFIGURE_CALIB_OK	Successfully configured robot-camera calibration.
	PICKIT_CONFIGURE_CALIB_FAILED	Failed to configure robot-camera calibration.

RC_PICKIT_FIND_CALIB_PLATE

Trigger Pickit to localize the calibration plate.

Response

Field	Value	Description
status	PICKIT_FIND_CALIB_PLATE_OK	Successfully localized calibration plate.
	PICKIT_FIND_CALIB_PLATE_FAILED	Failed to localize calibration plate.

RC_PICKIT_COMPUTE_CALIB

Trigger the computation of the robot-camera calibration.

Note

This command is intended for robot programs that do the entire calibration process from the robot program, and don’t use the Pickit web interface. See the documentation of the RC_PICKIT_CONFIGURE_CALIB command to learn more.

Response

The robot-camera calibration transform robot_T_camera (camera in robot base coordinates) is such that robot corresponds to the robot base (for camera fixed), or the robot flange (for camera on robot); and camera is the camera’s optical frame.

The error frame used in the response corresponds to the maximum difference observed when using the computed calibration to close a kinematic loop at the detected calibration plate locations.

Field	Value / Description
position	robot_T_camera position (XYZ, in meters). Each field has to be divided by MULT.
orientation	robot_T_camera orientation. The orientation encoding and units depend on the chosen orientation convention. Each field has to be divided by MULT.
payload[0]	distance of the error frame. After division by MULT, the result is expressed in meters.
payload[1]	angle of the error frame. After division by MULT, the result is expressed in degrees.
status	COMPUTE_CALIB_OK	Successfully computed the robot-camera calibration.
	COMPUTE_CALIB_FAILED	Failed to compute the robot-camera calibration.

RC_PICKIT_VALIDATE_CALIB

Validate the current robot-camera calibration.

Calibration validation requires the placement of the calibration plate as described in the validation procedure.

The error frame used in the response corresponds to the difference observed when using the current calibration to close a kinematic loop at the detected calibration plate location.

Response

Field	Value / Description
position	Error frame vector (XYZ, in meters), in robot base frame coordinates. Each field has to be divided by MULT.
orientation	Error frame orientation, in robot base frame coordinates. The orientation encoding and units depend on the chosen orientation convention. Each field has to be divided by MULT.
payload[0]	distance of the error frame. After division by MULT, the result is expressed in meters.
payload[1]	angle of the error frame. After division by MULT, the result is expressed in degrees.
status	PICKIT_VALIDATE_CALIB_OK	The validate command is successful.
	PICKIT_VALIDATE_CALIB_FAILED	The validate command failed.

Monitoring commands

This command is used to save data about a running application. This is specially useful when ramping up in production, as the stored data allows detecting eventual issues in the application.

RC_PICKIT_SAVE_SNAPSHOT

Request Pickit to save a snapshot with the last captured scene and the current configuration. Snapshots will be saved in the robot subfolder, which can be accessed from the web interface. If you wish to distinguish snapshots of different situations (e.g. mispicks and no detected objects), it is possible to optionally specify a subfolder inside the robot folder.

Request

Field	Description
payload[0]	Subfolder (inside robot) in which the snapshot should be saved. The subfolder is identified and named by a number between 1 and 255. If the value is outside this range, the snapshot is saved directly inside the robot folder.

Response

Field	Value	Description
status	PICKIT_SAVE_SNAPSHOT_OK	Successfully saved a snapshot.
	PICKIT_SAVE_SNAPSHOT_FAILED	Failed to save a snapshot.

Constants

Request command constants

RC_PICKIT_NO_COMMAND                    = -1
RC_PICKIT_CHECK_MODE                    = 0
RC_PICKIT_SHUTDOWN_SYSTEM               = 2
RC_PICKIT_FIND_CALIB_PLATE              = 10
RC_PICKIT_CONFIGURE_CALIB               = 11
RC_PICKIT_COMPUTE_CALIB                 = 12
RC_PICKIT_VALIDATE_CALIB                = 13
RC_PICKIT_LOOK_FOR_OBJECTS              = 20
RC_PICKIT_LOOK_FOR_OBJECTS_WITH_RETRIES = 21
RC_PICKIT_CAPTURE_IMAGE                 = 22
RC_PICKIT_PROCESS_IMAGE                 = 23
RC_PICKIT_NEXT_OBJECT                   = 30
RC_PICKIT_CONFIGURE                     = 40
RC_PICKIT_SET_CYLINDER_DIM              = 41
RC_SAVE_ACTIVE_SETUP                    = 42
RC_SAVE_ACTIVE_PRODUCT                  = 43
RC_PICKIT_SAVE_SCENE                    = 50
RC_PICKIT_BUILD_BACKGROUND              = 60
RC_PICKIT_GET_PICK_POINT_DATA           = 70

Response status constants

PICKIT_UNKNOWN_COMMAND                  = -99
PICKIT_ROBOT_MODE                       =   0
PICKIT_IDLE_MODE                        =   1
PICKIT_SHUTDOWN_REQUEST_ACCEPTED        =   5
PICKIT_SHUTDOWN_REQUEST_REJECTED        =   6
PICKIT_FIND_CALIB_PLATE_OK              =  10
PICKIT_FIND_CALIB_PLATE_FAILED          =  11
PICKIT_CONFIGURE_CALIB_OK               =  12
PICKIT_CONFIGURE_CALIB_FAILED           =  13
PICKIT_COMPUTE_CALIB_OK                 =  14
PICKIT_COMPUTE_CALIB_FAILED             =  15
PICKIT_VALIDATE_CALIB_OK                =  16
PICKIT_VALIDATE_CALIB_FAILED            =  17
PICKIT_OBJECT_FOUND                     =  20
PICKIT_NO_OBJECTS                       =  21
PICKIT_NO_IMAGE_CAPTURED                =  22
PICKIT_EMPTY_ROI                        =  23
PICKIT_IMAGE_CAPTURED                   =  26
PICKIT_INVALID_LICENSE                  =  27
PICKIT_CONFIG_OK                        =  40
PICKIT_CONFIG_FAILED                    =  41
PICKIT_SAVE_SNAPSHOT_OK                 =  50
PICKIT_SAVE_SNAPSHOT_FAILED             =  51
PICKIT_BUILD_BKG_CLOUD_OK               =  60
PICKIT_BUILD_BKG_CLOUD_FAILED           =  61
PICKIT_GET_PICK_POINT_DATA_OK           =  70
PICKIT_GET_PICK_POINT_DATA_FAILED       =  71

Object type constants

PICKIT_TYPE_SQUARE                =  21
PICKIT_TYPE_RECTANGLE             =  22
PICKIT_TYPE_CIRCLE                =  23
PICKIT_TYPE_ELLIPSE               =  24
PICKIT_TYPE_CYLINDER              =  32
PICKIT_TYPE_SPHERE                =  33
PICKIT_YTPE_POINTCLOUD            =  35
PICKIT_TYPE_BLOB                  =  50

Message metadata

To guarantee correct interpretation of the data on both the robot and the Pickit side, the following metadata is always sent along with both request and response messages:

Metadata message

Field	Value / Description
orientation convention	Convention that is being used to encode object or robot flange orientations. The following conventions are supported: Angle-axis (3D vector consisting of the unit axis multiplied by the angle in radians) → UNIVERSAL ROBOTS Quaternions (w,x,y,z) → GENERIC, ABB Euler Angles (x-y’-z”, in degrees) → STÄUBLI Fixed Angles (x-y-z, in degrees) → FANUC, NACHI, OMRON TM, YASKAWA Euler Angles (z-y’-x”, in degrees) → HANWHA, KUKA Euler Angles (z-y’-z”, in degrees) → COMAU, DOOSAN, OMRON
protocol version	The version of the robot-Pickit communication. The current version number is 11 and is not expected to change in the near future.

If your robot does not adhere to any of the above orientation conventions, it is recommended to use the GENERIC (quaternions) convention. The robot-side interface would then take the responsibility of converting back and forth between quaternions and the representation used by the robot.

Communication flow

An example communication flow is as follows:

1. The robot checks the Pickit mode using RC_PICKIT_CHECK_MODE.

2. After confirming that Pickit is in robot mode, the robot initiates a background thread that periodically sends robot pose updates (RC_PICKIT_NO_COMMAND) to Pickit.

3. To configure Pickit for the given product and workspace, the robot loads the desired product and setup configurations via RC_PICKIT_CONFIGURE.

4. The robot requests an object detection with RC_PICKIT_LOOK_FOR_OBJECTS. For this example, Pickit responds that two objects were found, of which the first object is part of the response message.

5. The robot requests specific pick point data for the first object with RC_PICKIT_GET_PICK_POINT_DATA.

6. After the first object has been picked, the robot requests the second and final object with RC_PICKIT_NEXT_OBJECT. This is again followed by RC_PICKIT_GET_PICK_POINT_DATA, to retrieve the pick point data of the last requested object.

Example communication flow between robot and Pickit.