The Pickit interface

This article proposes an Application Programming Interface (API) to be used by robot programs to perform vision-guided pick and place with Pickit.

It consists of a number of global variables and functions that encapsulate the socket communication between a robot and Pickit. The interface is meant to be robot-independent, and consists of global variables and functions written in pseudo-code, using a syntax similar to that of Python. Translating it to a specific robot programming language should consist of adapting to the syntax, features and best practices of the target language. Some deviation from the proposed interface might be necessary due to target language limitations, and the implementer is left with the responsibility of making the best compromise between what is possible and the proposed interface.

The following links point you to the API usage examples:

Picking

• Simple pick and place program

• Complete pick and place program

Calibration

• Calibration program

• Calibration validation program

Note

This article assumes an existing implementation of the socket communication layer.

Global variables

Detection variables

This section enumerates all output variables populated by Pickit when valid detection results are available, that is, when calling pickit_get_result() returns True.

These variables are typically used by a pick and place program in the pick and place hooks. They refer to the currently selected object for picking. PickitPick is the most commonly used variable, normally in the pick hook. All other variables are used for implementing smart picking or placing. Click on the entries below to expand them and learn more:

PickitPick

Object pick point, represented as a pose variable. From this point it’s possible to compute PickitPrePick and PickitPostPick for performing linear approach and retreat motions. Learn more about how they are computed here.

PickitPickId

ID of the pick point that was selected for picking, represented as an integer.

PickitPickRefId

ID of the reference pick point of the PickitPickId, represented as an integer. If PickitPickId was not created with respect to another pick point (that is, it has Origin as reference), this value will be the same as PickitPickId.

PickitPickOff

Pick point offset, represented as a pose variable. This is the relative transformation between the reference pick point (identified by PickitPickRefId) and the pick point that was selected for picking (identified by PickitPickId). This offset can be used to place an object consistently, regardless of how it was picked (which pick point, whether flexible pick orientation was used). If a Dropoff point was specified for an object picked from PickitPickRefId, it can be translated to PickitPickId by post-multiplying Dropoff by PickitPickOff. Learn more on how to use this variable in the smart placing examples.

PickitObjType

Object type, represented as an integer. The mapping between the object type and its identifier is the following: Pickit Teach Teach model ID Use this value to conditionally perform an action depending on the detected model. Pickit Flex and Pattern Square 21 Rectangle 22 Circle 23 Ellipse 24 Cylinder 32 Sphere 33 Blob 50 Pickit Bags An integer that holds information about the bag pattern and the detected layer orientation, according to this table.

PickitObjDim

Object dimensions, represented as a 3D array, in the units used by the robot. Depending on the object type, the array should be interpreted as follows: Pickit Teach [bbox x, bbox y, bbox z] Where bbox x represents the size of the object bounding box along its x-axis. Pickit Flex and Pattern Square [length, length, 0] Rectangle [length, width, 0] Circle [diameter, diameter, 0] Ellipse [length, width, 0] Cylinder [length, diameter, diameter] Sphere [diameter, diameter, diameter] Blob [bbox x, bbox y, bbox z]

PickitObjAge

Object age, represented as a floating-point number. The object age is the duration, in seconds, elapsed between the capturing of the camera image and the moment the object information is sent to the robot.

Calibration variables

This section enumerates all output variables populated by Pickit when computing a new calibration, or validating an existing calibration.

Click on the entries below to expand them and learn more:

PickitCal

The calibration transform, robot_T_camera, represented as a pose variable, where: robot is the robot base (for camera fixed), or the robot flange (for camera-on-robot). camera is the camera optical frame. This variable is updated by pickit_compute_calibration().

PickitCalErr

Calibration error, represented as a pose variable. This variable is updated by pickit_validate_calibration().

PickitCalErrPos

Position calibration error, represented as a floating-point number, in the units used by the robot. This value contains the norm of the calibration error translational component. This variable is updated by pickit_compute_calibration() and pickit_validate_calibration().

PickitCalErrAng

Orientation calibration error, represented as a floating-point number, in degrees. This value contains the angle of the calibration error rotational component, computed using the axis-angle representation. This variable is updated by pickit_compute_calibration() and pickit_validate_calibration().

Functions

System functions

The following functions send system-level commands to Pickit.

pickit_is_running()

Check whether robot mode is enabled in Pickit. Implementation Send RC_PICKIT_CHECK_MODE and wait for a response. Return True if the response status is PICKIT_ROBOT_MODE. All functions in this article except pickit_find_calibration_plate() require robot mode to be enabled, which can be done from the Pickit web interface.

pickit_shutdown()

Cleanly shut down the Pickit processor. Implementation Send RC_PICKIT_SHUTDOWN_SYSTEM and wait for a response. Return True if the response status is PICKIT_SHUTDOWN_REQUEST_ACCEPTED.

Detection functions

All functions in this section relate to performing object detection and collecting the results. Click on the entries below to expand them and learn more.

pickit_capture_image()

Capture a camera image without triggering object detection. This function blocks until image capture has completed (no need to call pickit_get_result()), and is especially useful when working with a robot-mounted camera, where the robot must remain stationary during image capture, but not during detection. This function is meant to be used prior to calling pickit_process_image(). If the camera moves during image capture, a warning will be shown in the web interface. For fixed camera mounts, it’s usually more convenient to call functions that combine image capture and processing, like pickit_find_objects() or pickit_find_objects_with_retries(retries). Implementation Send RC_PICKIT_CAPTURE_IMAGE and wait for a response. Return True if the response status is PICKIT_IMAGE_CAPTURED.

The functions below send a request to Pickit, but don’t wait for the response to arrive. Results from these requests are collected by calling pickit_get_result(), documented below, which waits until a response is ready.

pickit_find_objects()

Trigger a Pickit object detection using the currently active setup and product configuration. This call should be followed by a call to pickit_get_result(), which waits until a response for the detection request is ready. Implementation Send RC_PICKIT_LOOK_FOR_OBJECTS and don’t wait for a response.

pickit_find_objects_with_retries(retries)

Trigger a Pickit object detection with retries using the currently active setup and product configuration. As opposed to pickit_find_objects(), when no objects are found (but the Region of Interest (ROI) is not empty), Pickit will retry up to retries times to find objects before giving up. This call should be followed by a call to pickit_get_result(), which waits until a response for the detection request is ready. Parameters retries Maximum number of detection retries. Implementation Send RC_PICKIT_LOOK_FOR_OBJECTS_WITH_RETRIES and don’t wait for a response.

pickit_get_next_object()

Request the next detected object. This call should be followed by a call to pickit_get_result(), which waits until a response for the request is ready. A single object detection run might yield the detection of multiple objects. This function allows to request the next available object, if any, without the need of triggering a new detection and the time overhead it entails. It’s recommended to use this command only when objects in the detection region have not moved (significantly) since the last detection took place. A good example of when to use pickit_get_next_object() is when a detection is unreachable by the robot. An example of when using it is not recommended would be the following bin picking scenario: Trigger a Pickit detection that finds multiple objects. The first object is picked. Since objects are randomly placed in bin, neighboring objects move and fall into place. Call pickit_get_next_object() and attempt to pick next object. If the next object is one of the neighboring parts that moved, the pick is likely to fail. When the objects in the detection region have moved, it’s better to re-trigger object detection instead (by calling pickit_find_objects_with_retries(retries), for instance). Implementation Send RC_PICKIT_NEXT_OBJECT and don’t wait for a response.

pickit_process_image()

Trigger a Pickit object detection without image capture using the currently active setup and product configuration. This function uses the latest captured camera image, which typically comes from calling pickit_capture_image() just before. This call should be followed by a call to pickit_get_result(), which waits until a response for the detection request is ready. Refer to the documentation of pickit_capture_image() to learn more about the recommended usage of this function. Implementation Send RC_PICKIT_PROCESS_IMAGE and don’t wait for a response.

Note

You must collect the results of each request before sending a new one.

Tip

Note that all calls except pickit_get_next_object() (which is very fast) trigger a Pickit object detection run, which can potentially take multiple seconds.

In your robot program, it’s encouraged to perform robot motions or other non-Pickit-detection actions between calls to these functions and pickit_get_result() to save cycle time in your application. Refer to the pick and place loop of the simple pick and place program to learn more.

This function collects the results of the above requests:

pickit_get_result()

Wait for Pickit to reply with detection results. pickit_get_result() should always be paired to one of the above functions. Implementation Block until a reply from Pickit is received. When an object has been found, populate the output picking variables with valid content, which requires these steps: Populate PickitPick, PickitObjType, PickitObjDim, and PickitObjAge from the received response. Send RC_PICKIT_GET_PICK_POINT_DATA and wait for a new response. If the response status is PICKIT_GET_PICK_POINT_DATA_OK, populate from the newly received response these variables: PickitPickId, PickitPickRefId and PickitPickOff. Otherwise raise a non-recoverable error and halt program execution. The success of the original request can be queried by calling pickit_object_found() after pickit_get_result().

Configuration functions

These functions are used to change the active product and setup configuration.

pickit_configure(setup, product)

Loads the specified Pickit configuration (setup and product). Parameters IDs of valid setup and product configurations currently available in the connected Pickit system. Available configuration IDs are listed in the Pickit web interface. Implementation Send RC_PICKIT_CONFIGURE and wait for a response. If the response status is not PICKIT_CONFIG_OK, a non-recoverable error is raised and program execution is halted. This can happen, for instance, when the parameters correspond to non-existing setup or product IDs.

pickit_save_setup()

Save the active setup file. Implementation Send RC_SAVE_ACTIVE_SETUP and wait for a response. Return True if the response status is PICKIT_CONFIG_OK.

pickit_save_product()

Save the active product file. Implementation Send RC_SAVE_ACTIVE_PRODUCT and wait for a response. Return True if the response status is PICKIT_CONFIG_OK.

pickit_set_cylinder_dim(length, diameter)

Set the dimensions of a Teach cylinder model (diameter and length). Learn more about the conditions that apply to this request here. Parameters New length and diameter of the cylinder to detect. Implementation Send RC_PICKIT_SET_CYLINDER_DIM and wait for a response. Return True if the response status is PICKIT_CONFIG_OK.

pickit_build_background()

Build the background cloud used by some of the advanced Region of Interest filters. Calling this function will trigger a camera capture. So, if the camera mount is fixed, the robot must not occlude the camera view volume. If instead the camera is robot-mounted, the robot must be in the detection point (more). Implementation Send RC_PICKIT_BUILD_BACKGROUND and wait for a response. Return True if the response status is PICKIT_BUILD_BKG_CLOUD_OK.

Calibration functions

These functions are used to perform the robot-camera calibration.

Note

You don’t need to use these functions if your robot integration allows collecting calibration poses interactively from the calibration wizard, and you perform robot-camera calibration infrequently.

You should use these functions if you want to fully automate robot-camera calibration.

pickit_configure_calibration()

Set the robot-camera calibration configuration using the calibration method, camera mount and transform (if applicable) already set in the web interface calibration wizard. Equivalent to calling the advanced version (see below) like so: method = 3 # METHOD_DEFAULT. camera_mount = 0 # Ignored. transform = [0, 0, 0, 0, 0, 0] # Ignored. pickit_configure_calibration(method, camera_mount, transform)

pickit_configure_calibration(method, camera_mount, transform)

Set the robot-camera calibration configuration (advanced version). Parameters method: Calibration method (single-pose, multi-pose, or manual). camera_mount: Camera mount (fixed or camera on robot). transform: Required for manual and single-pose methods, not for multi-pose calibration. Learn more about the values of the parameters and the conditions that apply to them here. Implementation Send RC_PICKIT_CONFIGURE_CALIB and wait for a response. Return True if the response status is PICKIT_CONFIGURE_CALIB_OK.

pickit_find_calibration_plate()

Trigger detection of the robot-camera calibration plate. Robot mode doesn’t need to be enabled in Pickit to call this function. Implementation Send RC_PICKIT_FIND_CALIB_PLATE and wait for a response. If the respose status is not PICKIT_FIND_CALIB_PLATE_OK, a non-recoverable error is raised and program execution is halted.

pickit_compute_calibration()

Trigger the computation of the robot-camera calibration. Implementation Send RC_PICKIT_COMPUTE_CALIB and wait for a response. When the function returns True, the PickitCal, PickitCalErrPos and PickitCalErrAng variables are updated. Return True if the response status is PICKIT_COMPUTE_CALIB_OK.

pickit_validate_calibration(distance_tol, angle_tol)

Validate if the current robot-camera calibration is still valid. It’s considered valid if the location of the detected calibration plate is within the specified distance and angular tolerances. Parameters distance_tol: Distance tolerance, in the units used by the robot. angle_tol: Angular tolerance, in degrees. Implementation Send RC_PICKIT_VALIDATE_CALIB and wait for a response. When the function returns True, the PickitCalErr, PickitCalErrPos and PickitCalErrAng variables are updated. Return True if the response status is PICKIT_VALIDATE_CALIB_OK and PickitCalErrPos < distance_tol and PickitCalErrAng < angle_tol.

Monitoring functions

pickit_save_snapshot(subfolder_id=0)

Save a snapshot with the latest detection results. Snapshots will be saved in the robot subfolder of your Pickit system, which can be accessed from the web interface. If you wish to distinguish snapshots of different situations (e.g. mispicks and no detected objects), it is possible to optionally specify a subfolder inside the robot folder. For an example usage, refer to the description of the after_end hook of the pick and place program. Parameters Optional, ID of the subfolder where the snapshot will be saved. Allowed values are numbers between 1 and 255. Implementation Send RC_PICKIT_SAVE_SNAPSHOT and wait for a response. Return True if the response status is PICKIT_SAVE_SNAPSHOT_OK.

Helper functions

The following functions are helpers that don’t require extra communication with Pickit. They use the information received in the most recent detection results. These helpers are typically used as part of conditional expressions, such as an if statement.

pickit_object_found()

Check if the last call to pickit_get_result() produced valid detection results. Return True if detection results are available. When results are available, the output picking variables have valid contents. This function returns False when Pickit replied with no detection results (nominal usecase); or if pickit_get_result()  was called without a previous detection request to Pickit (should be avoided, as it makes no sense).

pickit_is_reachable(PickitPick, PrePick, PostPick)

Check if the pick point and the linear approach and retreat trajectories are reachable by the robot. Parameters PickitPick: The pick point. PrePick: Point determining the linear approach motion to the pick point. PostPick: Point determining the linear retreat motion from the pick point. This is an optional, but recommended function that requires the robot programming language to expose functionality like joint limits, inverse kinematics and safety plane checks. The exact implementation is open and left to the developer: The simplest implementation only checks the validity of the three input points. A more advanced implementation can additionally check points linearly sampled in the approach and retreat trajectories, that is, between PrePick → PickitPick and PickitPick → PostPick. Return True if the checked points are reachable by the robot.

pickit_is_reachable(Point)

Check if a point is reachable by the robot. Parameters Point: A robot TCP point. This function is an overload of pickit_is_reachable(PickitPick, PrePick, PostPick) that only checks the reachability of a single point, instead of a pick trajectory. This is an optional, but recommended function that requires the robot programming language to expose functionality like joint limits, inverse kinematics and safety plane checks. Return True if the checked point is reachable by the robot.

pickit_empty_roi()

Check if the last call to pickit_get_result() detected an empty Region of Interest (ROI). When pickit_object_found() returns False, it can be due to: The ROI is not empty, but Pickit doesn’t detect the active product. The ROI is empty. Use this function if you need to discriminate between these two situations. Return True if Pickit detected an empty ROI.

pickit_no_image_captured()

Check if object detection was unsuccessful due to a failure to capture a camera image. When this is the case, it typically indicates a hardware disconnection issue, such as a loose connector or broken cable. This function can be used as trigger to send an alarm to a higher level monitoring system. Return True if object detection was unsuccessful due to a failure to capture a camera image.

pickit_remaining_objects()

Get the number of remaining detected objects. After calling pickit_get_result(), this function returns the total number of object detections minus one, as the first object data is available through the output picking variables. This value is also equal to the number of times pickit_get_next_object() can be called. As such, the returned value decreases with each call to pickit_get_next_object(). Return Number of remaining object detections available for query.