Vision

HYPERLib contains classes to help you organize vision code and accomplish common tasks, like tracking reflective targets. Before getting into the details, consider what vision code should do:

1. Read images from camera(s).
2. Extract information from the image. This could be a common re-usable task like finding reflective targets, a user-supplied piece of code, or some combination. We may also want to extract multiple pieces of information from the same image.
3. Draw indicators on the image using the information collected in the previous step, and make them available to the dashboard.

This article will describe how to use hyperlib to connect all these peices together, but it won't get into the details of how to implement custom tasks for steps 2 and 3. To do that, look up any OpenCV tutorial online.

Modules, and pipelines, and processors, oh my!

Modules

This whole procedure should run on a separate thread from the rest of the robot code. In fact, since different cameras may operate independently, we should have one extra thread for each camera. HYPERLib wraps the camera, the thread, and the list of all tasks to run in steps (2) and (3) into a VisionModule.

Pipelines

Each task is represented by a VisionGUIPipeline object. This is an interface with two methods:

• process takes as input an image read directly from a camera, and extracts useful information from it. It saves this in instance variables for the next method, or for use in robot code.
• writeOutputs takes the saved information from earlier, or whatever other information it needs like sensor values, and draws indicators on the provided image. The given image might not be "clean" at this step, i.e. other pipelines may have already drawn things.

Note that a pipeline may only do something on one of these methods. For example, a simple pipeline may only draw indicators based on sensor or preference values, without needing to do anything extra.

FindTargetsPipeline and target processors

One of the most common types of pipelines involves finding colored rectangles and then further processing them in some way. This is done in FindTargetsPipeline. This takes as an argument a TargetProcessor, which is responsible for turning an array of rectangles into some useful information. Generally, an implementation will want to group the rectangles into targets in some way, find the biggest or closest target, and then provide a method for robot code to get the most recent position of the target. Two classes, ClosestTargetProcessor, and ClosestPairTargetProcessor, cover the most simple cases. An abstract class, AbstractTargetProcessor, is provided to help you get started with your own implementation.

TL;DR

For each camera on the robot, you want a VisionModule. This takes a list of VisionGUIPipelines describing what processing steps should be done. The most common one is FindTargetsPipeline. This finds a list of colored rectangles, and passes them to a TargetProcessor, which is in turn responsible for turning a list of rectangles into useful information, like where the closest target is. Depending on the shapes of targets, you might want to use a ClosestTargetProcessor (if the target is a single rectangle), a ClosestPairTargetProcessor (if the target it a pair rectangles with roughly the same y coordinate), or your own subclass of AbstractTargetProcessor. This will produce a VisionResult (or a custom sublclass of it) with the results. Code on the main thread can access this with the getLastResult method, or alternatively via xPID and yPID.

Example

For reference, here is a simplified version of the vision code from 2018. This used a single camera to track both cubes and the switch. Robot code would access the results from each through powerCubes().getLastResult() and switchTape().getLastResult().

public class VisionSystem {
    private final UsbCamera m_camera;
    private final VisionModule m_module;
    private final FindTargetsPipeline m_cubePipeline;
    private final FindTargetsPipeline m_switchPipeline;
    private final ClosestTargetProcessor m_cubeProcessor;
    private final ClosestPairTargetProcessor m_switchProcessor;
    private final CrosshairsPipeline m_cubeCrosshairs, m_switchCrosshairs;

    private static final int WIDTH = 160, HEIGHT = 120;

    private final PreferencesSet m_prefs = new PreferencesSet("Vision", this::onPreferencesUpdated);
    private final IntPreference m_cubeCrossY = m_prefs.addInt("Cubes Crosshairs Y", HEIGHT / 2);
    private final IntPreference m_switchCrossY = m_prefs.addInt("Switch Crosshairs Y", HEIGHT / 2);
    private final IntPreference m_cubeCrossX = m_prefs.addInt("Cubes Crosshairs X", WIDTH / 2);
    private final IntPreference m_switchCrossX = m_prefs.addInt("Switch Crosshairs X", HEIGHT / 2);

    /**
     * Construct all the objects related to vision. Call this in the initHelpers
     * method of the robot.
     */
    public VisionSystem() {
        // Set up camera
        m_camera = CameraServer.getInstance().startAutomaticCapture(0);
        m_camera.setResolution(WIDTH, HEIGHT);

        // Set up target processors
        m_cubeProcessor = new ClosestTargetProcessor(m_cubeCrossX::get, m_cubeCrossY::get);
        m_switchProcessor = new ClosestPairTargetProcessor(m_switchCrossX::get, m_switchCrossY::get);
        
        // Set up pipelines
        m_cubePipeline = new FindTargetsPipeline("Cubes", m_cubeProcessor);
        m_switchPipeline = new FindTargetsPipeline("Switch", m_switchProcessor);

        // These pipelines just draw, they don't do anything fancy
        m_cubeCrosshairs = new CrosshairsPipeline(m_cubeCrossX::get, m_cubeCrossY::get, 255, 255, 255);
        m_switchCrosshairs = new CrosshairsPipeline(m_switchCrossX::get, m_switchCrossY::get, 150, 150, 150);

        /*
         * Put it all together. We put the crosshairs pipelines last so they
         * will draw on top of everything else.
         */
        m_module = new VisionModule.Builder(m_camera)
                .addPipeline(m_switchPipeline)
                .addPipeline(m_cubePipeline)
                .addPipeline(m_switchCrosshairs)
                .addPipeline(m_cubeCrosshairs)
                .build();

        // Start the vision thread
        m_module.start();
    }

    public AbstractTargetProcessor<VisionResult> powerCubes() {
        return m_cubeProcessor;
    }

    public AbstractTargetProcessor<VisionResult> switchTape() {
        return m_switchProcessor;
    }
}