AprilTags

Build Status

codecov.io

This package is a ccall wrapper for the AprilTags library tailored for Julia.

Installation

AprilTags.jl can be installed in Julia 0.7 and Julia 1.0 with:

#enter ']' to get the package manager and then type:
(v1.0) pkg> add AprilTags
# or
using Pkg
Pkg.add("AprilTags")

Note that Julia 0.6 is no longer supprted going forward. Please use v0.0.2 for julia 0.6.

Usage

Examples

See examples and test folder for basic AprilTag usage examples.

Initialization

Initialize a detector with the default (tag36h11) tag family.

# Create default detector
detector = AprilTagDetector()

The tag detector parameters can be set as shown bellow. The default parameters are the recommended starting point.

detector.nThreads = 4 #number of threads to use
detector.quad_decimate =  1.0 #"Decimate input image by this factor"
detector.quad_sigma = 0.0 #"Apply low-pass blur to input; negative sharpens"
detector.refine_edges = 1 #"Set to 1 to spend more time to align edges of tags"
detector.decode_sharpening = 0.25

quad_decimate

Detection of quads can be done on a lower-resolution image, improving speed at a cost of pose accuracy and a slight decrease in detection rate. Decoding the binary payload is still done at full resolution. Increase the image decimation if faster processing is required. A factor of 1.0 means the full-size input image is used.

quad_sigma

What Gaussian blur should be applied to the segmented image (used for quad detection?). Parameter is the standard deviation in pixels. Very noisy images benefit from non-zero values (e.g. 0.8).

refine_edges

When non-zero, the edges of the each quad are adjusted to "snap to" strong gradients nearby. This is useful when decimation is employed, as it can increase the quality of the initial quad estimate substantially. Generally recommended to be on (1). Very computationally inexpensive. Option is ignored if quad_decimate = 1.

decode_sharpening

How much sharpening should be done to decoded images? This can help decode small tags but may or may not help in odd lighting conditions or low light conditions. The default value is 0.25.

Detection

Process an input image and return a vector of detections. The input image can be loaded with the Images package.

image = load("example_image.jpg")
tags = detector(image)
#do something with tags here

The caller is responsible for freeing the memmory by calling

freeDetector!(detector)

Creating the AprilTag Images

The AprilTag images can be created using the getAprilTagImage function. Eg. to create a tag image with id 1 from family 'tag36h11' run:

getAprilTagImage(1, AprilTags.tag36h11)

Visualizing Tags

Images can be updated to include the tag detections,

drawTagBox!(image, tag)

Or if the camera matrix K is known, the axes can be shown with

drawTagAxes!(image, tag, K)

Furthermore, the tag IDs can also be visualized by first loading a different package:

using FreeTypeAbstraction
using AprilTags
using ImageView

# get an image
img_ = drawTags(image, K)
imshow(img_)

# drawTags!(image, K, tags)

Example

An easy (synthetic) verification example to test:

using AprilTags
using Images

detector = AprilTagDetector()

projImg = zeros(Gray{N0f8}, 480,640)
tag0 = kron(getAprilTagImage(0), ones(Gray{N0f8}, 4,4))
projImg[221:260,301:340] = tag0
projImg

# `img[i,j]` implies `width == x == j`, and `height == y == i`, top left corner `(0,0)`
fx = 1000.
fy = 1000.
cx = 320.
cy = 240.

K = [fx 0  cx;
      0 fy cy]

tags = detector(projImg)

imCol = RGB.(projImg)
foreach(tag->drawTagBox!(imCol, tag), tags)

taglength = 0.1
(tags, poses) = detectAndPose(detector, projImg, fx, fy, cx, cy, taglength)

poses[1]

results

3×4 Array{Float64,2}:
  0.996868    0.00273632   0.079042   -0.000300523
  0.00256118  0.99776     -0.0668423  -0.000417185
 -0.0790478   0.0668353    0.994628    3.1166

[copied from an issue discussion]

Camera Calibration

Using a AprilTag grid, it is possible to take a series of photographs for estimating the camera intrinsic calibration parameters:

See the Calibration example file for more details, as well as function documentation:

AprilTags.calcCalibResidualAprilTags!Function
calcCalibResidualAprilTags!(images, allTags; tagList, taglength, VERT, HORI, f_width, f_height, c_width, c_height, s, boardPattern, dodraw)

Grid of AprilTags calibration helper function. This function sets up a squared cost to be minimized by any method, including Optim.opimize. The cost function is constructed by assuming a grid of 40 AprilTags are of equal size and on a common co-planar surface like a computer screen.

The cost function is constructed by predicting from each tag detection individually where the corners of all other 39 tags should be on the co-planar surface. The discrepanc_height between the predicted and detected tag positions are square accumulated.

Notes

  • Assume regular spacing grid of 40 AprilTags 36h11, ids=1:40,
    • down 1:5 by 8 columns 1:5:40 across where taglength and spacing gap are equal.
    • See grid file at AprilTags/data/CameraCalibration/AprilTagGrid1to40.png
  • Current implementation requires all 40 tags to be visible and detected
    • Contributions welcome to allow a few missed detections.
  • Take pictures of the grid with the camera you want to calibrate, making sure all tags are clearly visible.
    • Combinations of the grid nearer and further, centered and slanted around the edges of the field of view are best.
    • Any number of images can be used, the more the better.
    • Note that half a doen high-res images might take up to 20 mins or so to optimize.
  • Julia Images.jl follows the common `::Array column-major–-i.e. vertical-major–-index convention
    • That is img[vertical, horizontal]
    • See https://evizero.github.io/Augmentor.jl/images/#Vertical-Major-vs-Horizontal-Major-1
  • This function has the ability to draw the predicted tag corners, see keyword dodraw=true.
  • This is not a copy of any other AprilCal or such software, this was newly written code out of pure frustration when getting stuff to work. The more native Julia code the better, because Julia is much more mobile and versatile than any of the other calibration software out there. See DevNotes below for roadmap of features to add, contributions welcome.

Example

Also see AprilTags/examples/AprilTagsGridClibration.jl.

This example shows how a series of photos of the tag grid image (just use your computer screen, not a projector) can be used to calibrate a camera. This example only shows the basic pinhole parameters, although more are possible, see keyword arguments for which calibration parameters are available. The latter part shows how to draw crosses to see before and after result.

using AprilTags
using FileIO

# where are the photos of the calibration files
filepaths = ["photo1.jpg"; "photo2.jpg";...]
# load the images into memory
imgs = load.(filepaths)

# It's imporant that you measure and specify the tag length correctly here
# 30 mm is just a guess, insert your own correct tag measurements here.
taglength = 0.03

# rough guess of what calibration parameters might be
# img[rows,columns] <==> img[height, width]
c_width = size(imgs[1],2) / 2 # columns across in Images.jl
c_height = size(imgs[1],1) / 2 # rows down in Images.jl
f_width = size(imgs[1],1)
f_height=f_width

#  detect the tags and duplicate the memory before freeing the detector
detector = AprilTagDetector()
tags = detector.(imgs) .|> deepcopy
# remember to free detector later

# setup the cost function, you can add more parameters here if you like
obj = (f_width, f_height, c_width, c_height) -> calcCalibResidualAprilTags!( imgs, tags, taglength=taglength, f_width=f_width, f_height=f_height, c_width=c_width, c_height=c_height, dodraw=false )
obj_ = (fc_wh) -> obj(fc_wh...)

# check that it works
obj_([f_width, f_height, c_width, c_height])

## Bring in the Optim.jl optimization routines
using Optim

# Run the optimization. BFGS is slower by more precise, it's okay to mix and match as coarse and fine optimization stages
result = optimize(obj_, [f_width; f_height; c_width; c_height], BFGS(), Optim.Options(x_tol=1e-8))

# see the optimized calibration parameters
@show f_width_, f_height_, c_width_, c_height_ = (result.minimizer...,)

## show the before and after images to visually confirm things are working
using ImageView

# bad calibration
img1_before = deepcopy(imgs[1])
calcCornerProjectionsAprilTags!(img1_before, taglength=taglength, f_width=f_width, f_height=f_height, c_width=c_width, c_height=c_height, dodraw=true)
imshow(img1_before)

# new calibration
img1_after = deepcopy(imgs[1])
calcCornerProjectionsAprilTags!(img1_after, taglength=taglength, f_width=f_width_, f_height=f_height_, c_width=c_width_, c_height=c_height_, dodraw=true)
imshow(img1_after)

# free the detector memory
freeDetector!(detector) # could also use a deepcopy to duplicate the memory to a secondary location and free the primary immediately

DevNotes

  • FIXME common JuliaRobotics/CameraModels.jl package shoudl be made
  • TODO Radial distortion parameters should be added, see https://en.wikipedia.org/wiki/Distortion_(optics)
  • TODO allow missed detections on grid of 40 tags
  • TODO auto-detect the grid so that any grid can be used (still assuming regular spacing)

Related

calcCornerProjectionsAprilTags!

source
AprilTags.calcCornerProjectionsAprilTags!Function
calcCornerProjectionsAprilTags!(cimg_, tags_; tagList, taglength, f_width, f_height, c_width, c_height, s, VERT, HORI, boardPattern, dodraw)

April grid calibration helper function to assemble the cost for a single image. This function can also be used to draw the corner points on images to show how the cost function is constructed as well as the calibration performance. See calcCalibResidualAprilTags! for details.

source

Manual Outline