How to Create an ArUco Marker Using OpenCV Python

In this tutorial, I will show you how to create an ArUco Marker from scratch using OpenCV (Python). I will follow this tutorial.

If you do a search online for “aruco marker generator”, you can find a lot of free ArUco marker generators online. Here is one. We will create our own ArUco generator though.

Prerequisites

Create the Code

Open your favorite code editor, and write the following code. I will name my program generate_aruco_marker.py. This program generates an ArUco marker and saves it to your computer.

#!/usr/bin/env python
 
'''
Welcome to the ArUco Marker Generator!
 
This program:
  - Generates ArUco markers using OpenCV and Python
'''
 
from __future__ import print_function # Python 2/3 compatibility
import cv2 # Import the OpenCV library
import numpy as np # Import Numpy library
 
# Project: ArUco Marker Generator
# Date created: 12/17/2021
# Python version: 3.8
# Reference: https://www.pyimagesearch.com/2020/12/14/generating-aruco-markers-with-opencv-and-python/

desired_aruco_dictionary = "DICT_ARUCO_ORIGINAL"
aruco_marker_id = 1
output_filename = "DICT_ARUCO_ORIGINAL_id1.png"

# The different ArUco dictionaries built into the OpenCV library. 
ARUCO_DICT = {
  "DICT_4X4_50": cv2.aruco.DICT_4X4_50,
  "DICT_4X4_100": cv2.aruco.DICT_4X4_100,
  "DICT_4X4_250": cv2.aruco.DICT_4X4_250,
  "DICT_4X4_1000": cv2.aruco.DICT_4X4_1000,
  "DICT_5X5_50": cv2.aruco.DICT_5X5_50,
  "DICT_5X5_100": cv2.aruco.DICT_5X5_100,
  "DICT_5X5_250": cv2.aruco.DICT_5X5_250,
  "DICT_5X5_1000": cv2.aruco.DICT_5X5_1000,
  "DICT_6X6_50": cv2.aruco.DICT_6X6_50,
  "DICT_6X6_100": cv2.aruco.DICT_6X6_100,
  "DICT_6X6_250": cv2.aruco.DICT_6X6_250,
  "DICT_6X6_1000": cv2.aruco.DICT_6X6_1000,
  "DICT_7X7_50": cv2.aruco.DICT_7X7_50,
  "DICT_7X7_100": cv2.aruco.DICT_7X7_100,
  "DICT_7X7_250": cv2.aruco.DICT_7X7_250,
  "DICT_7X7_1000": cv2.aruco.DICT_7X7_1000,
  "DICT_ARUCO_ORIGINAL": cv2.aruco.DICT_ARUCO_ORIGINAL
}
 
def main():
  """
  Main method of the program.
  """
  # Check that we have a valid ArUco marker
  if ARUCO_DICT.get(desired_aruco_dictionary, None) is None:
    print("[INFO] ArUCo tag of '{}' is not supported".format(
      args["type"]))
    sys.exit(0)
    
  # Load the ArUco dictionary
  this_aruco_dictionary = cv2.aruco.Dictionary_get(ARUCO_DICT[desired_aruco_dictionary])
  
  # Allocate memory for the ArUco marker
  # We create a 300x300x1 grayscale image, but you can use any dimensions you desire.
  print("[INFO] generating ArUCo tag type '{}' with ID '{}'".format(
	desired_aruco_dictionary, aruco_marker_id))
    
  # Create the ArUco marker
  this_marker = np.zeros((300, 300, 1), dtype="uint8")
  cv2.aruco.drawMarker(this_aruco_dictionary, aruco_marker_id, 300, this_marker, 1)
  
  # Save the ArUco tag to the current directory
  cv2.imwrite(output_filename, this_marker)
  cv2.imshow("ArUco Marker", this_marker)
  cv2.waitKey(0)
  
if __name__ == '__main__':
  print(__doc__)
  main()

Save the file, and close it.

You need to have opencv-contrib-python installed and not opencv-python. Open a terminal window, and type:

pip uninstall opencv-python
pip install opencv-contrib-python

To run the program in Linux for example, type the following command:

python3 generate_aruco_marker.py
aruco_marker

If you want to restore OpenCV to the previous version after you’re finished creating the ArUco markers, type:

pip uninstall opencv-contrib-python
pip install opencv-python

To set the changes, I recommend rebooting your computer.

That’s it! Keep building!

Getting Started With OpenCV in ROS 2 Galactic (Python)

In this tutorial, we’ll learn the basics of how to interface ROS 2 Galactic with OpenCV, the popular computer vision library. These basics will provide you with the foundation to add vision to your robotics applications.

We’ll create an image publisher node to publish webcam data (i.e. video frames) to a topic, and we’ll create an image subscriber node that subscribes to that topic. 

If you’re interested in integrating OpenCV with ROS 2 Foxy, check out this tutorial.

Prerequisites

You can find all the code for this project at this link on my Google Drive.

Connect Your Built-in Webcam to Ubuntu 20.04 on a VirtualBox and Test OpenCV

Follow this tutorial to connect your built-in webcam to Ubuntu 20.04 on a Virtual Box and to test OpenCV on your machine.

Create a Package

Open a new terminal window, and navigate to the src directory of your workspace:

cd ~/dev_ws/src

Now let’s create a package named opencv_tools (you can name the package anything you want).

Type this command (this is all a single command):

ros2 pkg create --build-type ament_python opencv_tools --dependencies rclpy image_transport cv_bridge sensor_msgs std_msgs opencv2

Modify Package.xml

Go to the dev_ws/src/cv_basics folder.

cd ~/dev_ws/src/cv_basics

Make sure you have a text editor installed. I like to use gedit.

sudo apt-get install gedit

Open the package.xml file. 

gedit package.xml

Fill in the description of the cv_basics package, your email address and name on the maintainer line, and the license you desire (e.g. Apache License 2.0).

<description>Basic OpenCV-ROS2 nodes</description>
<maintainer email="automaticaddison@todo.todo">automaticaddison</maintainer>
<license>Apache License 2.0</license>

Save and close the file.

Build a Package

Return to the root of your workspace:

cd ~/dev_ws/

Build all packages in the workspace.

colcon build

Create the Image Publisher Node (Python)

Open a new terminal window, and type:

cd ~/dev_ws/src/opencv_tools/opencv_tools

Open a new Python file named basic_image_publisher.py.

gedit basic_image_publisher.py

Type the code below into it:

# Basic ROS 2 program to publish real-time streaming 
# video from your built-in webcam
# Author:
# - Addison Sears-Collins
# - https://automaticaddison.com
  
# Import the necessary libraries
import rclpy # Python Client Library for ROS 2
from rclpy.node import Node # Handles the creation of nodes
from sensor_msgs.msg import Image # Image is the message type
from cv_bridge import CvBridge # Package to convert between ROS and OpenCV Images
import cv2 # OpenCV library
 
class ImagePublisher(Node):
  """
  Create an ImagePublisher class, which is a subclass of the Node class.
  """
  def __init__(self):
    """
    Class constructor to set up the node
    """
    # Initiate the Node class's constructor and give it a name
    super().__init__('image_publisher')
      
    # Create the publisher. This publisher will publish an Image
    # to the video_frames topic. The queue size is 10 messages.
    self.publisher_ = self.create_publisher(Image, 'video_frames', 10)
      
    # We will publish a message every 0.1 seconds
    timer_period = 0.1  # seconds
      
    # Create the timer
    self.timer = self.create_timer(timer_period, self.timer_callback)
         
    # Create a VideoCapture object
    # The argument '0' gets the default webcam.
    self.cap = cv2.VideoCapture(0)
         
    # Used to convert between ROS and OpenCV images
    self.br = CvBridge()
   
  def timer_callback(self):
    """
    Callback function.
    This function gets called every 0.1 seconds.
    """
    # Capture frame-by-frame
    # This method returns True/False as well
    # as the video frame.
    ret, frame = self.cap.read()
          
    if ret == True:
      # Publish the image.
      # The 'cv2_to_imgmsg' method converts an OpenCV
      # image to a ROS 2 image message
      self.publisher_.publish(self.br.cv2_to_imgmsg(frame))
 
    # Display the message on the console
    self.get_logger().info('Publishing video frame')
  
def main(args=None):
  
  # Initialize the rclpy library
  rclpy.init(args=args)
  
  # Create the node
  image_publisher = ImagePublisher()
  
  # Spin the node so the callback function is called.
  rclpy.spin(image_publisher)
  
  # Destroy the node explicitly
  # (optional - otherwise it will be done automatically
  # when the garbage collector destroys the node object)
  image_publisher.destroy_node()
  
  # Shutdown the ROS client library for Python
  rclpy.shutdown()
  
if __name__ == '__main__':
  main()

Save and close the editor.

Modify Setup.py

Go to the following directory.

cd ~/dev_ws/src/opencv_tools/

Open setup.py.

gedit setup.py

Add the following line between the ‘console_scripts’: brackets:

'img_publisher = opencv_tools.basic_image_publisher:main',

The whole block should look like this:

entry_points={
    'console_scripts': [
      'img_publisher = opencv_tools.basic_image_publisher:main',
    ],
},

Save the file, and close it.

Create the Image Subscriber Node (Python)

Open a new terminal window, and type:

cd ~/dev_ws/src/opencv_tools/opencv_tools

Open a new Python file named basic_image_subscriber.py.

gedit basic_image_subscriber.py

Type the code below into it:

# Basic ROS 2 program to subscribe to real-time streaming 
# video from your built-in webcam
# Author:
# - Addison Sears-Collins
# - https://automaticaddison.com
  
# Import the necessary libraries
import rclpy # Python library for ROS 2
from rclpy.node import Node # Handles the creation of nodes
from sensor_msgs.msg import Image # Image is the message type
from cv_bridge import CvBridge # Package to convert between ROS and OpenCV Images
import cv2 # OpenCV library
 
class ImageSubscriber(Node):
  """
  Create an ImageSubscriber class, which is a subclass of the Node class.
  """
  def __init__(self):
    """
    Class constructor to set up the node
    """
    # Initiate the Node class's constructor and give it a name
    super().__init__('image_subscriber')
      
    # Create the subscriber. This subscriber will receive an Image
    # from the video_frames topic. The queue size is 10 messages.
    self.subscription = self.create_subscription(
      Image, 
      'video_frames', 
      self.listener_callback, 
      10)
    self.subscription # prevent unused variable warning
      
    # Used to convert between ROS and OpenCV images
    self.br = CvBridge()
   
  def listener_callback(self, data):
    """
    Callback function.
    """
    # Display the message on the console
    self.get_logger().info('Receiving video frame')
 
    # Convert ROS Image message to OpenCV image
    current_frame = self.br.imgmsg_to_cv2(data)
    
    # Display image
    cv2.imshow("camera", current_frame)
    
    cv2.waitKey(1)
  
def main(args=None):
  
  # Initialize the rclpy library
  rclpy.init(args=args)
  
  # Create the node
  image_subscriber = ImageSubscriber()
  
  # Spin the node so the callback function is called.
  rclpy.spin(image_subscriber)
  
  # Destroy the node explicitly
  # (optional - otherwise it will be done automatically
  # when the garbage collector destroys the node object)
  image_subscriber.destroy_node()
  
  # Shutdown the ROS client library for Python
  rclpy.shutdown()
  
if __name__ == '__main__':
  main()

Save and close the editor.

Modify Setup.py

Go to the following directory.

cd ~/dev_ws/src/opencv_tools/

Open setup.py.

gedit setup.py

Make sure the entry_points block looks like this:

entry_points={
    'console_scripts': [
      'img_publisher = opencv_tools.basic_image_publisher:main',
      'img_subscriber = opencv_tools.basic_image_subscriber:main',
    ],
},

Save the file, and close it.

Build the Package 

Return to the root of your workspace:

cd ~/dev_ws/

We need to double check that all the dependencies needed are already installed.

rosdep install -i --from-path src --rosdistro galactic -y

If you get the following error:

ERROR: the following packages/stacks could not have their rosdep keys resolved to system dependencies: cv_basics: Cannot locate rosdep definition for [opencv2]

To resolve the error, open your package.xml file.

cd ~/dev_ws/src/opencv_tools/
gedit package.xml

Change “opencv2” in your package.xml to “opencv-python” so that rosdep can find it. 

Save the file, and close it.

Build the package:

cd ~/dev_ws/
colcon build 

Run the Nodes

To run the nodes, open a new terminal window.

Make sure you are in the root of your workspace:

cd ~/dev_ws/

Run the publisher node. If you recall, its name is img_publisher.

ros2 run opencv_tools img_publisher
1-publishing-video-frame

Open a new terminal, and run the subscriber node.

ros2 run opencv_tools img_subscriber
2-receiving-video-frame

A window will pop up with the streaming video.

3-frame-appears

Check out the active topics.

ros2 topic list -t
4-ros2-topic-list

See the relationship between the active nodes.

rqt_graph
5-rqt_graph

Click the refresh button. The refresh button is that circular arrow at the top of the screen.

refresh

Press CTRL+C in all terminal windows when you’re ready to shut everything down.

That’s it! Keep building!

How to Load a TensorFlow Model Using OpenCV

In this tutorial, we will load a TensorFlow model (i.e. neural network) using the popular computer vision library known as OpenCV. To make things interesting, we will build an application to detect eating utensils (i.e. forks, knives, and spoons). Here is what the final output will look like:

eating-utensils-final

Our goal is to build an early prototype of a product that can make it easier and faster for a robotic chef arm, like the one created by Moley Robotics, to detect forks, knives, and spoons.

You Will Need

Directions

Open this page for the TensorFlow Object Detection API.

Download a weights and a config file for one of the pretrained object detection models. I will use Inception-SSD v2. You will want to right click and Save As.

2-go-to-this-page
1-two-files-config-weights

Use a program like 7-Zip to open the tar.gz archive file (i.e. click Open archive if using 7-Zip). 

Double click on ssd_inception_v2_coco_2017_11_17.tar.

3-double-click

Double click again on the folder name.

4-frozen-inference-graph

Locate a file inside the folder named frozen_inference_graph.pb. Move this file to your working directory (i.e. the same directory where we will write our Python program later in this tutorial).

I also have a pbtxt file named ssd_inception_v2_coco_2017_11_17.pbtxt that came from the config link. This file is of the form PBTXT. 

To create the ssd_inception_v2_coco_2017_11_17.pbtxt file, I right clicked on config, clicked Save As, saved to my Desktop, and then copied the contents of the pbtxt file on GitHub into this pbtxt file using Notepad++

Make sure the pb and pbtxt files are both in your working directory.

Alternatively, you can just download my pb file and my pbtxt file.

Now create a new Python program in your working directory called utensil_detector.py

Add the following code:

# Project: Eating Utensil Detector Using TensorFlow and OpenCV
# Author: Addison Sears-Collins
# Date created: August 1, 2021
# Description: This program detects forks, spoons, and knives

import cv2 as cv # OpenCV computer vision library
import numpy as np # Scientific computing library 

#  classes = ['person','bicycle','car','motorcycle','airplane' ,'bus','train','truck','boat' ,'traffic light','fire hydrant',
#    'stop sign','parking meter','bench','bird','cat','dog','horse','sheep','cow','elephant','bear','zebra','giraffe' ,
#    'backpack','umbrella','handbag' ,'tie','suitcase','frisbee' ,'skis','snowboard','sports ball' ,'kite',
#    'baseball bat','baseball glove','skateboard','surfboard','tennis rack','bottle','wine glass','cup','fork','knife',
#    'spoon','bowl','banana','apple' ,'sandwich','orange','broccoli','carrot','hot dog','pizza' ,'donut' ,'cake',
#    'chair' ,'couch' ,'potted plant','bed','dining table','toilet','tv','laptop','mouse','remote','keyboard',
#    'cell phone','microwave','oven','toaster','sink','refrigerator','book','clock','vase','scissors' ,'teddy bear',
#    'hair drier','toothbrush']

# Just use a subset of the classes
classes = ["background", "person", "bicycle", "car", "motorcycle",
  "airplane", "bus", "train", "truck", "boat", "traffic light", "fire hydrant",
  "unknown", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse",
  "sheep", "cow", "elephant", "bear", "zebra", "giraffe", "unknown", "backpack",
  "umbrella", "unknown", "unknown", "handbag", "tie", "suitcase", "frisbee", "skis",
  "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard",
  "surfboard", "tennis racket", "bottle", "unknown", "wine glass", "cup", "fork", "knife",
  "spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog",
  "pizza", "donut", "cake", "chair", "couch", "potted plant", "bed", "unknown", "dining table",
  "unknown", "unknown", "toilet", "unknown", "tv", "laptop", "mouse", "remote", "keyboard",
  "cell phone", "microwave", "oven", "toaster", "sink", "refrigerator", "unknown",
  "book", "clock", "vase", "scissors", "teddy bear", "hair drier", "toothbrush" ]

# Colors we will use for the object labels
colors = np.random.uniform(0, 255, size=(len(classes), 3))

# Open the webcam
cam = cv.VideoCapture(0)

pb  = 'frozen_inference_graph.pb'
pbt = 'ssd_inception_v2_coco_2017_11_17.pbtxt'

# Read the neural network
cvNet = cv.dnn.readNetFromTensorflow(pb,pbt)   

while True:

  # Read in the frame
  ret_val, img = cam.read()
  rows = img.shape[0]
  cols = img.shape[1]
  cvNet.setInput(cv.dnn.blobFromImage(img, size=(300, 300), swapRB=True, crop=False))

  # Run object detection
  cvOut = cvNet.forward()

  # Go through each object detected and label it
  for detection in cvOut[0,0,:,:]:
    score = float(detection[2])
    if score > 0.3:

      idx = int(detection[1])   # prediction class index. 

      # If you want all classes to be labeled instead of just forks, spoons, and knives, 
      # remove this line below (i.e. remove line 65)
      if classes[idx] == 'fork' or classes[idx] == 'spoon' or classes[idx] == 'knife':			
        left = detection[3] * cols
        top = detection[4] * rows
        right = detection[5] * cols
        bottom = detection[6] * rows
        cv.rectangle(img, (int(left), int(top)), (int(right), int(bottom)), (23, 230, 210), thickness=2)
           
        # draw the prediction on the frame
        label = "{}: {:.2f}%".format(classes[idx],score * 100)
        y = top - 15 if top - 15 > 15 else top + 15
        cv.putText(img, label, (int(left), int(y)),cv.FONT_HERSHEY_SIMPLEX, 0.5, colors[idx], 2)

  # Display the frame
  cv.imshow('my webcam', img)

  # Press ESC to quit
  if cv.waitKey(1) == 27: 
    break 

# Stop filming
cam.release()

# Close down OpenCV
cv.destroyAllWindows()

Save the code.

Here is what my working directory looks like.

here_is_what_my_working_directory

Run the code.

python utensil_detector.py

You should see object detection running in real time. If you place a fork, knife, or spoon in the camera, you will see it labeled accordingly.

That’s it! Keep building!