How to Make an Object Tracking Robot Using Raspberry Pi

In this tutorial, I will show you how to give your wheeled robot the ability to follow a colored ball. You will get your first taste of computer vision and image processing.

Special shout out to Matt Timmons-Brown for this project idea. He is the author of a really good book on Raspberry Pi robotics: (Learn Robotics with Raspberry Pi). Go check it out!

Video

Here is a video of what we will build in this tutorial.

Requirements

Here are the requirements:

  • Build a wheeled robot powered by Raspberry Pi that must identify and follow a yellow rubber ball using OpenCV, a library of programming functions for real-time computer vision and image processing.

You Will Need

The following components are used in this project. You will need:

Directions

Connecting the Raspberry Pi Camera Module

Make sure the Raspberry Pi is turned OFF.

Open the Camera Serial Interface on the Raspberry Pi. It is located next to the 3.5mm audio jack. Pull it upwards delicately from either side.

Insert the ribbon of the camera module into the Camera Serial Interface. Make sure the silver contacts face away from the 3.5mm audio jack.

2019-06-03-194539

Hold the ribbon in place while pushing down on the Camera Serial Interface port. Make sure it is closed.

2019-06-03-194547

Mount the camera to the front of the robot.

2019-06-03-200735

Power up Raspberry Pi.

Open up a configuration window:

sudo raspi-config

Interfacing Options –> ENTER –> Camera –> ENTER –> Yes

The camera is enabled.

Now, we need to set the resolution.

Advanced Options –> Resolution –> DMT Mode 82 1920×1080 60Hz 16: 9 –> ENTER –> Finish

Restart the Raspberry Pi by typing the following in a terminal window.

sudo reboot

Testing the Raspberry Pi Camera Module.

We need to take a test photo with our newly installed camera module.

Open a terminal window. Type the following command:

raspistill -o test_photo.jpg

Go to your home directory to see if the test photo is there. Here is the photo that mine took (back of my head).

test-camera

Setting Up Object Tracking

Now, we need to configure our system so the robot can track a yellow rubber ball.

Download the dependencies for OpenCV, a library of programming functions for real-time computer vision and image processing.

Type the following command into a terminal window:

sudo apt-get update
sudo apt-get install libblas-dev libatlas-base-dev libjasper-dev libqtgui4 libqt4-test

Y –> ENTER.

Wait a few moments while everything installs.

Install OpenCV using pip.

sudo pip3 install opencv-python
installing-open-cvPNG

Install the PiCamera library.

sudo apt-get install python3-picamera

Determining the HSV Value of the Yellow Ball

We need to select an appropriate HSV (hue, saturation, value) value for the yellow ball. HSV is an alternative color representation that is frequently used instead of the RGB (Red Green Blue) color model I covered in my light and sound wheeled robot post.

Here is the HSV table.

512px-HSV_color_solid_cylinder_saturation_gray

Since the ball is yellow, I’ll choose 60 as my starting number.

Open IDLE in your Raspberry Pi, and create a new file in your robot directory. Name it color_tester.py.

Here is the code for the program:

# Code source (Matt-Timmons Brown): https://github.com/the-raspberry-pi-guy/raspirobots
# import the necessary packages
from picamera.array import PiRGBArray
from picamera import PiCamera
import time
import cv2
import numpy as np


# initialize the camera and grab a reference to the raw camera capture
camera = PiCamera()
camera.resolution = (640, 480)
camera.framerate = 32
rawCapture = PiRGBArray(camera, size=(640, 480))

while True:
	while True:
		try:
			hue_value = int(input("Hue value between 10 and 245: "))
			if (hue_value < 10) or (hue_value > 245):
				raise ValueError
		except ValueError:
			print("That isn't an integer between 10 and 245, try again")
		else:
			break

	lower_red = np.array([hue_value-10,100,100])
	upper_red = np.array([hue_value+10, 255, 255])

	for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True):
		image = frame.array

		hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)

		color_mask = cv2.inRange(hsv, lower_red, upper_red)

		result = cv2.bitwise_and(image, image, mask= color_mask)

		cv2.imshow("Camera Output", image)
		cv2.imshow("HSV", hsv)
		cv2.imshow("Color Mask", color_mask)
		cv2.imshow("Final Result", result)

		rawCapture.truncate(0)

		k = cv2.waitKey(5) #& 0xFF
		if "q" == chr(k & 255):
			break

Place your ball about a yard in front of the camera.

2019-06-03-203156

Run the newly created program.

python3 color_tester.py

Choose 60.

You will see four windows.

  • Window 1. RGB representation
  • Window 2: HSV representation
  • Window 3: Show the portions of the frame that match a hue value of 60.
  • Window 4: Entire frame minus all portions that do NOT have a 60 hue value.
open-cv-robot-yellow-ballPNG

To try a different hue value, select any of the four windows above. Press Q to halt the output of the video.

Go to the terminal window, and try a new hue valve. I’ll try 29 this time. It worked!

You keep trying different numbers until Window 4 shows mostly your ball and nothing else. Be patient and try LOTS of numbers.

try-try-againPNG

Write down the hue value you ended up with on a sheet of paper.

Press CTRL-C in the terminal window to stop running color_tester.py.

Coding the Ball-Following Program

Open IDLE. Create a new file in your robot directory named:

ball_following_yellow. py

Here is the code (Credit to Matt Timmons-Brown, the author of a really good book on Raspberry Pi robotics: (Learn Robotics with Raspberry Pi):

# Code source (Matt-Timmons Brown): https://github.com/the-raspberry-pi-guy/raspirobots
from picamera.array import PiRGBArray
from picamera import PiCamera
import cv2
import numpy as np
import gpiozero

camera = PiCamera()
image_width = 640
image_height = 480
camera.resolution = (image_width, image_height)
camera.framerate = 32
rawCapture = PiRGBArray(camera, size=(image_width, image_height))
center_image_x = image_width / 2
center_image_y = image_height / 2
minimum_area = 250
maximum_area = 100000

robot = gpiozero.Robot(left=(22,27), right=(17,18))
forward_speed = 1.0
turn_speed = 0.8

HUE_VAL = 29

lower_color = np.array([HUE_VAL-10,100,100])
upper_color = np.array([HUE_VAL+10, 255, 255])

for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True):
	image = frame.array

	hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)

	color_mask = cv2.inRange(hsv, lower_color, upper_color)

	image2, countours, hierarchy = cv2.findContours(color_mask, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)

	object_area = 0
	object_x = 0
	object_y = 0

	for contour in countours:
		x, y, width, height = cv2.boundingRect(contour)
		found_area = width * height
		center_x = x + (width / 2)
		center_y = y + (height / 2)
		if object_area < found_area:
			object_area = found_area
			object_x = center_x
			object_y = center_y
	if object_area > 0:
		ball_location = [object_area, object_x, object_y]
	else:
		ball_location = None

	if ball_location:
		if (ball_location[0] > minimum_area) and (ball_location[0] < maximum_area):
			if ball_location[1] > (center_image_x + (image_width/3)):
				robot.right(turn_speed)
				print("Turning right")
			elif ball_location[1] < (center_image_x - (image_width/3)):
				robot.left(turn_speed)
				print("Turning left")
			else:
				robot.forward(forward_speed)
				print("Forward")
		elif (ball_location[0] < minimum_area):
			robot.left(turn_speed)
			print("Target isn't large enough, searching")
		else:
			robot.stop()
			print("Target large enough, stopping")
	else:
		robot.left(turn_speed)
		print("Target not found, searching")

	rawCapture.truncate(0)

Running the Ball – Following Program

Place your robot in an open space on the floor with the yellow rubber ball.

Run the program.

python3 ball_following_yellow.py

Whenever you want to stop the program, type CTRL-C.

How to Make a Line Following Robot Using Raspberry Pi

You all might remember the line-following robot I built using Arduino. Well today, we are going to do the same thing, but with a Raspberry Pi.

Special shout out to Matt Timmons-Brown for this project idea. He is the author of a really good book on Raspberry Pi robotics: (Learn Robotics with Raspberry Pi). Go check it out!

Video

Here is a video of what we will build in this tutorial.

Requirements

Here are the requirements:

  • Build a line-following robot using Raspberry Pi.

You Will Need

line-following-robot-6

The following components are used in this project. You will need:

Directions

Connecting the Infrared Line Sensor

Make sure the Raspberry Pi is turned OFF.

Connect the VCC pin of the IR line sensor to pin 1 of the Raspberry Pi.

Connect the GND pin of the line sensor to the blue (negative) ground rail of the solderless breadboard.

Connect the OUT pin of the line sensor to pin 21 (GPIO 9) of the solderless breadboard.

Testing the Infrared Line Sensor

Power up your Raspberry Pi, and open IDLE.

Create a new program called test_line_following.py.

Save it in to your robot directory.

Here is the code:

import gpiozero
import time
# Description: Code for testing the
# TCRT5000 IR Line Track Follower Sensor
# Author: Addison Sears-Collins
# Date: 05/29/2019


# Initialize line sensor to GPIO9
line_sensor = gpiozero.DigitalInputDevice(9)

while True:
  if line_sensor.is_active == False:
    print("Line detected")
  else:
    print("Line not detected")

  time.sleep(0.2) # wait for 0.2 seconds

Create a line-following course using your black electrical tape and your poster board. It should look something like this.

line-following-robot-3

I recommend leaving a 3-inch margin between the side of the poster board and the course. Don’t make curves that are too sharp.

line-following-robot-4

Run your test program from a terminal window.

cd robot
python3 test_line_following.py

Move your track in front of the sensor to see if the terminal prints out “Line detected. “

line-following-robot-1
line-following-robot-8

If you are running into issues, use a screwdriver to adjust the sensitivity of the sensor.

That white and blue potentiometer is what you should tweak.

Connect the other IR line sensor.

Connect the VCC pin of the IR line sensor to pin 17 of the Raspberry Pi using a female-to-female jumper wire.

Connect the GND pin of the IR line sensor to the blue (negative) ground rail of the solderless breadboard.

Connect the OUT pin of the IR line sensor to pin 23 (GPIO 11) of the Raspberry Pi.

Attaching the Sensors

Attach the first IR line sensor you wired up to the front, left side of the robot.

Attach the second IR line sensor to the right side of the robot.

Both sensors need to be just off the ground and can be mounted using 2×2 Lego blocks that extend downward from the body of the robot.

A piece of VELCRO is sufficient to attach both sensors.

Run the wires connected to the IR line sensors down through the gap in the robot body.

line-following-robot-2
line-following-robot-7

Create the Line-Following Program in Python

Open IDLE on your Raspberry Pi.

Create a new file inside your robot directory named

line_following_robot.py

Here is the code:

import gpiozero

# File name: line_following_robot.py
# Code source (Matt-Timmons Brown): https://github.com/the-raspberry-pi-guy/raspirobots
# Date created: 5/29/2019
# Python version: 3.5.3
# Description: Follow a line using a TCRT5000 IR
# Line Following Sensor

robot = gpiozero.Robot(left=(22,27), right=(17,18))

left = gpiozero.DigitalInputDevice(9)
right = gpiozero.DigitalInputDevice(11)

while True:
  if (left.is_active == True) and (right.is_active == True):
    robot.forward()
  elif (left.is_active == False) and (right.is_active == True):
    robot.right()
  elif (left.is_active == True) and (right.is_active == False):
    robot.left()
  else:
    robot.stop()

Deploying Your Line-Following Robot

Place your robot on your track. Make sure the line-following sensors are directly above the black line.

line-following-robot-5

Verify that your Raspberry Pi is connected to battery power, and your 4xAA battery pack is turned on.

Run your program from inside the robot directory.

cd robot
python3 line_following_robot.py

Watch your robot follow the line! Press CTRL-C anytime to stop the program.

How to Add Light to a Raspberry Pi Wheeled Robot

In this post, I will show you how to add light to a Raspberry Pi wheeled robot.

Special shout out to Matt Timmons-Brown for this project idea. He is the author of a really good book on Raspberry Pi robotics: (Learn Robotics with Raspberry Pi). Go check it out!

Requirements

Here are the requirements:

  • Add lights to a wheeled robot using the Adafruit NeoPixel Stick with 8 RGB LEDs.

You Will Need

The following components are used in this project. You will need:

Directions

Setting Up the RGB LED Stick

Cut four pins off one of the male pin header connectors with a pair of scissors.

Solder the four pins to the DIN (stands for “data-in”) side of the RGB LED stick.

lights-robot-rpi-1

If you don’t know how to solder, check out my video below.

Connect the 5VDC pin of the RGB LED stick to the positive (red) 5V rail of the solderless breadboard.

lights-robot-rpi-2

Connect the GND (Ground) pin of the RGB LED stick to the blue (negative) rail of the solderless breadboard.

lights-robot-rpi-3

Connect the DIN pin of the RGB LED stick to pin 19 (GPIO 10) of the Raspberry Pi. This is the Master Output Slave Input (MOSI) pin that the Pi uses to send information out via the SPI protocol (more on this in a second).

lights-robot-rpi-4

Mount the RGB LED on the Robot using Velcro. You can mount it anywhere you want.

lights-robot-rpi-5

Power up your Raspberry Pi.

Open a terminal window.

Verify that pip is installed. Pip is a tool that enables you to install and manage libraries that are not a part of Python’s standard library. Type the following commands in the terminal window.

sudo apt-get update
lights-robot-rpi-6
sudo apt-get install python3-pip
Install the rpi_ws 281x library.
sudo pip3 install rpi_ws281x

Make sure the SPI bus is enabled.

SPI is a communication interface built-in to several of the GPIO pins of the Raspberry Pi. It is good to use the SPI bus when you want data to be streamed over short distances, continuously with no interruptions.

Type this terminal command:

sudo raspi-config

Interfacing Options → SPI

lights-robot-rpi-7

Select Yes

Finish

Reboot the Raspberry Pi.

sudo reboot

Wait for the Raspberry Pi to reboot, then open a terminal window again in Raspberry Pi (I’m using Putty).

Modify the Graphics Programming Unit core frequency so that it is 250 MHz.

sudo nano /boot/config.txt

Scroll down, and add this to the bottom of the file.

core_freq = 250
lights-robot-rpi-8

CTRL-X –> Y –> ENTER to save

Reboot the Raspberry Pi.

sudo reboot 

Testing the rpi_ws281x Library

Open IDLE on your Raspberry Pi.

Create a new file.

Save it as rgb_library_test.py

Type the following code:

#!/usr/bin/env python3
# NeoPixel library strandtest example
# Author: Tony DiCola (tony@tonydicola.com)
#
# Direct port of the Arduino NeoPixel library strandtest example.  Showcases
# various animations on a strip of NeoPixels.
# Code source (Matt-Timmons Brown): https://github.com/the-raspberry-pi-guy/raspirobots
# Minor edits by Matt Timmons-Brown for "Learn Robotics with Raspberry Pi"

import time
from rpi_ws281x import *
import argparse

# LED strip configuration:
LED_COUNT      = 8      # Number of LED pixels.
#LED_PIN       = 18      # GPIO pin connected to the pixels (18 uses PWM!).
LED_PIN        = 10      # GPIO pin connected to the pixels (10 uses SPI /dev/spidev0.0).
LED_FREQ_HZ    = 800000  # LED signal frequency in hertz (usually 800khz)
LED_DMA        = 10      # DMA channel to use for generating signal (try 10)
LED_BRIGHTNESS = 255     # Set to 0 for darkest and 255 for brightest
LED_INVERT     = False   # True to invert the signal (when using NPN transistor level shift)
LED_CHANNEL    = 0       # set to '1' for GPIOs 13, 19, 41, 45 or 53
LED_STRIP      = ws.WS2811_STRIP_GRB # Strip type and color ordering

# Define functions which animate LEDs in various ways.
def colorWipe(strip, color, wait_ms=50):
    """Wipe color across display a pixel at a time."""
    for i in range(strip.numPixels()):
        strip.setPixelColor(i, color)
        strip.show()
        time.sleep(wait_ms/1000.0)

def theaterChase(strip, color, wait_ms=50, iterations=10):
    """Movie theater light style chaser animation."""
    for j in range(iterations):
        for q in range(3):
            for i in range(0, strip.numPixels(), 3):
                strip.setPixelColor(i+q, color)
            strip.show()
            time.sleep(wait_ms/1000.0)
            for i in range(0, strip.numPixels(), 3):
                strip.setPixelColor(i+q, 0)

def wheel(pos):
    """Generate rainbow colors across 0-255 positions."""
    if pos < 85:
        return Color(pos * 3, 255 - pos * 3, 0)
    elif pos < 170:
        pos -= 85
        return Color(255 - pos * 3, 0, pos * 3)
    else:
        pos -= 170
        return Color(0, pos * 3, 255 - pos * 3)

def rainbow(strip, wait_ms=20, iterations=1):
    """Draw rainbow that fades across all pixels at once."""
    for j in range(256*iterations):
        for i in range(strip.numPixels()):
            strip.setPixelColor(i, wheel((i+j) & 255))
        strip.show()
        time.sleep(wait_ms/1000.0)

def rainbowCycle(strip, wait_ms=20, iterations=5):
    """Draw rainbow that uniformly distributes itself across all pixels."""
    for j in range(256*iterations):
        for i in range(strip.numPixels()):
            strip.setPixelColor(i, wheel((int(i * 256 / strip.numPixels()) + j) & 255))
        strip.show()
        time.sleep(wait_ms/1000.0)

def theaterChaseRainbow(strip, wait_ms=50):
    """Rainbow movie theater light style chaser animation."""
    for j in range(256):
        for q in range(3):
            for i in range(0, strip.numPixels(), 3):
                strip.setPixelColor(i+q, wheel((i+j) % 255))
            strip.show()
            time.sleep(wait_ms/1000.0)
            for i in range(0, strip.numPixels(), 3):
                strip.setPixelColor(i+q, 0)

# Main program logic follows:
if __name__ == '__main__':
    # Process arguments
    parser = argparse.ArgumentParser()
    parser.add_argument('-c', '--clear', action='store_true', help='clear the display on exit')
    args = parser.parse_args()

    # Create NeoPixel object with appropriate configuration.
    strip = Adafruit_NeoPixel(LED_COUNT, LED_PIN, LED_FREQ_HZ, LED_DMA, LED_INVERT, LED_BRIGHTNESS, LED_CHANNEL)
    # Intialize the library (must be called once before other functions).
    strip.begin()

    print ('Press Ctrl-C to quit.')
    if not args.clear:
        print('Use "-c" argument to clear LEDs on exit')

    try:

        while True:
            print ('Color wipe animations.')
            colorWipe(strip, Color(255, 0, 0))  # Red wipe
            colorWipe(strip, Color(0, 255, 0))  # Blue wipe
            colorWipe(strip, Color(0, 0, 255))  # Green wipe
            print ('Theater chase animations.')
            theaterChase(strip, Color(127, 127, 127))  # White theater chase
            theaterChase(strip, Color(127,   0,   0))  # Red theater chase
            theaterChase(strip, Color(  0,   0, 127))  # Blue theater chase
            print ('Rainbow animations.')
            rainbow(strip)
            rainbowCycle(strip)
            theaterChaseRainbow(strip)

    except KeyboardInterrupt:
        if args.clear:
            colorWipe(strip, Color(0,0,0), 10)

Save it.

Open a new terminal window.

Run the test using the following command:

python3 rgb_library_test.py -c

The -c at the end makes sure that the RGB LEDs turn off when you type CTRL-C in the terminal window.

Managing the RGB LEDs Using the Nintendo Wii Remote Control

Now we will create a program to control the LEDs using our Wii remote. Open a new terminal window on your Raspberry Pi, and go to the robot directory.

cd robot

Open a new file in IDLE named rgb_wii_remote.py

Type the following code into the program:

import gpiozero
import cwiid
import time
from rpi_ws281x import *

# Author: Matt Timmons-Brown for "Learn Robotics with Raspberry Pi"
# Description: Enables control of the LED with the Wii Remote
# Minor modifications made by Addison Sears-Collins

robot = gpiozero.Robot(left=(22,27), right=(17,18))

print("Press and hold the 1+2 buttons on your Wiimote simultaneously")
wii = cwiid.Wiimote()
print("Connection established")
wii.rpt_mode = cwiid.RPT_BTN | cwiid.RPT_ACC

LED_COUNT      = 8
LED_PIN        = 10
LED_FREQ_HZ    = 800000
LED_DMA        = 10
LED_BRIGHTNESS = 150
LED_INVERT     = False
LED_CHANNEL    = 0
LED_STRIP      = ws.WS2811_STRIP_GRB

strip = Adafruit_NeoPixel(LED_COUNT, LED_PIN, LED_FREQ_HZ, LED_DMA, LED_INVERT, LED_BRIGHTNESS, LED_CHANNEL, LED_STRIP)
strip.begin()

def colorWipe(strip, color, wait_ms=50):
	"""Wipe color across display a pixel at a time."""
	for i in range(strip.numPixels()):
		strip.setPixelColor(i, color)
		strip.show()
		time.sleep(wait_ms/1000.0)

while True:
	buttons = wii.state["buttons"]
	if (buttons & cwiid.BTN_PLUS):
		colorWipe(strip, Color(255, 0, 0))  # Red wipe
	if (buttons & cwiid.BTN_HOME):
        colorWipe(strip, Color(0, 255, 0))  # Blue wipe
	if (buttons & cwiid.BTN_MINUS):
        colorWipe(strip, Color(0, 0, 255))  # Green wipe
	if (buttons & cwiid.BTN_B):
		colorWipe(strip, Color(0, 0, 0)) # Blank

	x = (wii.state["acc"][cwiid.X] - 95) - 25
	y = (wii.state["acc"][cwiid.Y] - 95) - 25

	if x < -25:
		x = -25
	if y < -25:
		y = -25
	if x > 25:
		x = 25
	if y > 25:
		y = 25

	forward_value = (float(x)/50)*2
	turn_value = (float(y)/50)*2

	if (turn_value < 0.3) and (turn_value > -0.3):
		robot.value = (forward_value, forward_value)
	else:
		robot.value = (-turn_value, turn_value)

Save the program.

Place your robot in an open space, and run the following command from inside your Pi’s robot directory.

python3 rgb_wii_remote.py

You can press the home, minus, and plus buttons to activate different lights.

Turn off the lights using the B button, and stop the program by typing CTRL-C.