How to Set Up Real-Time Video Using OpenCV on Raspberry Pi 4

In this tutorial, I will show you how to install OpenCV on Raspberry Pi 4 and then get a real-time video stream going. OpenCV is a library that has a bunch of programming functions that enable us to do real-time computer vision.

Prerequisites

You have set up Raspberry Pi with the Raspbian Operating System.

You Will Need

Install the Raspberry Pi Camera Module

Let’s install the Raspberry Pi Camera Module. Here are the official instructions, but I’ll walk through the whole process below.

Grab the Raspberry Pi Camera’s plastic clip. (optional)

Remove the ribbon cable that is currently in there. (optional)

Replace that small cable with the longer ribbon cable that came with the Flex Extension Cable Set. (optional)

Open the Camera Serial Interface on the Raspberry Pi by taking your fingers, pinching either side, and pulling up. The Camera Serial Interface is labeled “CAMERA”.

Push the long piece of ribbon into the interface. The silver pieces of the ribbon should be facing towards the CAMERA label on the Raspberry Pi board.

Hold the ribbon in place with one finger while you push down on the door. The door should snap into place.

This is how it should look at this stage (Ignore the other stuff in the photo).

Lightly pull on the ribbon to make sure that it is in their snugly. It shouldn’t come out when you pull on it.

Configure the Raspberry Pi

We now need to make sure the Raspberry Pi is configured properly to use the camera.

Start the Raspberry Pi.

Open a fresh terminal window, and type the following command:

sudo raspi-config

Go to Interfacing Options and press Enter.

Select Camera and press Enter to enable the camera.

Press Enter.

Go to Advanced Options and press Enter.

Select Resolution and press Enter.

Select a screen resolution. I’m using 1920 x 1080.

Press Enter.

Go to Finish and press Enter.

Press Enter on <Yes> to reboot the Raspberry Pi.

Test the Raspberry Pi Camera Module

Open up a new terminal window. Let’s take a test photo by typing the following command:

raspistill -o Desktop/image.jpg

Your photo should be on your Desktop.

Here is how the camera looks when it is right side up. The black bar needs to be on top above the camera lens.

Install OpenCV

Let’s install OpenCV on our Raspberry Pi. I will largely be following this excellent tutorial. The process has a lot of steps, so go slow.

Update the packages by typing this command:

sudo apt-get update

sudo apt-get upgrade

Install these packages that assist in the compilation of the OpenCV code:

sudo apt install cmake build-essential pkg-config git

These packages provide the support that enable OpenCV to use different formats for both images and videos.

sudo apt install libjpeg-dev libtiff-dev libjasper-dev libpng-dev libwebp-dev libopenexr-dev

sudo apt install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev libdc1394-22-dev libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev

Install some more OpenCV dependencies: You can learn about each of these packages by doing a search here at the Debian website:

sudo apt install libgtk-3-dev libqtgui4 libqtwebkit4 libqt4-test python3-pyqt5

Install these packages that help OpenCV run quickly on your Raspberry Pi.

sudo apt install libatlas-base-dev liblapacke-dev gfortran

Install the HDF5 packages that OpenCV will use to manage the data.

sudo apt install libhdf5-dev libhdf5-103

Install Python support packages.

sudo apt install python3-dev python3-pip python3-numpy

Increase the swap space. The swap space is the space that your operating system uses when RAM has reached its limit.

sudo nano /etc/dphys-swapfile

Locate this line:

CONF_SWAPSIZE=100

Change that to:

CONF_SWAPSIZE=2048

Save the file and close out using the following keystrokes, one after the other:

CTRL+X

Enter

Regenerate the swap file:

sudo systemctl restart dphys-swapfile

Get the latest version of OpenCV from GitHub.

git clone https://github.com/opencv/opencv.git

git clone https://github.com/opencv/opencv_contrib.git

Both of these commands above will take a while to finish executing, so be patient.

Now, we need to compile OpenCV. We’ll create a new directory for this purpose.

mkdir ~/opencv/build

cd ~/opencv/build

Generate the makefile. Copy and paste this entire command below into your terminal and then press Enter.

cmake -D CMAKE_BUILD_TYPE=RELEASE \
    -D CMAKE_INSTALL_PREFIX=/usr/local \
    -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
    -D ENABLE_NEON=ON \
    -D ENABLE_VFPV3=ON \
    -D BUILD_TESTS=OFF \
    -D INSTALL_PYTHON_EXAMPLES=OFF \
    -D OPENCV_ENABLE_NONFREE=ON \
    -D CMAKE_SHARED_LINKER_FLAGS=-latomic \
    -D BUILD_EXAMPLES=OFF ..

Compile OpenCV using the command below. -j$(nproc) makes sure that all of the processors that are available to us are used for the compilation. This speeds things up:

make -j$(nproc)

Wait for OpenCV to compile. It will take a while, so you can go to lunch or do something else, and then return. My Raspberry Pi took about an hour to finish everything.

Once that compilation process has completed, you need to make sure the files get installed.

sudo make install

Run this command so that Raspberry Pi can find OpenCV.

sudo ldconfig

Now, let’s go back to the swapfile and reset the size to something smaller.

sudo nano /etc/dphys-swapfile

Locate this line:

CONF_SWAPSIZE=2048

Change that to:

CONF_SWAPSIZE=100

Save the file and close out using the following keystrokes:

CTRL+X

Enter

Restart swap.

sudo systemctl restart dphys-swapfile

Now, make sure that picamera is installed. picamera is a Python package that enables your camera to interface with your Python code. This second array module that we’re installing enables us to use OpenCV.

pip3 install picamera

pip3 install "picamera[array]"

Test OpenCV

Launch python.

python3

Import OpenCV

import cv2

Check to see which version of OpenCV you have installed.

cv2.__version__

Here is what you should see:

To close out the window, type:

exit()

Capture Real-Time Video

Now, go to the Python IDE in your Raspberry Pi by clicking the logo -> Programming -> Thonny Python IDE.

Write the following code to capture a live video feed. Credit to Dr. Adrian Rosebrock for this source code. I’ll name the file test_video_capture.py.

Here is the code:

# Credit: Adrian Rosebrock
# https://www.pyimagesearch.com/2015/03/30/accessing-the-raspberry-pi-camera-with-opencv-and-python/

# import the necessary packages
from picamera.array import PiRGBArray # Generates a 3D RGB array
from picamera import PiCamera # Provides a Python interface for the RPi Camera Module
import time # Provides time-related functions
import cv2 # OpenCV library

# Initialize the camera
camera = PiCamera()

# Set the camera resolution
camera.resolution = (640, 480)

# Set the number of frames per second
camera.framerate = 32

# Generates a 3D RGB array and stores it in rawCapture
raw_capture = PiRGBArray(camera, size=(640, 480))

# Wait a certain number of seconds to allow the camera time to warmup
time.sleep(0.1)

# Capture frames continuously from the camera
for frame in camera.capture_continuous(raw_capture, format="bgr", use_video_port=True):
    
    # Grab the raw NumPy array representing the image
    image = frame.array
    
    # Display the frame using OpenCV
    cv2.imshow("Frame", image)
    
    # Wait for keyPress for 1 millisecond
    key = cv2.waitKey(1) & 0xFF
    
    # Clear the stream in preparation for the next frame
    raw_capture.truncate(0)
    
    # If the `q` key was pressed, break from the loop
    if key == ord("q"):
        break

When you’re ready, click Run. Here is what my output was:

That’s it for this tutorial. Keep building!

How to Configure Raspberry Pi to Run Programs On Startup

In this post, I’ll show you how to configure your Raspberry Pi so that your programs can run as soon as you plug in your Raspberry Pi to a power source.

Requirements

Here are the requirements:

Run a program as soon as the Raspberry Pi is turned on.

Directions

Power up your Raspberry Pi, and open up a terminal window.

Type:

sudo nano /etc/rc.local

Scroll down the file to the area right after fi but before exit 0. Type:

python3 /home/pi/robot/ball_following_yellow.py &

The format is

python3 /your/file/path/here/filename.py &

Press CTRL-X, and save the file.

Restart your Raspberry Pi.

sudo reboot

Your program, ball_following_yellow.py, should run on startup.

How to Make an Object Tracking Robot Using Raspberry Pi

In this tutorial, I will show you how to give your wheeled robot the ability to follow a colored ball. You will get your first taste of computer vision and image processing.

Special shout out to Matt Timmons-Brown for this project idea. He is the author of a really good book on Raspberry Pi robotics: (Learn Robotics with Raspberry Pi). Go check it out!

Video

Here is a video of what we will build in this tutorial.

Requirements

Here are the requirements:

Build a wheeled robot powered by Raspberry Pi that must identify and follow a yellow rubber ball using OpenCV, a library of programming functions for real-time computer vision and image processing.

You Will Need

The following components are used in this project. You will need:

Wheeled Robot
Yellow Rubber Ball (available from eBay)
Raspberry Pi Camera Module V2-8 (Standard)
2×2 Lego Brick (available from eBay)
VELCRO Brand Thin Clear Fasteners

Directions

Connecting the Raspberry Pi Camera Module

Make sure the Raspberry Pi is turned OFF.

Open the Camera Serial Interface on the Raspberry Pi. It is located next to the 3.5mm audio jack. Pull it upwards delicately from either side.

Insert the ribbon of the camera module into the Camera Serial Interface. Make sure the silver contacts face away from the 3.5mm audio jack.

Hold the ribbon in place while pushing down on the Camera Serial Interface port. Make sure it is closed.

Mount the camera to the front of the robot.

Power up Raspberry Pi.

Open up a configuration window:

sudo raspi-config

Interfacing Options –> ENTER –> Camera –> ENTER –> Yes

The camera is enabled.

Now, we need to set the resolution.

Advanced Options –> Resolution –> DMT Mode 82 1920×1080 60Hz 16: 9 –> ENTER –> Finish

Restart the Raspberry Pi by typing the following in a terminal window.

sudo reboot

Testing the Raspberry Pi Camera Module.

We need to take a test photo with our newly installed camera module.

Open a terminal window. Type the following command:

raspistill -o test_photo.jpg

Go to your home directory to see if the test photo is there. Here is the photo that mine took (back of my head).

Setting Up Object Tracking

Now, we need to configure our system so the robot can track a yellow rubber ball.

Download the dependencies for OpenCV, a library of programming functions for real-time computer vision and image processing.

Type the following command into a terminal window:

sudo apt-get update

sudo apt-get install libblas-dev libatlas-base-dev libjasper-dev libqtgui4 libqt4-test

Y –> ENTER.

Wait a few moments while everything installs.

Install OpenCV using pip.

sudo pip3 install opencv-python

Install the PiCamera library.

sudo apt-get install python3-picamera

Determining the HSV Value of the Yellow Ball

We need to select an appropriate HSV (hue, saturation, value) value for the yellow ball. HSV is an alternative color representation that is frequently used instead of the RGB (Red Green Blue) color model I covered in my light and sound wheeled robot post.

Here is the HSV table.

512px-HSV_color_solid_cylinder_saturation_gray

Since the ball is yellow, I’ll choose 60 as my starting number.

Open IDLE in your Raspberry Pi, and create a new file in your robot directory. Name it color_tester.py.

Here is the code for the program:

# Code source (Matt-Timmons Brown): https://github.com/the-raspberry-pi-guy/raspirobots
# import the necessary packages
from picamera.array import PiRGBArray
from picamera import PiCamera
import time
import cv2
import numpy as np


# initialize the camera and grab a reference to the raw camera capture
camera = PiCamera()
camera.resolution = (640, 480)
camera.framerate = 32
rawCapture = PiRGBArray(camera, size=(640, 480))

while True:
	while True:
		try:
			hue_value = int(input("Hue value between 10 and 245: "))
			if (hue_value < 10) or (hue_value > 245):
				raise ValueError
		except ValueError:
			print("That isn't an integer between 10 and 245, try again")
		else:
			break

	lower_red = np.array([hue_value-10,100,100])
	upper_red = np.array([hue_value+10, 255, 255])

	for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True):
		image = frame.array

		hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)

		color_mask = cv2.inRange(hsv, lower_red, upper_red)

		result = cv2.bitwise_and(image, image, mask= color_mask)

		cv2.imshow("Camera Output", image)
		cv2.imshow("HSV", hsv)
		cv2.imshow("Color Mask", color_mask)
		cv2.imshow("Final Result", result)

		rawCapture.truncate(0)

		k = cv2.waitKey(5) #& 0xFF
		if "q" == chr(k & 255):
			break

Place your ball about a yard in front of the camera.

Run the newly created program.

python3 color_tester.py

Choose 60.

You will see four windows.

Window 1. RGB representation
Window 2: HSV representation
Window 3: Show the portions of the frame that match a hue value of 60.
Window 4: Entire frame minus all portions that do NOT have a 60 hue value.

To try a different hue value, select any of the four windows above. Press Q to halt the output of the video.

Go to the terminal window, and try a new hue valve. I’ll try 29 this time. It worked!

You keep trying different numbers until Window 4 shows mostly your ball and nothing else. Be patient and try LOTS of numbers.

Write down the hue value you ended up with on a sheet of paper.

Press CTRL-C in the terminal window to stop running color_tester.py.

Coding the Ball-Following Program

Open IDLE. Create a new file in your robot directory named:

ball_following_yellow. py

Here is the code (Credit to Matt Timmons-Brown, the author of a really good book on Raspberry Pi robotics: (Learn Robotics with Raspberry Pi):

# Code source (Matt-Timmons Brown): https://github.com/the-raspberry-pi-guy/raspirobots
from picamera.array import PiRGBArray
from picamera import PiCamera
import cv2
import numpy as np
import gpiozero

camera = PiCamera()
image_width = 640
image_height = 480
camera.resolution = (image_width, image_height)
camera.framerate = 32
rawCapture = PiRGBArray(camera, size=(image_width, image_height))
center_image_x = image_width / 2
center_image_y = image_height / 2
minimum_area = 250
maximum_area = 100000

robot = gpiozero.Robot(left=(22,27), right=(17,18))
forward_speed = 1.0
turn_speed = 0.8

HUE_VAL = 29

lower_color = np.array([HUE_VAL-10,100,100])
upper_color = np.array([HUE_VAL+10, 255, 255])

for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True):
	image = frame.array

	hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)

	color_mask = cv2.inRange(hsv, lower_color, upper_color)

	image2, countours, hierarchy = cv2.findContours(color_mask, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)

	object_area = 0
	object_x = 0
	object_y = 0

	for contour in countours:
		x, y, width, height = cv2.boundingRect(contour)
		found_area = width * height
		center_x = x + (width / 2)
		center_y = y + (height / 2)
		if object_area < found_area:
			object_area = found_area
			object_x = center_x
			object_y = center_y
	if object_area > 0:
		ball_location = [object_area, object_x, object_y]
	else:
		ball_location = None

	if ball_location:
		if (ball_location[0] > minimum_area) and (ball_location[0] < maximum_area):
			if ball_location[1] > (center_image_x + (image_width/3)):
				robot.right(turn_speed)
				print("Turning right")
			elif ball_location[1] < (center_image_x - (image_width/3)):
				robot.left(turn_speed)
				print("Turning left")
			else:
				robot.forward(forward_speed)
				print("Forward")
		elif (ball_location[0] < minimum_area):
			robot.left(turn_speed)
			print("Target isn't large enough, searching")
		else:
			robot.stop()
			print("Target large enough, stopping")
	else:
		robot.left(turn_speed)
		print("Target not found, searching")

	rawCapture.truncate(0)

Running the Ball – Following Program

Place your robot in an open space on the floor with the yellow rubber ball.

Run the program.

python3 ball_following_yellow.py

Whenever you want to stop the program, type CTRL-C.