How to Set Up Real-Time Video Using OpenCV on Raspberry Pi 4

In this tutorial, I will show you how to install OpenCV on Raspberry Pi 4 and then get a real-time video stream going. OpenCV is a library that has a bunch of programming functions that enable us to do real-time computer vision.

rt_video_strea_rpi

Prerequisites

You Will Need

Install the Raspberry Pi Camera Module

Let’s install the Raspberry Pi Camera Module. Here are the official instructions, but I’ll walk through the whole process below.

Grab the Raspberry Pi Camera’s plastic clip. (optional)

Remove the ribbon cable that is currently in there. (optional)

2020-09-04-085435

Replace that small cable with the longer ribbon cable that came with the Flex Extension Cable Set. (optional)

2020-09-04-085651

Open the Camera Serial Interface on the Raspberry Pi by taking your fingers, pinching either side, and pulling up. The Camera Serial Interface is labeled “CAMERA”.

2020-09-04-085126

Push the long piece of ribbon into the interface. The silver pieces of the ribbon should be facing towards the CAMERA label on the Raspberry Pi board.

2020-09-04-090330

Hold the ribbon in place with one finger while you push down on the door. The door should snap into place.

This is how it should look at this stage (Ignore the other stuff in the photo).

2020-09-04-091651

Lightly pull on the ribbon to make sure that it is in their snugly. It shouldn’t come out when you pull on it.

Configure the Raspberry Pi

We now need to make sure the Raspberry Pi is configured properly to use the camera.

Start the Raspberry Pi.

Open a fresh terminal window, and type the following command:

sudo raspi-config

Go to Interfacing Options and press Enter.

1-interfacing-optionsJPG

Select Camera and press Enter to enable the camera.

Press Enter.

2-enable-cameraJPG

Go to Advanced Options and press Enter.

Select Resolution and press Enter.

Select a screen resolution. I’m using 1920 x 1080.

Press Enter.

3-screen-resolutionJPG

Go to Finish and press Enter.

Press Enter on <Yes> to reboot the Raspberry Pi.

Test the Raspberry Pi Camera Module

Open up a new terminal window. Let’s take a test photo by typing the following command:

raspistill -o Desktop/image.jpg

Your photo should be on your Desktop.

image

Here is how the camera looks when it is right side up. The black bar needs to be on top above the camera lens.

2020-09-04-101908

Install OpenCV

Let’s install OpenCV on our Raspberry Pi. I will largely be following this excellent tutorial. The process has a lot of steps, so go slow.

Update the packages by typing this command:

sudo apt-get update
sudo apt-get upgrade

Install these packages that assist in the compilation of the OpenCV code:

sudo apt install cmake build-essential pkg-config git

These packages provide the support that enable OpenCV to use different formats for both images and videos.

sudo apt install libjpeg-dev libtiff-dev libjasper-dev libpng-dev libwebp-dev libopenexr-dev
sudo apt install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev libdc1394-22-dev libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev

Install some more OpenCV dependencies: You can learn about each of these packages by doing a search here at the Debian website

sudo apt install libgtk-3-dev libqtgui4 libqtwebkit4 libqt4-test python3-pyqt5

Install these packages that help OpenCV run quickly on your Raspberry Pi.

sudo apt install libatlas-base-dev liblapacke-dev gfortran

Install the HDF5 packages that OpenCV will use to manage the data.

sudo apt install libhdf5-dev libhdf5-103

Install Python support packages.

sudo apt install python3-dev python3-pip python3-numpy

Increase the swap space. The swap space is the space that your operating system uses when RAM has reached its limit. 

sudo nano /etc/dphys-swapfile

Locate this line:

CONF_SWAPSIZE=100

Change that to:

CONF_SWAPSIZE=2048

Save the file and close out using the following keystrokes, one after the other:

CTRL+X 

Y

Enter

Regenerate the swap file:

sudo systemctl restart dphys-swapfile

Get the latest version of OpenCV from GitHub.

git clone https://github.com/opencv/opencv.git
git clone https://github.com/opencv/opencv_contrib.git

Both of these commands above will take a while to finish executing, so be patient.

Now, we need to compile OpenCV. We’ll create a new directory for this purpose.

mkdir ~/opencv/build
cd ~/opencv/build

Generate the makefile. Copy and paste this entire command below into your terminal and then press Enter.

cmake -D CMAKE_BUILD_TYPE=RELEASE \
    -D CMAKE_INSTALL_PREFIX=/usr/local \
    -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
    -D ENABLE_NEON=ON \
    -D ENABLE_VFPV3=ON \
    -D BUILD_TESTS=OFF \
    -D INSTALL_PYTHON_EXAMPLES=OFF \
    -D OPENCV_ENABLE_NONFREE=ON \
    -D CMAKE_SHARED_LINKER_FLAGS=-latomic \
    -D BUILD_EXAMPLES=OFF ..

Compile OpenCV using the command below. -j$(nproc) makes sure that all of the processors that are available to us are used for the compilation. This speeds things up:

make -j$(nproc)

Wait for OpenCV to compile. It will take a while, so you can go to lunch or do something else, and then return. My Raspberry Pi took about an hour to finish everything.

Once that compilation process has completed, you need to make sure the files get installed.

sudo make install

Run this command so that Raspberry Pi can find OpenCV.

sudo ldconfig

Now, let’s go back to the swapfile and reset the size to something smaller.

sudo nano /etc/dphys-swapfile

Locate this line:

CONF_SWAPSIZE=2048

Change that to:

CONF_SWAPSIZE=100

Save the file and close out using the following keystrokes:

CTRL+X 

Y

Enter

Restart swap.

sudo systemctl restart dphys-swapfile

Now, make sure that picamera is installed. picamera is a Python package that enables your camera to interface with your Python code. This second array module that we’re installing enables us to use OpenCV.

pip3 install picamera
pip3 install "picamera[array]"

Test OpenCV

Launch python.

python3

Import OpenCV

import cv2

Check to see which version of OpenCV you have installed.

cv2.__version__

Here is what you should see:

4-here-is-what-you-should-seeJPG

To close out the window, type:

exit()

Capture Real-Time Video

Now, go to the Python IDE in your Raspberry Pi by clicking the logo -> Programming -> Thonny Python IDE.

Write the following code to capture a live video feed. Credit to Dr. Adrian Rosebrock for this source code. I’ll name the file test_video_capture.py.

Here is the code:

# Credit: Adrian Rosebrock
# https://www.pyimagesearch.com/2015/03/30/accessing-the-raspberry-pi-camera-with-opencv-and-python/

# import the necessary packages
from picamera.array import PiRGBArray # Generates a 3D RGB array
from picamera import PiCamera # Provides a Python interface for the RPi Camera Module
import time # Provides time-related functions
import cv2 # OpenCV library

# Initialize the camera
camera = PiCamera()

# Set the camera resolution
camera.resolution = (640, 480)

# Set the number of frames per second
camera.framerate = 32

# Generates a 3D RGB array and stores it in rawCapture
raw_capture = PiRGBArray(camera, size=(640, 480))

# Wait a certain number of seconds to allow the camera time to warmup
time.sleep(0.1)

# Capture frames continuously from the camera
for frame in camera.capture_continuous(raw_capture, format="bgr", use_video_port=True):
    
    # Grab the raw NumPy array representing the image
    image = frame.array
    
    # Display the frame using OpenCV
    cv2.imshow("Frame", image)
    
    # Wait for keyPress for 1 millisecond
    key = cv2.waitKey(1) & 0xFF
    
    # Clear the stream in preparation for the next frame
    raw_capture.truncate(0)
    
    # If the `q` key was pressed, break from the loop
    if key == ord("q"):
        break

When you’re ready, click Run. Here is what my output was:

5_camera_video_streamJPG

That’s it for this tutorial. Keep building!

How to Install and Launch ROS2 Using Docker

In this tutorial, we will install and launch ROS2 using Docker within Ubuntu Linux 20.04. To test our installation, we will launch and run the popular Turtlebot3 software program.

What is Docker?

You can think of a Docker container on your PC as a computer inside your computer.

Docker is a service that provides portable stand-alone containers that come bundled with pre-installed software packages. These containers  make your life easier because they come with the right software versions and dependencies for whatever application you are trying to run.

Docker makes a lot more sense when you actually use it, so let’s get started.

Install Docker

Go to this link to download the appropriate Docker for your system.

Install Docker on your computer. Since I’m using Ubuntu 20.04 in Virtual Box on a Windows 10 machine, I’ll follow the Ubuntu instructions for “Install using the repository.”

If you’re using a Mac or Windows, follow those instructions. I actually find it easier to use Ubuntu, so if you’re using Windows, I highly suggest you do this tutorial first to get Ubuntu up and running, and then come back to this page.

Pull and Start the Docker Container With ROS2

Open a new terminal window, and create a new folder.

mkdir new_folder

Let’s pull a docker container. This one below comes with ROS2 already installed.

sudo docker pull /tiryoh/ros2-desktop-vnc:foxy

Now we need to start the docker container and mount the folder on it (the command below is all a single command).

sudo docker run -it -p 6080:80 -v /new_folder --name ros2_new_folder tiryoh/ros2-desktop-vnc:foxy

Note that ros2_new_folder is the name of the container.

Type the following in any browser: http://127.0.0.1:6080/

Here is the screen you should see:

1-screen-you-should-seeJPG

Install TurtleBot3

Click the menu icon in the very bottom left of the container.

Go to System Tools -> LX Terminal.

Download TurtleBot3.

mkdir -p ~/turtlebot3_ws/src
cd ~/turtlebot3_ws
wget https://raw.githubusercontent.com/ROBOTIS-GIT/turtlebot3/ros2/turtlebot3.repos
vcs import src < turtlebot3.repos

Wait a while while TurtleBot3 downloads into the container.

2-wait-a-whileJPG

Compile the source code.

colcon build --symlink-install

Wait for the source code to compile.

3-source-code-compileJPG

Let’s set the environment variables. Type these commands, one right after the other.

echo 'source ~/turtlebot3_ws/install/setup.bash' >> ~/.bashrc
echo 'export GAZEBO_MODEL_PATH=$GAZEBO_MODEL_PATH:~/turtlebot3_ws/src/turtlebot3/turtlebot3_simulations/turtlebot3_gazebo/models' >> ~/.bashrc
echo 'export TURTLEBOT3_MODEL=waffle_pi' >> ~/.bashrc
source ~/.bashrc

Launch an Empty World in Gazebo in ROS2

Now we will launch the simulation using the ros2 launch command.

ros2 launch turtlebot3_gazebo empty_world.launch.py

It will take a while to launch the first time, so be patient. You might see a bunch of warning messages, just ignore those.

When everything is done launching, you should see an empty world like this.

4-empty-gazebo-worldJPG

Go back to the terminal window by clicking the icon at the bottom of the screen.

Press CTRL+C to close it out.

For more simulations, check out this link at the TurtleBot3 website.

How to Stop the Docker Container

To stop the Docker container, go back to Ubuntu Linux and open a new terminal window. Type the following code.

sudo docker stop [name_of_container]

For example:

sudo docker stop ros2_new_folder

How to Restart the Docker Container

If you ever want to restart it in the future, you type:

sudo docker stop [name_of_container]

For example:

sudo docker restart ros2_new_folder

Type the following in any browser. http://127.0.0.1:6080/

That’s it.

How To Install Ubuntu and Raspbian on Your Raspberry Pi 4

In this tutorial, we will set up a Raspberry Pi 4 with both the Ubuntu 20.04 and Raspbian operating systems.

You Will Need

2020-08-29-150822

This section is the complete list of components you will need for this project.

Install Ubuntu

Prepare the SD Card

Grab the USB MicroSD Card Reader.

2020-08-29-151538

Take off the cap of the USB MicroSD Card Reader.

2020-08-29-151621

Stick the MicroSD card inside the Card Reader.

Stick the Card Reader into the USB drive on your computer.

Download the Raspberry Pi Imager for your operating system. I’m using Windows, so I will download Raspberry Pi Imager for Windows.

Open the Raspberry Pi Imager. Follow the instructions to install it on your computer.

When the installation is complete, click Finish.

Open the CHOOSE OS menu.

Scroll down, and click “Ubuntu”.

Select the Ubuntu 20.04 download (32-bit server).

Click CHOOSE SD. 

Select the microSD card you inserted. 

Click WRITE, and wait for the operating system to write to the card. It will take a while so be patient.

While you’re waiting, grab your Raspberry Pi 4 and the bag of heat sinks.

2020-08-29-154338
2020-08-29-154824

Peel off the backup of the heat sinks, and attach them to the corresponding chips on top of the Raspberry Pi.

2020-08-29-154344

Grab the cooling fan.

2020-08-29-155325

Connect the black wire to header pin 6 of the Raspberry Pi. Connect the red wire to header pin 1 of the Raspberry Pi.

2020-08-29-155750

Install the Raspberry Pi inside the case.

2020-08-29-155840

Connect the PiSwitch to the USB-C Power Supply. It should snap into place.

2020-08-29-160447

Once the installation of the operating system is complete, remove the microSD card reader from your laptop.

Set Up Wi-Fi

Reinsert the microSD card into your computer.

2020-08-29-161429

Open your File Manager, and find the network-config file. Mine is located on the F drive in Windows.

Open that file using Notepad or another plain text editor.

Uncomment (remove the “#” at the beginning) and edit the following lines to add your Wi-Fi credentials (don’t touch any of the other lines):

wifis:
  wlan0:
  dhcp4: true
  optional: true
  access-points:
    <wifi network name>:
      password: "<wifi password>"

For example:

wifis:
  wlan0:
  dhcp4: true
  optional: true
  access-points:
    "home network":
      password: "123456789"

Make sure the network name and password are inside quotes.

Save the file.

Set Up the Raspberry Pi

Safely remove the microSD Card Reader from your laptop.

Remove the microSD card from the card reader.

Insert the microSD card into the bottom of the Raspberry Pi.

Connect a keyboard and a mouse to the USB 3.0 ports of the Raspberry Pi.

2020-08-29-165525

Connect an HDMI monitor to the Raspberry Pi using the Micro HDMI cable connected to the Main MIcro HDMI port (which is labeled HDMI 0).

Connect the 3A USB-C Power Supply to the Raspberry Pi. You should see the computer boot.

Log in using “ubuntu” as both the password and login ID. You will have to do this multiple times.

You will then be asked to change your password.

Type:

sudo reboot

Type the command: 

hostname -I 

You will see the IP address of your Raspberry Pi. Mine is 192.168.254.68. Write this number down on a piece of paper because you will need it later.

Now update and upgrade the packages.

sudo apt update
sudo apt upgrade

Now, install a desktop.

sudo apt install xubuntu-desktop

Installing the desktop should take around 20-30 minutes or so.

Once that is done, it will ask you what you want as your default display manager. I’m going to use gdm3.

Wait for that to download.

Reboot your computer.

sudo reboot

Your desktop should show up.

Type in your password and press ENTER.

Click on Activities in the upper left corner of the screen to find applications.

If you want to see a Windows-like desktop, type the following commands:

cd ~/.cache/sessions/

Remove any files in there.

Type:

rm 

Then press the Tab key and press Enter.

Now type:

xfdesktop

Connect to Raspberry Pi from Your Personal Computer

Follow the steps for Putty under step 9b at this link to connect to your Raspberry Pi from your personal computer.

Install Raspbian

Now, we will install the Raspbian operating system. Turn off the Raspberry Pi, and remove the microSD card.

Insert the default microSD card that came with the kit.

Turn on the Raspberry Pi.

You should see an option to select “Raspbian Full [RECOMMENDED]”. Click the checkbox beside that.

Change the language to your desired language.

Click Wifi networks, and type in the password of your network.

Click Install.

Click Yes to confirm.

Wait while the operating system installs.

You’ll get a message that the operating system installed successfully.

Now follow all the steps from Step 7 of this tutorial. All the software updates at the initial startup take a really long time, so be patient. You can even go and grab lunch and return. It might not look like the progress bar is moving, but it is.

2020-08-29-212546

Keep building!