Neural networks are the workhorses of the rapidly growing field known as deep learning. Neural networks are used for all sorts of applications where a prediction of some sort is desired. Here are some examples:
Predicting the type of objects in an image or video
Sales forecasting
Speech recognition
Medical diagnosis
Risk management
and countless other applications…
In this post, I will explain how neural networks make those predictions by boiling these structures down to their fundamental parts and then building up from there.
Imagine you run a business that provides short online courses for working professionals. Some of your courses are free, but your best courses require the students to pay for a subscription.
You want to create a neural network to predict if a free student is likely to upgrade to a paid subscription. Let’s create the most basic neural network you can make.
OK, so there is our neural network. To implement this neural network on a computer, we need to translate this diagram into a software program. Let’s do that now using Python, the most popular language for machine learning.
# Declare a variable named weight and
# initiate it with a value
weight = 0.075
# Create a method called neural_network that
# takes as inputs, the input data (number of
free courses a student has taken during the
# last 30 days) and the weight of the connection.
# The method returns the prediction.
def neural_network(input, weight):
# The input data multiplied by the weight
# equals the prediction
prediction = input * weight
# This is the output
return prediction
So we currently have five students, all of whom are free students. The number of free courses these users have taken during the last 30 days is 12, 3, 5, 6, and 8. Let’s code this in Python as a list.
number_of_free_courses_taken = [12, 3, 5, 6, 8]
Let’s make a prediction for the first student, the one who has taken 12 free courses over the last 30 days.
Now let’s put the diagram into code.
# Extract the first value of the list...12...
# and store into a variable named input
first_student_input = number_of_free_courses_taken[0]
# Call the neural_network method and store the
# prediction result into a variable
first_student_prediction = neural_network(
first_student_input, weight)
# Print the prediction
print(first_student_prediction)
OK. We have finished the code. Let’s see how it looks all together.
Open a Jupyter Notebook and run the code above, or run the code inside your favorite Python IDE.
Here is what I got:
What did you get? Did you get 0.9? If so, congratulations!
Let’s see what is happening when we run our code. We called the neural_network method. The first operation performed inside that method is to multiply the input by the weight and return the result. In this case, the input is 12, and the weight is 0.075. The result is 0.9.
0.9 is stored in the first_student_prediction variable.
And this, my friend, is the most basic building block of a neural network. A neural network in its simplest form consists of one or more weights which you can multiply by input data to make a prediction.
Let’s take a look at some questions you might have at this stage.
What kind of input data can go into a neural network?
Real numbers that can be measured or calculated somewhere in the real world. Yesterday’s high temperature, a medical patient’s blood pressure reading, previous year’s rainfall, or average annual rainfall are all valid inputs into a neural network. Negative numbers are totally acceptable as well.
A good rule of thumb is, if you can quantify it, you can use it as an input into a neural network. It is best to use input data into a neural network that you think will be relevant for making the prediction you desire.
For example, if you are trying to create a neural network to predict if a patient has breast cancer or not, how many fingers a person has probably not going to be all that relevant. However, how many days per month a patient exercises is likely to be a relevant piece of input data that you would want to feed into your neural network.
What does a neural network predict?
A neural network outputs some real number. In some neural network implementations, we can do some fancy mathematics to limit the output to some real number between 0 and 1. Why would we want to do that? Well in some applications we might want to output probabilities. Let me explain.
Suppose you want to predict the probability that tomorrow will be sunny. The input into a neural network to make such a prediction could be today’s high temperature.
If the output is some number like 0.30, we can interpret this as a 30% change of the weather being sunny tomorrow given today’s high temperature. Pretty cool huh!
We don’t have to limit the output to between 0 and 1. For example, let’s say we have a neural network designed to predict the price of a house given the house’s area in square feet. Such a network might tell us, “given the house’s area in square feet, the predicted price of the house is $432,000.”
What happens if the neural network’s predictions are incorrect?
The neural network will adjust its weights so that the next time it makes a more accurate prediction. Recall that the weights are multiplied by the input values to make a prediction.
What is a neural network really learning?
A neural network is learning the best possible set of weights. “Best” in the context of neural networks means the weights that minimize the prediction error.
Remember, the core math operation in a neural network is multiplication, where the simplest neural network is:
Input Value * Weight = Prediction
How does the neural network find the best set of weights?
Short answer: Trial and error
Long answer: A neural network starts out with random numbers for weights. It then takes in a single input data point, makes a prediction, and then sees if its prediction was either too high or too low. The neural network then adjusts its weight(s) accordingly so that the next time it sees the same input data point, it makes a more accurate prediction.
Once the weights are adjusted, the neural network is fed the next data point, and so on. A neural network gets better and better each time it makes a prediction. It “learns” from its mistakes one data point at a time.
Do you notice something here?
Standard neural networks have no memory. They are fed an input data point, make a prediction, see how close the prediction was to reality, adjust the weights accordingly, and then move on to the next data point. At each step of the learning process of a neural network, it has no memory of the most recent prediction it made.
Standard neural networks focus on one input data point at a time. For example, in our subscriber prediction neural network we built earlier in this tutorial, if we feed our neural network number_of_free_courses_taken[1], it will have no clue what it predicted when number_of_free_courses_taken[0] was the input value.
In this tutorial, you will learn how to do histogram matchingusing OpenCV. Histogram matching (also known as histogram specification), is the transformation of an image so that its histogram matches the histogram of an image of your choice (we’ll call this image of your choice the “reference image”).
For example, consider this image below.
We want the image above to match the histogram of the reference image below.
After performing histogram matching, the output image needs to look like this:
Then, to make things interesting, we want to use this mask to mask the output image.
Below is the source code for the program that makes everything happen. Make sure you copy and paste this code into a single Python file (mine is named histogram_matching.py). Then put that file, as well as your source, reference, and mask images all in the same directory (or folder) in your computer. Once you have done that, run the code using the following command (note: mask image is optional):
#!/usr/bin/env python
'''
Welcome to the Histogram Matching Program!
Given a source image and a reference image, this program
returns a modified version of the source image that matches
the histogram of the reference image.
Image Requirements:
- Source image must be color.
- Reference image must be color.
- The sizes of the source image and reference image do not
have to be the same.
- The program supports an optional third image (mask) as
an argument.
- When the mask image is provided, it will be rescaled to
be the same size as the source image, and the resulting
matched image will be masked by the mask image.
Usage:
python histogram_matching.py <source_image> <ref_image> [<mask_image>]
'''
# Python 2/3 compatibility
from __future__ import print_function
import cv2 # Import the OpenCV library
import numpy as np # Import Numpy library
import matplotlib.pyplot as plt # Import matplotlib functionality
import sys # Enables the passing of arguments
# Project: Histogram Matching Using OpenCV
# Author: Addison Sears-Collins
# Date created: 9/27/2019
# Python version: 3.7
# Define the file name of the images
SOURCE_IMAGE = "aspens_in_fall.jpg"
REFERENCE_IMAGE = "forest_resized.jpg"
MASK_IMAGE = "mask.jpg"
OUTPUT_IMAGE = "aspens_in_fall_forest_output"
OUTPUT_MASKED_IMAGE = "aspens_in_fall_forest_output_masked.jpg"
def calculate_cdf(histogram):
"""
This method calculates the cumulative distribution function
:param array histogram: The values of the histogram
:return: normalized_cdf: The normalized cumulative distribution function
:rtype: array
"""
# Get the cumulative sum of the elements
cdf = histogram.cumsum()
# Normalize the cdf
normalized_cdf = cdf / float(cdf.max())
return normalized_cdf
def calculate_lookup(src_cdf, ref_cdf):
"""
This method creates the lookup table
:param array src_cdf: The cdf for the source image
:param array ref_cdf: The cdf for the reference image
:return: lookup_table: The lookup table
:rtype: array
"""
lookup_table = np.zeros(256)
lookup_val = 0
for src_pixel_val in range(len(src_cdf)):
lookup_val
for ref_pixel_val in range(len(ref_cdf)):
if ref_cdf[ref_pixel_val] >= src_cdf[src_pixel_val]:
lookup_val = ref_pixel_val
break
lookup_table[src_pixel_val] = lookup_val
return lookup_table
def match_histograms(src_image, ref_image):
"""
This method matches the source image histogram to the
reference signal
:param image src_image: The original source image
:param image ref_image: The reference image
:return: image_after_matching
:rtype: image (array)
"""
# Split the images into the different color channels
# b means blue, g means green and r means red
src_b, src_g, src_r = cv2.split(src_image)
ref_b, ref_g, ref_r = cv2.split(ref_image)
# Compute the b, g, and r histograms separately
# The flatten() Numpy method returns a copy of the array c
# collapsed into one dimension.
src_hist_blue, bin_0 = np.histogram(src_b.flatten(), 256, [0,256])
src_hist_green, bin_1 = np.histogram(src_g.flatten(), 256, [0,256])
src_hist_red, bin_2 = np.histogram(src_r.flatten(), 256, [0,256])
ref_hist_blue, bin_3 = np.histogram(ref_b.flatten(), 256, [0,256])
ref_hist_green, bin_4 = np.histogram(ref_g.flatten(), 256, [0,256])
ref_hist_red, bin_5 = np.histogram(ref_r.flatten(), 256, [0,256])
# Compute the normalized cdf for the source and reference image
src_cdf_blue = calculate_cdf(src_hist_blue)
src_cdf_green = calculate_cdf(src_hist_green)
src_cdf_red = calculate_cdf(src_hist_red)
ref_cdf_blue = calculate_cdf(ref_hist_blue)
ref_cdf_green = calculate_cdf(ref_hist_green)
ref_cdf_red = calculate_cdf(ref_hist_red)
# Make a separate lookup table for each color
blue_lookup_table = calculate_lookup(src_cdf_blue, ref_cdf_blue)
green_lookup_table = calculate_lookup(src_cdf_green, ref_cdf_green)
red_lookup_table = calculate_lookup(src_cdf_red, ref_cdf_red)
# Use the lookup function to transform the colors of the original
# source image
blue_after_transform = cv2.LUT(src_b, blue_lookup_table)
green_after_transform = cv2.LUT(src_g, green_lookup_table)
red_after_transform = cv2.LUT(src_r, red_lookup_table)
# Put the image back together
image_after_matching = cv2.merge([
blue_after_transform, green_after_transform, red_after_transform])
image_after_matching = cv2.convertScaleAbs(image_after_matching)
return image_after_matching
def mask_image(image, mask):
"""
This method overlays a mask on top of an image
:param image image: The color image that you want to mask
:param image mask: The mask
:return: masked_image
:rtype: image (array)
"""
# Split the colors into the different color channels
blue_color, green_color, red_color = cv2.split(image)
# Resize the mask to be the same size as the source image
resized_mask = cv2.resize(
mask, (image.shape[1], image.shape[0]), cv2.INTER_NEAREST)
# Normalize the mask
normalized_resized_mask = resized_mask / float(255)
# Scale the color values
blue_color = blue_color * normalized_resized_mask
blue_color = blue_color.astype(int)
green_color = green_color * normalized_resized_mask
green_color = green_color.astype(int)
red_color = red_color * normalized_resized_mask
red_color = red_color.astype(int)
# Put the image back together again
merged_image = cv2.merge([blue_color, green_color, red_color])
masked_image = cv2.convertScaleAbs(merged_image)
return masked_image
def main():
"""
Main method of the program.
"""
start_the_program = input("Press ENTER to perform histogram matching...")
# A flag to indicate if the mask image was provided or not by the user
mask_provided = False
# Pull system arguments
try:
image_src_name = sys.argv[1]
image_ref_name = sys.argv[2]
except:
image_src_name = SOURCE_IMAGE
image_ref_name = REFERENCE_IMAGE
try:
image_mask_name = sys.argv[3]
mask_provided = True
except:
print("\nNote: A mask was not provided.\n")
# Load the images and store them into a variable
image_src = cv2.imread(cv2.samples.findFile(image_src_name))
image_ref = cv2.imread(cv2.samples.findFile(image_ref_name))
image_mask = None
if mask_provided:
image_mask = cv2.imread(cv2.samples.findFile(image_mask_name))
# Check if the images loaded properly
if image_src is None:
print('Failed to load source image file:', image_src_name)
sys.exit(1)
elif image_ref is None:
print('Failed to load reference image file:', image_ref_name)
sys.exit(1)
else:
# Do nothing
pass
# Convert the image mask to grayscale
if mask_provided:
image_mask = cv2.cvtColor(image_mask, cv2.COLOR_BGR2GRAY)
# Calculate the matched image
output_image = match_histograms(image_src, image_ref)
# Mask the matched image
if mask_provided:
output_masked = mask_image(output_image, image_mask)
# Save the output images
cv2.imwrite(OUTPUT_IMAGE, output_image)
if mask_provided:
cv2.imwrite(OUTPUT_MASKED_IMAGE, output_masked)
## Display images, used for debugging
cv2.imshow('Source Image', image_src)
cv2.imshow('Reference Image', image_ref)
cv2.imshow('Output Image', output_image)
if mask_provided:
cv2.imshow('Mask', image_mask)
cv2.imshow('Output Image (Masked)', output_masked)
cv2.waitKey(0) # Wait for a keyboard event
if __name__ == '__main__':
print(__doc__)
main()
cv2.destroyAllWindows()
*** This tutorial is two years old and may no longer work properly. You can find an updated tutorial for object recognition at this link***
In this tutorial, we will develop a program that can recognize objects in a real-time video stream on a built-in laptop webcam using deep learning.
Object recognition involves two main
tasks:
Object Detection (Where are the objects?): Locate objects in a photo or video frame
Image Classification (What are the objects?): Predict the type of each object in a photo or video frame
Humans can do both tasks effortlessly, but computers cannot.
Computers require a lot of processing power to take full advantage of the state-of-the-art algorithms that enable object recognition in real time. However, in recent years, the technology has matured, and real-time object recognition is now possible with only a laptop computer and a webcam.
Real-time object recognition systems
are currently being used in a number of real-world applications, including the
following:
Self-driving cars: detection of pedestrians, cars, traffic lights, bicycles, motorcycles, trees, sidewalks, etc.
Sports: ball tracking in baseball, golf, and football.
Agriculture: disease detection in fruits.
Food: food identification.
There are a lot of steps in this tutorial. Have fun, be patient, and be persistent. Don’t give up! If something doesn’t work the first time around, try again. You will learn a lot more by fighting through to the end of this project. Stay relentless!
By the end of this tutorial, you will have the rock-solid confidence to detect and recognize objects in real time on your laptop’s GPU (Graphics Processing Unit) using deep learning.
We need to get all the required software set up on our computer. I will be following this really helpful tutorial.
Open an Anaconda command prompt terminal.
Type the command below to create a virtual environment named tensorflow_cpu that has Python 3.6 installed.
conda create -n tensorflow_cpu pip python=3.6
Press y and then ENTER.
A virtual environment is like an independent Python workspace which has its own set of libraries and Python version installed. For example, you might have a project that needs to run using an older version of Python, like Python 2.7. You might have another project that requires Python 3.7. You can create separate virtual environments for these projects.
Now, let’s activate the virtual environment by using this command:
conda activate tensorflow_cpu
Type the following command to install TensorFlow CPU.
You should see a message that says: “Your CPU supports instructions that this TensorFlow binary….”. Just ignore that. Your TensorFlow will still run fine.
Now run this command to complete the test of the installation:
print(sess.run(hello))
Press CTRL+Z. Then press ENTER to exit.
Type:
exit
That’s it for TensorFlow CPU. Now let’s install TensorFlow GPU.
Here is a good tutorial that walks through the installation, but I’ll outline all the steps below.
Install CUDA Toolkit v9.0
The first thing we need to do is to install the CUDA Toolkit v9.0. Go to this link.
Select your operating system. In my case, I will select Windows, x86_64, Version 10, and exe (local).
Download the Base Installer as well as all the patches. I downloaded all these files to my Desktop. It will take a while to download, so just wait while your computer downloads everything.
Open the folder where the downloads were saved to.
Double-click on the Base Installer program, the largest of the files that you downloaded from the website.
Click Yes to allow the program to make changes to your device.
Click OK to extract the files to your computer.
I saw this error window. Just click Continue.
Click Agree and Continue.
If you saw that error window earlier… “…you may not be able to run CUDA applications with this driver…,” select the Custom (Advanced) install option and click Next. Otherwise, do the Express installation and follow all the prompts.
Uncheck the Driver components, PhysX, and Visual Studio Integration options. Then click Next.
Install the NVIDIA CUDA Deep Neural Network library (cuDNN)
Now that we installed the CUDA 9.0 base installer and its four patches, we need to install the NVIDIA CUDA Deep Neural Network library (cuDNN). Official instructions for installing are on this page, but I’ll walk you through the process below.
Agree to the terms of the cuDNN Software License Agreement.
We have CUDA 9.0, so we need to click cuDNN v7.6.4 (September 27, 2019), for CUDA 9.0.
I have Windows 10, so I will download cuDNN Library for Windows 10.
In my case, the zip file downloaded to my Desktop. I will unzip that zip file now, which will create a new folder of the same name…just without the .zip part. These are your cuDNN files. We’ll come back to these in a second.
Before we get going, let’s double check what GPU we have. If you are on a Windows machine, search for the “Device Manager.”
Once you have the Device Manager open, you should see an option near the top for “Display Adapters.” Click the drop-down arrow next to that, and you should see the name of your GPU. Mine is NVIDIA GeForce GTX 1060.
If you are on Windows, you can also check what NVIDIA graphics driver you have by right-clicking on your Desktop and clicking the NVIDIA Control Panel. My version is 430.86. This version fits the requirements for cuDNN.
Ok, now that we have verified that our system meets the requirements, lets navigate to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0, your CUDA Toolkit directory.
Now go to your cuDNN files, that new folder that was created when you did the unzipping. Inside that folder, you should see a folder named cuda. Click on it.
Click bin.
Copy cudnn64_7.dll to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin. Your computer might ask you to allow Administrative Privileges. Just click Continue when you see that prompt.
Now go back to your cuDNN files. Inside the cuda folder, click on include. You should see a file named cudnn.h.
Copy that file to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\include. Your computer might ask you to allow Administrative Privileges. Just click Continue when you see that prompt.
Now go back to your cuDNN files. Inside the cuda folder, click on lib -> x64. You should see a file named cudnn.lib.
Copy that file to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\lib\x64. Your computer might ask you to allow Administrative Privileges. Just click Continue when you see that prompt.
If you are using Windows, do a search on your computer for Environment Variables. An option should pop up to allow you to edit the Environment Variables on your computer.
Click on Environment Variables.
Make sure you CUDA_PATH variable is set to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0.
Now let’s test the installation. Launch the Python interpreter.
python
Type this command.
import tensorflow as tf
If you don’t see an error, TensorFlow GPU is successfully installed.
Now type this:
hello = tf.constant('Hello, TensorFlow!')
And run this command. It might take a few minutes to run, so just wait until it finishes:
sess = tf.Session()
Now type this command to complete the test of the installation:
print(sess.run(hello))
You can further confirm whether TensorFlow can access the GPU, by typing the following into the Python interpreter (just copy and paste into the terminal window while the Python interpreter is running).
Now that we have everything setup, let’s install some useful libraries. I will show you the steps for doing this in my TensorFlow GPU virtual environment, but the steps are the same for the TensorFlow CPU virtual environment.
Open a new Anaconda terminal window. Let’s take a look at the list of virtual environments that we can activate.
conda env list
I’m going to activate the TensorFlow GPU virtual environment.
Once that is finished, you need to create a folder somewhere that has the TensorFlow Models (e.g. C:\Users\addis\Documents\TensorFlow). If you have a D drive, you can also save it there as well.
In your Anaconda terminal window, move to the TensorFlow directory you just created. You will use the cd command to change to that directory. For example:
Download the latest *-win32.zip release (assuming you are on a Windows machine).
Create a folder in C:\Program Files named it Google Protobuf.
Extract the contents of the downloaded *-win32.zip, inside C:\Program Files\Google Protobuf
Search for Environment Variables on your system. A window should pop up that says System Properties.
Click Environment Variables.
Go down to the Path variable and click Edit.
Click New.
Add C:\Program Files\Google Protobuf\bin
You can also add it the Path System variable.
Click OK a few times to close out all the windows.
Open a new Anaconda terminal window.
I’m going to activate the TensorFlow GPU virtual environment.
conda activate tensorflow_gpu
cd into your \TensorFlow\models\research\ directory and run the following command:
for /f %i in ('dir /b object_detection\protos\*.proto') do protoc object_detection\protos\%i --python_out=.
Now go back to the Environment Variables on your system. Create a New Environment Variable named PYTHONPATH (if you don’t have one already). Replace C:\Python27amd64 if you don’t have Python installed there. Also, replace <your_path> with the path to your TensorFlow folder.
Note: This section gets really technical. If you know the basics of computer vision and deep learning, it will make sense. Otherwise, it will not. You can skip this section and head straight to the Implementation section if you are not interested in what is going on under the hood of the object recognition application we are developing.
In this project, we use OpenCV and TensorFlow to create a system capable of automatically recognizing objects in a webcam. Each detected object is outlined with a bounding box labeled with the predicted object type as well as a detection score.
The detection score is the probability that a bounding box contains the object of a particular type (e.g. the confidence a model has that an object identified as a “backpack” is actually a backpack).
The particular SSD with Inception v2
model used in this project is the ssd_inception_v2_coco model. The ssd_inception_v2_coco model uses
the Single Shot MultiBox Detector (SSD) for its architecture and the Inception
v2 framework for feature extraction.
Single Shot MultiBox Detector (SSD)
Most state-of-the-art object detection methods involve the
following stages:
Hypothesize
bounding boxes
Resample
pixels or features for each box
Apply
a classifier
The Single Shot MultiBox Detector (SSD) eliminates the multi-stage process above and performs all object detection computations using just a single deep neural network.
Inception v2
Most state-of-the-art object detection
methods based on convolutional neural networks at the time of the invention of
Inception v2 added increasingly more convolution layers or neurons per layer in
order to achieve greater accuracy. The problem with this approach is that it is
computationally expensive and prone to overfitting. The Inception v2
architecture (as well as the Inception v3 architecture) was proposed in order
to address these shortcomings.
Rather than stacking multiple kernel
filter sizes sequentially within a convolutional neural network, the approach
of the inception-based model is to perform a convolution on an input with
multiple kernels all operating at the same layer of the network. By factorizing
convolutions and using aggressive regularization, the authors were able to
improve computational efficiency. Inception v2 factorizes the traditional 7 x 7
convolution into 3 x 3 convolutions.
Szegedy, Vanhoucke, Ioffe, Shlens, & Wojna, (2015) conducted an empirically-based demonstration in their landmark Inception v2 paper, which showed that factorizing convolutions and using aggressive dimensionality reduction can substantially lower computational cost while maintaining accuracy.
Data Set
The ssd_inception_v2_coco model used in this project is pretrained on the Common Objects in Context (COCO) data set (COCO data set), a large-scale data set that contains 1.5 million object instances and more than 200,000 labeled images. The COCO data required 70,000 crowd worker hours to gather, annotate, and organize images of objects in natural environments.
Software Dependencies
The
following libraries form the object recognition backbone of the application
implemented in this project:
OpenCV, a library of programming functions for computer vision.
Pillow, a library for manipulating images.
Numpy, a library for scientific computing.
Matplotlib, a library for creating graphs and visualizations.
TensorFlow Object Detection API, an open source framework developed by Google that enables the development, training, and deployment of pre-trained object detection models.
Now to the fun part, we will now recognize objects using our computer webcam.
Copy the following program, and save it to your TensorFlow\models\research\object_detection directory as object_detection_test.py .
# Import all the key libraries
import numpy as np
import os
import six.moves.urllib as urllib
import sys
import tarfile
import tensorflow as tf
import zipfile
import cv2
from collections import defaultdict
from io import StringIO
from matplotlib import pyplot as plt
from PIL import Image
from utils import label_map_util
from utils import visualization_utils as vis_util
# Define the video stream
cap = cv2.VideoCapture(0)
# Which model are we downloading?
# The models are listed here: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md
MODEL_NAME = 'ssd_inception_v2_coco_2018_01_28'
MODEL_FILE = MODEL_NAME + '.tar.gz'
DOWNLOAD_BASE = 'http://download.tensorflow.org/models/object_detection/'
# Path to the frozen detection graph.
# This is the actual model that is used for the object detection.
PATH_TO_CKPT = MODEL_NAME + '/frozen_inference_graph.pb'
# List of the strings that is used to add the correct label for each box.
PATH_TO_LABELS = os.path.join('data', 'mscoco_label_map.pbtxt')
# Number of classes to detect
NUM_CLASSES = 90
# Download Model
opener = urllib.request.URLopener()
opener.retrieve(DOWNLOAD_BASE + MODEL_FILE, MODEL_FILE)
tar_file = tarfile.open(MODEL_FILE)
for file in tar_file.getmembers():
file_name = os.path.basename(file.name)
if 'frozen_inference_graph.pb' in file_name:
tar_file.extract(file, os.getcwd())
# Load a (frozen) Tensorflow model into memory.
detection_graph = tf.Graph()
with detection_graph.as_default():
od_graph_def = tf.GraphDef()
with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
serialized_graph = fid.read()
od_graph_def.ParseFromString(serialized_graph)
tf.import_graph_def(od_graph_def, name='')
# Loading label map
# Label maps map indices to category names, so that when our convolution network
# predicts `5`, we know that this corresponds to `airplane`. Here we use internal
# utility functions, but anything that returns a dictionary mapping integers to
# appropriate string labels would be fine
label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
categories = label_map_util.convert_label_map_to_categories(
label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
category_index = label_map_util.create_category_index(categories)
# Helper code
def load_image_into_numpy_array(image):
(im_width, im_height) = image.size
return np.array(image.getdata()).reshape(
(im_height, im_width, 3)).astype(np.uint8)
# Detection
with detection_graph.as_default():
with tf.Session(graph=detection_graph) as sess:
while True:
# Read frame from camera
ret, image_np = cap.read()
# Expand dimensions since the model expects images to have shape: [1, None, None, 3]
image_np_expanded = np.expand_dims(image_np, axis=0)
# Extract image tensor
image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')
# Extract detection boxes
boxes = detection_graph.get_tensor_by_name('detection_boxes:0')
# Extract detection scores
scores = detection_graph.get_tensor_by_name('detection_scores:0')
# Extract detection classes
classes = detection_graph.get_tensor_by_name('detection_classes:0')
# Extract number of detectionsd
num_detections = detection_graph.get_tensor_by_name(
'num_detections:0')
# Actual detection.
(boxes, scores, classes, num_detections) = sess.run(
[boxes, scores, classes, num_detections],
feed_dict={image_tensor: image_np_expanded})
# Visualization of the results of a detection.
vis_util.visualize_boxes_and_labels_on_image_array(
image_np,
np.squeeze(boxes),
np.squeeze(classes).astype(np.int32),
np.squeeze(scores),
category_index,
use_normalized_coordinates=True,
line_thickness=8)
# Display output
cv2.imshow('object detection', cv2.resize(image_np, (800, 600)))
if cv2.waitKey(25) & 0xFF == ord('q'):
cv2.destroyAllWindows()
break
print("We are finished! That was fun!")
Open a new terminal window.
Activate the TensorFlow GPU virtual environment.
conda activate tensorflow_gpu
cd into your TensorFlow\models\research\object_detection directory.
At the time of this writing, we need to use Numpy version 1.16.4. Type the following command to see what version of Numpy you have on your system.
pip show numpy
If it is not 1.16.4, execute the following commands:
pip uninstall numpy
pip install numpy==1.16.4
Now run, your program:
python object_detection_test.py
In about 30 to 90 seconds, you should see your webcam power up and object recognition take action. That’s it! Congratulations for making it to the end of this tutorial!
