RAD Workshop

Objective

The objective of this workshop is to explore the capabilities of a Raspberry Pi, utilize computer vision to detect objects by color, and integrate servo motor control with the color detection output.

Overview

Raspberry Pi

A Raspberry Pi is a single-board, small computer that is used for electronic projects or as a component in larger systems (Figure 1). The default operating system is Raspberry Pi OS, a Debian-based operating system. A Raspberry Pi can also operate using Linux distros or Android. Internally, a Raspberry Pi has a central processing unit (CPU), graphics processing unit (GPU), and random-access memory (RAM) and can perform more advanced tasks, such as character recognition, object recognition, and machine learning, compared to Arduino.

Figure 1: Raspberry Pi 4

The base hardware of a Raspberry Pi 4 features USB 2.0/3.0 ports for peripherals, a wireless and ethernet port, Bluetooth, USB-C power, general-purpose input/output (GPIO) pins (Figure 2), an audio jack, two micro HDMI ports, and a camera module port.

Figure 2: Raspberry Pi GPIO Pin Courtesy of Raspberry Pi Foundation

A Raspberry Pi has 28 GPIO pins, two 5V pins, two 3.3V pins, and eight ground pins. The 28 GPIO pins are analogous to Arduino digital/analog I/O pins and allow connections to sensors and additional components, such as an Arduino board.

The data storage for a Raspberry Pi uses an SD card. Many programming languages can be used on the Raspberry Pi, including Python, Scratch, Java, and C/C++. The Python programming interface is the default for a Raspberry Pi. Thonny IDE is the environment for Raspberry Pi when using Python.

To further the capabilities of a Raspberry Pi, additional modules, such as a camera module, microcontrollers, and programming libraries, such as the Open Source Computer Vision Library (OpenCV), can be implemented.

Python

Python is a text-based programming language typically used for web development, software development, mathematics, and system scripting. Python is functional across several platforms and has syntax similar to the English language, which minimizes the amount of code needed to perform certain tasks compared to other languages.

Syntax

Unlike other programming languages, Python does not require a closing mark on lines of code. Lines of code are closed off by continuing onto the next line.

In Figure 3, the program will print “Hello World”, make a variable, x, equal 3, make a variable, y, equal 2, add the two variables together, and store the result in a variable named result, and print the result.

Figure 3: Python Syntax

Loops

For any loop made (if/elif/else statements, while/for loops), the lines following the initial declaration of the loop must be indented (Figure 4). The lack of indentation will cause a syntax error. In Figure 4, the program will determine if 5 is greater than 2, and if true, print “Five is greater than two!”.

Figure 4: If Statement

While Loops

A while loop executes the code within the loop so long as the condition is true. In Figure 5, the while loop will continuously print the result on a new line and add 1 to the result so long as the result is less than or equal to 10.

Figure 5: While Loop

A break statement is used to break a loop regardless of the condition (Figure 6).

Figure 6: Break Statement

For Loops

A for loop is used for iterating over a set sequence. The sequence can be a list, tuple, dictionary, set, or string. In Figure 7, the for loop will print each character in the string random_word with a space in between each character. Similar to the while loop, a for loop can be broken early with a break statement.

Figure 7: For Loop

Comments

Comments within code are typically used to explain the code and make the code more readable and easier to troubleshoot when errors arise. Line comments begin with a # and can only span a single line. Block comments begin and end with a ’’’ and can span multiple lines (Figure 8).

Figure 8: Comments

Computer Vision (CV)

Computer vision is an area of artificial intelligence (AI) that allows computers and systems to measure data from visual inputs and produce outputs based on that data. The OpenCV library is an extensive open source library used for computer vision. OpenCV contains interfaces for multiple programming languages.

OpenCV allows image/video processing and display, object detection, geometry-based monocular or stereo computer vision, computational photography, machine learning and clustering, and CUDA acceleration. With this functionality, many applications are possible including face recognition, street view image stitching, and driverless car navigation.

In OpenCV, color images are represented as three-dimensional arrays. An image consists of rows of pixels, and each pixel is represented by an array of values representing its color in BGR (Blue, Green, Red) format. The array can be transcribed into the color’s specific hue saturation value (HSV). The HSV model describes colors similar to how the human eye tends to perceive color. This format is more suitable for color detection purposes.

cv2.cvtColor()

cv2.cvtColor(frame, conversion code) is a method used to convert an image from one color space to another. In the OpenCV Library, there are over 150 different color-space conversion codes. The one used in this lab is cv2.COLOR_BGR2HSV, which converts an image from the default BGR format to the desired HSV format.

cv2.inRange()

cv2.inRange(frame, lower bound, upper bound) is a method used to identify pixels in the frame that fall within the range of values defined by the lower and upper boundaries. This command outputs a binary mask where white pixels represent areas that fall within the range, and black pixels represent areas that do not fall within the range. The right panel in Figure 9 shows the output for the cv2.inRange() method using the command orange_mask = cv2.inRange(nemo,lower,upper). The bright orange pixels fall within the range specified and have a corresponding white value in the output.

Figure 9: cv2.inRange() Method Courtesy of Real Python

cv2.bitwise_and()

cv2.bitwise_and(frame, frame, mask = mask) is a bitwise AND operation that delegates in making the frame only retain what is determined by the set mask. Only pixels in the frame with a corresponding white value in the mask would be preserved. The rightmost panel in Figure 10 shows the output of the command cv2.bitwise_and(nemo, nemo, mask=orange_mask).

Figure 10: Input, Mask, and Bitwise-And Output Courtesy of Real Python

cv2.bitwise_or()

cv2.bitwise_or(frame1, frame2) is a bitwise OR operation that delegates in combining multiple frames so that the output retains non-zero pixels from either frame.

cv2.imshow()

cv2.imshow(window name, image) is used to display a given image in a window. The name of the window is given by the first parameter in the command. The window automatically displays in the same size as the image.

NumPy

Numerical Python (NumPy) is a Python library used primarily for working with arrays. The library also contains functions for mathematical purposes, such as linear algebra, fourier transformation, and matrices. In this lab, the NumPy command used is np.array.

np.array()

np.array() creates an array object of the class N-dimensional NumPy array, or numpy.ndarray. A list, tuple, or a number can be entered in the parentheses to turn the data type into an array.

Raspberry Pi Camera

The Raspberry Pi Camera Module is a camera that can be used to capture high resolution and high definition images and video. The module’s library features commands to use the camera within code and modify how the camera outputs the frame rate, resolution, and timing of the image or video it is capturing. The camera module’s coding library (picamera) is automatically installed onto a Raspberry Pi once the camera is enabled and the Raspberry Pi is rebooted.

PiCamera()

PiCamera() is the function used to call the PiCamera within the code. Typically, PiCamera() is assigned to a variable to make calling the camera throughout the code easier.

PiRGBArray()

PiRGBArray(camera, options) produces a three-dimensional array from a BGR capture. After declaring the camera, the Options parameter can be used to set the size of the image captured.

camera.resolution

camera.resolution = (width, height) sets a specific resolution for the image captured by the module. The default image resolution is set to the resolution of the monitor being used. The maximum resolution for still photos is 2592x1944 pixels and 1920x1080 pixels for video recording. The minimum resolution is 64x64 pixels.

camera.framerate

camera.framerate = frame rate sets a specific frame rate or the speed at which the image is shown. The default frame rate and maximum frame rate are dependent on the resolution at which the video or image is being captured.

camera.capture()

camera.capture() enables the camera module to capture an image. The output file name, format type, video port usage, resizing, splitter port, bayer, and other parameters can be set.

camera.capture_continuous() captures images continuously until a defined end. For this lab, the format of the output is set to bgr, which means the image data is written to a file in BGR color space format. The output is a file-like object and each image is written to this object sequentially. The output must be cleared in preparation for the next image in the stream.

Raspberry Pi Setup

Before using a Raspberry Pi, the Raspberry Pi must be prepared with a formatted SD card with Raspberry Pi OS and the correct programming library packages installed. For this lab, these steps have been done. A more detailed explanation of the Raspberry Pi Setup can be found in the Appendix.

Servo Motor Setup

Continuous vs. Positional Servo Motor

A servo motor is a compact actuation device with high power output and energy efficiency. There are two kinds of servo motors used for common applications - continuous rotation and positional rotation servos. A continuous servo motor has a shaft that spins continuously allowing definite control over its speed and direction. A positional servo motor can only turn over a range of 180 degrees (90 degrees in each direction), allowing precise control over its position (Figure 11).

Figure 11: Servo Comparison Diagram

Programming Servo Motors (Raspberry Pi)

The following commands are used to program a continuous rotation servo motor once it’s wired to the GPIO pins on the Raspberry Pi:

GPIO.setmode(mode)

GPIO.setmode(mode) is used to dictate how the GPIO pins on the Pi are referenced in the program. Following are the two options for GPIO mode:

• GPIO.BOARD - Refers to the pins by the physical position of the pin on the board
• GPIO.BCM - Refers to the pins by the Broadcom SOC channel number, which is an intrinsically defined system that differs for various versions of the Pi (refer to Raspberry Pi GPIO Pinout)

In this lab, we will be referencing the GPIO pins with the GPIO.BCM method.

GPIO.setup(pin, mode)

GPIO.setup(servoPin, mode) sets the servoPin to either an input or output. The corresponding options for the mode are GPIO.IN and GPIO.OUT

GPIO.PWM(servoPin, frequency)

PWM stands for Pulse Width Modulation. Servos use a pulse-width modulated input signal that defines the state of rotation or position of the servo shaft. GPIO.PWM(servoPin, frequency) initializes PWM to a servo pin and a certain constant frequency.

pwm.start(duty cycle)

pwm.start(duty cycle) begins servo rotation with a specified duty cycle value, which is a measure of the speed and direction of rotation of the shaft. For the continuous rotation servos used in this lab, the shaft rotates:

• Clockwise for a duty cycle value of less than 14
• Counterclockwise for a duty cycle value of 14 or greater

pwm.stop()

pwm.stop() ceases servo motor rotation entirely.

GPIO.cleanup()

GPIO.cleanup() resets the ports and pins to the default state.

Materials and Equipment

• Raspberry Pi with SD card
• Raspberry Pi Camera Module
• Monitor with micro HDMI to HDMI cable
• USB keyboard
• USB mouse
• Power supply

Procedure

Ensure that the Raspberry Pi is connected to a power supply and a monitor. Verify that the camera module and the USB mouse and keyboard are plugged into their corresponding ports on the Raspberry Pi.

Setting up the Raspberry Pi

Before getting started with this procedure, ensure that an SD card with Raspberry Pi OS is inserted into the designated slot on the Raspberry Pi.

1. Connect a USB mouse and keyboard to the Pi.
2. Using a micro HDMI cable, connect the board to a monitor.
3. If required, connect an audio output device to the board using the audio jack.
4. Connect the board to a power supply using the Raspberry Pi charger.

Raspberry Pi will boot up and the monitor will display the desktop. This concludes the Pi setup.

Setting up the Camera Module

The following steps outline the procedure for connecting and enabling the camera module.

1. Locate the Camera Module port, as shown in Figure 20. Ensure the Raspberry Pi is powered off.

   

   Figure 20: Camera Module

2. Gently pull up on the edges of the port’s plastic clip, as illustrated in Figure 21. Insert the camera module ribbon cable into the port, ensuring that the connectors at the bottom of the cable are facing the contacts in the port.

   

   Figure 21: Camera Module Port

3. Push the plastic clip back into place.
4. Start Raspberry Pi and navigate to Main Menu > Preferences > Raspberry Pi Configuration . The main menu can be accessed from the Pi icon on the taskbar.
5. Select the Interfaces tab and enable all interfaces, as shown in Figure 22.

   

   Figure 22: Camera Module Enabled

6. Reboot the Raspberry Pi. The camera module should be ready for use.

Part 1: Testing the camera module

In this part, capture a still image using the camera module to verify that the camera has been connected and enabled correctly. While plugging the camera module, ensure the Raspberry Pi is powered off.

1. Open a terminal window by clicking the black monitor icon on the taskbar (Figure 12).

   

   Figure 12: Taskbar

2. To take a still picture, type in the following command raspistill -o testpic.jpg.

The camera should take a still picture and save it as testpic.jpg on the Raspberry Pi. A preview should be visible on the screen for a few seconds. If this command returns any errors, please notify a lab TA.

Part 2: Wiring the Servo Motor

1. Wire the servo motor according to the following diagram. Make sure to wire the servo pin to GPIO 17 on the Raspberry Pi, as shown in Figure 13

Figure 13: Servo Motor Connection to Raspberry Pi - Courtesy of Raspberry Pi Projects

Part 3: Color Detection using Pi Camera and Python-OpenCV

In this part, a Python script that isolates red and green colors from a live video captured by the camera module will be developed and this output will be used to create a prototype of a traffic light detection system.

1. Download the Python script for color detection using OpenCV. Open the camera module with the Thonny Python Editor installed on the Raspberry Pi.
2. There are missing components in the program. Complete the code following the instructions outlined in the next steps.
3. The code is divided into four sections. Figure 14 shows the workflow for this lab.
   1. Section 1 includes commands for importing all necessary packages and setting up the GPIO pins on the Pi.
   2. Section 2 sets up the camera module so it captures a continuous video.
   3. Section 3 contains code for color detection and isolation for red and green colors.
   4. Section 4 integrates the servo motor with the color detection output such that the servo rotates clockwise when red is detected, counterclockwise when green is detected, and stops rotating when neither red nor green is detected.

      

      Figure 14: Lab Workflow

4. Type the code in Figure 15 to import the required packages into the Python module.

   

   Figure 15: Importing Python Packages

   1. OpenCV represents images as NumPy arrays. Lines 3-5 import OpenCV, NumPy, and PiCamera libraries. The images obtained from the camera module must be in an array format so Line 6 is included.Line 7 imports the Raspberry Pi GPIO library.
   2. Assign a variable for the servo pin. Based on the wiring diagram, the servo should be wired to pin 17 on the Raspberry Pi.
   3. Set GPIO mode to GPIO.BCM
   4. Set the mode for the servo pin as output. This completes section 1 of the code.
5. Section 2 of the code sets up the camera module. The while(True): condition creates an infinite loop so that the code re-runs automatically to perform color detection on a set of objects. In the body of the while loop, type out a command for each of the following tasks using the background information of the camera module commands as reference.
   1. Initialize the camera object by setting a variable camera equal to PiCamera()
   2. Set camera resolution to (640,480)
   3. Set camera frame rate to 30
   4. Get the code approved by a TA and move to the next step.
6. Once the camera setup is done, the next step is to capture a continuous video with the camera. This is achieved by using the camera.capture_continuous(..) command with arguments as shown on Line 14 in the figure below. Line 15 converts every frame captured by the camera module to a NumPy array, ready to be manipulated with OpenCV to perform color detection. This concludes Section 2 of the code.

   

   Figure 16: Capturing Continuous Video and Converting BGR to HSV

7. Section 3 will perform color detection for red and green colors. Since it is easier to identify these colors with their HSV, each frame is converted from the default BGR format to the HSV format. Line 17 in Figure 16 performs this task using the cv2.cvtColor method. The newly created HSV frame (stored as variable hsv) will now be used for color detection.
8. The segment of code in Figure 17 will isolate red from the video. Lines 20-21 define the lower and upper boundaries of the range of HSV values within which the color would be classified as red. These values might have to be adjusted based on lighting and exposure conditions.

   

   Figure 17: Color Detection for Red

   1. Line 22, identifies all the pixels in the video that fall within the specified range by defining a variable red_mask, and using the cv2.InRange method using red_lower and red_upper as boundaries. This line outputs a binary mask where white pixels represent areas that fall within the range, and black pixels represent areas that do not.
   2. Line 22 applies the mask over the video to isolate red using the cv2.bitwise_and method taking frame as the input, and outputting frame using red_mask as the mask. This line outputs the frame so that only pixels that have a corresponding white value in the mask are shown.
9. Insert code for isolating green from the video. Use the lower boundary values [20,40,60] and upper boundary values [102, 255, 255] to define the range.
10. To isolate both red and green from the video at once, combine result_red and result_green. Think about how this could be achieved and type out the correct command in Line 32 (Figure 18). Which bitwise operation should be used to show pixels where either red or green are present?

    

    Figure 18: Result Statements

11. To display the final result (Line 34), type a line of code that creates a new window named "Red and Green Detection in Real-Time" and displays the final output. Use the cv2.imshow method.
    1. Note that Line 36 is required because it stops rawCapture to clear the stream in preparation for the next frame.
    2. This concludes Section 3 of the program. Get the code approved by a TA and move to Section 4.
12. Lines 45-61 represent the final section of the program. The objective of this section is to integrate the color detection output with servo rotation. The servo would rotate clockwise when a red object is detected, counterclockwise when a green object is detected, and would stop rotation when neither red nor green is detected.

    

    Figure 19: Traffic Light Detection System

    1. Lines 46-47 dictate that every time the key ‘q’ is pressed, the camera closes and stops capturing new frames. The final frame is checked for the presence of red and green using if statements (Lines 48-58).
    2. The following conditions are implemented for detecting red color (line 48)- the frame should have more red pixels than green, and the number of red pixels should be at least 25% of the total number of pixels. The cv2.countNonZero method is used on red_mask to obtain the number of red pixels.
    3. On line 50, insert a command to start servo rotation with a 5% duty cycle. This corresponds to clockwise rotation.
    4. Insert the following conditions for detecting green color (line 52)- the frame should have more green pixels than red, and the number of green pixels should be at least 25% of the total number of pixels.
    5. On line 54, insert a command to start servo rotation with a 55% duty cycle. This corresponds to counterclockwise rotation.
    6. On line 58, insert a command to stop servo rotation.
13. Run the Python script and place the red and green objects provided by the TAs in front of the camera module. Observe how the specified colors are isolated from the video output. Press the q key on the keyboard to pause the video at a specific frame and observe the output print statements. Figure 20 displays what the final output would look like when a red object is held up to the camera module.

    

    Figure 20: Final Output

This concludes the procedure for this workshop.

Assignment

Individual Lab Report

There is no lab report for Lab 4.

Team PowerPoint Presentation

There is no team presentation for Lab 4.

References

Raspberry Pi Foundation. “Getting Started with Raspberry Pi.” Raspberry Pi Foundation. Raspberry Pi Foundation, (n.d.). Retrieved 5 August 2021. https://projects.raspberrypi.org/en/projects/raspberry-pi-getting-started

Raspberry Pi Foundation. “Usage - Raspberry Pi Documentation.” Raspberry Pi Foundation. Raspberry Pi Foundation, (n.d.). Retrieved 5 August 2021. https://www.raspberrypi.org/documentation/usage/

IBM Corp. “What is Computer Vision.” IBM. International Business Machines Corporation, (n.d.). Retrieved 5 August 2021 https://www.ibm.com/topics/computer-vision

Real Python. (2020, November 7). Image segmentation using color spaces in OpenCV and Python. Real Python. https://realpython.com/python-opencv-color-spaces/. Retrieved 9 August 2021

Appendix

Installing Raspberry Pi OS on an SD card

This section outlines the procedure for installing an operating system on the Raspberry Pi using the Raspberry Pi Imager. A formatted SD card and a computer with an SD card reader are required to follow this procedure.

1. Download the latest version of Raspberry Pi Imager and install it on a computer.
2. Connect an SD card to the computer.
   1. Before this step, the SD card must be formatted to FAT32.
3. Open Raspberry Pi Imager and choose the operating system. For most general uses of a Raspberry Pi, Raspberry Pi OS (32-bit) should suffice.
4. Click on Storage and choose the SD card to write the image to.
5. Click on Write to install the operating system image on the SD card.

Installing OpenCV & Other Packages

This section outlines the procedure for installing the Python packages used in this lab.

• OpenCV
• NumPy
• PiCamera with [array] submodule

1. Open a terminal window by clicking the black monitor icon on the taskbar (Figure 23).

   

   Figure 23: Taskbar

2. Before installing the required packages, install PIP, update the system, and install Python 3 using the commands PIP for python3 - sudo apt-get install python 3-pip, sudo apt-get update && sudo apt-get upgrade, and sudo apt-get install python3.
3. To install OpenCV, NumPy, and PiCamera packages, type out the following commands sudo pip3 install opencv-python, sudo pip3 install numpy, and sudo pip3 install picamera[array].
4. Optionally, to utilize the GPIO pins on the Raspberry Pi, install the GPIO library using the following command sudo pip-3.2 install RPi.GPIO.