Ball Tracking Robot
The ball tracking robot utilizes a Raspberry Pi, camera, and open cv to identify and track a red ball. It also uses a YOLO (You Only Look Once) algorithim to identify animals and objects that enter its field of view. The robot has potential applications as a wildlife camera or other security and tracking device.
| Engineer | School | Area of Interest | Grade |
|---|---|---|---|
| Charlotte H | Culver Academies | Mechanical Engineering and Renewable Energy | Incoming Junior |
Final Milestone
My final milestone included adding a ton of modifications to my robot, including:
- Image recognition software
- Servo as a camera mount
- Ultrasonic distance sensors to prevent crashes
- Some smaller, sillier modifications
Image Recognition Software
I used pre-existing code (see link below) to run image recognition software on my Raspberry Pi. I used two different code for this to get different results. The first runs a live video feed in a browers that automatically identifies and displays what objects or animals are in frame. I added code so that when a certain thing is identified (I set it to humans but other animals/objects can be used) that frame is saved as a seperate picture. This also required using a timestamp as part of the file name and regularly calling upon it in the code so it would update. Someone could switch this out for a variable that regularly changes. This is very convenient but the feed is very slow. The second also runs a live feed from the camera in a browers but has buttons to manually take pictures, have those pictures analysed, then stored. This is less convenient but the feed is much faster. While using this I noticed that if you take multiple pictures before having it analyze the first one the newest picture will replace the previous ones (it does not save multiple unanalysed pictures). Both code utilize YOLO (You Only Look Once) algorithm for image detection. I am hoping that by ultilizing these image recognition capabilities that robot can serve as a wildlife camera.
Servo
I mounted the Raspberry Pi camera onto a servo (see link to item below). Servos work differently from the motors for the wheels as they move to a specific position rather than just spinning until stopped. You can adjust the speeed or torque of a servo but I did not for this project. The particular servo here only uses C code and as a result the coding process was a lot more complicated. I first had to learn and run the commands in C (which required a special virtual environment) then combine all the relevnt files into one, make that file a shared filed (so it can be called upon in python), then import the file and use specific python commands so the servo would respond. Once that was done it was fairly easy to adjust certain variables for the camera to easily track the ball. In my code I just used servo degree increase or decrease functions and had certain intervals where it would change the degree by more or less. One could also have the Rapsberry Pi calculate how much the servo angle needs to change based on how offset the ball is (using basic math and a proportionality constant). I had some issues with the camera ribbon getting caught on the bottom servo and am planning on flipping the camera upside down once I have a longer camera ribbon (and adding code for the camera to flip images verticially so images still apear normall).
Ultrasonic Distance Sensors
My final major modification was adding distance sensors. I noticed during Milestone 2 that the robot would slowly shift backwards and forwards as it tracked the ball. This would cause it to sometimes run into object and I wanted to prevent this from happening. I only used two distance sensors total, one in the front and one in the back because I was just trying to stop it from running into things, not accurately pin point distance. Ultrasonic distance sensors work by sending out sound waves which are then reflected of an object back to the sensor which measures the amount of time that takes to calculate distance. If you wanted to determine if an object is left or right, or dtermine how far it is with accuracy, more distance sensors in a variety of posistions would be needed. When coding the sensors I adjusted the threshold distance to be very small (I did not adjust the max distance because that was unnecessary) and created a sequence for the robot to follow when something was in this threshold distance (essentially just move forward or backward).
Miscellaneous Modifications
In addition to the larger modifcatiosn mentioned above, I also added a few smaller or sillier modifications. The first of these was adding headlights. They are connected directly to power and always remain on when the Raspberry Pi is on. These do actually have a function as the help to illuminate the ball from below. Earlier the camera had some difficulty recognizing the ball when it was directly above it and blocking overhead lights. Adding lights under makes it easier for the camera to recognize the red color. Another modifcation was adding red “backing” lights that only turn on when the robot is moving backwards. These are accompanied by an activate buzzer that is set to turn on and off (beeping) while backing. For these last two modifcations I had them only activate by adding them into the MotorTest.py code (see below) and they can be easily removed. I also made a number of modifications that made mounting the different components easier. The first was cutting pieces of cardboard to fit the robot and give the components a flat surface to rest. The robot has small plastic pieces that stick up and previously prevented me from placing the Raspberry Pi in the middle of the robot. I also cut a number of small cardboard rectangles to create a tower for the servo to sit. This was necessary as I wanted both the front distance sensor and camera to be in the center.
Final Thoughts
I really appreciated this project and all the fun things I got to add. There are still other modifications I am looking to make, like the camera flip mentioned above and swapping in different wheels. There is also currently an issue with how the camera is positioned. It struggles to look down and because I raised the camera in the final design that problem is even worse. It can still track the ball on the ground without issue but it does significantly better with objects in teh air. If there is a particular modifcation you do not what to make, simply remove it from the code.
Second Milestone
My Milestone 2 was having the robot track the ball as it moved. This included a number of components including:
- Setting up the Raspberry Pi camer, taking a picture and setting up a live feed
- Mounting all of the components onto the robot chassis
- Learning and setting up object tracking
Setting up Raspberry Pi Camera, Taking a Picture, and Live Feed:
The first step to completing this milestone was setting up the Raspberry Pi camera and installing the necessary packages to use it (like cv2). After this I moved on to taking a picture with the camera (see code below). After that, I set up a live feed that streamed from the camera into a browser. This gives you live updates and tracking (see code below).
Mounting:
Once every component was working I mounted it on the robot chassis. This is temporary, as it is currently mounted with pipe cleaners which are not strong and a potential fire hazard. I will be remounting the components later with double-sided tape. But, the components have stayed secure and mounting allows the robot to move and turn freely.
Object Tracking:
In order to track the red ball (or another object) the Raspberry Pi needs to follow a number of steps to first identify where the object is. These steps, or morphological transformations, include maksing, erosion, dilation, contouring and centriods (in that order). Masking, the first step, involves telling the Raspberry Pi what to focus on in the image. In this case that was the red ball. So, the mask I created focused on anything that was the color red. In the HSV spectrum that involved any colors with a hue value between 0-10 and 160-179. Because red is the start of the color wheel (0) it is also the end (180) and you therefore need multiple color arrays to include both of these ranges. The final image below was produced after applying this mask. The parts that were identified as red are in white and all other colors in black. The white is the area that will be detected and the black is the area that is hidden or ignored.
After the mask is appied the area that the Raspberry Pi needs to focus on is much smaller and erosion and dilation can be applied to further hone in on the object. Both erosion and dilation work by analyzing the image in small sections to identify which pixels are a part of the object or a part of the background. Then, the pixels that surround part of the outline or boundary of the object are either turned into the background or object (for either an erosion or dilation). This process will enlarge or reduce the size of the object depending on which transformation is applied. Erosions are useful for removing blobs, disconnecting objects, and remove other pizels that may get accidentally picked up. Dilations are useful for increasing the size of an object, joining broken pieces, and ensuring that the ball is recognized despite imperfections (that may cause parts to be removed). The impact of the erosion or dilation can be changed by changing the kernel. By changing the kernel to 5x5 instead of the default size. This means that the image will be analyzed in 5x5 pixel sections and if any pixel in that section is part of the object, all the pixels in that square will be chnaged (depending on whether it is an erosion or dilation). The second image (also in black and white) shows how erosion and dilation effects the quality of the mask, making it more precise and focused on the ball.
After maksing, eroding and dilating the image contouring can be applied to detect where the object is. The previous steps were focused on making it clear what the object is, contouring is the first that actually detects the object. It works by detecting continous edges in an image. In the third picture you can see this contour in action as the green line outlining the ball. You can also see that it is fairly accurate to the shape of the ball, because the eroding and dilating made the mask more accurate to the shape of the ball.
After contouring creates an outline of the object a centroid can be applied. This will take a mean of all the x and y values of the pixels in the object to find the center point. In the fourth image you can see the red ball outlined by contour with a blue dot in the middle, a result of the centroid. Once the center is identified and labled we can use that point to move the camera to track this center. The first code I created calculated and displayed how far offset the center of the ball was from the center of the camera and classified is this was “OffsetLEFT” or “OffsetRIGHT”. I then created code so that the robot would turn left or right until the object was in the center of its vision. This allowed the robot ot track the ball as it moved. (for more information on the code see below)
Here is an outline on how morphological transitions can effect an image:
(With the orginal image, the image after making, erosion and dilation, the image after contouring, and the image with the center identified)
*This last image is an example of an image with just a mask, no erosion or dilation. It is from a different orginal picture then the others
First Milestone
My major first milestone was creating code to control the robot chassis’ movements. I began by moving each wheel individually before condensing the code to easily call upon the chassis to move forward, backward, left and right (see code below). There were a number of smaller steps necessary to meet this first milestone including:
- Setting up, troubleshooting, and connecting to the Raspberry Pi
- Building the robot chassis
- Learning how to navigate Visual Studio Code
Schematics
Code (at Milestones)
This is my code from Milestone 1. It utilizes the library to import features like the GPIO pins and time. Defining variables and functions is the backbone of the code, beginning with defining various GPIO pins as motor inputs. I then do some important set-up things, including setting the GPIO pins as outputs (so they will respond to code telling them to output certain signals), setting the Hz for the PWM pins (so speed of motors can be adjusted), and defining how the motors should be at the beginning of the code. I then define the basic motor movements. The motor driver that I am using controls two motors and moves the wheel forward or backward based on what recieves power form the Raspberry Pi (ex. A_1A receiving power but A_1B not will move motor A backwards). Using these basic motor movemnets, I further condensed the code to create commands to move the robot forward, backward, left, and right. The code ends with a test sequence that allows one to test different parts of the code. I repeatedly edited thsi portion to ensure that all parts of the code was functioning. By using “try” and “finally” instead of “while True” ensured that the motors would stop after the test (as a ending sequence can be coded).
# Motor A and B pins
B_1A = 24
B_1B = 13
A_1A = 18
A_1B = 23
#Set up
import RPi.GPIO as GPIO
import time
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
for pin in [A_1A, A_1B, B_1A, B_1B]:
GPIO.setup(pin, GPIO.OUT)
#PWM at 100 Hz
pwm_A_1A = GPIO.PWM(A_1A, 100)
pwm_A_1B = GPIO.PWM(A_1B, 100)
pwm_B_1A = GPIO.PWM(B_1A, 100)
pwm_B_1B = GPIO.PWM(B_1B, 100)
pwm_A_1A.start(0)
pwm_A_1B.start(0)
pwm_B_1A.start(0)
pwm_B_1B.start(0)
#Define motor movements
def MotorA_forward(speed=100):
pwm_A_1A.ChangeDutyCycle(0)
pwm_A_1B.ChangeDutyCycle(speed)
def MotorA_backward(speed=100):
pwm_A_1A.ChangeDutyCycle(speed)
pwm_A_1B.ChangeDutyCycle(0)
def MotorB_forward(speed=100):
pwm_B_1A.ChangeDutyCycle(speed)
pwm_B_1B.ChangeDutyCycle(0)
def MotorB_backward(speed=100):
pwm_B_1A.ChangeDutyCycle(0)
pwm_B_1B.ChangeDutyCycle(speed)
def stop_all(speed=0):
for pwm in [pwm_A_1A, pwm_A_1B, pwm_B_1A, pwm_B_1B]:
pwm.ChangeDutyCycle(0)
pwm.ChangeDutyCycle(speed)
pwm_A_1A.stop()
pwm_A_1B.stop()
pwm_B_1A.stop()
pwm_B_1B.stop()
def forward (speed=100):
MotorB_forward()
MotorA_forward()
def backward (speed=100):
MotorA_backward()
MotorB_backward()
def right (speed=100):
MotorB_forward()
time.sleep(1)
pwm_B_1A.ChangeDutyCycle(0)
pwm_B_1B.ChangeDutyCycle(0)
def left (speed=100):
MotorA_forward()
time.sleep(1)
pwm_A_1A.ChangeDutyCycle(0)
pwm_A_1B.ChangeDutyCycle(0)
#test sequence
try:
time.sleep(7)
print("Motors forward and backward")
forward()
time.sleep(1)
backward()
time.sleep(1)
print("Motor right and left")
right()
time.sleep(1)
left()
time.sleep(1)
finally:
stop_all()
print("Test complete")
GPIO.cleanup()
Here is the code at the second Milestone (the names are what I have named the files). The first two sets of code set up what the robot draws upon to track the ball. The motor function (MotorTest.py) has been modified from the first Milestone to allow the robot to better adjust its position and track the ball more precisely. The second set of code (Camera2.py) starts by importing the necessary packages, takes a pictures, then applies the morphological transition discussed above (masking, eroding, dilating, contouring, and centroid) in order to identify where the ball is in that picture. The final set of code (Video.py) also begins by importing the necessary packages (imcluding MotorTest.py to draw upon the motor functions). It creates a function that tracks the red ball as it moves by defining how far offset the ball is the calling upon the correct motor function to move the necessary amount. It then creates a way to view the live video feed from the camera in a browser.
MotorTest.py
# Motor A and B pins
B_1A = 24
B_1B = 13
A_1A = 18
A_1B = 23
import RPi.GPIO as GPIO
import time
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
for pin in [A_1A, A_1B, B_1A, B_1B]:
GPIO.setup(pin, GPIO.OUT)
#PWM at 100 Hz
pwm_A_1A = GPIO.PWM(A_1A, 100)
pwm_A_1B = GPIO.PWM(A_1B, 100)
pwm_B_1A = GPIO.PWM(B_1A, 100)
pwm_B_1B = GPIO.PWM(B_1B, 100)
pwm_A_1A.start(0)
pwm_A_1B.start(0)
pwm_B_1A.start(0)
pwm_B_1B.start(0)
#Define motor movements
def MotorA_forward(speed=100):
pwm_A_1A.ChangeDutyCycle(0)
pwm_A_1B.ChangeDutyCycle(speed)
def MotorA_backward(speed=100):
pwm_A_1A.ChangeDutyCycle(speed)
pwm_A_1B.ChangeDutyCycle(0)
def MotorB_forward(speed=100):
pwm_B_1A.ChangeDutyCycle(speed)
pwm_B_1B.ChangeDutyCycle(0)
def MotorB_backward(speed=100):
pwm_B_1A.ChangeDutyCycle(0)
pwm_B_1B.ChangeDutyCycle(speed)
def stop_all(speed=100):
for pwm in [pwm_A_1A, pwm_A_1B, pwm_B_1A, pwm_B_1B]:
pwm.ChangeDutyCycle(speed)
def forward (speed=100):
MotorB_forward()
MotorA_forward()
def backward (speed=100):
MotorA_backward()
MotorB_backward()
def rightMajor (speed=50):
MotorB_forward()
MotorA_backward()
time.sleep(0.10)
MotorA_forward()
MotorB_forward()
time.sleep(.0001)
stop_all()
time.sleep(0.10)
def rightSmall (speed=1):
MotorB_forward()
MotorA_backward()
time.sleep(0.001)
stop_all()
time.sleep(0.06)
def leftMajor (speed=50):
MotorA_forward()
MotorB_backward()
time.sleep(0.10)
stop_all()
time.sleep(0.10)
def leftSmall (speed=1):
MotorA_forward()
MotorB_backward()
time.sleep(0.001)
stop_all()
time.sleep(0.06)
Camera2.py
from picamera2 import Picamera2
import cv2
import numpy as np
#Intilialize and configure the camera
picam2= Picamera2()
picam2.configure(picam2.create_preview_configuration(main={"format": "RGB888", "size": (640,480)}))
picam2.start()
#capture one frame
frame = picam2.capture_array()
#save to file
cv2.imwrite("ogframe.jpg", frame)
print("Frame saved as ogframe.jpg")
#save into var image
image = frame
hsv_image = cv2.cvtColor(image, cv2.COLOR_RGB2HSV)
lower_red = np.array([0, 50, 50])
upper_red = np.array([10, 255, 255])
lower_red2 = np.array([160, 100, 100])
upper_red2 = np.array([179, 255, 255])
kernel = np.ones((5,5), np.int8)
mask1 = cv2.inRange(hsv_image, lower_red, upper_red)
mask2 = cv2.inRange(hsv_image, lower_red2, upper_red2)
mask = cv2.bitwise_or(mask1, mask2)
mask = cv2.erode(mask, kernel, iterations=2)
mask = cv2.dilate(mask, kernel, iterations=2)
image_copy = image.copy()
contours, _ =cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
#find the largest contour by area
if contours:
largest_contour= max(contours, key=cv2.contourArea)
cv2.drawContours(image=image_copy, contours=[largest_contour], contourIdx=-1,color=(0,255,0), thickness=2, lineType=cv2.LINE_AA)
M = cv2.moments(largest_contour)
if M['m00']>0:
cx = int(M["m10"]/ M["m00"])
cy = int(M["m01"]/ M["m00"])
cv2.circle(image_copy, (cx, cy), 5, (255, 0, 0), -1)
#save to file
cv2.imwrite("resultframe.jpg", image_copy)
print("Frame saved as resultframe.jpg")
Video.py
from flask import Flask, Response, render_template_string
app = Flask(__name__)
import cv2
import numpy as np
from picamera2 import Picamera2
import RPi.GPIO as GPIO
import time
from MotorTest import *
app = Flask(__name__)
#initialize PiCam
picam2= Picamera2()
picam2.configure(picam2.create_preview_configuration(main={"format": "BGR888", "size": (640,480)}))
picam2.start()
FRAME_WIDTH = 640
CENTER_X = FRAME_WIDTH // 2
def track_red_ball(frame):
image = frame
hsv_image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
lower_red = np.array([0, 50, 50])
upper_red = np.array([10, 255, 255])
lower_red2 = np.array([160, 100, 100])
upper_red2 = np.array([179, 255, 255])
kernel = np.ones((5,5), np.int8)
mask1 = cv2.inRange(hsv_image, lower_red, upper_red)
mask2 = cv2.inRange(hsv_image, lower_red2, upper_red2)
mask = cv2.bitwise_or(mask1, mask2)
mask = cv2.erode(mask, kernel, iterations=2)
mask = cv2.dilate(mask, kernel, iterations=2)
image_copy = image.copy()
contours, _ =cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
if contours:
largest_contour= max(contours, key=cv2.contourArea)
cv2.drawContours(image=image_copy, contours=[largest_contour], contourIdx=-1,color=(0,255,0), thickness=2, lineType=cv2.LINE_AA)
M = cv2.moments(largest_contour)
if M['m00']>0:
cx = int(M["m10"]/ M["m00"])
cy = int(M["m01"]/ M["m00"])
cv2.circle(image_copy, (cx, cy), 5, (255, 0, 0), -1)
if( cx-140 > CENTER_X):
position = "OffsetRIGHTMajor"
rightMajor()
elif( cx-70 > CENTER_X):
position = "OffsetRIGHTSmall"
rightSmall()
elif(cx+140 < CENTER_X):
position = "OffsetLEFTMajor"
leftMajor()
elif(cx+70 < CENTER_X):
position = "OffsetLEFTSmall"
leftSmall()
else:
position = "Centered"
stop_all()
cv2.putText(image_copy, f"Offset: {cx-CENTER_X} ({position})", (10, 30),
cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 2)
return image_copy
def generate_frames():
while True:
frame = picam2.capture_array()
frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
frame = track_red_ball(frame)
ret, buffer = cv2.imencode('.jpg', frame)
jpg_frame = buffer.tobytes()
yield (b'--frame\r\n'
b'Content-Type: image/jpeg\r\n'
b'Content-Length: ' + f"{len(jpg_frame)}".encode() + b'\r\n\r\n' +
jpg_frame + b'\r\n')
@app.route('/')
def index():
return render_template_string('''
<html>
<head><title>Red Ball Tracking Stream</title></head>
<body>
<h2>Live Tracking</h2>
<img src="/video_feed">
</body>
</html>
''')
@app.route('/video_feed')
def video_feed():
return Response(generate_frames(),
mimetype='multipart/x-mixed-replace; boundary=frame')
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000)
Code (Final)
MotorTest.py
# Motor A and B pins
B_1A = 24
B_1B = 13
A_1A = 18
A_1B = 23
#Set up
import RPi.GPIO as GPIO
import time
from gpiozero import LED
led1 = LED(11)
led2 = LED(8)
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
GPIO.setup(21, GPIO.OUT)
#Set pins as outputs
for pin in [A_1A, A_1B, B_1A, B_1B]:
GPIO.setup(pin, GPIO.OUT)
#PWM at 100 Hz
pwm_A_1A = GPIO.PWM(A_1A, 100)
pwm_A_1B = GPIO.PWM(A_1B, 100)
pwm_B_1A = GPIO.PWM(B_1A, 100)
pwm_B_1B = GPIO.PWM(B_1B, 100)
pwm_A_1A.start(0)
pwm_A_1B.start(0)
pwm_B_1A.start(0)
pwm_B_1B.start(0)
#Define motor movements
def MotorA_forward(speed=100):
pwm_A_1A.ChangeDutyCycle(0)
pwm_A_1B.ChangeDutyCycle(speed)
def MotorA_backward(speed=100):
pwm_A_1A.ChangeDutyCycle(speed)
pwm_A_1B.ChangeDutyCycle(0)
def MotorB_forward(speed=100):
pwm_B_1A.ChangeDutyCycle(speed)
pwm_B_1B.ChangeDutyCycle(0)
def MotorB_backward(speed=100):
pwm_B_1A.ChangeDutyCycle(0)
pwm_B_1B.ChangeDutyCycle(speed)
def stop_all(speed=100):
for pwm in [pwm_A_1A, pwm_A_1B, pwm_B_1A, pwm_B_1B]:
pwm.ChangeDutyCycle(speed)
GPIO.output(21, GPIO.LOW)
def forward (speed=100):
MotorB_forward()
MotorA_forward()
def backward (speed=100):
MotorA_backward()
MotorB_backward()
led1.on()
led2.on()
GPIO.output(21, GPIO.HIGH)
time.sleep(0.5)
GPIO.output(21, GPIO.LOW)
time.sleep(0.5)
def rightMajor (speed=50):
MotorB_forward()
MotorA_backward()
time.sleep(0.10)
MotorA_forward()
MotorB_forward()
time.sleep(.0001)
stop_all()
time.sleep(0.10)
def rightMinor (speed=1):
MotorB_forward()
MotorA_backward()
time.sleep(0.001)
stop_all()
time.sleep(0.06)
def leftMajor (speed=50):
MotorA_forward()
MotorB_backward()
time.sleep(0.10)
stop_all()
time.sleep(0.10)
def leftMinor (speed=1):
MotorA_forward()
MotorB_backward()
time.sleep(0.001)
stop_all()
time.sleep(0.06)
Sensor.py
#set up
from gpiozero import DistanceSensor
ultrasonic2 = DistanceSensor(echo=19, trigger=12, threshold_distance=0.1)
ultrasonic1 = DistanceSensor(echo=6, trigger=5, threshold_distance=0.1)
from MotorTest import *
import time
#Defining what to do when the front of back sensor gets too close to an object
def closeF():
print("Too close")
backward()
time.sleep(0.4)
stop_all()
time.sleep(0.1)
print(ultrasonic1.distance)
def closeB():
print("Too close")
forward()
time.sleep(0.4)
stop_all()
time.sleep(0.1)
print(ultrasonic2.distance)
VideoMobile.py
#Set up
from flask import Flask, Response, render_template_string
app = Flask(__name__)
import cv2
import numpy as np
from picamera2 import Picamera2
import RPi.GPIO as GPIO
import time
from MotorTest import *
from ctypes import *
from Sensor import *
total = CDLL("./PCA9685/example/rpi/total2.so")
total.main()
app = Flask(__name__)
#initialize PiCam
picam2= Picamera2()
picam2.configure(picam2.create_preview_configuration(main={"format": "BGR888", "size": (640,480)}))
picam2.start()
FRAME_WIDTH = 640
CENTER_X = FRAME_WIDTH // 2
#Defining the main function that tracks the ball
def track_red_ball(frame):
#Finidng the correct object and it center
image = frame
hsv_image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
lower_red = np.array([0, 50, 50])
upper_red = np.array([10, 255, 255])
lower_red2 = np.array([160, 100, 100])
upper_red2 = np.array([179, 255, 255])
kernel = np.ones((5,5), np.int8)
mask1 = cv2.inRange(hsv_image, lower_red, upper_red)
mask2 = cv2.inRange(hsv_image, lower_red2, upper_red2)
mask = cv2.bitwise_or(mask1, mask2)
mask = cv2.erode(mask, kernel, iterations=2)
mask = cv2.dilate(mask, kernel, iterations=2)
image_copy = image.copy()
contours, _ =cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
if contours:
largest_contour= max(contours, key=cv2.contourArea)
cv2.drawContours(image=image_copy, contours=[largest_contour], contourIdx=-1,color=(0,255,0), thickness=2, lineType=cv2.LINE_AA)
M = cv2.moments(largest_contour)
if M['m00']>0:
cx = int(M["m10"]/ M["m00"])
cy = int(M["m01"]/ M["m00"])
cv2.circle(image_copy, (cx, cy), 5, (255, 0, 0), -1)
#Telling the servo how to turn after the center of the object has been identified
if( cx-140 > CENTER_X):
position = "OffsetRIGHTMajor"
rightMajor()
if ultrasonic1.in_range:
closeF()
elif ultrasonic2.in_range:
closeB()
elif( cx-70 > CENTER_X):
position = "OffsetRIGHTMinor"
rightMinor()
if ultrasonic1.in_range:
closeF()
elif ultrasonic2.in_range:
closeB()
elif(cx+140 < CENTER_X):
position = "OffsetLEFTMajor"
leftMajor()
if ultrasonic1.in_range:
closeF()
elif ultrasonic2.in_range:
closeB()
elif(cx+70 < CENTER_X):
position = "OffsetLEFTMinor"
leftMinor()
if ultrasonic1.in_range:
closeF()
elif ultrasonic2.in_range:
closeB()
else:
position = "Centered"
stop_all()
if ultrasonic1.in_range:
closeF()
elif ultrasonic2.in_range:
closeB()
cv2.putText(image_copy, f"Offset: {cx-CENTER_X} ({position})", (10, 30),
cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 2)
return image_copy
#Generating the live feed
def generate_frames():
while True:
frame = picam2.capture_array()
frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
frame = track_red_ball(frame)
ret, buffer = cv2.imencode('.jpg', frame)
jpg_frame = buffer.tobytes()
yield (b'--frame\r\n'
b'Content-Type: image/jpeg\r\n'
b'Content-Length: ' + f"{len(jpg_frame)}".encode() + b'\r\n\r\n' +
jpg_frame + b'\r\n')
#Putting that live feed in html for vieiwng in a browser
@app.route('/')
def index():
return render_template_string('''
<html>
<head><title>Red Ball Tracking Stream</title></head>
<body>
<h2>Live Tracking</h2>
<img src="/video_feed">
</body>
</html>
''')
@app.route('/video_feed')
def video_feed():
return Response(generate_frames(),
mimetype='multipart/x-mixed-replace; boundary=frame')
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000)
VideoStationary.py
#Set up
from flask import Flask, Response, render_template_string
app = Flask(__name__)
import cv2
import numpy as np
from picamera2 import Picamera2
import RPi.GPIO as GPIO
import time
from MotorTest import *
from ctypes import *
total = CDLL("./PCA9685/example/rpi/total2.so")
total.main()
app = Flask(__name__)
#initialize PiCam
picam2= Picamera2()
picam2.configure(picam2.create_preview_configuration(main={"format": "BGR888", "size": (640,480)}))
picam2.start()
FRAME_WIDTH = 640
FRAME_HEIGHT = 640
CENTER_X = FRAME_WIDTH // 2
#Defining the main function that tracks the ball
def track_red_ball(frame):
#Finidng the correct object and it center
image = frame
hsv_image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
lower_red = np.array([0, 50, 50])
upper_red = np.array([10, 255, 255])
lower_red2 = np.array([160, 100, 100])
upper_red2 = np.array([179, 255, 255])
kernel = np.ones((5,5), np.int8)
mask1 = cv2.inRange(hsv_image, lower_red, upper_red)
mask2 = cv2.inRange(hsv_image, lower_red2, upper_red2)
mask = cv2.bitwise_or(mask1, mask2)
mask = cv2.erode(mask, kernel, iterations=2)
mask = cv2.dilate(mask, kernel, iterations=2)
image_copy = image.copy()
contours, _ =cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
if contours:
largest_contour= max(contours, key=cv2.contourArea)
cv2.drawContours(image=image_copy, contours=[largest_contour], contourIdx=-1,color=(0,255,0), thickness=2, lineType=cv2.LINE_AA)
M = cv2.moments(largest_contour)
if M['m00']>0:
cx = int(M["m10"]/ M["m00"])
cy = int(M["m01"]/ M["m00"])
cv2.circle(image_copy, (cx, cy), 5, (255, 0, 0), -1)
#Telling the robot how to turn after the center of the object has been identified
if( cx-170 > CENTER_X):
position = "OffsetRIGHTMajor"
total.ServoDegreeIncrease(0,8)
elif( cx-140 > CENTER_X):
position = "OffsetRIGHTMinor"
total.ServoDegreeIncrease(0,6)
elif( cx-70 > CENTER_X):
position = "OffsetRIGHTMicro"
total.ServoDegreeIncrease(0,4)
elif(cx+170 < CENTER_X):
position = "OffsetLEFTMajor"
total.ServoDegreeDecrease(0,8)
elif(cx+140 < CENTER_X):
position = "OffsetLEFTMinor"
total.ServoDegreeDecrease(0,6)
elif(cx+70 < CENTER_X):
position = "OffsetLEFTMicro"
total.ServoDegreeDecrease(0,4)
elif(cy-120 > CENTER_X):
position = "OffsetDOWNMajor"
total.ServoDegreeDecrease(1,10)
elif(cy-50 > CENTER_X):
position = "OffsetDOWNMinor"
total.ServoDegreeDecrease(1,5)
elif(cy+140 < CENTER_X):
position = "OffsetUPMajor"
total.ServoDegreeIncrease(1,8)
elif(cy+70 < CENTER_X):
position = "OffsetUPMinor"
total.ServoDegreeIncrease(1,5)
else:
position = "Centered"
stop_all()
cv2.putText(image_copy, f"Offsetx: {cx-CENTER_X} ({position})", (10, 30),
cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 2)
return image_copy
#Generating the live feed
def generate_frames():
while True:
frame = picam2.capture_array()
frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
frame = track_red_ball(frame)
ret, buffer = cv2.imencode('.jpg', frame)
jpg_frame = buffer.tobytes()
yield (b'--frame\r\n'
b'Content-Type: image/jpeg\r\n'
b'Content-Length: ' + f"{len(jpg_frame)}".encode() + b'\r\n\r\n' +
jpg_frame + b'\r\n')
#Putting that lie feed in html for vieiwng in a browser
@app.route('/')
def index():
return render_template_string('''
<html>
<head><title>Red Ball Tracking Stream</title></head>
<body>
<h2>Live Tracking</h2>
<img src="/video_feed">
</body>
</html>
''')
@app.route('/video_feed')
def video_feed():
return Response(generate_frames(),
mimetype='multipart/x-mixed-replace; boundary=frame')
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000)
Please note that the code for the image recogition abilities are not here. You can access them using the following links (also at the bottom):
Bill of Materials
Here’s where you’ll list the parts in your project. To add more rows, just copy and paste the example rows below. Don’t forget to place the link of where to buy each component inside the quotation marks in the corresponding row after href =. Follow the guide here to learn how to customize this to your project needs.
| Part | Note | Price | Link |
|---|---|---|---|
| Raspberry Pi 4 Starter Kit | The microcontroller powering and controlling this project | $95.19 | Link |
| Robot Chassis | Basic body for the robot | $18.99 | Link |
| Screwdriver Kit | Screwing in screws of different sizes | $5.94 | Link |
| Ultrasonic Sensor | Sensor used to prevent crashes | $9.99 | Link |
| H Bridges | Motor driver to connect motors to the Rapsberry Pi and control them | $8.99 | Link |
| Pi Cam | Camera used to take pictures and stream live feed | $12.86 | Link |
| Electronics Kit | Basic electronic components in circuits and breadboard | $11.98 | Link |
| Motors | Used to turn wheels | $11.98 | Link |
| Digital Multimeter | Checking current in circuit | $11.00 | Link |
| Champion Sports Ball | The ball that the robot tracks | $16.73 | Link |
| AA Batteries | Batteries for the multimeter and alternative power for the robot | $18.74 | Link |
| USB Power Bank and Cable | Power for the robot | $16.19 | Link |
| Double Sided Foam Tape | Used in mounting different components | $6.99 | Link |
| Raspberry Pi Camera Servo | Servo mount for camera | $26.99 | Link |
| Cardboard | Added to create a flat surface on the robot for mounting | N/A | N/A |
| Clear Packing Tape | Wrapped around cardboard to make it more sturdy and waterproof | N/A | N/A |