Objectives

A robot, which has already been designed for this lab using an EV3 robotics kit, will be evaluated for accuracy and precision by utilizing two different sets of code, the first without the use of sensors and the second with sensors. The robot will be tested on one of the two provided mini-courses. The data from the product evaluation of both sets of code will determine if the robot should or should not use sensors.

Product Evaluation

Product evaluation is a common task for an engineer and for companies. Sometimes the product evaluated will be a company's product, and sometimes companies will be evaluating a competitor's design. In either case, this evaluation will use a standard (an accepted value), determine an average, and calculate accuracy and precision. ^[1]

Professional engineers must be sure of standards. For the purpose of this lab, the EG1004 standard will be 80% precision and accuracy. This means that the product has to achieve a percent accuracy of 80% and meet a tolerance of $\pm$ 1cm for distance and $\pm$ 10 degrees for angle. ^[1]

When taking measurements, the first step is to determine an average (Equation 1).

$Average\left(A_{p}\right)={\frac {P_{1}+P_{2}+P_{3}+...+P_{n}}{N}}\,$

Equation 1: Average.

In this equation, A_p is the average, P₁, P₂, P₃, ..., P_n are the results of the tests, and N is the number of tests performed.

The average is compared to the standard. Accuracy is a variable the must always be considered. The degree of accuracy required will allow the results to be verified. Engineers often work on projects that affect people's safety so it is critical that the solutions to the problems solved by technical professionals are accurate.

The comparison of the average to the standard is called accuracy, defined as Acc in Equation 2.

$Acc=\left|P_{s}-A_{p}\right|\,$

Equation 2: Accuracy.

In Equation 2, P_s is the standard value, and A_p is the value measured. Accuracy is always a positive number, so use the absolute value in case P_s is less than A_p.

Precision is the repeatability of a result, or how close the results are to each other, defined as Prec in Equation 3.

$Prec=\left|P_{high}-P_{low}\right|\,$

Equation 3: Precision.

In Equation 3, Prec is precision, P_high is the highest data value and P_low is the lowest data value. Like accuracy, precision is always a positive number, so use an absolute value here as well.

A product can be accurate but imprecise, or precise but inaccurate (Figure 1). The goal is to achieve a combination of the highest accuracy and precision. In order for engineers to make a recommendation based on experimental test data, it is important to determine how a product compares to a standard. Engineers need these benchmarks to determine the quality of a product.

Figure 1: Accuracy and precision.

By determining the percent accuracy of an experiment, and comparing this percentage to a standard, it can be determined, based on experimental evidence, if the product has passed or failed. The equation for percent accuracy is:

$\%Acc=\left(1-{\frac {\left|P_{s}-A_{p}\right|}{P_{s}}}\right)\times 100\%\,$

Equation 4: Percent accuracy.

In Equation 4, %Acc is the percent accuracy, P_s is the standard value, and A_p is the value measured.

Precision shows how precise a product is compared to the standard.

Adequate precision is determined by an acceptable tolerance, set by the manufacturer. In this lab, the precision of the robot must be compared to the tolerance allowed:

Distance Test:

T=\pm 1cm

Angle of Deviation Test:

T=\pm 10degrees

Calculating percent accuracy and precision and then comparing those results to a standard determines the quality of the product.

Quality Improvement

Quality improvement is a process of analyzing a design and testing it through physical modeling, computer simulation, or mathematical modeling. Quality improvement ensures that a product will perform as expected and allows improvements to be made to the design.

When deciding how to improve the design in this lab, keep these questions in mind:

Does the robot perform to standard?
- If no, what can be done to improve its functionality?
- If yes, what can be done to improve its performance beyond the standard?

Changes will be implemented after testing the initial code.

Reverse Engineering

Reverse engineering software involves reversing a program's machine code to obtain the original source code or determine the file structures the program uses. This is done when source code is lost or is not available. For example, when a project is reactivated after being dormant for years to add new features to a product, the original design documentation may have been lost. Also, when companies are writing a program that must interface with a second company's software, the second company may not have any incentive to cooperate by describing how the data in the files they use are organized, forcing the first company to deduce it on their own. Sometimes, this process is used to improve the performance of a program, to fix a bug, or to find a virus. When the source code is obtained in this way for any of these reasons, it is legal and necessary. Reverse engineering software to copy it constitutes a copy¬right violation and is illegal.^[2]

Hardware reverse engineering involves taking apart a device to see how it works. If a processor manufacturer wants to see how a competitor's processor works, the company can purchase the processor, disassemble it, and then make a new processor similar to it. In some countries, this process is illegal. Hardware reverse engineering is quite expensive and requires an expert in the field.^[2]

Design Considerations

Carefully consider the lab objectives and what needs to be done to achieve them:

What made or will make the results, inaccurate or imprecise?
If the robot failed a test, why did it fail?

EV3 Sensors

Sensor Basics

A sensor is a device that measures a specific value and is used as an input. Different properties can be measured by the EV3 sensors, such as light intensity, angle, or distance. The three sensors used in this lab are the light sensor, gyro sensor, and ultrasonic sensor.

Light Sensor

The light sensor or color sensor (Figure 2) can measure reflected light intensity, color, and ambient light, and is useful when the robot has the option to follow a line or use differences in color to perform different actions, such as stopping when detecting a different color. If measuring color, the sensor outputs a value (0-7) that corresponds to a specific color. Measuring reflected light outputs a value (0-100) depending on the intensity of the light that is reflected. The ambient light works the same way as the reflected light, except it measures only the surrounding light. The light sensor, as well as all other sensors, can be found in the yellow category the Mindstorms software.

Figure 2: Light sensor.

The type of value the sensor can measure can be changed by clicking the small box on the bottom left of the block (Figure 3).

Figure 3: Light sensor, value input.

Gyro Sensor

The gyro sensor can measure the degrees it turns or the rate at which it is turning. For example, if the robot is set to turn 90º using the gyro, the robot will keep trying to turn until 90º is reached, regardless of any bumps in its way (Figure 4).

Figure 4: Gyro sensor.

Ultrasonic Sensor

The ultrasonic sensor measures the distance between the robot and the closest object to it. It can measure distance in both centimeters and inches; the recommended range is 1 cm to 250 cm (Figure 5).

Figure 5: Ultrasonic sensor.

Switches and Loops

Switches and loops are used to tell the robot what actions to perform depending on the inputs from its surroundings, measured by the sensors.

Switch

A switch is an “if-else” statement: if the initial condition is true, the top action will occur, if false, the bottom action will occur. This block is used when the robot has to perform different actions depending on the input from the sensors (Figure 6).

Figure 6: Switch.

Loop

A switch will only run the code once, so if repeated action is desired, a loop must be used to repeat the switch’s statement. A loop continuously runs the code until the end statement occurs. The end statement can be determined with a sensor or hard-coded to a specific duration or rotations. Figure 7 shows an infinite loop.

Figure 7: Loop.

Materials and Equipment

A Mindstorms kit with pre-made robot
A computer with Mindstorms software
A mini-course with accuracy markings

Procedure

Problem Statement

Evaluate two types of code for a robot by creating a line of code without the use of sensors and then a separate line of code using sensors. For each trial, record the distance from the wall and angle of deviation from the center line. Test the first code by running four trials of Section 1 of the mini-course as marked on the physical course. Create a second line of code using switches and loops that incorporate the types of sensors found on the robot and features on the mini-course. Test the robot again with four trials of Section 1. This part of the procedure will have a total of 8 trials. Download the pre-made Datasheet and fill in the data for each trial.

Course Breakdown

The mini-course (Figure 8) is divided into two sections: one flat straight section of four tiles and a second section consisting of a right turn and ramp. Both sections have a wall at the end. In Figure 8, Section 1 is marked in red and Section 2 is marked in blue.

Figure 8: Mini-course.

Part 0: Pre-Lab Exercise

Download the instructions for the robot assigned to your group (found in the Documents section for Lab 3). Assemble the robot by following the directions using the kit assigned to your group. Test and modify the robot at home before coming to lab by following the steps below.

Reverse Engineering

Start the Mindstorms EV3 software. Click on the + in the top left corner to open up a blank program

Figure 9: Mindstorms screen.
The screen is shown in Figure 9. The program begins with the Start block

Figure 10: Start block.
Place a Steering block from the Action palette next to the Start block (Figure 10). Set the Port to reflect both motors plugged into the brick; both ports have to be correct

Figure 11: Steering block.
Along the bottom of the Steering block, there are four options. On the left, there is a Mode Selector button to select what units the motors will run for. Next are three values to decide what Direction, Power, and number of Rotations. The last input decides whether or not to Brake at End

Figure 12: Steering block mode.
Select On for Rotations for the Steering block. Test different values of rotations to watch the robot's performance. Does the robot do what its programmed to do?
Change the code to make the robot travel roughly four feet. If there are physical problems with the robot, modify the design and reprogram to travel about four feet. The robot also has to be able to make right turns. Bring the modified robot to the actual Lab 3 session to complete the rest of the lab.

Part 1: Hardcoding

Reverse Engineering

Upon receiving the pre-built robot, selected by the TA, analyze the design and performance of the robot by both physical inspection and basic testing (i.e. run the robot at least once to see how it works). Based off of these observations, make design changes to the robot in order to run trials on the first section (Section 1) of the mini-course.

Product Evaluation

Using the modified design, conduct four trials of the robot using a code without sensors to travel in a straight line across the first four tiles, turn 90 degrees to the right, and stop 5 cm from the wall. The values used in the sections of code below are only examples; the actual values must be determined through experimentation.

Start the Mindstorms EV3 software
Click on the + in the top left corner to open up a blank program

Figure 9: Mindstorms screen.
The screen is shown in Figure 9. The program begins with the Start block

Figure 10: Start block.
Place a Steering block from the Action palette next to the Start block (Figure 10). Set the Port to reflect both motors plugged into the brick; both ports have to be correct

Figure 11: Steering block.
Along the bottom of the Steering block, there are four options. On the left, there is a Mode Selector button followed by three values for the input parameters picture above the value from left to right, Power, Rotations, and Brake at End

Figure 12: Steering block mode.
Change the type of the Steering block to On for Seconds. This will change the Rotations input parameter to Seconds

Figure 13: Steering block rotations' menu.
Add a Steering or Tank block to turn the robot to the right 90 degrees. Then add another Steering block to travel straight until it is 5 cm away from the wall.
Test different values of Power and Seconds to find the best settings to travel the first four tiles of the course and stop five cm from the wall. When the optimal settings are found, alert the TA and run the program. Run the program for four trials. For each trial, record the distance between the robot and wall when the robot stops and the angle of deviation from the center line (draw a line in pencil from the tape to a point on the robot that started on the tape at the start of the trial and measure the angle with a protractor). Record the angle perpendicular to the tape to find the angle measurement.

Part 2: Sensors

Quality Improvement

Using the results of the original robot, incorporate the use of sensors in the robot's code. The following steps outline how to build a line-following code for the robot to stay on track. The values used in the sections of code below are only examples; the actual values must be determined through experimentation.

Click on the + in the top left corner to open up a blank program.
Insert a blank Loop. Insert a Switch within the loop.

Figure 14: Code for light sensor.
Set the input of the switch to a Light Sensor and the type to Reflected Light Intensity. Set the comparison to the “greater than” symbol and use Port View on the EV3 brick to measure the light intensity of the line. Set the value of the light intensity to the value displayed by Port View (not the value seen in the image below)

Figure 15: Light sensor output.
In the top half of the switch, insert two Move Motor blocks. Set one to stop and one to continuously moving. Duplicate this in the bottom half, but switch the ports. This way one wheel will move while the other is stationary, which will make the robot wiggle along the line

Figure 16: Move motor code.
Each time the Gyro Sensor is used, it has to be reset in order to set the direction the robot is facing before turning as zero. Insert a Gyro Sensor block and set it to reset:

Figure 18: Gyro Sensor Reset
Insert a Wait block right after the gyro reset. Set it to 2 seconds.

Figure 19: Wait Block
In order for the robot to continue turning until 90 degrees is reached, a Loop must be used to constantly run the robot until the gyro sensor reads 90 degrees. Insert a loop after the wait block and set the end condition to be activated by the gyro sensor using Measure and Angle. Change the comparison to Greater Than and the value of the angle to 90 degrees.

Figure 20: Gyro Sensor as End Statement
Insert a Motor block into the loop. Select the port that when activated will turn the robot to the right. Change the Motor type to Rotations and .05 rotations. This translates to the robot moving in .05 rotation increments until the gyro sensor reads 90 degrees:

Figure 21: Turning Program
After the loop, insert a Steering block for the robot to travel straight one tile. Use either rotations or time to move the robot and duplicate the first part of the program after the motor block.

Figure 22: Turning Program Continued
Use the Ultrasonic Sensor to make the robot stop five cm away from the first wall. Set the end statement of the loop to be controlled by the Ultrasonic Sensor and set the type to Distance Centimeters for five cm.

Figure 17: Ultrasonic sensor code.

Tabulation of Results

All data must be inputted into the Datasheet. From the datasheet, determine if the EG standards were met for accuracy and precision; recorded precision must be divided by 2 to get the corresponding tolerance:

Hardcode	Accuracy	%Acc	Pass/Fail	Precision	Tolerance	Pass/Fail
Distance Test
Angle of Deviation Test

Sensors Code	Accuracy	%Acc	Pass/Fail	Precision	Tolerance	Pass/Fail
Distance Test
Angle of Deviation Test

Part 3: Robot Competition

The robot competition consists of a single trial of the entire course to travel the most tiles as possible.

Procedure

The robot competition consists of attempting to travel the most amount of tiles possible in a single run. Only one trial counts as valid and must verbally be declared before the run starts.

Recreate the line following code to allow the robot to travel up the ramp. Test different values of power to overcome the ramp. The end statement should cause the robot to stop 10 cm away from the wall.

Figure 23: Full Competition Code

Competition Rules

Only parts provided in the EV3 kits are allowed (no regular Lego's)
No changes in physical robot design
No "re-tests"
The trial used for the competition must be declared before the robot runs and cannot be disputed

Competition Ratio

TBD

The lab work is now complete. Please clean up the workstation. Return the robot to the TA in exchange for your SLDP robot kit. Also return all unused materials to the TA. Refer to the Assignment section for the instructions needed to prepare the lab report.

Assignment

Individual Lab Report

Follow the lab report guidelines laid out in the page called Specifications for Writing Your Lab Reports in the Technical Communication section of this manual. The following points should be addressed in the appropriate section of the lab report.

Discuss product evaluation and the need for it
Discuss quality improvement and its importance
Explain why reverse engineering is important
Include examples of product evaluation, quality improvement, and reverse engineering
Discuss a standard, an average, accuracy, and precision
Describe loops and switches and how they are useful in coding
Describe the light sensor, gyro sensor, and ultrasonic sensor used in the lab
Discuss the importance of tolerance and %Acc in determining product quality
Discuss the EV3 kit and software used
Describe Mindstorms programming for only the sensors coding portion
Describe all robot modifications
Compare the final code with sensors and final code without sensors; determine which is better. Substantiate the decision with data, observations, and other facts collected
Make a final evaluation of the product: should it go into production? Describe its accuracy and precision
Assess the techniques used in the testing process
Discuss better testing methods for the robot design
Recommend if any design improvements should be made

Remember: Lab notes must be taken. Experimental details are easily forgotten unless written down. EG1004 Lab Notes Paper can be downloaded and printed from the EG1004 Website. Use the lab notes to write the Procedure section of the lab report. At the end of each lab, a TA will scan the lab notes and upload them to the Lab Documents section of the EG1004 Website. One point of extra credit is awarded if the lab notes are attached at the end of the lab report. Keeping careful notes is an essential component of all scientific practice.

Team PowerPoint Presentation

Follow the presentation guidelines laid out in the page called EG1004 Lab Presentation Format in the Introduction to Technical Presentations section of this manual. When preparing the presentation, consider the following points:

Based on the test results, what is the product's marketability?
If determinations were that the robot(s) did not meet the standard, what improvements could be made?
Why is product evaluation important?
Discuss why quality improvement is important
Discuss the importance of utilizing sensors

^ ^a ^b Oakes, W.C., L.L. Leone, and C.G. Gunn, Engineering Your Future, MI: Great Lakes Press, 2002.
^ ^a ^b What Is website. TechTarget Network. Retrieved July 29th, 2003.

[one-1] Oakes, W.C., L.L. Leone, and C.G. Gunn, Engineering Your Future, MI: Great Lakes Press, 2002.

[two-2] What Is website. TechTarget Network. Retrieved July 29th, 2003.

[1]

[2]

Navigation menu

Product Evaluation (Select Sections Only)

Contents

Objectives

Product Evaluation

Quality Improvement

Reverse Engineering

Design Considerations

EV3 Sensors

Sensor Basics

Light Sensor

Gyro Sensor

Ultrasonic Sensor

Switches and Loops

Switch

Loop

Materials and Equipment

Procedure

Problem Statement

Course Breakdown

Part 0: Pre-Lab Exercise

Reverse Engineering

Part 1: Hardcoding

Reverse Engineering

Product Evaluation

Part 2: Sensors

Quality Improvement

Tabulation of Results

Part 3: Robot Competition

Procedure

Competition Rules

Competition Ratio

Assignment

Individual Lab Report

Team PowerPoint Presentation