Product Evaluation

Objective

A robot will be evaluated for accuracy and precision when using two different sets of code, the first without the use of sensors and the second with sensors. The robot will be tested on one of the three provided mini-courses. The data from the product evaluation of both sets of code will determine if the robot should or should not use sensors in navigating the mini-course. Extra credit will be awarded if the robot is able to make it to the final tile of the mini-course.

Overview

Product Evaluation

Product evaluation is a common task for an engineer and for companies. Sometimes the product evaluated will be a company's product, and sometimes companies will be evaluating a competitor's design. In either case, this evaluation will use a standard (an accepted target value), determine an average, and calculate accuracy and precision (Oakes, 2002).

Professional engineers must be sure of standards. For the purpose of this lab, the EG1004 standard will be 80% precision and accuracy. This means that the product has to achieve a percent accuracy of 80% and meet a tolerance of ±1 cm for distance and ±10° for angle of deviation (Oakes, 2002).

When taking measurements, the first step is to determine an average (1).

$Average\left(A_{p}\right)={\frac {P_{1}+P_{2}+P_{3}+...+P_{n}}{N}}\,$

(1)

In (1), A_p is the average, P₁, P₂, P₃, ..., P_n are the results of the tests, and N is the number of tests performed. The average is compared to a standard.

Accuracy is a variable the must always be considered. The accuracy required will allow the results to be verified. Engineers often work on projects that affect people's safety so it is critical that the solutions to the problems solved by technical professionals are accurate. The comparison of the average to the standard (2) is called accuracy (Acc).

$Acc=\left|P_{s}-A_{p}\right|\,$

(2)

In (2), P_s is the standard value, and A_p is the average value measured. Accuracy is always a positive number so use the absolute value in case P_s is less than A_p.

In order for engineers to make a recommendation based on experimental test data, it is important to determine how a product compares to a standard. Engineers need these benchmarks to determine the quality of a product. By calculating the percent accuracy (3) of a set of testing data and comparing this percentage to a standard, it can be determined, based on experimental evidence, if the product has passed or failed.

$\%Acc=\left(1-{\frac {\left|P_{s}-A_{p}\right|}{P_{s}}}\right)\times 100\%\,$

(3)

In (4), %Acc is the percent accuracy, P_s is the standard value, and A_p is the average value measured.

Precision (4) is the repeatability of a result, or how close the results are to each other.

$Prec=\left|P_{high}-P_{low}\right|\,$

(4)

In (4), Prec is precision, P_high is the highest data value, and P_low is the lowest data value. Like accuracy, precision is always a positive number so use an absolute value here as well.

Precision shows how precise a product is compared to a standard. Adequate precision is determined by an acceptable tolerance, which is set by the manufacturer. In this lab, the precision of the robot must be compared to the tolerance (T) allowed.

• Distance Test: $T=\pm 1cm$
• Angle of Deviation Test: $T=\pm 10^{\circ }$

A product can be accurate, but imprecise, or precise but inaccurate (Figure 1). The goal is to achieve a combination of the highest accuracy and precision.

Figure 1: Accuracy and Precision

Calculating percent accuracy and precision and then comparing those results to a standard determines the quality of the product.

Quality Improvement

Quality improvement is a process of analyzing a design and testing it through physical modeling, computer simulation, or mathematical modeling. Quality improvement ensures that a product will perform as expected and allows improvements to be made to the design. When deciding how to improve the design in this lab, keep these questions in mind.

When deciding how to improve the design in this lab, keep these questions in mind:

• Does the robot perform to standard?
  • If no, what can be done to improve its functionality?
  • If yes, what can be done to improve its performance beyond the standard?

Quality improvement will be practiced in this lab by testing the initial code and implementing changes, if needed.

EV3 Sensors

Sensor Basics

A sensor is a device that measures a physical property and records, indicates, or responds to the measured value. This course uses EV3 Mindstorms robots and sensors. EV3 sensors can measure different properties, such as light intensity, angle, or distance. The three EV3 sensors used in this lab are the light sensor, gyro sensor, and ultrasonic sensor.

A light sensor or color sensor can measure reflected light intensity, color, and ambient light and is useful when the robot has the option to follow a line or use differences in color to perform different actions, such as stopping when detecting a different color (Figure 2). If measuring color, the sensor outputs a value (0-7) that corresponds to a specific color. Measuring reflected light outputs a value (0-100) depending on the intensity of the light that is reflected. Ambient light works in the same way as the reflected light except it measures only the surrounding light. The light sensor and other sensors can be found in the yellow category in the Mindstorms software. The light sensor should be placed so that it is 2 cm, at most, from whatever it is measuring.

Figure 2: Light Sensor

The value the light sensor can measure can be changed by clicking the small box on the bottom left of the block (Figure 3).

Figure 3: Light Sensor Value Input

A gyro sensor can measure the degrees it turns from a start point or the rate at which it is turning (Figure 4). For example, if the robot is set to turn 90° using the gyro sensor, the robot will keep turning until 90° is reached, regardless of any bumps in its way. The gyro sensor should be placed parallel to the ground to allow for the most accurate measurements.

Figure 4: Gyro Sensor

The ultrasonic sensor measures the distance between the robot and the object closest to it by emitting an ultrasonic sound wave and receiving back any reflected waves (Figure 5). It can measure distance in both centimeters and inches; the recommended range is 1 cm to 250 cm.

Figure 5: Ultrasonic Sensor

EV3 Programming

EV3 Mindstorms is a graphic-based programming language that directs the actions of the robot. In the EV3 programming software, switches and loops are programming instructions that tell the robot the actions to perform and when to stop performing those actions.

A switch is a block of program that tells the robot to perform certain actions depending on the input to the switch. Different inputs tell the robot to perform different actions. A switch is essentially an “if-else” statement: if the initial condition is met, the top action will occur; if it is not met, the bottom action will occur (Figure 6).

Figure 6: Switch

A switch will only run once, whereas a loop will repeat a desired action until an end condition is met. The end condition can be determined with a sensor or hardcoded to a specific duration or distance. Figure 7 shows an infinite loop – one that will run forever because there is no end condition to stop the program.

Figure 7: Loop

Design Considerations

Carefully consider the lab objectives and what needs to be done to achieve them.

• What made or will make the results inaccurate or imprecise?
• If the robot failed a test, why did it fail?

Materials and Equipment

• A Mindstorms kit with robot
• A computer with Mindstorms software
• A mini-course with accuracy markings

Procedure

Download the following files:

• Datasheet
• Mindstorms Program

Problem Statement

Evaluate two types of code for a robot in navigating a specific section of the mini-course by creating a line of code without the use of sensors (hardcoding) and then a separate line of code using sensors. For each trial, record the distance from the wall and angle of deviation from the center line from where the robot finishes running its code. Test the first code by running four trials on Section 1 of the mini-course as marked on the physical course (Figure 12). Create a second line of code using switches and loops that incorporates the sensors found on the robot and features on the mini-course. Test the robot again with four trials on Section 1. These two parts of the procedure will have eight trials total. Download the pre-made Datasheet and fill in the data for each trial.

Course Breakdown

The mini-course (Figure 8) has a designated Section 1 marked in red to complete the eight trials for the lab: one flat straight section of four tiles and a right turn with a wall at the end. The competition part of the lab will utilize the entire course.

Figure 8: Mini-Course

Tabulation of Results

All data must be inputted into the datasheet. From the datasheet, determine if the EG1004 standards were met for accuracy and precision. The measured precision must be divided by two to find the tolerance.

Table 1: Hardcode Data

Hardcode	Accuracy	%Acc	Pass/Fail	Precision	Tolerance	Pass/Fail
Distance Test
Angle of Deviation Test

Table 2: Sensors Code Data

Sensors Code	Accuracy	%Acc	Pass/Fail	Precision	Tolerance	Pass/Fail
Distance Test
Angle of Deviation Test

1. Hardcoding

Based on testing and observations, make design changes to the robot to run trials on Section 1 of the mini-course.

Every run counts as a trial unless the robot goes off the course. There is only one test run allowed after which all runs will be recorded as trials. When troubleshooting the code, please first cross check the port number on the code and the port the sensor/motor is connected.

Using the modified design, conduct four trials of the robot using a code without sensors to have the robot travel in a straight line across the first four tiles, turn 90° to the right, and stop 5 cm from the wall marked in yellow at Tile 5. The values used in the sections of code are only examples. The actual values must be determined through experimentation.

1. Start the Mindstorms EV3 software
2. Download the zipped folder containing the Mindstorms program
3. Go to the Hardware Trials tab of the Mindstorms program. The hardcoding program should look like the code shown in Figure 9. Please read the comments before running the program
   

   Figure 9: Part 1 Code
4. Change the ports to the configuration on the robot. Adjust the motor block power values to non-zero values. Run a sample trial to test the code
5. Evaluate the number of rotations the robot must travel by measuring the distance the robot must move and divide that by the circumference of the wheel. Change the values to the evaluated numbers
6. Try to stop the robot 5 cm before the wall
7. Run the program for four trials. For each trial, record the distance and angle of deviation using the pre-made markings on the sides of the tile
8. Distance is measured as distance from the wall. Use the ruler on the side of the tile. Denote it as y
9. Angle of deviation is calculated in the following way
   1. Measure deviation from the center of the wall. Denote it as x
   2. Use ${tan^{\text{-1}}}(x/y)$
   3. If there is confusion as to how to calculate angle of deviation, ask a TA for help

2. Sensors

Every run counts as a trial unless the robot goes off the course. There is only one test run allowed after which all runs will be recorded as trials. When troubleshooting the code, please first cross check the port number on the code and the port the sensor/motor is connected to.

Every run counts as a trial unless the robot goes off the course. There is only one test run allowed after which all runs will be recorded as trials. When troubleshooting the code, please first cross check the port number on the code and the port the sensor/motor is connected to.

1. Move to the Software Trials tab of the downloaded Mindstorms program. The program should look like Figure 10. Please read the comments before running the program
   

   Figure 10: Software Trials Code
2. Change the ports to the configuration on the robot. Adjust the motor block power values to a non-zero value. Adjust the wait block time values to a non-zero value
3. Using the port view shown in Figure 11, set the readings on the Mindstorms code for the light sensor and the gyro sensor blocks to the correct threshold values. The light sensor threshold value is an arbitrary value from 0 to 100 indicating the amount of light reflected (with 0 being no light reflected). The gyro sensor threshold value is the angle of rotation in degrees. The ultrasonic sensor threshold value is the distance away from an obstacle in inches or centimeters. For example, if the light sensor is moved over a black object, the port view reading will be around zero (because black objects absorb all light and there will be no reflected light to measure). In the switch, the initial condition should be set to a value that indicates that the color black is being read (< 2). The initial condition should not be set to (= 0) because the sensor is imprecise and a range should be provided for the initial condition. Adjust the number of rotations as done in the previous part. Ask a TA for assistance if the values cannot be found
   

   Figure 11: Port View
4. Take the same measurements as for 1. Hardcoding. Tabulate results in the datasheet

3. Extra Credit

The extra credit portion consists of the robot traversing the entire course. If the robot reaches the last tile, extra credit will be awarded. Again, the values used in the sections of code below are only examples. The actual values must be determined through experimentation.

Procedure

The extra credit portion consists of attempting to travel the greatest number of tiles in a single run.

1. Recreate the line of code in Figure 12 to allow the robot to travel up the ramp. Test different values of power to overcome the ramp. The end statement should cause the robot to stop 10 cm away from the wall
2. Notice that everything until the second loop is a replica of the Software Trials code with the last loop and switch added. Ask a TA for assistance if this cannot be solved

   

   Figure 12: Extra Credit Code

Rules

1. Only parts provided in the EV3 kits are allowed (no regular LEGO pieces)
2. No changes in physical robot design
3. A robot counts as clearing a tile if it stops on the tile with at least 50% of the robot on the tile

The lab work is now complete. Please clean up the workstation. Return all unused materials to the TA. Refer to the Assignment section for the instructions needed to prepare the lab report.

Assignment

Individual Lab Report

Follow the lab report guidelines laid out in the EG1004 Writing Style Guide in the Technical Writing section of the manual. Use the outline below to write this report.

• Discuss product evaluation and the need for it
• Discuss quality improvement and its importance
• Include examples of product evaluation and quality improvement
• Discuss a standard, an average, accuracy, and precision; include their equations
• Describe loops and switches and how they are useful in coding
• Describe the light sensor, gyro sensor, and ultrasonic sensor used in the lab and their optimal positions
• Discuss the importance of tolerance and percent accuracy in determining product quality
• Discuss the EV3 kit and software used
• Describe Mindstorms programming for only the sensors coding portion
• Discuss how robot behavior changed after sensors were added and code was revised
• Compare the final code with sensors and final code without sensors; determine which is better. Substantiate the decision with data, observations, and other facts collected
• Assess the techniques used in the testing process
• Discuss better testing methods for the robot design
• Recommend if any design improvements should be made
• State if the robot got to the last tile, using hardcoding or sensor code

Remember: Lab notes must be taken. Experimental details are easily forgotten unless written down. EG1004 Lab Notes paper can be downloaded and printed from the EG1004 Website. Use the lab notes to write the Procedure section of the lab report. At the end of each lab, a TA will scan the lab notes and upload them to the Lab Documents section of the EG1004 Website. One point of extra credit is awarded if the lab notes are attached at the end of the lab report. Keeping careful notes is an essential component of all scientific practice.

Team PowerPoint Presentation

Follow the presentation guidelines laid out in the EG1004 Lab Presentation Format in the Technical Presentations section of the manual. When preparing the presentation, consider the following points.

• Based on the test results, what is the product's marketability?
• If determinations were that the robot(s) did not meet the standard, what improvements could be made?
• Why is product evaluation important?
• Discuss why quality improvement is important
• Discuss the importance of utilizing sensors

References

Oakes, W.C., L.L. Leone, and C.G. Gunn. 2002. “Engineering Your Future.” Okemos, MI. Great Lakes Press

What Is website. TechTarget Network. Retrieved July 29, 2003