A pre-built robot will be evaluated for accuracy and precision by utilizing two different sets of code, the first without the use of sensors and the second with sensors. The robot will be tested on one of the three provided mini-courses. The data from the product evaluation of both sets of code will determine if the robot should or should not use sensors in navigating the mini-course.
Product evaluation is a common task for an engineer and for companies. Sometimes the product evaluated will be a company's product, and sometimes companies will be evaluating a competitor's design. In either case, this evaluation will use a standard (an accepted value), determine an average, and calculate accuracy and precision. 
Professional engineers must be sure of standards. For the purpose of this lab, the EG1003 standard will be 80% precision and accuracy. This means that the product has to achieve a percent accuracy of 80% and meet a tolerance of 1cm for distance and 10 degrees for angle. 
When taking measurements, the first step is to determine an average (Equation 1).
Equation 1: Average.
In this equation, Ap is the average, P1, P2, P3, ..., Pn are the results of the tests, and N is the number of tests performed.
The average is compared to the standard. Accuracy is a variable the must always be considered. The degree of accuracy required will allow the results to be verified. Engineers often work on projects that affect people's safety so it is critical that the solutions to the problems solved by technical professionals are accurate.
The comparison of the average to the standard is called accuracy, defined as Acc in Equation 2.
Equation 2: Accuracy.
In Equation 2, Ps is the standard value, and Ap is the value measured. Accuracy is always a positive number, so use the absolute value in case Ps is less than Ap.
Precision is the repeatability of a result, or how close the results are to each other, defined as Prec in Equation 3.
Equation 3: Precision.
In Equation 3, Prec is precision, Phigh is the highest data value and Plow is the lowest data value. Like accuracy, precision is always a positive number, so use an absolute value here as well.
A product can be accurate but imprecise, or precise but inaccurate (Figure 1). The goal is to achieve a combination of the highest accuracy and precision. In order for engineers to make a recommendation based on experimental test data, it is important to determine how a product compares to a standard. Engineers need these benchmarks to determine the quality of a product.
By determining the percent accuracy of an experiment, and comparing this percentage to a standard, it can be determined, based on experimental evidence, if the product has passed or failed. The equation for percent accuracy is:
Equation 4: Percent accuracy.
In Equation 4, %Acc is the percent accuracy, Ps is the standard value, and Ap is the value measured.
Precision shows how precise a product is compared to the standard.
Adequate precision is determined by an acceptable tolerance, set by the manufacturer. In this lab, the precision of the robot must be compared to the tolerance allowed:
- Distance Test:
- Angle of Deviation Test:
Calculating percent accuracy and precision and then comparing those results to a standard determines the quality of the product.
Quality improvement is a process of analyzing a design and testing it through physical modeling, computer simulation, or mathematical modeling. Quality improvement ensures that a product will perform as expected and allows improvements to be made to the design.
When deciding how to improve the design in this lab, keep these questions in mind:
- Does the robot perform to standard?
- If no, what can be done to improve its functionality?
- If yes, what can be done to improve its performance beyond the standard?
Changes will be implemented after testing the initial code.
A sensor is a device that measures a specific value and is used as an input. Different properties can be measured by the EV3 sensors, such as light intensity, angle, or distance. The three sensors used in this lab are the light sensor, gyro sensor, and an ultrasonic sensor.
Switches and Loops
Switches and loops are used to tell the robot what actions to perform depending on the inputs from its surroundings, measured by the sensors.
Materials and Equipment
- A Mindstorms kit with pre-made robot
- A computer with Mindstorms software
- A mini-course with accuracy markings
Download the following files:
Evaluate two types of code for a robot by creating a line of code without the use of sensors, and then a separate line of code using sensors. For each trial, record the distance from the wall and angle of deviation from the center line. Test the first code by running four trials of Section 1 of the mini-course as marked on the physical course. Create a second line of code using switches and loops that incorporate the types of sensors found on the robot and features on the mini-course. Test the robot again with four trials of Section 1. This part of the procedure will have a total of 8 trials. Download the pre-made datasheet and fill in the data for each trial.
Part 1: Hardcoding
Based off of your observations, make design changes to the robot in order to run trials on the first section (Section 1) of the mini-course.
Every run, unless the robot goes off the course, counts as a trial. There is only one test run allowed after which all runs will be recorded as trials. When troubleshooting the code, please cross check the port number on the code and the port the sensor/motor is connected to before anything else.
Using the modified design, conduct four trials of the robot using a code without sensors to travel in a straight line across the first four tiles, turn 90 degrees to the right, and stop 5 cm from the wall. The values used in the sections of code below are only examples - the actual values must be determined through experimentation.
- Start the Mindstorms EV3 software.
- Download the zipped folder containing the Mindstorms program.
- The hardcoding program should look like the one pictured below:
- Please read the comments before running the program.
- Change the ports to the configuration on your robot.
- Run a sample trial to understand the code.
- Evaluate the number of rotations by measuring the total distance that the robot has to move for a certain block of code and divide that by the perimeter of the wheel.
- Change the values to the evaluated numbers.
- Try to stop 5cm before the wall.
- Run the program for four trials. For each trial, record the distance and angle using pre-made markings on the sides of the tile.
- Distance is measured as the distance from the wall for which the ruler on the side of the tile can be used. (Denote it as y)
- Angle of deviation is calculated in the following way:
- Measure deviation from the center of the wall. (Denote it as x)
- tan^-1 (x/y)
- If you are confused as to how to calculate the angle of deviation, ask your TA for help.
Part 2: Sensors
Using the results of the original robot, incorporate the use of sensors in the robot's code. The following steps outline how to build a line-following code for the robot to stay on track. The values used in the sections of code below are only examples; the actual values must be determined through experimentation.
- Move to the Software Trials tab
- The program should look like this:
- Please read the comments before running the program
- Change the ports to the configuration on your robot
- Run a sample trial to understand the code
- Using the port view shown below, set the readings on the Mindstorms code for the light sensors and the gyro sensors blocks to the appropriate values. Adjust Rotations as done in the previous part. Ask a TA for assistance if you don't know what values to use.
- Take the same measurements as for Part 1: Harcoding
- Tabulate results neatly in the Datasheet
Tabulation of Results
All data must be inputted into the Datasheet. From the datasheet, determine if the EG standards were met for accuracy and precision; recorded precision must be divided by 2 to get the corresponding tolerance:
|Angle of Deviation Test|
|Angle of Deviation Test|
Part 3: Robot Competition
The robot competition consists of a single trial of the entire course to travel the most tiles as possible. Again, the values used in the sections of code below are only examples - the actual values must be determined through experimentation. The averages from the best type of coding (Part 1 vs Part 2) are also used to calculate the competition ratio.
The robot competition consists of attempting to travel the most amount of tiles possible in a single run. Only one trial counts as valid and must verbally be declared before the run starts.
- Recreate the line following code to allow the robot to travel up the ramp. Test different values of power to overcome the ramp. The end statement should cause the robot to stop 10 cm away from the wall.
- Notice that everything until the second loop is a replica of the Software Trial code. You only have to add the last loop. Ask a TA for assistance if you don't know how to do it.
- Only parts provided in the EV3 kits are allowed (no regular Lego's)
- No changes in physical robot design
- No "re-tests"
- The trial used for the competition must be declared before the robot runs and cannot be disputed
- A robot counts as clearing a tile if it stops on the tile with at least 50% of the robot on the tile
The team with the highest competition ratio wins the competition. Adistance is the robot's average distance from the wall for four trials, Aangle is the robot's average angle of deviation for four trials, and Ntiles is the number of tiles cleared during the single competition run:
The lab work is now complete. Please clean up the workstation. Return all unused materials to the TA. Refer to the Assignment section for the instructions needed to prepare the lab report.
Individual Lab Report
Follow the lab report guidelines laid out in the page called Specifications for Writing Your Lab Reports in the Technical Communication section of this manual. The following points should be addressed in the appropriate section of the lab report.
- Discuss product evaluation and the need for it
- Discuss quality improvement and its importance
- Include examples of product evaluation and quality improvement
- Discuss a standard, an average, accuracy, and precision; include their equations
- Describe loops and switches and how they are useful in coding
- Describe the light sensor, gyro sensor, and ultrasonic sensor used in the lab and their optimal position
- Discuss the importance of tolerance and percent accuracy in determining product quality
- Discuss the EV3 kit and software used
- Describe Mindstorms programming for only the sensors coding portion
- Discuss how robot behavior changed after sensors were added and code was revised
- Compare the final code with sensors and final code without sensors; determine which is better.
- Substantiate the decision with data, observations, and other facts collected
- Make a final evaluation of the product: should it go into production? Describe its accuracy and precision using the data
- Assess the techniques used in the testing process
- Discuss better testing methods for the robot design
- Recommend if any design improvements should be made
Remember: Lab notes must be taken. Experimental details are easily forgotten unless written down. EG1004 Lab Notes Paper can be downloaded and printed from the EG1004 Website. Use the lab notes to write the Procedure section of the lab report. At the end of each lab, a TA will scan the lab notes and upload them to the Lab Documents section of the EG1004 Website. One point of extra credit is awarded if the lab notes are attached at the end of the lab report. Keeping careful notes is an essential component of all scientific practice.
Team PowerPoint Presentation
Follow the presentation guidelines laid out in the page called EG1003 Lab Presentation Format in the Introduction to Technical Presentations section of this manual. When preparing the presentation, consider the following points:
- Based on the test results, what is the product's marketability?
- If determinations were that the robot(s) did not meet the standard, what improvements could be made?
- Why is product evaluation important?
- Discuss why quality improvement is important
- Discuss the importance of utilizing sensors