Product Evaluation and Quality Improvement

This page is currently under construction.

1 OBJECTIVES

The experimental goal of this lab is for your team to evaluate a robot design and see if it meets minimum standards for accuracy and precision and to make quality improvements to the design.         Lastly, you will determine the robot’s function. To do this your team will build and program a robot, and test it against the provided EG1004 standard. Next, make improvements on the design, then take apart the robot, measure the gear and velocity ratios, and describe the gear train and overall improvements that were made

The concepts of Product Evaluation and Quality Improvement will be defined. The importance of accurate measurements will be demonstrated and show how they relate to both precision and accuracy. From the data collected, a detailed analysis will be made of the design and improvements will be implemented and tested. A final analysis will be made upon disassembly.

2 OVERVIEW

Your team has been hired by an engineering firm to evaluate a robot design and make improvements.  A model will be assembled using a NXT robotics kit. The firm has specified the robot design they would like you to build and test. You will also make quality improvements on the robot design. Disassemble the robot and analyze the component parts. Your assignment is to build and test the design. Record the data on a spreadsheet. Make design changes and retest the robot with the design changes and record the data. Finally, disassemble the robot and evaluate for optimal design.

Design Analysis

Product evaluation is a common task for an engineer. Sometimes the product you evaluate will be one your team built, and sometimes you will be analyzing a competitor's design. In either case, you will need to decide on a standard (an accepted value), determine an average, and calculate accuracy and precision.¹

Professional engineers must be sure of standards. For the purpose of this lab, the EG1004 Standard that you will be testing to is 80% precision and accuracy. This means that the product you will be testing has to pass 80% of the tests that it undergoes. It is up to you to verify that your solutions are accurate based on the standard for your design project. ¹

When taking measurements, the first step is to determine an average.

$Average\left(A_{p}\right)={\frac {P_{1}+P_{2}+P_{3}+...+P_{n}}{N}}\,$

In this equation, $A_{p}\,$ is the average, $P_{1}\,$, $P_{2}\,$, $P_{3}\,$, ..., $P_{n}\,$ are the results of the tests and $N\,$ is the total number of tests to be performed.

The average is then compared to the standard. Accuracy is a variable you must always consider. The degree of accuracy required will allow you to verify your results. Engineers often work on projects that affect people's safety. Therefore, it is critical that the solutions to the problems you will solve as technical professionals are accurate.

The comparison of the average to the standard is called accuracy, defined as $Acc\,$ in the following equation:

$Acc=\left|P_{s}-A_{p}\right|\,$

In this equation, $P_{s}\,$ is the standard value, and $A_{p}\,$ is the value you measured. Accuracy is always a positive number, so we take the absolute value in case $P_{s}\,$ is less than $A_{p}\,$.

Precision is the repeatability of a result, or how close the results are to each other, defined as Prec in the following equation:

$Prec=\left|P_{high}-P_{low}\right|\,$

In this equation, $Prec\,$ is Precision, $P_{high}\,$ is the highest data value and $P_{low}\,$ is the lowest data value. Like accuracy, precision is always a positive number, so we use an absolute value here as well.

A product test can be accurate but imprecise, or precise but inaccurate. The goal is always the combination of accuracy and precision. In order for engineers to make a recommendation based on experimental test data, it is important to determine how a device measures against a standard. Engineers need these benchmarks to determine the quality of a product.

By determining the percent accuracy of an experiment, and comparing this percentage to a standard, you can determine, based on experimental evidence, whether the prototype has passed or failed. The equation for percent accuracy is:

$\%Acc=\left(1-{\frac {\left|P_{s}-A_{p}\right|}{P_{s}}}\right)\times 100\%\,$

In this equation, $\%Acc\,$ is the percent accuracy, $P_{s}\,$ is the standard value, and $A_{p}\,$ is the value you measured.

Percent accuracy allows an engineer to determine how precise a device is compared to the standard. The equation for percent precision is:

$\%Prec=100e^{-BP}\,$

In this equation, $\%Prec\,$ is the percent precision, $B\,$ is the exponential decay factor, and $P\,$ precision you measured.

Calculating percent accuracy and percent precision and then comparing your results to a standard will allow you to determine the quality of your prototype.

Adapted from http://www.whatis.com.¹

Quality Improvement

Quality Improvement is a process of analyzing a design and testing it either through physical modeling, computer simulation or mathematical modeling. Quality improvement ensures that a product will perform as expected and allows us to improve upon the design.

For this lab, you are given a design that will be using a model. When deciding how to improve the design, keep these questions in mind:

• Does the robot perform to standard?
  • If no, what can be done to improve its functionality?
  • If yes, what can be done to improve its performance beyond the standard?

Your changes will be implemented after testing the initial design.

Reverse Engineering

Reverse engineering software involves reversing a program's machine code to obtain the original source code or determine the file structures the program uses. This needs to be done when source code is lost or is not available. For example, when a project is reactivated after being dormant for years to add new features to a product, the original design documentation may have disappeared. Also, when companies are writing a program that must interface to a second company's software, the second company may not have any incentive to cooperate by describing how the data in the files they use are organized, forcing the first company to deduce it on their own. Sometimes, this process is also under­taken as a way to improve the performance of a program, to fix a bug, or to find a virus. When the source code is obtained in this way for any of these reasons it is legal and necessary. Reverse engineering software in order to copy it constitutes a copy­right violation and is illegal.

Hardware reverse engineering involves taking apart a device to see how it works. If a processor manufacturer wants to see how a competitor's processor works, the company can purchase the processor, disassemble it, and then make a new processor similar to it. In some countries, this process is illegal. Hardware reverse engineering is quite expensive and requires an expert in the field.

When you begin the process of reverse engineering your robot, identifying the gear trains and the individual gears used in the original design and in the final design is critical to your understanding of the robot's operation.

There are two types of gear trains: simple and compound. A simple gear train has its gears arranged in a line (see Figure 1).

Figure 1: Simple gear train

Compound gear trains use axles to connect the component gears (see Figure 2).

Figure 2: Compound gear train

Gear trains have measurable characteristics known as gear ratio and gear velocity. These characteristics are inversely proportional. Gear ratio equals output over input, while velocity ratio equals input over output. If multiple gears are being used, the overall gear ratio and velocity ratio is the product of these ratios for each individual gear. To determine input and output values, engineers would count the number of teeth on each gear. However, we can get an acceptably accurate value by measuring each gear's radius instead.

For example, to compute the gear ratio of the gear trains above, use these formulas, where the values are the radii of the gears in the gear train:

$GearRatio={\frac {2}{1}}\times {\frac {4}{2}}=4\,$ Simple gear train

$GearRatio={\frac {N_{2}}{N_{1}}}\times {\frac {N_{4}}{N_{3}}}\times {\frac {N_{6}}{N_{5}}}\times {\frac {N_{8}}{N_{7}}}\,$ Compound gear train

For example, to compute the velocity ratio of the gear trains above, use these formulas:

$VelocityRatio={\frac {1}{2}}\times {\frac {2}{4}}={\frac {1}{4}}\,$ Simple gear train

$VelocityRatio={\frac {N_{1}}{N_{2}}}\times {\frac {N_{3}}{N_{4}}}\times {\frac {N_{5}}{N_{6}}}\times {\frac {N_{7}}{N_{8}}}\,$ Compound gear train

It is important to note that the input is the source of rotation in a system, like a motor, and the output is the final gear of rotation in a system, like a wheel.

Figure 3: Types of Gears
Figures 3a-d courtesy of http://www.howthingswork.com
Figure 3f courtesy of Honda Motor Company, Inc.

Figure 3a: Rack gear. Its purpose is to change rotation into linear motion.	Figure 3b: Crown gear. Its purpose is to change the direction of rotation perpendicularly.
Figure 3c: Spur gear. Its purpose is to transmit torque through the gear train.	Figure 3d: Worm gear. Its purpose is to change the direction of rotation perpendicularly.
Figure 3e: Idler gear. Its purpose is to transfer force while not affecting the gear or velocity ratios.	Figure 3f: Pulley gear. Its purpose is to transmit rotation from one point to one or more other points.

Your objective as you reverse engineer your robot is to understand how it works and to calculate the gear and velocity ratios for its drive train (remember the wheels are part of the drive train). Remember that engineers reverse engineer products to reproduce or improve them. As you disassemble your device, consider how you would improve your robot's design.

3 YOUR ASSIGNMENT

Individual Lab Report

Follow the lab report guidelines laid out in the page called Specifications for Writing Your Lab Reports in the Technical Communication section of this manual. As you write, the following discussion points should be addressed in the appropriate section of your lab report:

• Discuss product evaluation and the need for it.
• Discuss Quality Improvement and its importance.
• Explain why reverse engineering is important.
• Discuss standard, average, accuracy, and precision.
• Discuss the NXT kit and software used.
• Assess the techniques used in the testing process.
• Discuss better testing methods for the robot design.
• Describe how the components fit together and how the device was constructed.
• Describe and show how you calculated the gear and velocity ratios, and discuss their importance in the overall robot design as well.
• Analyze the original robot design and the final design then make a decision which design to go with. Remember to substantiate your decision with the data and facts collected.
• Make a final evaluation of the product: should it go into production? Describe its accuracy and precision.
• Discuss the importance of $\%Prec\,$ and $\%Acc\,$ in determining product quality.
• Recommend if any design improvements should be made.

Team PowerPoint Presentation

Follow the presentation guidelines laid out in the page called [[EG1004 Lab Presentation Format]] in the Introduction to Technical Presentations section of this manual. When you are preparing your presentation, consider the following points:

• Based on the test results, what is your determination for the product's marketability?
• If determinations were negative, what improvements could be made?
• Why is product evaluation important to you?
• Discuss why Quality Improvement is important.
• Discuss the importance of gear and velocity ratios.

4 MATERIALS AND EQUIPMENT

• Mindstorms Kit
• Computer with Mindstorms Software
• Protractor and Ruler
• Graph Paper

Remember:You are required to take notes. Experimental details are easily forgotten unless written down. EG Standard Note Paper can be downloaded and printed from here: https://eg.poly.edu/downloads/Note_paper.zip. Use your lab notes to write the Procedure section of your lab report. At the end of each lab your TA will scan your lab notes and upload them to http://eg.poly.edu. You must attach your lab notes at the end of your lab report (use the "Insert Object" command in MS Word after your Conclusion). Keeping careful notes is an essential component of all scientific practice.

5 PROCEDURE

Problem Statement

You have been hired by an engineering firm to build and evaluate a robot design by building a specified robot design. You have been asked to test the robot design, make improvements on the design and retest it. While taking it apart, analyze the drive train.

1. Build a robot according to the lab handout provided by your TA.
2. Start the Mindstorms Program. On the welcome screen type in a name for your program and press Go.

   

3. Click on Complete Pallete to see all the programming icons. Turn off Robot Educator by clicking the small X.

   

4. Your screen should look like this. You are now ready to write a program.

   

5. Click on Actions, then on the Motor icon to select it.

   

6. Place the motor icon on to the "start" square.

   

7. Click the motor icon to see its properties. Here you can define exactly how you want the motor to spin and when to stop.

   

8. Continue writing your program by placing icons after each other. The execution of your program will run from left to right. In the sample program below, motor A will spin forward for 5 seconds, Stop, Wait 5 seconds, and then spin backwards for another 5 seconds.

   

9. For this lab you can use the Move icon to easily instruct your robot to move.

   

10. Whenever you need to, you can use Robot Educator to quickly learn how to program in Lego Mindstorms.

    

Note: Program and test your robot for each test specified. To do this, make a list of the tests in your lab notebook. Brainstorm what you want the robot to do in each test. Then, use your notes to write your program. Record the standard, average, accuracy, and precision. If your robot does not operate correctly, troubleshoot the problem. Your TA must initial your original data.

Distance Test

Note: Use the Metric system for this test.

In this test, the robot will go forward a set distance in a specified time. You are to test the robot's accuracy and precision for this distance. Before testing begins the standard must be found. To find the standard, program the robot to go forward for 5 seconds and measure the distance it travels. Divide your result by 5, and then multiply by 4. This number is your standard. Once the standard has been obtained, the testing can begin.

Warning: Read the section on the #Angle of Deviation Test before performing this test.

1. Upload a program that will set the robot to travel for 4 seconds.
2. Measure the distance traveled.
3. Repeat this test 5 times.

Angle of Deviation Test

Note:'' This test can be performed at the same time as the Distance Test. Since the robot is expected to travel in a straight line, the standard is 360°.

In this test, the robot will move forward for 4 seconds and the angle of deviation will be recorded. The angle of deviation is the angle between a straight line and a line drawn from the starting point to where the robot finishes.

1. Use a protractor to draw a 0° line from the start point.
2. Mark the start point and place your robot there. Identify a reference point on the robot.
3. Run the Distance Test.
4. Make a mark at the point where the robot finished, and draw a line from the start point to this mark.
5. Measure the positive angle of deviation.
6. Repeat this procedure 5 times.

Quality Improvement

1.)    'Assess Robot design from data collected and from observations

2.)    'Make necessary adjustments

3.)    'Retest robot and collect data

4.)    'Take pictures of both original design and finalized design

 

Reverse Engineering

1. Visually assess the robot's design. Consider how it works.
2. Sketch the front, top, and most detailed side of the robot. Be sure to include dimensions in your sketch.
3. Also sketch the most detailed view of the gear train.
4. Before disassembling your robot, ask your TA to take a picture of it.
5. Disassemble the robot. Analyze its inner workings and sketch the gear train(s). Note: All sketches in EG1004 must be done in pencil.
6. Have all sketches and original data signed by your TA.

You now have measurements for both tests you have performed and an analysis of the drive train of your robot. You must analyze this data. Calculate the average, accuracy, and precision for each test. Compare your results to the standard in order to determine if your robot is accurate and precise. Label all major components and describe their functions. Calculate the gear and velocity ratios. You must also calculate percent accuracy and percent precision. Use this formula to calculate percent accuracy:

<img border=0 width=196 height=59 id="Picture 141" src="Quality_files/image003.gif" alt="Image:Lab hardsyn 13.gif">

In this equation, %Acc is the percent accuracy, P_S is the standard value, and A_P is the value you measured.

Tabulation of Results

To decide if the product is ready to manufacture, arrange your data in a table designed to calculate its precision and accuracy. Remember it must pass all the tests at a rate of at least 80%.

	Accuracy	%Acc	Pass/Fail	Precision	%Prec	Pass/Fail
Distance Test
Angle of Deviation Test
Rotational Test

 Retest with new design using the same table, record your data. Your lab work is now complete. Please clean up your workstation. Return all unused materials to your TA. Refer to section 3 Your Assignment for the instructions you need to prepare your lab report.

Footnotes

¹ Oakes, W.C., L.L. Leone, and C.G. Gunn, Engineering Your Future w:st="on">MI: Great Lakes Press, 2002

¹ What Is website. TechTarget Network. Retrieved July 29th, 2003.