Confidence Levels

 

Data review, data analysis, data migration

Question

I would like to confirm if ASQ Z1.4-2008 attribute tables are calculated based on 95% confidence level? I am using Table II-A, on page 11.

Answer

ANSI/ASQ Z1.4 tables are not technically calculated based on a 95% confidence level.  The technical definition of AQL is the quality level that is the worst tolerable process average when a continuing series of lots is submitted for acceptance sampling.  Some interpret it to mean if a lot has AQL percent defective or less, a lot would have a high probability of being accepted based on the sampling plan.  The standard does not specify the probability of acceptance explicitly.  The operating characteristic curve (OC Curve and the tables define the AQL as the percent defective that has a 95% probability of acceptance.  So though it is not a 95% confidence level, it is a 95% probability of acceptance.

Steven Walfish

For more information about AQL, please view the resources here.

DPPM Calculation

Chart, graph, sampling, plan, calculation, z1.4

Question

Recently, there is a debate in my organization about Defective Parts Per Million (DPPM) computation.
Camp 1 – DPPM = (No of parts rejected / No of parts inspected) * 1,000,000
Camp 2 – DPPM = (No of parts rejected / No of parts received) * 1,000,000
We perform sampling inspection based on AQL.
Camp 1 insists they are correct and likewise for Camp 2.  Which is correct or more appropriate to reflect supplier quality?

Answer

This is not an uncommon question. If you look at the standard, they define the % nonconforming as the number of parts nonconforming/number of parts inspected x 100. If you are looking at DPPM, instead of multiplying by 100, you put in 1,000,000. This means that by your definition, Camp 1 is correct. This is also what was intended by the creators of the sampling scheme.

Jim Bossert
Sr Performance Improvement Specialist
JPS Hospital
ASQ Fellow, CQE, CQA, CMQ/OE, CSSBB, CSSMBB
Fort Worth, TX

AQL for Electricity Meter Testing

Chart, graph, sampling, plan, calculation, z1.4

Q: We have implemented a program to test electricity meters that are already in use. This would target approximately 28,000 electricity meters that have been in operation for more than 15 years. Under this program, we plan to test a sample of meters and come to a conclusion about the whole batch  —  whether replacement is required or not. As per ANSI/ISO/ASQ 2859-1:1999: Sampling procedures for inspection by attributes — Part 1: Sampling schemes indexed by acceptance quality limit (AQL) for lot-by-lot inspection, we have selected a sample of 315 to be in line with the total number of electricity meters in the batch.

Please advice us on how to select an appropriate acceptable quality level (AQL) value to accurately reflect the requirement of our survey and come in to a decision on whether the whole batch to be rejected and replaced. Thank you.

A: One of the least liked phrases uttered by statisticians is “it depends.” Unfortunately, in response to your question, the selection of the AQL depends on a number of factors and considerations.

If one didn’t have to sample from a population to make a decision, meaning we could perform 100% inspection accurately and economically, we wouldn’t need to set an AQL. Likewise, if we were not able to test any units from the population at all, we wouldn’t need the AQL. It’s the sampling and associated uncertainty that it provides that requires some thought in setting an AQL value.

As you may notice, the lower the AQL the more samples are required. Think of it as reflecting the size of a needle. A very large needle (say, the size of a telephone pole) is very easy to find in a haystack. An ordinary needle is proverbially impossible to find. If you desire to determine if all the units are faulty or not (100% would fail the testing if the hypothesis is true), that would be a large needle and only one sample would be necessary. If, on the other hand, you wanted to find if only one unit of the entire population is faulty, that would be a relatively small needle and 100% sampling may be required, as the testing has the possibility of finding all are good except for the very last unit tested in the population.

AQL is not the needle or, in your case, the proportion of faulty fielded units. It is the average quality level which is related to the proportion of bad units. The AQL is fixed by the probability of a random sample being drawn from a population with an unknown actual failure rate of the AQL (say 0.5%), creating a sample that has a sample failure rate of 0.5% or less. We set the probability of acceptance relatively high, often 95%. This means if the population is actually mostly as good as or better than our AQL, we have a 95% chance of pulling a sample that will result in accepting the batch as being good.

The probability of acceptance is built into the sampling plan. Drafting an operating characteristic curve of your sampling plan is helpful in understanding the relationship between AQL, probability of acceptance, and other sampling related values.

Now back to the comment of “it depends.” The AQL is the statement that basically says the population is good enough – an acceptable low failure rate. For an electrical meter, the number of out of specification may be defined by contract or agreement with the utility or regulatory body. As an end customer, I would enjoy a meter that under reports my electricity use as I would pay for less than I received. The utility company would not enjoy this situation, as it provides their service at a discount. And you can imagine the reverse situation and consequences. Some calculations and assumptions would permit you to determine the cost to the consumers or to the utility for various proportions of units out of specification, either over or under reporting. Balance the cost of testing to the cost to meter errors and you can find a reasonable sampling plan.

Besides the regulatory or contract requirements for acceptable percent defective, or the balance between costs, you should also consider the legal and publicity ramifications. If you accept 0.5% as the AQL, and there are one million end customers, that is 5,000 customers with possibly faulty meters. What is the cost of bad publicity or legal action? While not likely if the total number of faulty units is small, there does exist the possibility of a very expensive consequence.

Another consideration is the measurement error of the testing of the sampled units. If the measurement is not perfect, which is a reasonable assumption in most cases, then the results of the testing may have some finite possibilities to not represent the actual performance of the units. If the testing itself has repeatability and reproducibility issues, then setting a lower AQL may help to provide a margin to guard from this uncertainty. A good test (accurate, repeatable, reproducible, etc.) should have less of an effect on the AQL setting.

In summary, if the decision based on the sample results is important (major expensive recall, safety or loss of account, for example), then use a relatively lower AQL. If the test result is for an information gathering purpose which is not used for any major decisions, then setting a relatively higher AQL is fine.

If my meter is in the population under consideration, I am not sure I want my meter evaluated. There are three outcomes:

  • The meter is fine and in specification, which is to be expected and nothing changes.
  • The meter is overcharging me and is replaced with a new meter and my utility bill is reduced going forward. I may then pursue the return of past overcharging if the amount is worth the effort.
  • The meter is undercharging me, in which case I wouldn’t want the meter changed nor the back charging bill from the utility (which I doubt they would do unless they found evidence of tampering).

As an engineer and good customer, I would want to be sure my meter is accurate, of course.

Fred Schenkelberg
Voting member of U.S. TAG to ISO/TC 56
Voting member of U.S. TAG to ISO/TC 69
Reliability Engineering and Management Consultant
FMS Reliability
www.fmsreliability.com

Related Content:

To obtain more resources on sampling and statistics, read the featured ASQ journal articles below or browse ASQ Knowledge Center search results.

Rethinking Statistics for Quality Control, Quality Engineering

Setting Appropriate Fill Weight Targets — A Statistical Engineering Case Study, Quality Engineering

Compliance Testing for Random Effects Models With Joint Acceptance Criteria, Technometics

 

ISO 2859-3 Skip-lot Sampling 5.1.1, 5.2.1

Suppliers, supplier management

Q: Our quality team is trying to improve inspection efficiency and enhance supplier management by employing ISO 2859-3:2005 Sampling procedures for inspection by attributes — Part 3: Skip-lot sampling procedures.

Here are two questions on product qualification related to clause 5.2.1 Generic requirements for product qualification.

1. The standard requires that:

b) The product shall not have any critical classes of nonconforming items or nonconformities.

First, my understanding is that the risk level with any potential failure or nonconforming of the product should be low to customer — is this correct? Second, if a candidate product carries some critical features (dimension of mechanical product), but also carries a number of low risk features, can we apply the skip lot concept only to the non-critical features? And continue to perform lot-by-lot inspection with critical features? We are concerned the definition of “product” in the standard is a generic term and could be interpreted as feature of a physical product.

2. The standard requires that:

c) The specified AQL(s) shall be at least 0,025 %.

Does this mean the AQL value should be less than or greater than 0.025%? I assume “greater.” In our company, the most often used is AQL 1.0 and AQL 2.5, which I think meets the requirement.

We would greatly appreciate your help.

A: My name is Dean Neubauer and I am the U.S. Lead Delegate to Subcommittee 5 on Acceptance Sampling and co-author of Acceptance Sampling in Quality Control, 2nd edition., 2009. CRC Press. I hope I can help you.

Let’s start with question 1.

The general idea of skip-lot sampling is to reduce the number of times incoming lots inspected due to exceptional quality on behalf of the supplier.  ISO 2859-3 states this in the beginning as:

The purpose of these procedures is to provide a way of reducing the inspection effort on products of high quality submitted by a supplier who has a satisfactory quality assurance system and effective quality controls.

The reduction in inspection effort is achieved by determining at random, with a specified probability, whether a lot presented for inspection will be accepted without inspection.

A skip-lot sampling plan is also known as a cumulative results plan.  In general, such plans require certain assumptions to be met regarding the nature of the inspection process:

  • The lot should be one of a continuing series of lots
  • We expect these lots to be of the same quality
  • The consumer should not expect that any lot is any worse than any of the immediately preceding lots
  • The consumer must have confidence in the supplier not to pass a substandard lot even though other lots are of acceptable quality

Under these conditions, we can use the record of previous inspections as a means of reducing the amount of inspection on any given lot.  ISO 2859-3 states the above in 5.1.1 Requirements for supplier qualification:

The requirements for supplier qualification are as follows.

a) The supplier shall have implemented and maintained a documented system for controlling product quality and design changes. It is assumed that the system includes inspection by the supplier of each lot produced and the recording of inspection results.

b) The supplier shall have instituted a system that is capable of detecting and correcting shifts in quality levels and monitoring process changes that may adversely affect quality. The supplier’s personnel responsible for the application of the system shall demonstrate a clear understanding of the applicable standards, systems and procedures to be followed.

c) The supplier shall not have experienced any change that might adversely affect quality.

The underlying assumption here is that the supplier quality is exceptional (low nonconforming level).  The skip-lot plan is applied to each characteristic, or feature, separately.  If several characteristics are present, then try to test for at least one of them.  In your situation, if you have critical characteristics you should not be doing skip-lot inspection due to the risk of ignoring (skipping) a potentially dangerous lot.  On the other hand, you can use a skip-lot plans for non-critical (major and minor) nonconformities and they will have different AQL levels associated with them.  Your listing of AQLs of 1.0% and 2.5% are typical for major and minor nonconformities.  Critical defects will typically have an AQL less than 1.0%, such as 0.25% to 0.65% (0% is preferred but theoretically unattainable as you would have to do a perfect 100% inspection, i.e., no inspection error).

The subclause referenced in question 2 states that the AQL level must be greater than or equal to 0.025%. Your levels of 1.0% and 2.5% can be used.

Dean Neubauer

U.S. Lead delegate for Subcommittee 5 on Acceptance Sampling on ISO Technical Committee 69 on Applications of Statistical Methods.

Operational Qualification (OQ) Challenges; Cpk vs. AQL

Q: We’re completing a validation of a plastic extrusion process, which has raised a few questions with me.

This validation exercise encompasses the installation qualification (IQ), operational qualification (OQ), and the performance qualification (PQ). The IQ is self explanatory, but the OQ is challenging. The process is dependent on the batch resin properties which vary enough that the extrusion processing parameters cannot be setup where good parts are always produced. One resin batch can use processing parameters that will not work with the next batch. A justification will be written and included in the documentation package to explain this. Does the inability of defining an operating window void or limit the validation?

My second question has to do with PQ acceptance criteria. The PQ will be three production runs using at least two different material resins (the largest source of variation). While production acceptance will be on an AQL=1.0, C=0 basis, these initial validation lots will be accepted on a process capability index (Cpk) level. While on the surface the acceptance difference may seem benign, it is causing some changes. The tolerance is such that the process routinely passes the Acceptable Quality Limit (AQL) test criteria but fails a Cpk requirement. Is it possible to accept PQ runs as they would be accepted in production?

A related question is the power of a Cpk vs. an AQL sampling plan. A Cpk value can be calculated using the same number of samples on a 100-foot run vs. a 10,000-foot run, while an AQL sampling plan is size dependant. Is there a criterion on sample size or a rule of thumb as to when one plan should be used over another?

A: First, the plastic extrusion process is always a tricky one to qualify simply because each new batch of resin always requires adjustments no matter how controlled the storage conditions are. So yes, you will have to define what adjustments your organization has to make and how big an operating window you need to transition from batch to batch.  If you can demonstrate that it can be resolved within a certain time (say, 15-30 minutes), then it should be ok for validation.  This is assuming that the customer is in agreement with what your company is doing.

Cpk formula, Cpk indexThe second question is a bit more difficult in that the Cpk is assuming that the process is in control and performing at a steady rate.  Cpk is a long term measure and requires the use of control charts to really control the process.  You may be able to work with your customer on help to get validated to the Cpk requirement, but you have to show the plan to get here.  In the past, some customers have been willing to provide an extended period to attain validation. You may want to talk to your customer representative to find out what help they can provide.

The third question gets to the fundamental heart of the situation: the question of using Cpk vs. AQL.  Cpk is a measure of process capability and AQL is a measure of long-term, outgoing quality.  Are they the same?  On some studies I did early on with Cpk and specifications, it was not always clear.  I have not seen any criterion on sample size on when to use Cpk vs. AQL.

Jim Bossert
SVP Process Design Manger, Process Optimization
Bank of America
ASQ Fellow, CQE, CQA, CMQ/OE, CSSBB, CMBB
Fort Worth, TX

Z1.4 2008: AQL, Nonconformities, and Defects Explained

Pharmaceutical sampling

Q: My question is regarding the noncomformities per hundred units and percent nonconforming.  This topic is discussed in ANSI/ASQ Z1.4-2008 Sampling Procedures and Tables for Inspection by Attributes under sections 3.2 and 3.3 on page 2.  Regardless of the explanations provided, I find myself puzzled as to what the following numbers refer to in “Table II-A– Single sampling plans for normal inspection (Master table).”

Specifically, I am having problems understanding the following unit numbers just above the Acceptance and Rejection numbers (example, 0.010, 0.015, 0.025, 1000).  Do these represent percent noncomformities and if so,  does 0.010 = 0.01%, and conversely, how can 1000 = 1000%?

As you may see, I am very confused by these numbers, and I was hoping to have some light shed on this subject. Thank you for your answers in advance.

A: The numbers on the top of the table are just as the questioner stated: .0.010 = .01% defective.  That is the acceptable quality limit (AQL) number.  Generally, most companies want 1% or less, but as noted in the table, it does go up to 1000. It is extreme to think of something being more than 100%, but consider that it may be a minor or cosmetic defect that does not affect the function but just does not look good.  Scratch and dent sales are a common result of these higher numbers.

The AQL number is the worst quality level you would expect to find at this level.  The thing you have to remember is that these plans work best when the quality is very good or very bad.  If you are at the limit, you could end up taking more samples and spend a lot of time in tightened inspection.

Many people use percent nonconforming instead of percent defective, simply because of the connotation of “defective.” No one wants to say they shipped a defective product.  They may have shipped a nonconforming product that the customer could not use simply because their requirements were too strict, where another customer may be able to use the same thing because they have less stringent requirements.

Jim Bossert
SVP Process Design Manger, Process Optimization
Bank of America
ASQ Fellow, CQE, CQA, CMQ/OE, CSSBB, CMBB
Fort Worth, TX