AQL for Electricity Meter Testing

Chart, graph, sampling, plan, calculation, z1.4

Q: We have implemented a program to test electricity meters that are already in use. This would target approximately 28,000 electricity meters that have been in operation for more than 15 years. Under this program, we plan to test a sample of meters and come to a conclusion about the whole batch  —  whether replacement is required or not. As per ANSI/ISO/ASQ 2859-1:1999: Sampling procedures for inspection by attributes — Part 1: Sampling schemes indexed by acceptance quality limit (AQL) for lot-by-lot inspection, we have selected a sample of 315 to be in line with the total number of electricity meters in the batch.

Please advice us on how to select an appropriate acceptable quality level (AQL) value to accurately reflect the requirement of our survey and come in to a decision on whether the whole batch to be rejected and replaced. Thank you.

A: One of the least liked phrases uttered by statisticians is “it depends.” Unfortunately, in response to your question, the selection of the AQL depends on a number of factors and considerations.

If one didn’t have to sample from a population to make a decision, meaning we could perform 100% inspection accurately and economically, we wouldn’t need to set an AQL. Likewise, if we were not able to test any units from the population at all, we wouldn’t need the AQL. It’s the sampling and associated uncertainty that it provides that requires some thought in setting an AQL value.

As you may notice, the lower the AQL the more samples are required. Think of it as reflecting the size of a needle. A very large needle (say, the size of a telephone pole) is very easy to find in a haystack. An ordinary needle is proverbially impossible to find. If you desire to determine if all the units are faulty or not (100% would fail the testing if the hypothesis is true), that would be a large needle and only one sample would be necessary. If, on the other hand, you wanted to find if only one unit of the entire population is faulty, that would be a relatively small needle and 100% sampling may be required, as the testing has the possibility of finding all are good except for the very last unit tested in the population.

AQL is not the needle or, in your case, the proportion of faulty fielded units. It is the average quality level which is related to the proportion of bad units. The AQL is fixed by the probability of a random sample being drawn from a population with an unknown actual failure rate of the AQL (say 0.5%), creating a sample that has a sample failure rate of 0.5% or less. We set the probability of acceptance relatively high, often 95%. This means if the population is actually mostly as good as or better than our AQL, we have a 95% chance of pulling a sample that will result in accepting the batch as being good.

The probability of acceptance is built into the sampling plan. Drafting an operating characteristic curve of your sampling plan is helpful in understanding the relationship between AQL, probability of acceptance, and other sampling related values.

Now back to the comment of “it depends.” The AQL is the statement that basically says the population is good enough – an acceptable low failure rate. For an electrical meter, the number of out of specification may be defined by contract or agreement with the utility or regulatory body. As an end customer, I would enjoy a meter that under reports my electricity use as I would pay for less than I received. The utility company would not enjoy this situation, as it provides their service at a discount. And you can imagine the reverse situation and consequences. Some calculations and assumptions would permit you to determine the cost to the consumers or to the utility for various proportions of units out of specification, either over or under reporting. Balance the cost of testing to the cost to meter errors and you can find a reasonable sampling plan.

Besides the regulatory or contract requirements for acceptable percent defective, or the balance between costs, you should also consider the legal and publicity ramifications. If you accept 0.5% as the AQL, and there are one million end customers, that is 5,000 customers with possibly faulty meters. What is the cost of bad publicity or legal action? While not likely if the total number of faulty units is small, there does exist the possibility of a very expensive consequence.

Another consideration is the measurement error of the testing of the sampled units. If the measurement is not perfect, which is a reasonable assumption in most cases, then the results of the testing may have some finite possibilities to not represent the actual performance of the units. If the testing itself has repeatability and reproducibility issues, then setting a lower AQL may help to provide a margin to guard from this uncertainty. A good test (accurate, repeatable, reproducible, etc.) should have less of an effect on the AQL setting.

In summary, if the decision based on the sample results is important (major expensive recall, safety or loss of account, for example), then use a relatively lower AQL. If the test result is for an information gathering purpose which is not used for any major decisions, then setting a relatively higher AQL is fine.

If my meter is in the population under consideration, I am not sure I want my meter evaluated. There are three outcomes:

  • The meter is fine and in specification, which is to be expected and nothing changes.
  • The meter is overcharging me and is replaced with a new meter and my utility bill is reduced going forward. I may then pursue the return of past overcharging if the amount is worth the effort.
  • The meter is undercharging me, in which case I wouldn’t want the meter changed nor the back charging bill from the utility (which I doubt they would do unless they found evidence of tampering).

As an engineer and good customer, I would want to be sure my meter is accurate, of course.

Fred Schenkelberg
Voting member of U.S. TAG to ISO/TC 56
Voting member of U.S. TAG to ISO/TC 69
Reliability Engineering and Management Consultant
FMS Reliability
www.fmsreliability.com

Related Content:

To obtain more resources on sampling and statistics, read the featured ASQ journal articles below or browse ASQ Knowledge Center search results.

Rethinking Statistics for Quality Control, Quality Engineering

Setting Appropriate Fill Weight Targets — A Statistical Engineering Case Study, Quality Engineering

Compliance Testing for Random Effects Models With Joint Acceptance Criteria, Technometics

ISO 9001: Product Development and Customer Satisfaction

Manufacturing, inspection, exclusions

Q: Does a company certified to ANSI/ISO/ASQ Q9001-2008 Quality management systems — Requirements that produces raw materials for a customer according to their written specification also, as a raw material supplier, have a responsibility under ISO 9001 to meet the customer’s needs for their design intent and intended and known use?

In simple language, I sell a raw material to a customer who takes my raw material and then designs a product and sells it to a customer who uses it in the field. I wonder where does the ISO standard application stop for the raw material supplier?  How can a raw material supplier under ISO 9001 meet the needs of a customer’s trade secret designs, or further down the intended use of the product where the raw material supplier has no control over how it will be used or maintained?

A: Your question is more a legal one than a quality one. You are offering a product to a customer. This is your finished product and their raw material. When both parties agree to the terms and conditions (payment, form, fit, function, shipping, etc.) a contract exists. We call this a purchase order (PO) and part of that PO is the specification for your product. If they place an order to your spec, you have done the design work under ISO 9001 and they are accepting your design. END OF YOUR RESPONSIBILITY for future application and use. If you accept an order to their spec, they have done the design work and you are obligated to make sure your product meets the stated (and often implied) form/fit/function requirements. We call this quality control and you do this by testing in the lab prior to shipment.

Most firms address the issue of application by stating quite clearly in the contract terms that you are selling your product as-is and you do not warrant the product as fit for ultimate use. This is the kind of thing the lawyers require.

Having said all this, there is a requirement in ISO 9001 for you to measure customer satisfaction. You must state in your manual the concept (strategies) for doing this and have some defined processes – usually called procedures – to carry it out. Of course, part of this is the regular management review. Quality, marketing, and sales all provide input on how well the customer needs are being met. Your registrar should be examining how you do this.

If there is a trend showing that customers are unhappy with how the stuff performs under end-use conditions, ISO says you should address those issues. (Ignoring them is an option, if it is deliberate). Mature firms will work on building customer-supplier partnerships, getting their engineers to talk to your engineers. Although this is technically outside of the quality function, it is still part of your overall quality management system.

Charlie Cianfrani
Consulting Engineer
Green Lane Quality Management Services
Green Lane, PA
ASQ Fellow; ASQ CQE, CRE, CQA, RABQSA Certified QMS-Auditor (Q3558)
ASQ Quality Press Author
Related Content:

Open access resources about supplier quality and product development:

Two Sides of the Same Coin: Using Teams in Customer-Supplier Relationships, Journal for Quality and Participation

In the Know: A BoK dedicated to quality in outsourcing is essential in today’s global marketplace, Quality Progress

The Role of Quality Management Practice in the Performance of Integrated Supply Chains, Quality Management Journal

Has Information About Quality Become a Liability? Quality Progress

Product Liability: Beyond Loss Control — An Argument for Quality Assurance, Quality Management Journal

Customer Satisfaction and Loyalty

Suppliers, supplier management

Q: Can you give me more information about how organizations gain, measure, and retain customer satisfaction and loyalty?

A: The Quality Improvement Glossary, by Donald L. Siebels, defines customer loyalty/retention as “the result of an organization’s plans, processes, practice, and efforts designed to deliver their services or products in ways which create customer satisfaction so customers are retained and committed to remain loyal”.

ASQ has over 200 books, articles, and case studies that focus on the topic of Customer Satisfaction and Value. The following recommended books from ASQ may help to provide additional insight:

The Customer Advocate and The Customer Saboteur
Quality Press, 2012

Measuring Customer Satisfaction and Loyalty, 3rd Ed.
Quality Press, 2008

Strategic Customer Service: Managing the Customer Experience to Increase Positive Word of Mouth, Build Loyalty, and Maximize Profits

AMACOM, 2009

Managing the Customer Experience: A Measurement-Based Approach 

Quality Press, 2007

Beyond the Ultimate Question: A Systematic Approach to Improve Customer Loyalty
Quality Press, 2010

You also may want to take a look at the following articles and case studies:

“Your Customers Are Talking, But Are You Listening?”

Quality Progress, February 2006
The listen, collect, analyze, learn, improve (LCALI) process can help an organization capture important customer data for analysis and action.

“3M Entitlement Quality: Flawless Execution at the Speed of the Customer”

Case Study, April 2009
3M’s approach to enterprise-wide quality improvement for business results and customer satisfaction is a fusion of ISO 9000, Six Sigma, lean, business process management, commercialization, and supplier management, along with a homegrown model, process and product understanding (PPU).

“Challenges With Churn”

Six Sigma Forum Magazine, November 2011
For noncontractual businesses, identifying profitable customers before they take their business elsewhere is important but difficult. These businesses usually have limited information about when these customers churn. Taking a control chart approach similar to statistical process control (SPC) can help develop churn predictive models and identify early churn.

“Supporting Customers and Driving Excellence Through Quality”

Case Study, May 2011
When a key client entered a new line of business, Firstsource Solutions earned the contract to provide inbound customer service and technological support. Metrics showed that 15 percent of calls for the client’s new business were repeat calls, leading to higher costs and lower customer satisfaction scores. A cross-functional Six Sigma team implemented process improvements that lowered the repeat call rate to 9.6 percent.

“An Alternative Approach in Service Quality: An e-Banking Case Study”

Quality Management Journal, January 2008
To remain competitive in today’s business climate, organizations must offer services that not only satisfy their customers, but delight them. SERVQUAL and other measures have been used in the past to measure and improve service quality in the US and Europe, but Japanese quality systems such as Kansei Engineering (KE) and quality function deployment (QFD) offer an alternative way to include the voice of the customer in the development and improvement of service quality systems.

Other helpful web pages include:

ASQ Learn About Quality: Customer Satisfaction

ACSI: The American Customer Satisfaction Index

I hope that this information is helpful.  If you are interested in conducting your own search, you may want to visit the ASQ Knowledge Center.

Best regards,

ASQ Research Librarian
Milwaukee, WI