Q&A - Performance Data Reports

The Contact Us function at the top of every page on the tl9000.org website are the preferred means for asking questions and receiving answers from the subject matter experts of the TIA QuEST Forum. Over the last few years many questions have been answered through this means. The number of each question is the ticket number in the Contact Us tracking system.

These questions concern the Performance Data Reports.

Question 11037 — Question about benchmarking, why would the worst in class count be different from either the best in class or industry average?

Answer — For any specific measurement within a category, the Best In Class and Industry Average data point counts will always be the same. But it is possible for the Best In Class and Worst In Class counts to be different. That is because the Best in Class and Worst in Class data eligibility rules are different. In particular, there is an additional check used for identifying the Worst In Class which could exclude registrations that might have been included in Industry Average or Best In Class. It's all detailed in the TL 9000 Measurement Outputs and Calculations paper. See https://tl9000.org/pdrs/calculations.html

Question 11064 — I checked the performance data of category 8.5.2.3 Optical Subassemblies recently and find that for FR data, there is only ERI monthly average data published. Industry Average, Best in Class, and Worst in Class are not published. Please confirm if there is any problem about the performance data. Is there any special reason for these important figures not being published?

Answer — The tool that produces the PDRs is independent of category. That is, common software is used to publish all PDR data – there are no category specific calculations. The fact that ERI Monthly Average data was published and the other FR data was not published relates to the number of registrations/companies reporting. When the Monthly Average data is reported and the Industry Average, BIC, and/or WIC are not, it means that while 3 or more companies have reported data in the most recent month, 3 or more companies have not reported eligible data for the most recent 12 consecutive months.

As to why only ERI was published, remember that YRR, NYR, and LTR cover different time periods than ERI. ERI measures returns that had been shipped within the last six months while the other measurements span a longer period of time. It is likely that there were not enough companies reporting data over the longer period of time to allow reporting data in those measurements. Look at the count of registrations reporting. Data from three separate companies are required before any output is produced on the PDRs.

Question 11155 — I was told by my auditor that I should receive a quarterly report on our RMS data. I have not been receiving this report, what can I do to start receiving it?

Answer — The auditor is mistaken or else the information communicated wasn't understood. There are no quarterly reports. The TIA QuEST Forum publishes annual Performance Data Reports, made available to TL 9000 Registrants in March for the prior year’s data. These reports are available to those having administration rights for TL 9000 registrations. To view the report, you must log into the TL 9000 website, click on the “TL 9000 Registrations” section, and then click on the specific registration under the “Manage a TL 9000 Registration” heading. On that page, you will see a link under the heading “View Annual Data”. This report will show you for your specific category what the average performance was in the most recent calendar year for all published TL data.

Question 11935 — From the Performance data report on TIA QuEST Forum website, I found that the "Worst In Class" value of Line Item On Time Delivery (OTI) for category 7.7.4 is 67.30077%, which is actually higher than our actual company OTI value 0.64. I don't know why. Please give me some explanation.

Answer — The Best-In-Class and Worst-In-Class numbers are based on sustained performance. For the OTI measure, it is determined by the last six months on data. It is calculated by summing the numerators and denominators for the six months to arrive at the value. It is certainly possible for a single monthly value to be lower than the Worst-In-Class value. It’s not clear from your question whether your 64% performance was for the most recent month of for the most recent 6 months.

If the 64% performance was for the most recent 6 months, this could also be possible due to eligibility checks that are performed against data before publishing the PDR’s. Because we want to have representative data published for Industry Average, Best In Class, and Worst In Class, an organization’s denominator must represent at least 2% of the total of all data submitted during the 6-month period. The methodology used to calculate the industry statistics is described in the paper “TL 9000 Measurement Outputs and Calculations” available through a link on the tl9000.org web page at: https://tl9000.org/pdrs/calculations.html .

Question 11946 — From the website paper TL_9000_Measurement_Outputs_and_Calculations.pdf available through a link at https://tl9000.org/pdrs/calculations.html , there is one description after the Table 3, that is "even though OTS is identified as a 12-month window for smoothing, reporting starts after 6 consecutive months of data. How does the rule for minimum data points relate to the smoothing rules? In the case of OTI, NPR, FRT and OFR, the value is 6. Does it mean that the organization measurement data will be included after 3 consecutive months for measurements identified as requiring 6 months for smoothing?

Answer — For your first question: The rule for minimum data points does not relate for those for smoothing. The “minimum data points” referred to in the example is the requirement that there be data from at least 3 companies before any numbers are reported out. This is to protect the anonymity of the data submitters. Therefore, to report Best In Class, Worst In Class, and Industry Average there must be at least 3 companies that have submitted data for at least six consecutive months. Showing exactly how this works is the purpose of the example.

The smoothing of the submitted data used to determine the Best In Class or Worst In Class performance is performed by the system itself. There is nothing that the organization submitting the data does other than submit their required data monthly.

Question 12096 — It looks like the data for category 3.2.1.1 is tracking exactly like our company’s data. If you look at ERI for this category, there is a spike. It is the same spike that we saw due to a data entry error. It says that there are 7 companies reporting this category and this measurement, how can the data track exactly like ours?

Answer — There are seven registrations certified to TL 9000 in 3.2.1.1. You may know this already but if not, you can find out who is certified in a particular category by going first to tl9000.org and clicking on TL 9000 Certification in the main menu bar. Next click on Certified Organizations in the left menu that takes you to a page where you can make queries into our database. At this point the data is public so no login is needed. Click Advanced Search. On the page, under the 'Data on Certifications' heading, select "3. Certified Organizations by Category" to go the next page and select 3.2.1.1 in the top search box then click on Search by Category. That will give you a list of the certified organizations in the category and you'll find your certification there along with six others.

When the PDR results seem to track an individual certification, it is quite possible that the certification is the dominant (largest portion in normalization units) certification in the category. In addition, remember that ERI is a return measurement and it is possible to have no returns in a particular month. If the ERI Monthly Average is tracking your results, then it is possible that the other certified organizations reported very few returns. Due to anonymity rules, this can’t be verified but is the most likely scenario,

Question 12576 — Problem with downloading Performance Data Report of 8.3 Cable Assemblies. FRT3 & NPR3 is missing from download.

Answer — In category 8.3, only NPR4 and FRT4 is supposed to be published. See Tables 5.1-2 and 5.2-2 of the Measurements Handbook.