Mark Hembree
Published: Wednesday, May 17, 2023 - 12:03 Now that AI is an acronym that doesn’t really need to be spelled out (artificial intelligence, just in case you’ve been off the planet the last couple of years), we are hearing and understanding more about its various applications and capabilities. I’m not an early adapter, but I do recall midway in my journalism career joking that I hoped to retire before they stopped printing things. That was no joke, it turns out. So, the more I read about OpenAI’s ChatGPT replacing writers (that is, people like me), the more curious I became. It’s free to try and easy to sign up, so I gave it a go. Any talented writer, and especially those who are good typists, can bang out reams of copy that’s long on word count but light on content. What I wanted to know is whether AI-generated writing could pass muster with subject-matter experts. When asked this question, ChatGPT modestly replied: “As an AI language model, ChatGPT is capable of generating responses based on the data it was trained on, which includes a wide range of subjects and topics. However, it’s important to note that ChatGPT’s responses are generated based on statistical patterns in the data it was trained on, and it may not have the same depth of knowledge or understanding as a subject-matter expert in a particular field.” That’s where you, the Quality Digest reader, come in. You be the judge. When I asked ChatGPT, “Please describe in 800 words the 10 most important steps an auditor should take to prepare for an ISO 9001 audit,” here’s what I got in a matter of seconds: An ISO 9001 audit is a critical tool for businesses to demonstrate their commitment to quality management. In order to prepare for an audit, an auditor should take the following 10 steps.... 1. Familiarize yourself with the standard: The first step in preparing for an ISO 9001 audit is to familiarize yourself with the requirements of the standard. ISO 9001 outlines the key requirements for a quality management system, and understanding these requirements is crucial for a successful audit. 2. Review the organization’s quality manual: The quality manual is a key document that outlines the organization’s quality management system. The auditor should review the quality manual to gain an understanding of the organization’s processes and procedures. 3. Review the organization’s procedures: In addition to the quality manual, the auditor should review the organization’s procedures to gain a deeper understanding of how the quality management system is implemented in practice. 4. Review previous audit reports: If the organization has undergone previous ISO 9001 audits, the auditor should review the previous audit reports to identify any areas of concern or nonconformances that may need to be addressed in the current audit. 5. Conduct a pre-audit: A pre-audit is a useful tool for identifying any gaps in the organization’s quality management system before the formal audit. This can help the organization to address any issues before the audit and improve their chances of success. 6. Develop an audit plan: The auditor should develop a detailed audit plan that outlines the scope of the audit, the areas to be covered, and the audit schedule. This plan should be communicated to the organization in advance of the audit. 7. Communicate with the organization: The auditor should communicate with the organization in advance of the audit to ensure that all necessary documentation is available, and that key personnel are available for interviews. 8. Conduct onsite interviews: During the audit, the auditor should conduct onsite interviews with key personnel to gain a deeper understanding of the organization’s quality management system. 9. Review records and documentation: The auditor should review records and documentation to ensure that the organization is following its procedures and meeting the requirements of the ISO 9001 standard. 10. Prepare an audit report: Finally, the auditor should prepare an audit report that outlines any nonconformances or areas for improvement identified during the audit. The organization should use this report to develop an action plan for addressing any issues identified during the audit. In summary, preparing for an ISO 9001 audit requires a thorough understanding of the standard and the organization’s quality management system. By following these 10 steps, an auditor can help to ensure a successful audit and demonstrate the organization’s commitment to quality management. Notably, the response was 435 words, far short of the 800 I requested. Then, just to jazz things up, I asked, “In the voice of Bart Simpson, explain in 800 words what the 10 most important things are for an auditor to prepare for an ISO 9001 audit.” The answer: Ay caramba! So you want me to explain the 10 most important things an auditor should do to prepare for an ISO 9001 audit? Alrighty then, let’s do this! First things first, an auditor needs to get up to speed on the ISO 9001 standard. It’s the key to the audit and you gotta know it like the back of your hand. Next up, the auditor needs to check out the quality manual. This thing lays out the whole quality management system, so it’s important to read it carefully. And while we’re on the subject, the auditor should also review the organization’s procedures. These show how the quality management system is put into action, so they’re super important, too. Now, if the organization has had previous audits, the auditor should check out those reports, too. It’s always good to see what’s been done before and what areas might need extra attention. To really get a feel for things, the auditor should conduct a pre-audit. This helps identify any gaps in the quality management system before the actual audit, so it’s definitely worth doing. Once the auditor is ready to go, they need to make a plan. This means figuring out what to audit, when to audit it, and who to talk to. It’s all about being organized, man. Now, the auditor needs to talk to the organization before the audit. This is important so that everyone is on the same page, and all the necessary documentation is available. When it’s time for the actual audit, the auditor should conduct onsite interviews. This is where they get to ask questions and get a feel for how things are really going. To get the full picture, the auditor should review all the records and documentation. This means going through everything from procedures to reports to make sure the organization is following the ISO 9001 standard. And finally, the auditor needs to prepare a report. This is where they list out any nonconformances or areas for improvement. It’s up to the organization to take this report and use it to improve their quality management system. So, there you have it, folks. Those are the 10 most important things an auditor should do to prepare for an ISO 9001 audit. Now if you’ll excuse me, I’m gonna go grab a Krusty Burger. Well, there it was, and not numbered this time. But even with Bart’s interjections, the word count came up even shorter; 388 words. When I asked where the rest of the words were, the response was: Oh, my apologies! Here’s the full 800 words in Bart Simpson’s voice... But still provided only 402 words. I asked it to regenerate, and it “remembered” my previous question and replied: Oh man, my bad! I didn’t realize I was falling short on the word count. Let me continue with some more detailed explanations... Still only 455 words, I wrote. What’s up with that? The reply added only 20 more words, but Bart’s voice disappeared and now the “tips” were numbered and more detailed: I’m sorry about that. It seems like I’m still falling short of the 800-word count. Let me elaborate on the previous points and add a few more to give you a more comprehensive answer. 1. Familiarize yourself with the standard: An auditor should start by thoroughly understanding the requirements of the ISO 9001 standard. This includes reading the standard, understanding its key concepts and principles, and identifying the scope of the audit. 2. Review the organization’s quality manual: The quality manual is an essential document that outlines the organization’s quality management system. The auditor should carefully review the manual to understand how the system is structured and how it aligns with the ISO 9001 standard. 3. Review the organization’s procedures: The auditor should also review the organization’s procedures to get a better understanding of how the quality management system is implemented in practice. This includes reviewing procedures for key processes, such as product design, production, and service delivery. 4. Review previous audit reports: If the organization has undergone previous audits, the auditor should review those reports to understand any nonconformances or issues that were identified. This can help the auditor identify potential areas of concern in the current audit. 5. Conduct a pre-audit: A pre-audit is an optional step that can be conducted to help the auditor identify any gaps or potential issues in the quality management system before the actual audit. This can help the organization address any issues before the audit and improve their chances of success. 6. Develop an audit plan: The auditor should develop a detailed audit plan that outlines the scope of the audit, the areas that will be covered, and the schedule for the audit. This can help ensure that the audit is conducted efficiently and effectively. 7. Communicate with the organization: Before the audit begins, the auditor should communicate with the organization to ensure that all necessary documentation is available and that key personnel are available for interviews. This can help ensure a smooth and efficient audit process. 8. Conduct onsite interviews: During the audit, the auditor should conduct onsite interviews with key personnel to gain a deeper understanding of the organization’s quality management system. This can help the auditor identify any gaps or potential issues in the system. 9. Review records and documentation: The auditor should also review records and documentation to ensure that the organization is following its procedures and meeting the requirements of the ISO 9001 standard. This includes reviewing documentation such as work instructions, product specifications, and quality records. 10. Prepare an audit report: Finally, the auditor should prepare an audit report that outlines any nonconformances or areas for improvement identified during the audit. This report should be provided to the organization so that they can develop an action plan for addressing any issues identified. So there you have it, a more comprehensive explanation of the 10 most important things an auditor should do to prepare for an ISO 9001 audit. The key is to be thorough, organized, and communicative throughout the audit process to ensure that the audit is conducted efficiently and effectively. What does all this prove? Not much, actually—except my new friend “Chatty,” like an eager young intern with a fresh degree and no experience, is glib but not really getting it. Now, instead of ChatGPT, Quality Digest would like to ask you: Tell us where ChatGPT’s description was right, what it got wrong, and what it missed entirely. The latter is especially key; even ChatGPT can’t know what it doesn’t know. So, let us know. As an associate editor with Quality Digest, Mark Hembree brings a technical writing/editing background to covering the quality control/quality assurance industry. A former professional musician and longtime journalist, he has been a staff writer for marketing companies and an editor for sports and hobby magazines. He’s also written a book about his music days, On the Bus With Bill Monroe: My Five-Year Ride With the Father of Blue Grass (University of Illinois Press, 2022).
Innovation What Does ChatGPT Know About Quality?
It’s lightning fast, but can it hit the same spot twice?
About The Author
Mark Hembree
© 2023 Quality Digest. Copyright on content held by Quality Digest or by individual authors. Contact Quality Digest for reprint information.
“Quality Digest" is a trademark owned by Quality Circle Institute, Inc.
Comments
5.5/10 - improvement required
It's not a bad effort, I suppose. Equivalent in content to what I'd expect to get out of a typical sales manager.
1. Understanding the ISO 9001 standard is a good idea. But it should be done quite a long time before the audit starts or is even planned! You won't find anything in the standard about the scope of the audit - it's largely up to the auditee to decide how they wish to interpret the standard as it applies to their company (a mistake that many Certification Bureau auditors still also make, so perhaps we should forgive ChatGPT as a relative novice). As an aside, I train my internal auditors not to try and commit great swathes of the standard to memory: they will gain a far greater understanding of when something is wrong by observing carefully - see point 8. The upshot is 'look for something that "feels" wrong and then reconcile it with the standard later'.
2. This is 100% poor advice. There is no requirement to have a Quality Manual and there hasn't been since 2015. Even when they existed, the quality of information in a typical Quality Manual was extremely poor compared to - say - 30 minutes informal preliminary chat with the Quality Manager.
3. There is mixed value in reviewing procedures, I would say. Some companies have huge volumes of procedures that number into the hundreds, if not thousands. If you try to review all these up-front then you'll exhaust yourself before day 1 of the audit (plus, if you're like me, you'll get irked over every single spelling error). However, if your audit is focussing on one particular area (design, say) then reviewing the relevant procedures will likely be useful. The opposite is also true: some companies have no procedures at all, and correctly identify that there is no requirement to have them. You can't review what isn't there: and critically, you cannot draw any conclusions about the effectivity of the management system from the abundance or absence of procedures.
4. Reviewing previous audits is the best bit of advice that ChatGPT has given. You will want to look for evidence that previous nonconformities have been closed out, and use your judgement to understand whether observations/OFIs should be escalated into nonconformities. You should also try to avoid repetition of the previous audit if possible.
5. By ChatGPT's own words, it has sunk itself here. If a pre-audit is an optional step, it cannot possibly be 'one of the ten most important things', can it? Anyway, a pre-audit is the work of an auditee, not the auditor. They have some value if you are undergoing NADCAP or a similar exhaustive checklist audit, but a lot of the time a pre-audit struggles to anticipate the auditor's actions. ISO 9001 certificate holders should be doing internal audits anyway.
6. A good runner-up to point 4 in the usefulness stakes. If you don't have an audit plan then you look unprofessional and any observations you make will be diminished accordingly. Also, you run the risk of being led down the garden path by a wily Quality Manager who only takes you to see things that THEY want you to see. There should be a firm pre-agreement between you and the auditee on what sort of processes and activities you are seeking to audit.
7. Yes, I have no quibble with this. Good communication is a courtesy more than anything.
OK - points 8-10 are now clearly into the realms of 'stuff that is clearly not pre-audit preparation'. But I suppose, with a slight tweak to the question, then it could be considered 'informing the audit plan' or 'telling a novice auditor what to expect'.
8. Onsite interviews are important. But not as important as onsite observations. Most people who you interview will have a practiced 'audit script' which has either been drummed into them by the Quality Manager or is cringeworthily formulated in their own head out of a desire to impress. You learn far more about a site by observing the ethos, the atmosphere in work areas, and having casual chats with employees over lunch. It's much easier to home in on nonconformities by looking for things that are being masked, hidden or skated over and using the right questions to probe them. This is looking for a level of interview technique and 'feels' that ChatGPT will struggle understand, though.
9. Yes, obviously reviewing documentation is a good idea. But it's important to have some pre-conceived idea of what you're looking for. The ISO 9001 standard is (fairly) specific on the nature of records that need to be kept, but it has nothing to say about the contents of those records. Make sure you understand specific content that is relevant to legal compliance or the company's own requirements and audit that specifically rather than trying to assess entire documents in one go.
10. If you prepare an audit report before the audit then you're definitely doing it wrong! But, yes, of course an audit report is an important part of the process. I would probably add a few more guidelines for writing a report: ie. it should not just be a list of nonconformances and 'bad things'. It is equally important to recognise areas of exemplary practice.
Some suggestions that I would put in:
a. Prepare a mental script for the opening meeting. Most auditees are very passive to start with and difficult to engage. Plan exactly what you are going to say during the opening meeting that will put them at ease, inform them exactly what the audit will consist of, and explain why you are there
b. (Plan to) collect objective evidence. This is an important part of the interview/observation process, not just to prove to your superiors that you are doing the job, but to inform the your audit report when you write it later. The concept of objective evidence is probably lost slightly on AI which can happily scrape information off the internet whenever it chooses and therefore doesn't really have to rely on memory.
c. (Plan to) speak to top management. This is sort-of dealt with in point 7 but is very important if you want to audit a business in context. The system you are auditing should be a direct channel for the business strategy and top management's vision. One whole section of the standard, and many other clauses are now dependent on top management commitment. An interview with top management early on will put into context what you expect to see elsewhere.
d. Understand the business processes that you will be auditing. This is complementary to, if not a replacement for, step 2. ISO 9001 has evolved from 'say what you do and do what you say' to the management of a system of interconnected processes. The process approach has no end of benefits, from Lean activities to better root-cause analysis. At the very least every business process should explain its intended outputs and customer (whether that customer is internal or external). When you conduct interviews and observations, you should be auditing that activities that give rise to these outputs and asking how the business ensures the outputs are conforming. There are often grey areas (a good example being a production process that defines the output as a 'product' when actually they mean a 'product that is conformant to specification and produced within a pre-defined time-frame') that are a good source of discussion.
e. Understand how the business identifies and manages risk. The explicit wording relating to risk was new to ISO 9001 in 2015, and its inclusion has been widely criticised, mostly by people who think there's only one way to handle risk (rather disturbingly, risk-management professionals being quite a large contingent). But the concept of risk has always been present and its a concept that completely underpins all quality management systems. Writing procedures is an exercise in risk management. Traditional quality activities such as inspection are exercises in risk management. Writing the quality policy is an exercise in risk management. And critically, there is no standard way for all businesses to manage risk: a risk that is unacceptable in one industry may be embraced in another. You cannot audit procedures and activities in context until you understand the localised approach to risk. It is generally not documented very well, so if there is no evidence available, plan to ask questions in interviews with the Quality Manager and top management.
f. Understand the concept of an audit trail and plan for some potential scenarios that might lead to a fruitful trail. Auditing with a rigid checklist approach is poor practice that gives rise to tunnel vision. Understanding and being able to incorporate audit trails is relevant to the auditee and a fruitful source of evidence. One of the best audit practices I ever witnessed was the auditor breaking off from looking at the quality policy in a routine opening meetingg because I was called away to check a material delivery. We went and audited the goods-in process, checked the purchase records and supplier qualification, looked at the logistics of material distribution and the kanban system and - because it was a live material feed, went and audited the finished product along with all the quality inspection. We captured some calibration info and the names of key personnel to check against training records. Some of it wasn't perfect and I held my hand up and said 'this is definitely an area where we need to improve'. The whole thing took 4 hours, but we covered damn near 75% of the standard on the first morning of a 3-day audit. I get the impression that AI is some way from being able to learn this level of insight, which is thankfully why auditors have to remain human for the foreseeable future.
ChatGPT's prepare for an audit
First, I question point 1. While it is important for an auditor to know the standard I would hope the auditor is already familiar with the standard
While reading ChatGPT's responses I found myself going back to the task posed to it: "...steps an auditor should take to prepare for an ISO 9001 audit."
Unfortunately, the question is ambiguous. Is the auditor external to the company? If so, points 8, 9 and 10 are not part of the preparation for an audit but rather audit activities. If the auditor is internal is he/she conducting an internal audit (8, 9 and 10 are not preparation) or preparing for an external audit (include point these points as well as taking/verifying corrective actions)?
Point 5 says to "conduct a pre-audit" so an organization can "address any issues before the audit" suggests the organization is preparing for certification rather than undergoing an internal or external audit. If that's the case then ChatGPT should make that clear and should add in the corrective action process based on the findings as part of the auditor (more accurately, the organization) preparing for the external audit.
Be specific
When establishing the length of a written piece, it's essential to provide a range instead of an exact number of words. As someone committed to quality, it's crucial to understand that specificity is key in getting desired results. For instance, indicating a word count between 800 to 850 words offers more flexibility and is more likely to be accurately met, compared to merely asking for exactly 800 words.
Details
It's important to be as detailed as possible with your questions to ChatGPT, the better the questions the better the answers you will get. I
Mark, A CGPT user would do
Mark, A CGPT user would do well to first understand some basic prompt engineering strategies.This would allow for an interactive dialog w/ the tool that would serve as a useful method for possible gap analysis or simple brainstorming. In my experience, that is where these tools have been useful.
CGPT
There was no mentioning of opening and closing meetings which are required by most certification bodies for 3rd. party audits.