AI: Lean Six Sigma + Gemini

Coming Up: An intensive and exhaustive test of Google Sheets' Gemini used in Lean Six Sigma process improvement.

Introduction

There’s been a lot of speculation about a coming revolution with AI augmenting traditional strategies like Lean Six Sigma. It is confidently predicted that AI will make Lean Six Sigma more proactive, more predictive, more analytical, and yield results more quickly.

Let’s examine that prediction by using AI on several improvement projects. We’ll use Google’s Gemini because it claims to have strong data analysis capabilities when integrated with Google Sheets, and it comes with a free 14-day license.

If this is to benefit many of us, then it has to be simple and approachable. That’s supposed to be the benefit of AI: it’s going to make us smarter and more efficient without needing to be experts in data analytics.

We'll test these predictions for a step change in productivity by following a standard improvement project to see how current AI capabilities will handle it. The kind of projects that might lend themselves to AI support could be data-rich projects in either manufacturing or business processes.

Case: Manufacturing Quality Control

Case Study Details

This example is a standard quality control problem from the assembly process for a complex laboratory instrument and uses real data from a previous Lean Six Sigma project I was involved in.

The data I have is a month of defect logging on the final fault inspection of each instrument. The process was achieving a first-pass yield of around 30%. The project charter set a goal to improve First Pass Yield on assembly to 95%.

I’ll provide Gemini with the available data on fault descriptions and the cost of scrap or rework for each fault type. There were 121 faults in total listed in the first tab of this spreadsheet, including 15 different fault types. The second tab in the file contained the cost data representing the scrap and labour cost of replacement parts or adjustment and rework of each fault type.

Let’s start by asking simple questions and see where Gemini takes us.

Prompt:

Analyse the data in the Assembly Faults file.

Response:

Gemini responded with the classic Pareto chart showing priority of improvement effort by fault count. It includes the usual cumulative percentage line, which indicates that fixing the first six faults would eliminate 80% of all defects. That would probably have saved the inexperienced user time in finding and using the right software or manually creating this chart in a spreadsheet package.

Gemini suggested the next question in its previous answer:

Gemini quickly found the relevant data without prompts by reading the file name and column headers and completed the cost analysis. This saved a lot of time and effort over extracting the data and calculating the cost of each fault.

I would have expected a Pareto chart indexed by cost this time, rather than count. That’s a fairly standard option for prioritising faults by greatest saving, but it seems Gemini didn’t answer appropriately. Clearly, Gemini completed the cost analysis before, as shown by the table above. Perhaps I needed to be more specific in my question.

Prompt:

Repeat the Pareto analysis prioritised by cumulative cost from data in sheet 2.

Response:

Gemini provided a running commentary on its strategy, more detailed than I could keep pace with and then successfully created the Pareto- Chart by Cost.

Gemini has redrawn the Pareto chart, clearly identifying the priority of action using cost-benefit to the business rather than a simple count of faults. Let’s ask what strategy we should use to prioritise fault investigation.

A good answer, but remember this wasn’t Gemini’s first offer. It initially provided a Pareto analysis by count. If the fault data had been related to customer defects rather than inspection faults, then arguably, we should prioritise customer satisfaction and use the Pareto analysis of fault counts. Perhaps Gemini knew that all along?

These are all standard tools in the Six Sigma toolbox. Gemini can’t tell you which ones are more likely to be useful with the information it has, so it just presents a generic strategy. One or more of these tools may be useful, but there is no real analysis here.

I repeated the same question on the following day to test the continuity of the response.

Gemini has changed it’s advice from the previous response to this question. This is still very generic advice on a strategy to investigate Root Cause of faults. With little information Gemini is unable to provide a clear direction.

In the next case study, we will investigate how Gemini can contribute to analysing a much larger file of inspection and test data.

Case: Manufacturing Quality Control — Part 2

Case Study Details

In this next case study, we’ll present Gemini with a much larger data set to further investigate its analytical capabilities.

The data I’m using is from the inspection records of the final test on a complex product assembly. There are over 400 records with over 200 fields in each record. These include descriptive fields like name of the test engineer, date and time, serial number, product variants and also output results showing quality or performance metrics of elements or subsystems in the product.

No cleaning or pre-formatting has been carried out; the data is as it has been exported from the testing software, typos and errors included, as input by the test operators.

The first two issues are easily fixed by eliminating duplicates and junk data. Tester names like “L SMITH”, “Lesley Smith” and “L Smith” are easily standardised. Some column headers are repeated: “Max.” “Min.”, “Average”, so Gemini couldn’t isolate the specific parameters. These columns are always preceded by a unique parameter name, so fairly easily fixed. This provides a useful lesson in data integrity.

There is no particular insight from Gemini here. These conclusions were inferred from simple text flags in the file for correction and re-test: e.g. entries like “NOT IN SPEC” or “PASSED”. Gemini did not identify any other technical issues; it had no information on the acceptable ranges for test parameters.

This is useful and represents a huge saving in effort over running histograms on upwards of 100 data sets. Skew in noise metrics is to be expected, but the bimodal or multi-cluster distributions suggest an unstable process. The five anomalous fields can now be examined in more detail. One that stands out is RMS SNR (Root Mean Square Signal to Noise Ratio).

Prompt:

Show a histogram of RMS SNR

Response:

This clearly shows two groups of data which correspond to the same analysis when run in other software. The lower cluster is out of spec and would have been corrected and retested. Approximately 25% of products requiring rework and retest is definitely a problem worth investigating.

Prompt:

Analyse the Signal to Noise data

Response:

Signal-to-Noise Ratio shows variation by test engineer, see plot below.

Gemini responds with a box and whisker plot suggesting this as a potential cause for the variation in SNR data. However, the first two groups represent spurious individual data points, the third is a small sample, and the remaining three don’t really show enough variation to be remarkable. Gemini does eventually recognise this in its more detailed insights accopanying the plot above:

Gemini has also recognised that there is a correlation between the text flag “NOT IN SPEC” and low SNR (signal-to-noise ratio) values. That’s good, but this was already well known. The real question is: “What is the root cause of low SNR values and the high rate of rework?”

Gemini suggests there is no smoking gun here and that the root cause of system noise lies elsewhere. Although the correlation is weak, it may still be worth investigating.

A strong relationship suggested by Gemini is the one between SNR and RMS Noise.

Prompt

Create a scatter plot between SNR and RMS Noise

Response

Since these are both outputs and different measures of the same basic property of signal quality, that’s expected, so this doesn’t tell us anything new.

Let’s instead examine the relationship between the SNR of the output and the PSU noise level. I want to partition the data to include only the low-SNR values to see if the relationship is more than just what Gemini describes as “weak”.

The very strong correlation between SNR and Noise RMS is interpreted as two measures of the same output and may bias Gemini into considering weaker correlations insignificant. If this can be ignored, then the correlation between PSU noise and supply voltage seems to be worth investigating.

The next action in the real project was to identify PSUs with higher-than-usual noise levels and return them to the supplier for in-depth analysis. That eventually resulted in changes to the sourcing of some internal components, which led to a significant reduction in scrap and rework on assembly.

Gemini has produced an incorrect inference here that the high correlation was significant and that the weak correlations were not. Had Gemini known that the two Noise metrics were the same output, that conclusion may have been different. However, it is a well-known risk that correlation does not always imply cause.

Case: Measurement Systems Analysis

Case Study Details

A common need in manufacturing is to assess the quality of instrument measurements. These are often used in critical inspection requirements during product assembly or testing and give a pass/fail quality criterion.

Conducting a measurement systems analysis or gauge repeatability and reproducibility analysis (gauge R&R) is a fairly straightforward process, but it may not be familiar to many. Could AI guide and support such an analytical process?

Any instrument or measurement system would be suitable for the case study, but for simplicity, the test case chosen is a familiar teaching exercise using a small digital calliper to measure the diameter of coins. I will assume no prior knowledge or experience and ask Gemini to guide the data collection and analysis.

The common advice is 10 parts, 3 operators and 2 repeats. When asked for a recommendation, Gemini volunteered 10, 3, 3, which is a safe if conservative sample.

I proposed 10 parts, 3 operators and 2 repeats. Gemini responded by creating the following table.

Response

Note that other dedicated statistics software packages would randomise the data collection in a single column with the objective that the operator’s second measurement should not be influenced by their first. That is an important consideration; all measurements need to be independent and not subject to bias.

I used data from a previous MSA study here, and loaded the raw data into the Google sheet.

Prompt

Analyse the MSA data

Response

The current measurement system is unacceptable for this process, as the Gage R&R exceeds the standard threshold of 30%.

Key Takeaways

• Significant Interaction: The "Operator*Part" interaction contributes significantly to reproducibility (39.96% of study variation)
• System Performance is Unacceptable: The Total Gage R&R (GRR) accounts for 41.17% of the study variation. According to AIAG standards, a GRR over 30% indicates the measurement system needs improvement.
• Low Resolution: The Number of Distinct Categories (ndc) is 3.12. A reliable measurement system should have an NDC of 5 or higher to effectively distinguish between different parts.
• Balanced Error Sources: Both Repeatability (28.55%) and Reproducibility (29.67%) contribute significantly to the total error. This suggests that the issues are split between the equipment's consistency and differences in how the three operators perform the measurements.

(Note that Gemini does not know the specification limits, so it can only compare measurement repeatability and reproducibility with sample variation.)

The takeaway suggested by Gemini that there is significant interaction between “Operator” and “Part” data implies that operators measure different-sized parts with different repeatability. This is likely to be a false conclusion, with variation in measurement between operators more likely. There is a risk that this is accepted as a literal conclusion rather than just a feature of the data.

The inference that the measurement system is unacceptable is quite correct, although it is incomplete without reference to the part specification. So I’ll provide the specification tolerance and repeat the question.

Prompt

Part spec. is 20.3 – 20.5 long term Std Dev is 0.038, analyse the MSA data

Response

The MSA conducted with your new parameters reveals that the current measurement system is unacceptable for both process control and part specification assessment.

Key Takeaways

• High Measurement System Variation: The Gage R&R accounts for 63.15% of the Study Variation and 32.75% of the Tolerance. Both values exceed the standard 30% threshold for an acceptable measurement system.
• Operator Inconsistency (Reproducibility): Reproducibility (55.14% of Study Var) is significantly higher than Repeatability (30.79%). This indicates that the primary source of error is the variation between operators (Operators 1, 2, and 3) rather than the equipment itself.
• Insufficient Resolution (ndc): The Number of Distinct Categories (ndc) is 1.73. A value of at least 5 is required to ensure the measurement system can distinguish between different parts in the process.

Before reviewing the analysis and results provided by Gemini, I’ll repeat the analysis using an industry-standard statistics software package that includes MSA analysis as a built-in module.

The numerical analysis is identical, with some minor differences in presentation and format. Let’s test the key takeaways that Gemini gave:

1. The Measurement System is unacceptable. This is correct. The target % variation and % tolerance should be below 30% and ideally below 10%.
2. High system variation at 63% of the study. That’s not the important benchmark here. It compares measurement error with 6 times the standard deviation of the sample or study variation. The product's long-term standard deviation is much more important as a realistic benchmark of the process and should always be used. The statistics software calculates 29% of the study variation.
3. Operator inconsistency is high at 55% of study. This refers to reproducibility. Again, the wrong benchmark 25% using the correct benchmarks.
4. Number of distinct categories Gemini reports 1.7. While the statistics software reports 4, this is due to the different benchmarks used in the analysis.
5. Gemini recommends in the commentary that a training exercise should be carried out to improve consistency between operators. That would address note 3 above and is very likely to be useful in this case. This is clearly illustrated by the range chart for operators produced by the same statistics software used before.

It would seem likely that Alice is a very experienced operator with highly consistent measurement skills, while Colin and Barry would benefit from additional training.

Design of Experiments — Screening

Case Study Details

To investigate the use of AI in a screening design of experiments (Screening DoE), I’ll make use of the well-known teaching example, the Statapult. This is a small catapult that has become an industry-standard tool for teaching statistical modelling.

It allows for the adjustment of multiple control factors (launch angle, arm radius, input energy, from the rubber band tension, and stop pin settings) to evaluate the effect of those settings on the distance achieved and its repeatability or accuracy.

With good experimental technique and the right analysis software, a modelling equation can be developed for the Statapult with an R² (correlation coefficient squared) exceeding 0.99 - this represents a model that predicts 99% of the data with only 1% noise content. Could AI provide that same quality of model prediction to an inexperienced user? The process would normally start with a screening experiment before proceeding to a full modelling experiment.

Let’s assume little knowledge of DoE and ask Gemini for advice.

The use of an 8-run Resolution III design represents the most economical but least effective design in recognising interactions. Also, 32 runs or more are probably impractical. So let’s ask Gemini what it recommends. After a few follow-up questions, and my limit set at 16 runs, Gemini still recommended an 8-run Plackett-Burman design.

This would work as a first-level screening experiment. The choice between 8 or 12 run Plackett-Burman or a 16 run fractional factorial is probably not important here; all of them would work, but the greater the number of runs, the more data points and the more discriminating the design.

I am more familiar with a 12-run Placket Burman, so I’ll ask Gemini to analyse one that I have used many times before and compare its results to those from standard statistics software analysis. With 50% more data points, it should be more effective, but it is still only aimed at the first level of screening.

The standard advice in conducting this experiment is to randomise the run order. This is intended to spread the effects of any uncontrolled factors or environmental changes through the data. It may be of low relevance here and is often ignored. Gemini hasn’t mentioned it.

Conducting this experiment and collecting the output data, measuring distance to the point of impact, requires good practical experimental technique. This is the step that will make the difference between a useful model and a failed one. Repeatability and accuracy of the input settings and the response or output measure are vital to model quality.

The Statapult, with its pin positions, makes that easy to achieve for some of the settings. However, for the start angle, there is just a simple scale measured in degrees. That requires control using some kind of position stop to guarantee repeatability. Similarly, the distance to impact needs a clear, measurable indication, usually achieved with the aid of powdered talc to provide a “splash”.

Gemini doesn’t provide this kind of practical input or expertise, but let's see how it analyses the data collected. This was from a classroom training example with the operators well briefed on the importance of repeatability and accuracy.

Collecting the data for this takes at least half an hour with the aid of three people, so I’ll use the data from a recent training class. It is shown here with the standard descriptions and values of the input settings. Notice that some of these are discrete, and really only the Start Angle is a continuous metric. This is a feature of the Statapult with its fixed pin settings, but it won’t affect the analysis.

During analysis, these values will be substituted or coded as +1 and -1 to simplify the computation and then replaced with descriptions on completion. The Distance or output is measured in cm with a resolution of 0.1cm, that may imply a little more precision than is provided by a standard tape measure, but it won’t harm the analysis.

The charts or visuals provided by Gemini were not easy to interpret or particularly useful. The text responses above, while correct in principle, leave out the important information on statistical significance.

These are probability or p-values showing significance or relevance in the predictive model. The classic threshold used in hypothesis testing is 0.05, indicating a 5% probability that the factor is not significant in the model. Anything less than 5% is assumed to be significant and therefore relevant.

However, in DoE, the threshold is commonly set at 0.1 or 10% with the aim that a factor that is marginal is not discarded.

The next step in DoE is to eliminate factors with p-values above 0.1 and recompute the model and the p-value table. Eliminating factors provides more bandwidth for the model and will change the p-values used to assess significance.

This has changed the p-values, but it hasn’t changed the decision on whether to include or discard the factor in the model. The p-values computed by Gemini are identical to those provided by statistical software below after the model has been reduced to the significant terms.

The usual presentation of this data is to use a Pareto chart, which Gemini provided on prompting.

Prompt:

Create a Pareto Chart of Effects

Response:

Design of Experiments — Modelling

Case Study Details

In this next case study, I’ll investigate using AI in a DoE modelling experiment. The objective is to create an accurate prediction model using just 3 factors with a Full Factorial design. This will include sufficient repeat measurements to provide a high-quality model with R² at 0.99 or greater. A model that represents more than 99% of the data with less than 1% noise.

This is the natural next step after a screening experiment that determined the most important factors among a large number of possible inputs.

The example data that I have used comes from a classroom experiment with the Stataput in the hands of an experienced team. They developed their experimental process to eliminate variation, as far as is possible, from all input settings and from the response measurement. The model they achieved using statistics software has an R² value of 0.999, or less than 0.1% noise, a remarkable achievement. Note that this is a different Statapult with different set-up conditions than the previous screening experiment. We should not expect any similarity of models or significance of factors.

We will investigate the model they achieved using their input data in Gemin and compare it for accuracy and ease of use with their result from the statistics software.

We will again assume little knowledge of DoE and ask Gemini for advice.

This is the standard format for a coded model just using -1 and +1 input values and generic names for the inputs. It does not mention randomisation nor the number of repeats that should be applied. First we should update this with the common factor names and the real input high and low settings.

My textbook recommends 5 reps to aim for 95% confidence in the S model (model of standard deviation in the process). Since we didn’t ask for an S model but implied only a Y model (models the average of the process), let's set 4 reps with the aim of getting a good Y model.

This is good advice from Gemini, and when asked, it provided a single vertical table of 32 runs in random order. However, that considerably extends the time for conducting this experiment by a factor of 4 or more. It is much more efficient to collect all 4 reps one after the other. For that reason, most experimenters usually persuade themselves that the effects of external environmental changes are minimal and that randomisation can be dispensed with.

Our data was collected as an array table with four columns representing the repeats. The four measurements for each run D1, D2, D3, D4 were collected immediately one after the other. In this case the output was measured to the nearest cm. This whole experiment took around half an hour, similar to the first screening experiment. With randomisation it would have likely taken 2 hours.

Prompt:

Analyse the DoE modelling data representing 4 reps in D1, D2, D3, D4

Response:

Key Takeaway 1: Dominant Factors and Main Effects

The main effects of all three factors are highly statistically significant. The relative importance and magnitude of these effects can be visualized in the Pareto Chart of DoE Effects (alpha = 0.05).

• Start Angle (A) is the most dominant factor: Increasing the Start Angle from 150 to 170 degrees increases the response by 162.44 units on average (p < 0.001). This is by far the largest effect in the system.
• Stop Pin (B) has a strong negative effect: Increasing the Stop Pin from 3 to 4 decreases the response by 89.44 units on average (p < 0.001).
• Fixed Pin (C) has a moderate positive effect: Increasing the Fixed Pin from 1 to 2 increases the response by 44.31 units on average (p < 0.001).
• The Main Effects Plot illustrates these linear trends clearly, showing the steep positive slope for Start Angle, a steep negative slope for Stop Pin, and a moderate positive slope for Fixed Pin.

Key Takeaway 2: Significant Factor Interactions

The factors do not behave independently; instead, there are strong, statistically significant interactions that must be considered when optimizing the system.

• Start Angle & Stop Pin Interaction (A:B): This is the strongest interaction in the design (+25.06 effect, p < 0.001). As shown in Interaction: Start Angle & Stop Pin, the negative effect of changing the Stop Pin from 3 to 4 is much more severe at a Start Angle of 150 (decreasing the response by 114.50 units) than at a Start Angle of 170 (decreasing the response by 64.38 units).
• Start Angle & Fixed Pin Interaction (A:C): This interaction is also highly significant (+16.81 effect, p < 0.001). As shown in Interaction: Start Angle & Fixed Pin, the positive effect of changing the Fixed Pin from 1 to 2 is much stronger at a Start Angle of 170 (increasing the response by 61.13 units) than at a Start Angle of 150 (increasing the response by 27.50 units).
• Stop Pin & Fixed Pin Interaction (B:C): This interaction is not statistically significant (-1.56 effect, p = 0.256). The effect of the Stop Pin remains consistent regardless of which Fixed Pin level is used.
• Three-Way Interaction (A:B:C): The three-way interaction is statistically significant but very small (+3.94 effect, p = 0.007), suggesting a minor joint dependency between all three factors.

Key Takeaway 3: Exceptional Model Quality and Precision

The experiment was conducted with extremely high precision, making the conclusions highly reliable.

• High Reproducibility: The standard deviation across the 4 replicates (D1, D2, D3, D4) for each of the 8 runs is extremely low, ranging from a minimum of 1.00 (for Run 8: 170, 4, 2) to a maximum of only 5.91 (for Run 5: 150, 3, 2).
• Excellent Model Fit ($R^2 = 0.999$): A full factorial OLS regression model explains 99.9% of the total variance in the response. The residual mean square error is only 14.41, indicating that the average deviation of any individual replicate from its group mean is only about 3.79 units.

I have reproduced Gemini’s response at this stage in full, to show the text nature of its output. I found this quite difficult to interpret, being more used to tabulated values for coefficients and the significance of individual factors and interactions. I will ask Gemini to produce this in tabular form.

The table format is more familiar to me and easier in my opinion to interpret. For comparison the results from statistics software are below, showing precise agreement on all the coefficients and p values.

    $D = 256.7188 + 81.2188 A - 44.7188 B + 22.1563 C + 12.5313 AB + 8.4063 AC - 0.7813 BC + 1.9688 ABC$
    ${R^2 = 0.9988}$
  

For comparison here is the equivalent model equation and R² from other statistics software.