Quality Index (QI): Measure of Risk

Is it possible to capture the quality of an offering in a single metric, something like a Quality Index?

Before I answer this question, I'd like to reflect back on the definition of quality:

Quality: The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs. (ISO 8402: 1986, 3.1)

Note that the definition refers to stated and implied needs. What that means is Quality of a product which is considered "high" today (because it totally satisfies stated and implied needs) can go down tomorrow because of changes in implied needs. How do we measure that? We probably can't!

Let's re-write the questoin then,

Is it possible to capture the intrinsic quality of an offering in a single metric, something like a Quality Index?

This sounds more reasonable. But, there are over 100 metrics that can be argued to impact the quality of an offering. How do we make sure that we are capturing everything and our QI is based on the perfect algorithm?

Good news is that we don't need to look into hundreds of metrics. 80/20 rule applies here too! We can take 20% of the top variables to get the 80% of insight into intrinsic quality.

What if there is an error in the algorithm we choose? It is possible that our QI is off by 10% or even 20%. If e represents the error, then QI(Observed) = QI (Real) +/- e

If we use the consistent mechanism to capture QI, same error (e) will exist everytime we take a snapshot. And the best part is that majority of this error will cancel out when we plot QI numbers to look at its trend over time. The whole graph will be offset by e.

Note: The key to quality is consistency and repeatability. It is not really important what process we follow, as long as we can make sure that we can make it repeatable. Same concept applies to QI.

So, the answer is YES. We can capture the intrinsic quality of an offering in a single metric and use the trend analysis to keep tab on improvement. Remember TQM, it is all about continuous measurement and continuous improvement!!

Why calculate QI?

QI is not just a measure of quality, it is also a measure of risk associated with quality for your product offering. Here are some of the real advantages of having a QI for all your projects:

• Get an insight into release readiness, especially in the agile development
• QI dependency matrix can help identify problems much quicker in the SOA world
• Mapping QI ranges with your Customer escalation data can give you an insight into what to expect when you release a product with QI less than 60 as compared to a product with QI greater than 90.
• QI trend provides continuous feedback - required for control. It is easy to monitor when process is going out-of-control.
• Easy for management to digest one number and drill down, if required.

Remember:

• QI like number in itself is probably not meaningful. It's the trend that is relevant.
  
• QI number can't be compared across companies in the industry, as there are a lot of variables in the "error" part, which won't cancel each other!
• QI probably can't even be compared across different teams with-in the same company

Further Reading:

• Read It all boils down to Metrics! for more information!

Automation: TQM Perspective

Quality has two perspectives - Validation & Verification.

In validation, we are checking Are we building the right thing? Basically, we are checking for conformance to customer requirements and quality attributes.

In verification, we are checking Are we building the thing right? We are more concerned about the process conformance and quality activities. Verification is more on the TQM side.

Automation provides a lot of benefits like: Repeatability, Predicability, Higher productivity, Lower costs, Shorter cycle times, Faster turn arounds, Higher confidence, and Insurance against human blunders. So, in reality Automaiton is nothing but means to achieve higher quality of processes and hence higher quality of products.

All processes that are part of Application Life-Cycle Management (ALM) must be automated to assure repeatability and keep processes in-control.

Starting form the requirements to bug-filing - everything should be automated.

• Central Requirements Management
  
• Automated unit, integration and functional testing
• Central Code Repository, Version Control
  
• Automated Request to Integrate Process
• Automated builds (Nightly, weekly)
  
• Automated Deployment
  
• Automated installation of TestWare
• Automated testbed setup and configuration
• Automated Test execution (Functional & Non-functional both)
  
• Automated Reporting
• Automated Bug-filing!!
• CRM Solutions - Customer escalation management
• Project Program Management
  
• End-to-End Integration and Data flow
  

Well, that is the ideal scenario. The closer we can get, the more successfull we are with our automation initiative. It you are a TQM fan, you'd agree that we need to improve our processes and build more quality into them which inturn will improve our product quality.

Automation has following benefits, which goes hand-in-hand with TQM benefits:

• Provides a big picture of the overall process
• Achieve higher adoption rates (CAP)
• Easy to agree on standards
• Data Archiving leads to Data Mining Possibilities
• Historical data archives helps in decision making
• Provides Process Traceability
• Reports can reflect exact process state
• Treand reports can help in gaining process maturity
• Easy to monitor overall process
• Email alerts in case the process goes Out-of-control
• Defies Dilbert Model
• Brings Focus and Accountability
• Boosts Productivity
• Reduces Costs
• Saves Time
• Higer Process Quality leads to Higher Product Quality
• Easy to articulate SUCCESS

As I said before, it may not be possible to automate 100% of any process, but the closer we get, the better it is. Automation is like investing in infrastructure. The more solid our infrastructure is, more high we can rise. "We can't build an Empire State Building on a 6-inch foundation." If we want to achive high quality in our products, then we really need to have a very strong foundation, i.e very stable and high quality processes; and one way to achieve that is via Automation.

Final Thought:
Automation does not start and finish with Test Automation.