Showing posts with label Quality. Show all posts
Showing posts with label Quality. Show all posts

Wednesday, January 03, 2007

Seamless Integration: Testing 2.0!!

End-to-end seamless integration of tools is important in every domain, especially ALM. iTKO seem to have understood the concept and is delivering a Test Automation Platform (LISA) which not only allows automation of customer scenarios (spanning across various technologies), but also provides integration APIs to integrate with rest of the ALM tools.

For example:

A test can register a user on a web-portal, download a software, installs the bits on a local or remote system, runs commands to finish & verify installation, calls a web-service to confirm product registration and in the end queries the database to make sure all tables are correctly populated - all from within the context of the same test case!

There are even APIs available to expand the core functionality of LISA - like filters, assertions, companions, etc.

There is more... Once the test is automated, it can be automatically kicked off as part of cruise control, or other Ant integration, post result into a central database or TestDirector or homegrown test management system, automatically file a bug or integrate with requirement management system. Use of pure java, xml, open architecture and public APIs makes the seamless integration possible across the entire ALM domian.
The tool allows all data strategies I discussed in my August blog. The data driven testing is the key to leverage test scripts, lower maintenance and improve ROI.

Checkout Test Automation Tools blog. It nicely captures the automation framework requirements and the battle of build vs. buy. Remember, understanding the requirements and ability to clearly articulate the question is the first step in finding the answer!

Like Paris Hilton is Marketing 2.0, LISA seems to be Testing 2.0 for Web 2.0 :-)

Saturday, September 16, 2006

Quality Index (QI): Measure of Risk

Is it possible to capture the quality of an offering in a single metric, something like a Quality Index?

Before I answer this question, I'd like to reflect back on the definition of quality:

Quality: The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs. (ISO 8402: 1986, 3.1)
Note that the definition refers to stated and implied needs. What that means is Quality of a product which is considered "high" today (because it totally satisfies stated and implied needs) can go down tomorrow because of changes in implied needs. How do we measure that? We probably can't!

Let's re-write the questoin then,

Is it possible to capture the intrinsic quality of an offering in a single metric, something like a Quality Index?

This sounds more reasonable. But, there are over 100 metrics that can be argued to impact the quality of an offering. How do we make sure that we are capturing everything and our QI is based on the perfect algorithm?

Good news is that we don't need to look into hundreds of metrics. 80/20 rule applies here too! We can take 20% of the top variables to get the 80% of insight into intrinsic quality.

What if there is an error in the algorithm we choose? It is possible that our QI is off by 10% or even 20%. If e represents the error, then QI(Observed) = QI (Real) +/- e

If we use the consistent mechanism to capture QI, same error (e) will exist everytime we take a snapshot. And the best part is that majority of this error will cancel out when we plot QI numbers to look at its trend over time. The whole graph will be offset by e.

Note: The key to quality is consistency and repeatability. It is not really important what process we follow, as long as we can make sure that we can make it repeatable. Same concept applies to QI.

So, the answer is YES. We can capture the intrinsic quality of an offering in a single metric and use the trend analysis to keep tab on improvement. Remember TQM, it is all about continuous measurement and continuous improvement!!

Why calculate QI?

QI is not just a measure of quality, it is also a measure of risk associated with quality for your product offering. Here are some of the real advantages of having a QI for all your projects:
  • Get an insight into release readiness, especially in the agile development
  • QI dependency matrix can help identify problems much quicker in the SOA world
  • Mapping QI ranges with your Customer escalation data can give you an insight into what to expect when you release a product with QI less than 60 as compared to a product with QI greater than 90.
  • QI trend provides continuous feedback - required for control. It is easy to monitor when process is going out-of-control.
  • Easy for management to digest one number and drill down, if required.
Remember:
  • QI like number in itself is probably not meaningful. It's the trend that is relevant.
  • QI number can't be compared across companies in the industry, as there are a lot of variables in the "error" part, which won't cancel each other!
  • QI probably can't even be compared across different teams with-in the same company
Further Reading:

Tuesday, August 29, 2006

Automation: TQM Perspective

Quality has two perspectives - Validation & Verification.

In validation, we are checking Are we building the right thing? Basically, we are checking for conformance to customer requirements and quality attributes.

In verification, we are checking Are we building the thing right? We are more concerned about the process conformance and quality activities. Verification is more on the TQM side.

Automation provides a lot of benefits like: Repeatability, Predicability, Higher productivity, Lower costs, Shorter cycle times, Faster turn arounds, Higher confidence, and Insurance against human blunders. So, in reality Automaiton is nothing but means to achieve higher quality of processes and hence higher quality of products.

All processes that are part of Application Life-Cycle Management (ALM) must be automated to assure repeatability and keep processes in-control.

Starting form the requirements to bug-filing - everything should be automated.

  • Central Requirements Management
  • Automated unit, integration and functional testing
  • Central Code Repository, Version Control
  • Automated Request to Integrate Process
  • Automated builds (Nightly, weekly)
  • Automated Deployment
  • Automated installation of TestWare
  • Automated testbed setup and configuration
  • Automated Test execution (Functional & Non-functional both)
  • Automated Reporting
  • Automated Bug-filing!!
  • CRM Solutions - Customer escalation management
  • Project Program Management
  • End-to-End Integration and Data flow
Well, that is the ideal scenario. The closer we can get, the more successfull we are with our automation initiative. It you are a TQM fan, you'd agree that we need to improve our processes and build more quality into them which inturn will improve our product quality.

Automation has following benefits, which goes hand-in-hand with TQM benefits:
  • Provides a big picture of the overall process
  • Achieve higher adoption rates (CAP)
  • Easy to agree on standards
  • Data Archiving leads to Data Mining Possibilities
  • Historical data archives helps in decision making
  • Provides Process Traceability
  • Reports can reflect exact process state
  • Treand reports can help in gaining process maturity
  • Easy to monitor overall process
  • Email alerts in case the process goes Out-of-control
  • Defies Dilbert Model
  • Brings Focus and Accountability
  • Boosts Productivity
  • Reduces Costs
  • Saves Time
  • Higer Process Quality leads to Higher Product Quality
  • Easy to articulate SUCCESS
As I said before, it may not be possible to automate 100% of any process, but the closer we get, the better it is. Automation is like investing in infrastructure. The more solid our infrastructure is, more high we can rise. "We can't build an Empire State Building on a 6-inch foundation." If we want to achive high quality in our products, then we really need to have a very strong foundation, i.e very stable and high quality processes; and one way to achieve that is via Automation.

Final Thought:
Automation does not start and finish with Test Automation.

TQM in Software Development

Most of you may already know what TQM is and what its strengths are. But for those of you who don't and who always wonder, like me, how TQM can be applied in Software, I put-together this blog.

I always wondered what TQM (Total Quality Management) is and why we don't use it to imporve quality of Products and Services in Software. In Winter 2006, I took a course in Operations Management, as part of my MBA at Leavey School of Busineess, and got some answers. I hope this blog will help you in getting a deeper understanding of TQM and its role in Software Industry.

TQM Principle: Intrinsic Quality Control. Improve quality of processes to improve quality of process outcome, i.e. the product

TQM Origin and Background: In mid 1940s, Dr. W. Edward Deming picked up some of the ideas from Walter Shewhart (who discovered that quality can be measured, and that there are measures of variability) and developed what is known as TQM. In 1940s, Dr. Deming was working as an advisor in sampling at the Bureau of Census and later became a statistics professor at the New York University Business School. At that time, he had little success convincing American businesses to adopt TQM, but his management methods did gain a huge success in Japan.

While the Japanese were concentrating on producing quality products, businesses in the United States were more concerned with producing large quantities of products. Their emphasis on quantity at the expense of quality let the Japanese, with their inexpensive, high quality products, gain a substantial foothold in American markets.

In the 1970s and 1980s, many American companies, including Ford, IBM, and Xerox, began adopting Dr Deming’s principles of TQM. This gradually led to their regaining some of the markets previously lost to the Japanese.

TQM Definition: "TQM means that the organization's culture is defined by the constant attainment of satisfaction through an integrated system of tools, techniques, and training. Total Quality Management is a management style based upon producing quality service as defined by the customer. TQM is defined as a quality-centered, customer-focused, fact-based, team-driven, senior-management-led process to achieve an organization’s strategic imperative through continuous process improvement."
  • T = Total = everyone in the organization
  • Q = Quality = customer satisfaction
  • M = Management = people and processes
TQM Benefits: TQM has few short-term advantages. Most of its benefits are long-term and come into effect only after it is running smoothly for some time. In large organizations, it may take several years before long-term benefits are realized. Long-term benefits that may be expected from TQM are higher productivity, increased morale, reduced costs, and greater customer commitment. These benefits may lead to greater public support and improvement of an organization’s public image.

TQM & Software Industry:

Q. Can TQM be applied to Software Industry?

TQM was originated in the manifacturing sector but it has been successfuly adopted by almost every type of organization imaginable - for example - hotel management, highway maintenance, churches, schools & universities.

If you look closely at Software development, it is no different from any other industry. We develop software using processes. We know our processes are not perfect. What can we do to improve quality and bring more discpline into our processes? TQM has the answer.

Remember, we can't test quality into our software, we design in it. And the only way we can design quality in is by continuously monitoring and improving our processes.

Since TQM requires extensive statistical analysis to study processes and improve quality, we need be able to define a variable (or set of variables) to monitor a certain process. We should be able to then make a corrective action whenever we find that the process is going out-of-control. But, how do we know if the process is going out-of-control? Note: A process in-control means it's repeatable.

Q. How do we know if the process is going out-of-control?

Use Control-Charts. They are also known as XBAR and R-Charts. It's hard to explain without a diagram, but I'll try to explain. You measure a process variable and define the target mean i.e. it's acceptable average value. Note: To be able to begin monotoring a process, it must be in-control. Then define the Upper Control Limit (UCL) and Lower Control Limit (LCL). Use XBAR Charts to track if the process is going out-of-control. This URL provides more insight into how to define UCL and LCL. Note: if the LCL drops below some practical value, then choose zero.

Dilbert Syndrome: "When we treat symptoms, we get a step closer to Dilbert Syndrome. Making a wild guess at what went wrong and act on it to adopt a new unproven process."

PDCA Cycle: Plan-Do-Check-Act is also known as Shewhart Cycle. Instead of following a Dilbert Cycle, where one only react to changes, PDCA shows a more sophisticated way to welcome change. Look at the problem in detail and check what's wrong (Plan). Then try improvement in small steps (Do). Gather data and analyze results (Check). In the end, make a decision (Act) and repeat the cycle for next problem. It's common sense, isn't it? But we still find managers suffering from Dilbert Syndrome almost everyday!

Summary
  • We can't test quality into our software, we design in it
  • A chain is as strong as its weakest link. We must find and fix the bottlenecks first.
  • Processes needs to be continually improved
  • Don't treat symptoms. Don't follow Dilbert Model
  • The People who do the work generally knows best how to improve it
  • Use quantitative methods to support decisions, whenever possible
  • Plan a flight, and fly the plan

*Links and References*
  • Operations Management, OMIS 357 Class - Leavey School of Business
  • Software Practicum, CS 2335 Class - Georgia Institute of Technology
  • XBAR and R-Charts