Automation: TQM Perspective

Quality has two perspectives - Validation & Verification.

In validation, we are checking Are we building the right thing? Basically, we are checking for conformance to customer requirements and quality attributes.

In verification, we are checking Are we building the thing right? We are more concerned about the process conformance and quality activities. Verification is more on the TQM side.

Automation provides a lot of benefits like: Repeatability, Predicability, Higher productivity, Lower costs, Shorter cycle times, Faster turn arounds, Higher confidence, and Insurance against human blunders. So, in reality Automaiton is nothing but means to achieve higher quality of processes and hence higher quality of products.

All processes that are part of Application Life-Cycle Management (ALM) must be automated to assure repeatability and keep processes in-control.

Starting form the requirements to bug-filing - everything should be automated.

• Central Requirements Management
  
• Automated unit, integration and functional testing
• Central Code Repository, Version Control
  
• Automated Request to Integrate Process
• Automated builds (Nightly, weekly)
  
• Automated Deployment
  
• Automated installation of TestWare
• Automated testbed setup and configuration
• Automated Test execution (Functional & Non-functional both)
  
• Automated Reporting
• Automated Bug-filing!!
• CRM Solutions - Customer escalation management
• Project Program Management
  
• End-to-End Integration and Data flow
  

Well, that is the ideal scenario. The closer we can get, the more successfull we are with our automation initiative. It you are a TQM fan, you'd agree that we need to improve our processes and build more quality into them which inturn will improve our product quality.

Automation has following benefits, which goes hand-in-hand with TQM benefits:

• Provides a big picture of the overall process
• Achieve higher adoption rates (CAP)
• Easy to agree on standards
• Data Archiving leads to Data Mining Possibilities
• Historical data archives helps in decision making
• Provides Process Traceability
• Reports can reflect exact process state
• Treand reports can help in gaining process maturity
• Easy to monitor overall process
• Email alerts in case the process goes Out-of-control
• Defies Dilbert Model
• Brings Focus and Accountability
• Boosts Productivity
• Reduces Costs
• Saves Time
• Higer Process Quality leads to Higher Product Quality
• Easy to articulate SUCCESS

As I said before, it may not be possible to automate 100% of any process, but the closer we get, the better it is. Automation is like investing in infrastructure. The more solid our infrastructure is, more high we can rise. "We can't build an Empire State Building on a 6-inch foundation." If we want to achive high quality in our products, then we really need to have a very strong foundation, i.e very stable and high quality processes; and one way to achieve that is via Automation.

Final Thought:
Automation does not start and finish with Test Automation.

TQM in Software Development

Most of you may already know what TQM is and what its strengths are. But for those of you who don't and who always wonder, like me, how TQM can be applied in Software, I put-together this blog.

I always wondered what TQM (Total Quality Management) is and why we don't use it to imporve quality of Products and Services in Software. In Winter 2006, I took a course in Operations Management, as part of my MBA at Leavey School of Busineess, and got some answers. I hope this blog will help you in getting a deeper understanding of TQM and its role in Software Industry.

TQM Principle: Intrinsic Quality Control. Improve quality of processes to improve quality of process outcome, i.e. the product

TQM Origin and Background: In mid 1940s, Dr. W. Edward Deming picked up some of the ideas from Walter Shewhart (who discovered that quality can be measured, and that there are measures of variability) and developed what is known as TQM. In 1940s, Dr. Deming was working as an advisor in sampling at the Bureau of Census and later became a statistics professor at the New York University Business School. At that time, he had little success convincing American businesses to adopt TQM, but his management methods did gain a huge success in Japan.

While the Japanese were concentrating on producing quality products, businesses in the United States were more concerned with producing large quantities of products. Their emphasis on quantity at the expense of quality let the Japanese, with their inexpensive, high quality products, gain a substantial foothold in American markets.

In the 1970s and 1980s, many American companies, including Ford, IBM, and Xerox, began adopting Dr Deming’s principles of TQM. This gradually led to their regaining some of the markets previously lost to the Japanese.

TQM Definition: "TQM means that the organization's culture is defined by the constant attainment of satisfaction through an integrated system of tools, techniques, and training. Total Quality Management is a management style based upon producing quality service as defined by the customer. TQM is defined as a quality-centered, customer-focused, fact-based, team-driven, senior-management-led process to achieve an organization’s strategic imperative through continuous process improvement."

• T = Total = everyone in the organization
• Q = Quality = customer satisfaction
• M = Management = people and processes

TQM Benefits: TQM has few short-term advantages. Most of its benefits are long-term and come into effect only after it is running smoothly for some time. In large organizations, it may take several years before long-term benefits are realized. Long-term benefits that may be expected from TQM are higher productivity, increased morale, reduced costs, and greater customer commitment. These benefits may lead to greater public support and improvement of an organization’s public image.

TQM & Software Industry:

Q. Can TQM be applied to Software Industry?

TQM was originated in the manifacturing sector but it has been successfuly adopted by almost every type of organization imaginable - for example - hotel management, highway maintenance, churches, schools & universities.

If you look closely at Software development, it is no different from any other industry. We develop software using processes. We know our processes are not perfect. What can we do to improve quality and bring more discpline into our processes? TQM has the answer.

Remember, we can't test quality into our software, we design in it. And the only way we can design quality in is by continuously monitoring and improving our processes.

Since TQM requires extensive statistical analysis to study processes and improve quality, we need be able to define a variable (or set of variables) to monitor a certain process. We should be able to then make a corrective action whenever we find that the process is going out-of-control. But, how do we know if the process is going out-of-control? Note: A process in-control means it's repeatable.

Q. How do we know if the process is going out-of-control?

Use Control-Charts. They are also known as XBAR and R-Charts. It's hard to explain without a diagram, but I'll try to explain. You measure a process variable and define the target mean i.e. it's acceptable average value. Note: To be able to begin monotoring a process, it must be in-control. Then define the Upper Control Limit (UCL) and Lower Control Limit (LCL). Use XBAR Charts to track if the process is going out-of-control. This URL provides more insight into how to define UCL and LCL. Note: if the LCL drops below some practical value, then choose zero.

Dilbert Syndrome: "When we treat symptoms, we get a step closer to Dilbert Syndrome. Making a wild guess at what went wrong and act on it to adopt a new unproven process."

PDCA Cycle: Plan-Do-Check-Act is also known as Shewhart Cycle. Instead of following a Dilbert Cycle, where one only react to changes, PDCA shows a more sophisticated way to welcome change. Look at the problem in detail and check what's wrong (Plan). Then try improvement in small steps (Do). Gather data and analyze results (Check). In the end, make a decision (Act) and repeat the cycle for next problem. It's common sense, isn't it? But we still find managers suffering from Dilbert Syndrome almost everyday!

Summary

• We can't test quality into our software, we design in it
• A chain is as strong as its weakest link. We must find and fix the bottlenecks first.
• Processes needs to be continually improved
• Don't treat symptoms. Don't follow Dilbert Model
• The People who do the work generally knows best how to improve it
• Use quantitative methods to support decisions, whenever possible
• Plan a flight, and fly the plan

*Links and References*

• Operations Management, OMIS 357 Class - Leavey School of Business
• Software Practicum, CS 2335 Class - Georgia Institute of Technology
• XBAR and R-Charts

Data Strategy (DS)

A lot of test automation engineers get confused when it comes to Data Driven Testing (DDT)

• How tests should be structured?
• What seed data to use?
• When and How to load Data?
• When to flush data?
• How to bind test scripts with test data?

These are some of the very basic questions that must be answered in conjuction with the tooling and test automation strategies. Note: DDT is more than just passing arguments to a test library function.


When and How to load Data?
Well, it is imporant to understand data scope in conjunction with the test case and the automation strategy to address the "WHEN" question. There are several ways to address this.

• DS1: Consider most of your data as "seed data" and load it upfront before actually kicking off your test automation. In this case, data cleanup is done after test scripts exit. This is the most preferrred strategy for Load & Performance testing. Data can be loaded directly into the database using SQL scripts or by running data suite. Data suite executes a set of test cases that takes care of loading and polulating the seed data.
  

• DS2: Load data for every test suite and cleanup at the end of the suite. Make sure environment is clean for the next suite to run.

• DS3: Load data for every test case. Every test is independent and takes care of its own data. This strategy is very cumbersome as the data is not leveraged among test cases.

• DS4: Combo Strategy. Load high level data (which is going to be used most often) upfront and leave rest of the data to every test case.
  

In all of the above data strategies, test cases can use factory classes to get pre-populated data objects. These factory classes can encapsulate the whole data strategy, including creation and deletion of data.

Note: Remember, in DDT, you only need to worry about the end-user-data. Auto generated information, like session-ids, does not need to be part of the DS, unless you plan to skip some steps. Your automation framework must provide mechanisms to pass auto-generated data across test cases and test suites.

Final Thought:
Data Strategies can get really complex, and sometimes even more complex than your actual test cases. Try not to drift away too deep into fancy data strategies and loose sight of the real thing!

Data Driven Testing (DDT)

We all have heard about data driven testing and how it can improve our ROI exponentially and at the same time reduce maintenance costs. We all believe in it, but we still don't practice it to the extent we should. This blog describes the concepts using a pictorial notation in an effort to bringforth the structural thinking that one needs to implement DDT.

A picture is worth thousand words:-



Step 1:

You start your automation activity with some test procedures in hand. Refer to column 1. A test step is reflected by a donut, hexagon, triangle, etc. These different building blocks combine togather to form a structure, which depicts a test case. Different shapes and their combinations reflect various test cases in the test suite. As you can see, a test case can have many different steps as part of its procedure.

Step 2:

In next step, you start automating one test case at a time. When you encounter a second test case which contains the similar step that you automated before, you move that particular step into what we call "Test Library". You must filter out all common functionality into test libraries over time. Refer to column 2. It shows how you can take common shapes and make library out of them.

Step 3:

As part of Step 3, you separate variability from static functionality. This is the most important step in DDT. Most of the times the variability is the data input; but you may have more variability in your system that you may also want to identify and manage it as data. More the static functionality you can identify, more you'll be able to leverage your automated scripts across datasets. Refer to column 3. All colors depict data. In this step you also make your test libraries data driven.

Step 4:

Once you separate the variability from the core (static) functionality, you'll be able to identify opportunities for more test libraries or leverage existing libraries. Go through this iteration to make sure that you are leveraging your test case libraries to the fullest. Convert more common functionalities into libraries, if required. Refer to column 4.

The more you will think in terms of DDT and leveraging libraries, the easier it will get to automate more in less time. Apart from achieving higher productivity, you will realize that your automated test cases are a lot easier to maintain. Whenever some common functionality changes, you just need to update the corresponding test library. Just imagine updating hundreds of test cases, in case you are not using libraries. Similarly, when some data requirements changes, you will not have to change your automated scripts - just update your datasets and you will be done.

Final Thought:
Tools only provide mechanisms to accomplish DDT or create test libraries, but if we choose not to practice these concepts, we end up creating large autmation code which is both hard to maintain and sustain.