Meta-Data data driven testing!

If you are involved in test automation, you'll know the importance and challenges of managing data. In my experience, developers are the worst testers mainly because they fail to understand the concept of data driven testing. Even good developers, who care to test, hard code test data in their unit test cases. To me, coming up with a good Data Strategy is more than half of test automation challenge.

As part of my profession, I teach and train students in SOA lifecycle quality. Over period of time, I have realized that my basic theory of iteration (refer to my Data Driven Testing (DDT) blog) doesn't really explain all different types of data that one needs to take into consideration in order to define a good test strategy. It only explains, at a high level, how one can extract randomness and variability from a test and move it into external data asset.

Modular testing is the key to re-usability and lower maintenance. Apart from creating test libraries and using building blocks, it is also important to understand how to organize different types of data outside of a test script.

Following diagram illustrates how different types of data can be organized around a test procedure/logic. Different types of data must not be cobbled together, because it almost always leads to increased maintenance costs and chaos.

• Removing Meta-data from the test logic not only makes the logic more reusable, but also lowers its maintenance. Change in meta-data can be easily handled by updating appropriate properties files. Example: SQL scripts, tag names, component names. Meta-Data has 1-1 relationship with the test case.
• Input Test Data is core to the business functionality/scenario. Example: username and password to login into a website. A single test case can have multiple copies of the same data, generally referred to as rows inside a dataset.
  
• Output Test Data is used for validation purposes. Validation of responses can be done using regular expressions or against the actual data. In case, generic validations (like regular expression checking size of the confirmation id) are not used, output data is mapped 1-1 with the input data.
  
• Server Environment Data refers to the endpoint data which varies between environments. Ability to change this data (at the execution time) allows the same set of test cases to be run against multiple environments.
  
• Client Environment Data refers to local environment specific to the test framework and project.
  
• Staging data allows the same test asset to be staged in different ways.
  

Seamless Integration: Testing 2.0!!

End-to-end seamless integration of tools is important in every domain, especially ALM. iTKO seem to have understood the concept and is delivering a Test Automation Platform (LISA) which not only allows automation of customer scenarios (spanning across various technologies), but also provides integration APIs to integrate with rest of the ALM tools.

For example:

A test can register a user on a web-portal, download a software, installs the bits on a local or remote system, runs commands to finish & verify installation, calls a web-service to confirm product registration and in the end queries the database to make sure all tables are correctly populated - all from within the context of the same test case!

There are even APIs available to expand the core functionality of LISA - like filters, assertions, companions, etc.

There is more... Once the test is automated, it can be automatically kicked off as part of cruise control, or other Ant integration, post result into a central database or TestDirector or homegrown test management system, automatically file a bug or integrate with requirement management system. Use of pure java, xml, open architecture and public APIs makes the seamless integration possible across the entire ALM domian.

The tool allows all data strategies I discussed in my August blog. The data driven testing is the key to leverage test scripts, lower maintenance and improve ROI.

Checkout Test Automation Tools blog. It nicely captures the automation framework requirements and the battle of build vs. buy. Remember, understanding the requirements and ability to clearly articulate the question is the first step in finding the answer!

Like Paris Hilton is Marketing 2.0, LISA seems to be Testing 2.0 for Web 2.0 :-)

Understanding ALM

It is important to understand the complete life-cycle of an application to know and manage risk around it. ALM links all pieces of the puzzle to give us a complete picture and a bird-eye view. One can look at the life-cyle from different perspectives - For example: it can be read to reflect risk from quality standpoint.

Every software project starts with a simple concept and follows eight typical steps as shown in the following ALM diagram:


In the first step, Business Analysts gather requirements from customers (voice-of-customer forms, surveys, polls) and document them in the requirements database called Product Requirements Definition (PRD). Once the requirements are ironed out, Technical Analysts convert these requirements into Functional Specifications Definition (FSD) in terms of User Cases (UML diagrams). The process then goes through the development cycle, where developers, release engineers, and testers work together to deliver the requirements in form of a product or a service offering.

In this process, Quality Assurance (QA) team is responsible for the validation (Are we building the right thing?) and verification (Are we building the thing right?) of engineering deliverables.

The above ALM diagram summarizes different steps of a typical software development process. Companies have various adaptations of this overall process with the following steps being crucial and unifying.

• Requirements Management
• Test Management and Test Automation
• Release Management

There are different players in the ALM industry (top ones being Mercury and Borland) that provide tools for different pieces of the puzzle. More and more companies are focusing on automation frameworks, which aid testing. However, if you really want to improve quality of your products, you must "Cease dependence on inspection to achieve quality" (Deming's TQM principle)

Risk is injected into the system everytime some information (or some work-product) exchange hands. We must have tools that can measure the quality of the processes (i.e. ALM sub-processes) to be able to deliver a really good quality product.

Watts S. Humphrey is the poineer of TSP and PSP processes, which uses software engineering discipline to deliver higher quality products. Look at this video for more enlightenment.

Trackback URL: Understanding ALM

Quality Index (QI): Measure of Risk

Is it possible to capture the quality of an offering in a single metric, something like a Quality Index?

Before I answer this question, I'd like to reflect back on the definition of quality:

Quality: The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs. (ISO 8402: 1986, 3.1)

Note that the definition refers to stated and implied needs. What that means is Quality of a product which is considered "high" today (because it totally satisfies stated and implied needs) can go down tomorrow because of changes in implied needs. How do we measure that? We probably can't!

Let's re-write the questoin then,

Is it possible to capture the intrinsic quality of an offering in a single metric, something like a Quality Index?

This sounds more reasonable. But, there are over 100 metrics that can be argued to impact the quality of an offering. How do we make sure that we are capturing everything and our QI is based on the perfect algorithm?

Good news is that we don't need to look into hundreds of metrics. 80/20 rule applies here too! We can take 20% of the top variables to get the 80% of insight into intrinsic quality.

What if there is an error in the algorithm we choose? It is possible that our QI is off by 10% or even 20%. If e represents the error, then QI(Observed) = QI (Real) +/- e

If we use the consistent mechanism to capture QI, same error (e) will exist everytime we take a snapshot. And the best part is that majority of this error will cancel out when we plot QI numbers to look at its trend over time. The whole graph will be offset by e.

Note: The key to quality is consistency and repeatability. It is not really important what process we follow, as long as we can make sure that we can make it repeatable. Same concept applies to QI.

So, the answer is YES. We can capture the intrinsic quality of an offering in a single metric and use the trend analysis to keep tab on improvement. Remember TQM, it is all about continuous measurement and continuous improvement!!

Why calculate QI?

QI is not just a measure of quality, it is also a measure of risk associated with quality for your product offering. Here are some of the real advantages of having a QI for all your projects:

• Get an insight into release readiness, especially in the agile development
• QI dependency matrix can help identify problems much quicker in the SOA world
• Mapping QI ranges with your Customer escalation data can give you an insight into what to expect when you release a product with QI less than 60 as compared to a product with QI greater than 90.
• QI trend provides continuous feedback - required for control. It is easy to monitor when process is going out-of-control.
• Easy for management to digest one number and drill down, if required.

Remember:

• QI like number in itself is probably not meaningful. It's the trend that is relevant.
  
• QI number can't be compared across companies in the industry, as there are a lot of variables in the "error" part, which won't cancel each other!
• QI probably can't even be compared across different teams with-in the same company

Further Reading:

• Read It all boils down to Metrics! for more information!

Only 7% say SOA exceeded expectations!!

Interesting survey on SOA Adoption: "The Dark Side of SOA"

Out of 273 business and technical professionals:

1. 24% say projects fall short of expectations
2. 55% say SOA introduced more IT complexity
3. 41% say SOA projects cost more than expected
4. Only 7% say SOA exceeded expectations!!

According to HP's SOA executive director, Terri Schoenrock, these failures are because of "misalignment between business and IT goals." That's probably true. To add to this, I think, the program management teams are the ones that don't understand the power of SOA and how it impacts different functional groups. Since PMs are the facilitators and co-ordinators, the failure of SOA projects fall on their shoulders.

The article also mentions "As software grows over hundreds and thousands of components slated for reuse, and Web Services extend beyond company walls, having right tools to manage them is critical."

Probably it's not even management's fault. It is the lack of right tools that has crippled their ability manage SOA projects and their interdependencies effectively. SOA is about busines agility and continuous change!

By reducing the size of projects, and increasing the overall number of projects, SOA has created problems for the IT organization and the management. Companies don't have right tools that can help project managers to manage more projects, QA to test more efficiently and IT to deploy more frequently. Legacy tools are tied with legacy processes, which doesn't fit perfectly for SOA needs.

New technology (SOA) is pushing for new processes (in IT, Development, Project Management, and QA) and the new processes won't be efficient without the new tools.

The article also points at the SOA governance issues for the failures. According to IBM's VP of Web Services and SOA, Michael Liebow, companies with savvy executive teams tend to be most efective at governing SOA projects!

Tracback URL: Only 7% say SOA exceeded expectations!!

Understanding SOA

SOA is the buzz word these days. Gartner reports predict that more than 80% of applications will be based on SOA by year 2009. So, what is SOA and why is it picking up so fast?

SOA means different things to different people. Different parts of organizations have different motives behind adopting SOA. It has something for everyone!

We all know SOA is "Service Oriented Architecture", but what does it really mean? And more importantly what does it mean to an engineer? What does it mean to an IT organization? What does it mean to the top management? And what does it mean to a company as a whole?

Distributed Computing

Distributed computing is not new. Over the past 2 decades, we have seen it grown from simple 2-tier client-server architecture to 3-tier and then N-tier and p-2-p architectures. The goal of a distributed system has always been scalability, availability, and flexibility. Moreover, distributed architectures have helped companies to better align their resources (both human and hardware)

SOA evolution

Distributed computing allowed companies to break their systems into smaller components. This helped companies to realize the fact that a majority of these smaller components are not related to their core business and, in fact, are common across different companies and even industries. For example, entitlement and billing systems, payment systems, catalogue management, application infrastructure layers and even middle wares.

With the bust of dot-com era, companies started thinking about ways to lower costs. One of the most obvious was to out-source the non-core aspects of the business. The outsourcing and consulting companies find this really interesting, as they started leveraging their experience across companies and industries. The need for openness and ability to consume third party services led to the advent of web-services and Service Oriented Architecture (SOA).

Software as a Service (SaaS) and SOA

Software delivery as a service has grown rapidly as more and more companies have realized that they can keep the customer relationship active by delivering software as a service, rather than delivering one-time products. I see SOA as a SaaS enabler.

What does SOA mean to an Architect?

SOA provides loosely coupled components. Each component is implemented using open standards for flexibility and future expandability. Architects need to make sure that there is no Single Point of Failure (SPOF) in the end-to-end system and, in some cases, are forced to implement extra caching mechanisms to improve availability and reliability. SOA brings simplicity back to architectures.

What does SOA mean to Development?

To developers, SOA means smaller teams and shorter development cycles. However, SOA also means more coordination across different teams and requires developers to follow standards.

What does SOA mean to an IT organization?

SOA enables faster release trains. With SOA, not all components needs to be upgraded at once. That means Bug fixes can be delivered faster, safer and easier. Less downtime and happy customer. SOA provides business agility. SOA enables continuous change!

What does SOA mean to Management?

SOA means large number of smaller projects rather than one monolithic project. Till date, it has been a nightmare for management, as they have not been able to grasp the power of SOA and the concept of smaller projects. To management, it only means more projects. Since Software development has been famous for not following discipline and processes, SOA has caused more harm than good. SOA requires engineering discipline and standardization.

A lot of organizations have been caught in the messy middle - the market environmental pressures are forcing them to enable SaaS and adopt faster release trains, Technology office and architects are pushing SOA, and the managers are stuck with their dogmatic viewpoints around waterfall methodologies. Engineers are confused, not knowing what to do, and quality is the last thing on anyone's mind.

The irony is that SOA goes hand-in-hand with agile development methodologies like Scrum. If and only if the management can be trained in SOA and newer development mythologies like Scrum, the companies can dream of coming out of this messy middle.

Changing market environment and industry pressures have forced companies to adopt SOA, but not everyone in management is even aware of the change. Management needs to be trained with new ALM tools to be successful.

What does SOA mean to a Company?

SOA enabled applications live longer and the ROI curve becomes very lucrative very fast. For companies, SOA is a blessing (if only they can deliver it right).

Final Thought:
SOA requires better ALM tools and integration of processes. The dependency among different components of SOA is not trivial. Sophisticated tools are required to orchestrate the whole application life-cycle management.

Trackback URL: Understanding SOA

Automation: Beyond Testing

Automation is the key to productivity in any functional group in any organization. But for majority of people:

• "Automation" = "Test Automation"
  

The general perception is that we should spend all our resources in automating test cases to cut-down test-cycle times! Well, that's not entirely true. Test cases are a small piece of the overall puzzle. Automating test cases is important, but it alone does not gaurantee reduced cycle time and faster release trains.

We must automate not only the overall testing process (which is far more than just automating test cases), but also other processes that takes up considerable amount of resources and are prone to errors. The biggest problem with automating processes is getting buy-in from the top management and all the cross-functional teams they impact. Moreover, some people see automation as a threat to their job and hence never fully support the idea. However, the fact remains that to boost productivity, one must maximize the use of labor and capital - which can be best achieved via Automation!!

Automation has following benefits:

• An automated process tends to live longer and ROI follow an exponencial curve.
• Automated processes are a lot easy to follow.
• Automation provide repeatability and consistency and boosts the quality of the process itself.
  
• Good quality processes results in good quality products (TQM principle)
• Employees get more frustrated by the process relaeted issus. A lot of product related issues are there because of broken processes.
  
• Process Automation makes the upper managaemet's life easy. It generates data and trends which can significantly impact decision making.
  
• Automated process reduces the ramp-up time for new comers, as processes are a lot easier to understand, once automated.
  
• Process Automation is an investment in the infrastructure.
  

So, how imporant is it for a team to invest in automation beyond testing? The answer is "It Depends." It depends on the complexity of the team, its structure, size and the amount of decentralization. Processes becomes overhead when your teams are small and localized, like in a startup. Most startups hate processes. Whereas the processes become really important when team grows in size, members work from home and are distributed globally in different timezones. Processes also makes sense when team members are of mixed experience as a result of attrition and growth, and there is a growing need for cross-functional collaboration.

In case you are wondering which process to automate, here is a small list of ALM sub-processes, which are a good pick for automation:

• Requirements Management
• Version Control, Build and Release Management
  
• Change Management, Request to Integrate (RTI) Process
• Patch Management
• Request for Change (RFC) Process
• QA Acceptance, Sanity testing
• Overall Testing Process (includes test management and test automation)
• Continuous Integration
  
• Deployment
• Defect Management
  
• Customer Escalation Management, Issue Tracking
• Program Management
• Monitoring (servers, errors, thresholds, CPU, memory, I/O, response times, availability)
  
• End-to-end real-time reporting
• Dependency Management (must for SOA applications)
  

Process automation doesn't mean going out of limits and defining a process and then automating it. Start by identidying iterative activites that happen very frequently and are annoying. For large and distributed teams, where it's hard to get a handle on productivity, processes are a breeze. And if there are processes, they need to be automated to bring standardization, acceptance, speed, and higher overall productivity.

Final Thought:
If there is a process, it better be automated!

Automation: TQM Perspective

Quality has two perspectives - Validation & Verification.

In validation, we are checking Are we building the right thing? Basically, we are checking for conformance to customer requirements and quality attributes.

In verification, we are checking Are we building the thing right? We are more concerned about the process conformance and quality activities. Verification is more on the TQM side.

Automation provides a lot of benefits like: Repeatability, Predicability, Higher productivity, Lower costs, Shorter cycle times, Faster turn arounds, Higher confidence, and Insurance against human blunders. So, in reality Automaiton is nothing but means to achieve higher quality of processes and hence higher quality of products.

All processes that are part of Application Life-Cycle Management (ALM) must be automated to assure repeatability and keep processes in-control.

Starting form the requirements to bug-filing - everything should be automated.

• Central Requirements Management
  
• Automated unit, integration and functional testing
• Central Code Repository, Version Control
  
• Automated Request to Integrate Process
• Automated builds (Nightly, weekly)
  
• Automated Deployment
  
• Automated installation of TestWare
• Automated testbed setup and configuration
• Automated Test execution (Functional & Non-functional both)
  
• Automated Reporting
• Automated Bug-filing!!
• CRM Solutions - Customer escalation management
• Project Program Management
  
• End-to-End Integration and Data flow
  

Well, that is the ideal scenario. The closer we can get, the more successfull we are with our automation initiative. It you are a TQM fan, you'd agree that we need to improve our processes and build more quality into them which inturn will improve our product quality.

Automation has following benefits, which goes hand-in-hand with TQM benefits:

• Provides a big picture of the overall process
• Achieve higher adoption rates (CAP)
• Easy to agree on standards
• Data Archiving leads to Data Mining Possibilities
• Historical data archives helps in decision making
• Provides Process Traceability
• Reports can reflect exact process state
• Treand reports can help in gaining process maturity
• Easy to monitor overall process
• Email alerts in case the process goes Out-of-control
• Defies Dilbert Model
• Brings Focus and Accountability
• Boosts Productivity
• Reduces Costs
• Saves Time
• Higer Process Quality leads to Higher Product Quality
• Easy to articulate SUCCESS

As I said before, it may not be possible to automate 100% of any process, but the closer we get, the better it is. Automation is like investing in infrastructure. The more solid our infrastructure is, more high we can rise. "We can't build an Empire State Building on a 6-inch foundation." If we want to achive high quality in our products, then we really need to have a very strong foundation, i.e very stable and high quality processes; and one way to achieve that is via Automation.

Final Thought:
Automation does not start and finish with Test Automation.

TQM in Software Development

Most of you may already know what TQM is and what its strengths are. But for those of you who don't and who always wonder, like me, how TQM can be applied in Software, I put-together this blog.

I always wondered what TQM (Total Quality Management) is and why we don't use it to imporve quality of Products and Services in Software. In Winter 2006, I took a course in Operations Management, as part of my MBA at Leavey School of Busineess, and got some answers. I hope this blog will help you in getting a deeper understanding of TQM and its role in Software Industry.

TQM Principle: Intrinsic Quality Control. Improve quality of processes to improve quality of process outcome, i.e. the product

TQM Origin and Background: In mid 1940s, Dr. W. Edward Deming picked up some of the ideas from Walter Shewhart (who discovered that quality can be measured, and that there are measures of variability) and developed what is known as TQM. In 1940s, Dr. Deming was working as an advisor in sampling at the Bureau of Census and later became a statistics professor at the New York University Business School. At that time, he had little success convincing American businesses to adopt TQM, but his management methods did gain a huge success in Japan.

While the Japanese were concentrating on producing quality products, businesses in the United States were more concerned with producing large quantities of products. Their emphasis on quantity at the expense of quality let the Japanese, with their inexpensive, high quality products, gain a substantial foothold in American markets.

In the 1970s and 1980s, many American companies, including Ford, IBM, and Xerox, began adopting Dr Deming’s principles of TQM. This gradually led to their regaining some of the markets previously lost to the Japanese.

TQM Definition: "TQM means that the organization's culture is defined by the constant attainment of satisfaction through an integrated system of tools, techniques, and training. Total Quality Management is a management style based upon producing quality service as defined by the customer. TQM is defined as a quality-centered, customer-focused, fact-based, team-driven, senior-management-led process to achieve an organization’s strategic imperative through continuous process improvement."

• T = Total = everyone in the organization
• Q = Quality = customer satisfaction
• M = Management = people and processes

TQM Benefits: TQM has few short-term advantages. Most of its benefits are long-term and come into effect only after it is running smoothly for some time. In large organizations, it may take several years before long-term benefits are realized. Long-term benefits that may be expected from TQM are higher productivity, increased morale, reduced costs, and greater customer commitment. These benefits may lead to greater public support and improvement of an organization’s public image.

TQM & Software Industry:

Q. Can TQM be applied to Software Industry?

TQM was originated in the manifacturing sector but it has been successfuly adopted by almost every type of organization imaginable - for example - hotel management, highway maintenance, churches, schools & universities.

If you look closely at Software development, it is no different from any other industry. We develop software using processes. We know our processes are not perfect. What can we do to improve quality and bring more discpline into our processes? TQM has the answer.

Remember, we can't test quality into our software, we design in it. And the only way we can design quality in is by continuously monitoring and improving our processes.

Since TQM requires extensive statistical analysis to study processes and improve quality, we need be able to define a variable (or set of variables) to monitor a certain process. We should be able to then make a corrective action whenever we find that the process is going out-of-control. But, how do we know if the process is going out-of-control? Note: A process in-control means it's repeatable.

Q. How do we know if the process is going out-of-control?

Use Control-Charts. They are also known as XBAR and R-Charts. It's hard to explain without a diagram, but I'll try to explain. You measure a process variable and define the target mean i.e. it's acceptable average value. Note: To be able to begin monotoring a process, it must be in-control. Then define the Upper Control Limit (UCL) and Lower Control Limit (LCL). Use XBAR Charts to track if the process is going out-of-control. This URL provides more insight into how to define UCL and LCL. Note: if the LCL drops below some practical value, then choose zero.

Dilbert Syndrome: "When we treat symptoms, we get a step closer to Dilbert Syndrome. Making a wild guess at what went wrong and act on it to adopt a new unproven process."

PDCA Cycle: Plan-Do-Check-Act is also known as Shewhart Cycle. Instead of following a Dilbert Cycle, where one only react to changes, PDCA shows a more sophisticated way to welcome change. Look at the problem in detail and check what's wrong (Plan). Then try improvement in small steps (Do). Gather data and analyze results (Check). In the end, make a decision (Act) and repeat the cycle for next problem. It's common sense, isn't it? But we still find managers suffering from Dilbert Syndrome almost everyday!

Summary

• We can't test quality into our software, we design in it
• A chain is as strong as its weakest link. We must find and fix the bottlenecks first.
• Processes needs to be continually improved
• Don't treat symptoms. Don't follow Dilbert Model
• The People who do the work generally knows best how to improve it
• Use quantitative methods to support decisions, whenever possible
• Plan a flight, and fly the plan

*Links and References*

• Operations Management, OMIS 357 Class - Leavey School of Business
• Software Practicum, CS 2335 Class - Georgia Institute of Technology
• XBAR and R-Charts

Data Strategy (DS)

A lot of test automation engineers get confused when it comes to Data Driven Testing (DDT)

• How tests should be structured?
• What seed data to use?
• When and How to load Data?
• When to flush data?
• How to bind test scripts with test data?

These are some of the very basic questions that must be answered in conjuction with the tooling and test automation strategies. Note: DDT is more than just passing arguments to a test library function.


When and How to load Data?
Well, it is imporant to understand data scope in conjunction with the test case and the automation strategy to address the "WHEN" question. There are several ways to address this.

• DS1: Consider most of your data as "seed data" and load it upfront before actually kicking off your test automation. In this case, data cleanup is done after test scripts exit. This is the most preferrred strategy for Load & Performance testing. Data can be loaded directly into the database using SQL scripts or by running data suite. Data suite executes a set of test cases that takes care of loading and polulating the seed data.
  

• DS2: Load data for every test suite and cleanup at the end of the suite. Make sure environment is clean for the next suite to run.

• DS3: Load data for every test case. Every test is independent and takes care of its own data. This strategy is very cumbersome as the data is not leveraged among test cases.

• DS4: Combo Strategy. Load high level data (which is going to be used most often) upfront and leave rest of the data to every test case.
  

In all of the above data strategies, test cases can use factory classes to get pre-populated data objects. These factory classes can encapsulate the whole data strategy, including creation and deletion of data.

Note: Remember, in DDT, you only need to worry about the end-user-data. Auto generated information, like session-ids, does not need to be part of the DS, unless you plan to skip some steps. Your automation framework must provide mechanisms to pass auto-generated data across test cases and test suites.

Final Thought:
Data Strategies can get really complex, and sometimes even more complex than your actual test cases. Try not to drift away too deep into fancy data strategies and loose sight of the real thing!