Meta-Data data driven testing!

If you are involved in test automation, you'll know the importance and challenges of managing data. In my experience, developers are the worst testers mainly because they fail to understand the concept of data driven testing. Even good developers, who care to test, hard code test data in their unit test cases. To me, coming up with a good Data Strategy is more than half of test automation challenge.

As part of my profession, I teach and train students in SOA lifecycle quality. Over period of time, I have realized that my basic theory of iteration (refer to my Data Driven Testing (DDT) blog) doesn't really explain all different types of data that one needs to take into consideration in order to define a good test strategy. It only explains, at a high level, how one can extract randomness and variability from a test and move it into external data asset.

Modular testing is the key to re-usability and lower maintenance. Apart from creating test libraries and using building blocks, it is also important to understand how to organize different types of data outside of a test script.

Following diagram illustrates how different types of data can be organized around a test procedure/logic. Different types of data must not be cobbled together, because it almost always leads to increased maintenance costs and chaos.

• Removing Meta-data from the test logic not only makes the logic more reusable, but also lowers its maintenance. Change in meta-data can be easily handled by updating appropriate properties files. Example: SQL scripts, tag names, component names. Meta-Data has 1-1 relationship with the test case.
• Input Test Data is core to the business functionality/scenario. Example: username and password to login into a website. A single test case can have multiple copies of the same data, generally referred to as rows inside a dataset.
  
• Output Test Data is used for validation purposes. Validation of responses can be done using regular expressions or against the actual data. In case, generic validations (like regular expression checking size of the confirmation id) are not used, output data is mapped 1-1 with the input data.
  
• Server Environment Data refers to the endpoint data which varies between environments. Ability to change this data (at the execution time) allows the same set of test cases to be run against multiple environments.
  
• Client Environment Data refers to local environment specific to the test framework and project.
  
• Staging data allows the same test asset to be staged in different ways.
  

Automation: Beyond Testing

Automation is the key to productivity in any functional group in any organization. But for majority of people:

• "Automation" = "Test Automation"
  

The general perception is that we should spend all our resources in automating test cases to cut-down test-cycle times! Well, that's not entirely true. Test cases are a small piece of the overall puzzle. Automating test cases is important, but it alone does not gaurantee reduced cycle time and faster release trains.

We must automate not only the overall testing process (which is far more than just automating test cases), but also other processes that takes up considerable amount of resources and are prone to errors. The biggest problem with automating processes is getting buy-in from the top management and all the cross-functional teams they impact. Moreover, some people see automation as a threat to their job and hence never fully support the idea. However, the fact remains that to boost productivity, one must maximize the use of labor and capital - which can be best achieved via Automation!!

Automation has following benefits:

• An automated process tends to live longer and ROI follow an exponencial curve.
• Automated processes are a lot easy to follow.
• Automation provide repeatability and consistency and boosts the quality of the process itself.
  
• Good quality processes results in good quality products (TQM principle)
• Employees get more frustrated by the process relaeted issus. A lot of product related issues are there because of broken processes.
  
• Process Automation makes the upper managaemet's life easy. It generates data and trends which can significantly impact decision making.
  
• Automated process reduces the ramp-up time for new comers, as processes are a lot easier to understand, once automated.
  
• Process Automation is an investment in the infrastructure.
  

So, how imporant is it for a team to invest in automation beyond testing? The answer is "It Depends." It depends on the complexity of the team, its structure, size and the amount of decentralization. Processes becomes overhead when your teams are small and localized, like in a startup. Most startups hate processes. Whereas the processes become really important when team grows in size, members work from home and are distributed globally in different timezones. Processes also makes sense when team members are of mixed experience as a result of attrition and growth, and there is a growing need for cross-functional collaboration.

In case you are wondering which process to automate, here is a small list of ALM sub-processes, which are a good pick for automation:

• Requirements Management
• Version Control, Build and Release Management
  
• Change Management, Request to Integrate (RTI) Process
• Patch Management
• Request for Change (RFC) Process
• QA Acceptance, Sanity testing
• Overall Testing Process (includes test management and test automation)
• Continuous Integration
  
• Deployment
• Defect Management
  
• Customer Escalation Management, Issue Tracking
• Program Management
• Monitoring (servers, errors, thresholds, CPU, memory, I/O, response times, availability)
  
• End-to-end real-time reporting
• Dependency Management (must for SOA applications)
  

Process automation doesn't mean going out of limits and defining a process and then automating it. Start by identidying iterative activites that happen very frequently and are annoying. For large and distributed teams, where it's hard to get a handle on productivity, processes are a breeze. And if there are processes, they need to be automated to bring standardization, acceptance, speed, and higher overall productivity.

Final Thought:
If there is a process, it better be automated!