Seamless Integration: Testing 2.0!!

End-to-end seamless integration of tools is important in every domain, especially ALM. iTKO seem to have understood the concept and is delivering a Test Automation Platform (LISA) which not only allows automation of customer scenarios (spanning across various technologies), but also provides integration APIs to integrate with rest of the ALM tools.

For example:

A test can register a user on a web-portal, download a software, installs the bits on a local or remote system, runs commands to finish & verify installation, calls a web-service to confirm product registration and in the end queries the database to make sure all tables are correctly populated - all from within the context of the same test case!

There are even APIs available to expand the core functionality of LISA - like filters, assertions, companions, etc.

There is more... Once the test is automated, it can be automatically kicked off as part of cruise control, or other Ant integration, post result into a central database or TestDirector or homegrown test management system, automatically file a bug or integrate with requirement management system. Use of pure java, xml, open architecture and public APIs makes the seamless integration possible across the entire ALM domian.

The tool allows all data strategies I discussed in my August blog. The data driven testing is the key to leverage test scripts, lower maintenance and improve ROI.

Checkout Test Automation Tools blog. It nicely captures the automation framework requirements and the battle of build vs. buy. Remember, understanding the requirements and ability to clearly articulate the question is the first step in finding the answer!

Like Paris Hilton is Marketing 2.0, LISA seems to be Testing 2.0 for Web 2.0 :-)

Automation: Beyond Testing

Automation is the key to productivity in any functional group in any organization. But for majority of people:

• "Automation" = "Test Automation"
  

The general perception is that we should spend all our resources in automating test cases to cut-down test-cycle times! Well, that's not entirely true. Test cases are a small piece of the overall puzzle. Automating test cases is important, but it alone does not gaurantee reduced cycle time and faster release trains.

We must automate not only the overall testing process (which is far more than just automating test cases), but also other processes that takes up considerable amount of resources and are prone to errors. The biggest problem with automating processes is getting buy-in from the top management and all the cross-functional teams they impact. Moreover, some people see automation as a threat to their job and hence never fully support the idea. However, the fact remains that to boost productivity, one must maximize the use of labor and capital - which can be best achieved via Automation!!

Automation has following benefits:

• An automated process tends to live longer and ROI follow an exponencial curve.
• Automated processes are a lot easy to follow.
• Automation provide repeatability and consistency and boosts the quality of the process itself.
  
• Good quality processes results in good quality products (TQM principle)
• Employees get more frustrated by the process relaeted issus. A lot of product related issues are there because of broken processes.
  
• Process Automation makes the upper managaemet's life easy. It generates data and trends which can significantly impact decision making.
  
• Automated process reduces the ramp-up time for new comers, as processes are a lot easier to understand, once automated.
  
• Process Automation is an investment in the infrastructure.
  

So, how imporant is it for a team to invest in automation beyond testing? The answer is "It Depends." It depends on the complexity of the team, its structure, size and the amount of decentralization. Processes becomes overhead when your teams are small and localized, like in a startup. Most startups hate processes. Whereas the processes become really important when team grows in size, members work from home and are distributed globally in different timezones. Processes also makes sense when team members are of mixed experience as a result of attrition and growth, and there is a growing need for cross-functional collaboration.

In case you are wondering which process to automate, here is a small list of ALM sub-processes, which are a good pick for automation:

• Requirements Management
• Version Control, Build and Release Management
  
• Change Management, Request to Integrate (RTI) Process
• Patch Management
• Request for Change (RFC) Process
• QA Acceptance, Sanity testing
• Overall Testing Process (includes test management and test automation)
• Continuous Integration
  
• Deployment
• Defect Management
  
• Customer Escalation Management, Issue Tracking
• Program Management
• Monitoring (servers, errors, thresholds, CPU, memory, I/O, response times, availability)
  
• End-to-end real-time reporting
• Dependency Management (must for SOA applications)
  

Process automation doesn't mean going out of limits and defining a process and then automating it. Start by identidying iterative activites that happen very frequently and are annoying. For large and distributed teams, where it's hard to get a handle on productivity, processes are a breeze. And if there are processes, they need to be automated to bring standardization, acceptance, speed, and higher overall productivity.

Final Thought:
If there is a process, it better be automated!

Automation: TQM Perspective

Quality has two perspectives - Validation & Verification.

In validation, we are checking Are we building the right thing? Basically, we are checking for conformance to customer requirements and quality attributes.

In verification, we are checking Are we building the thing right? We are more concerned about the process conformance and quality activities. Verification is more on the TQM side.

Automation provides a lot of benefits like: Repeatability, Predicability, Higher productivity, Lower costs, Shorter cycle times, Faster turn arounds, Higher confidence, and Insurance against human blunders. So, in reality Automaiton is nothing but means to achieve higher quality of processes and hence higher quality of products.

All processes that are part of Application Life-Cycle Management (ALM) must be automated to assure repeatability and keep processes in-control.

Starting form the requirements to bug-filing - everything should be automated.

• Central Requirements Management
  
• Automated unit, integration and functional testing
• Central Code Repository, Version Control
  
• Automated Request to Integrate Process
• Automated builds (Nightly, weekly)
  
• Automated Deployment
  
• Automated installation of TestWare
• Automated testbed setup and configuration
• Automated Test execution (Functional & Non-functional both)
  
• Automated Reporting
• Automated Bug-filing!!
• CRM Solutions - Customer escalation management
• Project Program Management
  
• End-to-End Integration and Data flow
  

Well, that is the ideal scenario. The closer we can get, the more successfull we are with our automation initiative. It you are a TQM fan, you'd agree that we need to improve our processes and build more quality into them which inturn will improve our product quality.

Automation has following benefits, which goes hand-in-hand with TQM benefits:

• Provides a big picture of the overall process
• Achieve higher adoption rates (CAP)
• Easy to agree on standards
• Data Archiving leads to Data Mining Possibilities
• Historical data archives helps in decision making
• Provides Process Traceability
• Reports can reflect exact process state
• Treand reports can help in gaining process maturity
• Easy to monitor overall process
• Email alerts in case the process goes Out-of-control
• Defies Dilbert Model
• Brings Focus and Accountability
• Boosts Productivity
• Reduces Costs
• Saves Time
• Higer Process Quality leads to Higher Product Quality
• Easy to articulate SUCCESS

As I said before, it may not be possible to automate 100% of any process, but the closer we get, the better it is. Automation is like investing in infrastructure. The more solid our infrastructure is, more high we can rise. "We can't build an Empire State Building on a 6-inch foundation." If we want to achive high quality in our products, then we really need to have a very strong foundation, i.e very stable and high quality processes; and one way to achieve that is via Automation.

Final Thought:
Automation does not start and finish with Test Automation.

Data Strategy (DS)

A lot of test automation engineers get confused when it comes to Data Driven Testing (DDT)

• How tests should be structured?
• What seed data to use?
• When and How to load Data?
• When to flush data?
• How to bind test scripts with test data?

These are some of the very basic questions that must be answered in conjuction with the tooling and test automation strategies. Note: DDT is more than just passing arguments to a test library function.


When and How to load Data?
Well, it is imporant to understand data scope in conjunction with the test case and the automation strategy to address the "WHEN" question. There are several ways to address this.

• DS1: Consider most of your data as "seed data" and load it upfront before actually kicking off your test automation. In this case, data cleanup is done after test scripts exit. This is the most preferrred strategy for Load & Performance testing. Data can be loaded directly into the database using SQL scripts or by running data suite. Data suite executes a set of test cases that takes care of loading and polulating the seed data.
  

• DS2: Load data for every test suite and cleanup at the end of the suite. Make sure environment is clean for the next suite to run.

• DS3: Load data for every test case. Every test is independent and takes care of its own data. This strategy is very cumbersome as the data is not leveraged among test cases.

• DS4: Combo Strategy. Load high level data (which is going to be used most often) upfront and leave rest of the data to every test case.
  

In all of the above data strategies, test cases can use factory classes to get pre-populated data objects. These factory classes can encapsulate the whole data strategy, including creation and deletion of data.

Note: Remember, in DDT, you only need to worry about the end-user-data. Auto generated information, like session-ids, does not need to be part of the DS, unless you plan to skip some steps. Your automation framework must provide mechanisms to pass auto-generated data across test cases and test suites.

Final Thought:
Data Strategies can get really complex, and sometimes even more complex than your actual test cases. Try not to drift away too deep into fancy data strategies and loose sight of the real thing!