Strategic Test Optimization: Measurable Quality in Large Healthcare Pharma Software via Combinatorial Test Design for improved Test Coverage

In our case study, we detail how we significantly improved test coverage for a prominent healthcare pharma software through the strategic implementation of combinatorial test design. Facing the complex landscape of healthcare systems, we recognized the need for a comprehensive testing strategy that goes beyond conventional methods. By leveraging combinatorial techniques, we efficiently generated test scenarios that covered a multitude of input combinations, ensuring robust and thorough testing. The results were remarkable— Our scientific way of test case optimization, reduction & selection helped our client to get visibility into increased test coverage which improved defect detection, and ultimately, a higher quality healthcare software system. Dive into our case study to discover the transformative impact of combinatorial test design on healthcare/pharma software testing.

Problem Statement: A large test suite with almost 1.5 : 1.0 (dev:qa) ratio but still firefighting on production due to issues, without any visibility on test coverage.


                     * A Large highly configurable Software - with 8000 testcases
                     * Has a lot of regulatory requirements
                     * Enterprise customers
                     * Globally distributed team of dev & qa teams
                     * Technology Transformation phase


                     * Delivery - Over Cloud, but hosted for individually for customers [not True SaaS]
                         - Customers on different versions
                     * High number of config parameters
                     * 10+ modules
                         - Complexity - Rule engine, heavy db writes, dynamic UI
                         - Structure - Monolith, tightly coupled with UI
                         - Artifacts - heavy load of documents of all types with various dynamic topics
                         - Tribal knowledge with long time folks

Outcome: Transparent test coverage, quantify quality for decision making, reduced defects, improved CSAT & overall trained team for continuity

STAGE - 1 [Day Zero]

We don't often have a scenario, where management proactively prioritises quality and invest time and resources in its enhancement, not to mention engaging directly in quality-related matters. Typically, we find ourselves in the position of having to persuade stakeholders that the effort is worthwhile, especially when it comes to boosting velocity. However, in this particular instance, the management reached out to address the quality challenge. This was a significant advantage as it spared us the need to invest time and energy in convincing them of the vital connection between quality and improved velocity 

STAGE - 2 [3 days]

We embarked on a thorough exploration of the business, its organizational structure, functional intricacies, application layers, technology stack, and the nature of quality-related challenges. We delved into the pain points and issues affecting the system, examined the delivery methods in place, and differentiated between premium and all customers. Additionally, we conducted an in-depth assessment of the existing test cases. This comprehensive groundwork laid the foundation for our strategic approach. This took us about 3 days and a couple of important people who have the in-depth understanding.

STAGE - 3 [2 Days]

In response to our initial findings, we crafted a tailored orientation and training program designed to equip the team with the necessary knowledge and skills. This program aimed to elucidate the "Why" behind the entire endeavor, outline the specific steps to be taken, and establish clear methodologies for measuring the outcomes achieved. This phase was pivotal in ensuring that our team was fully aligned and empowered to drive the project forward. - 4 hour training for a batch of X people

STAGE - 4 [3 Weeks - 16 hours a week]

With all the inputs and the application flow, list out the most critical module as well as the most troublesome one. In this case it happened to be the same. there were 1200 tests in the sanity and we set out to optimise it. Ofcourse this was like on the job training as well with their application!

1. Uncover test parameters

  • Collected all known parameters and paths of the module in question

  • Sorted them into classes, options and criticality

  • Post this we should be able to generate the cartesian product combinations

2. Reduction

  • Reduced the valid parameters and paths- Using the functional rules

- constraints 

- Interaction Levels

3. Selection

  • Using the tribal knowledge, heuristics 

- Selected the Critical Path ones embedded with all data variations

- Required ones, must haves, trouble causing ones etc

4. Generation of Test Script

  • Script template to adjust and generate test cases that makes sense and inline with steps 1,2,3

5. Optimization

  • Remove redundancies

  • Include all validation requirements

  • Create required test data

6. Segregate to sanity & the rest

7. Execute for correctness, robustness and repeat the iteration to fine-tune.


Measure the outcomes

- We started with ~1200 sanity for a module and after stage 1 to 4, we were able to deduce 8 tests as sanity with 100% coverage for all known parameters.

- Execution resulted in some validation gaps identified by the long timers, they were bridged to solidify the suite. Even after all the additions it was around 170 tests for full suite for that module!

- Timetaken to execute was drastically reduced, less test uncovering more defects, visibility into what failed, less tests to automate!

A remarkable transformation started unfolding in front of us and in 4 weeks the team had the productive end result (suite) on hand! The time required for execution saw a dramatic reduction, while fewer tests managed to unearth more defects. The enhanced visibility into failure points allowed for more targeted problem-solving, and the need for test automation was significantly reduced. This set the stage for a substantial improvement in efficiency and effectiveness. The team was visibly happy that they were able to learn the science way of measuring quality!

The synergy of efficiency and comprehensiveness resulted in a more robust testing process, ensuring the highest quality for critical healthcare Pharma systems. By harnessing combinatorial techniques, we efficiently generated optimised test scenarios by encompassing a myriad of input combinations, guaranteeing thorough and robust testing. The results were truly remarkable, including heightened test coverage, enhanced defect detection, and, most importantly, an overall increase in the quality of healthcare software.

Explore the transformative impact of our combinatorial test design strategy by scheduling a call with us!

window.lintrk('track', { conversion_id: 9887532 }); window.lintrk('track', { conversion_id: 9887604 });