E-Commerce platforms consist of complex integrations of applications such as product information systems, payment integrations, warehouse apps, supply chain, logistics, fulfilment and vendor management systems etc. For the e-commerce platform to function efficiently, all the wheels must work in sync to generate memorable customer experiences of speed and quality. A robust quality engineering infrastructure must be in place, supported by automated processes and a dedicated talent pool to ensure sustainable performance.
Qapitol QA partnered with a large and leading e-commerce company for their platform testing and quality assurance requirements. As part of the engagement, Qapitol QA teams work closely with the client, including reporting into their quality leadership. Our team has been given responsibility for maintaining the quality and performance of a set of systems and sub-systems while working with the extended team that manages the overall platform. Warehouse management was one such critical function that was being looked after by the team, especially for optimization and stability requirements.
Sustainable quality engineering
As per routine quality engineering activities, the team conducts analysis and targeted testing to maximize code coverage, uncover blockages and maintain stability. One such instance is described here where the team analyses the challenges and optimises the time taken for executing tests.
Engagement context
Qapitol QA was tasked to improve the test coverage of unit test cases for the Warehouse Management System applications, while ensuring the stability of the existing services.
Problem identified
It was analysed that executing Unit tests consumed 4.5 Hrs to run for around 10000 test cases. Test coverage was still found insufficient.
Expected outcome
It was expected that after the fixes, the execution time for all the test cases would be brought down to less than 45 minutes. The gaps in test coverage were to be identified and plugged.
Application Technology Stack
Ruby on Rails (https://rubyonrails.org/)
Initial Analysis
Analysed current test coverage & identified Gaps
The team analysed the existing code coverage (58%) and identified the features with gaps in the test coverage
Analysed the test suite execution time – What’s taking time?
The source code was profiled and the time to execute the entire test suite was broken down for analysis.
- To profile the test server, the team Integrated a plugin to identify all the files that are getting called when the server starts and time consumed for the same.
- The team executed the specs with profiling and observed that DB deletion and truncation was being called after each spec. It was observed that it took more than 10 seconds to process each call. There are 1170 specs, exposing the potential of a considerable impact on time.
- The team used specialized tools while profiling to figure out the time taken by each method.
Initial Findings
- It was found that setting up the tests was consuming around one hour and test execution was taking 3.5 hours.
- The setup included extracting the latest Codebase from the repository, setting up the database and seed data plus loading transaction data.
- The test cases were executing external API calls and were querying the DB instead of calling the mocks.
- While starting the test server, it was observed that all libraries were getting loaded, including the unused ones.
Deep Dive Analysis
Analysed test coverage at feature level to identify gaps
- The team analyzed the coverage of unit tests at feature level, inspected the production Issues and studied defect leakages.
Further analysis on execution time
The team continued extensive analysis to unroot further pain points by performing the following activities:
- The DB was removed to identify all the cases that were performing DB calls and all the failed test cases were collated and the DB calls bucketed.
- DB calls were mocked one by one and the test case failures were fixed
- The test cases were segregated based on their execution time
- Cases that were consuming more than 60 seconds to execute were identified
- Cases that were consuming more than 10 seconds to execute were identified
- Cases that were consuming more than 1 second to execute were identified
More findings
- Few libraries were outdated and were in need of updates.
- Each time a migration script had to be executed to create and update the DB.
- Seed data was pre-populated whereas transactional data was loaded during the execution of respective cases
- A defect in the component was causing an increase in execution time in each test (Close to 1.5 hrs approximately)
Solution Provided
Improving test coverage
- The unit test cases were added as per the identified gaps in test coverage for the respective services, features and corresponding functionalities
- The test cases were added to identify product defect leakage accordingly
Rectifying the issues
- The component defect was fixed and the upgrade was merged into the development branch
- All the gems that were not getting used but were part of the script, were removed.
- Use of gem classes were removed in test cases to reduce DB dependencies in unit test execution and mocking was done on test cases where DB check was not required
- Developers were suggested to remove DB connection establishment gems from custom gems
Optimising the setup
- During setup, only delta is deployed
- Reduced the time taken for executing migration scripts by acquiring the latest DB data.
- Created pipeline with skip migration to reduce time while running pipeline on the same branch as the previous build.
Libraries updated
- The libraries were updated so as to reduce the execution time.
Impact analysis
- Performed impact analysis of upgraded gem version by running all pipelines & deploying changes on 1-1 machine of each VIP in production. No issues were observed
Final Outcome
- Test Coverage – Achieved 76% code coverage and improved testing efficiencies
- Execution Time – After this change, the unit spec pipeline which took 4.5 hours earlier for executing 10000 test cases, now took only around 50 minutes.
Write to [email protected] for quality engineering resources, automation strategies and testing talent.