Quality Trail

As a “proud” owner of a Jeep Wrangler, I’ve caught this bug many times, constantly customizing and upgrading my ride for the next off-road adventure. There’s a saying in the Jeep community — proudly plastered on window or bumper stickers and often shared with a wink — “real Jeeps are built, not bought.” While I agree entirely with this so called “Jeep Thing”, I’ve also learned the hard way that while some DIY upgrades might really shine off-road, they can also leave you stranded miles from any sort of civilization if you don’t take the time to ensure their quality. Trust me, roaming through the woods in search of a cell signal to make that embarrassing rescue call isn’t the adventure I had in mind! And, believe it or not, even those shiny factory features can occasionally leave you in the same position.

Navigating through the intricacies of software onboarding, I have encountered varied landscapes shaped by the critical decision: to buy or to build software. In some organizations, I observed a convoluted dance between external vendors, dedicated internal or external QA teams, and the uncharted territories of software architecture. Consulting has given me a front-row seat to the complex problems that businesses face when trying to implement third-party technologies. The lack of trained internal QA teams to test software from external vendors was a common problem. This necessity emerged when vendors did not provide adequate architectural documentation or test scripts, leading to reliance on manual or UI-automated testing.

This method, critical for confirming product integrity and compatibility, introduced a significant disconnect between development and user-end testing. The inability to shift left or perform agile in-sprint testing due to a lack of understanding of the product’s internal architecture and Dev/QA seperation also added unnecessary time to the development cycle.

In some cases, I have seen companies use one vendor to acquire software and another to conduct quality assurance and sistem integration testing. Having to deal with two separate suppliers slowed things down considerably. Limitations imposed by agreements and rules made it extremely difficult to optimize procedures and shift to the left.

These experiences underscore the intricate dynamics and hidden resource demands of the buy versus build decision. They highlight the imperative for a nuanced approach, considering the potential for unforeseen costs and complications, particularly from a QA or QE viewpoint.

Buying Software:

Pros:

1. Vendor Experience: In a perfect world, suppliers would have years of expertise and plenty of QA-specific resources, guaranteeing consistent quality. However, a careful selection of software vendors is essential to bridge gaps in domain-specific knowledge and requirement gathering, especially for custom-made solutions. Remember that your vendor is a software development company and may not have enough domain knowledge and understanding of your line of business such as banking, insurance, agriculture, telecommunication or any other industry. Remember your Jeep is a Jeep, a specific off-roading vehicle, you should not take it to a random mechanic who’s not familiar with your special domain usecases and off-roading concerns!
2. Standardized Testing Procedures: Software providers use industry-wide standards for product quality assurance and release frequent updates. However some vendors may not feel required to follow these practices unless customers specifically request and monitor their implementation in the contract.
3. Compliance Assurance: HIPAA, GDPR, PCI DSS, ISO 27001, Sarbanes-Oxley Act, FISMA, AODA, and WCAG are just some of the laws and regulations that must be followed depending on your state or province, and the risk of noncompliance can be reduced if the software you buy is in line with them. Failure to select a software vendor that follows all applicable compliance rules such as accessibility, diversity, and security, however, could have dire consequences and should not be taken lightly.

Cons:

1. Limited Insight and Control: Organizations often have restricted control and visibility over the vendor’s quality assurance processes. Reflecting on personal experience, a scenario comes to mind where a customer had to build an internal testing department. Without access to documents laying out the software’s internal architecture, the team faced test overkill on UI, missing out on the effectiveness and quality improvements of API testing.
2. Integration Testing Challenges: Externally developed software may pose integration challenges, necessitating additional in-house testing for compatibility with existing systems. It is vital that the vendor become responsible for the implementation and initial testing in the customer’s test lab and production environments. If this is ignored, serious integration problems may arise.
3. Vendor and Contract Dependency for Fixes: Having to rely on third-party providers for fixes can be a double-edged sword, resulting in frequent rollbacks and more work. Compare it to the possibility of optimizing time and quality by adopting a Test-Driven Development (TDD) methodology and using incremental releases to production within an in-house development strategy.

Building Software

Pros:

1. Customization and Control: Building software in-house offers unparalleled customization and control over all processes, from development to QA, and closely aligns the finished product with organizational requirements and standards.
2. Direct Communication: The ability to move testing left, perform in-sprint testing, and automate testing is increased by direct communication between developers and QA teams, which enables a more responsive and adaptive development cycle.
3. Adaptation to Changes: Internal development allows for continual software quality improvement and evolution since it is more flexible to changes in requirements or technology.

Cons:

1. Resource Intensive: In-house development is resource-intensive, requiring skilled personnel, time, and investment, which may be challenging for organizations to sustain.
2. Learning Curve: The software’s quality and timeliness may be impacted by the steep learning curve associated with new development and QA/QE processes.
3. Risk of Misalignment: There is a risk of misalignment between the software development team and industry standards or best practices, especially if the in-house team lacks experience in specific domains or technologies. Based on my own experience, as an example, I can say that banks are often conservative businesses that follow waterfall processes in a variety of areas, which frequently conflict with agile industry best practices for software teams. I have seen internal software teams at banks working hard to incorporate agile approaches. Despite their best efforts, the scrum-based, so-called agile teams encountered delays and obstacles since other departments’ organizational processes and security approvals could not be synchronized with the agile nature of the software teams. This gap was especially noticeable within the quality assurance teams, where the few hundred years old conservative culture of the banks made it difficult to embrace automation and have trust and confidence in robotic testing.

Similar to off-roading in a Jeep, designing or purchasing software is a journey filled with humps, twists, and unexpected turns. Whether you’re modifying your trusty Jeep for the umpteenth time or navigating the software terrain of your organization, remember: a well-tread path might be tempting, but sometimes the road less traveled (or the software built in-house) offers the thrill of discovery. However, without the proper QA in place, there is always the chance of getting stuck down in the software “mud”. So, fellow travellers, get ready! Make sure your “software Jeep” is equipped with robust quality assurance before you set off on your next tech adventure. If all else fails, keep in mind that every mistake is only a stop along the way to achievement, take some rest and hit the bumpy road again, now you have a fascinating tale for the campfire!