Quality Trail

Honoured to have shared a moment with Premier Doug Ford at a Nowruz gala last night. Briefly touched on the pivotal role of quality assurance and software security in government digital platforms, amidst Ontario’s efforts to safeguard against data-related challenges we’ve recently faced, especially in the increasing car thefts in the GTA. His awareness and response to these issues were truly commendable.

As a “proud” owner of a Jeep Wrangler, I’ve caught this bug many times, constantly customizing and upgrading my ride for the next off-road adventure. There’s a saying in the Jeep community — proudly plastered on window or bumper stickers and often shared with a wink — “real Jeeps are built, not bought.” While I agree entirely with this so called “Jeep Thing”, I’ve also learned the hard way that while some DIY upgrades might really shine off-road, they can also leave you stranded miles from any sort of civilization if you don’t take the time to ensure their quality. Trust me, roaming through the woods in search of a cell signal to make that embarrassing rescue call isn’t the adventure I had in mind! And, believe it or not, even those shiny factory features can occasionally leave you in the same position.

Navigating through the intricacies of software onboarding, I have encountered varied landscapes shaped by the critical decision: to buy or to build software. In some organizations, I observed a convoluted dance between external vendors, dedicated internal or external QA teams, and the uncharted territories of software architecture. Consulting has given me a front-row seat to the complex problems that businesses face when trying to implement third-party technologies. The lack of trained internal QA teams to test software from external vendors was a common problem. This necessity emerged when vendors did not provide adequate architectural documentation or test scripts, leading to reliance on manual or UI-automated testing.

This method, critical for confirming product integrity and compatibility, introduced a significant disconnect between development and user-end testing. The inability to shift left or perform agile in-sprint testing due to a lack of understanding of the product’s internal architecture and Dev/QA seperation also added unnecessary time to the development cycle.

In some cases, I have seen companies use one vendor to acquire software and another to conduct quality assurance and sistem integration testing. Having to deal with two separate suppliers slowed things down considerably. Limitations imposed by agreements and rules made it extremely difficult to optimize procedures and shift to the left.

These experiences underscore the intricate dynamics and hidden resource demands of the buy versus build decision. They highlight the imperative for a nuanced approach, considering the potential for unforeseen costs and complications, particularly from a QA or QE viewpoint.

Buying Software:

Pros:

1. Vendor Experience: In a perfect world, suppliers would have years of expertise and plenty of QA-specific resources, guaranteeing consistent quality. However, a careful selection of software vendors is essential to bridge gaps in domain-specific knowledge and requirement gathering, especially for custom-made solutions. Remember that your vendor is a software development company and may not have enough domain knowledge and understanding of your line of business such as banking, insurance, agriculture, telecommunication or any other industry. Remember your Jeep is a Jeep, a specific off-roading vehicle, you should not take it to a random mechanic who’s not familiar with your special domain usecases and off-roading concerns!
2. Standardized Testing Procedures: Software providers use industry-wide standards for product quality assurance and release frequent updates. However some vendors may not feel required to follow these practices unless customers specifically request and monitor their implementation in the contract.
3. Compliance Assurance: HIPAA, GDPR, PCI DSS, ISO 27001, Sarbanes-Oxley Act, FISMA, AODA, and WCAG are just some of the laws and regulations that must be followed depending on your state or province, and the risk of noncompliance can be reduced if the software you buy is in line with them. Failure to select a software vendor that follows all applicable compliance rules such as accessibility, diversity, and security, however, could have dire consequences and should not be taken lightly.

Cons:

1. Limited Insight and Control: Organizations often have restricted control and visibility over the vendor’s quality assurance processes. Reflecting on personal experience, a scenario comes to mind where a customer had to build an internal testing department. Without access to documents laying out the software’s internal architecture, the team faced test overkill on UI, missing out on the effectiveness and quality improvements of API testing.
2. Integration Testing Challenges: Externally developed software may pose integration challenges, necessitating additional in-house testing for compatibility with existing systems. It is vital that the vendor become responsible for the implementation and initial testing in the customer’s test lab and production environments. If this is ignored, serious integration problems may arise.
3. Vendor and Contract Dependency for Fixes: Having to rely on third-party providers for fixes can be a double-edged sword, resulting in frequent rollbacks and more work. Compare it to the possibility of optimizing time and quality by adopting a Test-Driven Development (TDD) methodology and using incremental releases to production within an in-house development strategy.

Building Software

Pros:

1. Customization and Control: Building software in-house offers unparalleled customization and control over all processes, from development to QA, and closely aligns the finished product with organizational requirements and standards.
2. Direct Communication: The ability to move testing left, perform in-sprint testing, and automate testing is increased by direct communication between developers and QA teams, which enables a more responsive and adaptive development cycle.
3. Adaptation to Changes: Internal development allows for continual software quality improvement and evolution since it is more flexible to changes in requirements or technology.

Cons:

1. Resource Intensive: In-house development is resource-intensive, requiring skilled personnel, time, and investment, which may be challenging for organizations to sustain.
2. Learning Curve: The software’s quality and timeliness may be impacted by the steep learning curve associated with new development and QA/QE processes.
3. Risk of Misalignment: There is a risk of misalignment between the software development team and industry standards or best practices, especially if the in-house team lacks experience in specific domains or technologies. Based on my own experience, as an example, I can say that banks are often conservative businesses that follow waterfall processes in a variety of areas, which frequently conflict with agile industry best practices for software teams. I have seen internal software teams at banks working hard to incorporate agile approaches. Despite their best efforts, the scrum-based, so-called agile teams encountered delays and obstacles since other departments’ organizational processes and security approvals could not be synchronized with the agile nature of the software teams. This gap was especially noticeable within the quality assurance teams, where the few hundred years old conservative culture of the banks made it difficult to embrace automation and have trust and confidence in robotic testing.

Similar to off-roading in a Jeep, designing or purchasing software is a journey filled with humps, twists, and unexpected turns. Whether you’re modifying your trusty Jeep for the umpteenth time or navigating the software terrain of your organization, remember: a well-tread path might be tempting, but sometimes the road less traveled (or the software built in-house) offers the thrill of discovery. However, without the proper QA in place, there is always the chance of getting stuck down in the software “mud”. So, fellow travellers, get ready! Make sure your “software Jeep” is equipped with robust quality assurance before you set off on your next tech adventure. If all else fails, keep in mind that every mistake is only a stop along the way to achievement, take some rest and hit the bumpy road again, now you have a fascinating tale for the campfire!

In today’s progressive landscape of software development, there’s a burgeoning enthusiasm around Quality Assurance (QA), Quality Engineering (QE), and test automation. Automating your current QA processes is unquestionably a commendable advancement and a crucial foundation. It streamlines and enhances what’s already in place, marking the start of a transformative journey towards quality. However, for organizations aiming to scale greater heights in software excellence, there’s an expansive horizon beyond just automation. Making the switch from automated QA to true QE involves more than just speeding up current procedures; it also represents an invitation to a more profound, all-encompassing evolution in how we view and integrate quality throughout the software lifecycle.

The Fundamental Differences

Traditional QA: This often acts as a checkpoint in the Software Development Life Cycle (SDLC). Once development is complete, the product is put through testing to find and fix any issues. While automation can improve the speed and accuracy of this stage, the underlying principle has not changed much from the conventionally reactive approach.

Quality engineering is a comprehensive approach that incorporates quality into every stage of the software development life cycle. In this setting, quality is “baked in” from the beginning of a project rather than being merely an added layer. It’s not about finding defects after they arise but embedding processes and practices that proactively prevent them, ensuring that every step, every decision, and every line of code inherently exudes quality.

Shifting Left: The Broad Spectrum

Proactivity is key to the “shift left” method. Shifting left emphasizes integrating quality checks considerably sooner in the SDLC process as opposed to waiting until the later phases to focus on quality. Quality is thus completely integrated into the development process, allowing for early detection and correction of possible faults.

By doing so, the development team adopts a mindset where quality is not just a checkpoint but a consistent guiding principle. Every design choice, every line of code, and every functionality introduced is viewed through the lens of quality from the very beginning.

Several “shift left” strategies have emerged as a result of applying this principle to different areas of software development. As one example, we can look to the field of performance engineering.

Performance Engineering vs. Performance Testing: Instead of relegating performance considerations to the tail-end, as done in traditional performance testing, performance engineering integrates them from the get-go. It guarantees that the program is designed from the ground up with peak performance in mind by proactively examining the potential consequences of architectural decisions and design patterns.

Catching Design Defects Early (aka Larvae Hunting!): In nature, those seemingly harmless little larvae might be overlooked. In a period of time though, they can develop into fully grown, destructive bugs. Similarly, in software development, small mistakes in the first phases may grow into major issues later on if nobody notices them. Why not become skilled larval hunters instead of bug exterminators? With a focus on early detection in QE, that’s exactly our role. We get involved in the preliminary stages of development to prevent the emergence of flying bugs. It’s a proactive strategy that improves productivity and injects excitement into the quality assurance procedure.

Design Testing and Prototype Testing: The areas of design testing and prototype testing stand out as crucial tools in the ‘larvae hunting’ toolkit. The primary goal of design testing is to verify that the preliminary design ideas satisfy the needs and expectations of the target audience. Verifying that everything is in order before writing a single line of code is essential. Prototype testing, on the other hand, entails making a basic but functional model of the software or the new feature. This prototype is then put through its paces, identifying potential usability, functionality, or integration issues. By subjecting early designs and prototypes to rigorous testing, we not only ensure that our foundation is robust but also that subsequent development phases are smoother, more predictable, and significantly more efficient. It’s similar to making sure the building’s blueprint is perfect before placing the first brick; this is an essential step in creating a masterpiece.

In Practice: The Nitty-Gritty of Early Stage Testing: When we talk about testing designs and prototypes, what does this look like in tangible terms? For instance, when designing a software interface, even before full functionality is developed, basic interfaces or empty classes and functions can be set up. As the exterior structure is being built, these crude structures offer a scaffold that can be tested right away. From a service-oriented perspective, consider an API-driven architecture: each web service can initially be introduced as a static mock, paving the way for early integration tests. As the development advances, these mocks then transform, gradually evolving into their fully-realized capabilities. This methodical procedure guarantees that the software is ready for testing at all times, allowing for continuous improvement and conformity to quality requirements.

In-Sprint Test Automation & The Advanced TDD Approach:

When we bring design and prototype testing into the mix, especially as discussed with our API example, a beautiful synchronization emerges between test automation and software development. The ideal scenario is for both processes to advance simultaneously throughout the same sprint. An our above example, as a static mock of an API evolves into a functional endpoint, its corresponding automated tests are being developed side-by-side, and evolving from prototype testing to actual component tests. This ensures that as a feature begins to arrive, its safety nets in the form of tests are already being woven (we’re talking about bugs right?! The net needs to be tight enough to keep the bugs and their larvae inside.

It’s not just about unit testing either. This approach allows for the simultaneous development of regression tests and even non-functional tests, painting a comprehensive picture of quality at every stage. This kind of coordination not only boosts productivity, but also drastically lowers the costs of production delays, helping us move closer to the utopian ideal of “bug-free software development” outlined in prior posts of this blog.

This concept isn’t new. It was developed in the aerospace industry, where the risk of human death makes a zero-bug tolerance absolutely essential. We are committed to producing software that is not only functional but also exceptional in its dependability and robustness, and we believe that adopting a similarly rigorous approach in software development is a key part of achieving this goal.

Lastly, In the dynamic world of software development, moving beyond mere automation to embrace Quality Engineering (QE) is imperative. It’s about integrating quality right from design to deployment, taking cues from precision-driven industries like aviation. As we work to create flawless software, it’s important to keep in mind that quality isn’t a destination, but rather the course we take to get there. Silkland Consulting Inc provides experienced guidance along this road for teams that are ready to take the plunge.

The Top 10 Signs That It’s Time to Switch from Quality Assurance to Quality Engineering

Knowing when to switch from quality assurance to quality engineering is crucial in the dynamic field of software development. The difference between the two can have a major impact on the quality of your product, the efficacy of your staff, and the happiness of your customers. Here are some indicators that your organization may be ready for the switch:

1. Reactive Testing Predominance: If your team’s primary modus operandi revolves around troubleshooting bugs after they manifest rather than forestalling them, it highlights the urgent requirement for QE’s proactive, preventive approach.
2. Delayed Testing Sessions: When testing is routinely pushed to the very end of the development cycle, it’s a sign that valuable time has been wasted. Fixing problems early on via testing can save time and money.
3. Longer Time to Production: Lengthy times between the development phase and the actual product deployment frequently indicate intrinsic inefficiencies in the present QA process. QE prioritizes frequent, small releases to cut down on lead times.

4. Regular Rollbacks Following Deployment: A red flag is raised if there is a history of rollbacks following deployment. It’s evidence that flaws are slipping through the cracks without being discovered during testing and highlights the need for thorough QE analysis.
5. Persistent Prod Issues: When end-users routinely encounter and report issues in a live environment, it is evidence that pre-release testing was not as thorough as it should have been.
6. Need for Early Fault Detection: The “shift left” attitude inherent to QE promotes identifying design flaws at their earliest stages. Here at Silkland Consulting Inc. we see it as dealing with “larvae” as opposed to fighting fully-formed bugs after production has begun. The time and money needed to fix defects at a later stage are reduced by this method. Kill the larvaes (design faults), you won’t need to deal with bugs!
7. Stark Silos Between Teams: The lack of communication and cooperation between the development and testing teams is a rallying cry for the integration-focused strategy favoured by QE.

8. Fragmented Tool Ecosystem: If your team struggles with tools that aren’t well-integrated with one another, you need to manually link various steps or your test architecture cannot technically be fully integrated, this is a good indicator that you could benefit from the unified tool integration that QE provides.
9. Significant Knowledge Deficits: When testers only focus on finding bugs and ignore more general concerns like design, architecture, and coding fundamentals, it’s clear they need to adopt the holistic, multi-faceted strategy that QE represents.
10. User Feedback Underscoring Usability Concerns: A steady stream of user grievances pertaining to usability, interface, or functionality highlights the necessity of the user-centric, holistic focus that’s at the heart of QE.

In conclusion, if you pay attention to these warnings, you can alter the future of your software’s quality. If your team has faced similar difficulties, now may be a good time to explore making a paradigm shift.

In addition, Silkland Consulting Inc is a trustable choice for reliable assistance during this change. Silkland Consulting Inc specializes in improving test automation and QE processes, thus we can help your team smoothly transition from QA to QE standards by developing individualized testing techniques. Put your faith in professionals that have mastered both worlds of development and testing and can smooth your way to top-notch software.

In the previous post of this blog, we discussed the role of AI in the future of test automation. This was an exploration of what the future will hold, but let’s take a step back and consider why test automation is so important and why current solutions and methodologies are not sufficient for this purpose.

The software industry is heading towards a more automated and intelligent future, with AI and machine learning becoming increasingly important components of every aspect of our modern lives. Governments and the private sector are investing heavily in research and development to create more software solutions for every aspect of life, many of which can have a direct impact on human health and wellbeing. Software is being used to improve banking, healthcare, education, and public safety, while autonomous vehicles, smart home devices, and virtual assistants are just a few of the many examples of how software is taking more responsibilities.

What happens if one of these goes wrong? The risks are becoming increasingly large. What if the software controlling a self-driving vehicle is updated to a new version which introduces a regression bug that prevents it from detecting taller pedestrians on the street? What if a public safety app fails to detect a potential threat because a negative test case had been forgotten, or if the software module controlling an artificial heart recently installed in the chest of one of our loved ones by a surgeon, fails while processing their blood pressure?

Software is taking on more and more responsibilities these days. This is no longer just a matter of a Notepad application crashing on your Windows XP, or an email being stuck in your outbox without being sent. These issues can now have serious implications for human lives. Escaped bugs are no longer acceptable, and software QA methodologies need to be updated and improved to ensure that products are 100% bug-free.

We will read more about how to head towards bug-free software development in the future posts of this blog.

Just five years ago when I was writing the abstract for my speech at Toronto Agile Testing and Automation Day conference on AI use cases in agile testing, I could hardly find any references about this topic. At the conference the topic was so fresh for all the audience to make them asking so many questions. I had to spend 20 minutes more on the stage.

In few months I started using an open source image comparison library in Java to implement my ideas in utilizing image processing to visually test user interfaces. I was hoping to replace fragile and flaky Selenium tests which are super costly in terms of maintenance. An idea which never worked out as expected before I heard about an AI testing tool provided by an offshore company.

Artificial Intelligence (AI) is expected to play a crucial role in the future of software test automation. With the increasing complexity of software systems, traditional manual testing methods, and even the automation built based on those methodologies are becoming more challenging and time-consuming. AI can help overcome these challenges by automating the testing process, reducing the time and effort required for manual testing, and increasing the efficiency and accuracy of the testing process. AI can be used to develop intelligent testing systems that can automatically detect software bugs and issues without human intervention. These systems can learn from past testing experiences, adapt to changing software environments, and improve their performance over time. This will not only save time and effort but also help identify bugs and issues that may have gone unnoticed using traditional testing methods. Moreover, AI can also be used to perform #PredictiveTesting, where the QA solution can predict potential software bugs and issues before they occur. This can help software developers to proactively fix these bugs and issues, resulting in higher quality software systems.

As an example, Applitools utilizes AI-powered image processing technology to compare images and detect differences in visual user interface testing. This technology helps to quickly identify even the most subtle changes in a user interface, and can be used to quickly detect bugs, regressions, and other issues.

Applitools does not seem to be using machine learning. In the current version, the user should review test results and validate changes categorizing them into bugs and expected updates, however the software saves this and uses exact same verifications on the next runs of the screenshots, it still does not learn from the user to improve the future experiences on new web pages or future features of the web application. Adding machine learning on top of image processing can take this to the next level.

AI has the potential to revolutionize the software testing process, making it faster, more efficient, and more effective, and it is likely to play an important role in shaping the future of software test automation.

My name is Armin Rajaee and I’m an experienced Software Development Engineer focused in Quality Assurance and Test Automation with 9 plus years of recent experience working in the banking, high tech, and several well-known Canadian and international organizations. Over the past 9 years, I have developed a strong background in software development and testing methodologies, enabling me to assess and improve the quality standards of any given software project.

My main areas of expertise include software quality assurance planning, test automation, test execution and result analysis, and quality management. I have experience working with a wide range of technologies, in different levels and stages of testing, and more. I’m also familiar with Agile and Waterfall methodologies and have performed in both successfully in past projects.

In this blog I will mostly write about the rolea and responsibilities of a QA engineer (SDET) in a software team. As an SDET it is your role to be the ambassador of quality for the organization. You are responsible for ensuring the highest level of quality from the initial stages of development all the way to the final code and product release. Your responsibilities include analyzing the requirements, creating and executing test cases, tracking the progress of the tests, and reporting any defects that may be encountered. You must also ensure that the software meets customer expectations and that customer feedback is incorporated into product releases.

The QA engineer is also responsible for communication between business stakeholders, engineers and quality assurance personnel. Your job is to mediate between different parties to ensure that everyone is on the same page and that they understand the consequences of their decisions, you are there to foresee what may go wrong as the result of a technical decision long before you catch it as a bug in the final product. You must also strive to ensure that the deadline is met and the product is released on time and no unexpected scenario is going to show up on the last days to delay the project.

This is not an easy role and requires you to understand the project from inside out and from outside in at the same time. You should speak end users’ non technical language to communicate to CX teams and be able to translate it into technical language of the engineers and follow, analyze, and debug the code to find the issue at the same time.

In this blog I am going to share my experiences with anyone who is interested in Software QA. I would be also super happy to hear from you. Please do not hesitate to contact me, write comments or message me here or on Twitter.