Over the years, change has been the only constant in the ever-growing market, which has developed the need to implement new or modified solutions. Though BAs may be involved during the project's initial phases, BA's opportunity to conduct a solution evaluation could be limited due to a project's duration. Evaluation tasks can be performed on solution components in varying stages of development to determine its value, provide insights into the solution delivered and explore areas of improvement. Evaluating the solution is based on the foundation laid; defining the problem and acceptance criteria will assist in baselining, which would serve during the on-going evaluation process. The key area for undertaking evaluation is to define and measure the solution, analyze the performance, assess the limitations, and provide recommendations.

So, what does a BA use to execute the solution evaluation task?

The fundamental is to understand and investigate what was the implemented v/s the anticipated solution. Some of the guidelines and tools that assist BAs in conducting the analysis are based on the variety of data captured, such as the business objective, scope defined, change strategy, current and future state, and risk analysis. In order to evaluate any solution, the measure to identify and assess the effectiveness is based on the solution itself, the circumstances of the organization, and how they define its value.

When does a BA evaluate the solution, and what are the techniques?

Depending on the nature of the business, the complexity of the program, and the stage the organization is at, the techniques used could vary, e.g., during the initial stage, in order to assess the vendors or to analyze the market, BA's can measure the competitors by defining the weightage and the criteria for selection.

Acceptance criteria, data mining, surveys, focus groups, key performance indicators are some of the other techniques that can be used to define the performance measures and evaluate the solution, which is either prototyped (proof of concept), in the pilot (beta) or in-operation, whereby establishing a point of reference.

Sample Diagram varying stages of development

How does a BA analyze the performance measures to identify limitations and evaluate the effectiveness of a solution?

The data collected (performance measures) is translated into meaningful insights for the stakeholders to consume. Analyzing the data also needs to take into account qualitative information, for example, customer experiences or changes in operational processes and staff impacts. This would help form a full picture of the solution that is accepted versus the solution implemented. Based on the current state, examining the trends, analyzing the performance variances, etc., would help realize the value and potential to be achieved by indicating its limitation, dependency, and strategic impact. Carrying out an impact assessment would help the BA understand the severity of the problem and define controls to mitigate the risks.

So, what happens after a solution is evaluated?

Providing recommendations and attaining sign-offs is one of the final steps a BA executes once the solution is evaluated. This results in understanding the value and potentially identify improvements that could be delivered as part of the enterprise analysis strategy, i.e., short, medium, or long term. Once the defects are identified, the BA should always ensure corrective and preventive actions are planned and undertaken, which consists of improvements to eliminate the shortcomings and increase the value of the solution delivered. The use of solutions should constantly be re-evaluated and should be an on-going process even after implementation.

Example of Short and long term evaluation

To conclude, solution evaluation is a critical component that can vary between organizations. If the task is not planned or executed, it can pose a significant risk to the business and outdate the solution in the ever-growing market. This could also result in a missed opportunity to identify external or internal factors to the organization, restricting the solution from realizing its full potential value. Other impacts of not performing a solution evaluation could be reduced customer and staff satisfaction, resulting in the organization investing further on defining workaround processes, thus increasing the operational costs, time, and resources. Therefore, it is critical for a BA to perform the solution evaluation to determine how well the solution performs or could potentially perform before the objective is deemed as achieved.