Verifying Functional Equivalence (FEQ)

:::note About this Page

We are actively investing in the development of a new capability that will provide two things to migration teams:

Automated testing procedures possibly using automated unit tests (AUT) and/or automated comparison tests (ACT) generated from legacy codes
A methodology of automatically inspecting source code to help with planning, executing, and tracking a functional equivalence tests for a migrated application using the AUTs and/or ACTs.

This page will organize knowledge and documentation about this capability.

:::

Problem Statement

In a typical upgrade effort there are many matters that can “break” the behavior of the new system. These breaks are much less of an issue if you are using tools to do a systematic transformation of a production tested source code, but it is there is still some risk of defects. If your upgrade plans include a lot of re-engineering or if you made aggressive use of VB6 language features that are not directly supported in .NET, you will want to test the new system carefully. Consequently, migration teams need an efficient means of verifying that an upgraded system is functionally equivalent (FEQ) to the original system. A fundamental challenge is preparing detailed procedures for using the original system. Procedures, including how to gather results, are very difficult to reverser engineer from code. They typically require expert user knowledge. However,if usage and results gathering procedures are available the operation of the old and new systems and the comparison of their results can be automated.

The Concept

The idea of AUT is not new, and most would agree that a valid, AUT capability has tremendous value if it can be sustained along with the application changes. However it is also commonly accepted that it is very difficult to retrospectively create valid unit tests for a large system, particularly if that system was not designed for automated unit testing or built using legacy technologies.

Furthermore, most of our customers’ idea of testing is to have the developer do adhoc manual testing of their change during development then to have users or testers do more manual system testing of the change along with some light weight regression testing. This manual “testing the change” approach works fine for incremental functional work, but it does not promote technical agility and it does scale well. Most importantly, in a migration project where arguably “everything is changing” the lack of scaleability for testing becomes a serious driver of risk and/or cost.

The need to prove functional equivalence is fundamental to our customers’ success in their migration efforts and we have repeatedly seen that customers need a lot of help with it. Time and time again, our customer’s testing problems become our problems either during the sales effort or when we try to close the project. So, in order to help meet this need, we intend to leverage the capabilities of our technology and provide customers with a more scalable AUT solution.

The idea was pitched to a client who was having particular difficulty with testing and could have benefited from this capability if we had it ready at the time.

From: Mark Juras
Sent: Thu, 2012-Nov-15 3:51 PM
To: client
Cc: George Juras
Subject: RE: Project termination

I admit I am disappointed that we are ending with 7 months still left in our original plan; particularly now that the product, the testing platform, and the processes are just starting to come together and overall progress is accelerating. However, I fully appreciate the gravity of factors challenging your efforts and I respect the decision.

The premature conclusion of our project has also inspired us to develop capabilities that will help our customers take on and overcome the challenge of upgrade risk and improve upgrade ROI. Specifically, we are enhancing our product so it can help identify/define and isolate technical test cases from code and then generate unit testing components that leverage mock dependency components. This will enable automated side-by-side testing and bottom-up testing of complex systems like yours. In a related effort, we are designing new analysis techniques and product enhancements that will allow migration teams to reorganize large, tightly-coupled systems into smaller more manageable layers and modular components that can be tested and upgraded incrementally and ultimately maintained more easily. The goal is improved scalability and effectiveness of testing processes to reduce risk and improved structural reengineering in constructing the new system. We hope these improved testing and maintainability aspects of our deliverables can be translated into in better ROI by customers. We will certainly reflect again and again on our experiences with you and your application as we conceive and create these capabilities. I may reach out to you to get your reaction and input.

If at some point you would like to pursue your testing process and design improvement opportunities with us, in the context of this application or some other system, please let us know; we would be happy to do a proof of concept with you to get started.

Prior AUT Efforts

There are many tools that can generate AUT code from the descriptions of the interfaces of those units. We suspect there are things we can do better than those tools because we know more than the interfaces, we know the operations and dependencies of the code behind the interfaces and this can potentially be used to:

Knowing more about dependencies of a unit can help a us initialize the state / data needed to allow the test to operate properly (or improperly for negative testinting)
Knowing operations, particularly conditional logic paths, can help us define the full set of test scenarios of a given unit.

Aaron Stemen did some work try and use gmBasic technology for AUT generation. Very little has been published about this effort other than he wanted to leverage mocking technology from Telerik, but some information is here:

Unit Test Generation
His work in progress is in GIT: gmslDll\DLL\UnitTests.

:::caution AUT and the State Problem

Developing an AUT for a function that has many external dependencies is very difficult. Most obvious dependencies are the infrastructure outside of the system. Dependencies such as databases, file system, OS environment, and web servers are needed to run the unit. They can be simulated using various mocking techniques, but more traditionally they must be satisfied by pain staking test environment setup procedures. In addition to environmental dependencies every almost unit of code depends on various variables and objects referenced by the unit but not initialized within the body of the unit. Note also if a unit calls external methods, then the state/data requirements of those external methods must also be valid for the unit test to be valid. You can see that this problem can quickly get very big and very difficult.

So while its not impossible to create a unit test it can be counter productive in typical cases, particularly if TDD was not carefully followed in developing the units.

This matter has been discussed extensively: just google for “legacy code automated unit testing challenges” or similar.

::: Introducing ACT

The idea of a automated comparison test is to automate the process of side-by-side testing. Conceptually the idea is simple:

setup inputs for the old system
run the old system
gather the results from running the old system in a formal structure in a text file
setup inputs for the new system - ideally these are identical to the inputs used for the old system
run the new system
gather the results from running the new system in a formal structure in a text file
compare the results from running the old system and the new system
If the results match the old and new system are declared functionally equivalent

ACT is a extremely simple and effective when the inputs, run, and outputs are well understood and easy to reproduce. Batch applications that run off an static input file in unattended mode and produce a detailed text output file are ideal. Many of the Fortran to C migrations we have done were batch systems and ACT was used with great success there. Our translation engines are also batch systems and we successfully use ACT on a routine basis to test them as well. Given how comfortable we are with ACT, it makes great sense that we would want to find away to apply it for verifying VB6/ASP/COM migration work.

Implementing ACT

The problem of implementing ACT for VB6/ASP/COM systems has three parts:

setting up inputs
running the systems
gathering the outputs

The immediate focus will be on solving the problem of gathering outputs. Specifically, the strategy is to have the old and new applications produce a detailed log of results as they run.

Trace Logging

The first step in supporting ACT is to get help with creating instrumented codes that log their behavior.

This will be done by adding a new instrumentation feature to the tool. Producing modified code is actualy a generally useful technique, and trace logging is just one application of the capability.

This enhancement is requested here: PRM-372 - Trace Logging

A copy of the initial version of the spec is below.

I would like the tool to help me instrument VB6 source code so that it will create an activity log at runtime.

1) Add Trace Logging commands to gmPL

The operation will be invoked using gmPL script with a couple new commands.

For consideration, here is a prototype gmPL script using the new commands, TraceLogging.xml:

    <gmBasic>
    <Storage Action="Create" Identifier="%JobId%" />
    <Select DeployLocation="%ProjFolder%\Instrumented" />
    <Select VirtualRoot="%SrcFolder%" />

    <Load "vbp1.vbp" />
    <Load "vbp2.vbp" />
    ...
    <Load "vbpN.vbp" />

    <AddTraceLogging identifier="scope" />

    <Output Status="New" Filename="Instrumented.bnd" />

    <AuthorTraceLogging/>

    <Storage Action="Close" />
    </gmBasic>

The AddTracelogging command indicates which methods in the loaded VBPs will be instrumented.

scope is an expression that indicates the scope of where trace logging statements should be added. It should follow other conventions for specifying fully qualified names. Something like this:

if scope is blank, all loaded VBPs are instrumented; this is the most common usage
if scope is a VBP Name then all methods in all files in the specified VBP will be instrumented (e.g. Debtors)
if scope is a file name all methods in the specified file will be instrumented (e.g. Debtors.frmSSOClientZoom)
if scope is a fully qualified method name, that method will be instrumented (e.g. Debtors.frmSSOClientZoom.SyncRSPointer_rsClients_grdClients)

The AuthorTraceLogging command will publish the instrumented VB6 code to the indicated output bundle file.

1. Author instrumented VBP Files
  Every VBP that contains an instrumented method will need to have a module reference added.

Module=gmTraceLogger; gmTraceLogger.bas

3) Author instrumented VB6 logic

Every method that is instrumented will have a block of code added before the first line. For example:

Private Sub SyncRSPointer_rsClients_grdClients(ClientNo As Long)
' This routine will make the current record of rsClients coincide with the current grid record
   On Error GoTo ErrorHandler
...

The tool will add modify all the methods in the VBPs as follows:

Private Sub SyncRSPointer_rsClients_grdClients(ClientNo As Long)

[trace logging block]

' This routine will make the current record of rsClients coincide with the current grid record
   On Error GoTo ErrorHandler
...

The trace logging code block will be specified by a new gmSL routine, AuthorTraceLoggerBlock
for example:

void AuthorTraceLoggerBlock()
{
    string className = ... gmsl for current filename;
    string methodName = ... gmsl to get current subroutine name;

    #TextStart
    gmTraceLogger_WriteLine("(%= className %).(%= methodName%)") ' added by gmBasic
    #TextEnd
}

The above is very generic ‘we were here’ tracing, but using gmSL allows the user to do more sophisticated dynamic and look deeper into runtime state.

This method could be added to the standard config, authortext.gmsl, or in a gmSL block in the gmPL (TraceLogging.xml) file.

4) Add a gmTraceLogger.bas file to the bundle

Every instrumented VBP will get a copy of gmTraceLogger.bas to be deployed in the instrumented VBP folder.

The content of this bas file should be specified by a new gmSL routine, AuthorTraceLoggerModule:

void AuthorTraceLoggerModule()
{
#TextStart
cat >(%= Project.DeployLocation %)\gmTraceLogger.bas <<'!)(!'
Attribute VB_Name = "gmTraceLogger"
Option Explicit

Private Const projectName = "(%= Project.Name %)"
Private Const logPath = "C:\temp\gmTraceLogger.log"

Public Sub gmTraceLogger_WriteLine(rpt as string)
    Open logPath "gmTraceLogger.log" For Append As 1
    Print #4, rpt
    Debug.Print projectName & "." & rpt
    Close 4
End Sub
!)(!
#TextEnd
}

5) folder structure of instrumented source tree

We will assume that the file references in the VBPs are relative to the VBP file.
We will preserve the filenames and folder structures form the original source tree in the instrumented source tree.
The variable VirtualRoot will be used as the base folder reference to help us keep our bearings.

For example, TraceLogging.xml specifies DeployLocation and virtualRoot

The orginal source tree is rooted at VirtualRoot. All original VBPs are in that tree and all orginal files are relative to the VBPs.

The instrumented source tree will be rooted at DeployLocation. All instrumented VBPs are in that tree and all instrumented files are relative to the VBPs.

The idea is that a folder comparison of these two trees will facilitate verification of the instrumentation process and show that there were no other changes to the source code.

Note the files that are included in the load VBPs but not instrumented should be passed through to the instrumented code tree. A file should not need to be instrumented more than once. A shared file that is instrumented should be instrumented in the instrumented tree even if all VBPs that use it are not marked for instrumentation. Its a rare case.

Please let me know what else we need for this specification and give it a try on a copy of the mertrix codes and send me the results.

Implementation Notes

See JIRA PRM-372