Solved Code coverage of unit tests in VBA

[Josef P.]

I'm working on AccUnit (unit test framework) again and would like to implement some kind of code coverage check.

Does anyone have an idea how to check something like this without having to make (big) changes in the source code?

Simplified example:

Public Function ProcedureToTest(ByVal X As Long) As Long

   If X Mod 2 = 1 Then
      ProcedureToTest = CalcWithOddParam(X)
   Else
      ProcedureToTest = CalcWithEvenParam(X)
   End If

End Function

Private Function CalcWithEvenParam(ByVal X As Long) As Long
   CalcWithEvenParam = X / 2
End Function

Private Function CalcWithOddParam(ByVal X As Long) As Long
   CalcWithOddParam = (X + 1) / 2
End Function

Unit test:

'AccUnit:Row(1, 1)
'AccUnit:Row(5, 3)
'AccUnit:Row(123, 62)
Public Sub Test_ProcedureToTest(ByVal X As Long, ByVal Expected As Long)

   Dim Actual As Long
   Actual = ProcedureToTest(X)

   Assert.That Actual, Iz.EqualTo(Expected)

End Sub

This test only tests with odd numbers as input. => CalcWithEvenParam is never called.
It would be great if I could show in the test framework that only 50% of the code was run.

Currently, the only option I can think of is to include a counter in the code.

Public Function ProcedureToTest(ByVal X As Long) As Long

   If X Mod 2 = 1 Then
#If UnitTest = 1 Then
   TestSuite.CodeCoverage "ProcedureToTest", 1
#End If
      ProcedureToTest = CalcWithOddParam(X)
   Else
#If UnitTest = 1 Then
   TestSuite.CodeCoverage "ProcedureToTest", 2
#End If
      ProcedureToTest = CalcWithEvenParam(X)
   End If

End Function

or

Public Function CodeCoverageLog(ByVal ProcName As String, ByVal Pos As Long) As Object
#If RunUnitTest = 1 Then
   TestSuite.CodeCoverage ProcName, Pos
#End If
End Function

Public Function ProcedureToTest(ByVal X As Long) As Long

   If X Mod 2 = 1 Then
      CodeCoverageLog "ProcedureToTest", 1
      ProcedureToTest = CalcWithOddParam(X)
   Else
      CodeCoverageLog "ProcedureToTest", 2
      ProcedureToTest = CalcWithEvenParam(X)
   End If

   CodeCoverageLog "ProcedureToTest", 3

End Function

But this is not really usable.

I also have the ability to insert code into the procedure from AccUnit and remove it after testing.
However, identifying where to insert a log line will not be so easy.

I'll probably have to cancel my wish, but maybe someone has a good idea.

 

[sonic8]

However, identifying where to insert a log line will not be so easy.

"not so easy" == impossible.
I don't think even an AI driven code coverage analyzer could currently reliably solve this task.

Even if it would work reliably, I'm very skeptical of the idea of automatically manipulating the code under test. It introduces quite a bit of risk of breaking stuff.

If you could insert helper-code reliably at the right places in the code under test, you (your analyzer) would need so in-depth understanding of the code that inserting the helper-code is unnecessary anyway. You could figure out code coverage without the helper-code already.

 

[Josef P.]

Even if it would work reliably, I'm very skeptical of the idea of automatically manipulating the code under test. It introduces quite a bit of risk of breaking stuff.

I definitely agree with you.
The safest variant would be the one with the If compiler statements. However, this makes the procedure look very ugly.

Another option would be:

Public Function ProcedureToTest(ByVal X As Long) As Long

   If X Mod 2 = 1 Then
1:    ProcedureToTest = CalcWithOddParam(X)
   Else
2:    ProcedureToTest = CalcWithEvenParam(X)
   End If

End Function

Before test generation, I could then replace the line numbers with log code.
=>

Public Function ProcedureToTest(ByVal X As Long) As Long

   If X Mod 2 = 1 Then
TestSuite.TraceLog "ProcedureToTest", 1:    ProcedureToTest = CalcWithOddParam(X)
   Else
TestSuite.TraceLog "ProcedureToTest", 2:    ProcedureToTest = CalcWithEvenParam(X)
   End If

End Function

But I don't really like it either.

 

[The_Doc_Man]

Part of the problem of automated testing is a program-flow analyzer that can identify branches in your code so that you can identify where you have alternative paths. Identifying such points would help you set logging points to determine the road not taken. (Apologies to Robert Frost.)

Essentially, such a tester would provide the start of an evaluation of whether your program's "automaton" is finite or non-finite. For Access, most of the time that should be easy, but for the small matter that you can do things that affect event code external to the routine being analyzed. E.g. if you use a form.SetFocus you trigger code in a different routine altogether, but... you don't call the control's .GotFocus routine. Access does.

 

[Josef P.]

For unit tests, I usually limit myself to methodes of classes and procedures in standard modules. I do not carry out UI tests automatically. Since I usually try to implement the single-responsibility principle, I think this is an acceptable compromise.

An example of tests: SqlToolsBuildCriteriaTests.cls (GitHub-Repro)
The procedure SqlTools.BuildCriteria is one cause of my interest for the installation of the code coverage. This is a very very long procedure with many sub-procedures. I am in the process of cleaning up this procedure a bit. This is a good moment to reflect on code coverage.

 

[Josef P.]

I'm very skeptical of the idea of automatically manipulating the code under test. It introduces quite a bit of risk of breaking stuff.

This gave me this idea: I separate the test run from the calculation of the code coverage by the tests.
I don't have to run the code coverage calculation as often compared to the tests.
This eliminates the risk of affecting the tests.

 

[Josef P.]

A first draft provides a result
I used the line numbers because you can insert and remove them with MZ-Tools, for example. That is an acceptable compromise for me.

This is how it currently looks:
a) Code for testing:

Public Function Method1() As Long

   Dim x As Long

'1 This line must be ignored.

1  x = 4 ' Line number with code

   x = x + 1

2   Method1 = x

End Function

Code during the test:

Public Function Method1() As Long

   Dim x As Long

'1 This line must be ignored.

1 CodeCoverageTracker.Track "ExampleClass", "Method1", 1: x = 4 ' Line number with code

   x = x + 1

2 CodeCoverageTracker.Track "ExampleClass", "Method1", 2:  Method1 = x

End Function

Video: https://accunit.access-codelib.net/videos/examples/CodeCoverageTest.mp4