Refacola Resource Based Tests

The refacola resource based tests are a special variant of JUnit tests. Instead of writing Java code for each new test case, a sample project is created, and for each test a property file is written. The property file contains all information on how to treat the sample project and what results are expected.

Dependencies

The refacola resource based tests are defined in plugin
de.feu.ps.refacola.restest
This plugin depends on plugin de.feu.ps.tests.javatests. Most functionality is implemented in that plugin, however you do not have to look into these plugins. The restest plugin reexports all plugins you need (except your test specific plugins), so you do not have to worry about that.

How it works

The basic idea is to create test cases based on folders (which are assumed to be mini-projects) instead of test methods. For that reason, a different test suite is to be created, which does not collect all test methods but all mini-project folders and some test properties files instead.

Note: This functionality is implemented in de.feu.ps.tests.javatests, but usually you do not have to think about that.

For each mini-project, or for each properties file found in a mini-project folder, a test case is created and added to the test suite. This is then similar to "normal" JUnit tests in Eclipse.

Create your own tests

All you need is to create a new plugin, let's say my.refacola.resourcetests, and add a single Java file in that project. This Java files need to have a static method which gets invoked in order to assemble the test suite. Within this method, you have to configure your overall test set up, that is where to find the mini-projects and which main test class to use. This is pretty simple and canonical. Here is an example:

public class MyRefacolaResourceBasedTest extends RefacolaResourceBasedTest {

	public static junit.framework.Test suite() throws Exception {
		TestSuiteBuilder testSuiteBuilder =
			new TestSuiteBuilder("resources", "My Tests", "mytests",
					MyRefacolaResourceBasedTest.class);
		TestSuite testSuite = testSuiteBuilder.createSuite();
		return testSuite;
	}

	public MyRefacolaResourceBasedTest(String i_testBasePath) throws Exception 
	{
		super(i_testBasePath);
	}

}
The following figure illustrates how to configure the test suite and the overall structure of the test project.

Fig. 1: Structure and test suite set up
That's all Java code you have to write!

Configure a test project

A test project looks as the two projects in Figure 1 (Simple and Eiffel). You need some sample (program files), these files are usually placed in a folder "in". The expected output can then be placed in a folder "expected", which will enable you to compare the created results (which will be added to a folder "out" within your project folder) with the expected results.

Note: Writing is not supported yet, so at the moment neither files in expected nor in out are read or written.

The test is set up via a properties file. For each test, you have to create a new properties file. In order to store local information (which are not expected to be submitted to the source code repository), a local properties file with the same name but a different extension is read additionally and overwrites settings defined in the original file: In de.feu.ps.refacola.restest, you will find a mini-project called _ResourceTestTemplate, its properties file contains all necessary information. Here is a table of the properties which can be defined in order to configure your test. If no default value is specified, the property is mandatory.
PropertyDefaultDecription
General settings
ignorefalseIf true, test is ignored. This is a flag usually used to disable tests in your local set up.
order1000Order in which this test is to be executed. E.g., for first starting with quick tests and run larger tests then.
Classes
abstractRefactoringClassmandatoryFully qualified class name of refactoring. Do not forget to add the plugin, in which the class is defined, to the project dependencies.
programInfoProviderClassmandatoryFully qualified class name of program info provider. Do not forget to add the plugin, in which the class is defined, to the project dependencies.
constraintGeneratorClassde.feu.ps.refacola.api.
MinimizedConstraint
SetGenerator
solverCHOCOV2CHOCOV2 | CHOCOV1 | CREAM
Refactoring Selection
testProgramFilemandatoryMini-project relative path to the program (information) which are to be refactored.
forcedValue.*mandatorynew values, mandatory for all forced changes. Pattern:
forcedValue.forceChangeName=externalID,newValue
Example: forcedValue.x=1,ANY
Refacola configuration
conf.disableConstantEvaluationfalsedisable constant evaluation in constraintset generation
conf.timeOut-1 (ignore)timeout of for whole refactoring
conf.solverTimeOut-1 (ignore)timeout of for solver only
conf.maxSolutions-1 (ignore)max. solutions created by solver
Expectations
expected.successnot defined (ignore)Boolean, expected result of refactoring. If not defined, this value is ignored.
expected.constantlySolvednot defined (ignore)Boolean, refactoring is expected to be constantly solvable, in this case solver is not activated. This value is ignored if conf.disableConstantEvaluation is set to false. If not defined, this value is ignored.
expected.constantlyInconsistentnot defined (ignore)Boolean, refactoring is expected to be impossible, in this case solver is not activated. This value is ignored if conf.disableConstantEvaluation is set to false.
expected.noSolutions-1 (ignore)Integer, number of expected solutions. If the refactoring can be constantly evaluated, a single solution with the forced changes is created. If -1, this value is ignored and not tested.
expected.noProgramElements-1Integer, number of expected program elements. If -1, this value is ignored and not tested.
expected.noProgramElements-1Integer, number of expected facts. If -1, this value is ignored and not tested.
expected.noConstraintsConstant-1Integer, number of expected constant constraints. If -1, this value is ignored and not tested.
expected.noConstraintsVariable-1Integer, number of expected variable constraint. If -1, this value is ignored and not tested.
expected.noConstraintsTotal-1Integer, number of expected total constraint (const+var). If -1, this value is ignored and not tested.
Output Generation
changeset.writefalseBoolean value, if true, the change set is written to testfolder/out/test.properties
factbase.writefalseif true, the output file is created in testfolder/out
factbase.write.useDataAsNametrueif true, IProgramElement.getData() is used to create the name of an element
factbase.write.usePropertyAsNamefalseBoolean value, if true, a property value is used as name
factbase.write.nameProperty"identifier"String value, name of property used to retrieve element name, if usePropertyAsName is true
Special modes
checkConsistencyfalseBoolean value, if true, no refactoring is actually performed. Instead, the program (represented by its elements and facts) is checked against the rule set used by the refactoring. The constraint system is expected to be solved, that is solvable, expectedSolutions, and expectedSolverConstraints are ignored. Also, forcedValue.* is ignored, as no changes are applied. constraintGeneratorClass is ignored as well, since the CheckConstraintSetGenerator is always used in this mode.
Notes:

Change Sets

Besides the numbers, the actual changes are to be compared with expected changes. For that purpose, the test can automatically read a file with change sets and compare these expected changes with the actual ones.

The expected changes are loaded from a file found in a subfolder "exp" of your resource based test, the name of the file must equal the name of the test file with extension ".change". The following tree structure shows an example:

    +resources
        + Simple
            +in
                ...
            +exp
                test.changes
            test.properties
The changes file contains a list of change sets, each change set is stored in a single line. A change set itself contains changes. The following grammar is used:
ChangeSet   ::= Change ( ";" Change )* "\n"
Change      ::= <externalProgramID> ";" <propertyName> ";" <valueAsString>
Other ";" are masked with the backslash. The change sets and the changes are sorted before comparison to avoid ordering problems. However, for manual comparison it is recommended to already sort the changes (which can be simply achieved by forcing a test failure, which will print out the actual results in a sorted manner). The following snippet shows a sample change set for a fact base test:
ConstructorA;identifier;X
ConstructorA;identifier;X;B;identifier;A;ConstructorB;identifier;A
ConstructorA;identifier;X;B;identifier;X;ConstructorB;identifier;X
This can be read more user friendly as:
ConstructorA.identifier=X
ConstructorA.identifier=X; B.identifier=A; ConstructorB.identifier=A
ConstructorA.identifier=X; B.identifier=X; ConstructorB.identifier=X
The current format is simply easily parsed.

If a comparison fails, it will print out the (first) difference, e.g.

Change set differ on line 3: Class B..identifier: U != X
The first value is the expected one, while the second one is the actual value.

Writing change sets

A computed change set is written to the out folder of the test, if changeset.write=true is set.

Test Properties Template

The following code can be used as a template for new tests, a shorter template with less comments and default is given below.

################################################################################
# Test properties to be shared via SVN, local settings
# should be defined in test.local.properties (with svn:ignore set).
#
################################################################################

################################################################################
# Test Suite Settings
#
# known properties: ignore, order
################################################################################

# ------------------------------------------------------------------------------
# ignore.=true that test in specific test class (simple name) or
# ignore=true for ignore that project in all test classes
# ------------------------------------------------------------------------------
#ignore=true

# ------------------------------------------------------------------------------
# order in which this test is to be executed, default: 1000
# ------------------------------------------------------------------------------
# order=1000

################################################################################
# Test Specific Settings
#
# Properties can be defined for all tests or for a specific
# test.
# E.g., a property "timeout=1000" is used for all tests, while
# "MySpeedyTest.timeout=10" is only used in test MySpeedTest.
# 
# Properties are test specific, there are no generic settings defined.
################################################################################

# ------------------------------------------------------------------------------
# general settings:
# ------------------------------------------------------------------------------
# programInfoProviderClass, class name of program info provider, mandatory
programInfoProviderClass=de.feu.ps.refacola.factbase.FactBaseReader
# additional parameters (properties of the provider class)
programInfoProviderClass.factory=de.feu.ps.refacola.lang.enums.EnumsFactory

# constraintGeneratorClass, class name of constraint generator, 
# 		default: de.feu.ps.refacola.api.MinimizedConstraintSetGenerator 
constraintGeneratorClass=de.feu.ps.refacola.api.MinimizedConstraintSetGenerator
# solver = CHOCOV2 | CHOCOV1 | CREAM, default: CHOCOV2
solver=CHOCOV2

# ------------------------------------------------------------------------------
# refactoring selection 
# ------------------------------------------------------------------------------
# abstractRefactoringClass, class name of refactoring, mandatory
abstractRefactoringClass=de.feu.ps.refacola.lang.enums.refactorings.RefacolaChangeAccessRefactoring
# testProgramFile, file name of test program 
testProgramFile=in/simple.factbase

# ------------------------------------------------------------------------------
# forced values, mandatory for all forced changes
# the name of the change is either the explicitly defined name,
# or a computed name from kind's language, kind and property type,
# e.g. EiffelClass_Identifier (from _).
# The kind must be the kind as defined in the refactoring, and
# not the actual kind (which may be a subkind).
# ------------------------------------------------------------------------------
forcedValue.x=B,drei

# ------------------------------------------------------------------------------
# configuration
# ------------------------------------------------------------------------------
# conf.disableConstantEvaluation: boolean, disable constant evaluation in 
#       constraintset generation
# 	default: false 
conf.disableConstantEvaluation = false
# conf.timeOut: int, timeout of for whole refactoring
#	default: -1 (ignore)
conf.timeOut=-1
# conf.solverTimeOut: int, timeout of for solver only
#	default: -1 (ignore)
conf.solverTimeOut=-1
# conf.maxSolutions: int, max. solutions created by solver
#	default: -1 (ignore)
conf.maxSolutions=-1

# ------------------------------------------------------------------------------
# expected results
# ------------------------------------------------------------------------------

# expected.success: boolean, expected result of refactoring
#	default: if not defined, this value is ignored
expected.success=true
# expected.constantlySolved: boolean, refactoring is expected to be
# 	constantly solvable, in this case solver is not activated. This
#	value is ignored if conf.disableConstantEvaluation is set to false.
#	default: if not defined, this value is ignored
expected.constantlySolved=false
# expected.constantlyInconsistent: boolean, refactoring is expected to be
#	impossible, in this case solver is not activated. This
#	value is ignored if conf.disableConstantEvaluation is set to false.
#	default: if not defined, this value is ignored
expected.constantlyInconsistent=false
# expected.noSolutions: int, expected number of solutions
#	default: -1 (ignore)
expected.noSolutions=2

# ------------------------------------------------------------------------------
# expected statistic data
# ------------------------------------------------------------------------------
# expected.noProgramElements: int, expected number of program elements,
#	default: -1 (ignore)
expected.noProgramElements=2
# expected.noProgramFacts: int, expected number of facts,
#	default: -1 (ignore)
expected.noProgramFacts=1
# expected.noConstraintsConstant: int, expected number of constant constraints,
#	default: -1 (ingore)
expected.noConstraintsConstant=0
# expected.noConstraintsVariable: int, expected number of variable constraints
#	default: -1 (ignore)
expected.noConstraintsVariable=1
# expected.noConstraintsTotal: int, expected total number of constraints
#	default: -1 (ignore)
expected.noConstraintsTotal=-1

# ------------------------------------------------------------------------------
# output
# ------------------------------------------------------------------------------
# changeset.write: boolean, write change set to out folder. If change set is
#	written, and a change set is found in exp, the changes are checked.
#	default: false
changeset.write = true
# factbase.write: boolean, write fact base to out folder
#	default: false
factbase.write = true
# factbase.write.useDataAsName: boolean, use AST data for element name
#	default: true
factbase.write.useDataAsName = true
# factbase.write.usePropertyAsName: boolean, use a property value (if defined)
#	for element name, the property is defined via factbase.write.nameProperty
#	default: false
factbase.write.usePropertyAsName = true
# factbase.write.nameProperty: String
#	default: identifier
factbase.write.nameProperty=identifier
Short template:

# ------------------------------------------------------------------------------
# general settings:
# ------------------------------------------------------------------------------
programInfoProviderClass=de.feu.ps.refacola.factbase.FactBaseReader
programInfoProviderClass.factory=de.feu.ps.refacola.lang.abc_lang.Abc_langFactory

# ------------------------------------------------------------------------------
# refactoring selection and forced values
# ------------------------------------------------------------------------------
abstractRefactoringClass=de.feu.ps.refacola.lang.abc_lang.refactorings.RefacolaChange_b_cRefactoring
testProgramFile=in/in.factbase
forcedValue.x=B1,C2

# ------------------------------------------------------------------------------
# expected results
# ------------------------------------------------------------------------------
expected.success=true
expected.noSolutions=2

# ------------------------------------------------------------------------------
# output
# ------------------------------------------------------------------------------
changeset.write = true
#factbase.write = true

Using the tests

Thanks to the test suite trick, the resource based refactorings can almost be used like "normal" JUnit tests. The only downside is you cannot directly run a single test. You always have to run the whole suite by selecting "Run As.. / JUnit test" from the context menu of your test file class, as shown in Figure 2. However, once you have run the suite, you can select single tests in the JUnit view and run them separately as shown in Figure 3.

Fig. 2: Run as Junit test

Fig. 3: Select and run a single test case

Note: At the moment, only some text files are read using the Eiffel reader. Still, it is even possible to use the JDT within the tests. In that case, the overall set up doesn't much differ from the description above. Of course, the Java project has to be configured somehow. Then the suite has to be run as Eclipse Plugin test. This has already been done in the congen project, the congen tests (congen.parsertest) are resource based as well, although they do not use the current Refacola API.

Logging

The resource base test supports configuration of JDK logging levels. That is, all property keys starting with "log." are assumed to refer to JDK loggers, the logger follows this prefix. The logging level of the logger is set to the value specified by this property. E.g.
log.de.feu.ps.refacola.api.refactorings=SEVERE
log.de.feu.ps.refacola.api.refactorings.AbstractRefactoring=FINEST
log.de.feu.ps.refacola.api=INFO
specifies the level for three loggers. These lines are similar to definining the levels in a logger properties file without the prefix "log.", that is
de.feu.ps.refacola.api.refactorings=SEVERE
de.feu.ps.refacola.api.refactorings.AbstractRefactoring=FINEST
de.feu.ps.refacola.api=INFO
The following levels are defined:

A logger is automatically defined when its level is set. Although the name of a logger can be any name, usually the class name is used. Loggers are defined hierarchically, that is, the level of logger de.feu.ps.refacola.api.refactorings is used by a logger named de.feu.ps.refacola.api.refactorings.AbstractRefactoring as long as the level for the later is not explicitly defined somewhere else.

A common practice to use JDK logging is illustrated by the following code snippet:

public abstract class AbstractRefactoring implements IRefactoring {
    /**
     * Logger for this class
     */
    private static final Logger log = Logger.getLogger(AbstractRefactoring.class.getName());
    
    ...
    protected void runRefactoring() {
        ...
        if (log.isLoggable(Level.INFO)) {
            log.info("numberOfsolutions=" + numberOfsolutions); //$NON-NLS-1$
        }
    }

As you probably want to configure logging for debugging purposes, the best location to configure logging is not in the test properties file but in the local properties file. Do not forget to add svn:ignore to your local properties files, if this is not already configured in the test project.

Further readings: