Publish TestStarted and TestCompleted events in the TestContext framework [SPR-8710]

[spring-projects-issues]

Stepan Koltsov opened SPR-8710 and commented

Status Quo

AFAIU currently test initialization and finalization code must be placed into a test class (not in the application context), so code cannot be easily reused between unrelated test classes.

Proposal

SpringJUnit4ClassRunner could publish a TestStartedEvent before each test method and a TestFinishedEvent after each test in the application context. Listeners to these events could be used to clean caches from the previous test run, recreate database state before a test method, etc.

---

Issue Links:

• Publish test execution events to the test ApplicationContext [SPR-13916] #18490 Publish TestExecutionListener events to the ApplicationContext under test
• Introduce support for test-scoped beans in the TestContext framework [SPR-14034] #18606 Introduce support for test-scoped beans in the TestContext framework

[spring-projects-issues]

Sam Brannen commented

Hi Stepan,

I'm resolving this issue as Won't Fix since the functionality you're are requesting already exists in the form of the TestExecutionListener SPI.

Please see the Javadoc for TestExecutionListener for details. There are also concrete implementations that you can study for examples, such as:

• DependencyInjectionTestExecutionListener
• DirtiesContextTestExecutionListener
• TransactionalTestExecutionListener
• and various third-party implementations that you can find on the Web

Regards,

Sam

[spring-projects-issues]

Stepan Koltsov commented

Sam,

TestExecutionListener does not solve the problem.

To use TestExecutionListener you have to annotate each test class with @TestExecutionListeners annotation.

I have ContextConfigurations shared between tests, and I'd like to "attach" those test execution listeners to those contexts.

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(loader = AnnotationConfigContextLoader.class)
public class Test1 {

    public static class Listener extends AbstractTestExecutionListener {
        @Override
        public void beforeTestMethod(TestContext testContext) throws Exception {
            throw new RuntimeException("won't be thrown");
        }
    }

    @Configuration
    public static class ContextConfiguration {

        public Listener listener() {
            return new Listener();
        }

    }

    @Test
    public void test() {
        // expecting to fail, but it succeeds
    }

}

Instead of publishing TestStartedEvent SpringJUnit4TestRunner could fire TestExecutionListener methods for listeners found in context, not only in annotations to test classes.

[spring-projects-issues]

Stepan Koltsov commented

Sam, please answer something. Or reopen the issue.

[spring-projects-issues]

Sam Brannen commented

Reopening for further discussion and brainstorming.

[spring-projects-issues]

Sam Brannen commented

Hi Stephan,

To use TestExecutionListener you have to annotate each test class with @TestExecutionListeners annotation.

That is partially true: to use a TestExecutionListener it must be registered via @TestExecutionListeners; however, you do not need to annotate each test class if you have a test class hierarchy. It is also possible to declare @TestExecutionListeners once in your test class hierarchy -- for example, on an abstract base test. Subclasses would then not need to redeclare @TestExecutionListeners.

Is this something you have considered?

Does this meet your reuse requirement?

Instead of publishing TestStartedEvent SpringJUnit4TestRunner could fire TestExecutionListener methods for listeners found in context, not only in annotations to test classes.

First, please keep in mind that the TestContext framework is agnostic of the underlying testing framework. For example, both JUnit and TestNG are supported. Any new features would therefore have to be supported in a generic manner. Consequently, if we add support to Spring in line with your proposal, that code would have to reside in TestContextManager, not in SpringJUnit4ClassRunner.

Second, note that both TestNG and JUnit provide support for running tests concurrently. So one issue I see with your proposal is that firing an event in the test's ApplicationContext (or executing callback methods on a TestExecutionListener that is registered as a bean in the context) assumes that the context is used by a single test in a single thread; however, this is not the case for concurrent tests. In other words, what you are proposing could cause undesirable side effects if a particular TestExecutionListener implementation is stateful and used concurrently by two separate tests that use the same ApplicationContext.

Third, another issue I see with your proposal involves the ordering of TestExecutionListeners. The status quo enforces a predictable ordering of listeners by requiring that they be configured in a single place (i.e., via the @TestExecutionListeners annotation). If we implement your proposal, how would TestExecutionListeners registered as beans in the ApplicationContext be ordered with regard to the default set of TestExecutionListeners and/or any listeners that are registered on test classes via \@TestExecutionListeners?

As an aside, note that JUnit and TestNG both support before and after callbacks at the method level as well as the class level. JUnit also now supports so called Rules that provide callbacks somewhat comparable to those in the TestExecutionListener API. However, these features are similar to @TestExecutionListeners in that each test class (or test class hierarchy) has to make use of the corresponding annotations in order to trigger these callbacks. So I am not saying that using these features would be equivalent to your proposals, but you might find that they suit your needs.

Last but not least, if we implement #12483, that would also address the issue of reuse of configuration within a project. If #12483 were implemented would you still want the feature you have proposed here?

Please let us know your thoughts.

Thanks,

Sam

[spring-projects-issues]

Sam Brannen commented

Awaiting feedback from reporter.

[spring-projects-issues]

Sam Brannen commented

FYI: this issue is closely related to #18490.

[sbrannen]

This issue has been superseded by gh-18490.

Please continue the discussion there.