A Discrete-Event Network Simulator
API
Loading...
Searching...
No Matches
example-as-test.h
Go to the documentation of this file.
1/*
2 * Copyright (c) 2020 Lawrence Livermore National Laboratory
3 *
4 * SPDX-License-Identifier: GPL-2.0-only
5 *
6 * Author: Peter D. Barnes, Jr. <pdbarnes@llnl.gov>
7 */
8
9#ifndef NS3_EXAMPLE_AS_TEST_SUITE_H
10#define NS3_EXAMPLE_AS_TEST_SUITE_H
11
12#include "test.h"
13
14#include <string>
15
16/**
17 * \file
18 * \ingroup testing
19 * Enable examples to be run as meaningful tests.
20 * Declaration of classes ns3::ExampleAsTestSuite and ns3::ExampleAsTestCase.
21 */
22
23namespace ns3
24{
25
26/**
27 * \ingroup testing
28 * Execute an example program as a test, by comparing the output
29 * to a reference file.
30 *
31 * User can subclass and override the GetCommandTemplate() and
32 * GetPostProcessingCommand() methods if more complex
33 * example invocation patterns are required.
34 *
35 * \see examples-as-tests-test-suite.cc
36 */
38{
39 public:
40 /**
41 * Constructor.
42 * \param [in] name The test case name, typically the program name
43 * and summary of the arguments, such as `my-example-foo`
44 * \param [in] program The actual example program names, such as `my-example`
45 * \param [in] dataDir The location of the reference file.
46 * This is normally provided by the symbol
47 * `NS_TEST_SOURCEDIR` in the `module-examples-test-suite.cc`
48 * file.
49 * The reference file should be named after
50 * the test case name,
51 * for example `my-example-foo.log`. If you use
52 * the `--update` argument to `test.py` or
53 * `test-runner` the reference file will be created
54 * with the correct name.
55 * \param [in] args Any additional arguments to the program.
56 * \param [in] shouldNotErr Whether an error return status should be
57 * considered a test failure. This is useful when testing
58 * error detection which might return a non-zero status.
59 * The output (on `std::cout` and `std::cerr`) will
60 * be compared to the reference logs as normal.
61 */
62 ExampleAsTestCase(const std::string name,
63 const std::string program,
64 const std::string dataDir,
65 const std::string args = "",
66 const bool shouldNotErr = true);
67
68 /** Destructor. */
69 ~ExampleAsTestCase() override;
70
71 /**
72 * Customization point for more complicated patterns
73 * to invoke the example program.
74 *
75 * \returns The string to be given to the `ns3 --command-template=` argument.
76 */
77 virtual std::string GetCommandTemplate() const;
78
79 /**
80 * Customization point for tests requiring post-processing of stdout.
81 *
82 * For example to sort return `"| sort"`
83 *
84 * One common case is to mask memory addresses, which can change
85 * when things are built on different platforms, recompiled locally,
86 * or even from run to run. A simple post-processing filter could be
87 *
88 * `"| sed -E 's/0x[0-9a-fA-F]{8,}/0x-address/g'"`
89 *
90 * Default is `""`, no additional processing.
91 *
92 * \returns The string of post-processing commands
93 */
94 virtual std::string GetPostProcessingCommand() const;
95
96 // Inherited
97 void DoRun() override;
98
99 protected:
100 std::string m_program; /**< The program to run. */
101 std::string m_dataDir; /**< The source directory for the test. */
102 std::string m_args; /**< Any additional arguments to the program. */
103 bool m_shouldNotErr; /**< Whether error return status is a test failure. */
104
105}; // class ExampleAsTestCase
106
107/**
108 * \ingroup testing
109 * Execute an example program as a test suite.
110 *
111 * You can use this TestSuite to add an example to the test suite with
112 * checking of the example output (`std::out` and `std::err`). This
113 * is an alternative to adding an example using the
114 * `examples-to-run.py` file. The key difference between the two
115 * methods is what criteria is used to for success. Examples added to
116 * `examples-to-run.py` will be run and the exit status checked
117 * (non-zero indicates failure). ExampleAsTestSuite adds checking of
118 * output against a specified known "good" reference file.
119 *
120 * \warning If you are thinking about using this class, strongly
121 * consider using a standard test instead. The TestSuite class has
122 * better checking using the NS_TEST_* macros and in almost all cases
123 * is the better approach. If your test can be done with a TestSuite
124 * class you will be asked by the reviewers to rewrite the test when
125 * you do a pull request.
126 *
127 * \par Test Addition
128 *
129 * To use an example program as a test you need to create a test suite
130 * file and add it to the appropriate list in your module CMakeLists.txt
131 * file. The "good" output reference file needs to be generated for
132 * detecting regressions.
133 *
134 * Let's assume your module is called `mymodule`, and the example
135 * program is `mymodule/examples/mod-example.cc`. First you should
136 * create a test file `mymodule/test/mymodule-examples-test-suite.cc`
137 * which looks like this:
138 *
139 * \code{.cpp}
140 * #include "ns3/example-as-test.h"
141 * static ns3::ExampleAsTestSuite g_modExampleOne("mymodule-example-mod-example-one",
142 * "mod-example", NS_TEST_SOURCEDIR, "--arg-one");
143 * static ns3::ExampleAsTestSuite g_modExampleTwo("mymodule-example-mod-example-two",
144 * "mod-example", NS_TEST_SOURCEDIR, "--arg-two"); \endcode
145 *
146 * The arguments to the constructor are the name of the test suite, the
147 * example to run, the directory that contains the "good" reference file
148 * (the macro `NS_TEST_SOURCEDIR` is normally the correct directory),
149 * and command line arguments for the example. In the preceding code
150 * the same example is run twice with different arguments.
151 *
152 * You then need to add that newly created test suite file to the list
153 * of test sources in `mymodule/CMakeLists.txt`. Building of examples
154 * is an option so you need to guard the inclusion of the test suite:
155 *
156 * \code{.py}
157 * if (bld.env['ENABLE_EXAMPLES']):
158 * module.source.append('model/mymodule-examples-test-suite.cc')
159 * \endcode
160 *
161 * Since you modified a CMakeLists.txt file you need to reconfigure and
162 * rebuild everything.
163 *
164 * You just added new tests so you will need to generate the "good"
165 * output reference files that will be used to verify the example:
166 *
167 * `./test.py --suite="mymodule-example-*" --update`
168 *
169 * This will run all tests starting with "mymodule-example-" and save
170 * new "good" reference files. Updating the reference file should be
171 * done when you create the test and whenever output changes. When
172 * updating the reference output you should inspect it to ensure that
173 * it is valid. The reference files should be committed with the new
174 * test.
175 *
176 * \par Test Verification
177 *
178 * You can run the test with the standard `test.py` script. For
179 * example to run the suites you just added:
180 *
181 * `./test.py --suite="mymodule-example-*"`
182 *
183 * This will run all `mymodule-example-...` tests and report whether they
184 * produce output matching the reference files.
185 *
186 * \par Writing good examples for testing
187 *
188 * When setting up an example for use by this class you should be very
189 * careful about what output the example generates. For example,
190 * writing output which includes simulation time (especially high
191 * resolution time) makes the test sensitive to potentially minor
192 * changes in event times. This makes the reference output hard to
193 * verify and hard to keep up-to-date. Output as little as needed for
194 * the example and include only behavioral state that is important for
195 * determining if the example has run correctly.
196 *
197 */
199{
200 public:
201 /**
202 * \copydoc ExampleAsTestCase::ExampleAsTestCase
203 * \param [in] duration Amount of time this test takes to execute
204 * (defaults to QUICK).
205 */
206 ExampleAsTestSuite(const std::string name,
207 const std::string program,
208 const std::string dataDir,
209 const std::string args = "",
210 const Duration duration = Duration::QUICK,
211 const bool shouldNotErr = true);
212}; // class ExampleAsTestSuite
213
214} // namespace ns3
215
216#endif /* NS3_EXAMPLE_TEST_SUITE_H */
Execute an example program as a test, by comparing the output to a reference file.
virtual std::string GetCommandTemplate() const
Customization point for more complicated patterns to invoke the example program.
bool m_shouldNotErr
Whether error return status is a test failure.
virtual std::string GetPostProcessingCommand() const
Customization point for tests requiring post-processing of stdout.
void DoRun() override
Implementation to actually run this TestCase.
ExampleAsTestCase(const std::string name, const std::string program, const std::string dataDir, const std::string args="", const bool shouldNotErr=true)
Constructor.
std::string m_args
Any additional arguments to the program.
std::string m_dataDir
The source directory for the test.
std::string m_program
The program to run.
~ExampleAsTestCase() override
Destructor.
Execute an example program as a test suite.
ExampleAsTestSuite(const std::string name, const std::string program, const std::string dataDir, const std::string args="", const Duration duration=Duration::QUICK, const bool shouldNotErr=true)
Constructor.
encapsulates test code
Definition test.h:1050
Duration
How long the test takes to execute.
Definition test.h:1054
A suite of tests to run.
Definition test.h:1267
Every class exported by the ns3 library is enclosed in the ns3 namespace.
ns3::TestCase, ns3::TestSuite, ns3::TestRunner declarations, and NS_TEST_ASSERT macro definitions.