source: mod_gnutls/test/README.md @ add6b31

Last change on this file since add6b31 was add6b31, checked in by Fiona Klute <fiona.klute@…>, 7 weeks ago

Documentation for building coverage reports

  • Property mode set to 100644
File size: 7.8 KB
Line 
1# Unit Tests for Apache's mod_gnutls
2
3Authors:
4Daniel Kahn Gillmor <dkg@fifthhorseman.net>,
5Fiona Klute <fiona.klute@gmx.de>
6
7There are a lot of ways that a TLS-capable web server can go wrong.  I
8want to at least test for some basic/common configurations.
9
10
11## Running the tests
12
13From the top level of the source, or from `test/` (where this README is),
14just run:
15
16```bash
17$ make check
18```
19
20You can also run specific test cases by passing their script names to
21make in the `TESTS` variable:
22
23```bash
24$ TESTS="test-03_cachetimeout_in_vhost.bash" make -e check
25```
26
27This should be handy when you're just trying to experiment with a new
28test and don't want to wait for the full test suite to run. Note that
29the test scripts are generated by `make`, using the names of
30directories in `tests/`.
31
32The default configuration assumes that a loopback device is available,
33and that `TEST_HOST="localhost"` resolves to the IPv6 and IPv4
34loopback addresses. `./configure` checks if `[::1]` and `127.0.0.1`
35are available, if one isn't it will be disabled. You can override the
36defaults by passing different values to `./configure`, e.g. to
37unconditionally use IPv4 only:
38
39```bash
40$ TEST_HOST="localhost" TEST_IP="127.0.0.1" ./configure
41```
42
43Note that having less than two addresses in `TEST_IP` will lead to
44some tests being skipped.
45
46If tests fail due to expired certificates or PGP signatures, run
47
48```bash
49$ make mostlyclean
50```
51
52to delete them and create fresh ones on the next test run. You could
53also use `make clean`, but in that case the keys will be deleted as
54well and have to be recreated, too, which takes more time.
55
56
57## Implementation
58
59Each test is defined by a directory in tests/, which the test suite
60uses to spin up an isolated Apache instance (or more, for tests that
61need a proxy or OCSP responder) and try to connect to it with
62gnutls-cli and make a simple HTTP 1.1 or 1.0 request. Make generates a
63minimal test script to start each test from `test-template.bash.in`.
64
65Test directories usually contain the following files:
66
67* `apache.conf` -- Apache configuration to be used
68
69* `test.yml` -- Defines the client connection(s) including parameters
70  for `gnutls-cli`, the request(s), and expected response(s). Please
71  see the module documentation of [mgstest.tests](./mgstest/tests.py)
72  for details, and [`sample_test.yml`](./sample_test.yml) and
73  [`sample_fail.yml`](./sample_fail.yml) for examples.
74
75* `backend.conf` [optional] -- Apache configuration for the proxy
76  backend server, if any
77
78* `ocsp.conf` [optional] -- Apache configuration for the OCSP
79  responder, if any
80
81* `fail.server` [optional] -- if this file exists, it means we expect
82  the web server to fail to even start due to some serious
83  configuration problem.
84
85* `hooks.py` [optional] -- Defines hook functions that modify or
86  override the default behavior of `runtest.py`. Please see the module
87  documentation of [mgstest.hooks](./mgstest/hooks.py) for details.
88
89The [`runtest.py`](./runtest.py) program is used to start the required
90services send a request (or more) based on the files described
91above.
92
93By default (if the `unshare` command is available and has the
94permissions required to create network and user namespaces), each test
95case is run inside its own network namespace. This avoids address and
96port conflicts with other tests as well has the host system. Otherwise
97the tests use a lock file to prevent port conflicts between
98themselves.
99
100
101## Robustness and Tuning
102
103Here are some things that you might want to tune about the tests based
104on your expected setup (along with the variables that can be passed to
105"make check" to adjust them):
106
107* They need a functioning loopback device.
108
109* They expect to have ports 9932 (`TEST_PORT` as defined in
110  `Makefile.am`) through 9936 available for test services to bind to,
111  and open for connections on the addresses listed in `TEST_IP`.
112
113* Depending on the compile time configuration of the Apache binary
114  installed on your system you may need to load additional Apache
115  modules. The recommended way to do this is to drop a configuration
116  file into the `apache-conf/` directory. Patches to detect such
117  situations and automatically configure the tests accordingly are
118  welcome.
119
120* If a machine is particularly slow or under heavy load, it's possible
121  that tests fail for timing reasons. [`TEST_QUERY_TIMEOUT` (timeout
122  for the HTTPS request in seconds)]
123
124The first two of these issues are avoided when the tests are isolated
125using network namespaces, which is the default (see "Implementation"
126above). The `./configure` script tries to detect if namespaces can be
127used (some Linux distributions disable them for unprivileged
128users). If this detection returns a false positive or you do not want
129to use namespace isolation for some other reason, you can run
130configure with the `--disable-test-namespaces` option.
131
132In some situations you may want to see the exact environment as
133configured by make, e.g. if you want to manually run an Apache
134instance with Valgrind using the same configuration as a test
135case. Use `make show-test-env` to dump `AM_TESTS_ENVIRONMENT` to
136stdout. If you want to load the test environment into the current bash
137instance, you can use:
138
139```bash
140$ eval $(make show-test-env)
141```
142
143If you are building on an exotic architecture which does not support
144flock (or timeouts using `flock -w`), `./configure` should detect that
145and disable locking, or you can disable it manually by passing
146`--disable-flock` to `./configure`. This will force serial execution
147of tests, including environment setup.
148
149
150## Testing with Valgrind memcheck
151
152The primary HTTPD instance will run under Valgrind if you run
153`./configure` with `--enable-valgrind-test`. While very slow that can
154be useful to catch memory leaks early.
155
156The [`suppressions.valgrind`](./suppressions.valgrind) file contains
157some suppressions for known reported errors that are deemed not to be
158mod\_gnutls issues. Note that the suppressions in that file are aimed
159at Debian x86_64 (or similar) systems, you may need to adjust them on
160other platforms. The Valgrind suppressions files to use are read from
161the `VALGRIND_SUPPRESS` variable in [`Makefile.am`](./Makefile.am).
162You can add suppression files on the command line by overriding
163`VALGRIND_SUPPRESS` like this:
164
165```bash
166VALGRIND_SUPPRESS="suppressions.valgrind extra.valgrind" make -e check
167```
168
169
170## Coverage reports
171
172You can create test coverage reports using [clang's source-based code
173coverage](https://clang.llvm.org/docs/SourceBasedCodeCoverage.html). This
174requires two things:
175
1761. Use the clang compiler to build the module (e.g. `CC=clang`).
1772. Pass `--enable-clang-coverage` to `./configure`, which adds the
178   required `CFLAGS` to the build.
179
180This will build mod\_gnutls with clang profiling instrumentation, and
181generate profiling data during tests. **Do not use a profiling build
182for production!**
183
184The gathered profiling data can then be compiled into a coverage
185report using `make coverage` (in this directory). The coverage report
186will be in [`coverage/index.html`](coverage/index.html). So the full
187process to build a coverage report might be, including opening the
188report (in this example using Firefox):
189
190```bash
191CC=clang ./configure --enable-clang-coverage
192make check
193cd test
194make coverage
195firefox coverage/index.html
196```
197
198If you'd like to work with the profiling data yourself, you can find
199the raw data in `outputs/coverage/`, the indexed and merged data in
200`outputs/coverage.profdata`.
201
202
203## Adding a Test
204
205Please add more tests!
206
207The simplest way to add a test is (from the directory containing this
208file):
209
210```bash
211$ ./newtest
212```
213
214This will prompt you for a simple name for the test, copy a starting
215set of files from `tests/00_basic`, and tell you the test script name
216you can use to run the test and add to the `test_scripts` variable in
217`Makefile.am` when your test is ready for inclusion in the test
218suite. The files in the test directory must be added to `EXTRA_DIST`
219in `tests/Makefile.am`.
Note: See TracBrowser for help on using the repository browser.