Home | History | Annotate | Download | only in unit-tests
History log of /src/usr.bin/make/unit-tests/Makefile
RevisionDateAuthorComments
 1.372  04-Aug-2025  sjg Use printf rather than echo -n
 1.371  04-Aug-2025  sjg sync-mi should do .inc files too?
 1.370  03-Aug-2025  sjg make: ensure output captured in meta file is accurate

In CollectOutput() a newline before the end of job->outBuf
results in a short write to stdout, but meta_job_output()
was recording the entire string in the meta file, so on
next call the left overs would be duplicated in the meta file.

Pass a length to meta_job_output() if it should only report part of
the data in meta file.
 1.369  29-Jun-2025  rillig tests/make: test backslash at the end of a line and in the :M modifier

As described in PR bin/46139 by David A. Holland.
 1.368  27-Jun-2025  rillig tests/make: test missing support for $^

Test provided by Diogo Santos via PR bin/59018; extended by me for .WAIT
and implicit rules.
 1.367  13-Jun-2025  rillig tests/make: fix race condition in debug output in parallel mode

Discovered in ATF run on evbearmv7hf.
 1.366  13-Jun-2025  rillig make: add on-demand inter-process stack traces

When debugging a build that heavily uses nested calls to sub-makes, the
chain of calls is useful to quickly assess the situation. If explicitly
requested, include the actions from any parent processes in the stack
traces.

Reviewed by: sjg
 1.365  13-Jun-2025  sjg Fix replacing ${TEST_MAKE} with make
 1.364  12-Jun-2025  rillig make: use a common style for unexpected error messages

In enomem, report the actual error instead of a fixed ENOMEM.
 1.363  05-Jun-2025  rillig make: remove empty line from diagnostic, add tests for output handling

The warning for the invalid internal option "-J" does not need a
trailing newline, as that is provided by Parse_Error already.

Having this newline in the output demonstrated a previously unknown bug
in handling the output of child commands in parallel mode. There, empty
lines are randomly discarded, depending on the chunks in which the
output from the child process is copied to make's stdout. See
job-output.mk for a demonstration.

The test for the Korn shell did not run the Korn shell in error handling
mode, as the error handling mode only affects the parallel mode but the
test didn't switch to that mode.

After fixing the shell-ksh.mk test, the single '"' in the output looked
suspicious, and indeed, the shell-csh.mk test had the same problem of
filtering more characters from the output than intended.
 1.362  09-May-2025  rillig make: unify debug messages for parallel mode

Always print the process status in hex, with a leading "0x", to avoid
confusion for status 127f, which looks suspiciously decimal.

Prefix all process IDs with "pid", to be able to filter them easily.

Omit trailing whitespace when printing a command.
 1.361  07-May-2025  rillig make: add end marker for -dg1, -dg2 and -dg3 debug log

The marker helps distinguish the output from sub-makes from the output
of the primary make.
 1.360  03-May-2025  rillig make: use uniform debug log messages for the token pool
 1.359  22-Apr-2025  rillig make: group the code for handling the job token pool
 1.358  13-Apr-2025  rillig tests/make: add tests for POSIX mode
 1.357  12-Apr-2025  sjg make/unit-tests: Fix expected results for meta-ignore tests
 1.356  30-Mar-2025  rillig make: use '"filename" line 123' for locations instead of 'filename:123'

The format 'filename:123' is commonly used for identifying a location in
a file. Text editors recognize it and allow quick navigation through it.
The previous format was specific to make and would have required custom
support in editors.

The new format was already used in stack traces, except for the first
line. Now all lines use the same format.
 1.355  25-Jan-2025  sjg make/unit-tests: avoid failure on NFS

The 'rm -rf ${TMPDIR}' at the end of the 'test' target, fails
when building on NFS because the temp file used for the target script
keeps the directory from being empty.

Move it to a separate target attached to '.END'

Reviewed by: rillig
 1.354  10-Jan-2025  rillig tests/make: test expressions based on undefined variables

When an undefined variable is used in a conditional, make complains about
"Malformed conditional", which is wrong since the conditional is
well-formed but errors out during evaluation.

In order to fix this wrong error message and preserve the remaining
behavior, cover the places in which an expression is evaluated with
undefined expressions being an error (VARE_EVAL_DEFINED), but with
neither producing an error message nor handling errors. This combination
results in the same behavior as evaluating the expressions in the mode
that allows undefined variables to be used in expressions.
 1.353  22-Nov-2024  rillig tests/make: speed up running the tests

Only 4 of the 389 tests need to postprocess their output, so spare the
extra 'cat' call in the majority of cases.
 1.352  31-Oct-2024  rillig make: in the debug log, always qualify line number with filename

Previously, to find out the filename corresponding to a line number from
a 'Parsing' line, it was necessary to find the preceding
'SetFilenameVars' line, which was unnecessarily time-consuming and
distracting.

The 'filename:line' format matches the one used in PrintStackTrace, and
it differs from the one used in PrintLocation, as the former format is
more common in other tools.

Most of the affected unit tests only use the 'Parsing' lines to give
additional context to their debug log, so no structural changes there.
 1.351  18-Aug-2024  rillig tests/make: clean up private temporary directory after running tests

Fixes PR bin/58569.
 1.350  07-Jul-2024  rillig make: don't track hash table chain lengths during lookup

The chain lengths are only used for debugging purposes, so avoid the
extra cost at each lookup. Instead, calculate the maximum chain length
only when it is actually requested in -dh mode.

The reported number changes slightly: Before, it was the length of the
chain that was actually traversed to find an entry, up to that entry,
now it is the length of the largest chain in the table, no matter if it
was actually accessed or not.
 1.349  04-Jul-2024  rillig make: on error, print the targets to be made

This helps to understand situations with several nested sub-makes in
varying directories.
 1.348  01-Jul-2024  sjg make: add :tc to capitalize first letter of each word

This is very hard to do without :tc

Reviewed by: rillig
 1.347  01-Jun-2024  sjg branches: 1.347.2;
Run check-expect.lua in .CURDIR
 1.346  01-Jun-2024  rillig tests/make: require all expected messages to be listed in the .mk files
 1.345  24-May-2024  rillig make: in -DCLEANUP mode, free variables and their values

The variables in the 3 scopes must be freed before the scopes themselves
are freed by Targ_End.

The test opt-m-include-dir creates a directory of the form '*.tmp', thus
it must be removed before attempting to only remove regular files of
this name.

POSIX requires setenv to copy the passed name and value, so there is no
need to keep that memory allocated any longer.
 1.344  30-Apr-2024  sjg Subst DEFSYSPATH in deptgt-phony
 1.343  20-Apr-2024  rillig make: provide more context information for parse/evaluate errors
 1.342  07-Jan-2024  sjg make: unit-tests handle TEST_MAKE:T != make

We need to allow for ${TEST_MAKE:T}[1-9]: etc when
TEST_MAKE is not 'make'
 1.341  09-Sep-2023  sjg Add .MAKE.JOBS.C to indicate wether -jC is supported
 1.340  17-Aug-2023  rillig tests/make: extend tests for the ':mtime' modifier
 1.339  20-Jun-2023  rillig make: allow targets to be used as multiple-inclusion guards

This style is used by FreeBSD, among others.
 1.338  16-Jun-2023  rillig tests/make: add test for multiple-inclusion guards
 1.337  16-Jun-2023  rillig tests/make: clean up tests for the ':M' and ':S' modifiers
 1.336  01-Jun-2023  rillig tests/make: clean up comments, extend a few tests
 1.335  10-May-2023  rillig tests/make: clean up tests for .for loops

Most of the tests from forloop.mk were already in directive-for.mk.
 1.334  09-May-2023  sjg make: add :mtime to provide mtime of file

The value of the variable is passed to stat(2)
and st_mtime is new value.
An optional arg can be used if stat(2) fails, otherwise
the current time is used.

See varmod-mtime.mk for usage examples.
 1.333  25-Feb-2023  sjg make: meta-ignore tests will not work if TMPDIR is /tmp

Since /tmp and /var/tmp are always part of .MAKE.META.IGNORE_PATHS
the varname-dot-make-meta-ignore_* tests will not work
if TMPDIR is either (or subdir) of those.
 1.332  23-Feb-2023  sjg Add unit-tests for .MAKE.META.IGNORE_*

These unit-tests should only be run if TEST_MAKE
has a valid .MAKE.PATH_FILEMON to indicate that filemon is
supported.

All three tests use the same meta-ignore.inc
which runs three sub-makes

one to initialize the target .meta file
two to ignore a file using .MAKE.META.IGNORE_{PATHS,PATTERNS,FILTER}
three to verify that without .MAKE.META.IGNORE_* the target is out of date.

Fix the order of _SED_CMDS to avoid errors when ${MAKE:T} appears in
.OBJDIR
 1.331  24-Jan-2023  sjg make: .SYSPATH: to add dirs to sysIncPath

.SYSPATH: with no sources will clear sysIncPath
otherwise sources are added

Reviewed by: rillig
 1.330  23-Jan-2023  sjg make: .[NO]READONLY for control of read-only variables

Reviewed by: rillig
 1.329  19-Jan-2023  rillig tests/make: rename files that are not test cases
 1.328  19-Jan-2023  rillig tests/make: add tests for whitespace in leading '@+-'
 1.327  25-Sep-2022  rillig tests/make: fix test for option '-e' (broken since 2022-01-23)
 1.326  10-Sep-2022  rillig tests/make: be more specific about the egrep compatibility
 1.325  09-Sep-2022  sjg Handle deprecation of egrep
 1.324  03-Sep-2022  sjg Only update distrib/sets/lists/tests/mi with valid files

sync-mi: use CVS/Entries to ensure we only update
distrib/sets/lists/tests/mi with files known to CVS.

Reviewed by: rillig
 1.323  02-Sep-2022  sjg Enable make/unit-tests/directive-for-break

Also ensure MAKE_VERSION is ignored.
 1.322  02-Sep-2022  sjg make: add .break to terminate .for loop early

When .break is encountered within a .for loop
it causes immediate termination.

Outside of a .for loop .break causes a parse error.

Reviewed by: christos
 1.321  25-Aug-2022  rillig tests/make: fix canonicalization for single-digit day of month
 1.320  24-Aug-2022  rillig make: prevent future out-of-bounds errors when parsing expressions

A modifier in an expression ends not only at the next ':' or at the
closing '}' or ')', but also at the end of the string.

Previously, testing for the end of the string had been done separately,
which was error-prone since 2006-05-11, when indirect modifiers were
introduced. Since then, it was possible that the string terminator '\0'
was accidentally skipped in cases where the loop condition only tested
for the ending character. When parsing indirect modifiers, the ending
character is indeed '\0', but when parsing direct modifiers, it is '}'
or ')'.

A welcome side effect is that in the case of unclosed expressions such
as '${VAR:Modifier', the amount of error messages is reduced from 2 or 3
to only 1. The removed error messages were wrong and thus confusing
anyway.
 1.319  26-Jul-2022  sjg Pass MAKEOBJDIR to opt-m-include-dir

Prevent makeing opt-m-include-dir.tmp in src tree.

Reviewed by: rillig
 1.318  10-Jun-2022  rillig tests/make: reword comment in Makefile to be more specific
 1.317  02-Jun-2022  skrll Increase the resource limit so these tests pass on hppa.

LGTM from Roland
 1.316  23-May-2022  rillig tests/make: document and demonstrate .for i containing .if empty(i)

PR bin/43821 describes the inconsistency that in a '.for i' loop, the
condition '.if ${i:M*.c}' works since 2009 while the seemingly
equivalent condition '.if !empty(i:M*.c)' does not access the variable
'i' from the .for loop but instead the global 'i'.

Resolving this situation in a backwards-compatible and non-surprising
way is hard, as make has grown several features during the last 20 years
that interact in various edge cases. For now, document the most obvious
pitfalls.
 1.315  08-May-2022  rillig tests/make: remove test varquote

The test varmod-quote-dollar covers the same topic.
 1.314  08-May-2022  rillig tests/make: migrate cond1 test to other, more specific tests

The tests in cond1 were a mixture of "everything related to conditions",
and the test cases were heavily dependent on each other, which made them
hard to understand. Move each test case to its corresponding
special-purpose test.
 1.313  07-May-2022  rillig make: allow to randomize build order of targets

In complex dependency structures, when a build fails, a probable cause
is a missing dependency declaration between some files. In compat mode,
the build order is deterministic, in jobs mode, it is somewhat
deterministic. To explore more edge cases, add the line ".MAKE.MODE +=
randomize-targets" somewhere in the makefile.

Fixes PR bin/45226 by riastradh. Reviewed by christos.
 1.312  18-Apr-2022  rillig make: only switch to POSIX mode if '.POSIX:' is the first line

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html
says that in order to make a makefile POSIX-conforming, its first
non-comment line must be the special dependency line '.POSIX:' without
any source dependencies.

Previously, make switched to POSIX mode even if such a line occurred
anywhere else, which was allowed by POSIX but was deep in the
"unspecified behavior" area. For NetBSD make, there is no big
difference since it doesn't ship any <posix.mk> file, this change mainly
affects the bmake distribution.

Previously, makefiles that contain '.POSIX:' somewhere in the middle
could fail due to <posix.mk> resetting .SUFFIXES, among other things.

Suggested by Simon J. Gerraty, who also reviewed an earlier version of
this change.
 1.311  26-Mar-2022  rillig make: fix crash on .undef of an environment variable (since 2020-10-06)
 1.310  25-Mar-2022  rillig tests/make: suppress -DCLEANUP output in test deptgt-phony
 1.309  12-Feb-2022  rillig tests/make: document the history of bugs in '-k' mode

Reported in PR#49720 in 2015, fixed independently in compat.c 1.199 from
2020-12-07.
 1.308  12-Feb-2022  rillig tests/make: demonstrate bug for .SILENT in jobs mode (since 2003)

Reported 2011 in PR#45356.
 1.307  09-Feb-2022  rillig tests/make: remove redundant 'echo' from variable assignments

Before main.c 1.231 from 2014-09-09, a variable assignment using the
operator '!=' generated a warning "Couldn't read shell's output" if the
output of the command was empty. The simplest way to suppress this
wrong warning was to add an empty 'echo' to the command. This hack is
no longer needed.
 1.306  09-Feb-2022  rillig make: fix mistakes, spelling and typos in comments and manual page

No binary change for -DNDEBUG.
 1.305  09-Feb-2022  rillig make: prefix the warning about read-only .OBJDIR with a colon

For consistency with the other warnings.
 1.304  09-Feb-2022  rillig tests/make: use more distinctive placeholder for TMPDIR
 1.303  07-Feb-2022  rillig tests/make: demonstrate combination of .USE with transformation rule

Reported in 2003 in PR toolchain/20993. Linking a transformation rule
with .USE or .USEBEFORE node makes the transformation rule fail.
 1.302  27-Jan-2022  sjg numEntries can vary
 1.301  27-Jan-2022  sjg Do not allow /usr/obj to cause failure.
 1.300  23-Jan-2022  rillig tests/make: migrate modts to varmod-to-separator and explain
 1.299  23-Jan-2022  rillig tests/make: rename var-class to var-scope

There is no such concept as a "variable class" in make, these tests
focus on the variable scope instead.
 1.298  23-Jan-2022  rillig tests/make: extend test suite, move old tests to 2020 scheme

The tests from envfirst.mk are now in opt-env.mk.
The tests from modword.mk are now in varmod-select-words.mk.
 1.297  22-Jan-2022  rillig tests/make: explore edge cases involving .ifmake
 1.296  19-Jan-2022  rillig tests/make: demonstrate wrong location in diagnostic (since 2018-12-22)

When a target has multiple places where commands are defined, the
diagnostics mixed up the filename in some cases.
 1.295  11-Jan-2022  rillig tests/make: clean up after test for option '-dF'

When the file for the debug log cannot be opened, make exits
immediately. This doesn't give the test a chance to clean up the
temporary log file.

Instead of trying to treat a regular file as a directory and create a
file in it, assume that the directory /nonexistent-$uuid is actually
nonexistent. This leads to the same kind of error message, independent
of strerror(3).
 1.294  09-Jan-2022  rillig tests/make: add placeholders for .PARSEDIR and .INCLUDEDFROMDIR

When run via 'cd usr.bin/make/unit-tests && make test', the tests are in
the current directory. When run via ATF, the tests are in
/usr/tests/usr.bin/make/unit-tests, while the current directory is a
temporary directory. Allow both variants, plus others that may occur in
the bmake distribution.
 1.293  28-Dec-2021  rillig tests/make: clean up after test 'opt-debug-file'
 1.292  28-Dec-2021  rillig tests/make: document inconsistencies between '!=' and '::!='

Found while trying to make the error messages from Cmd_Exec more
detailed.
 1.291  23-Dec-2021  rillig tests/make: explain the current behavior of the option '--version'
 1.290  14-Dec-2021  rillig make: remove unreachable code for parsing the dependency operator

At the point where ParseDependencyOp is called, cp is guaranteed to
point to either ':' or '!'.

No functional change.
 1.289  13-Dec-2021  rillig tests/make: extend tests for parsing makefiles
 1.288  12-Dec-2021  rillig tests/make: test and document the newly added "variable" .SUFFIXES
 1.287  09-Dec-2021  rillig make: add back support for jemalloc 100

The ports sun2 and vax default to jemalloc 100, as opposed to the rest
of the ports, which use jemalloc 510. Some developers choose to use
jemalloc 100, so let them run the tests with memory debugging as well.

Suggested by mrg on source-changes-d.
 1.286  05-Dec-2021  rillig make: fix use-after-free in modifier ':@'

Without memory allocator debugging, the newly added test doesn't show
any obvious failure.

With memory allocator debugging enabled, all make versions since
2016.02.27.16.20.06 crash with a segmentation fault.
 1.285  05-Dec-2021  rillig tests/make: migrate to jemalloc > 100

When I started working on usr.bin/make in 2020, I used NetBSD 8, which
used jemalloc 100.

After updating to NetBSD 8 to 9.99.x, the debugging options for jemalloc
stopped working since MALLOC_OPTIONS had been replaced with MALLOC_CONF
in 2019.

Enable malloc debugging again, to reliably reproduce the use-after-free
for the variable modifier ':@'.
 1.284  19-Oct-2021  sjg Replace ${.OBJDIR} with <curdir> for consistent results.

If we replace ${.OBJDIR} with <objdir> some tests produce
different results depending on the value of .OBJDIR

For similar reasons eliminate .OBJDIR and .PATH from output of
varname-empty

Reviewed by: rillig
 1.283  30-Aug-2021  rillig tests/make: explain where "${var}" in .for loops comes from
 1.282  31-Jul-2021  rillig tests/make: split tests for the variable modifier ':O'

The tests for parse errors are now in varmod-order, which lets the other
tests focus on the desired behavior of the modifiers.
 1.281  30-Jul-2021  sjg Add :On for numeric sort

Reviewed by: christos rillig
 1.280  29-Jun-2021  sjg Allow BROKEN_TESTS to cause TESTS to be skipped.

Some tests simply do not work in some environments.
Eg. shell-ksh on macos/arm64
Allow local site to set BROKEN_TESTS to skip those they know
will not work.

Reviewed by: rillig
 1.279  16-Jun-2021  rillig tests/make: demonstrate wrong error handling in jobs mode
 1.278  18-May-2021  sjg Do not trust that /nonexistent does not exist

Use /nonexistent.${.MAKE.PID} to avoid failure when
/nonexistent actually exists.
 1.277  27-Apr-2021  rillig branches: 1.277.2;
tests/make: test the combination of -de with -j1

The test cases are the same as in opt-debug-errors.mk. The output
differs in several details though.

Even though the option '-k' is given (which is the default for any tests
that don't override it in unit-tests/Makefile), there is no message
"(continuing)" anywhere.

The failed target is printed twice. Once before the failed commands,
once after. This redundancy is not necessary and may be removed in a
follow-up commit.

The printed commands are in their unexpanded form, which may or may not
be more helpful than the expanded and space-normalized form of compat
mode. Either way, this is an unnecessary inconsistency between compat
mode and jobs mode.

In jobs mode, the message "make: stopped in $dir" is printed for each
failure, which is helpful since each of the jobs may have started in a
separate directory.
 1.276  15-Apr-2021  rillig tests/make: demonstrate handling of null bytes
 1.275  04-Apr-2021  rillig tests/make: split test for modifier ':@' into separate files

The file varmod-loop.mk has grown too large to be single-purpose, plus
it combined parse-time and run-time tests. This has the downside that
as soon as a parse-time test results in an error, the run-time tests are
not run anymore.
 1.274  03-Apr-2021  rillig make: use C99 bool type instead of defining its own

No functional change.
 1.273  14-Mar-2021  rillig tests/make: add test for short-circuit evaluation of modifiers
 1.272  14-Mar-2021  rillig make: demonstrate wrong expression evaluation in -dL mode
 1.271  14-Feb-2021  rillig make: add test for the variable modifier ':sh'
 1.270  14-Feb-2021  rillig make: fix TMPDIR in unit-tests

Combining the assignment operator ':=' with the variable modifier ':U'
on the same variable does not work as intended. At the point where the
':U' is evaluated, the variable is guaranteed to be defined because the
code in VarAssign_EvalSubst says so.
 1.269  06-Feb-2021  sjg Avoid test failures when more than one user run them.

Use a private TMPDIR
a subdir of .OBJDIR if using it
otherwise /tmp/uid${.MAKE.UID}
 1.268  04-Feb-2021  rillig make: remove .PARSEDIR from test output of varname-empty

The .PARSEDIR has nothing to do with .CURDIR. This wrong assumption
made the NetBSD test suite fail since the actual .PARSEDIR was not
replaced with "<curdir>".
 1.267  03-Feb-2021  rillig make: in the test varname-empty, keep more debug output

Filtering out the interesting lines based on the very specific pattern
"Var_Set" was surprising.
 1.266  02-Feb-2021  rillig make: demonstrate that neither -dg2 nor -dg3 produces debug output
 1.265  01-Feb-2021  rillig make: add test for reading from make's stdin

Just out of curiosity whether that really works. It does.
 1.264  30-Jan-2021  rillig make(1): add test for combining the options -j, -n, -t

This is a preparation for refactoring the complicated condition in
JobStart.
 1.263  29-Jan-2021  rillig make(1): demonstrate unnecessary creation of empty files in jobs mode
 1.262  23-Jan-2021  rillig make(1): fix unit tests for dash and bash 5.1
 1.261  19-Jan-2021  rillig make(1): normalize shell error messages in tests

Newly supported is bash, which outputs "bash: line 123".
 1.260  31-Dec-2020  rillig make(1): add test for error handling and expansion in .for loops
 1.259  30-Dec-2020  rillig make(1): add test for .endfor without corresponding .for
 1.258  29-Dec-2020  rillig make(1): add test that explains how variables are exported

Exporting the variables at the right time and with the correct values is
a subtle issue. The current implementation carefully marks variables as
ready to be exported, then exports them and at the same time tries to
export as few variables as possible, to avoid memory leaks. This test
describes and explains how all this works in detail.

This test also justifies that the call to Var_ReexportVars happens in
the make process itself, not in the child processes, no matter whether
these are created with vfork or (only theoretically) with plain fork.
This has changed in compat.c 1.217, job.c 1.390 and main.c 1.504 from
2020-12-27.
 1.257  27-Dec-2020  rillig make(1): add test for expansion errors in jobs mode

Since compat mode and jobs mode are implemented separately and vary in
lots of small details, each of them needs to be tested on its own.
 1.256  21-Dec-2020  rillig make(1): align tests for :localtime with those for :gmtime

These tests had been almost the same before, now they are as similar as
possible again.
 1.255  21-Dec-2020  rillig make(1): move tests for :gmtime to parse time

It's easier to have both the expressions and the expected values in a
single file. This also allows for flexible handling of multiple
acceptable outputs, in this case for 32-bit time_t.
 1.254  21-Dec-2020  sjg Set default for .MAKE.OS once
 1.253  20-Dec-2020  rillig make(1): include line numbers in output of test varmod-edge.mk

Suppressing the line numbers does not provide much benefit, given that
the test file doesn't change often.
 1.252  19-Dec-2020  rillig make(1): add test for null byte in .for loop body
 1.251  19-Dec-2020  rillig make(1): demonstrate wrong line numbers in .for loops, since 2007-01-01
 1.250  14-Dec-2020  rillig make(1): add test for parsing the end of a condition
 1.249  13-Dec-2020  rillig make(1): demonstrate wrong error handling in compat mode
 1.248  13-Dec-2020  sjg Prune job debug output that may appear out of order.

A race between child and parent means that we cannot
guarantee whether all child output is seen before we call
JobClosePipes, thus intervening debug output can appear
before or after the last child output.
 1.247  13-Dec-2020  sjg Ensure we have a value for .MAKE.UID
 1.246  13-Dec-2020  rillig make(1): add test for misspelled directives

This test allows the other directive-* tests to focus on the purpose of
the individual directive, allowing these tests to continue after
parsing, without errors.
 1.245  12-Dec-2020  rillig make(1): add test for a shell with error control

None of the predefined shells has error control, and the corresponding
code had not been covered by the existing unit tests.
 1.244  10-Dec-2020  rillig make(1): test all combinations of switches for running commands

The code in JobPrintSpecials is rather complicated and contains
surprising interaction between some of the switches.

To see the exact effects of the switches, record the current state and
its output, to prevent accidental breakage during the upcoming
refactorings.
 1.243  09-Dec-2020  rillig make(1): remove noise from test output of opt-jobs-no-action
 1.242  09-Dec-2020  rillig make(1): add test for the -n option combined with RunFlags
 1.241  09-Dec-2020  rillig make(1): add test for combining -j1 with -n
 1.240  07-Dec-2020  rillig make(1): normalize output of test sh-dots for non-native mode
 1.239  07-Dec-2020  rillig make(1): add tests and tutorial for the ?= assignment operator
 1.238  07-Dec-2020  rillig make(1): add test for wrong exit status 0 after failed targets with -k
 1.237  06-Dec-2020  rillig make(1): add test for .END after failed main node
 1.236  05-Dec-2020  sjg Use .MAKE.OS if we can
 1.235  04-Dec-2020  rillig make(1): use fixed format for debug output of the directory cache

The previous output format had a %-20s conversion specifier. This
produced different output depending on the length of the pathname, which
was too difficult to normalize. By moving the directory name to the
end, it is no longer necessary to fill up any space, and the numbers are
always aligned properly.

As a result, 3 of the unit tests no longer need any special
postprocessing of their output.
 1.234  01-Dec-2020  rillig make(1): add test and tutorial for indirect modifiers
 1.233  01-Dec-2020  rillig make(1): add test for the special .MAKEFLAGS variable
 1.232  01-Dec-2020  rillig make(1): add tests for suppressing "stopped in"

These tests demonstrate the unwanted behavior described in PR bin/55578
and PR bin/55832.
 1.231  30-Nov-2020  rillig make(1): fix tests for :gmtime and :localtime for 32-bit time_t

https://bugs.freebsd.org/251465
 1.230  30-Nov-2020  sjg Add some tests for meta mode
 1.229  29-Nov-2020  rillig make(1): clean up memory management for CachedDirs

Previously, the reference count for a newly created CacheDir had been
set to 1 in CacheNewDir. This was wrong because at that point, the
object had not been referenced by any nonlocal variable. The reference
count is no longer incremented at this point.

All callers of CacheNewDir either append the newly created CachedDir to
a SearchPath via Lst_Append and CachedDir_Ref, or they assign it to a
global variable via CachedDir_Assign.

Since the reference count is no longer wrongly incremented, it does not
need to be decremented more than necessary in Dir_End. To keep the code
simple and maintainable, all assignments to global variables are now
handled by CachedDir_Assign. Adding a CachedDir to a list is still done
manually via Lst_Append, and the corresponding code for decrementing is
in SearchPath_Clean and SearchPath_Free. These details may be cleaned
up in a follow-up commit.

As a result, when OpenDirs_Done is called in the unit tests, the list of
open directories is empty. It had been non-empty in a single unit test
before (dep-wildcards.mk), as a result of calling Dir_Expand.

The additional debug logging for the reference counting is not enabled
by default since it contains memory addresses, which makes the output
dependent on the memory allocator.

The function CachedDir_Destroy has been merged into CachedDir_Undef,
which had only been used in Dir_End before. The new name emphasizes
that it corresponds to CachedDir_Ref.
 1.228  29-Nov-2020  rillig make(1): add debug logging for OpenDirs_Done
 1.227  29-Nov-2020  rillig make(1): add debug logging for reference counting of CachedDir
 1.226  25-Nov-2020  sjg Add .MAKE.UID and .MAKE.GID
 1.225  25-Nov-2020  sjg access(2) does not work for root, so skip objdir-writable
 1.224  24-Nov-2020  rillig make(1): add test for bug in error handling of .BEGIN in -k mode
 1.223  24-Nov-2020  rillig make(1): add test for error in dependency of .END node in -k mode
 1.222  24-Nov-2020  rillig make(1): add test for missing "Stop" after error in .END
 1.221  23-Nov-2020  rillig make(1): make output of test opt-debug-graph1 consistent

The output from the directory cache made the regular NetBSD build fail
because the pathname to the working directory differs, thus affecting
the spacing.

This time, document why the directory cache needs to be excluded from
the output.
 1.220  23-Nov-2020  rillig make(1): test that .PHONY targets are not resolved using suffix rules
 1.219  22-Nov-2020  rillig make(1): add test for debug output from transformation rules
 1.218  22-Nov-2020  rillig make(1): add another unit test for suffix handling
 1.217  22-Nov-2020  rillig make(1): document the selection of the main target
 1.216  22-Nov-2020  rillig make(1): make configuration of export tests simpler

The configuration section of unit-tests/Makefile is already complicated
enough to read, due to the excessive use of regular expressions.
Therefore, to keep the structuring elements at a minimum, inline the
.for loop.
 1.215  22-Nov-2020  rillig make(1): add more output to test for -dg1

Previously, the variables section had been omitted. This was because
the variables had been output in hashcode order until 2020-10-18, and
because some of the variable values are specific to the test environment
or the individual run (MACHINE_ARCH, MAKE.PPID).
 1.214  21-Nov-2020  rillig make(1): capture more output from test deptgt-suffixes

This will have an effect with the next commit.
 1.213  21-Nov-2020  rillig make(1): limit memory usage in tests

There is a suspicious condition in SuffUpdateTarget code that looks wrong
on the first sight. When removing it though, make allocates huge amounts
of memory. To stop freezing the whole machine in this situation, limit
the total memory.

The limit of 200000 has been determined experimentally on NetBSD 8.0
x86_64. With a limit of 100000, make wouldn't even start. 100 MB of
memory is really a lot for such a simple program that according to top(1)
only needs 8 MB. But 200 MB is still better than 5 GB.

Since the Makefile is used on other platforms as well, via the bmake
distribution, and since every operating system has its own list of ulimit
options, make this configurable.
 1.212  21-Nov-2020  rillig make(1): add test for incomplete transformation rule
 1.211  19-Nov-2020  rillig make(1): fix test opt-debug-jobs for native shell

Even after substituting the actual shell name with <shell>, the trailing
optional -q option needs to be normalized.
 1.210  19-Nov-2020  rillig make(1): fix test opt-debug-jobs when compiled with dash as shell

make USER_CPPFLAGS=-DDEFSHELL_CUSTOM='\"/usr/pkg/bin/dash\"'
 1.209  19-Nov-2020  rillig make(1): remove redundant postprocessing for test var-op-shell

The regular expression from the previous line already matches everything
that the removed line could ever match.
 1.208  19-Nov-2020  rillig make(1): remove postprocessing for test directive-export-gmake

Since revision 1.3, directive-export-gmake.mk takes care of handling this
case. Removing the "error" lines from the output of the tests only makes
the test harder to understand and may also hide future bugs.
 1.207  18-Nov-2020  sjg Cleanup - colliding fixes for same issue
 1.206  18-Nov-2020  sjg Some tests need help with dash
 1.205  17-Nov-2020  rillig make(1): make test vardebug independent from the shell
 1.204  17-Nov-2020  rillig make(1): fix test output of var-op-shell for dash

Dash is helpful and outputs the line number in case of an error. The
other shells don't do this. Accept both variants.
 1.203  16-Nov-2020  rillig make(1): add test for self-referencing suffix rule

Just to ensure that make doesn't run into an endless loop.
 1.202  15-Nov-2020  rillig make(1): fix typo in test for variable with empty name
 1.201  15-Nov-2020  rillig make(1): update and add comments in tests

As a result of the new comments, some line numbers have changed in the
output of the tests. No other changes.
 1.200  14-Nov-2020  rillig make(1): use progname instead of hard-coded 'make' in warning

This only affects the warning that chdir to objdir failed. In sub-makes
the progname includes the [n], allowing to narrow down the actual cause
of the problem.
 1.199  14-Nov-2020  rillig make(1): add test for the -t option in jobs mode
 1.198  12-Nov-2020  sjg Pass a writable flag to Main_SetObjdir to control writable check

For curdir and an explicit .OBJDIR target, we allow for
the directory to be read-only.
During InitObjdir we otherwise default to requiring objdir to be
writable - this can be controlled by env variable
MAKE_OBJDIR_CHECK_WRITABLE

Add unit-tests/objdir-writable

Reviewed by: christos rillig
 1.197  10-Nov-2020  rillig make(1): add more tests for parsing .if directives

The details of parsing are quite tricky and not documented in the manual
page. Record the current behavior to be a little safer in future
refactorings.
 1.196  10-Nov-2020  rillig make(1): add test for the obsolete variable VPATH
 1.195  09-Nov-2020  rillig make(1): in tests, move command line arguments into the test proper
 1.194  09-Nov-2020  rillig make(1): move test for != assignments to var-op-shell
 1.193  09-Nov-2020  sjg Control TZ for depsrc-optional
 1.192  08-Nov-2020  rillig make(1): in lint mode, only allow '&&' and '||', not '&' and '|'

These variants of the condition operators are neither documented in the
manual page nor are they used in practice.
 1.191  08-Nov-2020  rillig make(1): add test for expanding variable expressions
 1.190  07-Nov-2020  rillig make(1): add test for OP_SUBMAKE
 1.189  07-Nov-2020  rillig make(1): add test for job command flags
 1.188  04-Nov-2020  rillig make(1): add test for undefined variables in command line arguments

The variable discardUndefined has an implicit negation in its name,
which makes it hard to understand. Plus, most of the time it is true.
It's better to have a flag that is false most of the time and has a
positive name.

On the first attempt of inverting that variable, I stumbled upon
MainParseArgs, which initially leaves discardUndefined == FALSE, and
after handling the dashed options, sets it to TRUE. This would make a
difference when more command line arguments would be added later via the
.MAKEFLAGS special target.

Upon further inspection, the only place where discardUndefined is used
is in VarAssign_EvalSubst in parse.c, and that place is not reachable
from any of the dashed options. Therefore, discardUndefined could
already be set at the very beginning of MainParseArgs or even when
initializing the global variable itself, without any observable
difference.

Not even the ::= variable modifier could do anything about this since it
is not reachable from the dashed command line options as well, and in
addition, it expands its right-hand side in any case, always discarding
undefined variables. Oh, these little inconsistencies everywhere.
 1.187  03-Nov-2020  rillig make(1): move tests from directives.mk to separate tests
 1.186  02-Nov-2020  rillig make(1): remove obsolete POSTPROC.varname from unit tests

It was not needed anymore since 2020-10-18.
 1.185  02-Nov-2020  rillig make(1): add test for parse errors in commands in lint mode (-dL)

The difference to non-lint mode is that the exit status is now 2 instead
of 0.
 1.184  02-Nov-2020  rillig make(1): remove debugging code from sync-mi
 1.183  02-Nov-2020  rillig make(1): add test for parse errors in shell commands, compat mode
 1.182  02-Nov-2020  rillig make(1): handle errors when sync-mi fails because of a syntax error
 1.181  01-Nov-2020  rillig make(1): move test flags to test varmod-match-escape.mk
 1.180  31-Oct-2020  rillig make(1): add more tests for the variable modifier :localtime
 1.179  31-Oct-2020  rillig make(1): add test for recursive variable expressions
 1.178  31-Oct-2020  rillig make(1): merge the SysV modifier tests into varmod-sysv
 1.177  30-Oct-2020  rillig make(1): move test flags for varname-dot-shell to the test itself
 1.176  29-Oct-2020  rillig make(1): move MAKEFLAGS handling in vardebug test to the test itself
 1.175  24-Oct-2020  rillig make(1): add test for looking up suffixes
 1.174  23-Oct-2020  rillig make(1): add test for dependencies of the form '%.o: %.c'
 1.173  23-Oct-2020  rillig make(1): add test case in which .BEGIN depends on .END

Just for fun.
 1.172  23-Oct-2020  rillig make(1): add test for the '::' dependency operator
 1.171  23-Oct-2020  rillig make(1): move tests from cond2.mk to varmod-ifelse.mk (fixup)
 1.170  23-Oct-2020  rillig make(1): add test for .ERROR_CMD in jobs mode
 1.169  20-Oct-2020  rillig make(1): split test suffixes.mk into simpler, isolated tests

The code in suff.c is already hard to understand, and so were the tests
in suffixes.mk since several independent topics were merged into a
single test.

Splitting this test into a separate test per issue allows to document
the expected and actual behavior in more detail. That's complicated
enough already.

PR bin/49086
 1.168  18-Oct-2020  sjg In meta mode we do not have to re-run all tests when Makefile is touched
 1.167  18-Oct-2020  rillig make(1): add test for turning a target into a transformation
 1.166  17-Oct-2020  rillig make(1): document why the ampersand is not listed in export.exp
 1.165  10-Oct-2020  sjg Limit output of export tests to POSIX compatible env vars
 1.164  08-Oct-2020  rillig make(1): remove test misc.mk from Makefile
 1.163  08-Oct-2020  rillig make(1): fix test opt-debug-jobs on non-native platforms
 1.162  05-Oct-2020  rillig make(1): revert previous commit

It had accidentally reverted all the work from the past few days.
 1.161  05-Oct-2020  rillig make(1): fix double-free bug in -DCLEANUP mode (since 2020-10-02)

The bug had been introduced with dir.c 1.155 on 2020-10-02 22:20:25. In
that commit, openDirectories was replaced with a combination of a list
with a hash table, for more efficient lookup by name.

Upon cleanup, OpenDirs_Done is called, which in turn called
Dir_ClearPath. Dir_ClearPath takes full ownership of the given list and
empties it. This was no problem before since afterwards the list was
empty and calling Lst_Free just frees the remaining list pointer.

With OpenDirs, this list was combined with a hash table, and the hash
table contains the list nodes, assuming that the OpenDirs functions have
full ownership of both the list and the hash table. This assumption was
generally correct, except for the one moment during cleanup where full
ownership of the list was passed to Dir_ClearPath, while the hash table
still contained pointers to the (now freed) list nodes. This by itself
was not a problem since the hash table would be freed afterwards. But
as part of Dir_ClearPath, OpenDirs_Remove was called, which looked up
the freed directory by name and now found the freed list node, trying to
free it again. Boom.

Fixed by replacing the call to Dir_ClearPath with code that only frees
the directories, without giving up control over the list.
 1.160  04-Oct-2020  rillig make(1): add tests for parsing assignments, especially :sh

Luckily nobody uses the :sh variable assignment modifier since its
syntactical variant != is simpler.
 1.159  03-Oct-2020  rillig make(1): add test demonstrating the Towers of Hanoi puzzle

It's not the primary task of make to handle procedure calls with
parameters, combined with lexical scoping, therefore the code does not
look as straight-forward or clean as in other programming languages. It
feels more like squeezing a programming problem from the imperative
world into the world of declarative dependencies.

A more idiomatic way of implementing this puzzle should be as a
dependency graph since that's both the natural structure of the puzzle
and the primary domain of make. Something like having a main target
"hanoi-5" that depends on intermediate targets of the form
"move-1.2.3.4.5-_._._._._-_._._._._", each representing a single
configuration of the stacks. These targets could be generated
dynamically. A benefit of this implementation would be that the puzzle
could be resumed from an arbitrary configuration, just just from the
initial configuration.
 1.158  03-Oct-2020  rillig make(1): add tests for the various supported shells
 1.157  03-Oct-2020  rillig make(1): add test for the -dj debug option
 1.156  02-Oct-2020  rillig make(1): add tests for parsing and exporting variables

Once again, there are a few surprises deeply hidden inside the edge
cases.
 1.155  29-Sep-2020  rillig make(1): fix test job-output-long-lines

The test failed on Linux since the test relied on the two child
processes interleaving each other. This is not guaranteed to happen
though.
 1.154  27-Sep-2020  rillig make(1): add test for merging long lines of job output
 1.153  27-Sep-2020  rillig make(1): run tests with jemalloc debugging enabled

This protects against very simple memory allocation bugs such as
migrating Lst_ForEachUntil to Lst_ForEach without remembering that
Lst_ForEachUntil can handle the situation where the current list node is
removed from the list, but Lst_ForEach cannot. This happens in
Make_ExpandUse, for example.
 1.152  25-Sep-2020  rillig make(1): add test for harmless bug in Parse_File

When there is a dependency group at the end of a top-level makefile,
this dependency group is not finished properly. This allows to add
further commands to the targets of this dependency group, which was not
intended.
 1.151  25-Sep-2020  rillig make(1): add test for make() function in conditions
 1.150  25-Sep-2020  rillig make(1): add test for deleting the suffixes during parsing
 1.149  25-Sep-2020  rillig make(1): add test for partial expansion of undefined variables
 1.148  23-Sep-2020  rillig make(1): remove debug output from counter tests

Now that the tests work as intended, the debug information is no longer
necessary. It was only intended to track down the cause of the
unexpected behavior.
 1.147  23-Sep-2020  rillig make(1): add variant of the counter test

I had expected that using the ::+= modifier instead of the ::= modifier
would work, since the assignment modifier for COUNTER no longer contains
a reference to itself. But instead of ending up at 4, the counter even
goes up to 6.
 1.146  23-Sep-2020  rillig make(1): fix assertion failure in -j mode with .END node

There had been two separate global variables for the .END node, and in
parallel mode, only the one in jobs.c was initialized.

The code in JobRun heads over to Compat_Make without calling Compat_Run
first, which left the variable ENDNode uninitialized.
 1.145  21-Sep-2020  rillig make(1): run tests with absolute filenames as well

The tests varname-dot-parsedir and varname-dot-parsefile had been broken
before if they were run with the "-f $PWD/$test.mk" option. This way of
running the tests is used by FreeBSD.
 1.144  21-Sep-2020  rillig make(1): fix tests for .PARSEDIR/.PARSEFILE

The output of these tests previously depended upon whether the test
makefile was specified as relative or as absolute path. In NetBSD, it
is specified as relative path, in FreeBSD as absolute path.

Adjust the expected output to hide this difference; in the .rawout files
it is still visible.
 1.143  15-Sep-2020  rillig make(1): move flags for lint test into the test itself
 1.142  14-Sep-2020  rillig make(1): add test for undefined expressions in conditions in lint mode
 1.141  14-Sep-2020  rillig make(1): add tests for numbers in conditions
 1.140  13-Sep-2020  rillig make(1): add tests for the various .include directives
 1.139  13-Sep-2020  rillig make(1): fix sync-mi helper target, at least for adding tests
 1.138  07-Sep-2020  rillig make(1): explain why each test is run in a sub-make
 1.137  07-Sep-2020  rillig make(1): use consistent spelling for postprocessing
 1.136  05-Sep-2020  rillig make(1): add test for intentional hash collision for variable names

Hash collisions may slow down make in certain special situations. There
is no point though in maliciously triggering such a situation since
anyone who can inject values into makefiles can easily run shell
commands using the :!cmd! modifier or similar mechanisms. Crafting
variable names just to slow down make is thus not an attack vector.
 1.135  05-Sep-2020  rillig make(1): move test for -dg1 from opt-debug-g1 to opt-debug-graph1
 1.134  05-Sep-2020  rillig make(1): add tests for each debug option
 1.133  04-Sep-2020  rillig make(1): re-enable the archive test

The test had failed in the releng build because it assumed it were run
with .CURDIR == .PARSEDIR. This assumption is true when the tests are
run directly from usr.bin/make, but not when they are run from
tests/usr.bin/make.
 1.132  04-Sep-2020  rillig make(1): add test for the special variable MAKEFILE
 1.131  04-Sep-2020  rillig make(1): extend tests for the :hash variable modifier

The previous test vectors didn't contain any hash with a leading zero.
This could have been a simple programming mistake by using %8x instead
of the intended %08x. Using snprintf wouldn't have been possible anyway
since the hex digits are printed in little-endian order, but without
reversing the bits of each digit. Kind of unusual, but doesn't affect
the distribution of the hashes.
 1.130  02-Sep-2020  rillig make(1): add test for the .for directive

For a long time, I had assumed that the iteration variables of a .for
loop are just normal global variables. This assumption was wrong but
didn't have any consequences.

The iteration variables of a .for loop can just be accessed like global
variables, therefore it is not obvious that they are implemented in a
completely different way.

There are some edge cases in conditions used inside .for loops, in which
the iteration variables cannot be used like normal variables. An
example is brought up in https://gnats.netbsd.org/47888, which observes
that the defined() and empty() functions in conditions only work with
variables but ignore the iteration "variables", simply because these are
not variables but only expressions.
 1.129  29-Aug-2020  rillig make(1): fix sh-dots test

The exact output depends on the shell. Some shells prefix the error
message with their name, some don't.
 1.128  29-Aug-2020  rillig make(1): add test for the .ifmake directive
 1.127  29-Aug-2020  rillig make(1): add tests for .else and .for + .endif
 1.126  29-Aug-2020  rillig make(1): fix assertion failure for .SUFFIXES in archives

This occurred in the posix1.mk test, even though it is disabled in
unit-tests. But in tests/usr.bin/make it still runs. There, it should
have produced an "expected failure" but crashed instead.

The archive-suffix test is the stripped-down version of the posix1 test.
 1.125  29-Aug-2020  rillig make(1): fix test opt-debug-g1

Including the directory cache made the test output fragile.
 1.124  28-Aug-2020  rillig make(1): add test for interrupting a command
 1.123  28-Aug-2020  rillig make(1): add test for the undocumented .NULL special dependency target
 1.122  28-Aug-2020  rillig make(1): add tests for the special .INCLUDES and .LIBS variables
 1.121  28-Aug-2020  rillig make(1): disable the sync-mi convenience target
 1.120  27-Aug-2020  rillig make(1): add test for the -dg1 option
 1.119  23-Aug-2020  rillig make(1): add test for + commands with the -n command line option
 1.118  23-Aug-2020  rillig make(1): add tests for the -i -k -v -V -W options
 1.117  23-Aug-2020  rillig make(1): add tests for numeric comparisons in .if directives
 1.116  23-Aug-2020  rillig make(1): fix assertion failure in debug output for read-only variables
 1.115  23-Aug-2020  rillig make(1): add test for the newly added .SHELL variable
 1.114  22-Aug-2020  rillig make(1): add test for expanding wildcards in directories

This test covers DirExpandInt.
 1.113  22-Aug-2020  sjg bash says 'command not found'
 1.112  22-Aug-2020  rillig make(1): try to set the variable named "" using the command line

Before 2020-08-22, the command line argument '=cmline-plain' triggered
undefined behavior, just like the assignment '=assigned' in the Makefile.
 1.111  22-Aug-2020  sjg Use TEST_MAKE to run unit-tests

This lets us take advantage of new features like ${.SHELL}
 1.110  22-Aug-2020  rillig make(1): add test for assigning to the variable named ""

This has been possible at least since 2013, probably much longer.
 1.109  22-Aug-2020  rillig make(1): add test for deferred variable expansion in dependencies
 1.108  22-Aug-2020  sjg Supress /bin/sh: in error for sh-dots
 1.107  22-Aug-2020  rillig make(1): add test for "no target to make"
 1.106  22-Aug-2020  rillig make(1): add test for the special "..." in shell commands
 1.105  22-Aug-2020  rillig make(1): add test for combining :: and .USEBEFORE
 1.104  20-Aug-2020  rillig make(1): enable debug logging for archive test

This test succeeds locally on NetBSD 8.0 but not in the official test
runs on https://releng.netbsd.org/test-results.html. To see the
difference, run the test with full debug information.

This test is commented out in usr.bin/make/unit-tests/Makefile, but that
doesn't stop tests/usr.bin/make/t_make.sh from running it nevertheless,
since over there, all *.mk files are considered tests, be they commented
out or not.
 1.103  20-Aug-2020  rillig make(1): add test for parsing functions in .if conditions
 1.102  19-Aug-2020  rillig make(1): add test for the -n option
 1.101  19-Aug-2020  rillig make(1): add test for the -q option
 1.100  16-Aug-2020  rillig make(1): move tests for the :Ox modifier into separate file

The test has been extended by ensuring that the shuffled words are still
the same. Comparing two shuffled lists is probabilistic, but comparing
their sorted results is not, therefore that's completely sensible to do.

When writing this test, by coincidence I discovered how to generate the
"Undefined variable" error message. Unfortunately, the error message is
wrong since the variable NUMBERS is defined at that point. In summary,
that error message is shown when it shouldn't, and when it should it is
not shown. Still, I'm glad that I finally found it.
 1.99  16-Aug-2020  rillig make(1): move tests for the :M modifier into separate files

The test for the different escaping has been adjusted to actually show
the different parsing results in the test output. To do this, it had to
get its own file since it needs the -dv debug flag and specialized
post-processing.
 1.98  16-Aug-2020  rillig make(1): run tests in an almost empty, controlled environment

Several of the tests use simple variable names that might be influenced
by environment variables of the same name. Especially the tests for the
?= variable assignment operator need an empty environment to start with,
to produce reliable results.

The PATH must be in the base environment since several tests depend on
the usual tools like echo(1), sleep(1), grep(1). Setting the PATH to a
fixed value would have made it impossible to run the tests in a custom
environment that don't have a /bin directory. It's the user's
responsibility to provide a sane PATH.
 1.97  16-Aug-2020  rillig make(1): force all tests to be run with the -r flag

Without that flag, <sys.mk> is loaded before the tests. The tests are
not intended to either use or even test these rules and definitions,
therefore it is safe to omit this command line option.

If there should ever be tests for POSIX-conformance that need the
builtin rules and definitions, these tests can still ".include <sys.mk>"
at the beginning and be done with it, since the -m command line option
is not touched. The system-default rules and definitions are still
available, they are just not active by default.

Suggested by sjg.
 1.96  16-Aug-2020  rillig make(1): use consistent formatting for running the tests

With the many newly added tests, having two messages ("testing xzy" and
"postprocessing xyz") made the output too noisy since the words were not
aligned nicely. Use the same formatting as for MAKEVERBOSE=0, and
remove the message for postprocessing. If there should ever be problems
during postprocessing, it's trivial to run "../make TESTS=bad-test -dl"
to find out where the problem is.
 1.95  16-Aug-2020  rillig make(1): clean up cleaning of test files

The variables CLEANFILES and CLEANDIRS are not supposed to be
user-settable, thus use a simple "=" instead of "+=".

Since CLEANDIRS is always set, there is no point in making the rm
conditional.
 1.94  16-Aug-2020  rillig make(1): move tests for :S, :C, :@ from modmisc to their separate tests
 1.93  16-Aug-2020  rillig make(1): add dummies for fine-grained tests, one per single feature

The test names have been derived from the current manual page.

All these tests are dummies right now, and the code from the existing
tests will be moved into the new tests step by step.

This is done to prevent modmisc, escape, varmod-edge and varmisc from
growing without any bounds, and to reduce the side-effects of one test
to the others.
 1.92  15-Aug-2020  rillig make(1): fix archive test

Even though it is commented out in this Makefile, it is still run by
atf, on purpose. This test is broken on FreeBSD, but on NetBSD it
should still be run.

Without removing the archive first, it could be left over from a
previous test failure and thus be newer than all its members, leading to
${.OODATE} being empty.
 1.91  09-Aug-2020  rillig make(1): add test for .USE combined with inference rule without commands

Discovered by christos.
 1.90  09-Aug-2020  rillig make(1): add test for non-obvious .MAKE.EXPORTED edge case
 1.89  07-Aug-2020  rillig make(1): add test for exporting variables
 1.88  07-Aug-2020  rillig make(1): document why the suffixes test must stay disabled
 1.87  07-Aug-2020  rillig make(1): enable test phony-end

This at least documents the actual behavior. Not sure what the expected
behavior is and why.
 1.86  07-Aug-2020  rillig make(1): enable test for .IMPSRC

Having this test disabled was very confusing since its content didn't
match the actual behavior, and this also differs from the GNU make
behavior.
 1.85  06-Aug-2020  sjg Use -r for recursive
 1.84  06-Aug-2020  rillig make(1): add test for unintended "recursive variable" in -dL mode
 1.83  03-Aug-2020  rillig make(1): in lint mode, disallow dynamic variable names in :@ modifier

This is an extremely obscure feature that hopefully nobody ever
considered using.
 1.82  02-Aug-2020  rillig make(1): set more predictable environment for counter test

Having an environment variable or a global variable called "C" would
change the debug output since an existing variable is not set to the
empty value first. See parse.c, keyword "!Var_Exists".
 1.81  02-Aug-2020  rillig make(1): add test for nested VAR_SUBST assignments
 1.80  01-Aug-2020  rillig make(1): fix error message when regcomp fails

If regcomp fails, it's not the regex substitution that is erroneous, but
the regex compilation.
 1.79  01-Aug-2020  rillig make(1): improve output grouping in varmod-edge test

The generated error messages are now closer to the test cases that
produce them. To keep the expected output stable, the line numbers are
omitted from the .info directives.
 1.78  31-Jul-2020  rillig make(1): automate adding or removing test files
 1.77  31-Jul-2020  rillig make(1): add test for braces in dependency lines

The comment at DirExpandCurly says something about "recursively", but the
nested braces don't work.

The manual page only mentions that "{} may be used", but it keeps quiet
about the exact meaning and also doesn't mention any nesting.
 1.76  29-Jul-2020  rillig make(1): add enough tests to cover the ApplyModifier functions

Only a few return statements are still missing from the code coverage.

In ApplyModifier_Assign, the test for an empty variable name is skipped
for now since it segfaults.

In ApplyModifier_SysV after the second ParseModifierPart, the branch for
the missing delimiter is not reached since this case is already checked
for in the first part of the function. To trigger this branch, a
specially crafted, unrealistic string needs to be created, and that's too
complicated for the moment.
 1.75  28-Jul-2020  rillig make(1): add test for setting .OBJDIR via the command line
 1.74  28-Jul-2020  sjg Use cat if no POSTPROC.${.TARGET:R}

On Linux sed with no flags throws an error.
 1.73  28-Jul-2020  sjg Ensure .CURDIR is correct for tests that run sub-makes
 1.72  28-Jul-2020  rillig make(1): disable archive test

On FreeBSD, running the archive.mk test results in "create-archive is up
to date" and similar messages. The situation is unclear, and the
archive code seems to be practically unused. Furthermore the archive
test doesn't ensure that it is run in ${.PARSEDIR}, so it needs a bit
more work.
 1.71  28-Jul-2020  sjg Use -r for export.mk to minimize interference
 1.70  27-Jul-2020  rillig make(1): add tests for parsing directives like .if and .info
 1.69  27-Jul-2020  rillig make(1): prepare test for .export to show all exported variables

Currently the unit tests don't cover exporting of local variables like
.TARGET or @. To see the effects, all exported variables must be listed
by the tests.
 1.68  27-Jul-2020  rillig make(1): sort flags for test cases alphabetically
 1.67  27-Jul-2020  rillig make(1): add very basic test for archive handling

The whole code in arch.c had been uncovered by tests before.

The code coverage shows that neither the archive name nor the member
contain any $, even though archive.mk looks like it. It could be
necessary to place the variable assignments below the dependency line,
to force late evaluation.
 1.66  26-Jul-2020  rillig make(1): add test for off-by-one error in Var_Parse
 1.65  26-Jul-2020  rillig make(1): add test for appending and exporting a variable
 1.64  25-Jul-2020  rillig make(1): add tests for previously uncovered code
 1.63  09-Jul-2020  sjg Fix typo
 1.62  04-Jul-2020  rillig make(1): ignore system-specific error message from regcomp in test

NetBSD 8 has: repetition-operator operand invalid
Ubuntu has: Invalid preceding regular expression
 1.61  04-Jul-2020  rillig make(1): add post-processing for tests

By defining SED_CMDS.${test}, a test can do additional post-processing
on its output, for example to normalize paths or error messages.

The Makefile for the tests now follows the common style to have all
relevant definitions at the top of the file, and all the implementation
details at the bottom.

By using the += instead of backslashed lists, it becomes easier to
comment out a single test, either temporarily or permanently (such as
the POSIX tests).

The :C modifier has been replaced with the simpler :S, some :S modifiers
have been replaced with simpler := modifiers.

The postprocessing commands have been extracted from the actual command.
This makes editing and commenting them easier.
 1.60  04-Jul-2020  rillig make(1): improve documentation about adding new tests
 1.59  28-Jun-2020  rillig make(1): demonstrate bug when evaluating conditions
 1.58  17-May-2020  rillig usr.bin/make: demonstrate actual behavior of .INCLUDEDFROMFILE
 1.57  10-May-2020  rillig usr.bin/make: sort test cases alphabetically
 1.56  10-May-2020  rillig usr.bin/make: add tests for surprising dollar removal
 1.55  29-Apr-2020  rillig usr.bin/make: add test case for lazy conditions
 1.54  30-Nov-2019  rillig Add unit tests for variable modifiers like :M and :N
 1.53  24-May-2018  christos branches: 1.53.2;
unit test for $ underquoting in :Q modifier
 1.52  05-May-2015  sjg branches: 1.52.8; 1.52.14;
When evaluating condtionals from .if we want to require
that the lhs is a variable reference, a number or a quoted string.
This helps avoid subtle bugs caused by typos.

When conditionals are being evaluated during variable expansion
we cannot be as strict becuase lhs will already have been expanded.

We therefor pass a boolean to Cond_EvalExpression to tell it how
lhs should be treated.

Add unit-tests/cond2.mk to test the above

Reviewed by: christos, joerg
 1.51  20-Oct-2014  sjg Extract broken tests
 1.50  29-Aug-2014  sjg *.rawout are out-of-date if ${TEST_MAKE} is newer.
 1.49  29-Aug-2014  sjg posix1.mk and suffixes.mk need to cleanup in order to achieve
repeatable results.
posix1.mk's lib.a target still looks dubious.
 1.48  24-Aug-2014  apb Let "make clean" delete the files and dirs created by some of the tests.
 1.47  24-Aug-2014  apb Add tests for backslash escaping in make(1).

Many of these tests fail, and I have populated the escape.exp file with
the results that I expect, not with the results that make(1) actually
produces.

Also update the set lists for these tests.
 1.46  23-Aug-2014  christos Instead of adding the test in 48095 I added the tests in 48096. Add the
missing test now.
 1.45  23-Aug-2014  christos Convert test in PR/49085
 1.44  21-Aug-2014  apb Add a .PATH, so you can be in any other directory and run
"make -f /path/to/this/Makefile sometest.out"
and have it create sometest.out in your current directory.
 1.43  21-Aug-2014  apb Add varmisc.mk, for miscellaneous variable tests.

For now, the only test is copied from
src/tests/usr.bin/make/d_unmatchedvarparen.mk. This was
the only test in src/tests/usr.bin/make that was not also in
src/usr.bin/make/unit-tests.

XXX: src/tests/usr.bin/mk should be changed to reach over to
src/usr.bin/make/unit-tests, instead of keeping out of date copies
of the tests.
 1.42  21-Aug-2014  apb Give each group of tests its own output file.

* Rename each sub-makefile to *.mk;
* Add a *.exp file of expected output for each sub-makefile;
* Remove test.exp, which is replaced by all the other *.exp files.
* Use suffix rules to generate *.rawout and *.out files for
each test case.
* Rewrite the test and accept targets to adapt to the new way.

The old (now removed) test.exp file is almost identical to the
concatenation (in the correct order) of all the new *.exp files. There
are expected differences in makefile names embedded in the output, and
the new "exit status" lines. Some old "*** Error code 1 (ignored)"
lines are also removed (replaced by new "exit status 1" lines).
 1.41  21-Aug-2014  apb Stricter sed expression for matching an optional PID in square brackets.
 1.40  21-Aug-2014  apb When sanitising test output, replace "${TEST_MAKE}" with "make".
This string may appear in error messages that end up in the output.
 1.39  20-Aug-2014  apb Add tests for VAR != command.
 1.38  28-Aug-2013  sjg Fix parsing of SUNSHCMD style :sh = assigments.

Submitted by: Will Andrews at freebsd.org
 1.37  16-Jul-2013  sjg Fix unit-tests to cope with recent change to progname
 1.36  22-Mar-2013  sjg Add unit-test for export-env and gmake export
 1.35  09-Nov-2012  sjg Add a unit-test for .ORDER
 1.34  19-Jun-2012  sjg branches: 1.34.2;
Add unit-tests for .for
 1.33  29-Sep-2011  sjg branches: 1.33.2;
Make sure .PHONY targets get .TARGET set.
This wasn't happening when a dependency of .END.
Set .PREFIX too and add a unit-test.

PR: 43534
 1.32  07-Apr-2011  joerg Add the :hash modifier to compute a 32bit hash of an variable.
This uses MurmurHash3 to get a reasonable collission-free hash with
small code. The result is endian neutral.
 1.31  06-Mar-2011  sjg Add unit tests to check that exists(some/dir/) works

Also fix handling of sysV substitutions when lhs and variable are empty.

Also that modifiers do not cause errors during conditional tests
when undefined variables should otherwise be ok.
Ie. .if defined(nosuch) && ${nosuch:Mx} != ""
 1.30  10-Dec-2010  sjg fix style
 1.29  10-Dec-2010  sjg Need to export LC_ALL=C too
 1.28  22-Apr-2010  sjg Force LANG=C to ensure sort(1) behaves as expected
 1.27  22-Apr-2010  sjg PR: 42850
Reviewed by:

modmisc: since we apply an exists() test to $paths, be conservative
in what we expect.

Run the unit tests with -r -m / so that we do not fail if there
is no sys.mk present.
 1.26  08-Apr-2010  sjg Process .ERROR the same as .BEGIN, .END etc
so that it cannot be the default target.
Add unit-tests for .info - .error, and .ERROR.
 1.25  19-Nov-2009  sjg Add .unexport - the exact opposite of .export
and .unexport-env which unexport's all previously .export'd globals
as well as clearing environ[].
Allow's sys.mk near total controll.

Reviewed by: apb
 1.24  07-Oct-2009  sjg The parser used to break dependency lines at ';' without regard
for substitution patterns. This (perhaps coupled with the
new handling of .for variables in ${:U<value>...) caused interesting
results for lines like:

.for file in ${LIST}
for-subst: ${file:S;^;${here}/;g}

add a unit-test to keep an eye on this.
 1.23  25-Oct-2008  apb Use ${TOOL_SED} instead if plain sed in Makefiles.
 1.22  31-Mar-2008  sjg If the name passed to Var_Exists contains '$' run it through Var_Subst.
 1.21  05-Oct-2007  sjg Add the ability to .export variables to the environment.
 1.20  11-May-2006  sjg branches: 1.20.12;
Extract the variable modifier logic to a separate function.
This cuts Var_Parse in half! and allows the modifier logic to
be used recursively - when getting modifiers via variables.

Add new unit-test, to check that certain error cases are handled
correctly.
 1.19  26-Feb-2006  apb Make ".WAIT" apply recursively to all children of nodes on the right
hand side of the .WAIT, except when the recursive interpretation would
cause a cycle in the dependency graph.

Discussed in tech-toolchain. Reviewed by christos, sjg.
 1.18  26-Feb-2006  sjg Update man page and add test case for specifying modifiers via variable.
Also allow said variable to appear anywhere in the modifier list.
 1.17  13-Feb-2006  apb Escape dots in sed regexps. Without this patch, and without
an OBJDIR, "make test" in usr.bin/make fails.
 1.16  15-Jun-2005  rpaulo Add a unit-test for comments.
Addition to PR bin/17732.
Approved by sjg and christos.
 1.15  01-Jun-2005  sjg Add :Ox for random ordering, based on patch from
Mike M. Volokhov <mishka@apk.od.ua>
 1.14  30-Jun-2004  jmc branches: 1.14.2;
Add missing NetBSD ID tags
 1.13  07-May-2004  sjg Remove use of sh -e when running in compat mode.
Its not posix compliant and serves very little purpose.
With this change compat and jobs modes are consistent wrt how
they treat each line of a script.

Add support for the '+' command line prefix as required by posix.
Lines prefixed with '+' are executed even when -n is given.
[Actually posix says they should also be done for -q and -t]

PR:
Reviewed by: jmc
 1.12  08-Apr-2004  sjg Add check for extraneous .else's - based on patch from Arne H. Juul
in PR/24420.
Add a unit-test for conditionals.
Also in the unit-test makefile strip any .CURDIR in output.

PR: 24420
 1.11  20-Feb-2004  sjg branches: 1.11.2;
Fix :M so that modifiers in nested variables don't terminate parsing
early. Add a unit-test to verify it works and keeps working.
Re-jig the unit-tests so that all are sub makefiles.
 1.10  19-Jan-2004  sjg ternary is .PHONY
 1.9  09-Jan-2004  sjg Add unit test for ternary modifier
 1.8  27-Sep-2003  sjg Implement :[] modifier to allow picking a range of words out of a variable.
Also :tW and a W flag to :C and :S to allow treating value as a single word.
Add unit tests for the above, and fix some corner cases.
Based on patches supplied by Alan Barrett <apb@cequrux.com>
 1.7  08-Aug-2003  sjg Hook make unit-tests into regress
 1.6  01-Aug-2003  sjg Avoid \a as a test of invalid separator, since if we make ksh the default
shell, we get different results. \x seems safe.
 1.5  31-Jul-2003  sjg Test behavior of commandline variable assignments.
 1.4  29-Jul-2003  sjg First bug picked up by the unit tests - delim wasn't always initialized.
Avoid putting '\' in test case script lines since shell's like that on
SunOS insist on interpreting them.
 1.3  29-Jul-2003  sjg Use single quotes to protect everything from shell's that might
alter the results.
 1.2  29-Jul-2003  sjg Regardless of the name of TEST_MAKE, force it to make in test.exp
to avoid needless failures.
 1.1  28-Jul-2003  sjg Fix parsing bug for :ts - patch from Alan Barrett <apb@cequrux.com>
Also add simple unit-test jig (regress/usr.bin/make will use it too)
but having it local here makes inclusion in bmake simpler.
 1.11.2.2  10-May-2004  tron Pull up revision 1.13 (requested by sjg in ticket #282):
Remove use of sh -e when running in compat mode.
Its not posix compliant and serves very little purpose.
With this change compat and jobs modes are consistent wrt how
they treat each line of a script.
Add support for the '+' command line prefix as required by posix.
Lines prefixed with '+' are executed even when -n is given.
[Actually posix says they should also be done for -q and -t]
PR:
Reviewed by: jmc
 1.11.2.1  10-May-2004  tron Pull up revision 1.12 (requested by sjg in ticket #282):
Add check for extraneous .else's - based on patch from Arne H. Juul
in PR/24420.
Add a unit-test for conditionals.
Also in the unit-test makefile strip any .CURDIR in output.
PR: 24420
 1.14.2.3  02-Jul-2005  tron Pull up revision 1.16 (requested by rpaulo in ticket #491):
Add a unit-test for comments.
Addition to PR bin/17732.
Approved by sjg and christos.
 1.14.2.2  16-Jun-2005  tron Backout ticket 458, it causes PR toolchain/30536.
 1.14.2.1  16-Jun-2005  tron Pull up revision 1.16 (requested by rpaulo in ticket #458):
Add a unit-test for comments.
Addition to PR bin/17732.
Approved by sjg and christos.
 1.20.12.1  06-Nov-2007  matt sync with HEAD
 1.33.2.3  22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.33.2.2  16-Jan-2013  yamt sync with (a bit old) head
 1.33.2.1  30-Oct-2012  yamt sync with head
 1.34.2.3  20-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.34.2.2  23-Jun-2013  tls resync from head
 1.34.2.1  20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.52.14.1  25-Jun-2018  pgoyette Sync with HEAD
 1.52.8.1  09-Jun-2018  martin Pull up following revision(s) (requested by christos in ticket #870):

sys/lib/libgnuefi/Makefile.inc: revision 1.5
external/mit/xorg/xorg-pkg-ver.mk: revision 1.9
sys/arch/i386/stand/lib/Makefile.inc: revision 1.17
sys/compat/common/Makefile.inc: revision 1.25
usr.bin/make/unit-tests/varquote.exp: revision 1.1, 1.2
sys/lib/libsa/Makefile.inc: revision 1.24
usr.bin/make/var.c: revision 1.220
sys/arch/ia64/stand/efi/libefi/Makefile.inc: revision 1.5
sys/arch/hppa/spmath/Makefile.inc: revision 1.10
usr.bin/make/unit-tests/varquote.mk: revision 1.1, 1.2
usr.bin/make/unit-tests/Makefile: revision 1.53
sys/lib/libkern/Makefile.inc: revision 1.45
sys/arch/acorn32/stand/lib/Makefile.inc: revision 1.4
sys/arch/m68k/fpsp/Makefile.inc: revision 1.21
usr.bin/make/make.1: revision 1.273
sys/lib/libz/Makefile.inc: revision 1.17

- Introduce :q modifier for make variables and make it double escape $'s so
that passing variables to recursive makes with :q works as expected.
- Adjust makefiles that use recursive make to use :q

Discussed on tech-toolchain@

XXX: pullup 8
 1.53.2.1  08-Apr-2020  martin Merge changes from current as of 20200406
 1.277.2.1  31-May-2021  cjep sync with head
 1.347.2.1  02-Aug-2025  perseant Sync with HEAD

RSS XML Feed