make: unit-tests use .OBJDIR when appropriate

Always use .OBJDIR to refer to the directory make is running in.
Also, add some final filtering to cleanup references to /usr/obj

Reviewed by: rillig

make: add :sh1 to run command on first reference only

There are several ways to assign the output of a command to
a variable, two common ones:

CMD= command
VAR!= ${CMD}

is evaluated when the above line is read, so VAR is set
even if it is never used.

VAR = ${CMD:sh}

is only evaluated when referenced, but command will be run for
each reference.

We now add :sh1 which will only run command on first reference.
The output will be cached in .MAKE.SH1.VAR which will be used for
subsequent references.

Reviewed by: rillig

Use printf rather than echo -n

sync-mi should do .inc files too?

make: ensure output captured in meta file is accurate

In CollectOutput() a newline before the end of job->outBuf
results in a short write to stdout, but meta_job_output()
was recording the entire string in the meta file, so on
next call the left overs would be duplicated in the meta file.

Pass a length to meta_job_output() if it should only report part of
the data in meta file.

tests/make: test backslash at the end of a line and in the :M modifier

As described in PR bin/46139 by David A. Holland.

tests/make: test missing support for $^

Test provided by Diogo Santos via PR bin/59018; extended by me for .WAIT
and implicit rules.

tests/make: fix race condition in debug output in parallel mode

Discovered in ATF run on evbearmv7hf.

make: add on-demand inter-process stack traces

When debugging a build that heavily uses nested calls to sub-makes, the
chain of calls is useful to quickly assess the situation. If explicitly
requested, include the actions from any parent processes in the stack
traces.

Reviewed by: sjg

Fix replacing ${TEST_MAKE} with make

make: use a common style for unexpected error messages

In enomem, report the actual error instead of a fixed ENOMEM.

make: remove empty line from diagnostic, add tests for output handling

The warning for the invalid internal option "-J" does not need a
trailing newline, as that is provided by Parse_Error already.

Having this newline in the output demonstrated a previously unknown bug
in handling the output of child commands in parallel mode. There, empty
lines are randomly discarded, depending on the chunks in which the
output from the child process is copied to make's stdout. See
job-output.mk for a demonstration.

The test for the Korn shell did not run the Korn shell in error handling
mode, as the error handling mode only affects the parallel mode but the
test didn't switch to that mode.

After fixing the shell-ksh.mk test, the single '"' in the output looked
suspicious, and indeed, the shell-csh.mk test had the same problem of
filtering more characters from the output than intended.

make: unify debug messages for parallel mode

Always print the process status in hex, with a leading "0x", to avoid
confusion for status 127f, which looks suspiciously decimal.

Prefix all process IDs with "pid", to be able to filter them easily.

Omit trailing whitespace when printing a command.

make: add end marker for -dg1, -dg2 and -dg3 debug log

The marker helps distinguish the output from sub-makes from the output
of the primary make.

make: use uniform debug log messages for the token pool

make: group the code for handling the job token pool

tests/make: add tests for POSIX mode

make/unit-tests: Fix expected results for meta-ignore tests

make: use '"filename" line 123' for locations instead of 'filename:123'

The format 'filename:123' is commonly used for identifying a location in
a file. Text editors recognize it and allow quick navigation through it.
The previous format was specific to make and would have required custom
support in editors.

The new format was already used in stack traces, except for the first
line. Now all lines use the same format.

make/unit-tests: avoid failure on NFS

The 'rm -rf ${TMPDIR}' at the end of the 'test' target, fails
when building on NFS because the temp file used for the target script
keeps the directory from being empty.

Move it to a separate target attached to '.END'

Reviewed by: rillig

tests/make: test expressions based on undefined variables

When an undefined variable is used in a conditional, make complains about
"Malformed conditional", which is wrong since the conditional is
well-formed but errors out during evaluation.

In order to fix this wrong error message and preserve the remaining
behavior, cover the places in which an expression is evaluated with
undefined expressions being an error (VARE_EVAL_DEFINED), but with
neither producing an error message nor handling errors. This combination
results in the same behavior as evaluating the expressions in the mode
that allows undefined variables to be used in expressions.

tests/make: speed up running the tests

Only 4 of the 389 tests need to postprocess their output, so spare the
extra 'cat' call in the majority of cases.

make: in the debug log, always qualify line number with filename

Previously, to find out the filename corresponding to a line number from
a 'Parsing' line, it was necessary to find the preceding
'SetFilenameVars' line, which was unnecessarily time-consuming and
distracting.

The 'filename:line' format matches the one used in PrintStackTrace, and
it differs from the one used in PrintLocation, as the former format is
more common in other tools.

Most of the affected unit tests only use the 'Parsing' lines to give
additional context to their debug log, so no structural changes there.

tests/make: clean up private temporary directory after running tests

Fixes PR bin/58569.

make: don't track hash table chain lengths during lookup

The chain lengths are only used for debugging purposes, so avoid the
extra cost at each lookup. Instead, calculate the maximum chain length
only when it is actually requested in -dh mode.

The reported number changes slightly: Before, it was the length of the
chain that was actually traversed to find an entry, up to that entry,
now it is the length of the largest chain in the table, no matter if it
was actually accessed or not.

make: on error, print the targets to be made

This helps to understand situations with several nested sub-makes in
varying directories.

make: add :tc to capitalize first letter of each word

This is very hard to do without :tc

Reviewed by: rillig

branches: 1.347.2;
Run check-expect.lua in .CURDIR

tests/make: require all expected messages to be listed in the .mk files

make: in -DCLEANUP mode, free variables and their values

The variables in the 3 scopes must be freed before the scopes themselves
are freed by Targ_End.

The test opt-m-include-dir creates a directory of the form '*.tmp', thus
it must be removed before attempting to only remove regular files of
this name.

POSIX requires setenv to copy the passed name and value, so there is no
need to keep that memory allocated any longer.

Subst DEFSYSPATH in deptgt-phony

make: provide more context information for parse/evaluate errors

make: unit-tests handle TEST_MAKE:T != make

We need to allow for ${TEST_MAKE:T}[1-9]: etc when
TEST_MAKE is not 'make'

Add .MAKE.JOBS.C to indicate wether -jC is supported

tests/make: extend tests for the ':mtime' modifier

make: allow targets to be used as multiple-inclusion guards

This style is used by FreeBSD, among others.

tests/make: add test for multiple-inclusion guards

tests/make: clean up tests for the ':M' and ':S' modifiers

tests/make: clean up comments, extend a few tests

tests/make: clean up tests for .for loops

Most of the tests from forloop.mk were already in directive-for.mk.

make: add :mtime to provide mtime of file

The value of the variable is passed to stat(2)
and st_mtime is new value.
An optional arg can be used if stat(2) fails, otherwise
the current time is used.

See varmod-mtime.mk for usage examples.

make: meta-ignore tests will not work if TMPDIR is /tmp

Since /tmp and /var/tmp are always part of .MAKE.META.IGNORE_PATHS
the varname-dot-make-meta-ignore_* tests will not work
if TMPDIR is either (or subdir) of those.

Add unit-tests for .MAKE.META.IGNORE_*

These unit-tests should only be run if TEST_MAKE
has a valid .MAKE.PATH_FILEMON to indicate that filemon is
supported.

All three tests use the same meta-ignore.inc
which runs three sub-makes

one to initialize the target .meta file
two to ignore a file using .MAKE.META.IGNORE_{PATHS,PATTERNS,FILTER}
three to verify that without .MAKE.META.IGNORE_* the target is out of date.

Fix the order of _SED_CMDS to avoid errors when ${MAKE:T} appears in
.OBJDIR

make: .SYSPATH: to add dirs to sysIncPath

.SYSPATH: with no sources will clear sysIncPath
otherwise sources are added

Reviewed by: rillig

make: .[NO]READONLY for control of read-only variables

Reviewed by: rillig

tests/make: rename files that are not test cases

tests/make: add tests for whitespace in leading '@+-'

tests/make: fix test for option '-e' (broken since 2022-01-23)

tests/make: be more specific about the egrep compatibility

Handle deprecation of egrep

Only update distrib/sets/lists/tests/mi with valid files

sync-mi: use CVS/Entries to ensure we only update
distrib/sets/lists/tests/mi with files known to CVS.

Reviewed by: rillig

Enable make/unit-tests/directive-for-break

Also ensure MAKE_VERSION is ignored.

make: add .break to terminate .for loop early

When .break is encountered within a .for loop
it causes immediate termination.

Outside of a .for loop .break causes a parse error.

Reviewed by: christos

tests/make: fix canonicalization for single-digit day of month

make: prevent future out-of-bounds errors when parsing expressions

A modifier in an expression ends not only at the next ':' or at the
closing '}' or ')', but also at the end of the string.

Previously, testing for the end of the string had been done separately,
which was error-prone since 2006-05-11, when indirect modifiers were
introduced. Since then, it was possible that the string terminator '\0'
was accidentally skipped in cases where the loop condition only tested
for the ending character. When parsing indirect modifiers, the ending
character is indeed '\0', but when parsing direct modifiers, it is '}'
or ')'.

A welcome side effect is that in the case of unclosed expressions such
as '${VAR:Modifier', the amount of error messages is reduced from 2 or 3
to only 1. The removed error messages were wrong and thus confusing
anyway.

Pass MAKEOBJDIR to opt-m-include-dir

Prevent makeing opt-m-include-dir.tmp in src tree.

Reviewed by: rillig

tests/make: reword comment in Makefile to be more specific

Increase the resource limit so these tests pass on hppa.

LGTM from Roland

tests/make: document and demonstrate .for i containing .if empty(i)

PR bin/43821 describes the inconsistency that in a '.for i' loop, the
condition '.if ${i:M*.c}' works since 2009 while the seemingly
equivalent condition '.if !empty(i:M*.c)' does not access the variable
'i' from the .for loop but instead the global 'i'.

Resolving this situation in a backwards-compatible and non-surprising
way is hard, as make has grown several features during the last 20 years
that interact in various edge cases. For now, document the most obvious
pitfalls.

tests/make: remove test varquote

The test varmod-quote-dollar covers the same topic.

tests/make: migrate cond1 test to other, more specific tests

The tests in cond1 were a mixture of "everything related to conditions",
and the test cases were heavily dependent on each other, which made them
hard to understand. Move each test case to its corresponding
special-purpose test.

make: allow to randomize build order of targets

In complex dependency structures, when a build fails, a probable cause
is a missing dependency declaration between some files. In compat mode,
the build order is deterministic, in jobs mode, it is somewhat
deterministic. To explore more edge cases, add the line ".MAKE.MODE +=
randomize-targets" somewhere in the makefile.

Fixes PR bin/45226 by riastradh. Reviewed by christos.

make: only switch to POSIX mode if '.POSIX:' is the first line

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html
says that in order to make a makefile POSIX-conforming, its first
non-comment line must be the special dependency line '.POSIX:' without
any source dependencies.

Previously, make switched to POSIX mode even if such a line occurred
anywhere else, which was allowed by POSIX but was deep in the
"unspecified behavior" area. For NetBSD make, there is no big
difference since it doesn't ship any <posix.mk> file, this change mainly
affects the bmake distribution.

Previously, makefiles that contain '.POSIX:' somewhere in the middle
could fail due to <posix.mk> resetting .SUFFIXES, among other things.

Suggested by Simon J. Gerraty, who also reviewed an earlier version of
this change.

make: fix crash on .undef of an environment variable (since 2020-10-06)

tests/make: suppress -DCLEANUP output in test deptgt-phony

tests/make: document the history of bugs in '-k' mode

Reported in PR#49720 in 2015, fixed independently in compat.c 1.199 from
2020-12-07.

tests/make: demonstrate bug for .SILENT in jobs mode (since 2003)

Reported 2011 in PR#45356.

tests/make: remove redundant 'echo' from variable assignments

Before main.c 1.231 from 2014-09-09, a variable assignment using the
operator '!=' generated a warning "Couldn't read shell's output" if the
output of the command was empty. The simplest way to suppress this
wrong warning was to add an empty 'echo' to the command. This hack is
no longer needed.

make: fix mistakes, spelling and typos in comments and manual page

No binary change for -DNDEBUG.

make: prefix the warning about read-only .OBJDIR with a colon

For consistency with the other warnings.

tests/make: use more distinctive placeholder for TMPDIR

tests/make: demonstrate combination of .USE with transformation rule

Reported in 2003 in PR toolchain/20993. Linking a transformation rule
with .USE or .USEBEFORE node makes the transformation rule fail.

numEntries can vary

Do not allow /usr/obj to cause failure.

tests/make: migrate modts to varmod-to-separator and explain

tests/make: rename var-class to var-scope

There is no such concept as a "variable class" in make, these tests
focus on the variable scope instead.

tests/make: extend test suite, move old tests to 2020 scheme

The tests from envfirst.mk are now in opt-env.mk.
The tests from modword.mk are now in varmod-select-words.mk.

tests/make: explore edge cases involving .ifmake

tests/make: demonstrate wrong location in diagnostic (since 2018-12-22)

When a target has multiple places where commands are defined, the
diagnostics mixed up the filename in some cases.

tests/make: clean up after test for option '-dF'

When the file for the debug log cannot be opened, make exits
immediately. This doesn't give the test a chance to clean up the
temporary log file.

Instead of trying to treat a regular file as a directory and create a
file in it, assume that the directory /nonexistent-$uuid is actually
nonexistent. This leads to the same kind of error message, independent
of strerror(3).

tests/make: add placeholders for .PARSEDIR and .INCLUDEDFROMDIR

When run via 'cd usr.bin/make/unit-tests && make test', the tests are in
the current directory. When run via ATF, the tests are in
/usr/tests/usr.bin/make/unit-tests, while the current directory is a
temporary directory. Allow both variants, plus others that may occur in
the bmake distribution.

tests/make: clean up after test 'opt-debug-file'

tests/make: document inconsistencies between '!=' and '::!='

Found while trying to make the error messages from Cmd_Exec more
detailed.

tests/make: explain the current behavior of the option '--version'

make: remove unreachable code for parsing the dependency operator

At the point where ParseDependencyOp is called, cp is guaranteed to
point to either ':' or '!'.

No functional change.

tests/make: extend tests for parsing makefiles

tests/make: test and document the newly added "variable" .SUFFIXES

make: add back support for jemalloc 100

The ports sun2 and vax default to jemalloc 100, as opposed to the rest
of the ports, which use jemalloc 510. Some developers choose to use
jemalloc 100, so let them run the tests with memory debugging as well.

Suggested by mrg on source-changes-d.

make: fix use-after-free in modifier ':@'

Without memory allocator debugging, the newly added test doesn't show
any obvious failure.

With memory allocator debugging enabled, all make versions since
2016.02.27.16.20.06 crash with a segmentation fault.

tests/make: migrate to jemalloc > 100

When I started working on usr.bin/make in 2020, I used NetBSD 8, which
used jemalloc 100.

After updating to NetBSD 8 to 9.99.x, the debugging options for jemalloc
stopped working since MALLOC_OPTIONS had been replaced with MALLOC_CONF
in 2019.

Enable malloc debugging again, to reliably reproduce the use-after-free
for the variable modifier ':@'.

Replace ${.OBJDIR} with <curdir> for consistent results.

If we replace ${.OBJDIR} with <objdir> some tests produce
different results depending on the value of .OBJDIR

For similar reasons eliminate .OBJDIR and .PATH from output of
varname-empty

Reviewed by: rillig

tests/make: explain where "${var}" in .for loops comes from

tests/make: split tests for the variable modifier ':O'

The tests for parse errors are now in varmod-order, which lets the other
tests focus on the desired behavior of the modifiers.

Add :On for numeric sort

Reviewed by: christos rillig

Allow BROKEN_TESTS to cause TESTS to be skipped.

Some tests simply do not work in some environments.
Eg. shell-ksh on macos/arm64
Allow local site to set BROKEN_TESTS to skip those they know
will not work.

Reviewed by: rillig

tests/make: demonstrate wrong error handling in jobs mode

Do not trust that /nonexistent does not exist

Use /nonexistent.${.MAKE.PID} to avoid failure when
/nonexistent actually exists.

branches: 1.277.2;
tests/make: test the combination of -de with -j1

The test cases are the same as in opt-debug-errors.mk. The output
differs in several details though.

Even though the option '-k' is given (which is the default for any tests
that don't override it in unit-tests/Makefile), there is no message
"(continuing)" anywhere.

The failed target is printed twice. Once before the failed commands,
once after. This redundancy is not necessary and may be removed in a
follow-up commit.

The printed commands are in their unexpanded form, which may or may not
be more helpful than the expanded and space-normalized form of compat
mode. Either way, this is an unnecessary inconsistency between compat
mode and jobs mode.

In jobs mode, the message "make: stopped in $dir" is printed for each
failure, which is helpful since each of the jobs may have started in a
separate directory.

tests/make: demonstrate handling of null bytes

tests/make: split test for modifier ':@' into separate files

The file varmod-loop.mk has grown too large to be single-purpose, plus
it combined parse-time and run-time tests. This has the downside that
as soon as a parse-time test results in an error, the run-time tests are
not run anymore.

make: use C99 bool type instead of defining its own

No functional change.

tests/make: add test for short-circuit evaluation of modifiers

make: demonstrate wrong expression evaluation in -dL mode

make: add test for the variable modifier ':sh'

make: fix TMPDIR in unit-tests

Combining the assignment operator ':=' with the variable modifier ':U'
on the same variable does not work as intended. At the point where the
':U' is evaluated, the variable is guaranteed to be defined because the
code in VarAssign_EvalSubst says so.

Avoid test failures when more than one user run them.

Use a private TMPDIR
a subdir of .OBJDIR if using it
otherwise /tmp/uid${.MAKE.UID}

make: remove .PARSEDIR from test output of varname-empty

The .PARSEDIR has nothing to do with .CURDIR. This wrong assumption
made the NetBSD test suite fail since the actual .PARSEDIR was not
replaced with "<curdir>".

make: in the test varname-empty, keep more debug output

Filtering out the interesting lines based on the very specific pattern
"Var_Set" was surprising.

make: demonstrate that neither -dg2 nor -dg3 produces debug output

make: add test for reading from make's stdin

Just out of curiosity whether that really works. It does.

make(1): add test for combining the options -j, -n, -t

This is a preparation for refactoring the complicated condition in
JobStart.

make(1): demonstrate unnecessary creation of empty files in jobs mode

make(1): fix unit tests for dash and bash 5.1

make(1): normalize shell error messages in tests

Newly supported is bash, which outputs "bash: line 123".

make(1): add test for error handling and expansion in .for loops

make(1): add test for .endfor without corresponding .for

make(1): add test that explains how variables are exported

Exporting the variables at the right time and with the correct values is
a subtle issue. The current implementation carefully marks variables as
ready to be exported, then exports them and at the same time tries to
export as few variables as possible, to avoid memory leaks. This test
describes and explains how all this works in detail.

This test also justifies that the call to Var_ReexportVars happens in
the make process itself, not in the child processes, no matter whether
these are created with vfork or (only theoretically) with plain fork.
This has changed in compat.c 1.217, job.c 1.390 and main.c 1.504 from
2020-12-27.

make(1): add test for expansion errors in jobs mode

Since compat mode and jobs mode are implemented separately and vary in
lots of small details, each of them needs to be tested on its own.

make(1): align tests for :localtime with those for :gmtime

These tests had been almost the same before, now they are as similar as
possible again.

make(1): move tests for :gmtime to parse time

It's easier to have both the expressions and the expected values in a
single file. This also allows for flexible handling of multiple
acceptable outputs, in this case for 32-bit time_t.

Set default for .MAKE.OS once

make(1): include line numbers in output of test varmod-edge.mk

Suppressing the line numbers does not provide much benefit, given that
the test file doesn't change often.

make(1): add test for null byte in .for loop body

make(1): demonstrate wrong line numbers in .for loops, since 2007-01-01

make(1): add test for parsing the end of a condition

make(1): demonstrate wrong error handling in compat mode

Prune job debug output that may appear out of order.

A race between child and parent means that we cannot
guarantee whether all child output is seen before we call
JobClosePipes, thus intervening debug output can appear
before or after the last child output.

Ensure we have a value for .MAKE.UID

make(1): add test for misspelled directives

This test allows the other directive-* tests to focus on the purpose of
the individual directive, allowing these tests to continue after
parsing, without errors.

make(1): add test for a shell with error control

None of the predefined shells has error control, and the corresponding
code had not been covered by the existing unit tests.

make(1): test all combinations of switches for running commands

The code in JobPrintSpecials is rather complicated and contains
surprising interaction between some of the switches.

To see the exact effects of the switches, record the current state and
its output, to prevent accidental breakage during the upcoming
refactorings.

make(1): remove noise from test output of opt-jobs-no-action

make(1): add test for the -n option combined with RunFlags

make(1): add test for combining -j1 with -n

make(1): normalize output of test sh-dots for non-native mode

make(1): add tests and tutorial for the ?= assignment operator

make(1): add test for wrong exit status 0 after failed targets with -k

make(1): add test for .END after failed main node

Use .MAKE.OS if we can

make(1): use fixed format for debug output of the directory cache

The previous output format had a %-20s conversion specifier. This
produced different output depending on the length of the pathname, which
was too difficult to normalize. By moving the directory name to the
end, it is no longer necessary to fill up any space, and the numbers are
always aligned properly.

As a result, 3 of the unit tests no longer need any special
postprocessing of their output.

make(1): add test and tutorial for indirect modifiers

make(1): add test for the special .MAKEFLAGS variable

make(1): add tests for suppressing "stopped in"

These tests demonstrate the unwanted behavior described in PR bin/55578
and PR bin/55832.

make(1): fix tests for :gmtime and :localtime for 32-bit time_t

https://bugs.freebsd.org/251465

Add some tests for meta mode

make(1): clean up memory management for CachedDirs

Previously, the reference count for a newly created CacheDir had been
set to 1 in CacheNewDir. This was wrong because at that point, the
object had not been referenced by any nonlocal variable. The reference
count is no longer incremented at this point.

All callers of CacheNewDir either append the newly created CachedDir to
a SearchPath via Lst_Append and CachedDir_Ref, or they assign it to a
global variable via CachedDir_Assign.

Since the reference count is no longer wrongly incremented, it does not
need to be decremented more than necessary in Dir_End. To keep the code
simple and maintainable, all assignments to global variables are now
handled by CachedDir_Assign. Adding a CachedDir to a list is still done
manually via Lst_Append, and the corresponding code for decrementing is
in SearchPath_Clean and SearchPath_Free. These details may be cleaned
up in a follow-up commit.

As a result, when OpenDirs_Done is called in the unit tests, the list of
open directories is empty. It had been non-empty in a single unit test
before (dep-wildcards.mk), as a result of calling Dir_Expand.

The additional debug logging for the reference counting is not enabled
by default since it contains memory addresses, which makes the output
dependent on the memory allocator.

The function CachedDir_Destroy has been merged into CachedDir_Undef,
which had only been used in Dir_End before. The new name emphasizes
that it corresponds to CachedDir_Ref.

make(1): add debug logging for OpenDirs_Done

make(1): add debug logging for reference counting of CachedDir

Add .MAKE.UID and .MAKE.GID

access(2) does not work for root, so skip objdir-writable

make(1): add test for bug in error handling of .BEGIN in -k mode

make(1): add test for error in dependency of .END node in -k mode

make(1): add test for missing "Stop" after error in .END

make(1): make output of test opt-debug-graph1 consistent

The output from the directory cache made the regular NetBSD build fail
because the pathname to the working directory differs, thus affecting
the spacing.

This time, document why the directory cache needs to be excluded from
the output.

make(1): test that .PHONY targets are not resolved using suffix rules

make(1): add test for debug output from transformation rules

make(1): add another unit test for suffix handling

make(1): document the selection of the main target

make(1): make configuration of export tests simpler

The configuration section of unit-tests/Makefile is already complicated
enough to read, due to the excessive use of regular expressions.
Therefore, to keep the structuring elements at a minimum, inline the
.for loop.

make(1): add more output to test for -dg1

Previously, the variables section had been omitted. This was because
the variables had been output in hashcode order until 2020-10-18, and
because some of the variable values are specific to the test environment
or the individual run (MACHINE_ARCH, MAKE.PPID).

make(1): capture more output from test deptgt-suffixes

This will have an effect with the next commit.

make(1): limit memory usage in tests

There is a suspicious condition in SuffUpdateTarget code that looks wrong
on the first sight. When removing it though, make allocates huge amounts
of memory. To stop freezing the whole machine in this situation, limit
the total memory.

The limit of 200000 has been determined experimentally on NetBSD 8.0
x86_64. With a limit of 100000, make wouldn't even start. 100 MB of
memory is really a lot for such a simple program that according to top(1)
only needs 8 MB. But 200 MB is still better than 5 GB.

Since the Makefile is used on other platforms as well, via the bmake
distribution, and since every operating system has its own list of ulimit
options, make this configurable.

make(1): add test for incomplete transformation rule

make(1): fix test opt-debug-jobs for native shell

Even after substituting the actual shell name with <shell>, the trailing
optional -q option needs to be normalized.

make(1): fix test opt-debug-jobs when compiled with dash as shell

make USER_CPPFLAGS=-DDEFSHELL_CUSTOM='\"/usr/pkg/bin/dash\"'

make(1): remove redundant postprocessing for test var-op-shell

The regular expression from the previous line already matches everything
that the removed line could ever match.

make(1): remove postprocessing for test directive-export-gmake

Since revision 1.3, directive-export-gmake.mk takes care of handling this
case. Removing the "error" lines from the output of the tests only makes
the test harder to understand and may also hide future bugs.

Cleanup - colliding fixes for same issue

Some tests need help with dash

make(1): make test vardebug independent from the shell

make(1): fix test output of var-op-shell for dash

Dash is helpful and outputs the line number in case of an error. The
other shells don't do this. Accept both variants.

make(1): add test for self-referencing suffix rule

Just to ensure that make doesn't run into an endless loop.

make(1): fix typo in test for variable with empty name

make(1): update and add comments in tests

As a result of the new comments, some line numbers have changed in the
output of the tests. No other changes.

make(1): use progname instead of hard-coded 'make' in warning

This only affects the warning that chdir to objdir failed. In sub-makes
the progname includes the [n], allowing to narrow down the actual cause
of the problem.

make(1): add test for the -t option in jobs mode

Pass a writable flag to Main_SetObjdir to control writable check

For curdir and an explicit .OBJDIR target, we allow for
the directory to be read-only.
During InitObjdir we otherwise default to requiring objdir to be
writable - this can be controlled by env variable
MAKE_OBJDIR_CHECK_WRITABLE

Add unit-tests/objdir-writable

Reviewed by: christos rillig

make(1): add more tests for parsing .if directives

The details of parsing are quite tricky and not documented in the manual
page. Record the current behavior to be a little safer in future
refactorings.

make(1): add test for the obsolete variable VPATH

make(1): in tests, move command line arguments into the test proper

make(1): move test for != assignments to var-op-shell

Control TZ for depsrc-optional

make(1): in lint mode, only allow '&&' and '||', not '&' and '|'

These variants of the condition operators are neither documented in the
manual page nor are they used in practice.

make(1): add test for expanding variable expressions

make(1): add test for OP_SUBMAKE

make(1): add test for job command flags

make(1): add test for undefined variables in command line arguments

The variable discardUndefined has an implicit negation in its name,
which makes it hard to understand. Plus, most of the time it is true.
It's better to have a flag that is false most of the time and has a
positive name.

On the first attempt of inverting that variable, I stumbled upon
MainParseArgs, which initially leaves discardUndefined == FALSE, and
after handling the dashed options, sets it to TRUE. This would make a
difference when more command line arguments would be added later via the
.MAKEFLAGS special target.

Upon further inspection, the only place where discardUndefined is used
is in VarAssign_EvalSubst in parse.c, and that place is not reachable
from any of the dashed options. Therefore, discardUndefined could
already be set at the very beginning of MainParseArgs or even when
initializing the global variable itself, without any observable
difference.

Not even the ::= variable modifier could do anything about this since it
is not reachable from the dashed command line options as well, and in
addition, it expands its right-hand side in any case, always discarding
undefined variables. Oh, these little inconsistencies everywhere.

make(1): move tests from directives.mk to separate tests

make(1): remove obsolete POSTPROC.varname from unit tests

It was not needed anymore since 2020-10-18.

make(1): add test for parse errors in commands in lint mode (-dL)

The difference to non-lint mode is that the exit status is now 2 instead
of 0.

make(1): remove debugging code from sync-mi

make(1): add test for parse errors in shell commands, compat mode

make(1): handle errors when sync-mi fails because of a syntax error

make(1): move test flags to test varmod-match-escape.mk

make(1): add more tests for the variable modifier :localtime

make(1): add test for recursive variable expressions

make(1): merge the SysV modifier tests into varmod-sysv

make(1): move test flags for varname-dot-shell to the test itself

make(1): move MAKEFLAGS handling in vardebug test to the test itself

make(1): add test for looking up suffixes

make(1): add test for dependencies of the form '%.o: %.c'

make(1): add test case in which .BEGIN depends on .END

Just for fun.

make(1): add test for the '::' dependency operator

make(1): move tests from cond2.mk to varmod-ifelse.mk (fixup)

make(1): add test for .ERROR_CMD in jobs mode

make(1): split test suffixes.mk into simpler, isolated tests

The code in suff.c is already hard to understand, and so were the tests
in suffixes.mk since several independent topics were merged into a
single test.

Splitting this test into a separate test per issue allows to document
the expected and actual behavior in more detail. That's complicated
enough already.

PR bin/49086

In meta mode we do not have to re-run all tests when Makefile is touched

make(1): add test for turning a target into a transformation

make(1): document why the ampersand is not listed in export.exp

Limit output of export tests to POSIX compatible env vars

make(1): remove test misc.mk from Makefile

make(1): fix test opt-debug-jobs on non-native platforms

make(1): revert previous commit

It had accidentally reverted all the work from the past few days.

make(1): fix double-free bug in -DCLEANUP mode (since 2020-10-02)

The bug had been introduced with dir.c 1.155 on 2020-10-02 22:20:25. In
that commit, openDirectories was replaced with a combination of a list
with a hash table, for more efficient lookup by name.

Upon cleanup, OpenDirs_Done is called, which in turn called
Dir_ClearPath. Dir_ClearPath takes full ownership of the given list and
empties it. This was no problem before since afterwards the list was
empty and calling Lst_Free just frees the remaining list pointer.

With OpenDirs, this list was combined with a hash table, and the hash
table contains the list nodes, assuming that the OpenDirs functions have
full ownership of both the list and the hash table. This assumption was
generally correct, except for the one moment during cleanup where full
ownership of the list was passed to Dir_ClearPath, while the hash table
still contained pointers to the (now freed) list nodes. This by itself
was not a problem since the hash table would be freed afterwards. But
as part of Dir_ClearPath, OpenDirs_Remove was called, which looked up
the freed directory by name and now found the freed list node, trying to
free it again. Boom.

Fixed by replacing the call to Dir_ClearPath with code that only frees
the directories, without giving up control over the list.

make(1): add tests for parsing assignments, especially :sh

Luckily nobody uses the :sh variable assignment modifier since its
syntactical variant != is simpler.

make(1): add test demonstrating the Towers of Hanoi puzzle

It's not the primary task of make to handle procedure calls with
parameters, combined with lexical scoping, therefore the code does not
look as straight-forward or clean as in other programming languages. It
feels more like squeezing a programming problem from the imperative
world into the world of declarative dependencies.

A more idiomatic way of implementing this puzzle should be as a
dependency graph since that's both the natural structure of the puzzle
and the primary domain of make. Something like having a main target
"hanoi-5" that depends on intermediate targets of the form
"move-1.2.3.4.5-_._._._._-_._._._._", each representing a single
configuration of the stacks. These targets could be generated
dynamically. A benefit of this implementation would be that the puzzle
could be resumed from an arbitrary configuration, just just from the
initial configuration.

make(1): add tests for the various supported shells

make(1): add test for the -dj debug option

make(1): add tests for parsing and exporting variables

Once again, there are a few surprises deeply hidden inside the edge
cases.

make(1): fix test job-output-long-lines

The test failed on Linux since the test relied on the two child
processes interleaving each other. This is not guaranteed to happen
though.

make(1): add test for merging long lines of job output

make(1): run tests with jemalloc debugging enabled

This protects against very simple memory allocation bugs such as
migrating Lst_ForEachUntil to Lst_ForEach without remembering that
Lst_ForEachUntil can handle the situation where the current list node is
removed from the list, but Lst_ForEach cannot. This happens in
Make_ExpandUse, for example.

make(1): add test for harmless bug in Parse_File

When there is a dependency group at the end of a top-level makefile,
this dependency group is not finished properly. This allows to add
further commands to the targets of this dependency group, which was not
intended.

make(1): add test for make() function in conditions

make(1): add test for deleting the suffixes during parsing

make(1): add test for partial expansion of undefined variables

make(1): remove debug output from counter tests

Now that the tests work as intended, the debug information is no longer
necessary. It was only intended to track down the cause of the
unexpected behavior.

make(1): add variant of the counter test

I had expected that using the ::+= modifier instead of the ::= modifier
would work, since the assignment modifier for COUNTER no longer contains
a reference to itself. But instead of ending up at 4, the counter even
goes up to 6.

make(1): fix assertion failure in -j mode with .END node

There had been two separate global variables for the .END node, and in
parallel mode, only the one in jobs.c was initialized.

The code in JobRun heads over to Compat_Make without calling Compat_Run
first, which left the variable ENDNode uninitialized.

make(1): run tests with absolute filenames as well

The tests varname-dot-parsedir and varname-dot-parsefile had been broken
before if they were run with the "-f $PWD/$test.mk" option. This way of
running the tests is used by FreeBSD.

make(1): fix tests for .PARSEDIR/.PARSEFILE

The output of these tests previously depended upon whether the test
makefile was specified as relative or as absolute path. In NetBSD, it
is specified as relative path, in FreeBSD as absolute path.

Adjust the expected output to hide this difference; in the .rawout files
it is still visible.

make(1): move flags for lint test into the test itself

make(1): add test for undefined expressions in conditions in lint mode

make(1): add tests for numbers in conditions

make(1): add tests for the various .include directives

make(1): fix sync-mi helper target, at least for adding tests

make(1): explain why each test is run in a sub-make

make(1): use consistent spelling for postprocessing

make(1): add test for intentional hash collision for variable names

Hash collisions may slow down make in certain special situations. There
is no point though in maliciously triggering such a situation since
anyone who can inject values into makefiles can easily run shell
commands using the :!cmd! modifier or similar mechanisms. Crafting
variable names just to slow down make is thus not an attack vector.

make(1): move test for -dg1 from opt-debug-g1 to opt-debug-graph1

make(1): add tests for each debug option

make(1): re-enable the archive test

The test had failed in the releng build because it assumed it were run
with .CURDIR == .PARSEDIR. This assumption is true when the tests are
run directly from usr.bin/make, but not when they are run from
tests/usr.bin/make.

make(1): add test for the special variable MAKEFILE

make(1): extend tests for the :hash variable modifier

The previous test vectors didn't contain any hash with a leading zero.
This could have been a simple programming mistake by using %8x instead
of the intended %08x. Using snprintf wouldn't have been possible anyway
since the hex digits are printed in little-endian order, but without
reversing the bits of each digit. Kind of unusual, but doesn't affect
the distribution of the hashes.

make(1): add test for the .for directive

For a long time, I had assumed that the iteration variables of a .for
loop are just normal global variables. This assumption was wrong but
didn't have any consequences.

The iteration variables of a .for loop can just be accessed like global
variables, therefore it is not obvious that they are implemented in a
completely different way.

There are some edge cases in conditions used inside .for loops, in which
the iteration variables cannot be used like normal variables. An
example is brought up in https://gnats.netbsd.org/47888, which observes
that the defined() and empty() functions in conditions only work with
variables but ignore the iteration "variables", simply because these are
not variables but only expressions.

make(1): fix sh-dots test

The exact output depends on the shell. Some shells prefix the error
message with their name, some don't.

make(1): add test for the .ifmake directive

make(1): add tests for .else and .for + .endif

make(1): fix assertion failure for .SUFFIXES in archives

This occurred in the posix1.mk test, even though it is disabled in
unit-tests. But in tests/usr.bin/make it still runs. There, it should
have produced an "expected failure" but crashed instead.

The archive-suffix test is the stripped-down version of the posix1 test.

make(1): fix test opt-debug-g1

Including the directory cache made the test output fragile.

make(1): add test for interrupting a command

make(1): add test for the undocumented .NULL special dependency target

make(1): add tests for the special .INCLUDES and .LIBS variables

make(1): disable the sync-mi convenience target

make(1): add test for the -dg1 option

make(1): add test for + commands with the -n command line option

make(1): add tests for the -i -k -v -V -W options

make(1): add tests for numeric comparisons in .if directives

make(1): fix assertion failure in debug output for read-only variables

make(1): add test for the newly added .SHELL variable

make(1): add test for expanding wildcards in directories

This test covers DirExpandInt.

bash says 'command not found'

make(1): try to set the variable named "" using the command line

Before 2020-08-22, the command line argument '=cmline-plain' triggered
undefined behavior, just like the assignment '=assigned' in the Makefile.

Use TEST_MAKE to run unit-tests

This lets us take advantage of new features like ${.SHELL}

make(1): add test for assigning to the variable named ""

This has been possible at least since 2013, probably much longer.

make(1): add test for deferred variable expansion in dependencies

Supress /bin/sh: in error for sh-dots

make(1): add test for "no target to make"

make(1): add test for the special "..." in shell commands

make(1): add test for combining :: and .USEBEFORE

make(1): enable debug logging for archive test

This test succeeds locally on NetBSD 8.0 but not in the official test
runs on https://releng.netbsd.org/test-results.html. To see the
difference, run the test with full debug information.

This test is commented out in usr.bin/make/unit-tests/Makefile, but that
doesn't stop tests/usr.bin/make/t_make.sh from running it nevertheless,
since over there, all *.mk files are considered tests, be they commented
out or not.

make(1): add test for parsing functions in .if conditions

make(1): add test for the -n option

make(1): add test for the -q option

make(1): move tests for the :Ox modifier into separate file

The test has been extended by ensuring that the shuffled words are still
the same. Comparing two shuffled lists is probabilistic, but comparing
their sorted results is not, therefore that's completely sensible to do.

When writing this test, by coincidence I discovered how to generate the
"Undefined variable" error message. Unfortunately, the error message is
wrong since the variable NUMBERS is defined at that point. In summary,
that error message is shown when it shouldn't, and when it should it is
not shown. Still, I'm glad that I finally found it.

make(1): move tests for the :M modifier into separate files

The test for the different escaping has been adjusted to actually show
the different parsing results in the test output. To do this, it had to
get its own file since it needs the -dv debug flag and specialized
post-processing.

make(1): run tests in an almost empty, controlled environment

Several of the tests use simple variable names that might be influenced
by environment variables of the same name. Especially the tests for the
?= variable assignment operator need an empty environment to start with,
to produce reliable results.

The PATH must be in the base environment since several tests depend on
the usual tools like echo(1), sleep(1), grep(1). Setting the PATH to a
fixed value would have made it impossible to run the tests in a custom
environment that don't have a /bin directory. It's the user's
responsibility to provide a sane PATH.

make(1): force all tests to be run with the -r flag

Without that flag, <sys.mk> is loaded before the tests. The tests are
not intended to either use or even test these rules and definitions,
therefore it is safe to omit this command line option.

If there should ever be tests for POSIX-conformance that need the
builtin rules and definitions, these tests can still ".include <sys.mk>"
at the beginning and be done with it, since the -m command line option
is not touched. The system-default rules and definitions are still
available, they are just not active by default.

Suggested by sjg.

make(1): use consistent formatting for running the tests

With the many newly added tests, having two messages ("testing xzy" and
"postprocessing xyz") made the output too noisy since the words were not
aligned nicely. Use the same formatting as for MAKEVERBOSE=0, and
remove the message for postprocessing. If there should ever be problems
during postprocessing, it's trivial to run "../make TESTS=bad-test -dl"
to find out where the problem is.

make(1): clean up cleaning of test files

The variables CLEANFILES and CLEANDIRS are not supposed to be
user-settable, thus use a simple "=" instead of "+=".

Since CLEANDIRS is always set, there is no point in making the rm
conditional.

make(1): move tests for :S, :C, :@ from modmisc to their separate tests

make(1): add dummies for fine-grained tests, one per single feature

The test names have been derived from the current manual page.

All these tests are dummies right now, and the code from the existing
tests will be moved into the new tests step by step.

This is done to prevent modmisc, escape, varmod-edge and varmisc from
growing without any bounds, and to reduce the side-effects of one test
to the others.

make(1): fix archive test

Even though it is commented out in this Makefile, it is still run by
atf, on purpose. This test is broken on FreeBSD, but on NetBSD it
should still be run.

Without removing the archive first, it could be left over from a
previous test failure and thus be newer than all its members, leading to
${.OODATE} being empty.

make(1): add test for .USE combined with inference rule without commands

Discovered by christos.

make(1): add test for non-obvious .MAKE.EXPORTED edge case

make(1): add test for exporting variables

make(1): document why the suffixes test must stay disabled

make(1): enable test phony-end

This at least documents the actual behavior. Not sure what the expected
behavior is and why.

make(1): enable test for .IMPSRC

Having this test disabled was very confusing since its content didn't
match the actual behavior, and this also differs from the GNU make
behavior.

Use -r for recursive

make(1): add test for unintended "recursive variable" in -dL mode

make(1): in lint mode, disallow dynamic variable names in :@ modifier

This is an extremely obscure feature that hopefully nobody ever
considered using.

make(1): set more predictable environment for counter test

Having an environment variable or a global variable called "C" would
change the debug output since an existing variable is not set to the
empty value first. See parse.c, keyword "!Var_Exists".

make(1): add test for nested VAR_SUBST assignments

make(1): fix error message when regcomp fails

If regcomp fails, it's not the regex substitution that is erroneous, but
the regex compilation.

make(1): improve output grouping in varmod-edge test

The generated error messages are now closer to the test cases that
produce them. To keep the expected output stable, the line numbers are
omitted from the .info directives.

make(1): automate adding or removing test files

make(1): add test for braces in dependency lines

The comment at DirExpandCurly says something about "recursively", but the
nested braces don't work.

The manual page only mentions that "{} may be used", but it keeps quiet
about the exact meaning and also doesn't mention any nesting.

make(1): add enough tests to cover the ApplyModifier functions

Only a few return statements are still missing from the code coverage.

In ApplyModifier_Assign, the test for an empty variable name is skipped
for now since it segfaults.

In ApplyModifier_SysV after the second ParseModifierPart, the branch for
the missing delimiter is not reached since this case is already checked
for in the first part of the function. To trigger this branch, a
specially crafted, unrealistic string needs to be created, and that's too
complicated for the moment.

make(1): add test for setting .OBJDIR via the command line

Use cat if no POSTPROC.${.TARGET:R}

On Linux sed with no flags throws an error.

Ensure .CURDIR is correct for tests that run sub-makes

make(1): disable archive test

On FreeBSD, running the archive.mk test results in "create-archive is up
to date" and similar messages. The situation is unclear, and the
archive code seems to be practically unused. Furthermore the archive
test doesn't ensure that it is run in ${.PARSEDIR}, so it needs a bit
more work.

Use -r for export.mk to minimize interference

make(1): add tests for parsing directives like .if and .info

make(1): prepare test for .export to show all exported variables

Currently the unit tests don't cover exporting of local variables like
.TARGET or @. To see the effects, all exported variables must be listed
by the tests.

make(1): sort flags for test cases alphabetically

make(1): add very basic test for archive handling

The whole code in arch.c had been uncovered by tests before.

The code coverage shows that neither the archive name nor the member
contain any $, even though archive.mk looks like it. It could be
necessary to place the variable assignments below the dependency line,
to force late evaluation.

make(1): add test for off-by-one error in Var_Parse

make(1): add test for appending and exporting a variable

make(1): add tests for previously uncovered code

Fix typo

make(1): ignore system-specific error message from regcomp in test

NetBSD 8 has: repetition-operator operand invalid
Ubuntu has: Invalid preceding regular expression

make(1): add post-processing for tests

By defining SED_CMDS.${test}, a test can do additional post-processing
on its output, for example to normalize paths or error messages.

The Makefile for the tests now follows the common style to have all
relevant definitions at the top of the file, and all the implementation
details at the bottom.

By using the += instead of backslashed lists, it becomes easier to
comment out a single test, either temporarily or permanently (such as
the POSIX tests).

The :C modifier has been replaced with the simpler :S, some :S modifiers
have been replaced with simpler := modifiers.

The postprocessing commands have been extracted from the actual command.
This makes editing and commenting them easier.

make(1): improve documentation about adding new tests

make(1): demonstrate bug when evaluating conditions

usr.bin/make: demonstrate actual behavior of .INCLUDEDFROMFILE

usr.bin/make: sort test cases alphabetically

usr.bin/make: add tests for surprising dollar removal

usr.bin/make: add test case for lazy conditions

Add unit tests for variable modifiers like :M and :N

branches: 1.53.2;
unit test for $ underquoting in :Q modifier

branches: 1.52.8; 1.52.14;
When evaluating condtionals from .if we want to require
that the lhs is a variable reference, a number or a quoted string.
This helps avoid subtle bugs caused by typos.

When conditionals are being evaluated during variable expansion
we cannot be as strict becuase lhs will already have been expanded.

We therefor pass a boolean to Cond_EvalExpression to tell it how
lhs should be treated.

Add unit-tests/cond2.mk to test the above

Reviewed by: christos, joerg

Extract broken tests

*.rawout are out-of-date if ${TEST_MAKE} is newer.

posix1.mk and suffixes.mk need to cleanup in order to achieve
repeatable results.
posix1.mk's lib.a target still looks dubious.

Let "make clean" delete the files and dirs created by some of the tests.

Add tests for backslash escaping in make(1).

Many of these tests fail, and I have populated the escape.exp file with
the results that I expect, not with the results that make(1) actually
produces.

Also update the set lists for these tests.

Instead of adding the test in 48095 I added the tests in 48096. Add the
missing test now.

Convert test in PR/49085

Add a .PATH, so you can be in any other directory and run
"make -f /path/to/this/Makefile sometest.out"
and have it create sometest.out in your current directory.

Add varmisc.mk, for miscellaneous variable tests.

For now, the only test is copied from
src/tests/usr.bin/make/d_unmatchedvarparen.mk. This was
the only test in src/tests/usr.bin/make that was not also in
src/usr.bin/make/unit-tests.

XXX: src/tests/usr.bin/mk should be changed to reach over to
src/usr.bin/make/unit-tests, instead of keeping out of date copies
of the tests.

Give each group of tests its own output file.

* Rename each sub-makefile to *.mk;
* Add a *.exp file of expected output for each sub-makefile;
* Remove test.exp, which is replaced by all the other *.exp files.
* Use suffix rules to generate *.rawout and *.out files for
each test case.
* Rewrite the test and accept targets to adapt to the new way.

The old (now removed) test.exp file is almost identical to the
concatenation (in the correct order) of all the new *.exp files. There
are expected differences in makefile names embedded in the output, and
the new "exit status" lines. Some old "*** Error code 1 (ignored)"
lines are also removed (replaced by new "exit status 1" lines).

Stricter sed expression for matching an optional PID in square brackets.

When sanitising test output, replace "${TEST_MAKE}" with "make".
This string may appear in error messages that end up in the output.

Add tests for VAR != command.

Fix parsing of SUNSHCMD style :sh = assigments.

Submitted by: Will Andrews at freebsd.org

Fix unit-tests to cope with recent change to progname

Add unit-test for export-env and gmake export

Add a unit-test for .ORDER

branches: 1.34.2;
Add unit-tests for .for

branches: 1.33.2;
Make sure .PHONY targets get .TARGET set.
This wasn't happening when a dependency of .END.
Set .PREFIX too and add a unit-test.

PR: 43534

Add the :hash modifier to compute a 32bit hash of an variable.
This uses MurmurHash3 to get a reasonable collission-free hash with
small code. The result is endian neutral.

Add unit tests to check that exists(some/dir/) works

Also fix handling of sysV substitutions when lhs and variable are empty.

Also that modifiers do not cause errors during conditional tests
when undefined variables should otherwise be ok.
Ie. .if defined(nosuch) && ${nosuch:Mx} != ""

fix style

Need to export LC_ALL=C too

Force LANG=C to ensure sort(1) behaves as expected

PR: 42850
Reviewed by:

modmisc: since we apply an exists() test to $paths, be conservative
in what we expect.

Run the unit tests with -r -m / so that we do not fail if there
is no sys.mk present.

Process .ERROR the same as .BEGIN, .END etc
so that it cannot be the default target.
Add unit-tests for .info - .error, and .ERROR.

Add .unexport - the exact opposite of .export
and .unexport-env which unexport's all previously .export'd globals
as well as clearing environ[].
Allow's sys.mk near total controll.

Reviewed by: apb

The parser used to break dependency lines at ';' without regard
for substitution patterns. This (perhaps coupled with the
new handling of .for variables in ${:U<value>...) caused interesting
results for lines like:

.for file in ${LIST}
for-subst: ${file:S;^;${here}/;g}

add a unit-test to keep an eye on this.

Use ${TOOL_SED} instead if plain sed in Makefiles.

If the name passed to Var_Exists contains '$' run it through Var_Subst.

Add the ability to .export variables to the environment.

branches: 1.20.12;
Extract the variable modifier logic to a separate function.
This cuts Var_Parse in half! and allows the modifier logic to
be used recursively - when getting modifiers via variables.

Add new unit-test, to check that certain error cases are handled
correctly.

Make ".WAIT" apply recursively to all children of nodes on the right
hand side of the .WAIT, except when the recursive interpretation would
cause a cycle in the dependency graph.

Discussed in tech-toolchain. Reviewed by christos, sjg.

Update man page and add test case for specifying modifiers via variable.
Also allow said variable to appear anywhere in the modifier list.

Escape dots in sed regexps. Without this patch, and without
an OBJDIR, "make test" in usr.bin/make fails.

Add a unit-test for comments.
Addition to PR bin/17732.
Approved by sjg and christos.

Add :Ox for random ordering, based on patch from
Mike M. Volokhov <mishka@apk.od.ua>

branches: 1.14.2;
Add missing NetBSD ID tags

Remove use of sh -e when running in compat mode.
Its not posix compliant and serves very little purpose.
With this change compat and jobs modes are consistent wrt how
they treat each line of a script.

Add support for the '+' command line prefix as required by posix.
Lines prefixed with '+' are executed even when -n is given.
[Actually posix says they should also be done for -q and -t]

PR:
Reviewed by: jmc

Add check for extraneous .else's - based on patch from Arne H. Juul
in PR/24420.
Add a unit-test for conditionals.
Also in the unit-test makefile strip any .CURDIR in output.

PR: 24420

branches: 1.11.2;
Fix :M so that modifiers in nested variables don't terminate parsing
early. Add a unit-test to verify it works and keeps working.
Re-jig the unit-tests so that all are sub makefiles.

ternary is .PHONY

Add unit test for ternary modifier

Implement :[] modifier to allow picking a range of words out of a variable.
Also :tW and a W flag to :C and :S to allow treating value as a single word.
Add unit tests for the above, and fix some corner cases.
Based on patches supplied by Alan Barrett <apb@cequrux.com>

Hook make unit-tests into regress

Avoid \a as a test of invalid separator, since if we make ksh the default
shell, we get different results. \x seems safe.

Test behavior of commandline variable assignments.

First bug picked up by the unit tests - delim wasn't always initialized.
Avoid putting '\' in test case script lines since shell's like that on
SunOS insist on interpreting them.

Use single quotes to protect everything from shell's that might
alter the results.

Regardless of the name of TEST_MAKE, force it to make in test.exp
to avoid needless failures.

Fix parsing bug for :ts - patch from Alan Barrett <apb@cequrux.com>
Also add simple unit-test jig (regress/usr.bin/make will use it too)
but having it local here makes inclusion in bmake simpler.

make: add :sh1 to run command on first reference only

There are several ways to assign the output of a command to
a variable, two common ones:

CMD= command
VAR!= ${CMD}

is evaluated when the above line is read, so VAR is set
even if it is never used.

VAR = ${CMD:sh}

is only evaluated when referenced, but command will be run for
each reference.

We now add :sh1 which will only run command on first reference.
The output will be cached in .MAKE.SH1.VAR which will be used for
subsequent references.

Reviewed by: rillig

Use printf rather than echo -n

sync-mi should do .inc files too?

make: ensure output captured in meta file is accurate

In CollectOutput() a newline before the end of job->outBuf
results in a short write to stdout, but meta_job_output()
was recording the entire string in the meta file, so on
next call the left overs would be duplicated in the meta file.

Pass a length to meta_job_output() if it should only report part of
the data in meta file.

tests/make: test backslash at the end of a line and in the :M modifier

As described in PR bin/46139 by David A. Holland.

tests/make: test missing support for $^

Test provided by Diogo Santos via PR bin/59018; extended by me for .WAIT
and implicit rules.

tests/make: fix race condition in debug output in parallel mode

Discovered in ATF run on evbearmv7hf.

make: add on-demand inter-process stack traces

When debugging a build that heavily uses nested calls to sub-makes, the
chain of calls is useful to quickly assess the situation. If explicitly
requested, include the actions from any parent processes in the stack
traces.

Reviewed by: sjg

Fix replacing ${TEST_MAKE} with make

make: use a common style for unexpected error messages

In enomem, report the actual error instead of a fixed ENOMEM.

make: remove empty line from diagnostic, add tests for output handling

The warning for the invalid internal option "-J" does not need a
trailing newline, as that is provided by Parse_Error already.

Having this newline in the output demonstrated a previously unknown bug
in handling the output of child commands in parallel mode. There, empty
lines are randomly discarded, depending on the chunks in which the
output from the child process is copied to make's stdout. See
job-output.mk for a demonstration.

The test for the Korn shell did not run the Korn shell in error handling
mode, as the error handling mode only affects the parallel mode but the
test didn't switch to that mode.

After fixing the shell-ksh.mk test, the single '"' in the output looked
suspicious, and indeed, the shell-csh.mk test had the same problem of
filtering more characters from the output than intended.

make: unify debug messages for parallel mode

Always print the process status in hex, with a leading "0x", to avoid
confusion for status 127f, which looks suspiciously decimal.

Prefix all process IDs with "pid", to be able to filter them easily.

Omit trailing whitespace when printing a command.

make: add end marker for -dg1, -dg2 and -dg3 debug log

The marker helps distinguish the output from sub-makes from the output
of the primary make.

make: use uniform debug log messages for the token pool

make: group the code for handling the job token pool

tests/make: add tests for POSIX mode

make/unit-tests: Fix expected results for meta-ignore tests

make: use '"filename" line 123' for locations instead of 'filename:123'

The format 'filename:123' is commonly used for identifying a location in
a file. Text editors recognize it and allow quick navigation through it.
The previous format was specific to make and would have required custom
support in editors.

The new format was already used in stack traces, except for the first
line. Now all lines use the same format.

make/unit-tests: avoid failure on NFS

The 'rm -rf ${TMPDIR}' at the end of the 'test' target, fails
when building on NFS because the temp file used for the target script
keeps the directory from being empty.

Move it to a separate target attached to '.END'

Reviewed by: rillig

tests/make: test expressions based on undefined variables

When an undefined variable is used in a conditional, make complains about
"Malformed conditional", which is wrong since the conditional is
well-formed but errors out during evaluation.

In order to fix this wrong error message and preserve the remaining
behavior, cover the places in which an expression is evaluated with
undefined expressions being an error (VARE_EVAL_DEFINED), but with
neither producing an error message nor handling errors. This combination
results in the same behavior as evaluating the expressions in the mode
that allows undefined variables to be used in expressions.

tests/make: speed up running the tests

Only 4 of the 389 tests need to postprocess their output, so spare the
extra 'cat' call in the majority of cases.

make: in the debug log, always qualify line number with filename

Previously, to find out the filename corresponding to a line number from
a 'Parsing' line, it was necessary to find the preceding
'SetFilenameVars' line, which was unnecessarily time-consuming and
distracting.

The 'filename:line' format matches the one used in PrintStackTrace, and
it differs from the one used in PrintLocation, as the former format is
more common in other tools.

Most of the affected unit tests only use the 'Parsing' lines to give
additional context to their debug log, so no structural changes there.

tests/make: clean up private temporary directory after running tests

Fixes PR bin/58569.

make: don't track hash table chain lengths during lookup

The chain lengths are only used for debugging purposes, so avoid the
extra cost at each lookup. Instead, calculate the maximum chain length
only when it is actually requested in -dh mode.

The reported number changes slightly: Before, it was the length of the
chain that was actually traversed to find an entry, up to that entry,
now it is the length of the largest chain in the table, no matter if it
was actually accessed or not.

make: on error, print the targets to be made

This helps to understand situations with several nested sub-makes in
varying directories.

make: add :tc to capitalize first letter of each word

This is very hard to do without :tc

Reviewed by: rillig

branches: 1.347.2;
Run check-expect.lua in .CURDIR

tests/make: require all expected messages to be listed in the .mk files

make: in -DCLEANUP mode, free variables and their values

The variables in the 3 scopes must be freed before the scopes themselves
are freed by Targ_End.

The test opt-m-include-dir creates a directory of the form '*.tmp', thus
it must be removed before attempting to only remove regular files of
this name.

POSIX requires setenv to copy the passed name and value, so there is no
need to keep that memory allocated any longer.

Subst DEFSYSPATH in deptgt-phony

make: provide more context information for parse/evaluate errors

make: unit-tests handle TEST_MAKE:T != make

We need to allow for ${TEST_MAKE:T}[1-9]: etc when
TEST_MAKE is not 'make'

Add .MAKE.JOBS.C to indicate wether -jC is supported

tests/make: extend tests for the ':mtime' modifier

make: allow targets to be used as multiple-inclusion guards

This style is used by FreeBSD, among others.

tests/make: add test for multiple-inclusion guards

tests/make: clean up tests for the ':M' and ':S' modifiers

tests/make: clean up comments, extend a few tests

tests/make: clean up tests for .for loops

Most of the tests from forloop.mk were already in directive-for.mk.

make: add :mtime to provide mtime of file

The value of the variable is passed to stat(2)
and st_mtime is new value.
An optional arg can be used if stat(2) fails, otherwise
the current time is used.

See varmod-mtime.mk for usage examples.

make: meta-ignore tests will not work if TMPDIR is /tmp

Since /tmp and /var/tmp are always part of .MAKE.META.IGNORE_PATHS
the varname-dot-make-meta-ignore_* tests will not work
if TMPDIR is either (or subdir) of those.

Add unit-tests for .MAKE.META.IGNORE_*

These unit-tests should only be run if TEST_MAKE
has a valid .MAKE.PATH_FILEMON to indicate that filemon is
supported.

All three tests use the same meta-ignore.inc
which runs three sub-makes

one to initialize the target .meta file
two to ignore a file using .MAKE.META.IGNORE_{PATHS,PATTERNS,FILTER}
three to verify that without .MAKE.META.IGNORE_* the target is out of date.

Fix the order of _SED_CMDS to avoid errors when ${MAKE:T} appears in
.OBJDIR

make: .SYSPATH: to add dirs to sysIncPath

.SYSPATH: with no sources will clear sysIncPath
otherwise sources are added

Reviewed by: rillig

make: .[NO]READONLY for control of read-only variables

Reviewed by: rillig

tests/make: rename files that are not test cases

tests/make: add tests for whitespace in leading '@+-'

tests/make: fix test for option '-e' (broken since 2022-01-23)

tests/make: be more specific about the egrep compatibility

Handle deprecation of egrep

Only update distrib/sets/lists/tests/mi with valid files

sync-mi: use CVS/Entries to ensure we only update
distrib/sets/lists/tests/mi with files known to CVS.

Reviewed by: rillig

Enable make/unit-tests/directive-for-break

Also ensure MAKE_VERSION is ignored.

make: add .break to terminate .for loop early

When .break is encountered within a .for loop
it causes immediate termination.

Outside of a .for loop .break causes a parse error.

Reviewed by: christos

tests/make: fix canonicalization for single-digit day of month

make: prevent future out-of-bounds errors when parsing expressions

A modifier in an expression ends not only at the next ':' or at the
closing '}' or ')', but also at the end of the string.

Previously, testing for the end of the string had been done separately,
which was error-prone since 2006-05-11, when indirect modifiers were
introduced. Since then, it was possible that the string terminator '\0'
was accidentally skipped in cases where the loop condition only tested
for the ending character. When parsing indirect modifiers, the ending
character is indeed '\0', but when parsing direct modifiers, it is '}'
or ')'.

A welcome side effect is that in the case of unclosed expressions such
as '${VAR:Modifier', the amount of error messages is reduced from 2 or 3
to only 1. The removed error messages were wrong and thus confusing
anyway.

Pass MAKEOBJDIR to opt-m-include-dir

Prevent makeing opt-m-include-dir.tmp in src tree.

Reviewed by: rillig

tests/make: reword comment in Makefile to be more specific

Increase the resource limit so these tests pass on hppa.

LGTM from Roland

tests/make: document and demonstrate .for i containing .if empty(i)

PR bin/43821 describes the inconsistency that in a '.for i' loop, the
condition '.if ${i:M*.c}' works since 2009 while the seemingly
equivalent condition '.if !empty(i:M*.c)' does not access the variable
'i' from the .for loop but instead the global 'i'.

Resolving this situation in a backwards-compatible and non-surprising
way is hard, as make has grown several features during the last 20 years
that interact in various edge cases. For now, document the most obvious
pitfalls.

tests/make: remove test varquote

The test varmod-quote-dollar covers the same topic.

tests/make: migrate cond1 test to other, more specific tests

The tests in cond1 were a mixture of "everything related to conditions",
and the test cases were heavily dependent on each other, which made them
hard to understand. Move each test case to its corresponding
special-purpose test.

make: allow to randomize build order of targets

In complex dependency structures, when a build fails, a probable cause
is a missing dependency declaration between some files. In compat mode,
the build order is deterministic, in jobs mode, it is somewhat
deterministic. To explore more edge cases, add the line ".MAKE.MODE +=
randomize-targets" somewhere in the makefile.

Fixes PR bin/45226 by riastradh. Reviewed by christos.

make: only switch to POSIX mode if '.POSIX:' is the first line

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html
says that in order to make a makefile POSIX-conforming, its first
non-comment line must be the special dependency line '.POSIX:' without
any source dependencies.

Previously, make switched to POSIX mode even if such a line occurred
anywhere else, which was allowed by POSIX but was deep in the
"unspecified behavior" area. For NetBSD make, there is no big
difference since it doesn't ship any <posix.mk> file, this change mainly
affects the bmake distribution.

Previously, makefiles that contain '.POSIX:' somewhere in the middle
could fail due to <posix.mk> resetting .SUFFIXES, among other things.

Suggested by Simon J. Gerraty, who also reviewed an earlier version of
this change.