make: fix code coverage counts

See tests/usr.bin/gcov/t_gcov.sh.

branches: 1.127.2;
make: clean up code coverage report files

Remove trailing whitespace.

Remove the first line, as it only repeats the filename.

Remove the line numbers, as they are mostly the same as in the original
source file.

make: on cleandir, clean the unit-tests as well

make: sort files in coverage report alphabetically

bsd.own.mk: rename GCC_NO_* to CC_WNO_*

Rename compiler-warning-disable variables from
GCC_NO_warning
to
CC_WNO_warning
where warning is the full warning name as used by the compiler.

GCC_NO_IMPLICIT_FALLTHRU is CC_WNO_IMPLICIT_FALLTHROUGH

Using the convention CC_compilerflag, where compilerflag
is based on the full compiler flag name.

make: set WARNS to 6, from the default 5

No binary change on x86_64.

make: remove test scenarios for old GCC versions

These had been useful for NetBSD 8, which used GCC 5.

make: generate coverage summary on 'make test-coverage'

make: in Cmd_Exec, return error message instead of format string

This change leaves only literal format strings in parse.c. It allows
for more detailed error messages than the current "non-zero status" or
"exited on a signal".

No functional change.

make: do not report '?' for fully covered header files

make: only define target report-coverage if USE_COVERAGE is yes

Without USE_COVERAGE, GCOV was undefined, the '2>&1' passed all error
messages to GCOV_PERL, when then discarded them. If the error messages
had been left on stderr, the error message 'sh: arch.o.gcda: not found'
would have been a clear indicator of the actual cause of an empty
coverage report.

make: replace bloated bit-set-to-string code with simple code

It was a nice idea to implement a bit-set using an enum type and have a
generic ToString function for them. In the end, the implementation
involved really heavy preprocessor magic and was probably difficult to
understand. Replace all the code with a few bits of straight-forward
preprocessor magic that can be readily understood by just looking 5
lines around, instead of digging through 130 lines of lengthy macro
definitions.

Curiously, this reduces the binary size even though the 3 ToString
functions now have a few lines of duplicate code and there are more
explicit function calls.

The ToString functions are only seldom used, so the additional memory
allocation is acceptable.

No functional change.

make: fix lint warnings

The string functions from str.h are declared as 'static __unused' when
compiled with GCC, but lint explicitly undefines __GCC__ during
preprocessing. Therefore, make those functions inline, to prevent
warnings that they are unused.

The macro UNCONST is used in a few places, and (again) since lint
undefines __GCC__, that macro expanded to a simple type cast, which lint
warned about. To prevent this warning, implement UNCONST as a function
that works everywhere and hides the type cast.

In filemon_open, the code for closing F->in was obviously unreachable.

No functional change.

make: fix reported code coverage for *.h

In 'make test-coverage', the number of uncovered lines for inline
functions in headers was reported too high. The cause for this is that
gcov reports the coverage for these functions multiple times, once per
translation unit. If some of the translation units don't use these
inline functions, summing the lines containing '#####' quickly leads to
numbers that are obviously too high.

branches: 1.114.2;
make: clean up Makefile

The dependencies on the header files are handled by 'make depend'. They
do not need to be spelled out.

make: fix report-coverage

On NetBSD 8.0 it still worked. Maybe gcov doesn't support .c files as
arguments anymore. Using the .gcda files works and is more reliable
anyway since it covers the inline functions in the headers as well.

make(1): run lint with strict bool mode

The make code is not supposed to use constructs such as "if (strcmp(s1,
s2))" or "if (p && *p)", instead all boolean expressions have a form
that would be accepted by a C#, Go or Java compiler as well. This also
ensures that pre-C99 compilers generate equivalent code as C99
compilers, at least regarding boolean expressions.

make(1): add number of uncovered lines to coverage report

make(1): add target test-coverage

make(1): move .include lines as far down as possible

make(1): move COPTS above <bsd.prog.mk>

make(1): format coverage summary as a table

This is only used interactively, not in the official builds, therefore
the additional dependency on Perl doesn't matter. The same result could
have been achieved in any other programming language, but probably not
as concisely.

make(1): remove unused strlist_t

make(1): add target-specific COPTS when generating cpre or casm

make(1): fix GCC warning about small buffer for cohort_num

Since unmade_cohorts is a signed number (the code contains an underflow
check), the result of the '%' operator could be negative and result in
"#-999999", which would overflow the buffer, truncating the last digit.

make(1): enable format string truncation warnings for parse.c

No node is ever going to have a million or more cohorts. And even if
there were, this name is only used for printing informational messages.

make(1): revert previous commit

It had accidentally reverted all the work from the past few days.

make(1): fix double-free bug in -DCLEANUP mode (since 2020-10-02)

The bug had been introduced with dir.c 1.155 on 2020-10-02 22:20:25. In
that commit, openDirectories was replaced with a combination of a list
with a hash table, for more efficient lookup by name.

Upon cleanup, OpenDirs_Done is called, which in turn called
Dir_ClearPath. Dir_ClearPath takes full ownership of the given list and
empties it. This was no problem before since afterwards the list was
empty and calling Lst_Free just frees the remaining list pointer.

With OpenDirs, this list was combined with a hash table, and the hash
table contains the list nodes, assuming that the OpenDirs functions have
full ownership of both the list and the hash table. This assumption was
generally correct, except for the one moment during cleanup where full
ownership of the list was passed to Dir_ClearPath, while the hash table
still contained pointers to the (now freed) list nodes. This by itself
was not a problem since the hash table would be freed afterwards. But
as part of Dir_ClearPath, OpenDirs_Remove was called, which looked up
the freed directory by name and now found the freed list node, trying to
free it again. Boom.

Fixed by replacing the call to Dir_ClearPath with code that only frees
the directories, without giving up control over the list.

make(1): allow easy creation of preprocessed source and assembler

To be used during development, to see how changes in the source code
affect the generated machine code.

make(1): remove filemon object files on clean

make(1): fix compilation with GCC 8

make(1): fix copy-and-paste mistake for compiling with GCC10

make(1): remove redundant MAKEFLAGS from subdir make

The make flags are passed via the environment, not via the command line.
This is not as obvious and visible, but it works.

make(1): pass the command-line variables to the subdir make

make(1): remove header sprite.h

Make is independent of the Sprite operating system.

make(1): allow custom build options

make(1): allow to run only selected tests via the command line

Use TEST_MAKE to run unit-tests

This lets us take advantage of new features like ${.SHELL}

make(1): don't optimize when measuring the code coverage

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96622

make(1): fix archive test

Even though it is commented out in this Makefile, it is still run by
atf, on purpose. This test is broken on FreeBSD, but on NetBSD it
should still be run.

Without removing the archive first, it could be left over from a
previous test failure and thus be newer than all its members, leading to
${.OODATE} being empty.

make(1): enable -Wcast-qual for var.c again

When it became disabled 9 years ago, there were 15 instances of UNCONST
in the code. These have been removed in the latest cleanup sessions.

Tested with GCC 5 from NetBSD 8 and GCC 10 from pkgsrc.

make(1): allow optional compilation with GCC 10, use gcov from GCC

make(1): make gcov command line configurable

Some useful options are -a or -f.

make(1): clean up coverage files even if USE_COVERAGE is not set

make(1): allow to build using GCC 9 during development

make(1): add more detailed debug logging for variable modifiers

Before a modifier is applied to a variable, it is not yet parsed,
therefore it is only possible to log a rough estimate of the modifier.
But after applying it, the parsing position has advanced, and the full
modifier can be logged.

In addition, to fully understand how the modifiers work, it's not enough
to just know the variable names and values, there are also some flags
that influence how the modifiers behave. The most influential is
VARE_WANTRES.

Thanks to sjg for the extensive review and valuable feedback on the
first drafts.

make(1): one file per line in the Makefile

Suggested by sjg.

make(1): revert previous commit; "make retest" was already possible

What I really meant to allow was to run "make sync-mi" directly from the
main directory. But adding that to TARGETS didn't do anything.

make(1): allow "make retest" directly from the main directory

make(1): no declaration-after-statement anymore

NetBSD make is intended to be maximally portable, therefore it uses only
C89. This was not declared in the Makefile before.

There are still a few places in parse.c and metachar.c that use
end-of-line comments. These will be fixed in a follow-up commit.

make(1): remove line numbers from generated code coverage files

The line numbers make it difficult to spot changes in the code and
coverage if a line is added or removed from the code.

make(1): fix undefined behavior in malformed :S modifier

The unit tests failed on Ubuntu, and by looking closely at the code, it
was trivial to see the out-of-bounds memory read.

Other modifiers may have the same problem and will be inspected later.

make(1): use specialized return type for ApplyModifier functions

This makes it immediately obvious what happens after a modifier has been
applied, instead of having to translate single-character mnemonics or
booleans to their actual intention.

This also reduces the size of the binary since there are fewer jumps.

make(1): explicitly add dependencies on headers

This prevents partial builds after changing a header. The declared
dependencies are more than strictly necessary, but that's still better
than having inconsistent partial builds because too few dependencies are
declared.

make(1): condense the list library into a single file

The list library is only used in make(1). Having it spread out over 28
files made it look more complex than it really is. In fact, it's just a
versatile generic data type like in hash.c.

Having all the list functions in a single file reduces the code size,
both by omitting the many RCS Ids and by inlining commonly used code.

make(1): make it easy to run gcov on the code

There seems to be no gcov support in share/mk yes, but it's still good
to have, especially for a complex program like make(1).

make(1): indent Makefile directives like in pkgsrc

This makes it easier to see the corresponding .if/.endif, even without
comments.

make(1): format Makefile consistently

make(1): regroup source files

Fix meta_compat mode.

We need to call meta_job_output() from meta_compat_parent()
Also, the need for extra fds for meta mode only applies when
using filemon_ktrace.c.
Since we hope to see more interfaces like that, test for
USE_FILEMON_DEV being not defined.

Reviewed by: riastradh

Tweak makefile for USE_FILEMON=dev, from sjg.

Per sjg's suggestion, split filemon API into separate back ends.

By default we use the ktrace back end, but the /dev/filemon back end
is available as a compile-time option, by setting USE_FILEMON=dev in
make. sjg raised concerns about ktrace performance and would like to
continue using /dev/filemon on FreeBSD (which has seen more
maintenance kernel-side) without forking make.

Reimplement make(1) meta mode without filemon(4).

Revert the filemon removal in bmake, as pointed out by maya we do care
about not introducing divergence with FreeBSD, and the cost of unused
is acceptable here.

Retire filemon, discussed on tech-kern@.

introduce some common variables for use in GCC warning disables:

GCC_NO_FORMAT_TRUNCATION -Wno-format-truncation (GCC 7/8)
GCC_NO_STRINGOP_TRUNCATION -Wno-stringop-truncation (GCC 8)
GCC_NO_STRINGOP_OVERFLOW -Wno-stringop-overflow (GCC 8)
GCC_NO_CAST_FUNCTION_TYPE -Wno-cast-function-type (GCC 8)

use these to turn off warnings for most GCC-8 complaints. many
of these are false positives, most of the real bugs are already
commited, or are yet to come.


we plan to introduce versions of (some?) of these that use the
"-Wno-error=" form, which still displays the warnings but does
not make it an error, and all of the above will be re-considered
as either being "fix me" (warning still displayed) or "warning
is wrong."

convert HAVE_GCC == 7 to HAVE_GCC >= 7.

- use -Wno-error=format-truncation

branches: 1.60.2;
use SUBDIR.roff suggested by uwe@

branches: 1.59.14;
Adapt to changed Var_Subst()

Centralize the "is a meta char" test, instead of using two different arrays.

Rework /usr/share/doc.

Update the <bsd.doc.mk> infrastructure, and update the docs to match
the new infrastructure.

- Build and install text, ps, pdf, and/or html, not roff sources.

- Don't wire the chapter numbers into the build system, or use them in
the installed pathnames. This didn't matter much when the docs were a
museum, but now that we're theoretically going to start maintaining
them again, we're going to add and remove documents periodically and
having the chapter numbers baked in creates a lot of thrashing for no
purpose.

- Specify the document name explicitly, rather than implicitly in a
path. Use this name (instead of other random strings) as the name
of the installed files.

- Specify the document section, which is the subdirectory of
/usr/share/doc to install into.

- Allow multiple subdocuments. (That is, multiple documents in one
output directory.)

- Enumerate the .png files groff emits along with html so they can be
installed.

- Remove assorted hand-rolled rules for running roff and roff widgetry
and add enough variable settings to make these unnecessary. This
includes support for
- explicit use of soelim
- refer
- tbl
- pic
- eqn

- Forcibly apply at least minimal amounts of sanity to certain
autogenerated roff files.

- Don't exclude USD.doc, SMM.doc, and PSD.doc directories from the
build, as they now actually do stuff.

Note: currently we can't generate pdf. This turns out to be a
nontrivial problem with no immediate solution forthcoming. So for now,
as a workaround, install compressed .ps as the printable form.

branches: 1.56.2; 1.56.8;
Don't pass debug flags etc. to unit-tests

branches: 1.55.2;
- remove gcc-4.5 warnings
- simplify job printing code
- document non-literal format strings

apply some -Wno-error and/or -fno-strict-aliasing.


all of this should be looked at closer, but some of them are not
very trivial.

Back out a local change

If a makefile if a multiple of the page size and doesn't end with a
newline, then it all goes horribly wrong when make tries to terminate
the last input line.
This was all ok before the mmap changes (rev 1.169) because the last
32 bytes of the input buffer were never read into.
The non-mmap case has a similar problem.
Bug found because my .depend files don't have a newline at the end of
the .OPTIO|NAL line (don't know if that is a local change).

Add meta.c which implements "meta" mode for make.
In this mode, a .meta file is created for each target, capturing
the expanded commands used, any command output, and if filemon(9)
is available, a record of system calls which are of interest.
Not enabled unless USE_META=yes is set when building make.
Also, if FILEMON_H exists, meta.c will be compiled to use filemon(9).

PR: 42850
Reviewed by:

modmisc: since we apply an exists() test to $paths, be conservative
in what we expect.

Run the unit tests with -r -m / so that we do not fail if there
is no sys.mk present.

Enable WARNS=4 by default for usr.bin, except for:
awk bdes checknr compile_et error gss hxtool kgetcred kinit
klist ldd less lex locale login m4 man menuc mk_cmds
mklocale msgc openssl rpcgen rpcinfo sdiff spell ssh
string2key telnet tn3270 verify_krb5_conf xlint

remove unneeded special rule for main.o

The Makefile and main.c arranged to put "netbsd-${DATE}" into the
executable. Remove this so that different compiles can be binary
compared. rcsid's for all files are already embedded in the executable
so versions can be easily distinguished. (I didn't catch this on
previous passes because I did my builds on the same day.)

Note: there's a special rule for main.o in Makefile with a purpose I
can't actually discern -- I think it isn't needed, and I've flagged it
with a comment.

branches: 1.46.2;
Move the bmake_malloc() functions into their own .c and .h files.
Include instead of make.h in a few places.

Add functions for processing extendable arrays of pointers to strings.
Use for the .for variables and substution items - changing the latter from
make's all conquering lst.lib functions.
Being able to index everything makes the code simpler.
No functional changes intended.

Don't use emalloc and friends directly, but call them consistently
bmake_malloc and friends. Implement them via macros for the native case
and provide fallback implementations otherwise. Avoid polluting the
namespace by not defining enomem globally. Don't bother to provide
strdup and strndup, they were only used for the estrdup and estrndup
comapt code.

This addresses the presence of emalloc in system libraries on A/UX and
resulted strange issues as reported by Timothy E. Larson.

- use pid_t/size_t as appropriate instead of int.
- use %ld to print pids.
- fix a bit of lint.
- WARNS=4

Revert 1.34 (CFLAGS+=-g) that have been commited by mistake (hi, xtos).

branches: 1.41.2; 1.41.4; 1.41.8;
Use of libutil to provide emalloc et al, should be predicated on
a native NetBSD build using TOOLDIR etc.
Otherwise use the local versions.

Reviewed by: christos

Fix previous - need to add a lstPrev()

WFORMAT is no more...

protect libutil with HOSTPROG

More programs using efun.

Remove a hack that's no longer needed with gcc 3.3.6.

branches: 1.35.2;
Our gcc 3.3.3 (nb3) for hppa gets an "unrecognized insn" internal
compiler error when building buf.c with -O2, so reduce optimization
level to -O1 for this single file.
To be documented in docs/HACKS, discussed with skrll.

Coverity CID 529: Call VarFreeEnv to prevent leak.

Simplify build, no functional changes.

Instead of adding MAKE_BOOTSTRAP for hosted environments, i.e., when
you want things simple, instead add MAKE_NATIVE to get those hugely
important features like __RCSID().

It's now possible to build make on some hosts with: cc *.c */*.c

Instead of including from the wrong directory and then fixing it (-I)
in the Makefile, how about just doing the right include?

branches: 1.31.2;
Use ${.MAKE:S,^./,${.CURDIR}/,} so that when someone does
./make test
it works. Note use .CURDIR rather than .OBJDIR since the later
_may_ not always be correct (eg. symlink in .CURDIR for make).

Fix for TEST_MAKE from Alan Barrett

Fix parsing bug for :ts - patch from Alan Barrett <apb@cequrux.com>
Also add simple unit-test jig (regress/usr.bin/make will use it too)
but having it local here makes inclusion in bmake simpler.

build var.c with -Wno-cast-qual - gcc3.3 complains about VarWordCompare()

Pass WARNS=3

Fix major bug in make(1) ... due to shadowing of the dotLast path used for
the .DOTLAST primitive by a boolean variable with the same name, this whole
mechanism was broken ... it doesn't save much stat calls but it was wrong.

Thanks to Jason Thorpe for the other shadow-variable fixing patches he
made.

Overhaul the initialization and handling of .OBJDIR:

* Replace chdir_verify_path() with Main_SetObjdir(), which can be called
externally, and can take a "const char *". (There's a lot of non-const
"char *" passing around in var.c of what should be const strings....)

* Rewrite the initial "find my .OBJDIR" code to make use of the new
function. This still functions as it had in the past, but the comment
above this block was changed to reflect reality: if MAKEOBJDIRPREFIX
or MAKEOBJDIR are set in the environment, then *only that value* is
tried; make does not fall back to obj.MACHINE, obj, and /usr/obj/`pwd`
as it would without these env vars set.

* Add a new special target, .OBJDIR:, which when parsed will cause make to
change to a new object directory and reset .OBJDIR, and PWD in the
environment. This will allow some makefiles (mainly, src/tools)
to override the default objdir semantics in order to add custom logic.

Provide a useful? MAKE_VERSION.

Back out debug cruft.

Add a token-passing scheme to allow a recursive make to successfully
use -j; all make's in a recursive build cooperate to limit the total
number of jobs, using a token-passing scheme.

The current token passing algorithm is similar to the one implemented
by gmake; there is a single pipe which is inherited through the entire
process hierarchy; tokens are obtained by reading a byte from the
"read end" of the pipe, and are returned by writing them to the "write
end". This exact algorithm is likely to change in the future.

Implementation details:
- Use the new trace facility to allow measurement of the
effectiveness of different token-passing schemes
- Get a token in MakeStartJobs(), return it in Make_Update()
- Eliminate Job_Full() and the jobFull global since they are
redundant with token system.
- Add an "internal" -J option (to pass the token pipe fd's down to
submakes) and a -T option for tracing.
- Change how compatMake is forced so that -j means something when
inherited by submakes.
- When waiting for a token, poll the token-passing pipe as well as
the output pipes of existing jobs.

More format string cleanup by sommerfeld.

branches: 1.20.8;
Back out part of previous, only the parse.c mod was intended.

Trivial speed hack.

branches: 1.18.2;
use CPPFLAGS instead of CFLAGS

branches: 1.17.2;
don't define WARNS=1 here

Use bsd.subdir.mk as appropriate.

Add WARNS=1
RCSID police

Back out the .PATH.c changes. The .depend problem (and others)
will be fixed using the new .NOPATH make feature instead.

Use .PATH.c: ...

- Merge in FreeBSD and Lite2 changes.
- Fix bug where a non-archive target with a .a suffix would always
be considered to be out of date, since it does not have a TOC.

- Move -D flags from Makefile to config.h and explain what they do. Add
-Wall -Wno-unused to CFLAGS. Add new define SYSVVARSUB to enable SysV
style variable substitutions and enable them.
- Add SunOS style command substitutions via SUNSHCMD
- Fix core dump with '{variable = value'

branches: 1.10.4;
Jonathan Stone's fixes:
main.c: Remove double include of <sys/resource.h>; if MACHINE is defined, use
that instead of uname.
Makefile: Add util.c.

fix pr/1421 and pr/1997

- $NetBSD$ rcsids
- Fixed so that .[A-Z]* targets that do not match keywords are ignored as
Posix mandates
- Added .PHONY target keyword

None of the functions in util.c are needed by NetBSD, so don't
bother to compile it.

deal with new share/doc strategy

update for new doc-make/install strategy

fixes/improvements from Christos Zoulas <christos@deshaw.com>.

Compile with -DPOSIX --- enables reasonable behavior for MAKE and
MAKEFLAGS variables.

Add RCS indentifiers.

branches: 1.1.1;
Initial revision