allow XZ_OPT to be overriden.

Set the file creation mask

sets: enhance MAKEVERBOSE support

Add more messages for various sets operations.
Add ${_MKMSG_CREATE} to bsd.own.mk.

Enhance maketars to be a bit more verbose even with -q,
to make it more obvious why certain operations may be slow.
May help debugging PR 47562.

Deprecate extsrc/, EXTSRCSRCDIR, MKEXTSRC

Remove support for building extsrc/:
1. Makefile: remove do-extsrc target.
2. build.sh: remove options -y and -Y extsrcdir.
3. distrib/sets: remove support for extsrc in various tools
including the options -L ext and -y, and the extsrc sets.
4. doc/BUILDING.mdoc: remove docs for extsrc/, EXTSRCSRCDIR, MKEXTSRC (etc)
5. bsd.own.mk and various Makefiles: remove support for extsrc/,
EXTSRCSRCDIR, MKEXTSRC.

As proposed on tech-kern and tech-userlevel on 2022-01-07
and followed up on 2022-08-21.

convert make's HAVE_MESA_VER into mesa_ver for set lists

Redo xz vs. gzip selection via conditional make expressions, suggested
by joerg.

Another USE_XZ_SETS which probably should be fixed elsewhere
in a better way

Prepare for .tar.xz sets

branches: 1.103.12; 1.103.14;
support xorg_server_ver define, currently settable to 110 (1.10.x)
or 118 (1.18.x).

Fix SED issue differently.

Export the variables so that they propagate via sets.subr -> make -> mkvars.mk

Obey MKREPRO_TIMESTAMP when creating sets (create all the files in the
sets with that date).

checkflist requires an egrep that honours multiple pattern options, so
use ours. Fixes build on Solaris distributions.

Adapt the sortlists target to deal with commented file names, like
"#./dir/file". Such lines now compare equal to their uncommented
counterparts like "./dir/file".

branches: 1.97.2;
Ue ${GZIP_N_FLAG} instead of -n when setting GZIPFLAGS or GZIP_FLAGS.
This will allow -nT instead of just -n to be passed to pigz.

Execute sub-${MAKE} from within a source directory.

Teach "make sortlists" to ignore .#* files (sometimes created by cvs),
and to squeeze multiple blank lines.

branches: 1.94.2;
Pass the -n flag to gzip invocations.

This prevents it from embedding a timestamp in the output. We pass
"-n" unconditionally, not conditional on MKREPRO, because many other
invocations of gzip already passed the -n flag unconditionally.

Use ${TOOL_GZIP} instead of just gzip in all Makefiles
outside */dist/* subdirectories.

When USE_PIGZGZIP=yes, bsd.own.mk sets TOOL_GZIP=${TOOL_PIGZ},
so there's no need to test USE_PIGZGZIP in these Makefiles.

Remove time= keywords from METALOG.sanitised.

The times add unnecessary differences to the DESTDIR trees for MKREPRO
builds, and they are not needed by anything. (maketars, the chief
consumer of METALOG.sanitised, explicitly ignores the time fields.)

Also factor out some duplicated code and update comments.

eliminate use of bsd.sys.mk from Makefiles

In "make sortlists", if the file was already sorted then don't
change its timestamp.

Add "make sortlists", for sorting the set lists. This may be
run manually, before committing changes to the lists files.

Remove -s "stable" flag, since it is not posix.

Use sort -s to get a stable sort, even if that is the default for ours. We
prolly need a TOOL_SORT...

only do the keep last in update builds and explain why.

keep only the last entry for each file, before sorting.

branches: 1.84.2; 1.84.8;
when sorting the metalog for unpriv builds, first run it through sort so
mtree doesn't fail with:
nbmtree: .: missing directory in specification
nbmtree: failed at line 1 of the specification

Comment.

RCSMETALOG: treat initial clean_METALOG nicely.

Add a hook to version METALOG using rcs(1).

branches: 1.80.2;
add new option to build/use tools/pigz for compressing sets. for now
USE_PIGZGZIP defaults to "no". (it depends upon local <pthread.h> and -lz.)

you can set it to "yes" on most modern platforms just fine.

export PIGZ=${TOOL_PIGZ} (not yet used.)

Clean up debug knobs.

Set $rundir before calling sets.subr as other scripts do.

Style.

sh -> ${HOST_SH}

Add print_mkvars make target for debug.

Remove definition of GREP variable, which was unused. The only use
of this variable was removed in revision 1.67.

Pass ${TOOL_PKG_CREATE} to scripts for syspkg creation.

Support "extsrc", externally added programs and libraries. Users can write
their own reach-overs, cross-build, install, and get set files just like base
and X11 / X.org. (These sets are not included as TNF releases.)

Only generate SHA512 and MD5 checksums for releases.

If there are errors in creating a ${set}.tgz tar file, then don't
just press on regardless.

Add "-M" flag to the mtree invocation in sanitise_METALOG. This should
avoid errors in cases where a build poduct changes type (e.g. a file
becomes a symlink) between one build with older sources and a subsequent
update build with newer sources.

Clean up the handling of METALOG.sanitised:

* Don't change METALOG in place while creating METALOG.sanitised in the
sanitise_METALOG target. This makes it easier to manually inspect
METALOG to find the order in which lines were added during a build.

* Add a clean_METALOG target, to be invoked from src/Makefile early in
the build, to prevent unbounded growth of METALOG during multiple
MKUPDATE builds.

* Remove the old hack of incorporating etc/mtree/NetBSD.dist into
METALOG.sanitised. It was needed only during a short transition
period when people might have done MKUPDATE builds starting from
existing DESTDIR and OBJDIR trees that had been created before
revision 1.317 of src/etc/Makefile.

* Since etc/mtree/NetBSD.dist is no longer added, there's will not
be any "optional" lines, so there's no need to filter them from
METALOG.sanitised.

* There's no need for two different METALOG.unpriv and
METALOG.sanitised.unpriv variables; combine them into a single
METALOG.unpriv variable whose value is "-M ${METALOG}.sanitised".

* Reorder some lines in the Makefile, to bring related targets,
variables and comments together.

In various places that manipulate mtree specs, use "mtree -CS" to sort.
Remove some now-redundant standalone sort commands.

branches: 1.65.2;
Add support for new MAKEVERBOSE levels 3 and 4. The complete list is now:

0 Minimal output ("quiet")
1 Describe what is occurring
2 Describe what is occurring and echo the actual command
3 Ignore the effect of the "@" prefix in make commands
4 Trace shell commands using the shell's -x flag

The default remains MAKEVERBOSE=2.

- retire MKXORG from public use
- introduce X11FLAVOUR to choose src/x11 vs src/external/mit/xorg
for the X11 to build if MKX11=yes is set. it takes the values
of either Xorg or XFree86.
- default to Xorg on alpha, i386, macppc, shark, sparc and sparc64
- remove MKXORG_WITH_XSRC_XSERVER, unused and never useful

branches: 1.63.2;
Pass SED=${TOOL_SED:Q} in the environment to scripts run during the build.

Use ${TOOL_AWK} instead of ${AWK} or plain "awk" in make commands.
Pass AWK=${TOOL_AWK:Q} to shell scripts that use awk.

Remove set lists for the non-cross-compiled ("vendor provided") build of
XFree86; we've used our src/x11 framework for four years.

Improve support for MKXORG!=no to enable builds into external/mit/xorg.
This will be mutually exclusive to MKX11!=no.

Change remaining instances of ${RELEASEDIR}/${MACHINES} to
${RELEASEDIR}/${RELEASEMACHINEDIR}

branches: 1.58.2;
If MAKETARS_SERIAL is set to "YES" or "yes", then the tar sets will be
made in series, even if make's "-j" flag requests multiple parallel
jobs. This is useful on systems that exhibit poor performance when
running multiple parallel disk-intensive operations. The default is
MAKETARS_SERIAL=NO, which will obey the "-j" flag passed to the make
command.

Requested in tech-toolchain. Reverts the change made in revision 1.56.

branches: 1.57.2;
Add '-q' to maketars and makesrctars, to quieten them.
Improve MAKEVERBOSE support.

branches: 1.56.4; 1.56.10; 1.56.12;
Serialise making the "sets" tarballs in the maketars target.
On many systems, making them in parallel causes very poor
system performance due to disk I/O. (Requested by jmmv.)

Let do-${tar} depend on sanitise_METALOG.

branches: 1.54.2;
Ensure that ${METALOG} does not grow without bounds as an effect of
repeated unprivileged UPDATE builds.

Discussed with lukem@

remove support for building (with) GCC 2.95. also:
- always install <stdbool.h>
- don't generate a fake one for vax / gettext.

retire HAVE_GCC3/HAVE_GCC4 and introduce HAVE_GCC that is set to 2, 3 or 4.

Prepend etc/mtree/NetBSD.dist to the metalog before passing it to
${TOOL_MTREE}. Previously this was done by maketars. In the long
term there should be a better solution. In the short term, this
should fix the build for people who do unprivileged update builds and
who had an old METALOG from before the recent syspkg-related changes.

Reviewed by christos

Add a "makedeps" target, to update the "deps" file.
XXX: Why is "deps" checked in to the source tree, instead of
just being created as necessary?

Reviewed by agc

* Add new user-visible "syspkgs" target, and internal "makesyspkgs" and
"makesyspkgsums" targets for it to use internally.
* Add internal "sanitize_METALOG" target that uses "${TOOL_MTREE} -C ..."
to sanitise the METALOG file before invoking anything that will
want to parse the METALOG (checkflist, regpkgset, maketars, or
installsets). Previously, maketars used to do this itself, but its
better to do it only once.
* Add internal "checkflist_if_DESTDIR" target, which invokes checkflist iff
DESTDIR is set and not empty. Use it where appropriate.
* Add a few more host tools to SETSENV.
* A few minor related changes.

Reviewed by agc

Restructure how makesrctars works so it spits out source balls that look like
they would from a release (files owned by root:wsrc, perms set, etc)

Add SHA512 into sums lists

branches: 1.46.2;
``build.sh -m xen-i386 release'' now builds a release for NetBSD/xen
for i386. The resulting release consists of:
- NetBSD/xen for i386 kernel, loader and docuemntation
- NetBSD/i386 userland sets

makesrctars:
* Support '-x xsrcdir' to tar up xsrcdir as xsrc.tgz, with the
contents of the tarfile being relative to xsrcdir and
prefixed with 'usr/xsrc'
* Convert to getopts a la the other scripts in this directory

Makefile
* if ${MKX11} != no, call makesrctars with '-x ${X11SRCDIR}'

branches: 1.44.2;
makeflist is .PRECIOUS, do not delete it if make is interrupted.

* Add "makesetfiles" target, to populate etc/mtree/set.* with maketars -S
* Some MAKEVERBOSE support.
* Add some variables to simplify various rules.
* Simply script startup by replacing ${SETSENV} with ${SETSCMD}, which
cd's to ${.CURDIR} before doing the work.

Remove the set if maketars returns a non-zero exit code

* In installsets, pass ${MAKEFLIST_FLAGS} so that the X sets will also
be copied if MKX11 != no.
* Clean up .PHONY and my .PRECIOUS.
* Cosmetic tweaks.

If MKX11 != no, invoke makeflist & checkflist with -b.

Use "makeflist -l" to determine MAKETARSETS

Use sh -c '. ./sets.subr; echo $x11_version' to implement print_x11_version.

Simplify rules so tarfiles always attempt to rebuild (let maketars determine
whether or not to continue based on METALOG if it choses). Fixes bug where
tar files were rebuilding every other time

Significant overhaul of how the sets are used:

* Add an optional third field to the sets file which is a list of
comma separated keywords that control if the line is printed.
Currently supported keywords
kerberos4 ${MKKERBEROS4} != no
kerberos ${MKKERBEROS} != no
lint ${MKLINT} != no
obsolete ${obsolete} != 0.
In this case, non obsolete files are not printed.
(This will allow future support for builds with variables such as
MKHESIOD and MKYP set to "no".)

* Use sh(1)'s getopts where appropriate, and otherwise cleanup the
various scripts.

* Move defaults for sets.subr from sets.defaults into sets.subr.
Move replicated code for determining stuff such as shlibs type
from various scripts into sets.subr.

* Merge the obsolete.*, krb.*, krb4.* and lint.* into the appropriate
main lists with the relevant third field keyword(s).

Add a .ORDER so the checksums come out in the proper sequence.

Nuke the sets before building to make sure the original behavior of makesets
rebuilding all sets is preserved.

Forgot that makesums creates the checksum files anew each time it's run. Save
outputs from each step and concat into place when done.

Don't require '.' in PATH for sets.default

Redo creation of sets/sums so this can be parallelized if using -j.
Suggested in PR#20115 and PR#20116

Invoke ./checkflist with ${CHECKFLIST_FLAGS}, so that the end-user
can set CHECKFLIST_FLAGS="-e" in mk.conf so that extra files in
$DESTDIR are `non fatal'.

Remove unused '${CHECKFLISTFLAGS}' (which allowed selection of
different checkflist target in maketars target)

add MKTEMP=${TOOL_MKTEMP:Q} to SETSENV

Fix sets handling so that GCC specific lists are really used.

add initial support for gcc3

support XF4 on sparc. had to move parts of x{base,comp}4/mi into the
relevant ports as they do not apply to sparc (or other non-"hw/xfree86"
ports.)

Add MKUNPRIVED; if not no has the same semantics as if UNPRIVED was defined.
Replace defined(UNPRIVED) tests with ${MKUNPRIVED} != "no"

Add MKUPDATE; if not no has the same semantics as if UPDATE was defined.
Replace defined(UPDATE) tests with ${MKUPDATE} != "no"

Improve documentation for these and other make flags.

Rename a large chunk of the make(1) variables which refer to a
program/tool from "FOO" to "TOOL_FOO". The new variables are:
TOOL_ASN1_COMPILE TOOL_CAP_MKDB TOOL_CAT TOOL_CKSUM TOOL_COMPILE_ET
TOOL_CONFIG TOOL_CRUNCHGEN TOOL_CTAGS TOOL_DB TOOL_EQN TOOL_FGEN
TOOL_GENCAT TOOL_GROFF TOOL_HEXDUMP TOOL_INDXBIB TOOL_INSTALLBOOT
TOOL_INSTALL_INFO TOOL_M4 TOOL_MAKEFS TOOL_MAKEINFO TOOL_MAKEWHATIS
TOOL_MDSETIMAGE TOOL_MENUC TOOL_MKCSMAPPER TOOL_MKESDB
TOOL_MKLOCALE TOOL_MKMAGIC TOOL_MKTEMP TOOL_MSGC TOOL_MTREE
TOOL_PAX TOOL_PIC TOOL_PREPMKBOOTIMAGE TOOL_PWD_MKDB TOOL_REFER
TOOL_ROFF_ASCII TOOL_ROFF_DVI TOOL_ROFF_HTML TOOL_ROFF_PS
TOOL_ROFF_RAW TOOL_RPCGEN TOOL_SOELIM TOOL_SUNLABEL TOOL_TBL
TOOL_UUDECODE TOOL_VGRIND TOOL_ZIC

For each, provide default in <bsd.sys.mk> of the form:
TOOL_FOO?= foo
and for the ${USETOOLS}=="yes" case in <bsd.own.mk>, provide override:
TOOL_FOO= ${TOOLDIR}/bin/${_TOOL_PREFIX}foo

Document all of these in bsd.README.

This cleans up a chunk of potential (and actual) namespace collision
within our build infrastructure, as well as improves consistency in
the share/mk documentation and provision of appropriate defaults for
each of these variables.

- add makesrctars and sourcesets targets (latter just invokes former)
- use ${HOST_SH} instead of sh

In "sets" target, ensure ${RELEASEDIR}/${MACHINE}/binary/sets exists.
Fixes [toolchain/20120] from Lubomir Sedlacik.

Check if DESTDIR is defined before checking for non empty DESTDIR

Install release files under "${RELEASEDIR}/${MACHINE}/..." instead of
"${RELEASEDIR}/...".

${RELEASEDIR} is never cleaned , and ${RELEASEDIR}/${MACHINE} is only
cleaned if UPDATE is not defined.

Always call maketars (and subsequently pax) with -N ${NETBSDSRCDIR}/etc,
so that we always use the user & group databases from there rather than
the host systems.

maketars:
- add "-N etcdir", to be used for pax & mtree -N. defaults to ${DESTDIR}/etc

Makefile:
- if UNPRIVED, call maketars with "-N ${NETBSDSRCDIR}/etc", so that
"make UNPRIVED= DESTDIR=/foo installsets" can work with an empty
${DESTDIR}/etc. (i.e, after a make build without install-etc-files)

If ${DESTDIR}="", ensure `maketars -d ...' is called with an argument ("/").
Fixes problem noted by simonb@

Add "installsets" target, which copies the sets ${INSTALLSETS} from
${DESTDIR} into ${INSTALLDIR}.
INSTALLDIR is mandatory.
INSTALLSETS is optional, and if its not given, the following sets are used:
base comp games man misc text

switch from checking USE_NEW_TOOLCHAIN == yes to TOOLCHAIN_MISSING != yes,
and don't bother with tc.old any more

checkflist:
- add "-M metalog". if metalog starts with "${DESTDIR}/", it
will be skipped in the flist check, to prevent unnecessary
warnings about extraneous ./METALOG
- if the diff returned a non-zero exit code, print a warning message,
and for the default diff output, print a blurb describing the
meaning of the output.

Makefile:
- call checkflist with "-M ${METALOG}" if UNPRIVED
- run "${MAKE} checkflist" if ${DESTDIR} != ""

NOTE: this change will prevent maketars from running if the
sets lists are out of date. This is intentional.
Better to fail than to build tar files with missing files.

branches: 1.15.2;
yet another distrib overhaul:
- move guts of distrib/Makefile.inc to distrib/common/Makefile.distrib
(fixes problem caused by implicit include of ../Makefile.inc in certain
submake conditions triggered by makefiles not yet in tree)
- removed mkdir of ${RELEASEDIR}/*; rely upon "snap_pre" target of
etc/Makefile to create all the release directories
- renamed RELINSTALL to RELEASE_INSTALL
- renamed FLOPPYINSTDIR to FLOPPY_RELEASEDIR
- renamed MDSETDIR to MDSET_RELEASEDIR
- removed ITARGET
- move release target from top level to appropriate subdirectory
- ensure release target has correct depends
- replace miniroot's IMAGE_MD_POST with common/Makefile.image IMAGEPOSTBUILD
- Makefile.image: add realall: ${IMAGE}

Insert .WAIT between maketars and makesums in sets: dependancy.

An explicit makesums: maketars dependancy was not used because a
release engineer might want to regenerate the sums only and not the
tarballs; there's no explicit dependancy tracking here or elsewhere
between the input files and the output tarballs so everything gets
rebuild always.

XXX The existing "maketars" script is also not friendly to parallel
builds -- all of the sets files should be visible to the makefile so
they can be built in parallel.

replace _SRC_TOP_ with NETBSDSRCDIR, since the latter is the "public" version

call maketars with ${MAKETARSETS}; if empty (default), all sets are built

use ${MAKESUMS} from ../Makefile.inc

Remove a spurious crypto-related comment that should have gone long ago.

Set CKSUM in the environment when calling makesums.

Pull in <bsd.sys.mk> so that defaults for $PAX, $MTREE (etc) are
provided, which is required for USETOOLS=no because <bsd.own.mk>
doesn't set those in that case. Should fix [toolchain/16009]

provide check_DESTDIR and check_RELEASEDIR targets in <bsd.own.mk>
and use those rather than local versions

Fixes to allow unprivileged building of the binary tar files to
contain the correct permissions.

Makefile:
- set SETSENV to the list of variables we wish to pass to each shell
script and use that, rather than duplicating the effort below
- call maketars with "-M ${METALOG}" if UNPRIVED is set.

getdirs.awk:
- takes a list of filenames and outputs the list along with all the
directories used by those filenames

maketars:
- support "-M metalog"
- merge etc/mtree/NetBSD.dist with the "-M metalog" for use in the tgz build
- for a given set, find the union between the set list and the metalog
generated above and use that as the list of files to ${PAX} up as the
tgz file.

First step of supporting XFree86 4.x binary sets:
- Recognize version dependent X11 lists files.
- Move all files which only belong to XFree86 3.3.6 in lists for version 3.

add a checkflist-x11 target

Use the logic in bsd.own.mk to figure out the toolchain format,
and to convert MACHINE_ARCH to a CPU. Make this cross-build
friendly by specifying MACHINE, MACHINE_ARCH, and MAKE in the
makeflist/checkflist environment.

branches: 1.2.2;
Merge a bunch of things from crypto-us and crypto-intl into basesrc,
adding support for Heimdal/KTH Kerberos where easy to do so. Eliminate
bsd.crypto.mk.

There is still a bunch more work to do, but crypto is now more-or-less
fully merged into the base NetBSD distribution.

Make doing a few things here (e.g. checkflist) a little easier. This
determines which crypto set to build based on <bsd.crypto.mk> and invokes
the programs with the various correct options.

Not complete yet, but a good starting point for a few things, and a little
more flexible than the method used in the release: target in etc/Makefile.