xref: /src/UPDATING

$NetBSD: UPDATING,v 1.134 2005/01/09 07:31:31 christos Exp $

This file (UPDATING) is intended to be a brief reference to recent
changes that might cause problems in the build process, and a guide for
what to do if something doesn't work.

For a more detailed description of the recommended way to build NetBSD
using build.sh, see the BUILDING file.

Note that much of the advice in this UPDATING file was written before
build.sh existed.  Nevertheless, the advice here may be useful for
working around specific problems with build.sh.

See also: BUILDING, build.sh, Makefile.

Recent changes:
^^^^^^^^^^^^^^^
20050109:
	Since su is using pam by default now, make sure that you have
	/etc/pam.d populated (postinstall will do that automatically
	for you). Otherwise su will fail open (i.e. will not require
	a password).

20041229:
	Make had a path resolution bug that manifested itself as not
	being able to install openpam.3. This bug has been fixed, but
	you might need to rebuild make manually first to get through
	the build.

20041201:
	Userland programs have been changed to use /dev/bpf instead of
	/dev/bpfX. You need to create that device by installing a new
	MAKEDEV and running it, or mv /dev/bpf0 /dev/bpf && rm /dev/bpf[0-9]*

20041006:

	A bug was introduced into /bin/sh (var.c rev 1.35) which causes
	variables to not export correctly to subshells in all cases. This
	will cause builds to break if that version of /bin/sh is installed
	on the system. The proper version of /bin/sh can be verified with:

        ident /bin/sh | grep var

	Any /bin/sh w. version 1.35 will not work and needs to be updated
	before attempting a build.
	
20041001:
	The ipfilter kernel sources moved from sys/netinet to
	dist/sys/ipf/netinet. Due to the move some Makefile
	dependencies are now dangling requiring a make cleandir
	before they work again (kdump, ktruss, rescue, ipf, and
	ftp-proxy are the victims).

20040715:
	The rc.d/sendmail script now uses a heuristic to determine
	if sendmail should be started at boot time.  It checks the
	contents of /etc/mailer.conf, /etc/mail/submit.cf, and the
	owner and mode of the sendmail binary to see if any changes
	to the mail infrastructure have been made.  If no changes
	are detected, it will start an SMTP listener.

	Setting sendmail=NO in /etc/rc.conf will override this.

	If you are only using sendmail by default and only for
	local delivery, it is important that you also update your
	sendmail.cf so that the SMTP listener only listens on the
	loopback interface.

20040715:
	The method by which athhal-elf.o gets pulled into i386
	kernel builds has been changed again.  The latest version
	of bsd.files.mk is no longer required.

20040621:
	Due to the recent rototill of tools/compat it's crucial one starts
	from a clean objdir under tools/*.

	This is mostly due to generated files (yacc and lex sources) needing
	to be generated with new rules from bsd.hostprog.mk.

	The safest course is to rm -rf all objects under tools before building.

20040516:
	The end-user modifiable X11 configuration has been moved
	from /usr/X11R6/lib/X11/<dir> to /etc/X11/<dir>.
	Ensure that src and xsrc is up to date, and run
	"make cleandir" in src/x11 before your next build.

	postinstall currently doesn't migrate the files from
	/usr/X11R6/lib/X11/* to /etc/X11/* although it does
	detect that this needs to occur, so you'll have to
	manually move these files yourself.

20040426:
	Support for the original dynamic sysctl node structure has
	been removed in favor of the newer layout.  This affects
	consumers of the create and delete interface, as well as
	the dynamic discovery mechanism.  This is believed only to
	be the sysctl(8) binary itself, at this point in time, so
	the only effect of this should be that a sysctl binary
	built from sources dated between 2003/12/04 and 2004/03/24
	will not work on a kernel built from sources dated after
	2004/04/25.  If you need a new sysctl binary but build.sh
	does not work, make sure that your revision of
	src/sys/sys/sysctl.h is 1.112 (or later), and then the
	do the following:

	cd /usr/src (or wherever your source tree is)
	make USETOOLS=no includes
	cd lib/libc
	make USETOOLS=no dependall install
	cd ../../sbin/sysctl
	make USETOOLS=no dependall install

	If you are using older sysctl binary, GNU autoconf would fail to
	identify your machine architecture, and tries to build binary for
	"unknown-unknown-netbsd20F" or something like that.  if that happens,
	make sure to follow the above steps.

20040425:
	The ffs superblock issues listed below under 20040109 and 20030402
	are now automatically addressed by the /etc/rc.d/fixsb script or by
	sysinst when it checks a a filesystem. The manual fsck_ffs -b16 -c4
	invocation mentioned below will continue to work and is now
	automated by those scripts.  Note that under certain circumstances,
	affected filesystems upgraded to a -current kernel first before
	upgrading their userland with the fixsb and fsck_ffs fixes may
	encounter a 'freeing free inode' panic when writing to the affected
	filesystem, so it is a good idea to repair the filesystem as soon as
	possible.  For more details on the fixsb script, see pr install/25138.

20040418:
	statfs(2) and friends have been replaced with statvfs(2). Before
	installing a newly build userland make sure that you are running
	a newly built kernel with COMPAT_20 set. In addition your libc
	build might not work (undefined SYS_statfs symbol) because make
	clean does not know how to remove files it does not know about
	anymore. Manually remove all generated .S sources and objects
	from the libc build directory.

20040326:
	The method by which athhal-elf.o gets pulled into i386 kernel builds
	has been changed. The file is now stored as a uuencoded file in CVS
	and the generated Makefile will use the new .uue rules from bsd.file.mk
	to build it.

	This means you must have the latest bsd.files.mk installed when
	building a kernel without USETOOLS=yes.

20040318:
	A bug in the cgd(4) blowfish code was corrected, without
	provision of backwards compatibility, after several public
	notices over several months. Users of cgd with blowfish cipher
	ONLY must dump their data before updating their kernels, and
	recreate cgd's and restore data using the new kernel. See
	(recent message to current-users, URL when mail-index has updated)

20040313:
	On acorn32, the opms and qms drivers have been withdrawn, and
	the old wsqms driver is now called qms.  Kernel config files
	will need updating.  See the ones in sys/arch/acorn32/conf
	for examples.

20040125:
	On acorn32, the beep and sysbeep devices are no longer
	needed, and will need to be deleted from kernel configuration
	files.

20040109:
	Compatibility for old ffs superblock layouts has been
	added, and the restrictive fsck checks have been reenabled
	when using those layouts.  If you have been using -current
	since 20030402, you may find that fsck again signals fatal
	superblock mismatches.  To repair, make sure you have
	an updated fsck_ffs and then you can use fsck_ffs -b 16 -c 4
	to complete the filesystem upgrade.  A message has
	been added to the kernel which should detect this problem.
	See the following discussion for more information:
	http://mail-index.NetBSD.org/current-users/2004/01/11/0022.html

20031203:
	New binutils builds may fail due to old dependencies.
	It's necessary to "make cleandir" to ensure that
	the dependencies will be rebuilt correctly.

20031111:
	A newer mkdep is needed.  Error noting that is
		cc: Ambiguous abbreviation --

20031008:
	/usr/include/sys/disklabel_mbr.h was removed.
	It's necessary to "make cleandir" to ensure that
	the dependencies will be rebuilt correctly.

20031007:
	A sign exension bug was fixed which set all the high bits
	of our newly expanded ffs fs_flags.  This should only
	affect users who installed or upgraded in September of 2003.
	A small utility program was posted to tech-kern which
	should fix this problem, and a warning message was added
	to the kernel which should discover and warn about it.  See
	http://mail-index.NetBSD.org/tech-kern/2003/10/07/0005.html

20030906:
	With the addition of siginfo support the old signal trampoline
	code has been deprecated to COMPAT_16. Make sure that your running
	kernel has COMPAT_16 enabled before building userland.

20030801:
	With the new openssl, there is some header and library shuffling.
	rm -f /usr/include/des.h /usr/include/kerberosIV/* /lib/libdes* \
	/usr/lib/libdes* before building.

20030703:
	Texinfo was updated to 4.6.  To avoid failures when trying to
	build the included texinfo files, do:

	cd src/gnu/usr.bin/texinfo
	make MKINFO=no dependall install

20030630:
	Groff was updated to 1.19; it's probably necessary to do
		cd share/mk && make install
		cd src/gnu/usr.bin/groff
		make MKMAN=no dependall install
	(untested).

20030516:
	Due to bugs in the export handling code, invalid export lines
	were accepted before and caused the kernel to panic when
	mountd got restarted because it freed memory that had already
	been freed. This has been fixed and the kernel checks
	export addresses very strictly. If you upgrade your kernel,
	make sure you also upgrade mountd, because if your export
	file contains lines with an old inet4 address syntax (i.e.
	a.b.c or a.b or a), they will get rejected by the new kernel.

20030402:
	The superblock layout for FFS was changed.  If you have 1.6
	fsck binaries, they will signal a fatal superblock mismatch
	with the first alternate, because they compare too many
	fields (even ones that aren't useful).  If possible, upgrade
	your fsck_ffs binary before using a new kernel.
	None of this signals actual filesystem damage.

20030324:
	sendmail version 8.12.8 was imported.  Since sendmail is
	now setgid to the smmsp group, and runs in "collection"
	mode for most common activities, there is a new config
	file called submit.cf that needs to live in /etc/mail.
	The generic submit.cf sample in /usr/share/sendmail/cf
	is named netbsd-msp.cf.  Upgrading your regular sendmail
	configuration file is also strongly advised.

	See the section named "MESSAGE SUBMISSION PROGRAM" in
	the updated /usr/share/sendmail/README file for more
	information.

20030117:
	Texinfo was updated to 4.3.  To avoid failures when trying to
	build the included texinfo files, do:

	cd src/gnu/usr.bin/texinfo
	make MKINFO=no dependall install

20021223:
	The METALOG format changed slightly, to remove the leading
	"${DESTDIR}" from path names.
	This only affects people building with UNPRIVED.
	For complete safety, remove the DESTDIR entirely and
	update tools/mtree, before running make build.

20021219:
	CVS repository layout was changed.  See the following for details
	if you are using (anonymous) cvs to update your tree.

	http://mail-index.NetBSD.org/netbsd-announce/2002/12/19/0000.html

20021219:
	install(1) had a '-N dbdir' option added, to specify an
	alternate location to look up users & groups (instead
	of the host system passwd(5) and group(5) databases).

	The build system was modified to take advantage of
	this option (using ${NETBSDSRCDIR}/etc), so if you
	use USETOOLS==no, you may have to rebuild and
	reinstall usr.bin/xinstall first.

20021130:
	fparseln(3) moved from libutil to libc.
	If building to DESTDIR=/, reinstall the includes
	and rebuild libc:
		make includes
		make do-lib-libc
	If using build.sh, "cd tools/compat && make clean"
	before rebuilding the tools.

20021126:
	The mk.conf(5) variable SYS_INCLUDE has been deprecated,
	including the optional "SYS_INCLUDE=symlinks" support.
	All header files, including <sys/*.h> are copied into
	/usr/include.

20021121:
	The C run-time support files crtbegin.o and crtend.o
	(and their companions crtbeginS.o and crtendS.o) were
	split up, with new crti.o and crtn.o files resulting.
	This means that libtool needs to be rebuilt once the
	new libraries are installed.  The process of rebuilding
	libtool will cause it to automatically notice the new
	required files, but it *must* be rebuilt in order to
	do this.

	An out-of-date libtool will result in shared libraries
	which lack _init() and _fini() routines, which means that
	their global contructors/destructors will not be invoked.

20021121:
	A bug related to how ARM ELF objects were tagged has been
	corrected.

	NetBSD ARM ELF uses the soft-VFP floating point model by
	default.  However, the assembler lacked support for marking
	objects as using the VFP floating point format, and the
	compiler was not properly passing the flag indicating "soft-VFP"
	to the assembler.

	Unfortunately, this means that the linker will now consider
	old (i.e. not marked "softvfp") NetBSD ARM ELF objects to be
	incompatible with new (properly marked) objects.

	The problem will only manifest itself if you attempt to compile
	a new program using the fixed toolchain, and link that program
	against old libraries which do not have the proper "softvfp"
	markings.  ALL OF YOUR EXISTING BINARIES AND SHARED LIBRARIES
	WILL CONTINUE TO WORK PROPERLY.

	The only work-around for the problem is to recompile all of
	the libraries on the system.  The easiest way to do this for
	system libraries is to install a binary snapshot; they are
	generally available on releng.NetBSD.org.  Any packages you
	have installed which supply libraries will have to be recompiled
	if you wish to link new programs against those libraries.

	If you have questions about this matter, please contact
	port-arm (a] NetBSD.org.

20021011:
	Systrace has been improved to support privilege elevation.
	Updating the kernel requires the userland part of systrace
	to be rebuilt.

20021010:
	The config(8) grammar was changed to allow options to register
	dependencies on attributes, as well as other options.  Users
	must update and reinstall usr.sbin/config before building a new
	kernel.

20021009:
	A new attribute dependency syntax was introduced to config(8),
	which is now used by the SCSI configuration description.  Users
	must update and reinstall usr.sbin/config before building a new
	kernel.

20021003:
	Several changes have been made to the autoconfiguration
	framework.  Users must update and reinstall usr.sbin/config
	before building a new kernel.

20021001:
	The i386mp branch has been merged.  To compile a kernel, users
	will need to add the option 'cpu* at mainbus?' to their configuration
	file.  Multiprocessor kernels will need
	ioapic*		at mainbus? apid ?
	options		MULTIPROCESSOR
	options		COM_MPLOCK

20020922:
	MKDYNAMICROOT=yes enabled by default, which means that
	certain shared libraries are installed into /lib, the shared
	linker is installed into /libexec, and all programs in /bin
	and /sbin are dynamically linked.
	If you do not use "make build", you should ensure that
	you have the libraries and shared linker in the new locations,
	with:
		make do-lib-csu do-lib-libc do-lib do-gnu-lib do-ld.elf_so

20020917:
	USE_NEW_TOOLCHAIN has been replaced with:
	    -	TOOLCHAIN_MISSING -- set to "yes" on platforms for which
		there is no working in-tree toolchain (hppa, ns32k, sh5,
		x86_64).
	    -	EXTERNAL_TOOLCHAIN -- if defined by the user, points to the
		root of an external toolchain (e.g. /usr/local/gnu).  This
		enables the cross-build framework even for TOOLCHAIN_MISSING
		platforms.

20020906:
	gehenna-devsw has been merged into the trunk. Need to update and
	reinstall usr.sbin/config before building the kernel.

20020822:
	Crunched rescue tools (contents of /bin and /sbin, plus others)
	are now provided in /rescue.

	To ensure that these are built statically linked (no matter
	what the setting of LDSTATIC is), use a crunchgen(1) built
	from sources newer than 20020820 (see the next entry).

20020820:
	crunchgen(1) changed to ensure that the generated program
	is statically linked.

	Solution: update and reinstall usr.bin/crunch

20020605:
	smmsp user/group has been added for sendmail.

	Add the following into /etc/group:

	smmsp:*:17:

	and the following to /etc/master.passwd (via vipw):

	smmsp:*:17:17::0:0:Sendmail Message Submission Program:/nonexistent:/sbin/nologin

20020515:
	sshd user/group has been added.  Need to hand add this in, or sshd
	will not let you log in (with default, or UsePrivlegeSeparation=yes)

	Add the following into /etc/group:

	sshd:*:16:

	and the following to /etc/master.passwd (via vipw):

	sshd:*:16:16::0:0:& pseudo-user:/var/chroot/sshd:/sbin/nologin
	
	Also /var/chroot/sshd directory needs to be present (digged as part of
	the build process).

20020426:
	NBUILDJOBS obsoleted in favor of just using -j.

20020426:
	etc/postinstall added, which performs various checks for 
	configuration file updates and changes, and can fix most of
	the problems identified.
	This should make it much easier to upgrade a system's
	configuration from earlier systems (as far back as NetBSD 1.5).

20020320:
	<bsd.lib.mk> needs a new install(1) for its "-a cmd" support.
	Build and install at usr.bin/xinstall before the build.

20020319:
	Raw IPv6 socket now makes strict checking for sa_family and sa_len
	on send(2) operation.  Be sure to have sbin/rtsol and usr.sbin/rtsold
	newer than November 2001 when you upgrade the kernel.

20020311:
	ssh configuration files were moved from /etc to /etc/ssh.  Beware
	if you restart your machine from remote.  Note that sshd.conf needs
	to be changed (due to the use of "/etc" inside).

20020223:
	Users of the VAX port will need to rebuild and install gas
	so it deal with the now present register prefix used in all
	the VAX assembly files.

20020118:
	ntpd user/group has been added.  Need to hand add this in or builds
	will break as mtree aborts early.

	Add the following into /etc/group:

	ntpd:*:15:

	and the following to /etc/master.passwd (via vipw):

	ntpd:*:15:15::0:0:Ntpd pseudo-user:/var/chroot/ntpd:/sbin/nologin

20011207:
	If you're attempting to build a snapshot on sparc64 and are getting
	reloc errors from the toolchain groff binary this means your native
	toolchain has some broken C++ bits.

	To fix:

	Build a new toolchain (i.e. build.sh -t)
	Use the new toolchain to build and install natively (i.e. /usr/lib)

	gnu/lib/libgcc
	gnu/lib/libstdc++

	After this a snapshot will be able to be built.

20011201:
	In order for a sparc64 build to work you must have a working awk. If
	you've built and installed a system with the new toolchain up to this
	point you do not have a working awk as its ability to do floating
	point is broken. 

	To build:

	remake and install gnu/lib/libgcc
	remake and install gnu/usr.bin/gawk into /usr/bin (make sure it links
	against the new libgcc.a)

20011128:
	Kernel config information was changed to use defflag in
	the various "files" files.  Bug fixes to config(8) are
	required in order for this to work properly.  Make sure
	to build and install in usr.sbin/config before attempting
	to build a new kernel.

20011030:
	libc/locale/wcstod.c now needs new lint(1). Update lint(1)
	before building libc.

20011029:
	The new document BUILDING.mdoc (view with nroff | more, or
	see pre-generated .txt and .html versions) describes the build
	procedure in great detail.  BUILDING, and the USE_NEW_TOOLCHAIN
	build process, are intended in the long run to replace this
	manual update log.

	Users building a USE_NEW_TOOLCHAIN system should read the
	BUILDING document for caveats.  Generally, BUILDING supersedes
	UPDATING for these systems, as tool updating is taken care of
	by the new build system.

20011028:
	src/etc/Makefile now needs install to be able to handle
	symlinks that point to nowhere. A bug in install that
	prevented this was corrected.

	Solution: update and reinstall usr.bin/xinstall
	Better Solution: Use the new toolchain and it will just work
	for you.

20011006:
	/etc/mtree/NetBSD.dist has been updated to take advantage of
	absolute path support added to mtree(8). Older mtree(8)s don't
	understand the format.

	Solution: update and reinstall usr.sbin/mtree

20011004:
	Crunchgen has been updated to work via reach-over makefiles. Updating
	is suggested before running a snapshot build

20010915:
	The new "ubcperf" code committed by Chuck Silvers removed
	a header file, uvm/uvm_vnode.h.  There may be stale .depend
	files that still reference this file.

	Solution: "make cleandir && make dependall" in affected
	directories.

20010803:
	grep.info is now built from grep.texi using makeinfo.  Since it
	requires makeinfo v4.0, you need to install new texinfo before
	building gnu/usr.bin/grep.  To install new texinfo, please follow
	the instruction described in 20010726 entry.

20010803: 
	(i386 only): i386 kernel now uses new instructions like
	`fxsave' which old gas doesn't understand.  To build the
	kernel successfully, you need to build and install a new toolchain, 
	(i.e., build.sh -t) or 	(temporarily) comment out "options I686_CPU" 
	from your kernel configuration until you rebuild your userland.
	See 20011029 above and BUILDING file in this directory for more information.
	[updated 20020630 since i386 gas moved when USE_NEW_TOOLCHAIN enabled]

20010731:
	Bootloader update on ELF platforms.  DDB in kernels from before
	this will be unable to read symbol tables provided by newer
	bootloaders.

20010726:
	Texinfo was updated to 4.0.  To avoid failures when trying to
	build the included texinfo files, do:

	cd src/gnu/usr.bin/texinfo
	make MKINFO=no dependall install

20010718:
	Enabled correct .init/.fini processing in crt0.  The way this
	was done was to change a -I directive to cc(1), which means
	make(1) will have a stale dependency (it will be checking the
	timestamp on the wrong "dot_init.h").

	The symptom you will see is that new programs die with SIGSEGV
	if you have a stale dependency.

	Solution: "make cleandir" in both lib/csu and libexec/ld.elf_so
	before starting your build.

20010628:
	A construct was added to uvm_page.h that uncovered a bug
	in lint(1).  If you get a warning/error about a non-portable
	bitfield, update your lint(1) before proceeding.

20010226:
	Added named user/group to system. Need to hand add this in or builds
	will break as mtree aborts early.

	To work around add by hand:

	named:*:14:

	to /etc/group and add:

	named:*:14:14::0:0:Named pseudo-user:/var/named:/sbin/nologin

	to master.passwd (use vipw for instance if doing by hand).

	Now a make build should progress.

20010219:
	get/setprogname() added. Any hostprogs that may use this will need
	to be bootstrapped manually until the host system is current.

	Known problems: sys/arch/macppc/stand/fixcoff
			usr.sbin/config (adding -DMAKE_BOOTSTRAP to
			  CFLAGS and rebuilding should work)
			usr.sbin/mdsetimage - Build a static copy if
			  building a snapshot before fully bootstrapped.

20010204:
	prepare the code to compile with stricter gcc flags. in
	particular start eliminating redundant declarations. Yacc
	needs to be installed before make build.

20010114:
	introduce .if commands(target) in make(1). You need to
	bring everything up-to-date first, then without installing
	anything make and install in usr.bin/make, then proceed
	with make build.

20010101:
	bsd.subdir.mk committed 20001230 had a bug which caused
	afterinstall targets to run too soon; update again.

20001230:
	New share/mk files needed to support .WAIT in SUBDIR variables.
	If you get make errors, 
		(cd share/mk; make install)
	Also, PRINTOBJDIR has changed and is now used more heavily.

20001019:
	The `ca' device driver has been replaced by `ld'; although the
	major and minor numbers haven't changed, you should update your /dev
	directory.

20000929:
	The following make directives are obsoleted.
	MKCRYPTO_RSA NOCRYPTO_RSA NOCRYPTO_RC5 NOCRYPTO_IDEA 
	By default, RSA is built into libcrypto.  IDEA and RC5 will not be
	built into libcrypto.  By using MKCRYPTO_{RC5,IDEA}, you can build
	additional library libcrypto_{idea,rc5}.


Hints for a more successful build:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    Use build.sh, but do not use its "expert mode":
	This will automatically build the tools in the
	   correct order, and it will keep the tools and the
	   new build products from interfering with the running
	   system.  This will allow you to ignore most of the
	   other advice in this file.
    Build a new kernel first:
	This makes sure that any new system calls or features
	   expected by the new userland will be present.  This
	   helps to avoid critical errors when upgrading.
    Use object directories:
	This helps to keep stale object
	   files from polluting the build if a Makefile "forgets"
	   about one.  It also makes it easier to clean up after
	   a build.  It's also necessary if you want to use the
	   same source tree for multiple machines.
	   To use object directories with build.sh:
	    a) invoke build.sh with the "-M" or "-O" options.
	   To use object directories without using build.sh:
	    a) cd /usr/src ; make cleandir
	    b) Add "OBJMACHINE=yes" to /etc/mk.conf
	    c) Add "MKOBJDIRS=yes" to /etc/mk.conf
	    d) cd /usr/src ; make build
	   Note that running "make obj" in a directory will create
	   in obj.$MACHINE directory.
    Build to a DESTDIR:
	This helps to keep old installed files (especially libraries)
	   from interfering with the new build.
	   To build to a DESTDIR with build.sh, use the "-D" option.
	   To build to a DESTDIR without using build.sh, set the DESTDIR
	   environment variable before running make build.  It should be
	   set to the pathname of an initially empty directory.
	   Problems: if you do not use build.sh, you might need to
		update critical utilities without using DESTDIR since
		nothing is executed from what is installed in DESTDIR.
		(See critical utils, below.)
    Build often:
	This keeps critical utilities current enough to not choke
	on any other part of the source tree that depends on up to
	date functionality.  If you use build.sh, you should not have
	this problem.
 
What to do if things don't work:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When things don't work there is usually a few things that commonly
should be done.
    1)	make includes
	This should be done automatically by make build.
    2)  cd share/mk && make install
	Again, automatically done by make build.

Failsafe rebuild of a small part of the tree:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To make sure you rebuild something correctly you want to do
something like the following:
    1)  Make sure the includes and .mk files are up to date.
    2)  Make sure any program used to build the particular
	utility is up to date.  (yacc, lex, etc...)
    3)  cd ...path/to/util...
	make cleandir
	rm ...all obj directories...
	make cleandir			# yes, again
	make obj
	make depend && make

Failsafe rebuild of the entire tree:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you really want to make sure the source tree is clean and
ready for a build try the following.  Note that sourcing /etc/mk.conf
(a make(1) Makefile) in this manner is not right, and will not work
for anyone who uses any make(1) features in /etc/mk.conf.

---cut here---
#!/bin/sh
. /etc/mk.conf

if [ -z $NETBSDSRCDIR ] ; then
    NETBSDSRCDIR=/usr/src
fi
if [ \! -d $NETBSDSRCDIR ] ; then
    echo Unable to find sources
    exit 1
fi
find $NETBSDSRCDIR -name \*.o -o -name obj.\* -o -name obj -exec rm \{\} \;

if [ -z $BSDOBJDIR ] ; then
    BSDOBJDIR=/usr/obj
fi
if [ -d $BSDOBJDIR ] ; then
    rm -rf $BSDOBJDIR
fi

cd $NETBSDSRCDIR && make cleandir

---cut here---

Critical utilities:
^^^^^^^^^^^^^^^^^^^
	gnu/usr.bin/egcs
	usr.bin/compile_et
	usr.bin/make
	usr.bin/yacc
	usr.bin/lex
	usr.bin/xlint
	usr.sbin/config

Other problems and possible solutions:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Symptom:Unreasonable compiler errors.
Fix:	Rebuild gnu/usr.bin/egcs

Symptom:Complaints involving a Makefile.
Fix:	Rebuild usr.bin/make:
	cd usr.bin/make && make && make install
	Or, a failsafe method if that doesn't work:
	cd usr.bin/make && cc *.c */*.c -I . -o make && mv make /usr/bin

Fix:	Make sure .mk files are up to date.
	cd share/mk && make install

Symptom:Kernel `config' fails to configure any kernel, including GENERIC.
Fix:	Rebuild usr.sbin/config

Symptom:
Fix:	Rebuild usr.bin/yacc

Symptom:
Fix:	Rebuild usr.bin/lex

Symptom:
Fix:	rm /usr/lib/libbfd.a

Symptom:Obsolete intermediate files are used during compilation
Fix:	Try the following sequence of commands in the directory in question.
	make cleandir; rm `make print-objdir`; make cleandir; make obj
	(If you built the tree without "make obj" in the past, obsolete files
	may remain.  The command tries to clean everything up)

Symptom:.../sysinst/run.c:xx: warning: initialization from incompatible pointer type
Fix:	Rebuild and install usr.bin/menuc

Symptom:mklocale not found during build in share/locale/ctype
Fix:	Build and install usr.bin/mklocale

Symptom:undefined reference to `__assert13' or `__unsetenv13'
Fix:    Rebuild and install lib/libc

Symptom:usr.sbin/config fails to build.
Fix:	Try building with -DMAKE_BOOTSTRAP added to CFLAGS in Makefile.

Symptom:undefined reference to `getprogname' or `setprogname'
Fix:    Rebuild and install lib/libc

Symptom:lint does not understand the '-X' option
Fix:    May need to build & install libs with NOLINT=1 before rebuilding lint