remove GCC_NO_ADDR_OF_PACKED_MEMBER for several subdir builds
that are now handled by lfs_accessors.h internally.

add support for new GCC 9 warnings that may be too much to fix
right now. new address-of-packed-member and format-overflow
warnings have new GCC_NO_ADDR_OF_PACKED_MEMBER amd
GCC_NO_FORMAT_OVERFLOW variables to remove these warnings.

apply to a bunch of the tree. mostly, these are real bugs that
should be fixed, but in many cases, only by removing the 'packed'
attribute from some structure that doesn't really need it. (i
looked at many different ones, and while perhaps 60-80% were
already properly aligned, it wasn't clear to me that the uses
were always coming from sane data vs network alignment, so it
doesn't seem safe to remove packed without careful research for
each affect struct.) clang already warned (and was not erroring)
for many of these cases, but gcc picked up dozens more.

Use llabs(), not abs(), on the difference of daddr_t's.
Caught by clang, reported by joerg.

Turn off the clang-specific compiler flags that were making the report
nonfatal.

Make the abs() use check non-fatal for a bogus use in the LFS cleanerd.

Tidy up the LFS userland build hacks.
Don't use -I${NETBSDSRCDIR}/sys; don't include files other than the
exported LFS headers, which are lfs.h, lfs_inode.h, and (for now)
lfs_extern.h.

Cleanups and hacks to make lfs userland stuff build:
- lfs_cksum.c doesn't actually need ulfs_inode.h any more.
- neither does lfs_itimes.c.
- add hacks to fsck_lfs to make it compile.
- add hacks to newfs_lfs to make it compile.
- fix warning in ulfs_quota.c when quotas are fully disabled
(as I guess is happening with the rumpity version)

XXX: This commit adds -I${NETBSDSRCDIR}/sys to the Makefiles for
XXX: fsck_lfs, newfs_lfs, and lfs_cleanerd. This needs to be cleaned
XXX: up ASAP; but I consider this less problematic in the short term
XXX: than spewing ulfs_*.h into /usr/include.

branches: 1.15.6; 1.15.12;
Define syscalls of lfs userspace tools (cleaner, mainly) through
a struct called kernelops, which contains standard system calls
for the normal case and rump system calls for the rump case.

Make it possible to run the lfs cleaner in a library fashion (taking
the quick route with the implementation).

PR/40965: NAKAJIMA Yoshihiro: lfs_cleanerd isn't adapted to !MKDYNAMICROOT

Default to WARNS=4
Exceptions that need a lower level are:
getty ld.elf_so lfs_cleanerd makewhatis telnetd tftpd
hpropd ipropd-master ipropd-slave kadmind kpasswdd

branches: 1.12.12;
Make it build with src/sys from the vmlocking2 branch.

branches: 1.11.8;
Move lfs_cleanerd from /usr/libexec to /libexec. This is to allow putting
the root file system on a LFS volume.

Addresses PR bin/30407. No objections in tech-userlevel@.

A new version of the cleaner. In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

Use ${NETBSDSRCDIR}/some/path instead of ${.CURDIR}/../../some/path

First stab at file coalescing. When the cleaner detects that it might
be digging itself deeper into a hole, it forks off a subprocess
that locates files with too many discontinuities and rewrites them, if
there is enough room.

Optionally the user can manually coaleasce files by running with "-c".

The recent change to lfs_markv is required for the coalescer to do anything.

All of "digging itself deeper", "too many discontinuities", and "enough room"
need to be better defined.

Try to prevent running more than one active cleaner on a filesystem at a time.

Let lfs_cleanerd record its pid in /var/run like other daemons. Make
mount_lfs not start another cleaner when updating the mount, unless it is
being upgraded from read-only to read-write; when downgrading to read-only,
kill the cleaner using the recorded pids.

branches: 1.6.10;
use CPPFLAGS instead of CFLAGS

branches: 1.5.2;
merge lite-2 Makefiles (rcsids), and turn on WARNS for all of libexec.

WARNS?=1. RCS ids.

specify man pages the new way.

do not need -I/sys

branches: 1.1.1;
From 4.4-Lite, with local changes.

bsd.own.mk: rename GCC_NO_* to CC_WNO_*

Rename compiler-warning-disable variables from
GCC_NO_warning
to
CC_WNO_warning
where warning is the full warning name as used by the compiler.

GCC_NO_IMPLICIT_FALLTHRU is CC_WNO_IMPLICIT_FALLTHROUGH

Using the convention CC_compilerflag, where compilerflag
is based on the full compiler flag name.

libexec/lfs_cleanerd: Suppress -Werror=stringop-truncation error

This logic correctly uses strncpy(3) to fully initialize a fixed-width field, and also ensures
NUL-termination on the next line as other users of the field expect.

Add -Werror=stringop-truncation to prevent build failure, when run with MKSANITIZER=yes.

Error was reported when build.sh was run with MKSANITIZER=yes flag.

Reviewed by: kamil@, riastradh@

Missed one of the -I${NETBSDSRCDIR}/sys hacks yesterday; remove it now.

Redo these changes properly:
-r1.12 libexec/lfs_cleanerd/Makefile
-r1.15 sbin/fsck_lfs/Makefile
-r1.6 sbin/newfs_lfs/Makefile

hi ad@

Cleanups and hacks to make lfs userland stuff build:
- lfs_cksum.c doesn't actually need ulfs_inode.h any more.
- neither does lfs_itimes.c.
- add hacks to fsck_lfs to make it compile.
- add hacks to newfs_lfs to make it compile.
- fix warning in ulfs_quota.c when quotas are fully disabled
(as I guess is happening with the rumpity version)

XXX: This commit adds -I${NETBSDSRCDIR}/sys to the Makefiles for
XXX: fsck_lfs, newfs_lfs, and lfs_cleanerd. This needs to be cleaned
XXX: up ASAP; but I consider this less problematic in the short term
XXX: than spewing ulfs_*.h into /usr/include.

branches: 1.3.6; 1.3.12;
always add to SRCS

LDADD+= instead of LDADD=

Define syscalls of lfs userspace tools (cleaner, mainly) through
a struct called kernelops, which contains standard system calls
for the normal case and rump system calls for the rump case.

Make it possible to run the lfs cleaner in a library fashion (taking
the quick route with the implementation).

A new version of the cleaner. In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

64 bit inode changes.

Various minor LFS improvements:

* Adapt lfs_cleanerd to use the fcntl call to get the Ifile filehandle,
so it need not be in the namespace.
* Make lfs_cleanerd be more careful when there are very few available
segments.
* Remove the Ifile from the filesystem namespace. The cleaner now uses
a fcntl call on the root inode to find the Ifile filehandle.
* Make lfs_cleanerd less verbose when the filesystem is unmounted.

Replace the statfs() family of system calls with statvfs().
Retain binary compatibility.

branches: 1.18.4;
Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22284, verified by myself.

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Convert lfs_cleanerd over to use the new ioctl calls instead of the
lfs syscalls.

Bump daddr_t to 64 bits. Replace it with int32_t in all places where
it was used on-disk, so that on-disk formats remain the same.
Remove ufs_daddr_t and ufs_lbn_t for the time being.

Don't try to coalesce files that have fewer than NDADDR blocks, due to
a potential problem with cleaning fragments at all.

Better sanity checks when selecting files to coalesce; in particular don't
shift too far left when comparing the number of discontinuities to the log2
of the number of total blocks.

Better log messages: note beginning of coalescing correctly; also take
the log message from add_segment out of "if (debug)" for symmetry with the
"finished segment" message.

Use lfs_bmapv to find the inode, rather than looking it up manually in
the ifile; this should give more up-to-date information, since trolling
through every inode in the fs could take some time.

First stab at file coalescing. When the cleaner detects that it might
be digging itself deeper into a hole, it forks off a subprocess
that locates files with too many discontinuities and rewrites them, if
there is enough room.

Optionally the user can manually coaleasce files by running with "-c".

The recent change to lfs_markv is required for the coalescer to do anything.

All of "digging itself deeper", "too many discontinuities", and "enough room"
need to be better defined.

Handle segment 0 properly, if its offset is different from other segments
because of the disklabel.

Fix a problem with inode block handling that sometimes caused the wrong
blocks to be read, causing either cleaning failures or panics with v2 file
systems.

Merge the short-lived perseant-lfsv2 branch into the trunk.

Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default. Changes for the v2 layout include:

- Segments of non-PO2 size and arbitrary block offset, so these can be
matched to convenient physical characteristics of the partition (e.g.,
stripe or track size and offset).

- Address by fragment instead of by disk sector, paving the way for
non-512-byte-sector devices. In theory fragments can be as large
as you like, though in reality they must be smaller than MAXBSIZE in size.

- Use serial number and filesystem identifier to ensure that roll-forward
doesn't get old data and think it's new. Roll-forward is enabled for
v2 filesystems, though not for v1 filesystems by default.

- The inode free list is now a tailq, paving the way for undelete (undelete
is not yet implemented, but can be without further non-backwards-compatible
changes to disk structures).

- Inode atime information is kept in the Ifile, instead of on the inode;
that is, the inode is never written *just* because atime was changed.
Because of this the inodes remain near the file data on the disk, rather
than wandering all over as the disk is read repeatedly. This speeds up
repeated reads by a small but noticeable amount.

Other changes of note include:

- The ifile written by newfs_lfs can now be of arbitrary length, it is no
longer restricted to a single indirect block.

- Fixed an old bug where ctime was changed every time a vnode was created.
I need to look more closely to make sure that the times are only updated
during write(2) and friends, not after-the-fact during a segment write,
and certainly not by the cleaner.

branches: 1.10.2;
remove redundant declarations.

If the child cleaner dies repeatedly without doing anything, give up. Uses
similar logic to inetd to identify such looping.

Make datobyte do its arithmetic explicitly in 64 bits, so that segments
beyond the first 2G of disk can be cleaned.

branches: 1.7.2; 1.7.6;
Cleaner changes corresponding to in-kernel LFS changes. In particular, the
cleaner understands fragments; and it knows to change bi_bp to reflect a
change in bi_daddr, if lfs_bmapv says that there is one.

PR/6248: Konrad Schroder: lfs_cleanerd could use syslog() for notices,
instead of err()/warn()

type fixes, in part from PR#6032

Merge with Lite2 + local changes

WARNS?=1. RCS ids.

WARNS?=1. RCS ids.

branches: 1.1.1;
From 4.4-Lite, with local changes.

Add new accessors for the d_type and d_namlen fields of struct lfs_direct.
Napalm the old byteswap access logic for these.

Add byteswapping to the dinode accessors.

This prevents regressions in the ulfs code when switching to the new
accessors. Note that while adding byteswapping to the other accessors
is straightforward, I haven't done it yet; and that also is not enough
to make LFS_EI work, because there are places lying around that bypass
the accessors for one reason and another and all of them need to be
updated. That is going to have to wait for a later day as LFS_EI is
not on the critical path right now.

Make 32-bit and 64-bit versions of CLEANERINFO.

XXX: while this is written to disk, it seems like much of it would
XXX: be better set up as a commpage shared with the cleaner.

Fix assorted 64->32 truncations related to BLOCK_INFO.

Also make note of a cleaner limitation: it seems that when it goes to
coalesce discontiguous files, it mallocs an array with one BLOCK_INFO
for every block in the file. Therefore, with 64-bit LFS, on a 32-bit
platform it will be possible to have files large enough to overflow
the cleaner's address space. Currently these will be skipped and cause
warnings via syslog.

At some point someone should rewrite the logic to coalesce files to
use chunks of some reasonable size, as discontinuity between such
chunks is immaterial and mallocing this much space is silly and
fragile. Also, the kernel only accepts up to 65536 blocks at a time
for bmapv and markv, so processing more than this at once probably
isn't useful and may not even work currently. I don't want to change
this around just now as it's not entirely trivial.

Add a (draft) 64-bit superblock. Make things build again.

Add pieces of support for using both superblock types where
convenient, and specifically to the superblock accessors, but don't
actually enable it anywhere.

First substantive step on PR 50000.

Second batch of 64 -> 32 truncations in lfs, along with more minor
tidyups and corrections in passing.

Use lfs_accessors.h in conjunction with the cleaner's struct clfs.
Remove previous hacks.

Add a new lfs header file: lfs_accessors.h.

This contains all the accessor functions and macros out of lfs.h.
Add an include of lfs_accessors.h after all uses of lfs.h... except
for code that wants to define its own struct lfs-alike that the
accessors are supposed to play along with. For these, set STRUCT_LFS
and include lfs_accessors.h after the necessary structure has been
defined, so that lfs_accessors.h can emit functions in terms of it.

More lfs superblock accessors.
(This changes the rest of the code over; all the accessors were
already added.)

The difference between this commit and the previous one is arbitrary,
but the previous one passed the regression tests on its own so I'm
keeping it separate to help with any bisections that might be needed
in the future.

Switch to accessor functions for elements of the LFS on-disk
superblock. This will allow switching between 32/64 bit forms on the
fly; it will also allow handling LFS_EI reasonably tidily. (That
currently doesn't work on the superblock.)

It also gets rid of cpp abuse in the form of fake structure member
macros.

Also, instead of doing sleep/wakeup on &lfs_avail and &lfs_nextseg
inside the on-disk superblock, add extra elements to the in-memory
struct lfs for this. (XXX: these should be changed to condvars, but
not right now)

XXX: this migrates a structure needed by the lfs code in libsa (struct
salfs) into lfs.h, where it doesn't belong, but for the time being
this is necessary in order to allow the accessors (and the various
lfs macros and other goop that relies on them) to compile.

fix WARNS=3 issues;
* sprinkle const
* avoid tests < 0 on unsigned types
* test fwrite() returns nmemb not <0

branches: 1.4.28;
Make filehandles opaque to userland

Fix alignement problems for fhandle_t, exposed by gcc4.1.

While touching all vptofh/fhtovp functions, get rid of VFS_MAXFIDSIZ,
version the getfh(2) syscall and explicitly pass the size available in
the filehandle from userland.

Discussed on tech-kern, with lots of help from yamt (thanks!).

Be more careful about checking return value of {m,re}alloc().

In determining when to stop reading segments when counting bytes (-b flag),
total the sizes of the blocks we're actually writing instead of assuming
they are all full blocks: many could be fragments or inode blocks. This
increases the number of segments per Ifile write, markedly improving the
efficiency of the cleaner in the small file case.

A new version of the cleaner. In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

A new version of the cleaner. In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

64 bit inode changes.

Provide a resize_lfs(8), including kernel and cleaner support. The current
implementation requires the fs to be mounted while resizing. Tested in both
directions, and everything appears to work happily, but ymmv.

Add missing flags to usage()

branches: 1.52.2;
Various minor LFS improvements:

* Adapt lfs_cleanerd to use the fcntl call to get the Ifile filehandle,
so it need not be in the namespace.
* Make lfs_cleanerd be more careful when there are very few available
segments.
* Remove the Ifile from the filesystem namespace. The cleaner now uses
a fcntl call on the root inode to find the Ifile filehandle.
* Make lfs_cleanerd less verbose when the filesystem is unmounted.

Replace the statfs() family of system calls with statvfs().
Retain binary compatibility.

branches: 1.50.4;
realloc pedant

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22284, verified by myself.

use bounded string ops. check malloc failure. asprintf is easier than
complicated strcat/strcpy combination.

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Let the cleaner use LFCNRECLAIM to help empty segments along, if it
thinks it needs to clean and segments are tantalizingly lingering
in the "empty but dirty" state.

Make fs-specific fcntl macros take three arguments (approved wrstuden).
Let LFS use fcntl for cleaner functions.

Convert lfs_cleanerd over to use the new ioctl calls instead of the
lfs syscalls.

"Utilize" has exactly the same meaning as "use," but it is more
difficult to read and understand. Most manuals of English style
therefore say that you should use "use".

make this build on alpha after daddr_t->64bit

Bump daddr_t to 64 bits. Replace it with int32_t in all places where
it was used on-disk, so that on-disk formats remain the same.
Remove ufs_daddr_t and ufs_lbn_t for the time being.

for -b, use ssize instead of segshift.
segshift is invalid for v2 filesystems.

Don't try to coalesce files that have fewer than NDADDR blocks, due to
a potential problem with cleaning fragments at all.

Better sanity checks when selecting files to coalesce; in particular don't
shift too far left when comparing the number of discontinuities to the log2
of the number of total blocks.

Better log messages: note beginning of coalescing correctly; also take
the log message from add_segment out of "if (debug)" for symmetry with the
"finished segment" message.

Use lfs_bmapv to find the inode, rather than looking it up manually in
the ifile; this should give more up-to-date information, since trolling
through every inode in the fs could take some time.

First stab at file coalescing. When the cleaner detects that it might
be digging itself deeper into a hole, it forks off a subprocess
that locates files with too many discontinuities and rewrites them, if
there is enough room.

Optionally the user can manually coaleasce files by running with "-c".

The recent change to lfs_markv is required for the coalescer to do anything.

All of "digging itself deeper", "too many discontinuities", and "enough room"
need to be better defined.

branches: 1.37.2;
remove one of duplicated "bfree" from debug message.

use errx instead of err in some places
in order to avoid "Undefined error: 0"

daemon(3) should be used prior to file descriptor setups.

"than" instead of "then".

fix printf format on alpha

Handle segment 0 properly, if its offset is different from other segments
because of the disklabel.

Fix a problem with inode block handling that sometimes caused the wrong
blocks to be read, causing either cleaning failures or panics with v2 file
systems.

Merge the short-lived perseant-lfsv2 branch into the trunk.

Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default. Changes for the v2 layout include:

- Segments of non-PO2 size and arbitrary block offset, so these can be
matched to convenient physical characteristics of the partition (e.g.,
stripe or track size and offset).

- Address by fragment instead of by disk sector, paving the way for
non-512-byte-sector devices. In theory fragments can be as large
as you like, though in reality they must be smaller than MAXBSIZE in size.

- Use serial number and filesystem identifier to ensure that roll-forward
doesn't get old data and think it's new. Roll-forward is enabled for
v2 filesystems, though not for v1 filesystems by default.

- The inode free list is now a tailq, paving the way for undelete (undelete
is not yet implemented, but can be without further non-backwards-compatible
changes to disk structures).

- Inode atime information is kept in the Ifile, instead of on the inode;
that is, the inode is never written *just* because atime was changed.
Because of this the inodes remain near the file data on the disk, rather
than wandering all over as the disk is read repeatedly. This speeds up
repeated reads by a small but noticeable amount.

Other changes of note include:

- The ifile written by newfs_lfs can now be of arbitrary length, it is no
longer restricted to a single indirect block.

- Fixed an old bug where ctime was changed every time a vnode was created.
I need to look more closely to make sure that the times are only updated
during write(2) and friends, not after-the-fact during a segment write,
and certainly not by the cleaner.

branches: 1.30.2;
remove redundant declarations.

avoid C sequence point issues warned about by development version of gcc.

be more consistent about syslog usage. now it's more like:
err fatal errors
warning warnings
info status messages (-d), stats on SIGxxx
debug debug messages (-d), debug stats

Don't "compress" segment data if we were using mmap instead of malloc/copy
to read the segment.

Check for ENOENT return from lfs_{bmapv,markv} and do the right thing with it,
avoiding needless looping (possibly infinite looping) on certain kinds of
errors.

Get rid of erroneous free() in error return from add_segment.

Patch from Jesse Off <joff@gci-net.com> (PR #11547).

Try to prevent running more than one active cleaner on a filesystem at a time.

Let lfs_cleanerd record its pid in /var/run like other daemons. Make
mount_lfs not start another cleaner when updating the mount, unless it is
being upgraded from read-only to read-write; when downgrading to read-only,
kill the cleaner using the recorded pids.

Don't hold every segment that is being cleaned in memory in its entirety;
instead, if the segment doesn't have many live blocks, copy them to a
more appropriately sized chunk of memory and release the original.

This should prevent the cleaner from distending itself when cleaning many
segments with only one or two live blocks each, as when using the "-b" option.

Fix memory leak described in PR #11094 (patch from Jesse Off
<joff@gci-net.com>).

Various bug-fixes to LFS, to wit:


Kernel:

* Add runtime quantity lfs_ravail, the number of disk-blocks reserved
for writing. Writes to the filesystem first reserve a maximum amount
of blocks before their write is allowed to proceed; after the blocks
are allocated the reserved total is reduced by a corresponding amount.

If the lfs_reserve function cannot immediately reserve the requested
number of blocks, the inode is unlocked, and the thread sleeps until
the cleaner has made enough space available for the blocks to be
reserved. In this way large files can be written to the filesystem
(or, smaller files can be written to a nearly-full but thoroughly
clean filesystem) and the cleaner can still function properly.

* Remove explicit switching on dlfs_minfreeseg from the kernel code; it
is now merely a fs-creation parameter used to compute dlfs_avail and
dlfs_bfree (and used by fsck_lfs(8) to check their accuracy). Its
former role is better assumed by a properly computed dlfs_avail.

* Bounds-check inode numbers submitted through lfs_bmapv and lfs_markv.
This prevents a panic, but, if the cleaner is feeding the filesystem
the wrong data, you are still in a world of hurt.

* Cleanup: remove explicit references of DEV_BSIZE in favor of
btodb()/dbtob().

lfs_cleanerd:

* Make -n mean "send N segments' blocks through a single call to
lfs_markv". Previously it had meant "clean N segments though N calls
to lfs_markv, before looking again to see if more need to be cleaned".
The new behavior gives better packing of direct data on disk with as
little metadata as possible, largely alleviating the problem that the
cleaner can consume more disk through inefficient use of metadata than
it frees by moving dirty data away from clean "holes" to produce
entirely clean segments.

* Make -b mean "read as many segments as necessary to write N segments
of dirty data back to disk", rather than its former meaning of "read
as many segments as necessary to free N segments worth of space". The
new meaning, combined with the new -n behavior described above,
further aids in cleaning storage efficiency as entire segments can be
written at once, using as few blocks as possible for segment summaries
and inode blocks.

* Make the cleaner take note of segments which could not be cleaned due
to error, and not attempt to clean them until they are entirely free
of dirty blocks. This prevents the case in which a cleanerd running
with -n 1 and without -b (formerly the default) would spin trying
repeatedly to clean a corrupt segment, while the remaining space
filled and deadlocked the filesystem.

* Update the lfs_cleanerd manual page to describe all the options,
including the changes mentioned here (in particular, the -b and -n
flags were previously undocumented).

fsck_lfs:

* Check, and optionally fix, lfs_avail (to an exact figure) and
lfs_bfree (within a margin of error) in pass 5.

newfs_lfs:

* Reduce the default dlfs_minfreeseg to 1/20 of the total segments.

* Add a warning if the sgs disklabel field is 16 (the default for FFS'
cpg, but not usually desirable for LFS' sgs: 5--8 is a better range).

* Change the calculation of lfs_avail and lfs_bfree, corresponding to
the kernel changes mentioned above.

mount_lfs:

* Add -N and -b options to pass corresponding -n and -b options to
lfs_cleanerd.

* Default to calling lfs_cleanerd with "-b -n 4".


[All of these changes were largely tested in the 1.5 branch, with the
idea that they (along with previous un-pulled-up work) could be applied
to the branch while it was still in ALPHA2; however my test system has
experienced corruption on another filesystem (/dev/console has gone
missing :^), and, while I believe this unrelated to the LFS changes, I
cannot with good conscience request that the changes be pulled up.]

cleaner changes corrseponding to kernel changes

User-level changes corrseponding to my latest kernel changes.
newfs_lfs gives lfs_minfreeseg a value of 1/8 of the total segments on
the disk, based on rough empirical data, but this should be refined in
the future.

Make sure to segunmap segments on error in lfs_bmapv or lfs_markv. Prevents
a memory leak of by default 1 Mb per error. May fix PR #9149.

branches: 1.18.4;
Take care of memory leaks

If the child cleaner dies repeatedly without doing anything, give up. Uses
similar logic to inetd to identify such looping.

Remove extraneous colons and newlines from perror(3) arguments.

branches: 1.15.2;
Correct wrong conversion specifications in calls to syslog(3).

The cleaner now marks empty segments clean without having to read their
contents, a substantial optimization if the work load is right: if enough
empty segments are available, the cleaner never has to read or write *any*
blocks except those on the Ifile. When the cleaner wakes up it marks all
empty segments clean before deciding whether any further segments need to
be cleaned.

Fixed overflow bugs in the cleaner's handling of the cost/benefit metric
for empty segments.

branches: 1.13.2;
work around printf format warnings on alpha

Cleaner changes corresponding to in-kernel LFS changes. In particular, the
cleaner understands fragments; and it knows to change bi_bp to reflect a
change in bi_daddr, if lfs_bmapv says that there is one.

PR/6248: Konrad Schroder: lfs_cleanerd could use syslog() for notices,
instead of err()/warn()

PR/6249: Konrad Schroder lfs_cleanerd uses wrong units when computing which
segments to clean

PR/6247: Konrad Schroder: lfs_cleanerd overflows char register counting live
bytes

type fixes, in part from PR#6032

use correct getopt parameters, as per PR 5480 from Konrad Schroder <perseant@hhhh.org>

Need <time.h> for time() and ctime() prototypes.

Merge with Lite2 + local changes

No need to decalre optind.

Compare a return value of getopt() against -1 instead of EOF.

WARNS?=1. RCS ids.

branches: 1.1.1;
From 4.4-Lite, with local changes.

ufs -> ulfs for lfs_cleanerd

branches: 1.3.4; 1.3.26;
Remove clause 3 and 4 from TNF licenses

branches: 1.2.18;
Be more careful about checking return value of {m,re}alloc().

In determining when to stop reading segments when counting bytes (-b flag),
total the sizes of the blocks we're actually writing instead of assuming
they are all full blocks: many could be fragments or inode blocks. This
increases the number of segments per Ifile write, markedly improving the
efficiency of the cleaner in the small file case.

A new version of the cleaner. In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

Use llabs(), not abs(), on the difference of daddr_t's.
Caught by clang, reported by joerg.

Turn off the clang-specific compiler flags that were making the report
nonfatal.

Hack up dinode usage to be 64 vs. 32 as needed. Part 1.

(This part changes the native lfs code; the ufs-derived code already
has 64 vs. 32 logic, but as aspects of it are unsafe, and don't
entirely interoperate cleanly with the lfs 64/32 stuff, pass 2 will be
rehashing that.)

Add IFILE32 and IFILE64 structures for the on-disk ifile entries.
Add and use accessors. There are also a bunch of places that cast and
I hope I've found them all...

Make 32-bit and 64-bit versions of CLEANERINFO.

XXX: while this is written to disk, it seems like much of it would
XXX: be better set up as a commpage shared with the cleaner.

Fix assorted 64->32 truncations related to BLOCK_INFO.

Also make note of a cleaner limitation: it seems that when it goes to
coalesce discontiguous files, it mallocs an array with one BLOCK_INFO
for every block in the file. Therefore, with 64-bit LFS, on a 32-bit
platform it will be possible to have files large enough to overflow
the cleaner's address space. Currently these will be skipped and cause
warnings via syslog.

At some point someone should rewrite the logic to coalesce files to
use chunks of some reasonable size, as discontinuity between such
chunks is immaterial and mallocing this much space is silly and
fragile. Also, the kernel only accepts up to 65536 blocks at a time
for bmapv and markv, so processing more than this at once probably
isn't useful and may not even work currently. I don't want to change
this around just now as it's not entirely trivial.

Use lfs_accessors.h in conjunction with the cleaner's struct clfs.
Remove previous hacks.

Add a new lfs header file: lfs_accessors.h.

This contains all the accessor functions and macros out of lfs.h.
Add an include of lfs_accessors.h after all uses of lfs.h... except
for code that wants to define its own struct lfs-alike that the
accessors are supposed to play along with. For these, set STRUCT_LFS
and include lfs_accessors.h after the necessary structure has been
defined, so that lfs_accessors.h can emit functions in terms of it.

More lfs superblock accessors.
(This changes the rest of the code over; all the accessors were
already added.)

The difference between this commit and the previous one is arbitrary,
but the previous one passed the regression tests on its own so I'm
keeping it separate to help with any bisections that might be needed
in the future.

Switch to accessor functions for elements of the LFS on-disk
superblock. This will allow switching between 32/64 bit forms on the
fly; it will also allow handling LFS_EI reasonably tidily. (That
currently doesn't work on the superblock.)

It also gets rid of cpp abuse in the form of fake structure member
macros.

Also, instead of doing sleep/wakeup on &lfs_avail and &lfs_nextseg
inside the on-disk superblock, add extra elements to the in-memory
struct lfs for this. (XXX: these should be changed to condvars, but
not right now)

XXX: this migrates a structure needed by the lfs code in libsa (struct
salfs) into lfs.h, where it doesn't belong, but for the time being
this is necessary in order to allow the accessors (and the various
lfs macros and other goop that relies on them) to compile.

- Fix fallout for recent bread() change (removing cred arg).

Prefix most of the cpp macros with lfs_ and LFS_ to avoid conflicts with ffs.
This was done so that boot blocks that want to compile both FFS and LFS in
the same file work.

Missed one of the -I${NETBSDSRCDIR}/sys hacks yesterday; remove it now.

ufs -> ulfs for lfs_cleanerd

Stuff UFS_ in front of a few of ufs's symbols to reduce namespace
pollution. Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN -> UFS_MAXSYMLINKLEN
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Sort out ext2fs's misuse of NDADDR and NIADDR; fortunately, these have
the same values in ext2fs and ffs.

No functional change intended.

branches: 1.19.6;
In lfs_cleanerd, check errno earlier, to avoid premature exit when the
real errno was EAGAIN.

When coalescing, check against NULL to avoid seg fault.

branches: 1.18.6;
Define syscalls of lfs userspace tools (cleaner, mainly) through
a struct called kernelops, which contains standard system calls
for the normal case and rump system calls for the rump case.

Make it possible to run the lfs cleaner in a library fashion (taking
the quick route with the implementation).

fix WARNS=3 issues;
* sprinkle const
* avoid tests < 0 on unsigned types
* test fwrite() returns nmemb not <0

branches: 1.16.6;
Make sure all cached buffers with valid, not yet written data have been
run through copy-on-write. Call fscow_run() with valid data where possible.

The LP_UFSCOW hack is no longer needed to protect ffs_copyonwrite() against
endless recursion.

- Add a flag B_MODIFY to bread(), breada() and breadn(). If set the caller
intends to modify the buffer returned.

- Always run copy-on-write on buffers returned from ffs_balloc().

- Add new function ffs_getblk() that gets a buffer, assigns a new blkno,
may clear the buffer and runs copy-on-write. Process possible errors
from getblk() or fscow_run(). Part of PR kern/38664.

Welcome to 4.99.63

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.15.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.14.6;
brelse() now takes two arguments.

branches: 1.13.10;
Be more careful about checking return value of {m,re}alloc().

In determining when to stop reading segments when counting bytes (-b flag),
total the sizes of the blocks we're actually writing instead of assuming
they are all full blocks: many could be fragments or inode blocks. This
increases the number of segments per Ifile write, markedly improving the
efficiency of the cleaner in the small file case.

Coverity CID 2792: Fix memory leaks

A new version of the cleaner. In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

64 bit inode changes.

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Make fs-specific fcntl macros take three arguments (approved wrstuden).
Let LFS use fcntl for cleaner functions.

Convert lfs_cleanerd over to use the new ioctl calls instead of the
lfs syscalls.

Bump daddr_t to 64 bits. Replace it with int32_t in all places where
it was used on-disk, so that on-disk formats remain the same.
Remove ufs_daddr_t and ufs_lbn_t for the time being.

- fix memory leak.
- add more error checks.
- spaces -> tab

fix calculation bugs that prevents coalescing from working properly.
PR 19133.

Note each type of failure in clean_inode and provide statistics on
failures as well as successes when a run of clean_all_inodes completes.

Explicitly cast to off_t in get_dinode and get_rawblock, to make sure we
read the right block.

Don't try to coalesce files that have fewer than NDADDR blocks, due to
a potential problem with cleaning fragments at all.

Better sanity checks when selecting files to coalesce; in particular don't
shift too far left when comparing the number of discontinuities to the log2
of the number of total blocks.

Better log messages: note beginning of coalescing correctly; also take
the log message from add_segment out of "if (debug)" for symmetry with the
"finished segment" message.

Use lfs_bmapv to find the inode, rather than looking it up manually in
the ifile; this should give more up-to-date information, since trolling
through every inode in the fs could take some time.

First stab at file coalescing. When the cleaner detects that it might
be digging itself deeper into a hole, it forks off a subprocess
that locates files with too many discontinuities and rewrites them, if
there is enough room.

Optionally the user can manually coaleasce files by running with "-c".

The recent change to lfs_markv is required for the coalescer to do anything.

All of "digging itself deeper", "too many discontinuities", and "enough room"
need to be better defined.

Prefix most of the cpp macros with lfs_ and LFS_ to avoid conflicts with ffs.
This was done so that boot blocks that want to compile both FFS and LFS in
the same file work.

branches: 1.9.2;
Use the same loop order for freeing as for allocation. Simpler code and
easier to process for analyzers.

branches: 1.8.2;
Fix memset usage.

Define syscalls of lfs userspace tools (cleaner, mainly) through
a struct called kernelops, which contains standard system calls
for the normal case and rump system calls for the rump case.

Make it possible to run the lfs cleaner in a library fashion (taking
the quick route with the implementation).

Remove clause 3 and 4 from TNF licenses

branches: 1.5.6;
brelse() now takes two arguments.

branches: 1.4.10;
Coverity CID 3766: Avoid null pointer deref

Be more careful about checking return value of {m,re}alloc().

In determining when to stop reading segments when counting bytes (-b flag),
total the sizes of the blocks we're actually writing instead of assuming
they are all full blocks: many could be fragments or inode blocks. This
increases the number of segments per Ifile write, markedly improving the
efficiency of the cleaner in the small file case.

Coverity CID 2846: Don't use LIST_FOREACH if you are going to free the cursor

A new version of the cleaner. In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

A new version of the cleaner. In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

s/threshhold/threshold

New sentence, new line. Punctuation improvements.

Add support for passing the raw device name separate from the
filesystem. This is useful in the case where the cleaner is compiled
into code, such as rump_lfs and the ATF tests. This helps to fix
bin/54488

branches: 1.18.46; 1.18.48;
Describe -D. Bump date.

Bump date for previous.

Temporarily disable -c (file coalescing) until data corruption issues can
be resolved.

Document -i option, update date; at the request of wiz@.

A new version of the cleaner. In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22284, verified by myself.

Begin new sentences on new lines.
Patch from Robert Elz (kre at munnari oz au).

update lfs_cleanerd manual page for new -c option

Whitespace nits, sort SEE ALSO.

Merge the short-lived perseant-lfsv2 branch into the trunk.

Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default. Changes for the v2 layout include:

- Segments of non-PO2 size and arbitrary block offset, so these can be
matched to convenient physical characteristics of the partition (e.g.,
stripe or track size and offset).

- Address by fragment instead of by disk sector, paving the way for
non-512-byte-sector devices. In theory fragments can be as large
as you like, though in reality they must be smaller than MAXBSIZE in size.

- Use serial number and filesystem identifier to ensure that roll-forward
doesn't get old data and think it's new. Roll-forward is enabled for
v2 filesystems, though not for v1 filesystems by default.

- The inode free list is now a tailq, paving the way for undelete (undelete
is not yet implemented, but can be without further non-backwards-compatible
changes to disk structures).

- Inode atime information is kept in the Ifile, instead of on the inode;
that is, the inode is never written *just* because atime was changed.
Because of this the inodes remain near the file data on the disk, rather
than wandering all over as the disk is read repeatedly. This speeds up
repeated reads by a small but noticeable amount.

Other changes of note include:

- The ifile written by newfs_lfs can now be of arbitrary length, it is no
longer restricted to a single indirect block.

- Fixed an old bug where ctime was changed every time a vnode was created.
I need to look more closely to make sure that the times are only updated
during write(2) and friends, not after-the-fact during a segment write,
and certainly not by the cleaner.

branches: 1.8.2;
in SEE also, xref newfs_lfs(8), sort entries and seperate by ","

Various bug-fixes to LFS, to wit:


Kernel:

* Add runtime quantity lfs_ravail, the number of disk-blocks reserved
for writing. Writes to the filesystem first reserve a maximum amount
of blocks before their write is allowed to proceed; after the blocks
are allocated the reserved total is reduced by a corresponding amount.

If the lfs_reserve function cannot immediately reserve the requested
number of blocks, the inode is unlocked, and the thread sleeps until
the cleaner has made enough space available for the blocks to be
reserved. In this way large files can be written to the filesystem
(or, smaller files can be written to a nearly-full but thoroughly
clean filesystem) and the cleaner can still function properly.

* Remove explicit switching on dlfs_minfreeseg from the kernel code; it
is now merely a fs-creation parameter used to compute dlfs_avail and
dlfs_bfree (and used by fsck_lfs(8) to check their accuracy). Its
former role is better assumed by a properly computed dlfs_avail.

* Bounds-check inode numbers submitted through lfs_bmapv and lfs_markv.
This prevents a panic, but, if the cleaner is feeding the filesystem
the wrong data, you are still in a world of hurt.

* Cleanup: remove explicit references of DEV_BSIZE in favor of
btodb()/dbtob().

lfs_cleanerd:

* Make -n mean "send N segments' blocks through a single call to
lfs_markv". Previously it had meant "clean N segments though N calls
to lfs_markv, before looking again to see if more need to be cleaned".
The new behavior gives better packing of direct data on disk with as
little metadata as possible, largely alleviating the problem that the
cleaner can consume more disk through inefficient use of metadata than
it frees by moving dirty data away from clean "holes" to produce
entirely clean segments.

* Make -b mean "read as many segments as necessary to write N segments
of dirty data back to disk", rather than its former meaning of "read
as many segments as necessary to free N segments worth of space". The
new meaning, combined with the new -n behavior described above,
further aids in cleaning storage efficiency as entire segments can be
written at once, using as few blocks as possible for segment summaries
and inode blocks.

* Make the cleaner take note of segments which could not be cleaned due
to error, and not attempt to clean them until they are entirely free
of dirty blocks. This prevents the case in which a cleanerd running
with -n 1 and without -b (formerly the default) would spin trying
repeatedly to clean a corrupt segment, while the remaining space
filled and deadlocked the filesystem.

* Update the lfs_cleanerd manual page to describe all the options,
including the changes mentioned here (in particular, the -b and -n
flags were previously undocumented).

fsck_lfs:

* Check, and optionally fix, lfs_avail (to an exact figure) and
lfs_bfree (within a margin of error) in pass 5.

newfs_lfs:

* Reduce the default dlfs_minfreeseg to 1/20 of the total segments.

* Add a warning if the sgs disklabel field is 16 (the default for FFS'
cpg, but not usually desirable for LFS' sgs: 5--8 is a better range).

* Change the calculation of lfs_avail and lfs_bfree, corresponding to
the kernel changes mentioned above.

mount_lfs:

* Add -N and -b options to pass corresponding -n and -b options to
lfs_cleanerd.

* Default to calling lfs_cleanerd with "-b -n 4".


[All of these changes were largely tested in the 1.5 branch, with the
idea that they (along with previous un-pulled-up work) could be applied
to the branch while it was still in ALPHA2; however my test system has
experienced corruption on another filesystem (/dev/console has gone
missing :^), and, while I believe this unrelated to the LFS changes, I
cannot with good conscience request that the changes be pulled up.]

branches: 1.6.10;
More and more .Os cleanups. .Os is defined in the tmac.doc-common file,
so we shouldn't override it with versions in the manpages. Many more to
come.

macroize BSD, NetBSD, FreeBSD and misc cleanup

Fix usage of .Nm.

WARNS?=1. RCS ids.

Fix broken uses of Dd. Both the mdoc and mdoc.samples pages agree:
.Dd is supposed to be invoked like:
.Dd month day, year
e.g. ".Dd January 25, 1989", rather than:
.Dd "month day, year"
which is what these pages did.

branches: 1.1.1;
From 4.4-Lite, with local changes.

Add support for passing the raw device name separate from the
filesystem. This is useful in the case where the cleaner is compiled
into code, such as rump_lfs and the ATF tests. This helps to fix
bin/54488

Use getdiskrawname to find the device name

Reviewed by Christos

branches: 1.58.16; 1.58.18;
move the compiler hack closer to the source of the fail

make the GCC 4.8 specific hack for ci.clean being set GCC >= 4.8.

Need <stdbool.h> for true/false.

Enable lfs64 in the cleaner.

Use IINFO in the cleaner.

Add new accessors for the d_type and d_namlen fields of struct lfs_direct.
Napalm the old byteswap access logic for these.

The ifile's inode number is constant. (it is always 1)

Therefore, storing the value in the superblock and reading it out
again is silly and offers the opportunity for it to become corrupted.
So, don't do that (most of the code already didn't) and use the
existing constant instead. Initialize new 32-bit superblocks with
the value for the sake of old userland programs, but don't keep the
value in the 64-bit superblock at all.

(approved by Margo Seltzer)

Add byteswapping to the dinode accessors.

This prevents regressions in the ulfs code when switching to the new
accessors. Note that while adding byteswapping to the other accessors
is straightforward, I haven't done it yet; and that also is not enough
to make LFS_EI work, because there are places lying around that bypass
the accessors for one reason and another and all of them need to be
updated. That is going to have to wait for a later day as LFS_EI is
not on the critical path right now.

Hack up dinode usage to be 64 vs. 32 as needed. Part 1.

(This part changes the native lfs code; the ufs-derived code already
has 64 vs. 32 logic, but as aspects of it are unsafe, and don't
entirely interoperate cleanly with the lfs 64/32 stuff, pass 2 will be
rehashing that.)

Provide 32-bit and 64-bit versions of FINFO.

This also entailed sorting out part of struct segment, as that
contains a pointer into the current FINFO data.

Make 32-bit and 64-bit versions of SEGSUM.
Also fix some of the FINFO handling as it's closely entangled.

Add IFILE32 and IFILE64 structures for the on-disk ifile entries.
Add and use accessors. There are also a bunch of places that cast and
I hope I've found them all...

Make 32-bit and 64-bit versions of CLEANERINFO.

XXX: while this is written to disk, it seems like much of it would
XXX: be better set up as a commpage shared with the cleaner.

Fix assorted 64->32 truncations related to BLOCK_INFO.

Also make note of a cleaner limitation: it seems that when it goes to
coalesce discontiguous files, it mallocs an array with one BLOCK_INFO
for every block in the file. Therefore, with 64-bit LFS, on a 32-bit
platform it will be possible to have files large enough to overflow
the cleaner's address space. Currently these will be skipped and cause
warnings via syslog.

At some point someone should rewrite the logic to coalesce files to
use chunks of some reasonable size, as discontinuity between such
chunks is immaterial and mallocing this much space is silly and
fragile. Also, the kernel only accepts up to 65536 blocks at a time
for bmapv and markv, so processing more than this at once probably
isn't useful and may not even work currently. I don't want to change
this around just now as it's not entirely trivial.

Add a (draft) 64-bit superblock. Make things build again.

Add pieces of support for using both superblock types where
convenient, and specifically to the superblock accessors, but don't
actually enable it anywhere.

First substantive step on PR 50000.

Use accessor functions for the version field of the lfs superblock.
I thought at first maybe the cases that test the version should be
rolled into the accessors, but on the whole I think the conclusion on
that is no.

Second batch of 64 -> 32 truncations in lfs, along with more minor
tidyups and corrections in passing.

Use lfs_accessors.h in conjunction with the cleaner's struct clfs.
Remove previous hacks.

More lfs superblock accessors.
(This changes the rest of the code over; all the accessors were
already added.)

The difference between this commit and the previous one is arbitrary,
but the previous one passed the regression tests on its own so I'm
keeping it separate to help with any bisections that might be needed
in the future.

Switch to accessor functions for elements of the LFS on-disk
superblock. This will allow switching between 32/64 bit forms on the
fly; it will also allow handling LFS_EI reasonably tidily. (That
currently doesn't work on the superblock.)

It also gets rid of cpp abuse in the form of fake structure member
macros.

Also, instead of doing sleep/wakeup on &lfs_avail and &lfs_nextseg
inside the on-disk superblock, add extra elements to the in-memory
struct lfs for this. (XXX: these should be changed to condvars, but
not right now)

XXX: this migrates a structure needed by the lfs code in libsa (struct
salfs) into lfs.h, where it doesn't belong, but for the time being
this is necessary in order to allow the accessors (and the various
lfs macros and other goop that relies on them) to compile.

- Fix fallout for recent bread() change (removing cred arg).

fix unused variable warnings

Prefix most of the cpp macros with lfs_ and LFS_ to avoid conflicts with ffs.
This was done so that boot blocks that want to compile both FFS and LFS in
the same file work.

Missed one of the -I${NETBSDSRCDIR}/sys hacks yesterday; remove it now.

Move the dinode (on-disk inode) structures to lfs.h, since they are
and will be obviously required by userland tools that need to read
the on-disk structures.

Also, DINODE{1,2}_SIZE -> LFS_DINODE{1,2}_SIZE.

ufs -> ulfs for lfs_cleanerd

Stuff UFS_ in front of a few of ufs's symbols to reduce namespace
pollution. Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN -> UFS_MAXSYMLINKLEN
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Sort out ext2fs's misuse of NDADDR and NIADDR; fortunately, these have
the same values in ext2fs and ffs.

No functional change intended.

branches: 1.31.2;
Make load_threshold double and restore original value of 0.2.
From NAKAJIMA Yoshihiro.

Don't assign non-integral constants to integral variables.

Shut down the cleaner earlier if the filesystem is unmounted.

In lfs_cleanerd, check errno earlier, to avoid premature exit when the
real errno was EAGAIN.

When coalescing, check against NULL to avoid seg fault.

branches: 1.27.6;
Deduce raw device correctly by prepending "r" to basename.

Don't exit when running as a lib after FS is unmounted, just return
from the mainloop.

Do the semaphore trick only if the cleaner is run as a lib -- the
standard installation doesn't get linked against librt or libpthread.

Add a method for posting a semaphore when cleaner init has reached
the main loop. This is useful for very quickly executed lfs runs
such as those in tests, as an initialization phase cleaner may
prevent file system unmount.

Read the padded superblocks to avoid problems with disks that have
larger sectors than 512 Bytes.

Attempt to recover slightly more gracefully if the kernel socket is shutdown.

Define syscalls of lfs userspace tools (cleaner, mainly) through
a struct called kernelops, which contains standard system calls
for the normal case and rump system calls for the rump case.

Make it possible to run the lfs cleaner in a library fashion (taking
the quick route with the implementation).

add D to getopt so that previous actually works

add -D, which prompts to stay on foreground without debug spewage

Check main loop fcntl return value and error out if it fails.
Prevents e.g. from 5.0 cleaner looping forever against -current
lfs driver (fcntl data structure sizes are different).

fix WARNS=3 issues;
* sprinkle const
* avoid tests < 0 on unsigned types
* test fwrite() returns nmemb not <0

avoid a shadowed variable

branches: 1.15.6;
Make sure all cached buffers with valid, not yet written data have been
run through copy-on-write. Call fscow_run() with valid data where possible.

The LP_UFSCOW hack is no longer needed to protect ffs_copyonwrite() against
endless recursion.

- Add a flag B_MODIFY to bread(), breada() and breadn(). If set the caller
intends to modify the buffer returned.

- Always run copy-on-write on buffers returned from ffs_balloc().

- Add new function ffs_getblk() that gets a buffer, assigns a new blkno,
may clear the buffer and runs copy-on-write. Process possible errors
from getblk() or fscow_run(). Part of PR kern/38664.

Welcome to 4.99.63

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.14.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.13.6;
brelse() now takes two arguments.

branches: 1.12.8;
Temporarily disable -c (file coalescing) until data corruption issues can
be resolved.

Pay attention to the new LFS_CLEANER_MUST_CLEAN flag in the Ifile.

Make filehandles opaque to userland

Fix alignement problems for fhandle_t, exposed by gcc4.1.

While touching all vptofh/fhtovp functions, get rid of VFS_MAXFIDSIZ,
version the getfh(2) syscall and explicitly pass the size available in
the filehandle from userland.

Discussed on tech-kern, with lots of help from yamt (thanks!).

Place an upper bound on the number of segments to clean at one time, no
matter how empty they are.

Note that if two blocks have the same inode and block number, they sort
the same (this should never happen, but if it does there's no reason to
have qsort scramble the list).

Add some diagnostic syslog messages for unusual cases.

Be more careful about checking return value of {m,re}alloc().

In determining when to stop reading segments when counting bytes (-b flag),
total the sizes of the blocks we're actually writing instead of assuming
they are all full blocks: many could be fragments or inode blocks. This
increases the number of segments per Ifile write, markedly improving the
efficiency of the cleaner in the small file case.

avoid some char * vs. u_char * issues.

When checking hidden cleaning costs, don't erroneously add the first
indirect block when considering the cleaning of block numbers less
than NDADDR (which do not use indirect blocks).

Also, note the loss of only half a block per segment to fragmentation
when considering the benefit function, rather than a whole block.

Look for the inode pointers in the right place if lfs_bsize != lfs_fsize.

Handle the case of inode blocks misidentified by the segment headers
correctly, a problem reported by Kurt Schreiner; as well as similar
error cases (realloc returning NULL or no cleanable segments found).

Coverity CID 2847: Don't use LIST_FOREACH if you are going to free the
current pointer!

A new version of the cleaner. In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

A new version of the cleaner. In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

Change PRiu64 with a cast and %llu. I am not using PRIu32 and no cast for
now. I will do so in a separate pass that will fix all the inode printing.

fix compilation problem on LP64

64 bit inode changes.

remove bogus frees on uninitialized variable.

Provide a resize_lfs(8), including kernel and cleaner support. The current
implementation requires the fs to be mounted while resizing. Tested in both
directions, and everything appears to work happily, but ymmv.

branches: 1.41.2;
Various minor LFS improvements:

* Adapt lfs_cleanerd to use the fcntl call to get the Ifile filehandle,
so it need not be in the namespace.
* Make lfs_cleanerd be more careful when there are very few available
segments.
* Remove the Ifile from the filesystem namespace. The cleaner now uses
a fcntl call on the root inode to find the Ifile filehandle.
* Make lfs_cleanerd less verbose when the filesystem is unmounted.

Replace the statfs() family of system calls with statvfs().
Retain binary compatibility.

branches: 1.39.4;
fix debug output.

realloc pedannt

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22284, verified by myself.

use bounded string ops. check malloc failure. asprintf is easier than
complicated strcat/strcpy combination.

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Convert lfs_cleanerd over to use the new ioctl calls instead of the
lfs syscalls.

Use int32_t for block adresses in segment summary structures.

Bump daddr_t to 64 bits. Replace it with int32_t in all places where
it was used on-disk, so that on-disk formats remain the same.
Remove ufs_daddr_t and ufs_lbn_t for the time being.

fix a typo in previous.

PR 19278 from Ryo HAYASAKA.

clean this up a bit. avoid annoying code duplication on opening files,
and make error messages consistent.

Note each type of failure in clean_inode and provide statistics on
failures as well as successes when a run of clean_all_inodes completes.

Explicitly cast to off_t in get_dinode and get_rawblock, to make sure we
read the right block.

Don't try to coalesce files that have fewer than NDADDR blocks, due to
a potential problem with cleaning fragments at all.

Better sanity checks when selecting files to coalesce; in particular don't
shift too far left when comparing the number of discontinuities to the log2
of the number of total blocks.

Better log messages: note beginning of coalescing correctly; also take
the log message from add_segment out of "if (debug)" for symmetry with the
"finished segment" message.

Use lfs_bmapv to find the inode, rather than looking it up manually in
the ifile; this should give more up-to-date information, since trolling
through every inode in the fs could take some time.

First stab at file coalescing. When the cleaner detects that it might
be digging itself deeper into a hole, it forks off a subprocess
that locates files with too many discontinuities and rewrites them, if
there is enough room.

Optionally the user can manually coaleasce files by running with "-c".

The recent change to lfs_markv is required for the coalescer to do anything.

All of "digging itself deeper", "too many discontinuities", and "enough room"
need to be better defined.

fix a reversed condition.

Cast arg to long, and print with %ld, so that this compiles on some of
the more esoteric architectures.

Correct my previous lfs_cleanerd commit so that it works properly on
v1 filesystems as well (use segtod instead of lfs_ssize / lfs_fsize).

Tested on i386.

Handle segment 0 properly, if its offset is different from other segments
because of the disklabel.

Fix a problem with inode block handling that sometimes caused the wrong
blocks to be read, causing either cleaning failures or panics with v2 file
systems.

Merge the short-lived perseant-lfsv2 branch into the trunk.

Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default. Changes for the v2 layout include:

- Segments of non-PO2 size and arbitrary block offset, so these can be
matched to convenient physical characteristics of the partition (e.g.,
stripe or track size and offset).

- Address by fragment instead of by disk sector, paving the way for
non-512-byte-sector devices. In theory fragments can be as large
as you like, though in reality they must be smaller than MAXBSIZE in size.

- Use serial number and filesystem identifier to ensure that roll-forward
doesn't get old data and think it's new. Roll-forward is enabled for
v2 filesystems, though not for v1 filesystems by default.

- The inode free list is now a tailq, paving the way for undelete (undelete
is not yet implemented, but can be without further non-backwards-compatible
changes to disk structures).

- Inode atime information is kept in the Ifile, instead of on the inode;
that is, the inode is never written *just* because atime was changed.
Because of this the inodes remain near the file data on the disk, rather
than wandering all over as the disk is read repeatedly. This speeds up
repeated reads by a small but noticeable amount.

Other changes of note include:

- The ifile written by newfs_lfs can now be of arbitrary length, it is no
longer restricted to a single indirect block.

- Fixed an old bug where ctime was changed every time a vnode was created.
I need to look more closely to make sure that the times are only updated
during write(2) and friends, not after-the-fact during a segment write,
and certainly not by the cleaner.

branches: 1.21.2;
remove redundant declarations.

be more consistent about syslog usage. now it's more like:
err fatal errors
warning warnings
info status messages (-d), stats on SIGxxx
debug debug messages (-d), debug stats

Don't qsort() by the segcreate field. Prevents potentially serious filesystem
corruption if the clock jumps backwards.

use more standard %ll_ in favour of %q_

Fix memory leak described in PR #11094 (patch from Jesse Off
<joff@gci-net.com>).

Various bug-fixes to LFS, to wit:


Kernel:

* Add runtime quantity lfs_ravail, the number of disk-blocks reserved
for writing. Writes to the filesystem first reserve a maximum amount
of blocks before their write is allowed to proceed; after the blocks
are allocated the reserved total is reduced by a corresponding amount.

If the lfs_reserve function cannot immediately reserve the requested
number of blocks, the inode is unlocked, and the thread sleeps until
the cleaner has made enough space available for the blocks to be
reserved. In this way large files can be written to the filesystem
(or, smaller files can be written to a nearly-full but thoroughly
clean filesystem) and the cleaner can still function properly.

* Remove explicit switching on dlfs_minfreeseg from the kernel code; it
is now merely a fs-creation parameter used to compute dlfs_avail and
dlfs_bfree (and used by fsck_lfs(8) to check their accuracy). Its
former role is better assumed by a properly computed dlfs_avail.

* Bounds-check inode numbers submitted through lfs_bmapv and lfs_markv.
This prevents a panic, but, if the cleaner is feeding the filesystem
the wrong data, you are still in a world of hurt.

* Cleanup: remove explicit references of DEV_BSIZE in favor of
btodb()/dbtob().

lfs_cleanerd:

* Make -n mean "send N segments' blocks through a single call to
lfs_markv". Previously it had meant "clean N segments though N calls
to lfs_markv, before looking again to see if more need to be cleaned".
The new behavior gives better packing of direct data on disk with as
little metadata as possible, largely alleviating the problem that the
cleaner can consume more disk through inefficient use of metadata than
it frees by moving dirty data away from clean "holes" to produce
entirely clean segments.

* Make -b mean "read as many segments as necessary to write N segments
of dirty data back to disk", rather than its former meaning of "read
as many segments as necessary to free N segments worth of space". The
new meaning, combined with the new -n behavior described above,
further aids in cleaning storage efficiency as entire segments can be
written at once, using as few blocks as possible for segment summaries
and inode blocks.

* Make the cleaner take note of segments which could not be cleaned due
to error, and not attempt to clean them until they are entirely free
of dirty blocks. This prevents the case in which a cleanerd running
with -n 1 and without -b (formerly the default) would spin trying
repeatedly to clean a corrupt segment, while the remaining space
filled and deadlocked the filesystem.

* Update the lfs_cleanerd manual page to describe all the options,
including the changes mentioned here (in particular, the -b and -n
flags were previously undocumented).

fsck_lfs:

* Check, and optionally fix, lfs_avail (to an exact figure) and
lfs_bfree (within a margin of error) in pass 5.

newfs_lfs:

* Reduce the default dlfs_minfreeseg to 1/20 of the total segments.

* Add a warning if the sgs disklabel field is 16 (the default for FFS'
cpg, but not usually desirable for LFS' sgs: 5--8 is a better range).

* Change the calculation of lfs_avail and lfs_bfree, corresponding to
the kernel changes mentioned above.

mount_lfs:

* Add -N and -b options to pass corresponding -n and -b options to
lfs_cleanerd.

* Default to calling lfs_cleanerd with "-b -n 4".


[All of these changes were largely tested in the 1.5 branch, with the
idea that they (along with previous un-pulled-up work) could be applied
to the branch while it was still in ALPHA2; however my test system has
experienced corruption on another filesystem (/dev/console has gone
missing :^), and, while I believe this unrelated to the LFS changes, I
cannot with good conscience request that the changes be pulled up.]

branches: 1.15.4;
Take care of memory leaks

Don't complain that we can't fstat the Ifile, if it's because the filesystem
has been unmounted. (I.e., don't give errors every time an LFS is unmounted.)

branches: 1.13.2;
Remove partial-segment timestamp optimization from lfs_segmapv. If "time"
is not monotonically increasing (e.g. clock is slaved to another system)
the optimization will result in segments being treated as corrupt
(uncleanable). If enough such "bad" segments were created, the cleaner would
clean continuously, and after some time the system would panic with "no
clean segments".

(Legitimately old partial-segments are relatively rare, and will have their
blocks culled by lfs_bmapv.)

branches: 1.12.2;
work around egcs prints format warning on alpha
(XXX bad fix, but egcs doesn't like a "quad_t" here)

Cleaner changes corresponding to in-kernel LFS changes. In particular, the
cleaner understands fragments; and it knows to change bi_bp to reflect a
change in bi_daddr, if lfs_bmapv says that there is one.

Cast quad_t and off_t objects to (long long) for printing with %q.
(Fix a sort-of-LP64 egcs printf warning.)

It's unfortunate that off_t and quad_t don't print with %q. I wonder
what would happen if alpha changed these from long -> long long? It's
the same actual size in bits either way.

PR/6248: Konrad Schroder: lfs_cleanerd could use syslog() for notices,
instead of err()/warn()

type fixes, in part from PR#6032

bzero->memset, bcopy->memcpy, bcmp->memcmp

Merge with Lite2 + local changes

Fill in missing (default) mmap(2) flags.

WARNS?=1. RCS ids.

avoid void pointer arithmetic

* NULL => 0 (Arne Juul; PR#3629)
* Nuke spacoids.

branches: 1.1.1;
From 4.4-Lite, with local changes.

A new version of the cleaner. In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22284, verified by myself.

Merge the short-lived perseant-lfsv2 branch into the trunk.

Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default. Changes for the v2 layout include:

- Segments of non-PO2 size and arbitrary block offset, so these can be
matched to convenient physical characteristics of the partition (e.g.,
stripe or track size and offset).

- Address by fragment instead of by disk sector, paving the way for
non-512-byte-sector devices. In theory fragments can be as large
as you like, though in reality they must be smaller than MAXBSIZE in size.

- Use serial number and filesystem identifier to ensure that roll-forward
doesn't get old data and think it's new. Roll-forward is enabled for
v2 filesystems, though not for v1 filesystems by default.

- The inode free list is now a tailq, paving the way for undelete (undelete
is not yet implemented, but can be without further non-backwards-compatible
changes to disk structures).

- Inode atime information is kept in the Ifile, instead of on the inode;
that is, the inode is never written *just* because atime was changed.
Because of this the inodes remain near the file data on the disk, rather
than wandering all over as the disk is read repeatedly. This speeds up
repeated reads by a small but noticeable amount.

Other changes of note include:

- The ifile written by newfs_lfs can now be of arbitrary length, it is no
longer restricted to a single indirect block.

- Fixed an old bug where ctime was changed every time a vnode was created.
I need to look more closely to make sure that the times are only updated
during write(2) and friends, not after-the-fact during a segment write,
and certainly not by the cleaner.

branches: 1.6.12;
work around printf format warnings on alpha

Cleaner changes corresponding to in-kernel LFS changes. In particular, the
cleaner understands fragments; and it knows to change bi_bp to reflect a
change in bi_daddr, if lfs_bmapv says that there is one.

Cast quad_t and off_t objects to (long long) for printing with %q.
(Fix a sort-of-LP64 egcs printf warning.)

It's unfortunate that off_t and quad_t don't print with %q. I wonder
what would happen if alpha changed these from long -> long long? It's
the same actual size in bits either way.

PR/6248: Konrad Schroder: lfs_cleanerd could use syslog() for notices,
instead of err()/warn()

WARNS?=1. RCS ids.

branches: 1.1.1;
From 4.4-Lite, with local changes.

A new version of the cleaner. In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22284, verified by myself.

Add code to UBCify LFS. This is still behind "#ifdef LFS_UBC" for now
(there are still some details to work out) but expect that to go
away soon. To support these basic changes (creation of lfs_putpages,
lfs_gop_write, mods to lfs_balloc) several other changes were made, to
wit:

* Create a writer daemon kernel thread whose purpose is to handle page
writes for the pagedaemon, but which also takes over some of the
functions of lfs_check(). This thread is started the first time an
LFS is mounted.

* Add a "flags" parameter to GOP_SIZE. Current values are
GOP_SIZE_READ, meaning that the call should return the size of the
in-core version of the file, and GOP_SIZE_WRITE, meaning that it
should return the on-disk size. One of GOP_SIZE_READ or
GOP_SIZE_WRITE must be specified.

* Instead of using malloc(...M_WAITOK) for everything, reserve enough
resources to get by and use malloc(...M_NOWAIT), using the reserves if
necessary. Use the pool subsystem for structures small enough that
this is feasible. This also obsoletes LFS_THROTTLE.

And a few that are not strictly necessary:

* Moves the LFS inode extensions off onto a separately allocated
structure; getting closer to LFS as an LKM. "Welcome to 1.6O."

* Unified GOP_ALLOC between FFS and LFS.

* Update LFS copyright headers to correct values.

* Actually cast to unsigned in lfs_shellsort, like the comment says.

* Keep track of which segments were empty before the previous
checkpoint; any segments that pass two checkpoints both dirty and
empty can be summarily cleaned. Do this. Right now lfs_segclean
still works, but this should be turned into an effectless
compatibility syscall.

Bump daddr_t to 64 bits. Replace it with int32_t in all places where
it was used on-disk, so that on-disk formats remain the same.
Remove ufs_daddr_t and ufs_lbn_t for the time being.

Correct my previous lfs_cleanerd commit so that it works properly on
v1 filesystems as well (use segtod instead of lfs_ssize / lfs_fsize).

Tested on i386.

Fix error in how much memory needed to be allocated to check data cksum
to proceed with adding segments.

Use fixed-width types to compute checksum, so LP64 machines can do it too.

Tested on alpha; test-compiled on arm32.

Merge the short-lived perseant-lfsv2 branch into the trunk.

Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default. Changes for the v2 layout include:

- Segments of non-PO2 size and arbitrary block offset, so these can be
matched to convenient physical characteristics of the partition (e.g.,
stripe or track size and offset).

- Address by fragment instead of by disk sector, paving the way for
non-512-byte-sector devices. In theory fragments can be as large
as you like, though in reality they must be smaller than MAXBSIZE in size.

- Use serial number and filesystem identifier to ensure that roll-forward
doesn't get old data and think it's new. Roll-forward is enabled for
v2 filesystems, though not for v1 filesystems by default.

- The inode free list is now a tailq, paving the way for undelete (undelete
is not yet implemented, but can be without further non-backwards-compatible
changes to disk structures).

- Inode atime information is kept in the Ifile, instead of on the inode;
that is, the inode is never written *just* because atime was changed.
Because of this the inodes remain near the file data on the disk, rather
than wandering all over as the disk is read repeatedly. This speeds up
repeated reads by a small but noticeable amount.

Other changes of note include:

- The ifile written by newfs_lfs can now be of arbitrary length, it is no
longer restricted to a single indirect block.

- Fixed an old bug where ctime was changed every time a vnode was created.
I need to look more closely to make sure that the times are only updated
during write(2) and friends, not after-the-fact during a segment write,
and certainly not by the cleaner.

branches: 1.9.2;
use more standard %ll_ in favour of %q_

Take care of memory leaks

branches: 1.7.2;
Cleaner changes corresponding to in-kernel LFS changes. In particular, the
cleaner understands fragments; and it knows to change bi_bp to reflect a
change in bi_daddr, if lfs_bmapv says that there is one.

Cast quad_t and off_t objects to (long long) for printing with %q.
(Fix a sort-of-LP64 egcs printf warning.)

It's unfortunate that off_t and quad_t don't print with %q. I wonder
what would happen if alpha changed these from long -> long long? It's
the same actual size in bits either way.

PR/6248: Konrad Schroder: lfs_cleanerd could use syslog() for notices,
instead of err()/warn()

type fixes, in part from PR#6032

Need <time.h> for ctime() prototype.

WARNS?=1. RCS ids.

branches: 1.1.1;
From 4.4-Lite, with local changes.