s/remaing/remaining/ s/containg/containing/, mainly in comments.

branches: 1.6.2;
fix spelling mistakes, mainly in comments and log messages.

merge ktrace-lwp.

parameter with two es. From Peter Postma.

branches: 1.3.22;
Fix typo. s/extention/extension/

branches: 1.2.14;
Change the reference to "newlfs" in the CHANGES file to the correct "newfs_lfs"

branches: 1.1.4;
New CHANGES files that describes briefly all nontrivial changes made to
the LFS since the 4.4lite2 code was merged into NetBSD.

TODO updated to remove everything marked DONE in 4.4, and add in a list
of more current things to do.

Get rid of comments about the cleaner syscall code and missing fragment
support from README.

Add a new lfs header file: lfs_accessors.h.

This contains all the accessor functions and macros out of lfs.h.
Add an include of lfs_accessors.h after all uses of lfs.h... except
for code that wants to define its own struct lfs-alike that the
accessors are supposed to play along with. For these, set STRUCT_LFS
and include lfs_accessors.h after the necessary structure has been
defined, so that lfs_accessors.h can emit functions in terms of it.

branches: 1.2.10;
Split the definitions suitable for userland out of ulfs_inode.h into
lfs_inode.h. Since fsck_lfs, newfs_lfs, and lfs_cleanerd want to reuse
the inode structure for their own internal use, and some of them share
parts of the kernel code as well, the best way forward is to provide a
relatively sanitized header that doesn't bring in stray material.

Shuffle a few other definitions around so that lfs_inode.h depends
only on lfs.h.

Install lfs_inode.h into /usr/include.

branches: 1.1.188; 1.1.198;
Rework the way kernel include files are installed. In the new method,
as with user-land programs, include files are installed by each directory
in the tree that has includes to install. (This allows more flexibility
as to what gets installed, makes 'partial installs' easier, and gives us
more options as to which machines' includes get installed at any given
time.) The old SYS_INCLUDES={symlinks,copies} behaviours are _both_
still supported, though at least one bug in the 'symlinks' case is
fixed by this change. Include files can't be build before installation,
so directories that have includes as targets (e.g. dev/pci) have to move
those targets into a different Makefile.

New CHANGES files that describes briefly all nontrivial changes made to
the LFS since the 4.4lite2 code was merged into NetBSD.

TODO updated to remove everything marked DONE in 4.4, and add in a list
of more current things to do.

Get rid of comments about the cleaner syscall code and missing fragment
support from README.

New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD'

branches: 1.1.1;
Update to 4.4-Lite fs code, with local changes.

branches: 1.2.2; 1.2.10;
Update the line-count standings.

Split lfs from ufs, part 1: cut and paste 15000 lines of ufs as "ulfs".

These are verbatim copies except that I've preserved the ufs rcsids
for reference. Also,
ufs/quota.h -> ulfs_quotacommon.h
ufs/ufs_quota.h -> ulfs_quota.h

Splitting lfs from ufs was ok'd by core some years ago. This is not
from my original tree, which became unmergeable after the several sets
of quota changes; I've done the work over again over the last couple
days.

merge ktrace-lwp.

branches: 1.9.2;
Protect various per-fs structures with fs->lfs_interlock simple_lock, to
improve behavior in the multiprocessor case. Add debugging segment-lock
assertion statements.

branches: 1.8.2;
Various minor LFS improvements:

* Note when lfs_putpages(9) thinks it is not going to be writing any
pages before calling genfs_putpages(9). This prevents a situation in
which blocks can be queued for writing without a segment header.
* Correct computation of NRESERVE(), though it is still a gross
overestimate in most cases. Note that if NRESERVE() is too high, it
may be impossible to create files on the filesystem. We catch this
case on filesystem mount and refuse to mount r/w.
* Allow filesystems to be mounted whose block size is == MAXBSIZE.
* Somewhere along the line, ufs_bmaparray(9) started mangling UNWRITTEN
entries in indirect blocks again, triggering a failed assertion "daddr
<= LFS_MAX_DADDR". Explicitly convert to and from int32_t to correct
this.
* Add a high-water mark for the number of dirty pages any given LFS can
hold before triggering a flush. This is settable by sysctl, but off
(zero) by default.
* Be more careful about the MAX_BYTES and MAX_BUFS computations so we
shouldn't see "please increase to at least zero" messages.
* Note that VBLK and VCHR vnodes can have nonzero values in di_db[0]
even though their v_size == 0. Don't panic when we see this.
* Change lfs_bfree to a signed quantity. The manner in which it is
processed before being passed to the cleaner means that sometimes it
may drop below zero, and the cleaner must be aware of this.
* Never report bfree < 0 (or higher than lfs_dsize) through
lfs_statvfs(9). This prevents df(1) from ever telling us that our full
filesystems have 16TB free.
* Account space allocated through lfs_balloc(9) that does not have
associated buffer headers, so that the pagedaemon doesn't run us out
of segments.
* Return ENOSPC from lfs_balloc(9) when bfree drops to zero.
* Address a deadlock in lfs_bmapv/lfs_markv when the filesystem is being
unmounted. Because vfs_busy() is a shared lock, and
lfs_bmapv/lfs_markv mark the filesystem vfs_busy(), the cleaner can be
holding the lock that umount() is blocking on, then try to vfs_busy()
again in getnewvnode().

branches: 1.7.2; 1.7.8; 1.7.10; 1.7.12;
Fix a buffer overflow bug in the LFS_UBC case that manifested itself
either as a mysterious UVM error or as "panic: dirty bufs". Verify
maximum size in lfs_malloc.

Teach lfs_updatemeta and lfs_shellsort about oversized cluster blocks from
lfs_gop_write.

When unwiring pages in lfs_gop_write, deactivate them, under the theory
that the pagedaemon wanted to free them last we knew.

Add code to UBCify LFS. This is still behind "#ifdef LFS_UBC" for now
(there are still some details to work out) but expect that to go
away soon. To support these basic changes (creation of lfs_putpages,
lfs_gop_write, mods to lfs_balloc) several other changes were made, to
wit:

* Create a writer daemon kernel thread whose purpose is to handle page
writes for the pagedaemon, but which also takes over some of the
functions of lfs_check(). This thread is started the first time an
LFS is mounted.

* Add a "flags" parameter to GOP_SIZE. Current values are
GOP_SIZE_READ, meaning that the call should return the size of the
in-core version of the file, and GOP_SIZE_WRITE, meaning that it
should return the on-disk size. One of GOP_SIZE_READ or
GOP_SIZE_WRITE must be specified.

* Instead of using malloc(...M_WAITOK) for everything, reserve enough
resources to get by and use malloc(...M_NOWAIT), using the reserves if
necessary. Use the pool subsystem for structures small enough that
this is feasible. This also obsoletes LFS_THROTTLE.

And a few that are not strictly necessary:

* Moves the LFS inode extensions off onto a separately allocated
structure; getting closer to LFS as an LKM. "Welcome to 1.6O."

* Unified GOP_ALLOC between FFS and LFS.

* Update LFS copyright headers to correct values.

* Actually cast to unsigned in lfs_shellsort, like the comment says.

* Keep track of which segments were empty before the previous
checkpoint; any segments that pass two checkpoints both dirty and
empty can be summarily cleaned. Do this. Right now lfs_segclean
still works, but this should be turned into an effectless
compatibility syscall.

Merge the short-lived perseant-lfsv2 branch into the trunk.

Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default. Changes for the v2 layout include:

- Segments of non-PO2 size and arbitrary block offset, so these can be
matched to convenient physical characteristics of the partition (e.g.,
stripe or track size and offset).

- Address by fragment instead of by disk sector, paving the way for
non-512-byte-sector devices. In theory fragments can be as large
as you like, though in reality they must be smaller than MAXBSIZE in size.

- Use serial number and filesystem identifier to ensure that roll-forward
doesn't get old data and think it's new. Roll-forward is enabled for
v2 filesystems, though not for v1 filesystems by default.

- The inode free list is now a tailq, paving the way for undelete (undelete
is not yet implemented, but can be without further non-backwards-compatible
changes to disk structures).

- Inode atime information is kept in the Ifile, instead of on the inode;
that is, the inode is never written *just* because atime was changed.
Because of this the inodes remain near the file data on the disk, rather
than wandering all over as the disk is read repeatedly. This speeds up
repeated reads by a small but noticeable amount.

Other changes of note include:

- The ifile written by newfs_lfs can now be of arbitrary length, it is no
longer restricted to a single indirect block.

- Fixed an old bug where ctime was changed every time a vnode was created.
I need to look more closely to make sure that the times are only updated
during write(2) and friends, not after-the-fact during a segment write,
and certainly not by the cleaner.

branches: 1.4.2; 1.4.4; 1.4.6;
Correct accounting of lfs_avail, locked_queue_count, and locked_queue_bytes.
(PR #11468). In the case of fragment allocation, check to see if enough
space is available before extending a fragment already scheduled for writing.

The locked_queue_* variables indicate the number of buffer headers and bytes,
respectively, that are unavailable to getnewbuf() because they are locked up
waiting for LFS to flush them; make sure that that is actually what we're
counting, i.e., never count malloced buffers, and always use b_bufsize instead
of b_bcount.

If DEBUG is defined, the periodic calls to lfs_countlocked will now complain
if either counter is incorrect. (In the future lfs_countlocked will not need
to be called at all if DEBUG is not defined.)

branches: 1.3.8;
New CHANGES files that describes briefly all nontrivial changes made to
the LFS since the 4.4lite2 code was merged into NetBSD.

TODO updated to remove everything marked DONE in 4.4, and add in a list
of more current things to do.

Get rid of comments about the cleaner syscall code and missing fragment
support from README.

New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD'

branches: 1.1.1;
Update to 4.4-Lite fs code, with local changes.

Update inode disk addresses and do their segment accounting together,
rather than trying to coordinate two separate events.

Undo previous commit; it needs further refinement.

Track individual inodes between the time that their accounting is
subtracted from one segment before it is added to the new segment.
Counting duplicates is not sufficient. Addresses a "negative bytes"
panic.

Use a red/black tree, rather than a linked list, to store deferred
sergment byte count changes in truncation.

* Use the LFS_IFDIRTY flag to keep track of whether the Ifile has
ungathered changes during checkpoint. Modify LFS_WRITESEGENTRY and
make a new LFS_WRITEIENTRY macro to help track this. Prevents
an infinite loop in lfs_writeinode, previously caught with a panic.

* Fix a false-positive "negative bytes" panic when writing inodes into
the same segment they were in before, and consolidate the accounting
logic into new function lfs_subtract_inode.

* Ensure that we do not clean any inodes that were unavailable during
the first pass of cleaning, so that we never need to mix cleaning and
non-cleaning blocks (except for the Ifile) when writing.

* Straighten out the logic in lfs_writefile to make it more readable.

* Include IMNT_WANTRDONLY in tests for closing out writes, not just
IMNT_UNMOUNT.

* Keep track of whether the kernel cleaner is active on a file system
with a status flag; use a condvar to synchronously turn this on
or off.

Add in-kernel cleaner. Move cleaning-related routines into their own file.
Document the fcntl calls better in lfs.h. Bounds-check inode numbers in
segment parsing functions. Use the in-kernel cleaner in a greedy mode
for tests in tests/fs/vfs, and add a test of the kernel cleaner.

Remove su_flags array, replacing it with a new flag SEGUSE_READY.
Segments progress from having su_nbytes==0 to SEGUSE_EMPTY to SEGUSE_READY
to clean, progressing to the nest step after a checkpoint.

* Generalize the partial-segment parser introduced for roll-forward,
using it to facilitate an in-kernel segment rewriter (cleaner), and a
mechanism to check whether a segment is in fact empty (only used
with DEBUG).

* Add these new fcntl calls:
- LFCNFILESTATS: For each inode given, report its number of direct
blocks, how many gaps (discontinuities) there are between direct
blocks, and how large the total gap distance is. This will be
useful for a coalescing agent.
- LFCNREWRITEFILE: For each inode given, rewrite its direct blocks,
effectively coalescing it into as compact a form as possible.
- LFCNSCRAMBLE: As above, except that it only rewrites every other
block. This causes the file to have many gaps that can be
measured with LFCNFILESTATS and addressed with LFCNREWRITEFILE,
for testing purposes.
- LFCNREWRITESEGS: Rewrite any live data in the given segments.
This is intended to simplify the cleaner API and facilitate an
in-kernel cleaner.
- LFCNCLEANERINFO: Get the most current CLEANERINFO data from the
kernel.
- LFCNSEGUSE: Retrieve segment usage data from the kernel.

* Vnodes marked IN_CLEANING now take a reference. Add a new "cleaner
lock", which must be taken by the cleaner before the segment lock,
and before marking nodes IN_CLEANING. This allows us to flush
vnodes, if necessary, before the cleaning segment is written, and
never to flush vnodes being cleaned. When the cleaner lock is
released, the vnodes are cleared of IN_CLEANING and the reference
dropped.

* Track a potential infinite loop in lfs_gatherblock.

* Pull "needs to flush" and "needs to wait for flush" into functions
instead of inlining their definitions.

Add routines to check freelist consistency if compiled with DEBUG and
conditional on a kernel variable manipulated via sysctl.
Add checks before and after each routine that modifies the free list.
#if 0 a section of lfs_vfree() that was intended to keep the free list ordered
but instead corrupted it.

If setting the head (or tail) of the inode free list to LFS_UNUSED_INUM, also
set the tail (resp. head) to LFS_UNUSED_INUM, as the list is now empty.

Add a check to ensure that lfs_valloc_fixed will always terminate, even
if the free list should contain a loop. Extend the ifile at the end if it
is empty, to match the assumption of lfs_valloc() that the free list is
never empty.

Needed for roll-forward.

Comment out some of the CTASSERTS for lint until I fix lint.

Avoid misaligned access to lfs64 on-disk records in memory.

lfs64 directory entries are only 32-bit aligned in order to conserve
space in directory blocks, and we had a hack to stuff a 64-bit inode
in them. This replaces the hack by __aligned(4) __packed, and goes
further:

1. It's not clear that all the other lfs64 data structures are 64-bit
aligned on disk to begin with. We can go through these later and
upgrade them from

struct foo64 {
...
} __aligned(4) __packed;

union foo {
struct foo64 f64;
...
};

to

struct foo64 {
...
};

union foo {
struct foo64 f64 __aligned(8);
...
} __aligned(4) __packed;

if we really want to take advantage of 64-bit memory accesses.

However, the __aligned(4) __packed must remain on the union
because:

2. We access even the lfs32 data structures via a union that has
lfs64 members, and it turns out that compilers will assume access
through a union with 64-bit aligned members implies the whole
union has 64-bit alignment, even if we're only accessing a 32-bit
aligned member.

CTASSERT lfs on-disk structure sizes.

Teach LFS_ORPHAN_NEXTFREE about lfs64.

branches: 1.204.4; 1.204.6;
Update comment (overlooked in r1.179).
From Jos� Luis Rodr�guez Garc�a in PR kern/53849.

branches: 1.203.2; 1.203.4;
change lfs_nextsegsleep and lfs_allclean_wakeup to use condvar

XXX had to use lfs_lock in lfs_segwait, removed kernel_lock, is this
appropriate?

Move definition of IN_ALLMOD near the flag it's a mask for.

Now we can see that it doesn't match all the flags, but changing that will
require more careful thought.

branches: 1.201.6;
switch lfs_dirops to condvar (from mtsleep)

switch lfs_sleepers to condvar (from mtsleep)

branches: 1.199.2; 1.199.4;
Massedit u_int{8,16,32,64}_t to uint{8,16,32,64}_t. This effectively
merges ufs/dinode.h 1.25.

we are actually synced with ufs/dinode.h 1.24 and ufs/dir.h 1.25.

Update now-lying comment.

For now bitflip the lfs64 magic number.

This will be unflipped when the format is finalized - right now I
still have pending changes to the superblock in mind (to reduce the
number of redundant fields) so anything created now is not future-
proof. However, the code's also nearing being ready for testing; so
I'm doing this before turning it on as a precaution.

Move stuff from struct ulfsmount to struct lfs.

Add lfs_checkword type for reading checksum data out of structures.
This is always uint32_t, but having a name for it both makes things
clearer and avoids confusion about whether it should be 32 or 64 bit.

Note: deployed in only one place (that was erroneously tagged
ondisk32) so far.

Add an IINFO struct, which is like the FINFO struct but for the inode
blocks portion of the segment summary.

A segment summary block begins with a header (SEGSUM); the rest of the
block contains FINFO structures describing file blocks growing upward
from the bottom (after the header), and IINFO structures describing
inode blocks grown downward from the end of the block. (When they meet
the segment is full regardless of how many blocks might be left.)

IINFO contains just a block number, and until now this information was
handled by just using uint32_t*; switching to a structure will make
the code a lot easier to read, and also make it easier to have 32-bit
and 64-bit versions without making a mess.

This commit just adds the structures and accessors; they'll be
deployed into the code in subsequent commits.

Oops, I forgot to make the atime in the 64-bit IFILE 64 bits.
Correct that. Incompatible change, but no LFS64 volumes can have been
created yet.

Add 64-bit directory entry structures, and adjust accessors accordingly.

The LFS64 directory entry has a 64-bit inode number. This is stored as
two 32-bit values to avoid inducing 64-bit alignment requirements.

The exposed type for manipulating directory entries is now
LFS_DIRHEADER, following the same convention as e.g. IFILE and SEGUSE.
(But with LFS_ on it, because.)

Clean up struct lfs_dirtemplate.

Remove struct lfs_direct (no longer used) and update the big block
comment about directories.

Add an accessor function for directory names.

Move the header part of struct lfs_direct to its own structure.
(lfs_dirheader)

Take the opportunity to improve the directory generation code in
make_lfs.c. (Everything else was unaffected by virtue of using
accessor functions.)

Add and use accessor functions for more of the directory entry fields.

Add new accessors for the d_type and d_namlen fields of struct lfs_direct.
Napalm the old byteswap access logic for these.

Comments on directories.

This includes a description of the struct direct byteswap horrors that
ought to be propagated to ufs/ufs.

The ifile's inode number is constant. (it is always 1)

Therefore, storing the value in the superblock and reading it out
again is silly and offers the opportunity for it to become corrupted.
So, don't do that (most of the code already didn't) and use the
existing constant instead. Initialize new 32-bit superblocks with
the value for the sake of old userland programs, but don't keep the
value in the 64-bit superblock at all.

(approved by Margo Seltzer)

Make the inode fields in the 64-bit superblock 64 bits wide.
Reasoning as before.

Note that I am not going through and checking for 64->32 truncations
in inode numbers; I'm sure there are quite a few, but that's a project
for later.

Add byteswapping to the dinode accessors.

This prevents regressions in the ulfs code when switching to the new
accessors. Note that while adding byteswapping to the other accessors
is straightforward, I haven't done it yet; and that also is not enough
to make LFS_EI work, because there are places lying around that bypass
the accessors for one reason and another and all of them need to be
updated. That is going to have to wait for a later day as LFS_EI is
not on the critical path right now.

Hack up dinode usage to be 64 vs. 32 as needed. Part 1.

(This part changes the native lfs code; the ufs-derived code already
has 64 vs. 32 logic, but as aspects of it are unsafe, and don't
entirely interoperate cleanly with the lfs 64/32 stuff, pass 2 will be
rehashing that.)

Make the inode number in the 64-bit dinode 64 bits wide, like the
other lfs64 on-disk inode numbers; I've been doing that since this is
a new format and we may as well take the opportunity. This does assume
that more than 4 billion files on a single volume becomes desirable;
but for an average file size of 10K all that takes is a 40 TB volume,
and it's not that hard to make one of those these days if you want to
badly enough.

Provide 32-bit and 64-bit versions of FINFO.

This also entailed sorting out part of struct segment, as that
contains a pointer into the current FINFO data.

Make 32-bit and 64-bit versions of SEGSUM.
Also fix some of the FINFO handling as it's closely entangled.

Add IFILE32 and IFILE64 structures for the on-disk ifile entries.
Add and use accessors. There are also a bunch of places that cast and
I hope I've found them all...

Make 32-bit and 64-bit versions of CLEANERINFO.

XXX: while this is written to disk, it seems like much of it would
XXX: be better set up as a commpage shared with the cleaner.

Widen several of the fields of BLOCK_INFO to 64 bits.

Keep the old BLOCK_INFO as BLOCK_INFO_70, and version the fcntls that
use it.

Note that BLOCK_INFO_70 has 64-bit padding issues so that it's
different on 32-bit and 64-bit machines. This has been fixed. However,
BLOCK_INFO also contains a pointer, so compat32 stuff for 32-on-64 is
still needed and doesn't currently exist.

Fix assorted 64->32 truncations related to BLOCK_INFO.

Also make note of a cleaner limitation: it seems that when it goes to
coalesce discontiguous files, it mallocs an array with one BLOCK_INFO
for every block in the file. Therefore, with 64-bit LFS, on a 32-bit
platform it will be possible to have files large enough to overflow
the cleaner's address space. Currently these will be skipped and cause
warnings via syslog.

At some point someone should rewrite the logic to coalesce files to
use chunks of some reasonable size, as discontinuity between such
chunks is immaterial and mallocing this much space is silly and
fragile. Also, the kernel only accepts up to 65536 blocks at a time
for bmapv and markv, so processing more than this at once probably
isn't useful and may not even work currently. I don't want to change
this around just now as it's not entirely trivial.

Pass the fs object to LFS_MAX_DADDR so it can check lfs_is64.

Remove some hackish intentional 64->32 truncations next to the checks
using LFS_MAX_DADDR, and tackle the problem they handled in bmap
instead.

The problem: the magic block pointer value UNWRITTEN has magic value
-2, and if it's not handled specifically, uint32 -> uint64 promotion
turns it into 4294967294, which then causes consternation and
monkeyhouse downstream.

What's here is still kind of a hack, but it's a step forward.

Add a (draft) 64-bit superblock. Make things build again.

Add pieces of support for using both superblock types where
convenient, and specifically to the superblock accessors, but don't
actually enable it anywhere.

First substantive step on PR 50000.

lfs_cleanint[] in the in-memory superblock needs to have 64-bit entries.

Second batch of 64 -> 32 truncations in lfs, along with more minor
tidyups and corrections in passing.

Fix assorted 64 -> 32 truncations in lfs. Also, some minor tidyups and
corrections in passing.

Move struct salfs back inside libsa now that lfs_accessors.h is separate.

Add a new lfs header file: lfs_accessors.h.

This contains all the accessor functions and macros out of lfs.h.
Add an include of lfs_accessors.h after all uses of lfs.h... except
for code that wants to define its own struct lfs-alike that the
accessors are supposed to play along with. For these, set STRUCT_LFS
and include lfs_accessors.h after the necessary structure has been
defined, so that lfs_accessors.h can emit functions in terms of it.

More lfs superblock accessors.
(This changes the rest of the code over; all the accessors were
already added.)

The difference between this commit and the previous one is arbitrary,
but the previous one passed the regression tests on its own so I'm
keeping it separate to help with any bisections that might be needed
in the future.

Switch to accessor functions for elements of the LFS on-disk
superblock. This will allow switching between 32/64 bit forms on the
fly; it will also allow handling LFS_EI reasonably tidily. (That
currently doesn't work on the superblock.)

It also gets rid of cpp abuse in the form of fake structure member
macros.

Also, instead of doing sleep/wakeup on &lfs_avail and &lfs_nextseg
inside the on-disk superblock, add extra elements to the in-memory
struct lfs for this. (XXX: these should be changed to condvars, but
not right now)

XXX: this migrates a structure needed by the lfs code in libsa (struct
salfs) into lfs.h, where it doesn't belong, but for the time being
this is necessary in order to allow the accessors (and the various
lfs macros and other goop that relies on them) to compile.

ulfs2_dinode, having never actually been used with lfs, doesn't have a
di_inumber field. Fix that. First preliminary step on PR 50000.

Use VFS_PROTOS() for lfs.
Rename conflicting struct lfs field "lfs_start" to "lfs_s0addr".

No functional change.

Remove the 'cred' argument from bread(). Remove a now unused var in
ffs_snapshot.c. Update the man page accordingly.

ok hannken@

branches: 1.160.6;
Bring in a copy of ffs_quota2_mount() for reference.
Add stuff to struct lfs that it needs to initialize.
Clear these fields in mount as there's no on-disk support for quota2;
but this increases the chances of being able to add it (or something
like it) in the future.

Migrate the miscellaneous ulfs-level info from struct ulfsmount to
struct lfs.

Put them inside #ifdef _KERNEL there. They are not the only such
members, gross as that is. Unfortunately, moving struct lfs to
lfs_kernel.h does not work.

Add lfs_kernel.h for declarations that don't need to be exposed to userland.

lfs currently has the following headers:
lfs.h - on-disk structures and stuff needed for userlevel tools
lfs_inode.h - additional restricted materials for userlevel tools
that operate the fs (newfs_lfs, fsck_lfs, lfs_cleanerd)
lfs_kernel.h - stuff needed only in the kernel

and the following legacy headers that are expected to be mopped up and
folded into one of the above:
lfs_extern.h - function prototypes
ulfs_bswap.h - endian-independent support
ulfs_dinode.h - now contains very little
ulfs_dirhash.h - dirhash support
ulfs_extattr.h - extattr support
ulfs_extern.h - more function prototypes
ulfs_inode.h - assorted kernel-only declarations
ulfs_quota.h - quota support
ulfs_quota1.h - more quota support
ulfs_quota2.h - more quota support
ulfs_quotacommon.h - more quota support
ulfsmount.h - legacy copy of ufsmount material

branches: 1.157.2;
Remove duplicate define of LFS_MAXNAMLEN

typo in comment

Rename ambiguous macros:
MAXDIRSIZE -> UFS_MAXDIRSIZE or LFS_MAXDIRSIZE
NINDIR -> FFS_NINDIR, EXT2_NINDIR, LFS_NINDIR, or MFS_NINDIR
INOPB -> FFS_INOPB, LFS_INOPB
INOPF -> FFS_INOPF, LFS_INOPF
blksize -> ffs_blksize, ext2_blksize, or lfs_blksize
sblksize -> ffs_blksize

These are not the only ambiguously defined filesystem macros, of
course, there's a pile more. I may not have found all the ambiguous
definitions of blksize(), too, as there are a lot of other things
called 'blksize' in the system.

Prefix most of the cpp macros with lfs_ and LFS_ to avoid conflicts with ffs.
This was done so that boot blocks that want to compile both FFS and LFS in
the same file work.

Tuck away a bunch of symbols that don't need to be public.

Move struct lfs_inode_ext to lfs_inode.h; it doesn't need to be public.

Remove stale union and accessor macros.

ulfs_dir.h has been emptied; remove it.

Move a comment to lfs.h that belongs better there.

As nearly all the content of ulfs_dir.h and ulfs_dinode.h has migrated
to lfs.h, propagate the copyright notices too.

Move more symbols to lfs.h:
LFS_DIRBLKSIZ
LFS_DIRECTSIZ
LFS_DIRSIZ
LFS_OLDDIRFMT
LFS_NEWDIRFMT
LFS_IFTODT
LFS_DTTOIF
ULFS{,1,2}_MAXSYMLINKLEN

DIRBLKSIZ -> LFS_DIRBLKSIZ
DIRECTSIZ -> LFS_DIRECTSIZ
DIRSIZ -> LFS_DIRSIZ
OLDDIRFMT -> LFS_OLDDIRFMT
NEWDIRFMT -> LFS_NEWDIRFMT
IFTODT -> LFS_IFTODT
DTTOIF -> LFS_DTTOIF

Move stuff to lfs.h that's needed by userland:
LFS_DT_*
ULFS_ROOTINO
ULFS_WINO
struct lfs_direct
struct lfs_dirtemplate
struct lfs_odirtemplate
struct ulfs_args

Also fix FFS_MAXNAMLEN -> LFS_MAXNAMLEN in several places.

Now move LFS_IFMT and friends from ulfs_dinode.h to lfs.h.

Move the dinode (on-disk inode) structures to lfs.h, since they are
and will be obviously required by userland tools that need to read
the on-disk structures.

Also, DINODE{1,2}_SIZE -> LFS_DINODE{1,2}_SIZE.

Split the definitions suitable for userland out of ulfs_inode.h into
lfs_inode.h. Since fsck_lfs, newfs_lfs, and lfs_cleanerd want to reuse
the inode structure for their own internal use, and some of them share
parts of the kernel code as well, the best way forward is to provide a
relatively sanitized header that doesn't bring in stray material.

Shuffle a few other definitions around so that lfs_inode.h depends
only on lfs.h.

Install lfs_inode.h into /usr/include.

Fix some exposed symbols:
LOSTFOUNDINO -> LFS_LOSTFOUNDINO
struct ufid -> struct ulfs_ufid

Cleanups to reduce symbol and header exposure:
- move struct ufid from ulfs_inode.h to lfs.h
- lfs.h needs sys/mount.h and sys/pool.h
- ulfs_quota2_subr.c needs lfs_inode.h
- remove ulfs_inode.h from lfs.h in favor of ulfs_dinode.h
- move ULFS_NDADDR, ULFS_NIADDR, ULFS_NXADDR from ulfs_dinode.h to lfs.h
- remove ulfs_dinode.h from lfs.h
- add lfs.h to ulfs_dinode.h

Remove stray references to ext2fs, chfs, ffs, and mfs.

Split lfs from ufs step 4:

Massedit all ufs symbols to be "ulfs" instead, to make sure there are
no conflicts with ufs. Confirmed with grep.

(This required changing a few comments that maybe should have been
left alone to say "ulfs", but we'll survive that.)

Stuff UFS_ in front of a few of ufs's symbols to reduce namespace
pollution. Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN -> UFS_MAXSYMLINKLEN
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Sort out ext2fs's misuse of NDADDR and NIADDR; fortunately, these have
the same values in ext2fs and ffs.

No functional change intended.

branches: 1.136.2;
Pass t_renamerace and t_rmdirrace tests.

Adapt dholland@'s fix to ufs_rename to fix PR kern/43582. Address several
other MP locking issues discovered during the course of investigating the
same problem.

Removed extraneous vn_lock() calls on the Ifile, since the Ifile writes
are controlled by the segment lock.

Fix PR kern/45982 by deemphasizing the estimate of how much metadata
will fill the empty space on disk when the disk is nearly empty
(t_renamerace crates a lot of inode blocks on a tiny empty disk).

branches: 1.135.2;

* Remove PGO_RECLAIM during lfs_putpages()' call to genfs_putpages(),
to avoid a live lock in the latter when reclaiming a vnode with
dirty pages.

* Add a new segment flag, SEGM_RECLAIM, to note when a segment is
being written for vnode reclamation, and record which inode is being
reclaimed, to aid in forensic debugging.

* Add a new segment flag, SEGM_SINGLE, so that opportunistic writes
can write a single segment's worth of blocks and then stop, rather
than writing all the way up to the cleaner's reserved number of
segments.

* Add assert statements to check mutex ownership is the way it ought
to be, mostly in lfs_putpages; fix problems uncovered by this.

* Don't clear VU_DIROP until the inode actually makes its way to disk,
avoiding a problem where dirop inodes could become separated
(uncovered by a modified version of the "ckckp" forensic regression
test).

* Move the vfs_getopsbyname() call into lfs_writerd. Prepare code to
make lfs_writerd notice when there are no more LFSs, and exit losing
the reference, so that, in theory, the module can be unloaded. This
code is not enabled, since it causes a crash on exit.

* Set IN_MODIFIED on inodes flushed by lfs_flush_dirops. Really we
only need to set IN_MODIFIED if we are going to write them again
(e.g., to write pages); need to think about this more.

Finally, several changes to help avoid "no clean segments" panics:

* In lfs_bmapv, note when a vnode is loaded only to discover whether
its blocks are live, so it can immediately be recycled. Since the
cleaner will try to choose ~empty segments over full ones, this
prevents the cleaner from (1) filling the vnode cache with junk, and
(2) squeezing any unwritten writes to disk and running the fs out of
segments.

* Overestimate by half the amount of metadata that will be required
to fill the clean segments. This will make the disk appear smaller,
but should help avoid a "no clean segments" panic.

* Rearrange lfs_writerd. In particular, lfs_writerd now pays
attention to the number of clean segments available, and holds off
writing until there is room.

branches: 1.134.2; 1.134.6;
Change VOP_BWRITE() to take a vnode as its first argument like all other
VOPs do. Layered file systems no longer have to modify bp->b_vp and run
into trouble when an async VOP_BWRITE() uses the wrong vnode.

- change all occurences of VOP_BWRITE(bp) to VOP_BWRITE(bp->b_vp, bp).
- remove layer_bwrite().
- welcome to 5.99.55

Adresses PR kern/38762 panic: vwakeup: neg numoutput

No objections from tech-kern@.

Three changes in a single commit.

- drop the notion of frags (LFS fragments) vs fsb (FFS fragments)
The code uses a complicated unity function that just makes the
code difficult to understand.

- support larger sector sizes. Fix disk address computations
to use DEV_BSIZE in the kernel as required by device drivers
and to use sector sizes in userland.

- Fix several locking bugs in lfs_bio.c and lfs_subr.c.

branches: 1.132.2;
... actually, define compat only for the kernel. Userlandia should
see only one version of the interfaces.

Include compat/sys/time_types.h instead of compat/sys/time.h.
Fixes lint drama with interface name collisions.

Include compat code by default.

PR/42246: NAKAJIMA Yoshihiro: provide COMPAT_50 for LFS

typo in comment

branches: 1.127.12;
Make sure all cached buffers with valid, not yet written data have been
run through copy-on-write. Call fscow_run() with valid data where possible.

The LP_UFSCOW hack is no longer needed to protect ffs_copyonwrite() against
endless recursion.

- Add a flag B_MODIFY to bread(), breada() and breadn(). If set the caller
intends to modify the buffer returned.

- Always run copy-on-write on buffers returned from ffs_balloc().

- Add new function ffs_getblk() that gets a buffer, assigns a new blkno,
may clear the buffer and runs copy-on-write. Process possible errors
from getblk() or fscow_run(). Part of PR kern/38664.

Welcome to 4.99.63

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.126.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.125.6; 1.125.8; 1.125.10;
The buffer LOCKED flag need not be under the protection of bufcache_lock,
BUSY is enough.

Use pool_cache.

Merge vmlocking2 to head.

branches: 1.122.4; 1.122.6; 1.122.10;
Merge from vmlocking:

- Split vnode::v_flag into three fields, depending on field locking.
- simple_lock -> kmutex in a few places.
- Fix some simple locking problems.

Merge ffs locking & brelse changes from the vmlocking branch.

branches: 1.120.6; 1.120.8; 1.120.10;
Change references to SEGM_W_DIROPS to SEGM_CKP, and replace the logic that
formerly used SEGM_W_DIROPS in lfs_segwrite() appropriately. This prevents
a problem in which processes could get stuck in "buffers" sleep forever.

Install a new sysctl, vfs.lfs.ignore_lazy_sync, which causes LFS to ignore
the "smooth" syncer, as if vfs.sync.*delay = 0, but only for LFS. The
default is "on", i.e., ignore lazy sync.

Reduce the amount of polling/busy-waiting done by lfs_putpages(). To
accomplish this, copied genfs_putpages() and modified it to indicate which
page it was that caused it to return with EDEADLK. fsync()/fdatasync()
should no longer ever fail with EAGAIN, and should not consume huge
quantities of cpu.

Also, try to make dirops less likely to be written as the result of a
VOP_PUTPAGES(), while ensuring that they are written regularly.

branches: 1.118.2; 1.118.6; 1.118.8;
Replace some uses of lockmgr() / simplelocks.

Use lockstatus instead of a homebrewed locking system to control
LFCNWRAPSTOP and LFCNWRAPGO.

Be less verbose about the various looping checks: use log() rather than
printf(), and only log anything if we are really looping ("count = 2" is
not an error condition).

Allow dirops sleeping on available space to be interruptible.

branches: 1.116.2;
Don't remark a locked inode with IN_MODIFIED after writing it to disk,
if we ourselves hold the lock. This prevents e.g. mknod from hanging
indefinitely.

Also, always use the return value from VOP_ISLOCKED to determine whether
we hold the lock or someone else does, rather than looking into the lock
structure ourselves.

merge yamt-pdpolicy branch.
- separate page replacement policy from the rest of kernel
- implement an alternative replacement policy

branches: 1.114.2;
Changes to help the roll-forward agent, to wit:

* Mark being-deleted files in the Ifile so we can finish deleting them
at fs mount time.
* Flag the Ifile with "cleaner must clean" when writers are waiting for
the cleaner, rather than relying solely on the cleaner's estimation of
whether it should clean or not.
* Note partial segments written by a user agent (in particular,
fsck_lfs) so that repeated rolls forward don't interfere with one
another.
* Add a new fcntl, LFCNPASS, that allows the log to wrap exactly once,
for better testing of the validity of checkpoints.
* Keep track of the on-disk nlink count when cleaning, so that we don't
partially complete directory operations while cleaning.
* Ensure that every single Ifile inode write represents a consistent
view of the filesystem. In particular, the accounting for the segment
we are writing the inode into must be correct, and the accounting for
the segment that inode used to reside in must be correct. Rather than
just rewriting the inode if we wrote it wrong, rewrite the necessary
ifile blocks before writing the inode so we never write it wrong.
* Don't unmark any VDIROP vnodes if we haven't written them to disk,
avoiding yet another problem with the "wait for the cleaner" error
return from lfs_putpages().

Also, move the last callback to an aiodone call, so we no longer do any
memory management from interrupt context.

Fix size confusion with lfs_fhandle - and as it now turns out to be the same
as the lfs compat_30_fhandle, g/c the latter.
Add an alias for the LFCNIFILEFH fcntl, so that binaries compiled in the
meantime (with too large lfs_fhandle) continue to work.

This makes vfs_cleanerd work again after the kernel checks filehandle size
more strictly (problem reported by Kurt Schreiner on current-users).

Make filehandles opaque to userland

Note partial segments that are written by the cleaner, to help out the
roll-forward agent.

Version the lfs_cleanerd internal fcntl() for filehandles too,
so old cleaners should work with newer kernels.

Fix alignement problems for fhandle_t, exposed by gcc4.1.

While touching all vptofh/fhtovp functions, get rid of VFS_MAXFIDSIZ,
version the getfh(2) syscall and explicitly pass the size available in
the filehandle from userland.

Discussed on tech-kern, with lots of help from yamt (thanks!).

Change LFCNWRAP{STOP,GO} to make them more suitable for snapshotting; in
particular, the caller can now choose whether to wait for the condition
to be met, and if the caller of LFCNWRAPSTOP dies or otherwise closes
the descriptor, the filesystem is started again. Updated the ckckp
regression test to use the new semantics.

dump_lfs(8) now uses the fcntls to implement LFS-style snapshotting through
the -X flag, addressing PR#33457 albeit not using fss(4). Fixed a couple
other problems with dump_lfs that manifested themselves during testing.

branches: 1.107.4;
integrate kauth.

Fixes to address the "vinvalbuf: dirty blocks" panic that can occur when
many inodes are cleaned at once. Make sure that we write all the pages
on vnodes that are being flushed, even if we don't think there's room;
drain v_numoutput before lfs_vflush() completes.

Also, don't allow a vnode that is in the process of being cleaned to be
chosen by getnewvnode(); this avoids a segment accounting panic in the case
that a large number of inodes are fed to lfs_markv() all at once.

Introduce another per-filesystem parameter, lfs_resvseg, to separate the
notion of "how many segments are reserved for the cleaner" from that of
"how many segments are not counted in lfs_bfree". The default value
used for existing filesystems is the same as the previous implicit value
of (lfs_minfreeseg / 2 + 1), modulo some sanity checking.

Count pending dirops on a per-filesystem basis, since once we start
writing them we can't stop until we're done. This seems to help stave off
the "no clean segments" panic in the case of filling the filesystem with
directories and small files (e.g. simultaneously unpacking more copies of
pkgsrc than will fit).

Postpone the segment accounting changes coming from truncation until the
inode that makes those changes valid is either written to disk by
lfs_writeinode() or discarded by lfs_vfree().

A couple of locking fixes are also included as well.

Introduce two fcntl calls that freeze the filesystem right at the point
where segment 0 is being considered for writing. This allows for automated
checkpoint vailidity scanning, and could be used (in conjunction with the
existing LFCNREWIND) for e.g. snapshot dumps as well.

Include a regression test that does such scanning.

When writing the Ifile, loop through the dirty block list three times to
make sure that the checkpoint is always consistent (the first and second
times the Ifile blocks can cross a segment boundary; not so the third time
unless the segments are very small). Discovered by using the aforementioned
regression test.

Make lfs_vref/lfs_vunref not need to know about VXLOCK and VFREEING
explicitly (especially since we didn't know about VFREEING at all before),
but notice the EBUSY return from vget() instead.

Fix some more MP locking protocol issues, most of which were pointed out by
Christian Ehrhardt this morning on tech-kern.

Optimize the free list search a little more; in particular use words
instead of bytes for the index, and never search below fs->lfs_freehd.

Fix a bug in the previous version of the search (an erroneous assumption
that ino_t was signed).

Free the bitmap when we unmount the filesystem.

Implement a somewhat finer-grained mechanism for paging LFS-backed pages.
The writer daemon, if it does not need to flush the whole filesystem,
now only writes the vnodes for which the pagedaemon has requested pageouts
(although it does not pay attention to the page ranges the pagedaemon
supplies).

Keep the free list ordered. This solves a problem first pointed out to me
by Michel Oey, in which an aged LFS writes up to an extra Ifile block for
every file created; and paves the way for the truncation of the Ifile when
many files are deleted.

Make the segment lock aware of LWPs. Fixes a (somewhat confusing)
"lockmgr: pid 3997, not exclusive lockholder 3997, unlocking" panic I
encountered while running blogbench on an LFS.

Improvements to LFS's paging mechanism, to wit:

* Acknowledge that sometimes there are more dirty pages to be written to
disk than clean segments. When we reach the danger line,
lfs_gop_write() now returns EAGAIN. The caller of VOP_PUTPAGES(), if
it holds the segment lock, drops it and waits for the cleaner to make
room before continuing.

* Note and avoid a three-way deadlock in lfs_putpages (a writer holding
a page busy blocks on the cleaner while the cleaner blocks on the
segment lock while lfs_putpages blocks on the page).

From Konrad Schroeder, in response to strange df output on anoncvs.netbsd.org:
We were returning the wrong value for free space. Now we're not.

branches: 1.95.4; 1.95.6; 1.95.8; 1.95.10; 1.95.12;
merge ktrace-lwp.

split out lfs_itimes(). It is used in fsck_lfs.

Use nanotime() to update the time fields in filesystems. Convert the code
from macros to real functions. Original patch and review from chuq.
Note: ext2fs only keeps seconds in the on-disk inode, and msdosfs does not
have enough precision for all fields, so this is not very useful for those
two.

Don't overload MAXNAMLEN, use a separate constant for each filesystem type.

whitespace.

change ino_t to u_int32_t for syscall compatibility.

Move extern kernel variable declarations, into a _KERNEL protected session
so that the don't pollute userland's namespace.

branches: 1.88.2;
- sprinkle const
- avoid shadow variables.

Keep track of the number of segments reclaimed, since the cleaner doesn't
do this anymore (it hasn't for quite some time). Add a couple of conditional
debugging messages to indicate why segments are not cleaned, in the event
that lfs_segclean is used.

Make the LFCNSEGWAITALL fcntl work again.

Provide a resize_lfs(8), including kernel and cleaner support. The current
implementation requires the fs to be mounted while resizing. Tested in both
directions, and everything appears to work happily, but ymmv.

Keep per-inode, per-fs, and subsystem-wide counts of blocks allocated through
lfs_balloc(), and use that to estimate the number of dirty pages belonging
to LFS (subsystem or filesystem). This is almost certainly wrong for
the case of a large mmap()ed region, but the accounting is tighter than
what we had before, and performs much better in the typical case of pages
dirtied through write().

Make userland compile again.

Use splay trees, rather than a hash table, to manage the accounting of
blocks allocated through VOP_BALLOC() for pages to be written to disk.
This accounting no longer takes a noticeable fraction of the system CPU.

Use lfs_malloc() to manage the blkiov arrays that the cleaner functions use,
since the cleaner is likely to operate in a low-memory condition.

Tabify leading whitespace

Consolidate the hash table we use to maintain the integrity of lfs_avail
into a single, system-wide table, rather than having a separate hash table
per inode. Significantly reduces the "system" cpu usage of your average
file write.

Keep track of the highest block held by an LFS inode, so that we can
be assured that the last byte of a file is always allocated. Previously
a file extension could cause the filesystem to be flushed, writing an
inconsistent inode to disk. Although this condition would be corrected
the next time blocks were written to disk, an intervening crash would leave
the filesystem in an inconsistent state, leaving fsck_lfs to complain
of an inode "partially truncated".

Protect various per-fs structures with fs->lfs_interlock simple_lock, to
improve behavior in the multiprocessor case. Add debugging segment-lock
assertion statements.

branches: 1.77.2;
Straighten out the maze of ifdefs. Instead, consolidate all the debugging
stuff under '#ifdef DEBUG', and use sysctl knobs to turn on/off particular
parts of the debugging reporting (if DEBUG is enabled). Re-enable the LFS
statistics in sysctl, while I'm there. A bit of a rototill.

nuke trailing whitespace

Various minor LFS improvements:

* Note when lfs_putpages(9) thinks it is not going to be writing any
pages before calling genfs_putpages(9). This prevents a situation in
which blocks can be queued for writing without a segment header.
* Correct computation of NRESERVE(), though it is still a gross
overestimate in most cases. Note that if NRESERVE() is too high, it
may be impossible to create files on the filesystem. We catch this
case on filesystem mount and refuse to mount r/w.
* Allow filesystems to be mounted whose block size is == MAXBSIZE.
* Somewhere along the line, ufs_bmaparray(9) started mangling UNWRITTEN
entries in indirect blocks again, triggering a failed assertion "daddr
<= LFS_MAX_DADDR". Explicitly convert to and from int32_t to correct
this.
* Add a high-water mark for the number of dirty pages any given LFS can
hold before triggering a flush. This is settable by sysctl, but off
(zero) by default.
* Be more careful about the MAX_BYTES and MAX_BUFS computations so we
shouldn't see "please increase to at least zero" messages.
* Note that VBLK and VCHR vnodes can have nonzero values in di_db[0]
even though their v_size == 0. Don't panic when we see this.
* Change lfs_bfree to a signed quantity. The manner in which it is
processed before being passed to the cleaner means that sometimes it
may drop below zero, and the cleaner must be aware of this.
* Never report bfree < 0 (or higher than lfs_dsize) through
lfs_statvfs(9). This prevents df(1) from ever telling us that our full
filesystems have 16TB free.
* Account space allocated through lfs_balloc(9) that does not have
associated buffer headers, so that the pagedaemon doesn't run us out
of segments.
* Return ENOSPC from lfs_balloc(9) when bfree drops to zero.
* Address a deadlock in lfs_bmapv/lfs_markv when the filesystem is being
unmounted. Because vfs_busy() is a shared lock, and
lfs_bmapv/lfs_markv mark the filesystem vfs_busy(), the cleaner can be
holding the lock that umount() is blocking on, then try to vfs_busy()
again in getnewvnode().

branches: 1.74.4; 1.74.6;
Push atime/mtime updates even further -- into the reclaim path, so they happen
rarely in the normal case. (Note: This happens at reboot/shutdown time because
all file systems are unmounted.)

Also, for IN_MODIFY, use IN_ACCESSED, not IN_MODIFIED; otherwise "ls -l" of
your device node or FIFO would cause the time stamps to get written too
quickly.

Add a new flag, IN_MODIFY. This is like IN_UPDATE|IN_CHANGE, but unlike
setting those flags, it does not cause the inode to be written in the periodic
sync. This is used for writes to special files (devices and named pipes) and
FIFOs.

Do not preemptively sync updates to access times and modification times. They
are now updated in the inode only opportunistically, or when the file or device
is closed. (Really, it should be delayed beyond close, but this is enough to
help substantially with device nodes.)

And the most amusing part:
Trickle sync was broken on both FFS and ext2fs, in different ways. In FFS, the
periodic call to VFS_SYNC(MNT_LAZY) was still causing all file data to be
synced. In ext2fs, it was causing the metadata to *not* be synced. We now
only call VOP_UPDATE() on the node if we're doing MNT_LAZY. I've confirmed
that we do in fact trickle correctly now.

branches: 1.72.4;
use correct segment size. this fixes memory corruption when using lfsv1.

use bufmem instead of bufpages to make lfs a little less broken.

- raise spl to bio in lfs_countlocked() rather than having callers to do so.
- buffer cache MP locks.
- assert B_CALL buffers are not on the free queue.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

using normal bufcache buffer for cluster buffer head.

- protect global resource counts with lfs_subsys_lock.
- clean up scattered externs a little.

use queue.h macros.

- add a new functions, lfs_writer_enter/leave, and use them instead of
duplicated code fragments.
- add an assertion.

branches: 1.64.2;
Make LFS work better (though still not "well") as an NFS-exported
filesystem (and other things that needed to be fixed before the tests
would complete), to wit:

* Include the fs ident in the filehandle; improve stale filehandle checks.

* Change definition of blksize() to use the on-dinode size instead of
the inode's i_size, so that fsck_lfs will work properly again.

* Use b_interlock in lfs_vtruncbuf.

* Postpone dirop reclamation until after the seglock has been released,
so that lfs_truncate is not called with the segment lock held.

* Don't loop in lfs_fsync(), just write everything and wait.

* Be more careful about the interlock/uobjlock in lfs_putpages: when we
lose this lock, we have to resynchronize dirtiness of pages in each
block.

* Be sure to always write indirect blocks and update metadata in
lfs_putpages; fixes a bug that caused blocks to be accounted to the
wrong segment.

Use PAGE_SIZE rather than NBPG.

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Add a sleeper count, to prevent the cleaner from panicing the kernel
when the filesystem is unmounted, relocking the Ifile when its lock is
draining. (We can't use vfs_busy() since the process is sleeping for a
good long time.) Clean up / organize lfs.h, while I'm here.

In lfs_update_single, assert that disk addresses are either negative, or
are still positive when converted to int32_t, to prevent recurrence of a
negative/positive block problem.

KNF (space after keywords).

Use VONWORKLST as a heuristic for vnode emptiness, rather than exhaustively
checking the memq.

Take greater care not to dirty the Ifile vnode when unmounting the filesystem.
This should fix a "(vp->v_flag & VONWORKLST) == 0" assertion panic in vgonel
that could occur when unmounting.

Do not allow the Ifile to be mapped for writing.

Add simple_lock protection for lfs_seglock and lfs_subsys_pages; these will
be expanded to cover other per-fs and subsystem-wide data as well.

Fix a case of IN_MODIFIED being set without updating lfs_uinodes, resulting
in a "lfs_uinodes < 0" panic.

Fix a deadlock in lfs_putpages arising from the need to busy all pages in a
block; unbusy any that had already been busied before starting over.

- Get rid of unused #ifdefs LFS_NO_PAGEMOVE and LFS_MALLOC_SUMMARY (both
always true) and accompanying dead code.

- When constructing write clusters in lfs_writeseg, if the block we are
about to add is itself a cluster from GOP_WRITE, don't put a cluster
in a cluster, just write the GOP_WRITE cluster on its own. This seems
to represent a slight performance gain on my test machine.

- Charge someone's rusage for writes on LFSes. It's difficult to tell
who the "right" process to charge is; just charge whoever triggered
the write.

Take away "#ifdef LFS_UBC".

Add an lfs_strategy() that checks to make sure we're not trying to read
where the cleaner is trying to write, instead of tying up the "live"
buffers (or pages).

Fix a bug in the LFS_UBC case where oversized buffers would not be
checksummed correctly, causing uncleanable segments.

Make sure that wakeup(fs->lfs_iocount) is done if fs->lfs_iocount is 1
as well as 0, since we wait in some places for it to drop to 1.

Activate all pages that make it into lfs_gop_write without the segment
lock held, since they must have been dirtied very recently, even if
PG_DELWRI is not set.

Account SEGUSE_ACTIVE correctly so that the automatic segment cleaning
actually happens.

Add a new fcntl call that will write the minimum necessary to checkpoint
(i.e., for on-disk directory structure to be consistent, not including
updates to file data) so that the cleaner can clean segments more quickly
without sacrificing three-way commit for cleaning.

Do roundup and offset arithmetic in 64 bits, to allow >=2G files.

Make fs-specific fcntl macros take three arguments (approved wrstuden).
Let LFS use fcntl for cleaner functions.

Add lfs_ioctl vnode op, with ioctls to take over cleaner system call
functionality (not including segment clean, since that is now done
automatically as checkpoints happen).

Fix a buffer overflow bug in the LFS_UBC case that manifested itself
either as a mysterious UVM error or as "panic: dirty bufs". Verify
maximum size in lfs_malloc.

Teach lfs_updatemeta and lfs_shellsort about oversized cluster blocks from
lfs_gop_write.

When unwiring pages in lfs_gop_write, deactivate them, under the theory
that the pagedaemon wanted to free them last we knew.

Tabify, and fix some comment alignment problems.

workaround for "another flush is..." infinity loop in writerd.
if we're writerd, sleep in lfs_flush until another writer goes away
instead of busy loop in writed.

Make libsa compile again.

Add code to UBCify LFS. This is still behind "#ifdef LFS_UBC" for now
(there are still some details to work out) but expect that to go
away soon. To support these basic changes (creation of lfs_putpages,
lfs_gop_write, mods to lfs_balloc) several other changes were made, to
wit:

* Create a writer daemon kernel thread whose purpose is to handle page
writes for the pagedaemon, but which also takes over some of the
functions of lfs_check(). This thread is started the first time an
LFS is mounted.

* Add a "flags" parameter to GOP_SIZE. Current values are
GOP_SIZE_READ, meaning that the call should return the size of the
in-core version of the file, and GOP_SIZE_WRITE, meaning that it
should return the on-disk size. One of GOP_SIZE_READ or
GOP_SIZE_WRITE must be specified.

* Instead of using malloc(...M_WAITOK) for everything, reserve enough
resources to get by and use malloc(...M_NOWAIT), using the reserves if
necessary. Use the pool subsystem for structures small enough that
this is feasible. This also obsoletes LFS_THROTTLE.

And a few that are not strictly necessary:

* Moves the LFS inode extensions off onto a separately allocated
structure; getting closer to LFS as an LKM. "Welcome to 1.6O."

* Unified GOP_ALLOC between FFS and LFS.

* Update LFS copyright headers to correct values.

* Actually cast to unsigned in lfs_shellsort, like the comment says.

* Keep track of which segments were empty before the previous
checkpoint; any segments that pass two checkpoints both dirty and
empty can be summarily cleaned. Do this. Right now lfs_segclean
still works, but this should be turned into an effectless
compatibility syscall.

don't use daddr_t for segment summary since it's an on-disk structure.

make these compilable with lfs debug options.
(follow daddr_t change)

XXX maybe segment number should be 64bit.

Bump daddr_t to 64 bits. Replace it with int32_t in all places where
it was used on-disk, so that on-disk formats remain the same.
Remove ufs_daddr_t and ufs_lbn_t for the time being.

Add multiple inclusion protection for headers. Fix mismatched
variable declarations (missing const's) as needed.

Deal with fragment size changes better. For each fragment that can
exist on an on-disk inode, we keep a record of its size in struct inode,
which is updated when we write the block to disk. The cleaner routines
thus have ready access to what size is the correct size for this block,
on disk.

Fixed a related bug: if a file with fragments is being cleaned
(fragments being cleaned) at the same time it is being extended beyond
NDADDR blocks, we could write a bogus FINFO record that has a frag in the
middle; when it was cleaned this would give back bogus file data. Don't
write the indirect blocks in this case, since there is no need.

lfs_fragextend and lfs_truncate no longer require the seglock, but instead
take a shared lock, which the seglock locks exclusively.

For synchronous writes, keep separate i/o counters for each write, so
processes don't have to wait for one another to finish (e.g., nfsd seems
to be a little happier now, though I haven't measured the difference).
Synchronous checkpoints, however, must always wait for all i/o to finish.

Take the contents of the callback functions and have them run in thread
context instead (aiodoned thread). lfs_iocount no longer has to be
protected in splbio(), and quite a bit less of the segment construction
loop needs to be in splbio() as well.

If lfs_markv is handed a block that is not the correct size according to
the inode, refuse to process it. (Formerly it was extended to the "correct"
size.) This is possibly more prone to deadlock, but less prone to corruption.

lfs_segclean now outright refuses to clean segments that appear to have live
bytes in them. Again this may be more prone to deadlock but avoids
corruption.

Replace ufsspec_close and ufsfifo_close with LFS equivalents; this means
that no UFS functions need to know about LFS_ITIMES any more. Remove
the reference from ufs/inode.h.

Tested on i386, test-compiled on alpha.

branches: 1.39.2; 1.39.4;
Phase one of my three-phase plan to make LFS play nice with UBC, and bug-fixes
I found while making sure there weren't any new ones.

* Make the write clusters keep track of the buffers whose blocks they contain.
This should make it possible to (1) write clusters using a page mapping
instead of malloc, if desired, and (2) schedule blocks for rewriting
(somewhere else) if a write error occurs. Code is present to use
pagemove() to construct the clusters but that is untested and will go away
anyway in favor of page mapping.
* DEBUG now keeps a log of Ifile writes, so that any lingering instances of
the "dirty bufs" problem can be properly debugged.
* Keep track of whether the Ifile has been dirtied by various routines that
can be called by lfs_segwrite, and loop on that until it is clean, for
a checkpoint. Checkpoints need to be squeaky clean.
* Warn the user (once) if the Ifile grows larger than is reasonable for their
buffer cache. Both lfs_mountfs and lfs_unmount check since the Ifile can
grow.
* If an inode is not found in a disk block, try rereading the block, under
the assumption that the block was copied to a cluster and then freed.
* Protect WRITEINPROG() with splbio() to fix a hang in lfs_update.

add spaces for KNF. confirmed to produce identical objects.

Merge the short-lived perseant-lfsv2 branch into the trunk.

Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default. Changes for the v2 layout include:

- Segments of non-PO2 size and arbitrary block offset, so these can be
matched to convenient physical characteristics of the partition (e.g.,
stripe or track size and offset).

- Address by fragment instead of by disk sector, paving the way for
non-512-byte-sector devices. In theory fragments can be as large
as you like, though in reality they must be smaller than MAXBSIZE in size.

- Use serial number and filesystem identifier to ensure that roll-forward
doesn't get old data and think it's new. Roll-forward is enabled for
v2 filesystems, though not for v1 filesystems by default.

- The inode free list is now a tailq, paving the way for undelete (undelete
is not yet implemented, but can be without further non-backwards-compatible
changes to disk structures).

- Inode atime information is kept in the Ifile, instead of on the inode;
that is, the inode is never written *just* because atime was changed.
Because of this the inodes remain near the file data on the disk, rather
than wandering all over as the disk is read repeatedly. This speeds up
repeated reads by a small but noticeable amount.

Other changes of note include:

- The ifile written by newfs_lfs can now be of arbitrary length, it is no
longer restricted to a single indirect block.

- Fixed an old bug where ctime was changed every time a vnode was created.
I need to look more closely to make sure that the times are only updated
during write(2) and friends, not after-the-fact during a segment write,
and certainly not by the cleaner.

branches: 1.36.2; 1.36.4; 1.36.6;
replace \<space(s)><newline> (wrong!) with \<newline>

Correct accounting of lfs_avail, locked_queue_count, and locked_queue_bytes.
(PR #11468). In the case of fragment allocation, check to see if enough
space is available before extending a fragment already scheduled for writing.

The locked_queue_* variables indicate the number of buffer headers and bytes,
respectively, that are unavailable to getnewbuf() because they are locked up
waiting for LFS to flush them; make sure that that is actually what we're
counting, i.e., never count malloced buffers, and always use b_bufsize instead
of b_bcount.

If DEBUG is defined, the periodic calls to lfs_countlocked will now complain
if either counter is incorrect. (In the future lfs_countlocked will not need
to be called at all if DEBUG is not defined.)

Remove debugging code that accidentally went in with yesterday's commit.

Do not needlessly dirty segment table blocks during lfs_segwrite,
preventing needless disk activity when the filesystem is idle. (PR #10979.)

Cast back to int32_t in LFS_EST_BFREE and LFS_EST_RSVD macros, for
consistency with their arguments.

Change the debugging printf in lfs_reserve to match, and enclose it in
#ifdef DEBUG.

Tested on alpha, arm32, sparc.

Various bug-fixes to LFS, to wit:


Kernel:

* Add runtime quantity lfs_ravail, the number of disk-blocks reserved
for writing. Writes to the filesystem first reserve a maximum amount
of blocks before their write is allowed to proceed; after the blocks
are allocated the reserved total is reduced by a corresponding amount.

If the lfs_reserve function cannot immediately reserve the requested
number of blocks, the inode is unlocked, and the thread sleeps until
the cleaner has made enough space available for the blocks to be
reserved. In this way large files can be written to the filesystem
(or, smaller files can be written to a nearly-full but thoroughly
clean filesystem) and the cleaner can still function properly.

* Remove explicit switching on dlfs_minfreeseg from the kernel code; it
is now merely a fs-creation parameter used to compute dlfs_avail and
dlfs_bfree (and used by fsck_lfs(8) to check their accuracy). Its
former role is better assumed by a properly computed dlfs_avail.

* Bounds-check inode numbers submitted through lfs_bmapv and lfs_markv.
This prevents a panic, but, if the cleaner is feeding the filesystem
the wrong data, you are still in a world of hurt.

* Cleanup: remove explicit references of DEV_BSIZE in favor of
btodb()/dbtob().

lfs_cleanerd:

* Make -n mean "send N segments' blocks through a single call to
lfs_markv". Previously it had meant "clean N segments though N calls
to lfs_markv, before looking again to see if more need to be cleaned".
The new behavior gives better packing of direct data on disk with as
little metadata as possible, largely alleviating the problem that the
cleaner can consume more disk through inefficient use of metadata than
it frees by moving dirty data away from clean "holes" to produce
entirely clean segments.

* Make -b mean "read as many segments as necessary to write N segments
of dirty data back to disk", rather than its former meaning of "read
as many segments as necessary to free N segments worth of space". The
new meaning, combined with the new -n behavior described above,
further aids in cleaning storage efficiency as entire segments can be
written at once, using as few blocks as possible for segment summaries
and inode blocks.

* Make the cleaner take note of segments which could not be cleaned due
to error, and not attempt to clean them until they are entirely free
of dirty blocks. This prevents the case in which a cleanerd running
with -n 1 and without -b (formerly the default) would spin trying
repeatedly to clean a corrupt segment, while the remaining space
filled and deadlocked the filesystem.

* Update the lfs_cleanerd manual page to describe all the options,
including the changes mentioned here (in particular, the -b and -n
flags were previously undocumented).

fsck_lfs:

* Check, and optionally fix, lfs_avail (to an exact figure) and
lfs_bfree (within a margin of error) in pass 5.

newfs_lfs:

* Reduce the default dlfs_minfreeseg to 1/20 of the total segments.

* Add a warning if the sgs disklabel field is 16 (the default for FFS'
cpg, but not usually desirable for LFS' sgs: 5--8 is a better range).

* Change the calculation of lfs_avail and lfs_bfree, corresponding to
the kernel changes mentioned above.

mount_lfs:

* Add -N and -b options to pass corresponding -n and -b options to
lfs_cleanerd.

* Default to calling lfs_cleanerd with "-b -n 4".


[All of these changes were largely tested in the 1.5 branch, with the
idea that they (along with previous un-pulled-up work) could be applied
to the branch while it was still in ALPHA2; however my test system has
experienced corruption on another filesystem (/dev/console has gone
missing :^), and, while I believe this unrelated to the LFS changes, I
cannot with good conscience request that the changes be pulled up.]

Change dlfs_dmeta and dlfs_avail to signed quantities, to prevent
underflow errors, visible in userland as impossibly high values
returned from df(1).

Clean up accounting of lfs_uinodes (dirty but unwritten inodes).

Make lfs_uinodes a signed quantity for debugging purposes, and set it to
zero as fs mount time.

Enclose setting/clearing of the dirty flags (IN_MODIFIED, IN_ACCESSED,
IN_CLEANING) in macros, and use those macros everywhere. Make
LFS_ITIMES use these macros; updated the ITIMES macro in inode.h to know
about this. Make ufs_getattr use ITIMES instead of FFS_ITIMES.

Fix errors observed while trying to fill the filesystem with yesterday's
fixes:

- Write copies of bfree and avail in the CLEANERINFO block, so the
cleaner doesn't have to guess which superblock has the current
information (if indeed any do).

- Tighten up accounting of lfs_avail (more needs to be done).

- When cleansing indirect blocks of UNWRITTEN, make sure not to mark
them clean, since they'll need to be rewritten later.

Allow the number of free segments reserved for the cleaner to be
parametrized in the filesystem, defaulting to MIN_FREE_SEGS = 2 but set
to something more reasonable at newfs_lfs time.

Note the number of blocks that have been scheduled for writing but which
are not yet on disk in an inode extension, i_lfs_effnblks. Move
i_ffs_effnlink out of the ffs extension and onto the main inode, since
it's used all over the shared code and the lfs extension would clobber
it.

At inode write time, indirect blocks and inode-held blocks of inodes
that have i_lfs_effnblks != i_ffs_blocks are cleansed of UNWRITTEN disk
addresses, so that these never make it to disk.

Fixes associated with filling an LFS:

Change the space computation to appear to change the size of the *disk*
rather than the *bytes used* when more segment summaries and inode
blocks are written. Try to estimate the amount of space that these will
take up when more files are written, so the disk size doesn't change too
much.

Regularize error returns from lfs_valloc, lfs_balloc, lfs_truncate: they
now fail entirely, rather than succeeding half-way and leaving the fs in
an inconsistent state.

Rewrite lfs_truncate, mostly stealing from ffs_truncate. The old
lfs_truncate had difficulty truncating a large file to a non-zero size
(indirect blocks were not handled appropriately).

Unmark VDIROP on fvp after ufs_remove, ufs_rmdir, so these can be
reclaimed immediately: this vnode would not be written to disk again
anyway if the removal succeeded, and if it failed, no directory
operation occurred.

ufs_makeinode and ufs_mkdir now remove IN_ADIROP on error.

branches: 1.25.2;
Protect inode free list with seglock, instead of separate lock, so that
the head of the inode free list (on the superblock) always matches the
rest of the free list (in the ifile).

Protect lfs_fragextend with seglock, to prevent the segment byte count
fudging from making its way to disk.

Don't try to inactivate dirop vnodes that are still in the middle of
their dirop (may address PR#10285).

update for IN_ACCESSED changes

branches: 1.23.2;
Prevent dirops from getting around lfs_check and wedging the buffer cache.
All the dirop vnops now mark the inodes with a new flag, IN_ADIROP, which
is removed as soon as the dirop is done (as opposed to VDIROP which stays
until the file is written). To address one issue raised in PR#9357.

Change the sementics of the last parameter from a boolean ("waitfor") to
a set of flags ("flags"). Two flags are defined, UPDATE_WAIT and
UPDATE_DIROP.

Under the old semantics, VOP_UPDATE would block if waitfor were set,
under the assumption that directory operations should be done
synchronously. At least LFS and FFS+softdep do not make this
assumption; FFS+softdep got around the problem by enclosing all relevant
calls to VOP_UPDATE in a "if(!DOINGSOFTDEP(vp))", while LFS simply
ignored waitfor, one of the reasons why NFS-serving an LFS filesystem
did not work properly.

Under the new semantics, the UPDATE_DIROP flag is a hint to the
fs-specific update routine that the call comes from a dirop routine, and
should be wait for, or not, accordingly.

Closes PR#8996.

Change the way LFS does block accounting, from trying to infer from the
buffer cache flags, to marking the inode and/or indirect blocks with a
special disk address UNWRITTEN==-2 when a block is accounted for. (This
address is never written to disk, but only used in-core. This is essentially
the same method of block accounting as on the UBC branch, where the buffer
headers don't exist.) Make sure that truncation is handled properly,
especially in the case of holey files.

Fixes PR#9994.

Changes to stabilize LFS. The first two of these should also apply to the
1.4 branch.

* Use a separate per-fs lock, instead of ufs_hashlock, to protect the Inode
free list. This seems to prevent the "lockmgr: %d, not exclusive lock holder
%d, unlocking" message I was mis-attributing last night to an unlocked vnode
being passed to vrele.

* Change calling semantics of lfs_ifind, to give better error reporting:
If fed a struct buf, it can report the block number of the offending inode
block as well as the inode number.

* Back out rev 1.10 of lfs_subr.c, since the replacement code was slightly
uglier while being functionally identical.

* Make lfs_vunref use the same free list convention as vrele/vput, so that
vget does not remove vnodes from a hash list they are not on.

In lfs_bwrite, don't mark buffers dirty if lfs is mounted read-only.
(Previously buffers could be marked dirty by the cleaner, and possibly by
other means.)

Also check for softdep mount in vfs_shutdown before trying to bawrite
buffers, since other filesystems don't need it and lfs doesn't bawrite.
(This fragment reviewed by fvdl.)

Partially addresses PR#8964.

Use an explicitly sized type (u_int32_t) for inode numbers in the super
block instead of ino_t. Reviewed by Konrad Schroder.

branches: 1.17.2;
Address ufs_hashlock/ufs_ihashins protocol bug, discovered while doing a
post-mortem of a production machine. Also, take the active dirop
count off of the fs and make it global (since it is measuring a global
resource) and tie the threshold value LFS_MAXDIROP to desiredvnodes.

branches: 1.16.2; 1.16.4; 1.16.6;
Minor changes to the segment live bytes calculation. In particular, fixed
a bug in fragment extension that could run the count negative. Also, don't
overcount for inodes, and don't count segment summaries. Thus, for empty
segments the live bytes count should now be exactly zero.

Fixed lfs_update (and related functions) so that calls from lfs_fsync
will DTRT with vnodes marked VDIROP. In particular, the message
"flushing VDIROP" will no longer appear, and the filesystem will remain
stable in the event of a crash.

This was particularly a problem with NFS-exported LFSes, since fsync
was called on every file close.

branches: 1.14.2; 1.14.4; 1.14.6;
clean up unused/required #ifdefs

Move dlfs_pad to the end of struct dlfs (after the pad), for upward
compatibility.

Fix pad on lfs.h so it is really 512 bytes, as advertized

New sources should leave the LFS in a more-or-less working state. Changes
include:

- DIROP segregation is enabled, and greater care is taken
to make sure that a checkpoint completes. Fsck is not
needed to remount the filesystem.
- Several checks to make sure that the LFS subsystem does not
overuse various resources (memory, in particular).
- The cleaner routines, lfs_markv in particular, are completely
rewritten. A buffer overflow is removed. Greater care is taken
to ensure that inodes come from where lfs_cleanerd say they come
from (so we know nothing has changed since lfs_bmapv was called).
- Fragment allocation is fixed, so that writes beyond end-of-file
do the right thing.

PR#6032: define fixed sized on-disk superblock structure.

Merge with Lite2 + local changes

Make the struct lfs 512 bytes long on 32bit machines whose compiler doesn't
align 32bit integers. Use explicit sized typing at some other places.

XXX This still won't fix lfs for 64bit machines, as we have some
assumptions about sizeof(pointer)=sizeof(u_int32_t) in here, and (if I
looked right) a misaligned u_int64_t. The right fix (to cite cgd) will
be to seperate on-disk-representation from in-core, but I don't have
the time (at the moment) to do this.

lfs prototypes

Add RCS ids where missing.

Sync with CSRG.

Round struct lfs to 512 bytes.

update for new syscall args description mechanism, and deal safely
with wider types.

New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD'

branches: 1.1.1;
Update to 4.4-Lite fs code, with local changes.

* Use the LFS_IFDIRTY flag to keep track of whether the Ifile has
ungathered changes during checkpoint. Modify LFS_WRITESEGENTRY and
make a new LFS_WRITEIENTRY macro to help track this. Prevents
an infinite loop in lfs_writeinode, previously caught with a panic.

* Fix a false-positive "negative bytes" panic when writing inodes into
the same segment they were in before, and consolidate the accounting
logic into new function lfs_subtract_inode.

* Ensure that we do not clean any inodes that were unavailable during
the first pass of cleaning, so that we never need to mix cleaning and
non-cleaning blocks (except for the Ifile) when writing.

* Straighten out the logic in lfs_writefile to make it more readable.

* Include IMNT_WANTRDONLY in tests for closing out writes, not just
IMNT_UNMOUNT.

* Keep track of whether the kernel cleaner is active on a file system
with a status flag; use a condvar to synchronously turn this on
or off.

remove an extra whitespace before else keyword.

Add in-kernel cleaner. Move cleaning-related routines into their own file.
Document the fcntl calls better in lfs.h. Bounds-check inode numbers in
segment parsing functions. Use the in-kernel cleaner in a greedy mode
for tests in tests/fs/vfs, and add a test of the kernel cleaner.

Remove su_flags array, replacing it with a new flag SEGUSE_READY.
Segments progress from having su_nbytes==0 to SEGUSE_EMPTY to SEGUSE_READY
to clean, progressing to the nest step after a checkpoint.

* Generalize the partial-segment parser introduced for roll-forward,
using it to facilitate an in-kernel segment rewriter (cleaner), and a
mechanism to check whether a segment is in fact empty (only used
with DEBUG).

* Add these new fcntl calls:
- LFCNFILESTATS: For each inode given, report its number of direct
blocks, how many gaps (discontinuities) there are between direct
blocks, and how large the total gap distance is. This will be
useful for a coalescing agent.
- LFCNREWRITEFILE: For each inode given, rewrite its direct blocks,
effectively coalescing it into as compact a form as possible.
- LFCNSCRAMBLE: As above, except that it only rewrites every other
block. This causes the file to have many gaps that can be
measured with LFCNFILESTATS and addressed with LFCNREWRITEFILE,
for testing purposes.
- LFCNREWRITESEGS: Rewrite any live data in the given segments.
This is intended to simplify the cleaner API and facilitate an
in-kernel cleaner.
- LFCNCLEANERINFO: Get the most current CLEANERINFO data from the
kernel.
- LFCNSEGUSE: Retrieve segment usage data from the kernel.

* Vnodes marked IN_CLEANING now take a reference. Add a new "cleaner
lock", which must be taken by the cleaner before the segment lock,
and before marking nodes IN_CLEANING. This allows us to flush
vnodes, if necessary, before the cleaning segment is written, and
never to flush vnodes being cleaned. When the cleaner lock is
released, the vnodes are cleared of IN_CLEANING and the reference
dropped.

* Track a potential infinite loop in lfs_gatherblock.

* Pull "needs to flush" and "needs to wait for flush" into functions
instead of inlining their definitions.

If setting the head (or tail) of the inode free list to LFS_UNUSED_INUM, also
set the tail (resp. head) to LFS_UNUSED_INUM, as the list is now empty.

Add a check to ensure that lfs_valloc_fixed will always terminate, even
if the free list should contain a loop. Extend the ifile at the end if it
is empty, to match the assumption of lfs_valloc() that the free list is
never empty.

Needed for roll-forward.

lfs: fix lint warning about empty declaration

Suppress -Waddress-of-packed-member just for lfs_accessors.h.

We can remove -Wno-error=address-of-packed-member from various
makefiles now.

Avoid misaligned access to lfs64 on-disk records in memory.

lfs64 directory entries are only 32-bit aligned in order to conserve
space in directory blocks, and we had a hack to stuff a 64-bit inode
in them. This replaces the hack by __aligned(4) __packed, and goes
further:

1. It's not clear that all the other lfs64 data structures are 64-bit
aligned on disk to begin with. We can go through these later and
upgrade them from

struct foo64 {
...
} __aligned(4) __packed;

union foo {
struct foo64 f64;
...
};

to

struct foo64 {
...
};

union foo {
struct foo64 f64 __aligned(8);
...
} __aligned(4) __packed;

if we really want to take advantage of 64-bit memory accesses.

However, the __aligned(4) __packed must remain on the union
because:

2. We access even the lfs32 data structures via a union that has
lfs64 members, and it turns out that compilers will assume access
through a union with 64-bit aligned members implies the whole
union has 64-bit alignment, even if we're only accessing a 32-bit
aligned member.

branches: 1.48.4; 1.48.8; 1.48.12;
Rename i_flag to i_state.

The similarity to i_flags has previously caused errors.

branches: 1.47.8;
fix sign confusion

branches: 1.46.2;
Massedit u_int{8,16,32,64}_t to uint{8,16,32,64}_t. This effectively
merges ufs/dinode.h 1.25.

we are actually synced with ufs/dinode.h 1.24 and ufs/dir.h 1.25.

Explicitly cast between char and unsigned char here.

Various housekeeping.

- Include <ufs/lfs/lfs.h> for union lfs_dinode &c.
- Include <string.h> or <sys/systm.h> for memcpy.
- Avoid signedness mismatch in lfs dino accessor for `rdev'.
- Avoid shadowing global `index'.

there is no reason to use __unused here.

Fix two functions that were accidentally "static __unused" instead of
"static __unused inline". Oops; but probably not actually harmful.

fix stupid typo in the 64-bit branch of the d_namlen accessor

improve some panic messages

Remove stray #define of lfs_magic
(the last of the fake superblock structure field macros)

Add byteswapping to the inode block-pointer accessors.

Drop an explicit sign-extension in fsck that shouldn't be needed any
more.

Use IINFO in lfs_writeinode().
(both the kernel and the userland copies)

Add an IINFO struct, which is like the FINFO struct but for the inode
blocks portion of the segment summary.

A segment summary block begins with a header (SEGSUM); the rest of the
block contains FINFO structures describing file blocks growing upward
from the bottom (after the header), and IINFO structures describing
inode blocks grown downward from the end of the block. (When they meet
the segment is full regardless of how many blocks might be left.)

IINFO contains just a block number, and until now this information was
handled by just using uint32_t*; switching to a structure will make
the code a lot easier to read, and also make it easier to have 32-bit
and 64-bit versions without making a mess.

This commit just adds the structures and accessors; they'll be
deployed into the code in subsequent commits.

branches: 1.33.2;
Fix some assorted 32-bit assumptions not yet otherwise handled.

Also apply patch to fix the overt problem in PR 50246: newfs was
calculating ifpb wrong for volumes with non-default block sizes.

Oops, I forgot to make the atime in the 64-bit IFILE 64 bits.
Correct that. Incompatible change, but no LFS64 volumes can have been
created yet.

Add 64-bit directory entry structures, and adjust accessors accordingly.

The LFS64 directory entry has a 64-bit inode number. This is stored as
two 32-bit values to avoid inducing 64-bit alignment requirements.

The exposed type for manipulating directory entries is now
LFS_DIRHEADER, following the same convention as e.g. IFILE and SEGUSE.
(But with LFS_ on it, because.)

Oops; LFS_DIRECTSIZ() is going to need the fs as an argument.

Also, it turns out that dirhash needs a compile-time-constant version
of LFS_DIRECTSIZ(LFS_MAXNAMLEN+1), independent of 64-vs-32, so create
LFS_MAXDIRENTRYSIZE for this. Sigh.

Clean up struct lfs_dirtemplate.

Fix glaringly stupid overflow/sizing bug in -r1.25. The part I don't
get is how it passed testing...

Pass around struct lfs_dirheader instead of struct lfs_direct.

Add an accessor function for directory names.

Add a function lfs_copydirname() to copy directory names in place; use
it in place of (variously) memcpy and strlcpy. (The latter isn't even
correct; was probably changed blindly from strncpy at some point.)

The new function zeroes the padding in the directory entry instead of
leaving trash behind.

Move the header part of struct lfs_direct to its own structure.
(lfs_dirheader)

Take the opportunity to improve the directory generation code in
make_lfs.c. (Everything else was unaffected by virtue of using
accessor functions.)

Add and use accessor functions for more of the directory entry fields.

Add new accessors for the d_type and d_namlen fields of struct lfs_direct.
Napalm the old byteswap access logic for these.

Fix up indirect block handling in truncate to be 32/64 clean.

Tidy the MAXSYMLINKLEN macros.

The ifile's inode number is constant. (it is always 1)

Therefore, storing the value in the superblock and reading it out
again is silly and offers the opportunity for it to become corrupted.
So, don't do that (most of the code already didn't) and use the
existing constant instead. Initialize new 32-bit superblocks with
the value for the sake of old userland programs, but don't keep the
value in the 64-bit superblock at all.

(approved by Margo Seltzer)

Make the inode fields in the 64-bit superblock 64 bits wide.
Reasoning as before.

Note that I am not going through and checking for 64->32 truncations
in inode numbers; I'm sure there are quite a few, but that's a project
for later.

Add byteswapping to the dinode accessors.

This prevents regressions in the ulfs code when switching to the new
accessors. Note that while adding byteswapping to the other accessors
is straightforward, I haven't done it yet; and that also is not enough
to make LFS_EI work, because there are places lying around that bypass
the accessors for one reason and another and all of them need to be
updated. That is going to have to wait for a later day as LFS_EI is
not on the critical path right now.

Use the lfs dinode accessors in place of the ufs-derived ones.
(Mostly.)

The ufs-derived ones are fake structure member macros, which are gross
and not very safe. Also, it seems that a lot of places in the lfs code
were using the ffsv1 branch of them unconditionally, and this way it's
guaranteed all those places have been updated.

Found while doing this: for non-devices, have getattr produce NODEV
in the rdev field instead of leaking the address of the first direct
block.

Fix IFILE pointer calculation when scanning freelist.

Part two of dinodes; use the same union everywhere.
(previously the ufs-derived code had things set up slightly different)

Remove a bunch of associated mess.

Hack up dinode usage to be 64 vs. 32 as needed. Part 1.

(This part changes the native lfs code; the ufs-derived code already
has 64 vs. 32 logic, but as aspects of it are unsafe, and don't
entirely interoperate cleanly with the lfs 64/32 stuff, pass 2 will be
rehashing that.)

Provide 32-bit and 64-bit versions of FINFO.

This also entailed sorting out part of struct segment, as that
contains a pointer into the current FINFO data.

Make 32-bit and 64-bit versions of SEGSUM.
Also fix some of the FINFO handling as it's closely entangled.

Add IFILE32 and IFILE64 structures for the on-disk ifile entries.
Add and use accessors. There are also a bunch of places that cast and
I hope I've found them all...

Make 32-bit and 64-bit versions of CLEANERINFO.

XXX: while this is written to disk, it seems like much of it would
XXX: be better set up as a commpage shared with the cleaner.

Pass the fs object to LFS_MAX_DADDR so it can check lfs_is64.

Remove some hackish intentional 64->32 truncations next to the checks
using LFS_MAX_DADDR, and tackle the problem they handled in bmap
instead.

The problem: the magic block pointer value UNWRITTEN has magic value
-2, and if it's not handled specifically, uint32 -> uint64 promotion
turns it into 4294967294, which then causes consternation and
monkeyhouse downstream.

What's here is still kind of a hack, but it's a step forward.

Add a (draft) 64-bit superblock. Make things build again.

Add pieces of support for using both superblock types where
convenient, and specifically to the superblock accessors, but don't
actually enable it anywhere.

First substantive step on PR 50000.

Use accessor functions for the version field of the lfs superblock.
I thought at first maybe the cases that test the version should be
rolled into the accessors, but on the whole I think the conclusion on
that is no.

Second batch of 64 -> 32 truncations in lfs, along with more minor
tidyups and corrections in passing.

Fix assorted 64 -> 32 truncations in lfs. Also, some minor tidyups and
corrections in passing.

Allow superblock accessors that widen 32-bit disk fields to 64-bit
memory values.

Use lfs_accessors.h in conjunction with the cleaner's struct clfs.
Remove previous hacks.

Add a new lfs header file: lfs_accessors.h.

This contains all the accessor functions and macros out of lfs.h.
Add an include of lfs_accessors.h after all uses of lfs.h... except
for code that wants to define its own struct lfs-alike that the
accessors are supposed to play along with. For these, set STRUCT_LFS
and include lfs_accessors.h after the necessary structure has been
defined, so that lfs_accessors.h can emit functions in terms of it.

Simplify orphaned file recovery by using vget/vput instead of explicitly
truncating the files, following a helpful comment from riastradh@.

Update inode disk addresses and do their segment accounting together,
rather than trying to coordinate two separate events.

Undo previous commit; it needs further refinement.

Track individual inodes between the time that their accounting is
subtracted from one segment before it is added to the new segment.
Counting duplicates is not sufficient. Addresses a "negative bytes"
panic.

Use a red/black tree, rather than a linked list, to store deferred
sergment byte count changes in truncation.

* Use the LFS_IFDIRTY flag to keep track of whether the Ifile has
ungathered changes during checkpoint. Modify LFS_WRITESEGENTRY and
make a new LFS_WRITEIENTRY macro to help track this. Prevents
an infinite loop in lfs_writeinode, previously caught with a panic.

* Fix a false-positive "negative bytes" panic when writing inodes into
the same segment they were in before, and consolidate the accounting
logic into new function lfs_subtract_inode.

* Ensure that we do not clean any inodes that were unavailable during
the first pass of cleaning, so that we never need to mix cleaning and
non-cleaning blocks (except for the Ifile) when writing.

* Straighten out the logic in lfs_writefile to make it more readable.

* Include IMNT_WANTRDONLY in tests for closing out writes, not just
IMNT_UNMOUNT.

* Keep track of whether the kernel cleaner is active on a file system
with a status flag; use a condvar to synchronously turn this on
or off.

Be more careful about only setting IN_CLEANING in lfs_setclean() and clearing
it in lfs_clrclean(). Prevents a crash from re-removing an entry from the
lfs_cleanhd TAILQ.

Create a new LFS inode flag, IN_DEAD, to indicate that a file's last
reference, other than those that come with VU_DIROP or IN_CLEANING and
the one the caller holds, has been dropped. Check and apply this flag
in lfs_orphan(), and call lfs_orphan() on close if the link count is
zero. Change the signature of lfs_orphan to facilitate.

Make test t_vfsops:lfs_tfhremove expect success.

Closes PR kern/43745.

lfs_freelist_prev is unused

Add routines to check freelist consistency if compiled with DEBUG and
conditional on a kernel variable manipulated via sysctl.
Add checks before and after each routine that modifies the free list.
#if 0 a section of lfs_vfree() that was intended to keep the free list ordered
but instead corrupted it.

Initialize nextfree, to placate gcc.

If setting the head (or tail) of the inode free list to LFS_UNUSED_INUM, also
set the tail (resp. head) to LFS_UNUSED_INUM, as the list is now empty.

Add a check to ensure that lfs_valloc_fixed will always terminate, even
if the free list should contain a loop. Extend the ifile at the end if it
is empty, to match the assumption of lfs_valloc() that the free list is
never empty.

Needed for roll-forward.

Dust off the orphan detection code and try to make it work.

Teach LFS_ORPHAN_NEXTFREE about lfs64.

Avoid undefined behavior in *_BITMAP_FREE() macros

left shift of 1 by 31 places cannot be represented in type 'int'

VFS_VGET(), VFS_ROOT(), VFS_FHTOVP(): give them a "int lktype" argument, to
allow us to get shared locks (or no lock) on the returned vnode. Matches
FreeBSD.

branches: 1.137.4; 1.137.8; 1.137.10;
Consistently use {,UN}MARK_VNODE macros rather than function calls.

Rename i_flag to i_state.

The similarity to i_flags has previously caused errors.

branches: 1.135.6;
Fill in some XXXs with the exact action described in them. match
lfs_valloc behaviour.

#if DIAGNOSTIC panic ---> KASSERT

Replace some #if DEBUG by this too. DEBUG is only for expensive
assertions; these are not.

branches: 1.133.2;
Remove unused <sys/tree.h>

Comments

branches: 1.131.2;
Fix minor bitrot in #if 0 or otherwise disabled code.

Fix wrong code in lfs_valloc_fixed(). It was overwriting the inode
number it was supposed to be allocating with the head of the inode
freelist, then applying the wrong test to that result. Net result:
unless the freelist was empty (in which case it would always fail),
it would in general drop a bunch of entries from the freelist.

This code seems to have been broken when the first version of lfsv2
was imported onto the perseant-lfsv2 branch in -r1.47.2.1, and
remained broken since, in spite of having been moved to lfs_rfw.c and
back and rearranged quite a bit in the meantime.

Sigh.

Found by Coverity in a rather confusing way as CID 1316545.

Use the lfs dinode accessors in place of the ufs-derived ones.
(Mostly.)

The ufs-derived ones are fake structure member macros, which are gross
and not very safe. Also, it seems that a lot of places in the lfs code
were using the ffsv1 branch of them unconditionally, and this way it's
guaranteed all those places have been updated.

Found while doing this: for non-devices, have getattr produce NODEV
in the rdev field instead of leaking the address of the first direct
block.

Fix IFILE pointer calculation when scanning freelist.

Hack up dinode usage to be 64 vs. 32 as needed. Part 1.

(This part changes the native lfs code; the ufs-derived code already
has 64 vs. 32 logic, but as aspects of it are unsafe, and don't
entirely interoperate cleanly with the lfs 64/32 stuff, pass 2 will be
rehashing that.)

Add IFILE32 and IFILE64 structures for the on-disk ifile entries.
Add and use accessors. There are also a bunch of places that cast and
I hope I've found them all...

Use accessor functions for the version field of the lfs superblock.
I thought at first maybe the cases that test the version should be
rolled into the accessors, but on the whole I think the conclusion on
that is no.

Add a new lfs header file: lfs_accessors.h.

This contains all the accessor functions and macros out of lfs.h.
Add an include of lfs_accessors.h after all uses of lfs.h... except
for code that wants to define its own struct lfs-alike that the
accessors are supposed to play along with. For these, set STRUCT_LFS
and include lfs_accessors.h after the necessary structure has been
defined, so that lfs_accessors.h can emit functions in terms of it.

More lfs superblock accessors.
(This changes the rest of the code over; all the accessors were
already added.)

The difference between this commit and the previous one is arbitrary,
but the previous one passed the regression tests on its own so I'm
keeping it separate to help with any bisections that might be needed
in the future.

Switch to accessor functions for elements of the LFS on-disk
superblock. This will allow switching between 32/64 bit forms on the
fly; it will also allow handling LFS_EI reasonably tidily. (That
currently doesn't work on the superblock.)

It also gets rid of cpp abuse in the form of fake structure member
macros.

Also, instead of doing sleep/wakeup on &lfs_avail and &lfs_nextseg
inside the on-disk superblock, add extra elements to the in-memory
struct lfs for this. (XXX: these should be changed to condvars, but
not right now)

XXX: this migrates a structure needed by the lfs code in libsa (struct
salfs) into lfs.h, where it doesn't belong, but for the time being
this is necessary in order to allow the accessors (and the various
lfs macros and other goop that relies on them) to compile.

Don't cast the return value of malloc.

Change lfs from hash table to vcache.

- Change lfs_valloc() to return an inode number and version instead of
a vnode and move lfs_ialloc() and lfs_vcreate() to new lfs_init_vnode().

- Add lfs_valloc_fixed() to allocate a known inode, used by kernel
roll forward.

- Remove lfs_*ref(), these functions cannot coexist with vcache and
their commented behaviour is far away from their implementation.

- Add the cleaner lwp and blockinfo to struct ulfsmount so lfs_loadvnode()
may use hints from the cleaner.

- Remove vnode locks from ulfs_lookup() like we did with ufs_lookup().

branches: 1.119.6;
Add more of the bits for supporting quotas.

Add lfs_kernel.h for declarations that don't need to be exposed to userland.

lfs currently has the following headers:
lfs.h - on-disk structures and stuff needed for userlevel tools
lfs_inode.h - additional restricted materials for userlevel tools
that operate the fs (newfs_lfs, fsck_lfs, lfs_cleanerd)
lfs_kernel.h - stuff needed only in the kernel

and the following legacy headers that are expected to be mopped up and
folded into one of the above:
lfs_extern.h - function prototypes
ulfs_bswap.h - endian-independent support
ulfs_dinode.h - now contains very little
ulfs_dirhash.h - dirhash support
ulfs_extattr.h - extattr support
ulfs_extern.h - more function prototypes
ulfs_inode.h - assorted kernel-only declarations
ulfs_quota.h - quota support
ulfs_quota1.h - more quota support
ulfs_quota2.h - more quota support
ulfs_quotacommon.h - more quota support
ulfsmount.h - legacy copy of ufsmount material

branches: 1.117.2;
Prefix most of the cpp macros with lfs_ and LFS_ to avoid conflicts with ffs.
This was done so that boot blocks that want to compile both FFS and LFS in
the same file work.

Split lfs from ufs step 4:

Massedit all ufs symbols to be "ulfs" instead, to make sure there are
no conflicts with ufs. Confirmed with grep.

(This required changing a few comments that maybe should have been
left alone to say "ulfs", but we'll survive that.)

Split lfs from ufs step 3: rearrange config stuff.
Add new options:
LFS_EI
LFS_DIRHASH
LFS_EXTATTR
LFS_EXTATTR_AUTOSTART
LFS_QUOTA
LFS_QUOTA2

and update code referring to the corresponding FFS and UFS config
symbols to use the LFS versions. Disable the one extant reference
to APPLE_UFS in the ulfs files. Use opt_lfs.h only, not opt_ffs.h.

Split lfs from ufs, part 2:

Change all <ufs/ufs/foo.h> includes to <ufs/lfs/ulfs_foo.h>.

Stuff UFS_ in front of a few of ufs's symbols to reduce namespace
pollution. Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN -> UFS_MAXSYMLINKLEN
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Sort out ext2fs's misuse of NDADDR and NIADDR; fortunately, these have
the same values in ext2fs and ffs.

No functional change intended.

branches: 1.112.2;
Pass t_renamerace and t_rmdirrace tests.

Adapt dholland@'s fix to ufs_rename to fix PR kern/43582. Address several
other MP locking issues discovered during the course of investigating the
same problem.

Removed extraneous vn_lock() calls on the Ifile, since the Ifile writes
are controlled by the segment lock.

Fix PR kern/45982 by deemphasizing the estimate of how much metadata
will fill the empty space on disk when the disk is nearly empty
(t_renamerace crates a lot of inode blocks on a tiny empty disk).

branches: 1.111.2; 1.111.6; 1.111.8;
Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.110.6;
Clean up vnode lock operations pass 2:

VOP_UNLOCK(vp, flags) -> VOP_UNLOCK(vp): Remove the unneeded flags argument.

Welcome to 5.99.32.

Discussed on tech-kern.

branches: 1.109.2; 1.109.4;
The VATTR_NULL/VREF/VHOLD/HOLDRELE() macros lost their will to live
years ago when the kernel was modified to not alter ABI based on
DIAGNOSTIC, and now just call the respective function interfaces
(in lowercase). Plenty of mix'n match upper/lowercase has creeped
into the tree since then. Nuke the macros and convert all callsites
to lowercase.

no functional change

Move declaration of ufs_hashlock into <ufs/ufs_extern.h> from each c source.

Remove clause 3 and 4 from TNF licenses

branches: 1.106.6; 1.106.8; 1.106.10;
Replace struct lock on vnodes with a simpler lock object built on
krwlock_t. This is a step towards removing lockmgr and simplifying
vnode locking. Discussed on tech-kern.

Merge vmlocking2 to head.

Fix a use of lfs_truncate() inside an #ifdef notyet (so no resulting change);
lfs_truncate() has lost its lwp argument.

branches: 1.103.4; 1.103.6; 1.103.8; 1.103.10;
Merge from vmlocking:

- Split vnode::v_flag into three fields, depending on field locking.
- simple_lock -> kmutex in a few places.
- Fix some simple locking problems.

Merge ffs locking & brelse changes from the vmlocking branch.

branches: 1.101.6; 1.101.8; 1.101.10;
Move `struct dquot' and its supporting functions from quota.h to ufs_quota.c.

- Make quota-internal functions static.
- Clean up declarations in quota.h and ufs_extern.h. quota.h now has the
description of quota criterions, on-disk structure, user-kernel interface and
declaration of init/done functions. All ufs quota related function
prototypes go to ufs_extern.h.
- New functions ufsquota_init() and ufsquota_free() create or destroy the
quota fields of `struct inode'.
- chkdq() and chkiq() always update the quota fields of `struct inode' first.
- Only ufs_access() explicitely calls getinoquota().

No objections on tech-kern@

branches: 1.100.6; 1.100.8;
Replace some uses of lockmgr() / simplelocks.

__unused removal on arguments; approved by core.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.97.2; 1.97.4;
Changes to help the roll-forward agent, to wit:

* Mark being-deleted files in the Ifile so we can finish deleting them
at fs mount time.
* Flag the Ifile with "cleaner must clean" when writers are waiting for
the cleaner, rather than relying solely on the cleaner's estimation of
whether it should clean or not.
* Note partial segments written by a user agent (in particular,
fsck_lfs) so that repeated rolls forward don't interfere with one
another.
* Add a new fcntl, LFCNPASS, that allows the log to wrap exactly once,
for better testing of the validity of checkpoints.
* Keep track of the on-disk nlink count when cleaning, so that we don't
partially complete directory operations while cleaning.
* Ensure that every single Ifile inode write represents a consistent
view of the filesystem. In particular, the accounting for the segment
we are writing the inode into must be correct, and the accounting for
the segment that inode used to reside in must be correct. Rather than
just rewriting the inode if we wrote it wrong, rewrite the necessary
ifile blocks before writing the inode so we never write it wrong.
* Don't unmark any VDIROP vnodes if we haven't written them to disk,
avoiding yet another problem with the "wait for the cleaner" error
return from lfs_putpages().

Also, move the last callback to an aiodone call, so we no longer do any
memory management from interrupt context.

Separate the (non-working) LFS kernel roll-forward code into its own file,
lfs_rfw.c.

Protect lfs_order_freelist() with the segment lock.

branches: 1.94.4;
integrate kauth.

Fixes to address the "vinvalbuf: dirty blocks" panic that can occur when
many inodes are cleaned at once. Make sure that we write all the pages
on vnodes that are being flushed, even if we don't think there's room;
drain v_numoutput before lfs_vflush() completes.

Also, don't allow a vnode that is in the process of being cleaned to be
chosen by getnewvnode(); this avoids a segment accounting panic in the case
that a large number of inodes are fed to lfs_markv() all at once.

Introduce another per-filesystem parameter, lfs_resvseg, to separate the
notion of "how many segments are reserved for the cleaner" from that of
"how many segments are not counted in lfs_bfree". The default value
used for existing filesystems is the same as the previous implicit value
of (lfs_minfreeseg / 2 + 1), modulo some sanity checking.

Count pending dirops on a per-filesystem basis, since once we start
writing them we can't stop until we're done. This seems to help stave off
the "no clean segments" panic in the case of filling the filesystem with
directories and small files (e.g. simultaneously unpacking more copies of
pkgsrc than will fit).

Add an explicit list initialization that was missing from my last commit.

Postpone the segment accounting changes coming from truncation until the
inode that makes those changes valid is either written to disk by
lfs_writeinode() or discarded by lfs_vfree().

A couple of locking fixes are also included as well.

Fix a fencepost error in the bitmap handling in extend_ifile(), and another
in lfs_freelist_prev().

Optimize the free list search a little more; in particular use words
instead of bytes for the index, and never search below fs->lfs_freehd.

Fix a bug in the previous version of the search (an erroneous assumption
that ino_t was signed).

Free the bitmap when we unmount the filesystem.

Keep the free list ordered. This solves a problem first pointed out to me
by Michel Oey, in which an aged LFS writes up to an extra Ifile block for
every file created; and paves the way for the truncation of the Ifile when
many files are deleted.

branches: 1.86.4; 1.86.6; 1.86.8; 1.86.10; 1.86.12;
merge ktrace-lwp.

merge yamt-vop branch. remove following VOPs.

VOP_BLKATOFF
VOP_VALLOC
VOP_BALLOC
VOP_REALLOCBLKS
VOP_VFREE
VOP_TRUNCATE
VOP_UPDATE

branches: 1.84.2;
64 bit inode changes.

branches: 1.83.2;
- sprinkle const
- avoid shadow variables.

Keep per-inode, per-fs, and subsystem-wide counts of blocks allocated through
lfs_balloc(), and use that to estimate the number of dirty pages belonging
to LFS (subsystem or filesystem). This is almost certainly wrong for
the case of a large mmap()ed region, but the accounting is tighter than
what we had before, and performs much better in the typical case of pages
dirtied through write().

Use splay trees, rather than a hash table, to manage the accounting of
blocks allocated through VOP_BALLOC() for pages to be written to disk.
This accounting no longer takes a noticeable fraction of the system CPU.

Consolidate the hash table we use to maintain the integrity of lfs_avail
into a single, system-wide table, rather than having a separate hash table
per inode. Significantly reduces the "system" cpu usage of your average
file write.

Keep track of the highest block held by an LFS inode, so that we can
be assured that the last byte of a file is always allocated. Previously
a file extension could cause the filesystem to be flushed, writing an
inconsistent inode to disk. Although this condition would be corrected
the next time blocks were written to disk, an intervening crash would leave
the filesystem in an inconsistent state, leaving fsck_lfs to complain
of an inode "partially truncated".

Protect various per-fs structures with fs->lfs_interlock simple_lock, to
improve behavior in the multiprocessor case. Add debugging segment-lock
assertion statements.

Make LFS dirops get their vnode first, before incrementing the dirop count,
to prevent a deadlock trying to call VOP_PUTPAGES() on a VDIROP vnode.
This can happen when a stacked filesystem is mounted on top of an LFS: an
LFS dirop needs to get a vnode, which is available from the upper layer.
The corresponding lower layer vnode, however, is VDIROP, so the upper layer
can't be cleaned out since its VOP_PUTPAGES() is passed through to the lower
layer, which waits for dirops to drain before it can proceed. Deadlock.

Tweak ufs_makeinode() and ufs_mkdir() to pass the a_vpp argument through
to VOP_VALLOC().

Partially addresses PR # 26043, though it probably does not completely fix
the problem described there.

branches: 1.76.2;
Straighten out the maze of ifdefs. Instead, consolidate all the debugging
stuff under '#ifdef DEBUG', and use sysctl knobs to turn on/off particular
parts of the debugging reporting (if DEBUG is enabled). Re-enable the LFS
statistics in sysctl, while I'm there. A bit of a rototill.

nuke trailing whitespace

Various minor LFS improvements:

* Note when lfs_putpages(9) thinks it is not going to be writing any
pages before calling genfs_putpages(9). This prevents a situation in
which blocks can be queued for writing without a segment header.
* Correct computation of NRESERVE(), though it is still a gross
overestimate in most cases. Note that if NRESERVE() is too high, it
may be impossible to create files on the filesystem. We catch this
case on filesystem mount and refuse to mount r/w.
* Allow filesystems to be mounted whose block size is == MAXBSIZE.
* Somewhere along the line, ufs_bmaparray(9) started mangling UNWRITTEN
entries in indirect blocks again, triggering a failed assertion "daddr
<= LFS_MAX_DADDR". Explicitly convert to and from int32_t to correct
this.
* Add a high-water mark for the number of dirty pages any given LFS can
hold before triggering a flush. This is settable by sysctl, but off
(zero) by default.
* Be more careful about the MAX_BYTES and MAX_BUFS computations so we
shouldn't see "please increase to at least zero" messages.
* Note that VBLK and VCHR vnodes can have nonzero values in di_db[0]
even though their v_size == 0. Don't panic when we see this.
* Change lfs_bfree to a signed quantity. The manner in which it is
processed before being passed to the cleaner means that sometimes it
may drop below zero, and the cleaner must be aware of this.
* Never report bfree < 0 (or higher than lfs_dsize) through
lfs_statvfs(9). This prevents df(1) from ever telling us that our full
filesystems have 16TB free.
* Account space allocated through lfs_balloc(9) that does not have
associated buffer headers, so that the pagedaemon doesn't run us out
of segments.
* Return ENOSPC from lfs_balloc(9) when bfree drops to zero.
* Address a deadlock in lfs_bmapv/lfs_markv when the filesystem is being
unmounted. Because vfs_busy() is a shared lock, and
lfs_bmapv/lfs_markv mark the filesystem vfs_busy(), the cleaner can be
holding the lock that umount() is blocking on, then try to vfs_busy()
again in getnewvnode().

branches: 1.73.4; 1.73.6;
Add a new flag, IN_MODIFY. This is like IN_UPDATE|IN_CHANGE, but unlike
setting those flags, it does not cause the inode to be written in the periodic
sync. This is used for writes to special files (devices and named pipes) and
FIFOs.

Do not preemptively sync updates to access times and modification times. They
are now updated in the inode only opportunistically, or when the file or device
is closed. (Really, it should be delayed beyond close, but this is enough to
help substantially with device nodes.)

And the most amusing part:
Trickle sync was broken on both FFS and ext2fs, in different ways. In FFS, the
periodic call to VFS_SYNC(MNT_LAZY) was still causing all file data to be
synced. In ext2fs, it was causing the metadata to *not* be synced. We now
only call VOP_UPDATE() on the node if we're doing MNT_LAZY. I've confirmed
that we do in fact trickle correctly now.

branches: 1.72.4;
cleanup IN_ADIROP/VDIROP handling a little.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

- protect global resource counts with lfs_subsys_lock.
- clean up scattered externs a little.

branches: 1.69.2;
Back out the lwp/ktrace changes. They contained a lot of colateral damage,
and need to be examined and discussed more.

Undo part of the ktrace/lwp changes. In particular:
* Remove the "lwp *" argument that was added to vget(). Turns out
that nothing actually used it!
* Remove the "lwp *" arguments that were added to VFS_ROOT(), VFS_VGET(),
and VFS_FHTOVP(); all they did was pass it to vget() (which, as noted
above, didn't use it).
* Remove all of the "lwp *" arguments to internal functions that were added
just to appease the above.

Pass lwp pointers throughtout the kernel, as required, so that the lwpid can
be inserted into ktrace records. The general change has been to replace
"struct proc *" with "struct lwp *" in various function prototypes, pass
the lwp through and use l_proc to get the process pointer when needed.

Bump the kernel rev up to 1.6V

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Add simple_lock protection for lfs_seglock and lfs_subsys_pages; these will
be expanded to cover other per-fs and subsystem-wide data as well.

Fix a case of IN_MODIFIED being set without updating lfs_uinodes, resulting
in a "lfs_uinodes < 0" panic.

Fix a deadlock in lfs_putpages arising from the need to busy all pages in a
block; unbusy any that had already been busied before starting over.

Tabify, and fix some comment alignment problems.

Add code to UBCify LFS. This is still behind "#ifdef LFS_UBC" for now
(there are still some details to work out) but expect that to go
away soon. To support these basic changes (creation of lfs_putpages,
lfs_gop_write, mods to lfs_balloc) several other changes were made, to
wit:

* Create a writer daemon kernel thread whose purpose is to handle page
writes for the pagedaemon, but which also takes over some of the
functions of lfs_check(). This thread is started the first time an
LFS is mounted.

* Add a "flags" parameter to GOP_SIZE. Current values are
GOP_SIZE_READ, meaning that the call should return the size of the
in-core version of the file, and GOP_SIZE_WRITE, meaning that it
should return the on-disk size. One of GOP_SIZE_READ or
GOP_SIZE_WRITE must be specified.

* Instead of using malloc(...M_WAITOK) for everything, reserve enough
resources to get by and use malloc(...M_NOWAIT), using the reserves if
necessary. Use the pool subsystem for structures small enough that
this is feasible. This also obsoletes LFS_THROTTLE.

And a few that are not strictly necessary:

* Moves the LFS inode extensions off onto a separately allocated
structure; getting closer to LFS as an LKM. "Welcome to 1.6O."

* Unified GOP_ALLOC between FFS and LFS.

* Update LFS copyright headers to correct values.

* Actually cast to unsigned in lfs_shellsort, like the comment says.

* Keep track of which segments were empty before the previous
checkpoint; any segments that pass two checkpoints both dirty and
empty can be summarily cleaned. Do this. Right now lfs_segclean
still works, but this should be turned into an effectless
compatibility syscall.

make these compilable with lfs debug options.
(follow daddr_t change)

XXX maybe segment number should be 64bit.

Use PRId64 instead of hard coding "%lld" to fix build problems under
LP64 ports.

Fix printf() format strings problems caused by "daddr_t" change.

Bump daddr_t to 64 bits. Replace it with int32_t in all places where
it was used on-disk, so that on-disk formats remain the same.
Remove ufs_daddr_t and ufs_lbn_t for the time being.

use lfs_unmark_vnode instead of duplicated code fragments.

make sure i_lfs_fragsize is initialized.
fix panic "lfs_writefile: more than one fragment!"
PR 18974.

Phase one of my three-phase plan to make LFS play nice with UBC, and bug-fixes
I found while making sure there weren't any new ones.

* Make the write clusters keep track of the buffers whose blocks they contain.
This should make it possible to (1) write clusters using a page mapping
instead of malloc, if desired, and (2) schedule blocks for rewriting
(somewhere else) if a write error occurs. Code is present to use
pagemove() to construct the clusters but that is untested and will go away
anyway in favor of page mapping.
* DEBUG now keeps a log of Ifile writes, so that any lingering instances of
the "dirty bufs" problem can be properly debugged.
* Keep track of whether the Ifile has been dirtied by various routines that
can be called by lfs_segwrite, and loop on that until it is clean, for
a checkpoint. Checkpoints need to be squeaky clean.
* Warn the user (once) if the Ifile grows larger than is reasonable for their
buffer cache. Both lfs_mountfs and lfs_unmount check since the Ifile can
grow.
* If an inode is not found in a disk block, try rereading the block, under
the assumption that the block was copied to a cluster and then freed.
* Protect WRITEINPROG() with splbio() to fix a hang in lfs_update.

Correct free list tail pointer, when adding blocks of new inodes to v2
filesystems. Should fix PR #14408.

use the new compatibility routines to allow mmap() to work
(in the same non-coherent fashion that it worked pre-UBC)
until someone has time to do it the right way.

add spaces for KNF. confirmed to produce identical objects.

add RCSID

branches: 1.51.2;
initialize the vnode's copy of the size in lfs_ialloc().

don't depend on other headers to include sys/proc.h for us.

a whole bunch of changes to improve performance and robustness under load:

- remove special treatment of pager_map mappings in pmaps. this is
required now, since I've removed the globals that expose the address range.
pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
no longer any need to special-case it.
- eliminate struct uvm_vnode by moving its fields into struct vnode.
- rewrite the pageout path. the pager is now responsible for handling the
high-level requests instead of only getting control after a bunch of work
has already been done on its behalf. this will allow us to UBCify LFS,
which needs tighter control over its pages than other filesystems do.
writing a page to disk no longer requires making it read-only, which
allows us to write wired pages without causing all kinds of havoc.
- use a new PG_PAGEOUT flag to indicate that a page should be freed
on behalf of the pagedaemon when it's unlocked. this flag is very similar
to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
pageout fails due to eg. an indirect-block buffer being locked.
this allows us to remove the "version" field from struct vm_page,
and together with shrinking "loan_count" from 32 bits to 16,
struct vm_page is now 4 bytes smaller.
- no longer use PG_RELEASED for swap-backed pages. if the page is busy
because it's being paged out, we can't release the swap slot to be
reallocated until that write is complete, but unlike with vnodes we
don't keep a count of in-progress writes so there's no good way to
know when the write is done. instead, when we need to free a busy
swap-backed page, just sleep until we can get it busy ourselves.
- implement a fast-path for extending writes which allows us to avoid
zeroing new pages. this substantially reduces cpu usage.
- encapsulate the data used by the genfs code in a struct genfs_node,
which must be the first element of the filesystem-specific vnode data
for filesystems which use genfs_{get,put}pages().
- eliminate many of the UVM pagerops, since they aren't needed anymore
now that the pager "put" operation is a higher-level operation.
- enhance the genfs code to allow NFS to use the genfs_{get,put}pages
instead of a modified copy.
- clean up struct vnode by removing all the fields that used to be used by
the vfs_cluster.c code (which we don't use anymore with UBC).
- remove kmem_object and mb_object since they were useless.
instead of allocating pages to these objects, we now just allocate
pages with no object. such pages are mapped in the kernel until they
are freed, so we can use the mapping to find the page to free it.
this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

branches: 1.48.2;
Merge the short-lived perseant-lfsv2 branch into the trunk.

Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default. Changes for the v2 layout include:

- Segments of non-PO2 size and arbitrary block offset, so these can be
matched to convenient physical characteristics of the partition (e.g.,
stripe or track size and offset).

- Address by fragment instead of by disk sector, paving the way for
non-512-byte-sector devices. In theory fragments can be as large
as you like, though in reality they must be smaller than MAXBSIZE in size.

- Use serial number and filesystem identifier to ensure that roll-forward
doesn't get old data and think it's new. Roll-forward is enabled for
v2 filesystems, though not for v1 filesystems by default.

- The inode free list is now a tailq, paving the way for undelete (undelete
is not yet implemented, but can be without further non-backwards-compatible
changes to disk structures).

- Inode atime information is kept in the Ifile, instead of on the inode;
that is, the inode is never written *just* because atime was changed.
Because of this the inodes remain near the file data on the disk, rather
than wandering all over as the disk is read repeatedly. This speeds up
repeated reads by a small but noticeable amount.

Other changes of note include:

- The ifile written by newfs_lfs can now be of arbitrary length, it is no
longer restricted to a single indirect block.

- Fixed an old bug where ctime was changed every time a vnode was created.
I need to look more closely to make sure that the times are only updated
during write(2) and friends, not after-the-fact during a segment write,
and certainly not by the cleaner.

branches: 1.47.2; 1.47.4;
use _KERNEL_OPT

branches: 1.46.2;
Get rid of some old unnecessary code that cleared B_NEEDCOMMIT from buffers in
lfs_writeseg (possibly after they had been freed).

If MALLOCLOG is defined, make lfs_newbuf and lfs_freebuf pass along the
caller's file and line to _malloc and _free.

Initial integration of the Unified Buffer Cache project.

If LFS_DO_ROLLFORWARD is defined, roll forward from the older checkpoint
on mount, through the newer checkpoint and on through any newer
partial-segments that may have been written but not checkpointed because
of an intervening crash.

LFS_DO_ROLLFORWARD is not defined by default.

Various bug-fixes to LFS, to wit:


Kernel:

* Add runtime quantity lfs_ravail, the number of disk-blocks reserved
for writing. Writes to the filesystem first reserve a maximum amount
of blocks before their write is allowed to proceed; after the blocks
are allocated the reserved total is reduced by a corresponding amount.

If the lfs_reserve function cannot immediately reserve the requested
number of blocks, the inode is unlocked, and the thread sleeps until
the cleaner has made enough space available for the blocks to be
reserved. In this way large files can be written to the filesystem
(or, smaller files can be written to a nearly-full but thoroughly
clean filesystem) and the cleaner can still function properly.

* Remove explicit switching on dlfs_minfreeseg from the kernel code; it
is now merely a fs-creation parameter used to compute dlfs_avail and
dlfs_bfree (and used by fsck_lfs(8) to check their accuracy). Its
former role is better assumed by a properly computed dlfs_avail.

* Bounds-check inode numbers submitted through lfs_bmapv and lfs_markv.
This prevents a panic, but, if the cleaner is feeding the filesystem
the wrong data, you are still in a world of hurt.

* Cleanup: remove explicit references of DEV_BSIZE in favor of
btodb()/dbtob().

lfs_cleanerd:

* Make -n mean "send N segments' blocks through a single call to
lfs_markv". Previously it had meant "clean N segments though N calls
to lfs_markv, before looking again to see if more need to be cleaned".
The new behavior gives better packing of direct data on disk with as
little metadata as possible, largely alleviating the problem that the
cleaner can consume more disk through inefficient use of metadata than
it frees by moving dirty data away from clean "holes" to produce
entirely clean segments.

* Make -b mean "read as many segments as necessary to write N segments
of dirty data back to disk", rather than its former meaning of "read
as many segments as necessary to free N segments worth of space". The
new meaning, combined with the new -n behavior described above,
further aids in cleaning storage efficiency as entire segments can be
written at once, using as few blocks as possible for segment summaries
and inode blocks.

* Make the cleaner take note of segments which could not be cleaned due
to error, and not attempt to clean them until they are entirely free
of dirty blocks. This prevents the case in which a cleanerd running
with -n 1 and without -b (formerly the default) would spin trying
repeatedly to clean a corrupt segment, while the remaining space
filled and deadlocked the filesystem.

* Update the lfs_cleanerd manual page to describe all the options,
including the changes mentioned here (in particular, the -b and -n
flags were previously undocumented).

fsck_lfs:

* Check, and optionally fix, lfs_avail (to an exact figure) and
lfs_bfree (within a margin of error) in pass 5.

newfs_lfs:

* Reduce the default dlfs_minfreeseg to 1/20 of the total segments.

* Add a warning if the sgs disklabel field is 16 (the default for FFS'
cpg, but not usually desirable for LFS' sgs: 5--8 is a better range).

* Change the calculation of lfs_avail and lfs_bfree, corresponding to
the kernel changes mentioned above.

mount_lfs:

* Add -N and -b options to pass corresponding -n and -b options to
lfs_cleanerd.

* Default to calling lfs_cleanerd with "-b -n 4".


[All of these changes were largely tested in the 1.5 branch, with the
idea that they (along with previous un-pulled-up work) could be applied
to the branch while it was still in ALPHA2; however my test system has
experienced corruption on another filesystem (/dev/console has gone
missing :^), and, while I believe this unrelated to the LFS changes, I
cannot with good conscience request that the changes be pulled up.]

Clean up accounting of lfs_uinodes (dirty but unwritten inodes).

Make lfs_uinodes a signed quantity for debugging purposes, and set it to
zero as fs mount time.

Enclose setting/clearing of the dirty flags (IN_MODIFIED, IN_ACCESSED,
IN_CLEANING) in macros, and use those macros everywhere. Make
LFS_ITIMES use these macros; updated the ITIMES macro in inode.h to know
about this. Make ufs_getattr use ITIMES instead of FFS_ITIMES.

i_lfs_effnblks fixes. Put debugging printfs under #ifdef DEBUG_LFS.

Rearrange code around getnewvnode as was already done for ffs, to avoid
locking against oneself because getnewvnode recycles a softdep-using vnode.

remove include of <vm/vm.h> and <uvm/uvm_extern.h>

Fixes associated with filling an LFS:

Change the space computation to appear to change the size of the *disk*
rather than the *bytes used* when more segment summaries and inode
blocks are written. Try to estimate the amount of space that these will
take up when more files are written, so the disk size doesn't change too
much.

Regularize error returns from lfs_valloc, lfs_balloc, lfs_truncate: they
now fail entirely, rather than succeeding half-way and leaving the fs in
an inconsistent state.

Rewrite lfs_truncate, mostly stealing from ffs_truncate. The old
lfs_truncate had difficulty truncating a large file to a non-zero size
(indirect blocks were not handled appropriately).

Unmark VDIROP on fvp after ufs_remove, ufs_rmdir, so these can be
reclaimed immediately: this vnode would not be written to disk again
anyway if the removal succeeded, and if it failed, no directory
operation occurred.

ufs_makeinode and ufs_mkdir now remove IN_ADIROP on error.

fix my own typo, grr....

Read i_ffs_gen from the version number in the Ifile during lfs_valloc,
instead of keeping it always == 1. (The ifile version number is
increased on vfree.) May address PR #7213, but I haven't been able to
test thoroughly enough to say for sure.

Update lfs_vunref for the fact that now a vnode can be locked with no
references (locked for VOP_INACTIVE at the end of vrele) and it's okay.
Check the return value of lfs_vref where appropriate.
Fixes PR #s 10285 and 10352.

branches: 1.34.2;
Protect inode free list with seglock, instead of separate lock, so that
the head of the inode free list (on the superblock) always matches the
rest of the free list (in the ifile).

Protect lfs_fragextend with seglock, to prevent the segment byte count
fudging from making its way to disk.

Don't try to inactivate dirop vnodes that are still in the middle of
their dirop (may address PR#10285).

update for IN_ACCESSED changes

branches: 1.32.2;
Prevent dirops from getting around lfs_check and wedging the buffer cache.
All the dirop vnops now mark the inodes with a new flag, IN_ADIROP, which
is removed as soon as the dirop is done (as opposed to VDIROP which stays
until the file is written). To address one issue raised in PR#9357.

Changes to stabilize LFS. The first two of these should also apply to the
1.4 branch.

* Use a separate per-fs lock, instead of ufs_hashlock, to protect the Inode
free list. This seems to prevent the "lockmgr: %d, not exclusive lock holder
%d, unlocking" message I was mis-attributing last night to an unlocked vnode
being passed to vrele.

* Change calling semantics of lfs_ifind, to give better error reporting:
If fed a struct buf, it can report the block number of the offending inode
block as well as the inode number.

* Back out rev 1.10 of lfs_subr.c, since the replacement code was slightly
uglier while being functionally identical.

* Make lfs_vunref use the same free list convention as vrele/vput, so that
vget does not remove vnodes from a hash list they are not on.

Fix error returns on lfs vnops so that locks and reference counts are
preserved. Handle dirop accounting in lfs_vfree for this case as well.
May address PR#8823.

Add Kirk McKusick's soft updates code to the trunk. Not enabled by
default, as the copyright on the main file (ffs_softdep.c) is such
that is has been put into gnusrc. options SOFTDEP will pull this
in. This code also contains the trickle syncer.

Bump version number to 1.4O

Back out my patch of the 8th (to address unreferenced inode problem).
Apparently this needs more thought.

If ifile blocks were written before dirops were complete, and then the
system crashed, inodes could be allocated that were not referenced. (Though
not a serious problem, it evidences itself in phase 4 of fsck_lfs.) Fix
this by marking if_daddr with UNASSIGNED before the inodes are actually
written; at mount time the ifile is checked for UNASSIGNED entries and
any that are found are linked back into the free list. (The latter
functionality should move into the roll-forward agent when it materializes.)

branches: 1.26.2;
Address ufs_hashlock/ufs_ihashins protocol bug, discovered while doing a
post-mortem of a production machine. Also, take the active dirop
count off of the fs and make it global (since it is measuring a global
resource) and tie the threshold value LFS_MAXDIROP to desiredvnodes.

branches: 1.25.2; 1.25.4; 1.25.6;
Make changes that will allow an LFS filesystem to be used as the root
filesystem. In particular,

- Fix mknod deadlock, described in PR 8172.
- Enable lfs_mountroot.
- Make lfs_writevnodes treat filesystems mounted on lfs device nodes properly,
by flushing that device rather than trying to add blocks to the device inode.

This, in combination with lfs boot blocks, will allow operation of an all-lfs
system.

Modify file systems to deal with struct lock in struct vnode. All leaf
fs's other than nfs use genfs_lock() for locking.

Modify lookup routines to set PDIRUNLOCK when they unlock the parrent.

squash some compiler warnings on debug printfs by casting to int

Minor changes to the segment live bytes calculation. In particular, fixed
a bug in fragment extension that could run the count negative. Also, don't
overcount for inodes, and don't count segment summaries. Thus, for empty
segments the live bytes count should now be exactly zero.

Other half of the ufs_hashlock locking fix (oops)

Fix locking panic on ufs_hashlock

Take out the `#ifdef USE_UFSHASH'; use ufs_hashlock to lock the inode free
list instead of free_lock.

branches: 1.18.2;
completely remove Mach VM support. all that is left is the all the
header files as UVM still uses (most of) these.

New sources should leave the LFS in a more-or-less working state. Changes
include:

- DIROP segregation is enabled, and greater care is taken
to make sure that a checkpoint completes. Fsck is not
needed to remount the filesystem.
- Several checks to make sure that the LFS subsystem does not
overuse various resources (memory, in particular).
- The cleaner routines, lfs_markv in particular, are completely
rewritten. A buffer overflow is removed. Greater care is taken
to ensure that inodes come from where lfs_cleanerd say they come
from (so we know nothing has changed since lfs_bmapv was called).
- Fragment allocation is fixed, so that writes beyond end-of-file
do the right thing.

Use DINODE_SIZE rather than sizeof(struct dinode).

Use the pool allocator and the "nointr" pool page allocator for LFS inodes.

Always include fifos; "not an option any more".

Protect various config(8)-generated files from inclusion while
building LKMs. Fixes PR 5557.

Use the newly-defined opt_quota.h.

Merge with Lite2 + local changes

add UVM stuff.

Don't cast 64bit (off_t) file sizes to vm_offset_t (32bit on many
architectures), truncate them intelligently instead.
The truncation is done centralized in vnode_pager.c.
This prevents from wrap-over effects when parts of large (>2^32 byte) files
are mmapped.
Don't allow to mmap above the numerical range of vm_offset_t.
This is considered a temporary solution until the vm system handles the
object sizes/offsets more cleanly.

Add support for ext2fs, this needed a few modifications to ufs/ufs/inode.h:
- added an "union inode_ext" to struct inode, for the per-fs extentions.
For now only ext2fs uses it.
- i_din is now an union:
union {
struct dinode ffs_din; /* 128 bytes of the on-disk dinode. */
struct ext2fs_dinode e2fs_din; /* 128 bytes of the on-disk dinode. */
} i_din
Added a lot of #define i_ffs_* and i_e2fs_* to access the fields.
- Added two macros: FFS_ITIMES and EXT2FS_ITIMES. ITIMES calls the rigth
macro, depending on the time of the inode. ITIMES is used where necessary,
FFS_ITIMES and EXT2FS_ITIMES in other places.

Just increment the generation count. Using the time is bogus and defeats
fsirand(8).

branches: 1.6.6;
revert previous kprintf changes

printf -> kprintf, sprintf -> ksprintf

Appease gcc: unused variables if !QUOTA

lfs prototypes

New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD'

branches: 1.1.1;
Update to 4.4-Lite fs code, with local changes.

* Generalize the partial-segment parser introduced for roll-forward,
using it to facilitate an in-kernel segment rewriter (cleaner), and a
mechanism to check whether a segment is in fact empty (only used
with DEBUG).

* Add these new fcntl calls:
- LFCNFILESTATS: For each inode given, report its number of direct
blocks, how many gaps (discontinuities) there are between direct
blocks, and how large the total gap distance is. This will be
useful for a coalescing agent.
- LFCNREWRITEFILE: For each inode given, rewrite its direct blocks,
effectively coalescing it into as compact a form as possible.
- LFCNSCRAMBLE: As above, except that it only rewrites every other
block. This causes the file to have many gaps that can be
measured with LFCNFILESTATS and addressed with LFCNREWRITEFILE,
for testing purposes.
- LFCNREWRITESEGS: Rewrite any live data in the given segments.
This is intended to simplify the cleaner API and facilitate an
in-kernel cleaner.
- LFCNCLEANERINFO: Get the most current CLEANERINFO data from the
kernel.
- LFCNSEGUSE: Retrieve segment usage data from the kernel.

* Vnodes marked IN_CLEANING now take a reference. Add a new "cleaner
lock", which must be taken by the cleaner before the segment lock,
and before marking nodes IN_CLEANING. This allows us to flush
vnodes, if necessary, before the cleaning segment is written, and
never to flush vnodes being cleaned. When the cleaner lock is
released, the vnodes are cleared of IN_CLEANING and the reference
dropped.

* Track a potential infinite loop in lfs_gatherblock.

* Pull "needs to flush" and "needs to wait for flush" into functions
instead of inlining their definitions.

Round of uvm.h cleanup.

The poorly named uvm.h is generally supposed to be for uvm-internal
users only.

- Narrow it to files that actually need it -- mostly files that need
to query whether curlwp is the pagedaemon, which should maybe be
exposed by an external header.

- Use uvm_extern.h where feasible and uvm_*.h for things not exposed
by it. We should split up uvm_extern.h but this will serve for now
to reduce the uvm.h dependencies.

- Use uvm_stat.h and #ifdef UVMHIST uvm.h for files that use
UVMHIST(ubchist), since ubchist is declared in uvm.h but the
reference evaporates if UVMHIST is not defined, so we reduce header
file dependencies.

- Make uvm_device.h and uvm_swap.h independently includable while
here.

ok chs@

Serialize access to the splay tree with lfs_lock.

branches: 1.94.6; 1.94.10; 1.94.12;
Rename i_flag to i_state.

The similarity to i_flags has previously caused errors.

move some buffer cache internals declarations from buf.h to vfs_bio.c.
this is needed to avoid name conflicts with ZFS and also
makes it clearer that other code shouldn't be messing with these.
remove the LFS debug code that poked around in bufqueues and
remove the BQ_EMPTY bufqueue since nothing uses it anymore.
provide a function to let LFS and wapbl read the value of nbuf for now.

branches: 1.92.6;
Provide a LFS_ENTER_LOG (__nothing) in the !DEBUG case.
so I can drop lots of #ifdef DEBUG around this macro. NFCI

branches: 1.91.2;
Fix stupid thinko.

comments

use static properly

branches: 1.88.2;
Use accessors for some more indirect block manipulations.

Use the lfs dinode accessors in place of the ufs-derived ones.
(Mostly.)

The ufs-derived ones are fake structure member macros, which are gross
and not very safe. Also, it seems that a lot of places in the lfs code
were using the ffsv1 branch of them unconditionally, and this way it's
guaranteed all those places have been updated.

Found while doing this: for non-devices, have getattr produce NODEV
in the rdev field instead of leaking the address of the first direct
block.

Pass the fs object to LFS_MAX_DADDR so it can check lfs_is64.

Remove some hackish intentional 64->32 truncations next to the checks
using LFS_MAX_DADDR, and tackle the problem they handled in bmap
instead.

The problem: the magic block pointer value UNWRITTEN has magic value
-2, and if it's not handled specifically, uint32 -> uint64 promotion
turns it into 4294967294, which then causes consternation and
monkeyhouse downstream.

What's here is still kind of a hack, but it's a step forward.

Fix assorted 64 -> 32 truncations in lfs. Also, some minor tidyups and
corrections in passing.

Add a new lfs header file: lfs_accessors.h.

This contains all the accessor functions and macros out of lfs.h.
Add an include of lfs_accessors.h after all uses of lfs.h... except
for code that wants to define its own struct lfs-alike that the
accessors are supposed to play along with. For these, set STRUCT_LFS
and include lfs_accessors.h after the necessary structure has been
defined, so that lfs_accessors.h can emit functions in terms of it.