merge the bouyer-quota2 branch. This adds a new on-disk format
to store disk quota usage and limits, integrated with ffs
metadata. Usage is checked by fsck_ffs (no more quotacheck)
and is covered by the WAPBL journal. Enabled with kernel
option QUOTA2 (added where QUOTA was enabled in kernel config files),
turned on with tunefs(8) on a per-filesystem
basis. mount_mfs(8) can also turn quotas on.

See http://mail-index.netbsd.org/tech-kern/2011/02/19/msg010025.html
for details.

branches: 1.6.16; 1.6.22; 1.6.24;
Merge the simonb-wapbl branch. From the original branch commit:

Add Wasabi System's WAPBL (Write Ahead Physical Block Logging)
journaling code. Originally written by Darrin B. Jewell while
at Wasabi and updated to -current by Antti Kantee, Andy Doran,
Greg Oster and Simon Burge.

OK'd by core@, releng@.

branches: 1.5.32; 1.5.70; 1.5.74; 1.5.76; 1.5.78; 1.5.80;
merge ktrace-lwp.

Experimental support for extended attributes on UFS1 file systems, using a
backing file per attribute type indexed by inode number to hold the extended
attributes.

This is working pretty well on my test systems, except for the "autostart"
feature. I need someone with a better handle on the VFS locking protocol
to go over that.

This is a work-in-progress. There are parts of this that could be re-factored
allowing this approach to be used on other types of file systems.

Adapted from FreeBSD.

branches: 1.3.8;
Remove dirhash.h.

Bring in Ian Dowse's Dirhash from FreeBSD. Hash tables of
directories are created on the fly and used to increase
performance by circumventing ufs_lookup's linear search.

Dirhash is enabled by the UFS_DIRHASH option, but not
by default.

branches: 1.1.50; 1.1.58;
Rework the way kernel include files are installed. In the new method,
as with user-land programs, include files are installed by each directory
in the tree that has includes to install. (This allows more flexibility
as to what gets installed, makes 'partial installs' easier, and gives us
more options as to which machines' includes get installed at any given
time.) The old SYS_INCLUDES={symlinks,copies} behaviours are _both_
still supported, though at least one bug in the 'symlinks' case is
fixed by this change. Include files can't be build before installation,
so directories that have includes as targets (e.g. dev/pci) have to move
those targets into a different Makefile.

s/isses/issues/

Add ACL support for FFS. From FreeBSD.

Add ACL support for FFS. From FreeBSD.

u_int{16,32,64}_t -> uint{16,32,64}_t, for the benefit of userland.

branches: 1.24.10;
Remove lfs-only inumber field (and its supporting union) from struct
ufs1_dinode.

Get rid of this copy of the accessor macro for di_u.inumber; it breaks
the build now.

Stuff UFS_ in front of a few of ufs's symbols to reduce namespace
pollution. Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN -> UFS_MAXSYMLINKLEN
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Sort out ext2fs's misuse of NDADDR and NIADDR; fortunately, these have
the same values in ext2fs and ffs.

No functional change intended.

branches: 1.21.12; 1.21.22;
+/*
+ * NOTE: COORDINATE ON-DISK FORMAT CHANGES WITH THE FREEBSD PROJECT.
+ */

Add di_modrev to the inode, for NFSv4. From FreeBSD.

branches: 1.19.74; 1.19.90;
merge ktrace-lwp.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

branches: 1.17.2;
Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Bump daddr_t to 64 bits. Replace it with int32_t in all places where
it was used on-disk, so that on-disk formats remain the same.
Remove ufs_daddr_t and ufs_lbn_t for the time being.

writable, not writeable.

Add support for the Apple UFS variation on ffs
This is the bulk of PR #17345

The general approach is to use a run time deteriminable value
for DIRBLKSIZ. Additional allowances are included for using
MAXSYMLINKLEN with FS_42INODEFMT and a shift in the cylinder group
cluster summary count array. Support is added for managing
the Apple UFS volume label.

multiple include protection

branches: 1.12.4; 1.12.6; 1.12.10;
Add Kirk McKusick's soft updates code to the trunk. Not enabled by
default, as the copyright on the main file (ffs_softdep.c) is such
that is has been put into gnusrc. options SOFTDEP will pull this
in. This code also contains the trickle syncer.

Bump version number to 1.4O

branches: 1.11.2; 1.11.4; 1.11.8;
clean up inclusion of "opt_ffs.h" and use of "FFS_EI" a bit

Modify file systems to deal with struct lock in struct vnode. All leaf
fs's other than nfs use genfs_lock() for locking.

Modify lookup routines to set PDIRUNLOCK when they unlock the parrent.

branches: 1.9.8;
Define a symbolic constant to represent the size of a dinode.

Merge with Lite2 + local changes

compensate for timeval/timespec/stat structure changes.

Add RCS ids where missing.

Sync with CSRG.

update for new syscall args description mechanism, and deal safely
with wider types.

New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD'

Fix compatibility with old fastlinks.

branches: 1.1.1;
Update to 4.4-Lite fs code, with local changes.

Add more comments to explain what we are doing.

Zero out all the dirent padding not just one byte, to avoid kernel memory
disclosure (from https://svnweb.freebsd.org/base?view=revision&revision=347066)

branches: 1.25.10; 1.25.18;
Pull over comments on struct direct's type/reclen byte swapping from LFS.

branches: 1.24.10;
Rename ambiguous macros:
MAXDIRSIZE -> UFS_MAXDIRSIZE or LFS_MAXDIRSIZE
NINDIR -> FFS_NINDIR, EXT2_NINDIR, LFS_NINDIR, or MFS_NINDIR
INOPB -> FFS_INOPB, LFS_INOPB
INOPF -> FFS_INOPF, LFS_INOPF
blksize -> ffs_blksize, ext2_blksize, or lfs_blksize
sblksize -> ffs_blksize

These are not the only ambiguously defined filesystem macros, of
course, there's a pile more. I may not have found all the ambiguous
definitions of blksize(), too, as there are a lot of other things
called 'blksize' in the system.

Stick UFS_ in front of these symbols:
DIRBLKSIZ
DIRECTSIZ
DIRSIZ
OLDDIRFMT
NEWDIRFMT

Part of PR 47909.

typo in comment

branches: 1.21.12; 1.21.22;
typo in comment

branches: 1.20.74; 1.20.90;
merge ktrace-lwp.

Don't overload MAXNAMLEN, use a separate constant for each filesystem type.

now that we've changed the _DIRENT_ALIGN macro, provide a d_fileno for struct
direct

branches: 1.17.16;
Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

branches: 1.16.2;
PR/2996: Greg Oster: Fix comment to match reality. Names are padded to
four bytes, but the bytes are not null.

Add multiple inclusion protection for headers. Fix mismatched
variable declarations (missing const's) as needed.

Add support for the Apple UFS variation on ffs
This is the bulk of PR #17345

The general approach is to use a run time deteriminable value
for DIRBLKSIZ. Additional allowances are included for using
MAXSYMLINKLEN with FS_42INODEFMT and a shift in the cylinder group
cluster summary count array. Support is added for managing
the Apple UFS volume label.

#undef DIRBLKSIZ before #define-ing it

#undef MAXNAMLEN before defining it; this lets ufs/ufs/dir.h be used
properly on non-NetBSD hosts with makefs(8).

move code to calculate size of direct for a given namlen to separate
DIRECTSIZ() macro, and use this to implement a (now shorter) DIRSIZ().
inspired by freebsd

branches: 1.10.20; 1.10.26; 1.10.30;
Add support for reading/writing FFS in non-native byte order, conditioned
to "options FFS_EI". The superblock and inodes (without blk addr) are
byteswapped at disk read/write time, other metadatas are byteswapped
when used (as they are acceeded directly in the buffer cache).
This required the addition of a "um_flags" field to struct ufsmount.
ffs_bswap.c contains superblock and inode byteswap routines also used
by userland utilities.

Merge with Lite2 + local changes

Remove hack to work around <sys/dirent.h> DIRSIZ conflict

Need to #undef DIRSIZ from <sys/dirent.h> before we redefine it.

compensate for timeval/timespec/stat structure changes.

Add RCS ids where missing.

Sync with CSRG.

update for new syscall args description mechanism, and deal safely
with wider types.

New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD'

branches: 1.1.1;
Update to 4.4-Lite fs code, with local changes.

s/memry/memory+s/softare/software/+s/grapics/graphics+s/ouput/output

s/inital/initial/

branches: 1.7.34;
Stick UFS_ in front of these symbols:
DIRBLKSIZ
DIRECTSIZ
DIRSIZ
OLDDIRFMT
NEWDIRFMT

Part of PR 47909.

branches: 1.6.32; 1.6.42;
- Tidy up the locking a bit.
- Use atomics/kmem_alloc/pool_cache.

branches: 1.5.28; 1.5.30; 1.5.32; 1.5.34;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

branches: 1.4.30; 1.4.32;
merge ktrace-lwp.

Don't overload MAXNAMLEN, use a separate constant for each filesystem type.

branches: 1.2.4; 1.2.6;
nuke trailing whitespace

branches: 1.1.2; 1.1.4;
Bring in Ian Dowse's Dirhash from FreeBSD. Hash tables of
directories are created on the fly and used to increase
performance by circumventing ufs_lookup's linear search.

Dirhash is enabled by the UFS_DIRHASH option, but not
by default.

Extended attribute support for ffsv2, from FreeBSD.

branches: 1.11.18; 1.11.28;
Bump UFS1 extended attribute max name length to 256

For extended attribute name max length, kernel filesystem-independant
code use either EXTATTR_MAXNAMELEN (BSD API) or XATTR_NAME_MAX (Linux API),
which are both defined as KERNEL_NAME_MAX and fits Linux limit of 255
without training \0.

UFS1 code had a lower limit that broke Linux compatibility. We can bump
the limit without sacrifying backward compatibility, because:

1) There is no API exposing this limit outside the kernel. Upper kernel
layers have a larger limit handle the increase without a hitch

2) Each attribute has its own backing store in the fileystem, the name
of the backing store matching the attribute name. A newer kernel can
create/read/write backing store for longer attribute names and will
have no problem with existing shorter names.

branches: 1.10.8; 1.10.12; 1.10.28; 1.10.30;
add forward declarations for the VOP args structures
so that fstat can include this file.

Add mount -o extattr option to enable extended attributs (corrently only
for UFS1).
Remove kernel option for EA backing store autocreation and do it by
default. Add a sysctl so that autocreated attriutr size can be modified.

branches: 1.8.42;
Replace use of lockmgr.

Destroy extattr lock when destroying extattrs associated with the
mountpoint. Make stopping extattrs always succesful to facilitate
always being able to free resources.

Remove the "struct lwp *" argument from all VFS and VOP interfaces.
The general trend is to remove it from all kernel interfaces and
this is a start. In case the calling lwp is desired, curlwp should
be used.

quick consensus on tech-kern

branches: 1.5.6; 1.5.8; 1.5.14;
Using POOL_INIT here makes no sense, since file systems always have
an init method. So get rid of it and #ifdef _LKM and just always
init in the init method. Give malloc types the same treatment.
Makes file systems nicer to work with in linksetless environments
and fixes a few LKM discrepancies.

branches: 1.4.6; 1.4.20; 1.4.22;
integrate kauth.

branches: 1.3.4; 1.3.6; 1.3.8; 1.3.10; 1.3.12;
Convert UFS_EXTATTR to struct lwp.

merge ktrace-lwp.

branches: 1.1.6;
Experimental support for extended attributes on UFS1 file systems, using a
backing file per attribute type indexed by inode number to hold the extended
attributes.

This is working pretty well on my test systems, except for the "autostart"
feature. I need someone with a better handle on the VFS locking protocol
to go over that.

This is a work-in-progress. There are parts of this that could be re-factored
allowing this approach to be used on other types of file systems.

Adapted from FreeBSD.

fix few typos for word "previous(ly)" in comments.

Don't cache id's for vnodes that have ACLs. ok chs@

Extended attribute support for ffsv2, from FreeBSD.

branches: 1.76.4; 1.76.14;
update the comment to the current IFMT/permissions location

again remove IN_E4EXTENTS; it's not used anywhere any more, and it's better to keep fs-specific flags out of generic headers anyway

move i_e2fs_* defines from ufs/inode.h to ext2fs/ext2fs_dinode.h, where they belong; they don't seem to be used anywhere else then ext2fs code any more

rename struct ext2fs_dinode attribute e2di_dacl to correct
e2di_size_high; even Linux ext2 filesystem code actually uses it
unconditionally this way and ext4 code finally also calls it that way
in their struct definition too; if there was any trace of this for other
purpose it's long gone

branches: 1.72.2;
ext4 extents glue

branches: 1.71.4;
Fix previous. Anyone have a brown paper bag?

Close comments

Remove lfs-only inode flags.

Reorder struct ufid members to avoid padding (and save 4 bytes) on some
architectures.

Make filehandles on UFS based filesystems use proper 64bit inodes.
32bit restriction noticed by Taylor R Campbell.

Add a global vnode cache:

- vcache_get() retrieves a referenced and initialised vnode / fs node pair.
- vcache_remove() removes a vnode / fs node pair from the cache.

On cache miss vcache_get() calls new vfs operation vfs_loadvnode() to
initialise a vnode / fs node pair. This call is guaranteed exclusive,
no other thread will try to load this vnode / fs node pair.

Convert ufs/ext2fs, ufs/ffs and ufs/mfs to use this interface.

Remove now unused ufs/ufs_ihash

Discussed on tech-kern.

Welcome to 6.99.41

branches: 1.65.2; 1.65.6;
Remove lfs-only inumber field (and its supporting union) from struct
ufs1_dinode.

- Add e2di_version, e2di_nblock_high, e2di_facl_high fields to ext2fs_dinode.

- Update i_e2fs_ aliases to match.

- ext2fs_bswap support for these ext2fs_dinode fields.

(e2di_version and e2di_facl_high replace previously reserved fields.
e2di_nblock_high was formerly e2di_nfrag and e2di_fsize, however these
are currently defined in e2fsprogs as only being relevant for HURD.)

Move i_e2fs_rdev define to be adjacent to the field it aliases.

branches: 1.62.2;
Use two separate comments for stub where IN_RENAME was.

Kill the IN_RENAME in-core inode flag in ufs and ext2fs.

Now that rename works we need not to wave this sort of voodoo at it.

ok dholland

comments and cosmetics. no functional changes.

* Remove PGO_RECLAIM during lfs_putpages()' call to genfs_putpages(),
to avoid a live lock in the latter when reclaiming a vnode with
dirty pages.

* Add a new segment flag, SEGM_RECLAIM, to note when a segment is
being written for vnode reclamation, and record which inode is being
reclaimed, to aid in forensic debugging.

* Add a new segment flag, SEGM_SINGLE, so that opportunistic writes
can write a single segment's worth of blocks and then stop, rather
than writing all the way up to the cleaner's reserved number of
segments.

* Add assert statements to check mutex ownership is the way it ought
to be, mostly in lfs_putpages; fix problems uncovered by this.

* Don't clear VU_DIROP until the inode actually makes its way to disk,
avoiding a problem where dirop inodes could become separated
(uncovered by a modified version of the "ckckp" forensic regression
test).

* Move the vfs_getopsbyname() call into lfs_writerd. Prepare code to
make lfs_writerd notice when there are no more LFSs, and exit losing
the reference, so that, in theory, the module can be unloaded. This
code is not enabled, since it causes a crash on exit.

* Set IN_MODIFIED on inodes flushed by lfs_flush_dirops. Really we
only need to set IN_MODIFIED if we are going to write them again
(e.g., to write pages); need to think about this more.

Finally, several changes to help avoid "no clean segments" panics:

* In lfs_bmapv, note when a vnode is loaded only to discover whether
its blocks are live, so it can immediately be recycled. Since the
cleaner will try to choose ~empty segments over full ones, this
prevents the cleaner from (1) filling the vnode cache with junk, and
(2) squeezing any unwritten writes to disk and running the fs out of
segments.

* Overestimate by half the amount of metadata that will be required
to fill the clean segments. This will make the disk appear smaller,
but should help avoid a "no clean segments" panic.

* Rearrange lfs_writerd. In particular, lfs_writerd now pays
attention to the number of clean segments available, and holds off
writing until there is room.

branches: 1.58.2; 1.58.6;
Currently, ufs_lookup produces five auxiliary results that are left in
the vnode when lookup returns and fished out again later.

1. Create struct ufs_lookup_results to hold these.

2. Call the ufs_lookup_results instance in struct inode "i_crap" to be
clear about exactly what's going on, and to distinguish the lookup
results from respectable members of struct inode.

3. Update references to these members in the directory access
subroutines.

4. Include preliminary infrastructure for checking that the i_crap
being used is still valid when it's used. This doesn't actually do
anything yet.

5. Update the way ufs_wapbl_rename manipulates these elements to use
the new data structures. I have not changed the manipulation; it may
or may not be correct but I continue to suspect that it is not.

The word of the day is "stigmergy".

ext2fs,ffs: free on disk inodes in the reclaim routine.
Remove now unneeded vnode flag VI_FREEING.

Welcome to 5.99.38.

Ok: Andrew Doran <ad@netbsd.org>

branches: 1.56.2; 1.56.4;
PR kern/26878 FFSv2 + softdep = livelock (no free ram)
PR kern/16942 panic with softdep and quotas
PR kern/19565 panic: softdep_write_inodeblock: indirect pointer #1 mismatch
PR kern/26274 softdep panic: allocdirect_merge: ...
PR kern/26374 Long delay before non-root users can write to softdep partitions
PR kern/28621 1.6.x "vp != NULL" panic in ffs_softdep.c:4653 while unmounting a softdep (+quota) filesystem
PR kern/29513 FFS+Softdep panic with unfsck-able file-corruption
PR kern/31544 The ffs softdep code appears to fail to write dirty bits to disk
PR kern/31981 stopping scsi disk can cause panic (softdep)
PR kern/32116 kernel panic in softdep (assertion failure)
PR kern/32532 softdep_trackbufs deadlock
PR kern/37191 softdep: locking against myself
PR kern/40474 Kernel panic after remounting raid root with softdep

Retire softdep, pass 2. As discussed and later formally announced on the
mailing lists.

branches: 1.55.4;
add support for 32 bit uid/gid fields in ext2, but only do so for
when the revision is > REV0.

branches: 1.54.2; 1.54.4;
fix reversed comment, from anon ymous

Be consistent with #define<tab>.

Merge the simonb-wapbl branch. From the original branch commit:

Add Wasabi System's WAPBL (Write Ahead Physical Block Logging)
journaling code. Originally written by Darrin B. Jewell while
at Wasabi and updated to -current by Antti Kantee, Andy Doran,
Greg Oster and Simon Burge.

OK'd by core@, releng@.

branches: 1.51.6; 1.51.10; 1.51.12; 1.51.14; 1.51.16;
Go back to freeing on disk inodes in the inactive routine. It would be
better not to do this, but it rules out potential side effects with softdep.

Fix 'panic: softdep_update_inodeblock: update failed'.

Merge vmlocking2 to head.

branches: 1.48.10; 1.48.16; 1.48.18; 1.48.22;
fix comment: struct fid is in fstypes.h now

branches: 1.47.2; 1.47.4;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.46.26;
merge ktrace-lwp.

introduce "ufs_ops" and use it for ITIMES.

- access the ffs and ext2fs itimes functions through a pointer, so that
if the filesystem is not compiled in the kernel still links. Probably
a better solution is to use weak symbols.
- move the filesystem-specific itime macros to the filesystem header files.

Use nanotime() to update the time fields in filesystems. Convert the code
from macros to real functions. Original patch and review from chuq.
Note: ext2fs only keeps seconds in the on-disk inode, and msdosfs does not
have enough precision for all fields, so this is not very useful for those
two.

64 bit inode changes.

branches: 1.41.4;
nuke trailing whitespace

branches: 1.40.2;
Bring in Ian Dowse's Dirhash from FreeBSD. Hash tables of
directories are created on the fly and used to increase
performance by circumventing ufs_lookup's linear search.

Dirhash is enabled by the UFS_DIRHASH option, but not
by default.

branches: 1.39.4;
Push atime/mtime updates even further -- into the reclaim path, so they happen
rarely in the normal case. (Note: This happens at reboot/shutdown time because
all file systems are unmounted.)

Also, for IN_MODIFY, use IN_ACCESSED, not IN_MODIFIED; otherwise "ls -l" of
your device node or FIFO would cause the time stamps to get written too
quickly.

Add a new flag, IN_MODIFY. This is like IN_UPDATE|IN_CHANGE, but unlike
setting those flags, it does not cause the inode to be written in the periodic
sync. This is used for writes to special files (devices and named pipes) and
FIFOs.

Do not preemptively sync updates to access times and modification times. They
are now updated in the inode only opportunistically, or when the file or device
is closed. (Really, it should be delayed beyond close, but this is enough to
help substantially with device nodes.)

And the most amusing part:
Trickle sync was broken on both FFS and ext2fs, in different ways. In FFS, the
periodic call to VFS_SYNC(MNT_LAZY) was still causing all file data to be
synced. In ext2fs, it was causing the metadata to *not* be synced. We now
only call VOP_UPDATE() on the node if we're doing MNT_LAZY. I've confirmed
that we do in fact trickle correctly now.

Add ffs internal snapshots. Written by Marshall Kirk McKusick for FreeBSD.

- Not enabled by default. Needs kernel option FFS_SNAPSHOT.
- Change parameters of ffs_blkfree.
- Let the copy-on-write functions return an error so spec_strategy
may fail if the copy-on-write fails.
- Change genfs_*lock*() to use vp->v_vnlock instead of &vp->v_lock.
- Add flag B_METAONLY to VOP_BALLOC to return indirect block buffer.
- Add a function ffs_checkfreefile needed for snapshot creation.
- Add special handling of snapshot files:
Snapshots may not be opened for writing and the attributes are read-only.
Use the mtime as the time this snapshot was taken.
Deny mtime updates for snapshot files.
- Add function transferlockers to transfer any waiting processes from
one lock to another.
- Add vfsop VFS_SNAPSHOT to take a snapshot and make it accessible through
a vnode.
- Add snapshot support to ls, fsck_ffs and dump.

Welcome to 2.0F.

Approved by: Jason R. Thorpe <thorpej@netbsd.org>

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

branches: 1.35.2;
The C language does not permit statements of the form
(X ? Y : Z) = 0;
even though gcc handles this by a stupid extension.

Transform these to correct C.

Approved by fvdl.

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Add code to UBCify LFS. This is still behind "#ifdef LFS_UBC" for now
(there are still some details to work out) but expect that to go
away soon. To support these basic changes (creation of lfs_putpages,
lfs_gop_write, mods to lfs_balloc) several other changes were made, to
wit:

* Create a writer daemon kernel thread whose purpose is to handle page
writes for the pagedaemon, but which also takes over some of the
functions of lfs_check(). This thread is started the first time an
LFS is mounted.

* Add a "flags" parameter to GOP_SIZE. Current values are
GOP_SIZE_READ, meaning that the call should return the size of the
in-core version of the file, and GOP_SIZE_WRITE, meaning that it
should return the on-disk size. One of GOP_SIZE_READ or
GOP_SIZE_WRITE must be specified.

* Instead of using malloc(...M_WAITOK) for everything, reserve enough
resources to get by and use malloc(...M_NOWAIT), using the reserves if
necessary. Use the pool subsystem for structures small enough that
this is feasible. This also obsoletes LFS_THROTTLE.

And a few that are not strictly necessary:

* Moves the LFS inode extensions off onto a separately allocated
structure; getting closer to LFS as an LKM. "Welcome to 1.6O."

* Unified GOP_ALLOC between FFS and LFS.

* Update LFS copyright headers to correct values.

* Actually cast to unsigned in lfs_shellsort, like the comment says.

* Keep track of which segments were empty before the previous
checkpoint; any segments that pass two checkpoints both dirty and
empty can be summarily cleaned. Do this. Right now lfs_segclean
still works, but this should be turned into an effectless
compatibility syscall.

Bump daddr_t to 64 bits. Replace it with int32_t in all places where
it was used on-disk, so that on-disk formats remain the same.
Remove ufs_daddr_t and ufs_lbn_t for the time being.

Add multiple inclusion protection for headers. Fix mismatched
variable declarations (missing const's) as needed.

eliminate i_ino from in-core inode
and use local variable instead.

ok'ed by Frank van der Linden.

Deal with fragment size changes better. For each fragment that can
exist on an on-disk inode, we keep a record of its size in struct inode,
which is updated when we write the block to disk. The cleaner routines
thus have ready access to what size is the correct size for this block,
on disk.

Fixed a related bug: if a file with fragments is being cleaned
(fragments being cleaned) at the same time it is being extended beyond
NDADDR blocks, we could write a bogus FINFO record that has a frag in the
middle; when it was cleaned this would give back bogus file data. Don't
write the indirect blocks in this case, since there is no need.

lfs_fragextend and lfs_truncate no longer require the seglock, but instead
take a shared lock, which the seglock locks exclusively.

For synchronous writes, keep separate i/o counters for each write, so
processes don't have to wait for one another to finish (e.g., nfsd seems
to be a little happier now, though I haven't measured the difference).
Synchronous checkpoints, however, must always wait for all i/o to finish.

Take the contents of the callback functions and have them run in thread
context instead (aiodoned thread). lfs_iocount no longer has to be
protected in splbio(), and quite a bit less of the segment construction
loop needs to be in splbio() as well.

If lfs_markv is handed a block that is not the correct size according to
the inode, refuse to process it. (Formerly it was extended to the "correct"
size.) This is possibly more prone to deadlock, but less prone to corruption.

lfs_segclean now outright refuses to clean segments that appear to have live
bytes in them. Again this may be more prone to deadlock but avoids
corruption.

Replace ufsspec_close and ufsfifo_close with LFS equivalents; this means
that no UFS functions need to know about LFS_ITIMES any more. Remove
the reference from ufs/inode.h.

Tested on i386, test-compiled on alpha.

branches: 1.27.8; 1.27.10;
Bring over fixes from FreeBSD that weren't incorporated yet, mainly
from Kirk McKusick. They implement taking pending block/inode frees
into account for the sake of correct statfs() numbers, and adding
a new softdep type (newdirblk) to correctly handle newly allocated
directory blocks.

Minor additional changes: 1) swap the newly introduced fs_pendinginodes
and fs_pendingblock fields in ffs_sb_swap, and 2) declare lkt_held
in the debug version of the softdep lock structure volatile, as it
can be modified from interrupt context #ifdef DEBUG.

this needs <ufs/ufs/quota.h>, so pull it in

branches: 1.25.2;
a whole bunch of changes to improve performance and robustness under load:

- remove special treatment of pager_map mappings in pmaps. this is
required now, since I've removed the globals that expose the address range.
pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
no longer any need to special-case it.
- eliminate struct uvm_vnode by moving its fields into struct vnode.
- rewrite the pageout path. the pager is now responsible for handling the
high-level requests instead of only getting control after a bunch of work
has already been done on its behalf. this will allow us to UBCify LFS,
which needs tighter control over its pages than other filesystems do.
writing a page to disk no longer requires making it read-only, which
allows us to write wired pages without causing all kinds of havoc.
- use a new PG_PAGEOUT flag to indicate that a page should be freed
on behalf of the pagedaemon when it's unlocked. this flag is very similar
to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
pageout fails due to eg. an indirect-block buffer being locked.
this allows us to remove the "version" field from struct vm_page,
and together with shrinking "loan_count" from 32 bits to 16,
struct vm_page is now 4 bytes smaller.
- no longer use PG_RELEASED for swap-backed pages. if the page is busy
because it's being paged out, we can't release the swap slot to be
reallocated until that write is complete, but unlike with vnodes we
don't keep a count of in-progress writes so there's no good way to
know when the write is done. instead, when we need to free a busy
swap-backed page, just sleep until we can get it busy ourselves.
- implement a fast-path for extending writes which allows us to avoid
zeroing new pages. this substantially reduces cpu usage.
- encapsulate the data used by the genfs code in a struct genfs_node,
which must be the first element of the filesystem-specific vnode data
for filesystems which use genfs_{get,put}pages().
- eliminate many of the UVM pagerops, since they aren't needed anymore
now that the pager "put" operation is a higher-level operation.
- enhance the genfs code to allow NFS to use the genfs_{get,put}pages
instead of a modified copy.
- clean up struct vnode by removing all the fields that used to be used by
the vfs_cluster.c code (which we don't use anymore with UBC).
- remove kmem_object and mb_object since they were useless.
instead of allocating pages to these objects, we now just allocate
pages with no object. such pages are mapped in the kernel until they
are freed, so we can use the mapping to find the page to free it.
this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

branches: 1.24.4; 1.24.6;
only include "fs_lfs.h" if _KERNEL_OPT.

branches: 1.23.2;
attach the softdep pagecache pseudo-buffers to the inode
so we can find them quickly in the softdep truncate path.

Fix so non-kernel code will compile (_LKM)

Clean up accounting of lfs_uinodes (dirty but unwritten inodes).

Make lfs_uinodes a signed quantity for debugging purposes, and set it to
zero as fs mount time.

Enclose setting/clearing of the dirty flags (IN_MODIFIED, IN_ACCESSED,
IN_CLEANING) in macros, and use those macros everywhere. Make
LFS_ITIMES use these macros; updated the ITIMES macro in inode.h to know
about this. Make ufs_getattr use ITIMES instead of FFS_ITIMES.

Allow the number of free segments reserved for the cleaner to be
parametrized in the filesystem, defaulting to MIN_FREE_SEGS = 2 but set
to something more reasonable at newfs_lfs time.

Note the number of blocks that have been scheduled for writing but which
are not yet on disk in an inode extension, i_lfs_effnblks. Move
i_ffs_effnlink out of the ffs extension and onto the main inode, since
it's used all over the shared code and the lfs extension would clobber
it.

At inode write time, indirect blocks and inode-held blocks of inodes
that have i_lfs_effnblks != i_ffs_blocks are cleansed of UNWRITTEN disk
addresses, so that these never make it to disk.

branches: 1.19.2;
Pull in IN_ACCESSED changes and some MNT_LAZY `bug fixes' from FFS to EXT2FS.

Add a new inode flags called IN_ACCESSED. This used in place of IN_MODIFIED
to record that the atime was updated. In ffs_update(), we only do synchronous
writes if something *other* than the atime was changed.

branches: 1.17.2;
Prevent dirops from getting around lfs_check and wedging the buffer cache.
All the dirop vnops now mark the inodes with a new flag, IN_ADIROP, which
is removed as soon as the dirop is done (as opposed to VDIROP which stays
until the file is written). To address one issue raised in PR#9357.

Define i_e2fs_rdev.

Cosmetic changes; fix indentation and usage of white spaces.

Add Kirk McKusick's soft updates code to the trunk. Not enabled by
default, as the copyright on the main file (ffs_softdep.c) is such
that is has been put into gnusrc. options SOFTDEP will pull this
in. This code also contains the trickle syncer.

Bump version number to 1.4O

branches: 1.13.2; 1.13.4; 1.13.8;
Modify file systems to deal with struct lock in struct vnode. All leaf
fs's other than nfs use genfs_lock() for locking.

Modify lookup routines to set PDIRUNLOCK when they unlock the parrent.

branches: 1.12.4;
Add IN_CLEANING flag for LFS

Merge with Lite2 + local changes

Add support for ext2fs, this needed a few modifications to ufs/ufs/inode.h:
- added an "union inode_ext" to struct inode, for the per-fs extentions.
For now only ext2fs uses it.
- i_din is now an union:
union {
struct dinode ffs_din; /* 128 bytes of the on-disk dinode. */
struct ext2fs_dinode e2fs_din; /* 128 bytes of the on-disk dinode. */
} i_din
Added a lot of #define i_ffs_* and i_e2fs_* to access the fields.
- Added two macros: FFS_ITIMES and EXT2FS_ITIMES. ITIMES calls the rigth
macro, depending on the time of the inode. ITIMES is used where necessary,
FFS_ITIMES and EXT2FS_ITIMES in other places.

Add a set of generic file system operations that most file systems use.
Also, fix some time stamp bogosities.

compensate for timeval/timespec/stat structure changes.

KERNEL -> _KERNEL

Add RCS ids where missing.

Sync with CSRG.

update for new syscall args description mechanism, and deal safely
with wider types.

fix the definition of a dev_t

New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD'

branches: 1.1.1;
Update to 4.4-Lite fs code, with local changes.

Remove extraneous files from Lite2 merge.

branches: 1.1.1;
Initial revision

Move INITQFNAMES to the right header file.

Rename static inline "helper" functions:
ufsclass2qtype -> quota_idtype_to_ufs
qtype2ufsclass -> quota_idtype_from_ufs

The reason for the direction of "ufs" changing is that the old names
were among the symbols using "ufs" to mean "fs-independent". So the
old names were for translating "ufsclass" (fs-independent quota id
type) to "qtype" (ufs-specific quota id type) and vice versa.

These functions are used in only two places, both of which are
inappropriate, so at some point they should probably be removed.
They're also identity transformations so not particularly helpful,
unless one were to make a careful and concerted effort to distinguish
the ufs quota code numbers from the fs-independent ones. This has not
been done and is probably impossible without support from a program
verifier, and maybe not even then.

They are static inline, so no compat concerns arise.

Also adjust the symbols they use to avoid <quota/quotaprop.h>.

branches: 1.28.4; 1.28.8;
Don't include quota/quotaprop.h for tools.

Add a new libquota library, which contains some blocks to build and/or
parse quota plists; as well as a getfsquota() function to retrieve quotas
for a single id from a single filesystem (whatever filesystem this is:
a local quota-enabled fs or NFS). This is build on functions getufsquota()
(for local filesystems with UFS-like quotas) and getnfsquota();
which are also available to userland programs.
move functions from quota2_subr.c to libquota or libprop as appropriate,
and ajust in-tree quota tools.
move some declarations from kernel headers to either sys/quota.h or
quota/quota.h as appropriate. ufs/ufs/quota.h still installed because
it's needed by other installed ufs headers.
ufs/ufs/quota1.h still installed as a quick&dirty way to get a code
using the old quotactl() to compile (just include ufs/ufs/quota1.h instead of
ufs/ufs/quota.h - old code won't compile without this change and this is
on purpose).
Discussed on tech-kern@ and tech-net@ (long thread, but not much about
libquota itself ...)

merge the bouyer-quota2 branch. This adds a new on-disk format
to store disk quota usage and limits, integrated with ffs
metadata. Usage is checked by fsck_ffs (no more quotacheck)
and is covered by the WAPBL journal. Enabled with kernel
option QUOTA2 (added where QUOTA was enabled in kernel config files),
turned on with tunefs(8) on a per-filesystem
basis. mount_mfs(8) can also turn quotas on.

See http://mail-index.netbsd.org/tech-kern/2011/02/19/msg010025.html
for details.

branches: 1.25.56; 1.25.62; 1.25.64;
Move `struct dquot' and its supporting functions from quota.h to ufs_quota.c.

- Make quota-internal functions static.
- Clean up declarations in quota.h and ufs_extern.h. quota.h now has the
description of quota criterions, on-disk structure, user-kernel interface and
declaration of init/done functions. All ufs quota related function
prototypes go to ufs_extern.h.
- New functions ufsquota_init() and ufsquota_free() create or destroy the
quota fields of `struct inode'.
- chkdq() and chkiq() always update the quota fields of `struct inode' first.
- Only ufs_access() explicitely calls getinoquota().

No objections on tech-kern@

If a quota-enabled file system has 65536 active vnodes for one uid
the reference counter of the corresponding struct dquot will overflow.

Change the type of the reference counter from u_int16_t to u_int32_t and
add an assertion to check for overflow.

Observed and tested by Edgar Fu�.

Welcome to 4.99.21 (struct dquot and therefore struct inode changed layout)

branches: 1.23.2; 1.23.4;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.22.12; 1.22.14; 1.22.18;
integrate kauth.

branches: 1.21.4; 1.21.6; 1.21.8; 1.21.10; 1.21.12;
changes for making DIAGNOSTIC not change the kernel ABI:
- for structure fields that are conditionally present,
make those fields always present.
- for functions which are conditionally inline, make them never inline.
- remove some other functions which are conditionally defined but
don't actually do anything anymore.
- make a lock-debugging function conditional on only LOCKDEBUG.

as discussed on tech-kern some time back.

merge ktrace-lwp.

- Use ANSI function decls.
- Sprinkle some static.

branches: 1.18.10; 1.18.12; 1.18.14; 1.18.16;
First pass for some caddr_t removal and changes to get rid of it where we
no longer use and/or need it

- removed casts from unionfs, deadfs and fdesc
(there are more to hunt down still)
- changed vfs_quotactl args argumet from caddr_t to void *
- changed vfs_quotactl structures/callers to reflect the api change

Compiled fine and ran for about a day. Approved/reviewed by
christos@netbsd.org and gimpy@netbsd.org.

branches: 1.17.4;
Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

branches: 1.16.2;
Back out the lwp/ktrace changes. They contained a lot of colateral damage,
and need to be examined and discussed more.

Pass lwp pointers throughtout the kernel, as required, so that the lwpid can
be inserted into ktrace records. The general change has been to replace
"struct proc *" with "struct lwp *" in various function prototypes, pass
the lwp through and use l_proc to get the process pointer when needed.

Bump the kernel rev up to 1.6V

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Add multiple inclusion protection for headers. Fix mismatched
variable declarations (missing const's) as needed.

Remove superfluous semicolon.

rename inclusion protection define from _QUOTA_ to _UFS_UFS_QUOTA_H_

branches: 1.10.2;
add a new VFS op, vfs_reinit, which is called when desiredvnodes is
adjusted via sysctl. file systems that have hash tables which are
sized based on the value of this variable now resize those hash tables
using the new value. the max number of FFS softdeps is also recalculated.

convert various file systems to use the <sys/queue.h> macros for
their hash tables.

branches: 1.9.2; 1.9.6; 1.9.8;
Use int32_t for on-disk time_t values.

branches: 1.8.4;
Change ufs_init() to keep global count of how many times it was called.
Resources are initialized still just once (on first call).

Add ufs_done(), which takes care of freeing all resources allocated in
ufs_init(). The resources are freed only when last user of the code exits.

branches: 1.7.28;
Add prototype for quotactl.

KERNEL -> _KERNEL

#include sys/queue.h, but also hide kernel structures in #ifdef KERNEL.

Sync with CSRG.

update for new syscall args description mechanism, and deal safely
with wider types.

New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD'

branches: 1.1.1;
Update to 4.4-Lite fs code, with local changes.

Move INITQFNAMES to the right header file.

Change dqblk_to_quotaval() from quota1_subr.c to dqblk_to_quotavals(),
and pass in two single quotaval structs (for blocks and inodes)
instead of an array of (implicitly) QUOTA_NLIMITS quotaval structs
indexed by constants from quotaprop.h.

Note: because this code is used by COMPAT_50 as well as ufs, this
change requires a kernel version bump. (The code is also used by
edquota, but via .PATH so it's not ABI-sensitive there.)

branches: 1.5.2;
Rename struct ufs_quota_entry -> struct quotaval.

Reshuffle decls among the quota headers so everything is in the place
it should be:
- stuff for the proplib interface goes in <quota/quotaprop.h>
- stuff for userlevel only goes in <quota/quota.h>
- stuff shared between user and kernel goes in <sys/quota.h>

Note that <quota/quota.h> and <quota/quotaprop.h> are expected to be
moved or removed later on... one thing at a time.

Update include directives in other files as needed.

branches: 1.3.2; 1.3.6; 1.3.8;
Add a new libquota library, which contains some blocks to build and/or
parse quota plists; as well as a getfsquota() function to retrieve quotas
for a single id from a single filesystem (whatever filesystem this is:
a local quota-enabled fs or NFS). This is build on functions getufsquota()
(for local filesystems with UFS-like quotas) and getnfsquota();
which are also available to userland programs.
move functions from quota2_subr.c to libquota or libprop as appropriate,
and ajust in-tree quota tools.
move some declarations from kernel headers to either sys/quota.h or
quota/quota.h as appropriate. ufs/ufs/quota.h still installed because
it's needed by other installed ufs headers.
ufs/ufs/quota1.h still installed as a quick&dirty way to get a code
using the old quotactl() to compile (just include ufs/ufs/quota1.h instead of
ufs/ufs/quota.h - old code won't compile without this change and this is
on purpose).
Discussed on tech-kern@ and tech-net@ (long thread, but not much about
libquota itself ...)

merge the bouyer-quota2 branch. This adds a new on-disk format
to store disk quota usage and limits, integrated with ffs
metadata. Usage is checked by fsck_ffs (no more quotacheck)
and is covered by the WAPBL journal. Enabled with kernel
option QUOTA2 (added where QUOTA was enabled in kernel config files),
turned on with tunefs(8) on a per-filesystem
basis. mount_mfs(8) can also turn quotas on.

See http://mail-index.netbsd.org/tech-kern/2011/02/19/msg010025.html
for details.

branches: 1.1.2;
file quota1.h was initially added on branch bouyer-quota2.

ufs: Nix trailing whitespace and tidy up some other minor KNF.

Change dqblk_to_quotaval() from quota1_subr.c to dqblk_to_quotavals(),
and pass in two single quotaval structs (for blocks and inodes)
instead of an array of (implicitly) QUOTA_NLIMITS quotaval structs
indexed by constants from quotaprop.h.

Note: because this code is used by COMPAT_50 as well as ufs, this
change requires a kernel version bump. (The code is also used by
edquota, but via .PATH so it's not ABI-sensitive there.)

branches: 1.6.2;
Rename struct ufs_quota_entry -> struct quotaval.

Reshuffle decls among the quota headers so everything is in the place
it should be:
- stuff for the proplib interface goes in <quota/quotaprop.h>
- stuff for userlevel only goes in <quota/quota.h>
- stuff shared between user and kernel goes in <sys/quota.h>

Note that <quota/quota.h> and <quota/quotaprop.h> are expected to be
moved or removed later on... one thing at a time.

Update include directives in other files as needed.

branches: 1.4.2;
Fix bad cut'n'paste in copyright. Pointed out by dyoung@

branches: 1.3.2; 1.3.4; 1.3.6;
Add a new libquota library, which contains some blocks to build and/or
parse quota plists; as well as a getfsquota() function to retrieve quotas
for a single id from a single filesystem (whatever filesystem this is:
a local quota-enabled fs or NFS). This is build on functions getufsquota()
(for local filesystems with UFS-like quotas) and getnfsquota();
which are also available to userland programs.
move functions from quota2_subr.c to libquota or libprop as appropriate,
and ajust in-tree quota tools.
move some declarations from kernel headers to either sys/quota.h or
quota/quota.h as appropriate. ufs/ufs/quota.h still installed because
it's needed by other installed ufs headers.
ufs/ufs/quota1.h still installed as a quick&dirty way to get a code
using the old quotactl() to compile (just include ufs/ufs/quota1.h instead of
ufs/ufs/quota.h - old code won't compile without this change and this is
on purpose).
Discussed on tech-kern@ and tech-net@ (long thread, but not much about
libquota itself ...)

merge the bouyer-quota2 branch. This adds a new on-disk format
to store disk quota usage and limits, integrated with ffs
metadata. Usage is checked by fsck_ffs (no more quotacheck)
and is covered by the WAPBL journal. Enabled with kernel
option QUOTA2 (added where QUOTA was enabled in kernel config files),
turned on with tunefs(8) on a per-filesystem
basis. mount_mfs(8) can also turn quotas on.

See http://mail-index.netbsd.org/tech-kern/2011/02/19/msg010025.html
for details.

branches: 1.1.2;
file quota1_subr.c was initially added on branch bouyer-quota2.

ufs: Nix trailing whitespace and tidy up some other minor KNF.

fix tyop, PR kern/52653 by Edgar Fuss

branches: 1.9.6;
Migrate one last leftover bit (used only by the kernel now) to
sys/ufs/ufs and remove the old quota headers and no-longer-used shared
code. Ok by releng.

quota2_check_limit() is used in only one place, so don't stuff it in a
header file.

Remove now-unused declarations from quota2.h.

Remove references to <quota/quotaprop.h> in src/sys/ufs.
The remaining references in the kernel are in vfs_quotactl.c, the
compat_50 code for the old quotactl (to be fixed up), and the
code compiled from src/common/lib/libquota.

branches: 1.5.2; 1.5.6;
Fix bad cut'n'paste in copyright. Pointed out by dyoung@

branches: 1.4.2; 1.4.4; 1.4.6;
Add a new libquota library, which contains some blocks to build and/or
parse quota plists; as well as a getfsquota() function to retrieve quotas
for a single id from a single filesystem (whatever filesystem this is:
a local quota-enabled fs or NFS). This is build on functions getufsquota()
(for local filesystems with UFS-like quotas) and getnfsquota();
which are also available to userland programs.
move functions from quota2_subr.c to libquota or libprop as appropriate,
and ajust in-tree quota tools.
move some declarations from kernel headers to either sys/quota.h or
quota/quota.h as appropriate. ufs/ufs/quota.h still installed because
it's needed by other installed ufs headers.
ufs/ufs/quota1.h still installed as a quick&dirty way to get a code
using the old quotactl() to compile (just include ufs/ufs/quota1.h instead of
ufs/ufs/quota.h - old code won't compile without this change and this is
on purpose).
Discussed on tech-kern@ and tech-net@ (long thread, but not much about
libquota itself ...)

typo in comment

merge the bouyer-quota2 branch. This adds a new on-disk format
to store disk quota usage and limits, integrated with ffs
metadata. Usage is checked by fsck_ffs (no more quotacheck)
and is covered by the WAPBL journal. Enabled with kernel
option QUOTA2 (added where QUOTA was enabled in kernel config files),
turned on with tunefs(8) on a per-filesystem
basis. mount_mfs(8) can also turn quotas on.

See http://mail-index.netbsd.org/tech-kern/2011/02/19/msg010025.html
for details.

branches: 1.1.2;
file quota2.h was initially added on branch bouyer-quota2.

Add a new libquota library, which contains some blocks to build and/or
parse quota plists; as well as a getfsquota() function to retrieve quotas
for a single id from a single filesystem (whatever filesystem this is:
a local quota-enabled fs or NFS). This is build on functions getufsquota()
(for local filesystems with UFS-like quotas) and getnfsquota();
which are also available to userland programs.
move functions from quota2_subr.c to libquota or libprop as appropriate,
and ajust in-tree quota tools.
move some declarations from kernel headers to either sys/quota.h or
quota/quota.h as appropriate. ufs/ufs/quota.h still installed because
it's needed by other installed ufs headers.
ufs/ufs/quota1.h still installed as a quick&dirty way to get a code
using the old quotactl() to compile (just include ufs/ufs/quota1.h instead of
ufs/ufs/quota.h - old code won't compile without this change and this is
on purpose).
Discussed on tech-kern@ and tech-net@ (long thread, but not much about
libquota itself ...)

merge the bouyer-quota2 branch. This adds a new on-disk format
to store disk quota usage and limits, integrated with ffs
metadata. Usage is checked by fsck_ffs (no more quotacheck)
and is covered by the WAPBL journal. Enabled with kernel
option QUOTA2 (added where QUOTA was enabled in kernel config files),
turned on with tunefs(8) on a per-filesystem
basis. mount_mfs(8) can also turn quotas on.

See http://mail-index.netbsd.org/tech-kern/2011/02/19/msg010025.html
for details.

branches: 1.1.2;
file quota2_prop.c was initially added on branch bouyer-quota2.

Add a new libquota library, which contains some blocks to build and/or
parse quota plists; as well as a getfsquota() function to retrieve quotas
for a single id from a single filesystem (whatever filesystem this is:
a local quota-enabled fs or NFS). This is build on functions getufsquota()
(for local filesystems with UFS-like quotas) and getnfsquota();
which are also available to userland programs.
move functions from quota2_subr.c to libquota or libprop as appropriate,
and ajust in-tree quota tools.
move some declarations from kernel headers to either sys/quota.h or
quota/quota.h as appropriate. ufs/ufs/quota.h still installed because
it's needed by other installed ufs headers.
ufs/ufs/quota1.h still installed as a quick&dirty way to get a code
using the old quotactl() to compile (just include ufs/ufs/quota1.h instead of
ufs/ufs/quota.h - old code won't compile without this change and this is
on purpose).
Discussed on tech-kern@ and tech-net@ (long thread, but not much about
libquota itself ...)

merge the bouyer-quota2 branch. This adds a new on-disk format
to store disk quota usage and limits, integrated with ffs
metadata. Usage is checked by fsck_ffs (no more quotacheck)
and is covered by the WAPBL journal. Enabled with kernel
option QUOTA2 (added where QUOTA was enabled in kernel config files),
turned on with tunefs(8) on a per-filesystem
basis. mount_mfs(8) can also turn quotas on.

See http://mail-index.netbsd.org/tech-kern/2011/02/19/msg010025.html
for details.

branches: 1.1.2;
file quota2_prop.h was initially added on branch bouyer-quota2.

s/defaut/default/ in comments.

ufs: Nix trailing whitespace and tidy up some other minor KNF.

Migrate one last leftover bit (used only by the kernel now) to
sys/ufs/ufs and remove the old quota headers and no-longer-used shared
code. Ok by releng.

branches: 1.4.2; 1.4.6;
Fix bad cut'n'paste in copyright. Pointed out by dyoung@

branches: 1.3.2; 1.3.4; 1.3.6;
Add a new libquota library, which contains some blocks to build and/or
parse quota plists; as well as a getfsquota() function to retrieve quotas
for a single id from a single filesystem (whatever filesystem this is:
a local quota-enabled fs or NFS). This is build on functions getufsquota()
(for local filesystems with UFS-like quotas) and getnfsquota();
which are also available to userland programs.
move functions from quota2_subr.c to libquota or libprop as appropriate,
and ajust in-tree quota tools.
move some declarations from kernel headers to either sys/quota.h or
quota/quota.h as appropriate. ufs/ufs/quota.h still installed because
it's needed by other installed ufs headers.
ufs/ufs/quota1.h still installed as a quick&dirty way to get a code
using the old quotactl() to compile (just include ufs/ufs/quota1.h instead of
ufs/ufs/quota.h - old code won't compile without this change and this is
on purpose).
Discussed on tech-kern@ and tech-net@ (long thread, but not much about
libquota itself ...)

merge the bouyer-quota2 branch. This adds a new on-disk format
to store disk quota usage and limits, integrated with ffs
metadata. Usage is checked by fsck_ffs (no more quotacheck)
and is covered by the WAPBL journal. Enabled with kernel
option QUOTA2 (added where QUOTA was enabled in kernel config files),
turned on with tunefs(8) on a per-filesystem
basis. mount_mfs(8) can also turn quotas on.

See http://mail-index.netbsd.org/tech-kern/2011/02/19/msg010025.html
for details.

branches: 1.1.2;
file quota2_subr.c was initially added on branch bouyer-quota2.

ufs: Nix trailing whitespace and tidy up some other minor KNF.

use MNT_NFS4ACLS instead of MNT_ACLS (which was changed before to mean
MNT_POSIX1EACLS)

Overhaul of the EVFILT_VNODE kevent(2) filter:

- Centralize vnode kevent handling in the VOP_*() wrappers, rather than
forcing each individual file system to deal with it (except VOP_RENAME(),
because VOP_RENAME() is a mess and we currently have 2 different ways
of handling it; at least it's reasonably well-centralized in the "new"
way).
- Add support for NOTE_OPEN, NOTE_CLOSE, NOTE_CLOSE_WRITE, and NOTE_READ,
compatible with the same events in FreeBSD.
- Track which kevent notifications clients are interested in receiving
to avoid doing work for events no one cares about (avoiding, e.g.
taking locks and traversing the klist to send a NOTE_WRITE when
someone is merely watching for a file to be deleted, for example).

In support of the above:

- Add support in vnode_if.sh for specifying PRE- and POST-op handlers,
to be invoked before and after vop_pre() and vop_post(), respectively.
Basic idea from FreeBSD, but implemented differently.
- Add support in vnode_if.sh for specifying CONTEXT fields in the
vop_*_args structures. These context fields are used to convey information
between the file system VOP function and the VOP wrapper, but do not
occupy an argument slot in the VOP_*() call itself. These context fields
are initialized and subsequently interpreted by PRE- and POST-op handlers.
- Version VOP_REMOVE(), uses the a context field for the file system to report
back the resulting link count of the target vnode. Return this in tmpfs,
udf, nfs, chfs, ext2fs, lfs, and ufs.

NetBSD 9.99.92.

Use VN_KNOTE() to send our NOTE_ATTRIB and NOTE_REVOKE events.

Add ACL support for FFS. From FreeBSD.

Restore backward compatibility of UFS2 with previous NetBSD releases by
disabling support in UFS2 for extended attributes (including ACLs).
Add a new variant of UFS2 called "UFS2ea" that does support extended attributes.
Add new fsck_ffs operations "-c ea" and "-c no-ea" to convert file systems
from UFS2 to UFS2ea and vice-versa (both of which delete all existing extended
attributes in the process).

handle negative small block numbers for extattr

branches: 1.52.14; 1.52.24;
#if DIAGNOSTIC panic ---> KASSERT

Remove now redundant calls to fstrans_start()/fstrans_done().

branches: 1.50.14; 1.50.18; 1.50.22;
Stuff UFS_ in front of a few of ufs's symbols to reduce namespace
pollution. Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN -> UFS_MAXSYMLINKLEN
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Sort out ext2fs's misuse of NDADDR and NIADDR; fortunately, these have
the same values in ext2fs and ffs.

No functional change intended.

branches: 1.49.4; 1.49.14;
merge the bouyer-quota2 branch. This adds a new on-disk format
to store disk quota usage and limits, integrated with ffs
metadata. Usage is checked by fsck_ffs (no more quotacheck)
and is covered by the WAPBL journal. Enabled with kernel
option QUOTA2 (added where QUOTA was enabled in kernel config files),
turned on with tunefs(8) on a per-filesystem
basis. mount_mfs(8) can also turn quotas on.

See http://mail-index.netbsd.org/tech-kern/2011/02/19/msg010025.html
for details.

branches: 1.48.26; 1.48.32; 1.48.34;
Make rusage collection per-LWP and collate in the appropriate places.
cloned threads need a little bit more work but the locking needs to
be fixed first.

branches: 1.47.6;
Merge vmlocking2 to head.

branches: 1.46.4; 1.46.6; 1.46.10;
Merge ffs locking & brelse changes from the vmlocking branch.

branches: 1.45.6; 1.45.8; 1.45.10;
Fstrans_start() always returns zero, so change its type to void.

branches: 1.44.4; 1.44.6;
TRUE -> true, FALSE -> false

Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

branches: 1.42.2;
Change fstrans enum types to upper case.
No functional change.

From Antti Kantee <pooka@netbsd.org>

New file system suspension API to replace vn_start_write and vn_finished_write.
The suspension helpers are now put into file system specific operations.
This means every file system not supporting these helpers cannot be suspended
and therefore snapshots are no longer possible.

Implemented for file systems of type ffs.

The new API is enabled on a kernel option NEWVNGATE. This option is
not enabled by default in any kernel config.

Presented and discussed on tech-kern with much input from
Bill Studenmund <wrstuden@netbsd.org> and YAMAMOTO Takashi <yamt@netbsd.org>.

Welcome to 4.99.9 (new vfs op vfs_suspendctl).

branches: 1.40.8;
Correct typo in a panic message.

branches: 1.39.4; 1.39.6; 1.39.8; 1.39.10; 1.39.12;
merge ktrace-lwp.

avoid the need for a bogus initializer.

Drop extra word from comment.

- Use ANSI function decls.
- Sprinkle some static.

branches: 1.35.4;
nuke trailing whitespace

branches: 1.34.2; 1.34.4;
Remove some unnecessary (int32_t) casts that would cause us to screw up the
top bit in block addresses.

Also, change some daddr_t->int32_t casts (mostly as arguments to ufs_rw32(),
where they would get promoted anyway) to u_int32_t.

ufs_getlbns:
- fix an integer overflow when calculating lbns of indirect blocks.
- remove a redundant calculation of blockcnt.

Repair some FFS_EI code for ufsmount changes.

Fixing age old cruft:
* Rather than using mnt_maxsymlinklen to indicate that a file systems returns
d_type fields(!), add a new internal flag, IMNT_DTYPE.

Add 3 new elements to ufsmount:
* um_maxsymlinklen, replaces mnt_maxsymlinklen (which never should have existed
in the first place).
* um_dirblksiz, which tracks the current directory block size, eliminating the
FS-specific checks littered throughout the code. This may be used later to
make the block size variable.
* um_maxfilesize, which is the maximum file size, possibly adjusted lower due
to implementation issues.

Sync some bug fixes from FFS into ext2fs, particularly:
* ffs_lookup.c 1.21, 1.28, 1.33, 1.48
* ffs_inode.c 1.43, 1.44, 1.45, 1.66, 1.67
* ffs_vnops.c 1.84, 1.85, 1.86

Clean up some crappy pointer frobnication.

remove incorrect casts that limit some uses of daddr_t to 31 bits
this fixes problems using ffs2 with more than 2^31 sectors (~1tb)

Add ffs internal snapshots. Written by Marshall Kirk McKusick for FreeBSD.

- Not enabled by default. Needs kernel option FFS_SNAPSHOT.
- Change parameters of ffs_blkfree.
- Let the copy-on-write functions return an error so spec_strategy
may fail if the copy-on-write fails.
- Change genfs_*lock*() to use vp->v_vnlock instead of &vp->v_lock.
- Add flag B_METAONLY to VOP_BALLOC to return indirect block buffer.
- Add a function ffs_checkfreefile needed for snapshot creation.
- Add special handling of snapshot files:
Snapshots may not be opened for writing and the attributes are read-only.
Use the mtime as the time this snapshot was taken.
Deny mtime updates for snapshot files.
- Add function transferlockers to transfer any waiting processes from
one lock to another.
- Add vfsop VFS_SNAPSHOT to take a snapshot and make it accessible through
a vnode.
- Add snapshot support to ls, fsck_ffs and dump.

Welcome to 2.0F.

Approved by: Jason R. Thorpe <thorpej@netbsd.org>

branches: 1.28.2;
Shut up gcc3 warning that `metalbn' might be used uninitialized.
XXX: The warning is bogus and only triggered on sh3.

Make VOP_STRATEGY(bp) a real VOP as discussed on tech-kern.

VOP_STRATEGY(bp) is replaced by one of two new functions:

- VOP_STRATEGY(vp, bp) Call the strategy routine of vp for bp.
- DEV_STRATEGY(bp) Call the d_strategy routine of bp->b_dev for bp.

DEV_STRATEGY(bp) is used only for block-to-block device situations.

store a i/o priority hint in struct buf for buffer queue discipline.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

cast UFS1 on-disk block pointers to int32_t before assign it to daddr_t.
it's needed for LFS because UNWRITTEN is a negative number.

branches: 1.23.2;
make is_sequential a callback in order to achieve better lfs write clustering.

since lfs always rewrite blocks into the new segment,
current on-disk place of the block doesn't affect to write clustering.

ok'ed by Konrad Schroder.

Correct use of MAXBSIZE where MAXPHYS was intended. This is a necessary
first step towards per-device MAXPHYS, and has the beneficial side effect
of allowing clustering to MAXPHYS even on systems that need to run with
a reduced MAXBSIZE to get more metadata buffers.

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

LFS likes to store negative values in the dinode block pointers, so
make sure to cast the value back to int32_t after it was changed
by ufs_rw32, before passing it to blkptrtodb.

Allow negative values other than UNASSIGNED to be returned from ufs_bmap;
fixes a bug introduced in the 64-bit daddr_t conversion, that manifests
itself in LFS with kernels compiled with the FFS_EI option.

Bump daddr_t to 64 bits. Replace it with int32_t in all places where
it was used on-disk, so that on-disk formats remain the same.
Remove ufs_daddr_t and ufs_lbn_t for the time being.

branches: 1.17.4;
Add comment that getblk() in ufs_bmaparray() returns an error only if
we are pagedaemon.

some callers of ufs_bmaparray() in LFS depend on *nump being set to 0 for
direct blocks, so restore that behaviour.

fix the previous change: use the correct test for a block number
referring to a direct block.

only call ufs_getlbns() for blocks which involve indirects, and assert
that this is so. use a shift instead of a multiply in one place.

add RCSID

Remove some variables that are set but never used.

remove #include <ufs/ufs/quota.h> where it was just to appease
<ufs/ufs/inode.h>, since the latter now includes the former. leave the former
in source that obviously uses specific bits of it (for completeness.)

branches: 1.10.2; 1.10.6; 1.10.10;
Initial integration of the Unified Buffer Cache project.

Remove register declarations.

branches: 1.8.14;
KNF, mostly of FFS_EI changes.

Add support for reading/writing FFS in non-native byte order, conditioned
to "options FFS_EI". The superblock and inodes (without blk addr) are
byteswapped at disk read/write time, other metadatas are byteswapped
when used (as they are acceeded directly in the buffer cache).
This required the addition of a "um_flags" field to struct ufsmount.
ffs_bswap.c contains superblock and inode byteswap routines also used
by userland utilities.

Merge with Lite2 + local changes

When allocating a new block, store the result obtained through counting
indirect blocks in a 64 bit integer, to prevent overflows when computing
NINDIR^3

Add support for ext2fs, this needed a few modifications to ufs/ufs/inode.h:
- added an "union inode_ext" to struct inode, for the per-fs extentions.
For now only ext2fs uses it.
- i_din is now an union:
union {
struct dinode ffs_din; /* 128 bytes of the on-disk dinode. */
struct ext2fs_dinode e2fs_din; /* 128 bytes of the on-disk dinode. */
} i_din
Added a lot of #define i_ffs_* and i_e2fs_* to access the fields.
- Added two macros: FFS_ITIMES and EXT2FS_ITIMES. ITIMES calls the rigth
macro, depending on the time of the inode. ITIMES is used where necessary,
FFS_ITIMES and EXT2FS_ITIMES in other places.

ufs prototype changes

New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD'

branches: 1.1.1;
Update to 4.4-Lite fs code, with local changes.

s/static inline/static __inline/g for consistency.

branches: 1.22.12;
Add smaller versions of fsck_ffs(8) and newfs(8) for install media, where
support for Endian-Independent FFS and Apple UFS is disabled unless FFS_EI=1
and APPLE_UFS=1 are added to CRUNCHENV, respectively.

This reduces the size of ramdisk image for atari by over 15KB.

Thanks tsutsui and christos for their useful comments.

branches: 1.21.2; 1.21.4;
use variables that could be unused.

branches: 1.20.6;
convert ufs_rw{16,32,64}() into real inline functions in all cases,
so that they consume their second arguments properly.

branches: 1.19.12; 1.19.22; 1.19.26;
Remove closes 3 & 4 from my licence. Lots of thanks to Soren Jacobsen
for the booring work !

branches: 1.18.72;
Make almost everything #include <sys/bswap.h> instead of <machine/bswap.h>
The bswap.h and endian.h files are all rather incestuous, but I want to
get the constant folding stuff into one place - sys/bswap.h

branches: 1.17.2;
Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

- Use ANSI function decls.
- Sprinkle some static.

branches: 1.14.12;
Fixing age old cruft:
* Rather than using mnt_maxsymlinklen to indicate that a file systems returns
d_type fields(!), add a new internal flag, IMNT_DTYPE.

Add 3 new elements to ufsmount:
* um_maxsymlinklen, replaces mnt_maxsymlinklen (which never should have existed
in the first place).
* um_dirblksiz, which tracks the current directory block size, eliminating the
FS-specific checks littered throughout the code. This may be used later to
make the block size variable.
* um_maxfilesize, which is the maximum file size, possibly adjusted lower due
to implementation issues.

Sync some bug fixes from FFS into ext2fs, particularly:
* ffs_lookup.c 1.21, 1.28, 1.33, 1.48
* ffs_inode.c 1.43, 1.44, 1.45, 1.66, 1.67
* ffs_vnops.c 1.84, 1.85, 1.86

Clean up some crappy pointer frobnication.

Remove references to University of California from my copyright notices.

branches: 1.12.2;
ufs_rwXX: cast to unsigned even without FFS_EI to avoid
possible nasty signed vs unsigned problems.

Add multiple inclusion protection for headers. Fix mismatched
variable declarations (missing const's) as needed.

Revert previous. This is actually being done a better way.

For makefs, only include <machine/bswap.h> if it exists.

fix spelo

branches: 1.7.4;
use _KERNEL_OPT

branches: 1.6.6;
Sync copyrigth notice.

Add Kirk McKusick's soft updates code to the trunk. Not enabled by
default, as the copyright on the main file (ffs_softdep.c) is such
that is has been put into gnusrc. options SOFTDEP will pull this
in. This code also contains the trickle syncer.

Bump version number to 1.4O

branches: 1.4.2; 1.4.4; 1.4.8;
clean up inclusion of "opt_ffs.h" and use of "FFS_EI" a bit

Move the bswap functions from libutil to libc (this bups the
minor of libc and the major of libutil). For little-endian architectures
merge the bnswap() assembly versions with nto* and hton* using symbols
aliasing. Use symbol renaming for the bswap function in this case to avoid
namespace pollution.
Declare bswap* in machine/bswap.h, not machine/endian.h. For little-endian
machines, common code for inline macros go in machine/byte_swap.h
Sync libkern with libc.
Adjust #include in kernel sources for machine/bswap.h.

defopt FFS_EI

Add support for reading/writing FFS in non-native byte order, conditioned
to "options FFS_EI". The superblock and inodes (without blk addr) are
byteswapped at disk read/write time, other metadatas are byteswapped
when used (as they are acceeded directly in the buffer cache).
This required the addition of a "um_flags" field to struct ufsmount.
ffs_bswap.c contains superblock and inode byteswap routines also used
by userland utilities.

If UFS or LFS dirhash is enabled in the kernel, set the dirhash cache
size dependant on memory size. If less than 128MB of memory, default
to no cache. With 128MB of memory or more, use a maximum cache size of
1/64th of memory; cap maximum default cache size to 32MB (for systems
with 2GB of memory or more).

The dirhash cache sizes are still explicityly setable by sysctl(8) or
by adding relevant entry(s) to sysctl.conf(5).

Use the module subsystem's ability to process SYSCTL_SETUP() entries to
automate installation of sysctl nodes.

Note that there are still a number of device and pseudo-device modules
that create entries tied to individual device units, rather than to the
module itself. These are not changed.

- Hide the details of SPCF_SHOULDYIELD and related behind a couple of small
functions: preempt_point() and preempt_needed().

- preempt(): if the LWP has exceeded its timeslice in kernel, strip it of
any priority boost gained earlier from blocking.

in ufsdirhash_free(), only examine dh->dh_onlist after taking the
dirhashlist lock. if we skip the lock then we might see that
dh_onlist is zero while ufsdirhash_recycle() is still working on
the dirhash. the symptom I saw was that ufsdirhash_free() would
try to destroy the dh_lock mutex while it was still held.

branches: 1.37.18; 1.37.22;
clear i_dirhash sooner, but what lock protects it?

branches: 1.36.6;
Ensure that the top level sysctl nodes (kern, vfs, net, ...) exist before
the sysctl link sets are processed, and remove redundancy.

Shaves >13kB off of an amd64 GENERIC, not to mention >1k duplicate
lines of code.

branches: 1.35.2;
Stick UFS_ in front of these symbols:
DIRBLKSIZ
DIRECTSIZ
DIRSIZ
OLDDIRFMT
NEWDIRFMT

Part of PR 47909.

branches: 1.34.12; 1.34.22;
ufsdirhash_recycle(): modify ufs_dirhashmem atomically.

ufsdirhash_lookup(): call ufs_blkatoff() with "modify == false".
This buffer is used read-only here and from caller.

Revert previous until problem will be understood.

ufsdirhash_recycle():
- Fix ufs_dirhashmem modification (do it atomically).
- Fix a memory leak.

OK by <ad>.

bcmp -> memcmp

Ansify function definitions w/o arguments. Generated with sed.

PR kern/26878 FFSv2 + softdep = livelock (no free ram)
PR kern/16942 panic with softdep and quotas
PR kern/19565 panic: softdep_write_inodeblock: indirect pointer #1 mismatch
PR kern/26274 softdep panic: allocdirect_merge: ...
PR kern/26374 Long delay before non-root users can write to softdep partitions
PR kern/28621 1.6.x "vp != NULL" panic in ffs_softdep.c:4653 while unmounting a softdep (+quota) filesystem
PR kern/29513 FFS+Softdep panic with unfsck-able file-corruption
PR kern/31544 The ffs softdep code appears to fail to write dirty bits to disk
PR kern/31981 stopping scsi disk can cause panic (softdep)
PR kern/32116 kernel panic in softdep (assertion failure)
PR kern/32532 softdep_trackbufs deadlock
PR kern/37191 softdep: locking against myself
PR kern/40474 Kernel panic after remounting raid root with softdep

Retire softdep, pass 2. As discussed and later formally announced on the
mailing lists.

branches: 1.27.4; 1.27.10;
ufsdirhash_build: missing unlock in failure path.

Create sysctl entries during module initialisation and destroy them
appropriately.

Many of these file systems are now ready for modularisation.

Add some locking, runs with DIAGNOSTIC.

Fix two cases where we would panic locking against ourselves.

branches: 1.23.2;
- Tidy up the locking a bit.
- Use atomics/kmem_alloc/pool_cache.

Make sure all cached buffers with valid, not yet written data have been
run through copy-on-write. Call fscow_run() with valid data where possible.

The LP_UFSCOW hack is no longer needed to protect ffs_copyonwrite() against
endless recursion.

- Add a flag B_MODIFY to bread(), breada() and breadn(). If set the caller
intends to modify the buffer returned.

- Always run copy-on-write on buffers returned from ffs_balloc().

- Add new function ffs_getblk() that gets a buffer, assigns a new blkno,
may clear the buffer and runs copy-on-write. Process possible errors
from getblk() or fscow_run(). Part of PR kern/38664.

Welcome to 4.99.63

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.21.6; 1.21.8; 1.21.10; 1.21.12;
Use pool_cache.

branches: 1.20.4; 1.20.10;
Merge ffs locking & brelse changes from the vmlocking branch.

branches: 1.19.4; 1.19.6; 1.19.8; 1.19.10;
Add missing RCSID.

branches: 1.18.2;
Move `struct dquot' and its supporting functions from quota.h to ufs_quota.c.

- Make quota-internal functions static.
- Clean up declarations in quota.h and ufs_extern.h. quota.h now has the
description of quota criterions, on-disk structure, user-kernel interface and
declaration of init/done functions. All ufs quota related function
prototypes go to ufs_extern.h.
- New functions ufsquota_init() and ufsquota_free() create or destroy the
quota fields of `struct inode'.
- chkdq() and chkiq() always update the quota fields of `struct inode' first.
- Only ufs_access() explicitely calls getinoquota().

No objections on tech-kern@

Fix merge botch.

Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

Using POOL_INIT here makes no sense, since file systems always have
an init method. So get rid of it and #ifdef _LKM and just always
init in the init method. Give malloc types the same treatment.
Makes file systems nicer to work with in linksetless environments
and fixes a few LKM discrepancies.

branches: 1.14.2;
Pass an ipl argument to pool_init/POOL_INIT to be used when initializing
the pool's lock.

branches: 1.13.2;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.12.2;
Merge newlock2 to head.

branches: 1.11.8;
More MALLOC -> malloc changes.

branches: 1.10.2; 1.10.4; 1.10.6; 1.10.8; 1.10.10;
- unify ffs_blkatoff and lfs_blkatoff.
- remove ufs_ops::uo_blkatoff.
- add directory read-ahead code. (disabled for now.)

make ufsdirhash_pool static.

ufsdirhash_build: yield cpu when looping on directory entries.

branches: 1.7.2;
merge ktrace-lwp.

merge yamt-vop branch. remove following VOPs.

VOP_BLKATOFF
VOP_VALLOC
VOP_BALLOC
VOP_REALLOCBLKS
VOP_VFREE
VOP_TRUNCATE
VOP_UPDATE

branches: 1.5.2;
Defflag UFS_DIRHASH.

branches: 1.4.2;
Change the rest of the sysctl subsystem to use const consistently.
The __UNCONST macro is now used only where necessary and the RW macros
are gone. Most of the changes here are consumers of the
sysctl_createv(9) interface that now takes a pair of const pointers
which used not to be.

s/buf/sbuf.

branches: 1.2.4;
nuke trailing whitespace

branches: 1.1.2; 1.1.4;
Bring in Ian Dowse's Dirhash from FreeBSD. Hash tables of
directories are created on the fly and used to increase
performance by circumventing ufs_lookup's linear search.

Dirhash is enabled by the UFS_DIRHASH option, but not
by default.

Remove extraneous files from Lite2 merge.

branches: 1.1.1;
Initial revision

Fix various typos in comments, log messages and documentation.

ufs: Nix trailing whitespace and tidy up some other minor KNF.

Add containment for the cloning devices hack in vn_open.

Cloning devices (and also things like /dev/stderr) work by allocating
a struct file, stuffing it in the file table (which is a layer
violation), stuffing the file descriptor number for it in a magic
field of struct lwp (which is gross), and then "failing" with one of
two magic errnos, EDUPFD or EMOVEFD.

Before this commit, all callers of vn_open in the kernel (there are
quite a few) were expected to check for these errors and handle the
situation. Needless to say, none of them except for open() itself did,
resulting in internal negative errnos being returned to userspace.

This hack is fairly deeply rooted and cannot be eliminated all at
once. This commit adds logic to handle the magic errnos inside
vn_open; now on success vn_open returns either a vnode or an integer
file descriptor, along with a flag that says whether the underlying
code requested EDUPFD or EMOVEFD. Callers not prepared to cope with
file descriptors can pass NULL for the extra return values, in which
case if a file descriptor would be produced vn_open fails with
EOPNOTSUPP.

Since I'm rearranging vn_open's signature anyway, stop exposing struct
nameidata. Instead, take three arguments: an optional vnode to use as
the starting point (like openat()), the path, and additional namei
flags to use, restricted to NOCHROOT and TRYEMULROOT. (Other namei
behavior, e.g. NOFOLLOW, can be requested via the open flags.)

This change requires a kernel bump. Ride the one an hour ago.
(That was supposed to be coordinated; did not intend to let an hour
slip by. My fault.)

branches: 1.52.6;
Add ACL support for FFS. From FreeBSD.

VFS_VGET(), VFS_ROOT(), VFS_FHTOVP(): give them a "int lktype" argument, to
allow us to get shared locks (or no lock) on the returned vnode. Matches
FreeBSD.

branches: 1.50.2;
- KNF more
- print the full path in error messages

- return (foo) -> return foo
- normalize all error messages to use __func__
- add more messages for startup failure (should perhaps auto-create
the attributes directory and the user and system subdirs?)
- factor out common code

branches: 1.48.16;
Explain why the lock in here needs to be recursive. Related to PR 46997.

branches: 1.47.2;
KNF. Remove extra spaces. No functional change.

branches: 1.46.2;
Fix uninitialized mutex usage

We use extended attribute mount mutex before testing if it had been
initialized, and as reported by Christos, this caused panic with
LOCKDEBUG. Fix it by testing before using.

Fix UFS1 extended attribute backend autocreation deadlock

UFS1 extended attribute backend autocration goes through a vn_open()
to create the backend file, and this forces us to release the lock
on the target node, in case the target is within the parents of the
backend file. That created a window within which another thread could
acquire a lock on the target vnode and deadlock awaiting for the
mount extended attribute lock.

We fix the problem by also releasing the mount extended attribute lock
when calling vn_open(), but that lets another thread race us for backend
creation. We just detect this using O_EXCL for vn_open() and by checking
for EEXIST return code. If we are raced, we fail backend creation but
this is not a problem since another thread succeeded on it: we just have
to use the result.

Fix use-after-free on failed unmount with extended attribute enabled

When unmount failed, for instance because the mount is still busy,
UFS1 extended attributes structures were left freed while the kernel
assumes extended attributes were still enabled. This led to using
UFS1 extended attributes structures after free. With LOCKDEBUG, with
quickly triggers a panic.

The problem is fixed by:
1) clear MNT_EXTATTR flag after extended attributes structures are freed
2) attempt to restart extended attributes after failed unmount
2) set MNT_EXTATTR correctly after extended attributes restart

As a side effect, extended attribute structures are now only initialized
when extended attributes are started for the filesystem.

branches: 1.43.4;
Change vnode operation lookup to return the resulting vnode *vpp unlocked.
Change cache_lookup() to return an unlocked vnode.

Discussed on tech-kern@

Welcome to 6.99.31

branches: 1.42.2;
Stick UFS_ in front of these symbols:
DIRBLKSIZ
DIRECTSIZ
DIRSIZ
OLDDIRFMT
NEWDIRFMT

Part of PR 47909.

Remove always-true condition and note that the current code is suboptimal.

branches: 1.40.2;
Fix unmount returnign EBUSY if an attribute was autocreated: we hold
a useless reference that we never gave back

Return ENODATA when no attribute is found, like Linux does. After
all we decided to adopt the Linux API, therefore there is rationale
to stick to it.

No standard tells us what to do, and our extended attribute API has not
been used in a release, therefore we do not break anything, and we get
more easily compatible with programs using the Linux extended attribute
API.

Note that FreeBSD and MacOS X return ENOATTR. FreeBSD has its own API
and MacOS X has a Linux-like API. How did the world get so complicated?

Fix wrong variable in snprintf. From Henning Petersen in PR 46259,
ok manu@

Replace the remaining KAUTH_GENERIC_ISSUSER authorization calls with
something meaningful. All relevant documentation has been updated or
written.

Most of these changes were brought up in the following messages:

http://mail-index.netbsd.org/tech-kern/2012/01/18/msg012490.html
http://mail-index.netbsd.org/tech-kern/2012/01/19/msg012502.html
http://mail-index.netbsd.org/tech-kern/2012/02/17/msg012728.html

Thanks to christos, manu, njoly, and jmmv for input.

Huge thanks to pgoyette for spinning these changes through some build
cycles and ATF.

branches: 1.36.2;
converting readdir in ffs ext2fs from malloc(9) to kmem(9)
while there allocate ufs mount structs from kmem(9) too
preceding kmem-vmem-pool-patch

releng@ acknowledged

branches: 1.35.2; 1.35.6;
Fix locking protocol to avoid a panic on extattrctl stop and on umount.

Add a flag to VOP_LISTEXTATTR(9) so that the vnode interface can tell the
filesystem in which format extended attribute shall be listed.

There are currently two formats:
- NUL-terminated strings, used for listxattr(2), this is the default.
- one byte length-pprefixed, non NUL-terminated strings, used for
extattr_list_file(2), which is obtanined by setting the
EXTATTR_LIST_PREFIXLEN flag to VOP_LISTEXTATTR(9)

This approach avoid the need for converting the list back and forth, except
in libperfuse, since FUSE uses NUL-terminated strings, and the kernel may
have requested EXTATTR_LIST_PREFIXLEN.

Implement extended attribute listing for UFS1.

Modify lsextattr(8) so that it does not expect each attribute name to be
prefixed by its length. This enable extattr_list_(file|link|fd) to
return a buffer matching its documentation. This also makes the interface
similar to what Linux and FUSE do, which is nice for interoperability.

Note that since we had no EA implementation supporting listing, we do
not break anything.

Add mount -o extattr option to enable extended attributs (corrently only
for UFS1).
Remove kernel option for EA backing store autocreation and do it by
default. Add a sysctl so that autocreated attriutr size can be modified.

Improve UFS1 extended attributes usability
- autocreate attribute backing file for new attributes
- autoload attributes when issuing extattrctl start
- when autoloading attributes, do not display garbage warning when looking
up entries that got ENOENT

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.29.2;
Fix filesystem root leaked lock when using UFS_EXTATTR_AUTOSTART.
This way, statvfs(2) calls obtained by df(1) or umount(8) will no
longer sleep forever in the kernel to acquire the lock.

branches: 1.28.2;
Abolish struct componentname's cn_pnbuf. Use the path buffer in the
pathbuf object passed to namei as work space instead. (For now a pnbuf
pointer appears in struct nameidata, to support certain unclean things
that haven't been fixed yet, but it will be going away in the future.)

This removes the need for the SAVENAME and HASBUF namei flags.

Clean up vnode lock operations pass 2:

VOP_UNLOCK(vp, flags) -> VOP_UNLOCK(vp): Remove the unneeded flags argument.

Welcome to 5.99.32.

Discussed on tech-kern.

branches: 1.26.2; 1.26.4;
The lwp member of struct componentname was removed a long time ago.
Fix broken build with UFS_EXTATTR_AUTOSTART by removing it here as well.

bzero -> memset

Ansify function definitions w/o arguments. Generated with sed.

branches: 1.23.2;
kill MALLOC and FREE macros.

_KERNEL_OPT

branches: 1.21.4; 1.21.10; 1.21.12; 1.21.14;
Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

branches: 1.20.6;
Replace use of lockmgr.

- Fix a couple of locking botches.
- Remove calls to VOP_LEASE().

Destroy extattr lock when destroying extattrs associated with the
mountpoint. Make stopping extattrs always succesful to facilitate
always being able to free resources.

use __KERNEL_RCSID() instead of __RCSID()

branches: 1.16.2; 1.16.4; 1.16.6;
Remove the "struct lwp *" argument from all VFS and VOP interfaces.
The general trend is to remove it from all kernel interfaces and
this is a start. In case the calling lwp is desired, curlwp should
be used.

quick consensus on tech-kern

branches: 1.15.6; 1.15.8; 1.15.14;
Move `struct dquot' and its supporting functions from quota.h to ufs_quota.c.

- Make quota-internal functions static.
- Clean up declarations in quota.h and ufs_extern.h. quota.h now has the
description of quota criterions, on-disk structure, user-kernel interface and
declaration of init/done functions. All ufs quota related function
prototypes go to ufs_extern.h.
- New functions ufsquota_init() and ufsquota_free() create or destroy the
quota fields of `struct inode'.
- chkdq() and chkiq() always update the quota fields of `struct inode' first.
- Only ufs_access() explicitely calls getinoquota().

No objections on tech-kern@

Using POOL_INIT here makes no sense, since file systems always have
an init method. So get rid of it and #ifdef _LKM and just always
init in the init method. Give malloc types the same treatment.
Makes file systems nicer to work with in linksetless environments
and fixes a few LKM discrepancies.

branches: 1.13.2; 1.13.4;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.12.2;
Consistent usage of KAUTH_GENERIC_ISSUSER.

a smorgasbord of improvements to vnode locking and path lookup:
- LOCKPARENT is no longer relevant for lookup(), relookup() or VOP_LOOKUP().
these now always return the parent vnode locked. namei() works as before.
lookup() and various other paths no longer acquire vnode locks in the
wrong order via vrele(). fixes PR 32535.
as a nice side effect, path lookup is also up to 25% faster.
- the above allows us to get rid of PDIRUNLOCK.
- also get rid of WANTPARENT (just use LOCKPARENT and unlock it).
- remove an assumption in layer_node_find() that all file systems implement
a recursive VOP_LOCK() (unionfs doesn't).
- require that all file systems supply vfs_vptofh and vfs_fhtovp routines.
fill in eopnotsupp() for file systems that don't support being exported
and remove the checks for NULL. (layerfs calls these without checking.)
- in union_lookup1(), don't change refcounts in the ISDOTDOT case, just
adjust which vnode is locked. fixes PR 33374.
- apply fixes for ufs_rename() from ufs_vnops.c rev. 1.61 to ext2fs_rename().

branches: 1.10.4; 1.10.6; 1.10.8;
Use the LWP cached credentials where sane.

branches: 1.9.4;
Include <sys/kauth.h> because it's needed.

branches: 1.8.2;
integrate kauth.

branches: 1.7.2; 1.7.4; 1.7.6;
merge yamt-uio_vmspace branch.

- use vmspace rather than proc or lwp where appropriate.
the latter is more natural to specify an address space.
(and less likely to be abused for random purposes.)
- fix a swdmover race.

branches: 1.6.2; 1.6.4; 1.6.6;
Convert UFS_EXTATTR to struct lwp.

merge ktrace-lwp.

branches: 1.4.6; 1.4.8;
In ufs_extattr_stop(), if we haven't started yet, errno must be set
before bailing out.

From FreeBSD.

Add missing '$' in __RCSID().

In ufs_extattr_start(), unlock uepm_lock when bailing out.

Ok'd Jason Thorpe.

Experimental support for extended attributes on UFS1 file systems, using a
backing file per attribute type indexed by inode number to hold the extended
attributes.

This is working pretty well on my test systems, except for the "autostart"
feature. I need someone with a better handle on the VFS locking protocol
to go over that.

This is a work-in-progress. There are parts of this that could be re-factored
allowing this approach to be used on other types of file systems.

Adapted from FreeBSD.

Overhaul of the EVFILT_VNODE kevent(2) filter:

- Centralize vnode kevent handling in the VOP_*() wrappers, rather than
forcing each individual file system to deal with it (except VOP_RENAME(),
because VOP_RENAME() is a mess and we currently have 2 different ways
of handling it; at least it's reasonably well-centralized in the "new"
way).
- Add support for NOTE_OPEN, NOTE_CLOSE, NOTE_CLOSE_WRITE, and NOTE_READ,
compatible with the same events in FreeBSD.
- Track which kevent notifications clients are interested in receiving
to avoid doing work for events no one cares about (avoiding, e.g.
taking locks and traversing the klist to send a NOTE_WRITE when
someone is merely watching for a file to be deleted, for example).

In support of the above:

- Add support in vnode_if.sh for specifying PRE- and POST-op handlers,
to be invoked before and after vop_pre() and vop_post(), respectively.
Basic idea from FreeBSD, but implemented differently.
- Add support in vnode_if.sh for specifying CONTEXT fields in the
vop_*_args structures. These context fields are used to convey information
between the file system VOP function and the VOP wrapper, but do not
occupy an argument slot in the VOP_*() call itself. These context fields
are initialized and subsequently interpreted by PRE- and POST-op handlers.
- Version VOP_REMOVE(), uses the a context field for the file system to report
back the resulting link count of the target vnode. Return this in tmpfs,
udf, nfs, chfs, ext2fs, lfs, and ufs.

NetBSD 9.99.92.

Abolish all the silly indirection macros for initializing vnode ops tables.

These are things of the form #define foofs_op genfs_op, or #define
foofs_op genfs_eopnotsupp, or similar. They serve no purpose besides
obfuscation, and have gotten cutpasted all over everywhere.

branches: 1.86.6;
Add ACL support for FFS. From FreeBSD.

Extended attribute support for ffsv2, from FreeBSD.

branches: 1.84.4;
VFS_VGET(), VFS_ROOT(), VFS_FHTOVP(): give them a "int lktype" argument, to
allow us to get shared locks (or no lock) on the returned vnode. Matches
FreeBSD.

branches: 1.83.16; 1.83.22;
reorganize ffs_truncate()/ffs_indirtrunc() to be able to partially
succeed; change wapbl_register_deallocation() to return EAGAIN
rather than panic when code hits the limit

callers changed to either loop calling ffs_truncate() using new
utility ufs_truncate_retry() if their semantics requires it, or
just ignore the failure; remove ufs_wapbl_truncate()

this fixes possible user-triggerable panic during truncate, and
resolves WAPBL performance issue with truncates of large files

PR kern/47146 and kern/49175

branches: 1.82.2;
Remove gcc hack, it does not help.
Add more const.

Provide reason to be printed in panic string.

misc cleanups, no functional change

Disentangle buffer-cached I/O from page-cached I/O in UFS.

Page-cached I/O is used for regular files, and is initiated by VFS
users such as userland and NFS.

Buffer-cached I/O is used for directories and symlinks, and is issued
only internally by UFS.

New UFS routine ufs_bufio replaces vn_rdwr for internal use.
ufs_bufio is implemented by new UFS operations uo_bufrd/uo_bufwr,
which sit in ufs_readwrite.c alongside the VOP_READ/VOP_WRITE
implementations.

I preserved the code as much as possible and will leave further
simplification for future commits. I kept the ulfs_readwrite.c
copypasta close to ufs_readwrite.c in case we ever want to merge them
back; likewise ext2fs_readwrite.c.

No externally visible semantic change. All atf fs tests still pass.

Change ffs to use vcache_new:
- Change ffs_valloc to return an inode number.
- Remove now obsolete UFS operations UFS_VALLOC and UFS_VFREE.
- Make ufs_makeinode private to ufs_vnops.c and pass vattr instead of mode.

branches: 1.77.2;
simplify and correct.

Move and unify indirect block truncate algorithm into a separate function.

Reviewed by joerg.

branches: 1.75.2;
Remove ufs_checkpath() and ufs_readdotdot(). These are relics
from the pre-genfs_rename era.

Add a global vnode cache:

- vcache_get() retrieves a referenced and initialised vnode / fs node pair.
- vcache_remove() removes a vnode / fs node pair from the cache.

On cache miss vcache_get() calls new vfs operation vfs_loadvnode() to
initialise a vnode / fs node pair. This call is guaranteed exclusive,
no other thread will try to load this vnode / fs node pair.

Convert ufs/ext2fs, ufs/ffs and ufs/mfs to use this interface.

Remove now unused ufs/ufs_ihash

Discussed on tech-kern.

Welcome to 6.99.41

branches: 1.73.2; 1.73.6;
Add an UFS_SNAPGONE() ufs op replacing the calls
to ffs_snapgone() in ufs_lookup.c.

Ok: David Holland <dholland@netbsd.org>

Welcome to 6.99.22

branches: 1.72.2;
Adapt ffs, lfs, and ext2fs to use genfs_rename.

ok dholland, rmind

Change the syscall API for quotas over to the new non-proplib one.

- struct vfs_quotactl_args -> struct quotactl_args
- add sys/stdint.h to sys/quotactl.h for clean userland build
- install sys/quotactl.h in /usr/include
- update set lists for same
- add new marshalling code in libquota
- add new unmarshalling code in vfs_syscalls.c
- discard proplib interpreter code in vfs_quotactl.c
- add dispatching code for the 14 quotactl ops in vfs_quotactl.c
- mark the proplib quotactl syscall obsolete
- add a new syscall number for the new quotactl syscall
- change the name of the syscall to __quotactl()
- remove the decl of the old quotactl from quota/quotaprop.h
- add a decl of the new quotactl to sys/quotactl.h
- update the libc build
- update ktruss
- remove proplib marshalling code from libquota
- update copy of syscall table in gdb ppc sources
- hack rumphijack to accomodate new quotactl name (as I recall,
pooka wanted such a name change to simplify something, but I
don't really see what/how)

This change appears to require a kernel version bump for rumpish
reasons.

Remove the extra op argument to VFS_QUOTACTL() - the op is now stored
purely in the args structure.

This change requires a kernel version bump.

Introduce struct vfs_quotactl_args. Use it.

This change uglifies vfs_quotactl some in order to make room for
moving operation-specific but FS-independent logic out of ufs_quota.c.

Note: this change requires a kernel version bump.

Move the proplib-based quota command dispatching (that is, the code
that knows the magic string names for the allowed actions) out of
UFS-specific code and to fs-independent code.

This introduces QUOTACTL_* operation codes and changes the signature
of VFS_QUOTACTL() again for compile safety.

Note: this change requires a kernel version bump.

Move the code for iterating over the multiple RPC calls in a quota
proplib XML packet to vfs_quotactl.c out of sys/ufs/ufs.

Add a dummy extra arg to VFS_QUOTACTL for compile safety.

Note: this change requires a kernel version bump.

branches: 1.66.2; 1.66.6;
Provide correct locking for ufs_wapbl_rename. Note that this does not
fix the non-wapbl rename; that will be coming soon. This patch also
leaves a lot of the older locking-related code around in #if 0 blocks,
and there's a lot of leftover redundant logic. All that will be going
away later.

Relates to at least these PRs:

PR kern/24887
PR kern/41417
PR kern/42093
PR kern/43626

and possibly others.

Pass the ufs_lookup_results pointer around instead of fetching it from
the inode in the guts of ufs. Now, in VOPs where i_crap is used it is
used (directly) only immediately on entry to the VOP call and then
passed around by reference.

Except for rename, which needs explicit sorting out. The code in
ufs_wapbl_rename is unchanged in behavior but I'm increasingly
inclined to think it's wrong.

add "struct ufid;" so we can include it without ufs/inode.h

merge the bouyer-quota2 branch. This adds a new on-disk format
to store disk quota usage and limits, integrated with ffs
metadata. Usage is checked by fsck_ffs (no more quotacheck)
and is covered by the WAPBL journal. Enabled with kernel
option QUOTA2 (added where QUOTA was enabled in kernel config files),
turned on with tunefs(8) on a per-filesystem
basis. mount_mfs(8) can also turn quotas on.

See http://mail-index.netbsd.org/tech-kern/2011/02/19/msg010025.html
for details.

branches: 1.62.4; 1.62.6; 1.62.8;
Move declaration of ufs_hashlock into <ufs/ufs_extern.h> from each c source.

PR kern/26878 FFSv2 + softdep = livelock (no free ram)
PR kern/16942 panic with softdep and quotas
PR kern/19565 panic: softdep_write_inodeblock: indirect pointer #1 mismatch
PR kern/26274 softdep panic: allocdirect_merge: ...
PR kern/26374 Long delay before non-root users can write to softdep partitions
PR kern/28621 1.6.x "vp != NULL" panic in ffs_softdep.c:4653 while unmounting a softdep (+quota) filesystem
PR kern/29513 FFS+Softdep panic with unfsck-able file-corruption
PR kern/31544 The ffs softdep code appears to fail to write dirty bits to disk
PR kern/31981 stopping scsi disk can cause panic (softdep)
PR kern/32116 kernel panic in softdep (assertion failure)
PR kern/32532 softdep_trackbufs deadlock
PR kern/37191 softdep: locking against myself
PR kern/40474 Kernel panic after remounting raid root with softdep

Retire softdep, pass 2. As discussed and later formally announced on the
mailing lists.

branches: 1.60.6; 1.60.8; 1.60.12;
XXX softdep:

If the number of deletes in progress is getting too high, newdirrem()
requests the syncer to flush faster, and in some cases will block to
prevent deletes accumulating faster than the disk can service them.

The syncer will try to lock vnodes that the remover holds locked, leading
to the syncer and remover proceeding in lockstep and making very little
overall forward progress.

Put a hook into ufs_rmdir() and ufs_remove() so that the softdep code
can pace itself without holding vnode locks if the number of deletes is
running out of control.

Make sure all cached buffers with valid, not yet written data have been
run through copy-on-write. Call fscow_run() with valid data where possible.

The LP_UFSCOW hack is no longer needed to protect ffs_copyonwrite() against
endless recursion.

- Add a flag B_MODIFY to bread(), breada() and breadn(). If set the caller
intends to modify the buffer returned.

- Always run copy-on-write on buffers returned from ffs_balloc().

- Add new function ffs_getblk() that gets a buffer, assigns a new blkno,
may clear the buffer and runs copy-on-write. Process possible errors
from getblk() or fscow_run(). Part of PR kern/38664.

Welcome to 4.99.63

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.58.6; 1.58.8; 1.58.10; 1.58.12;
Remove VOP_LEASE. Discussed on tech-kern.

Use pool_cache.

Merge vmlocking2 to head.

branches: 1.55.2; 1.55.6;
Remove the "struct lwp *" argument from all VFS and VOP interfaces.
The general trend is to remove it from all kernel interfaces and
this is a start. In case the calling lwp is desired, curlwp should
be used.

quick consensus on tech-kern

branches: 1.54.2; 1.54.8;
Move the fstrans-aware lock vnops from ufs to ffs. Other ufs file systems
do not need them.

Ride on 4.99.28

branches: 1.53.2; 1.53.6;
Move `struct dquot' and its supporting functions from quota.h to ufs_quota.c.

- Make quota-internal functions static.
- Clean up declarations in quota.h and ufs_extern.h. quota.h now has the
description of quota criterions, on-disk structure, user-kernel interface and
declaration of init/done functions. All ufs quota related function
prototypes go to ufs_extern.h.
- New functions ufsquota_init() and ufsquota_free() create or destroy the
quota fields of `struct inode'.
- chkdq() and chkiq() always update the quota fields of `struct inode' first.
- Only ufs_access() explicitely calls getinoquota().

No objections on tech-kern@

branches: 1.52.2; 1.52.4;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

branches: 1.50.2;
New file system suspension API to replace vn_start_write and vn_finished_write.
The suspension helpers are now put into file system specific operations.
This means every file system not supporting these helpers cannot be suspended
and therefore snapshots are no longer possible.

Implemented for file systems of type ffs.

The new API is enabled on a kernel option NEWVNGATE. This option is
not enabled by default in any kernel config.

Presented and discussed on tech-kern with much input from
Bill Studenmund <wrstuden@netbsd.org> and YAMAMOTO Takashi <yamt@netbsd.org>.

Welcome to 4.99.9 (new vfs op vfs_suspendctl).

branches: 1.49.8;
integrate kauth.

branches: 1.48.2; 1.48.4; 1.48.6; 1.48.8; 1.48.10;
- unify ffs_blkatoff and lfs_blkatoff.
- remove ufs_ops::uo_blkatoff.
- add directory read-ahead code. (disabled for now.)

branches: 1.47.2;
merge ktrace-lwp.

Apply the NFS exports list rototill patch:

- Remove all NFS related stuff from file system specific code.
- Drop the vfs_checkexp hook and generalize it in the new nfs_check_export
function, thus removing redundancy from all file systems.
- Move all NFS export-related stuff from kern/vfs_subr.c to the new
file sys/nfs/nfs_export.c. The former was becoming large and its code
is always compiled, regardless of the build options. Using the latter,
the code is only compiled in when NFSSERVER is enabled. While doing this,
also make some functions in nfs_subs.c conditional to NFSSERVER.
- Add a new command in nfssvc(2), called NFSSVC_SETEXPORTSLIST, that takes a
path and a set of export entries. At the moment it can only clear the
exports list or append entries, one by one, but it is done in a way that
allows setting the whole set of entries atomically in the future (see the
comment in mountd_set_exports_list or in doc/TODO).
- Change mountd(8) to use the nfssvc(2) system call instead of mount(2) so
that it becomes file system agnostic. In fact, all this whole thing was
done to remove a 'XXX' block from this utility!
- Change the mount*, newfs and fsck* userland utilities to not deal with NFS
exports initialization; done internally by the kernel when initializing
the NFS support for each file system.
- Implement an interface for VFS (called VFS hooks) so that several kernel
subsystems can run arbitrary code upon receipt of specific VFS events.
At the moment, this only provides support for unmount and is used to
destroy NFS exports lists from the file systems being unmounted, though it
has room for extension.

Thanks go to yamt@, chs@, thorpej@, wrstuden@ and others for their comments
and advice in the development of this patch.

update file timestamps for nfsd loaned-read and mmap.
PR/25279. discussed on tech-kern@.

- Use ANSI function decls.
- Sprinkle some static.

branches: 1.43.2;
- sprinkle const
- avoid shadow variables.

branches: 1.42.2;
nuke trailing whitespace

branches: 1.41.4; 1.41.6;
Use a pool for struct direct instead of kernel stack.
Reduces the kernel stack usage by 264 bytes.

Approved by: Jason R. Thorpe <thorpej@netbsd.org>

Add ffs internal snapshots. Written by Marshall Kirk McKusick for FreeBSD.

- Not enabled by default. Needs kernel option FFS_SNAPSHOT.
- Change parameters of ffs_blkfree.
- Let the copy-on-write functions return an error so spec_strategy
may fail if the copy-on-write fails.
- Change genfs_*lock*() to use vp->v_vnlock instead of &vp->v_lock.
- Add flag B_METAONLY to VOP_BALLOC to return indirect block buffer.
- Add a function ffs_checkfreefile needed for snapshot creation.
- Add special handling of snapshot files:
Snapshots may not be opened for writing and the attributes are read-only.
Use the mtime as the time this snapshot was taken.
Deny mtime updates for snapshot files.
- Add function transferlockers to transfer any waiting processes from
one lock to another.
- Add vfsop VFS_SNAPSHOT to take a snapshot and make it accessible through
a vnode.
- Add snapshot support to ls, fsck_ffs and dump.

Welcome to 2.0F.

Approved by: Jason R. Thorpe <thorpej@netbsd.org>

First pass for some caddr_t removal and changes to get rid of it where we
no longer use and/or need it

- removed casts from unionfs, deadfs and fdesc
(there are more to hunt down still)
- changed vfs_quotactl args argumet from caddr_t to void *
- changed vfs_quotactl structures/callers to reflect the api change

Compiled fine and ran for about a day. Approved/reviewed by
christos@netbsd.org and gimpy@netbsd.org.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

Pass the inode flags to set as an argument to ufs_dirrewrite().
Use it to restore the behaviour of not updating the modified time of a
directory that moves to a new parent.

branches: 1.36.2;
Back out the lwp/ktrace changes. They contained a lot of colateral damage,
and need to be examined and discussed more.

Undo part of the ktrace/lwp changes. In particular:
* Remove the "lwp *" argument that was added to vget(). Turns out
that nothing actually used it!
* Remove the "lwp *" arguments that were added to VFS_ROOT(), VFS_VGET(),
and VFS_FHTOVP(); all they did was pass it to vget() (which, as noted
above, didn't use it).
* Remove all of the "lwp *" arguments to internal functions that were added
just to appease the above.

Pass lwp pointers throughtout the kernel, as required, so that the lwpid can
be inserted into ktrace records. The general change has been to replace
"struct proc *" with "struct lwp *" in various function prototypes, pass
the lwp through and use l_proc to get the process pointer when needed.

Bump the kernel rev up to 1.6V

make is_sequential a callback in order to achieve better lfs write clustering.

since lfs always rewrite blocks into the new segment,
current on-disk place of the block doesn't affect to write clustering.

ok'ed by Konrad Schroder.

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Add code to UBCify LFS. This is still behind "#ifdef LFS_UBC" for now
(there are still some details to work out) but expect that to go
away soon. To support these basic changes (creation of lfs_putpages,
lfs_gop_write, mods to lfs_balloc) several other changes were made, to
wit:

* Create a writer daemon kernel thread whose purpose is to handle page
writes for the pagedaemon, but which also takes over some of the
functions of lfs_check(). This thread is started the first time an
LFS is mounted.

* Add a "flags" parameter to GOP_SIZE. Current values are
GOP_SIZE_READ, meaning that the call should return the size of the
in-core version of the file, and GOP_SIZE_WRITE, meaning that it
should return the on-disk size. One of GOP_SIZE_READ or
GOP_SIZE_WRITE must be specified.

* Instead of using malloc(...M_WAITOK) for everything, reserve enough
resources to get by and use malloc(...M_NOWAIT), using the reserves if
necessary. Use the pool subsystem for structures small enough that
this is feasible. This also obsoletes LFS_THROTTLE.

And a few that are not strictly necessary:

* Moves the LFS inode extensions off onto a separately allocated
structure; getting closer to LFS as an LKM. "Welcome to 1.6O."

* Unified GOP_ALLOC between FFS and LFS.

* Update LFS copyright headers to correct values.

* Actually cast to unsigned in lfs_shellsort, like the comment says.

* Keep track of which segments were empty before the previous
checkpoint; any segments that pass two checkpoints both dirty and
empty can be summarily cleaned. Do this. Right now lfs_segclean
still works, but this should be turned into an effectless
compatibility syscall.

Bump daddr_t to 64 bits. Replace it with int32_t in all places where
it was used on-disk, so that on-disk formats remain the same.
Remove ufs_daddr_t and ufs_lbn_t for the time being.

Add multiple inclusion protection for headers. Fix mismatched
variable declarations (missing const's) as needed.

Bring over fixes from FreeBSD that weren't incorporated yet, mainly
from Kirk McKusick. They implement taking pending block/inode frees
into account for the sake of correct statfs() numbers, and adding
a new softdep type (newdirblk) to correctly handle newly allocated
directory blocks.

Minor additional changes: 1) swap the newly introduced fs_pendinginodes
and fs_pendingblock fields in ffs_sb_swap, and 2) declare lkt_held
in the debug version of the softdep lock structure volatile, as it
can be modified from interrupt context #ifdef DEBUG.