Home | History | Annotate | Download | only in lfs
History log of /src/sys/ufs/lfs/lfs_cksum.c
RevisionDateAuthorComments
 1.30  02-Aug-2015  dholland Add a (draft) 64-bit superblock. Make things build again.

Add pieces of support for using both superblock types where
convenient, and specifically to the superblock accessors, but don't
actually enable it anywhere.

First substantive step on PR 50000.
 1.29  06-Jun-2013  dholland branches: 1.29.10;
Cleanups and hacks to make lfs userland stuff build:
- lfs_cksum.c doesn't actually need ulfs_inode.h any more.
- neither does lfs_itimes.c.
- add hacks to fsck_lfs to make it compile.
- add hacks to newfs_lfs to make it compile.
- fix warning in ulfs_quota.c when quotas are fully disabled
(as I guess is happening with the rumpity version)

XXX: This commit adds -I${NETBSDSRCDIR}/sys to the Makefiles for
XXX: fsck_lfs, newfs_lfs, and lfs_cleanerd. This needs to be cleaned
XXX: up ASAP; but I consider this less problematic in the short term
XXX: than spewing ulfs_*.h into /usr/include.
 1.28  06-Jun-2013  dholland Split lfs from ufs, part 2:

Change all <ufs/ufs/foo.h> includes to <ufs/lfs/ulfs_foo.h>.
 1.27  28-Apr-2008  martin branches: 1.27.34; 1.27.44;
Remove clause 3 and 4 from TNF licenses
 1.26  11-Dec-2005  christos branches: 1.26.70; 1.26.72; 1.26.74;
merge ktrace-lwp.
 1.25  26-Feb-2005  perry nuke trailing whitespace
 1.24  09-Mar-2004  yamt branches: 1.24.8; 1.24.10;
calculate data checksum inline.
 1.23  07-Aug-2003  agc Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.
 1.22  20-Feb-2003  perseant branches: 1.22.2;
Tabify, and fix some comment alignment problems.
 1.21  17-Feb-2003  perseant Add code to UBCify LFS. This is still behind "#ifdef LFS_UBC" for now
(there are still some details to work out) but expect that to go
away soon. To support these basic changes (creation of lfs_putpages,
lfs_gop_write, mods to lfs_balloc) several other changes were made, to
wit:

* Create a writer daemon kernel thread whose purpose is to handle page
writes for the pagedaemon, but which also takes over some of the
functions of lfs_check(). This thread is started the first time an
LFS is mounted.

* Add a "flags" parameter to GOP_SIZE. Current values are
GOP_SIZE_READ, meaning that the call should return the size of the
in-core version of the file, and GOP_SIZE_WRITE, meaning that it
should return the on-disk size. One of GOP_SIZE_READ or
GOP_SIZE_WRITE must be specified.

* Instead of using malloc(...M_WAITOK) for everything, reserve enough
resources to get by and use malloc(...M_NOWAIT), using the reserves if
necessary. Use the pool subsystem for structures small enough that
this is feasible. This also obsoletes LFS_THROTTLE.

And a few that are not strictly necessary:

* Moves the LFS inode extensions off onto a separately allocated
structure; getting closer to LFS as an LKM. "Welcome to 1.6O."

* Unified GOP_ALLOC between FFS and LFS.

* Update LFS copyright headers to correct values.

* Actually cast to unsigned in lfs_shellsort, like the comment says.

* Keep track of which segments were empty before the previous
checkpoint; any segments that pass two checkpoints both dirty and
empty can be summarily cleaned. Do this. Right now lfs_segclean
still works, but this should be turned into an effectless
compatibility syscall.
 1.20  16-Jun-2002  perseant For synchronous writes, keep separate i/o counters for each write, so
processes don't have to wait for one another to finish (e.g., nfsd seems
to be a little happier now, though I haven't measured the difference).
Synchronous checkpoints, however, must always wait for all i/o to finish.

Take the contents of the callback functions and have them run in thread
context instead (aiodoned thread). lfs_iocount no longer has to be
protected in splbio(), and quite a bit less of the segment construction
loop needs to be in splbio() as well.

If lfs_markv is handed a block that is not the correct size according to
the inode, refuse to process it. (Formerly it was extended to the "correct"
size.) This is possibly more prone to deadlock, but less prone to corruption.

lfs_segclean now outright refuses to clean segments that appear to have live
bytes in them. Again this may be more prone to deadlock but avoids
corruption.

Replace ufsspec_close and ufsfifo_close with LFS equivalents; this means
that no UFS functions need to know about LFS_ITIMES any more. Remove
the reference from ufs/inode.h.

Tested on i386, test-compiled on alpha.
 1.19  15-Nov-2001  lukem branches: 1.19.8; 1.19.10;
don't need <sys/types.h> when including <sys/param.h>
 1.18  08-Nov-2001  lukem add RCSID
 1.17  26-Oct-2001  lukem remove #include <ufs/ufs/quota.h> where it was just to appease
<ufs/ufs/inode.h>, since the latter now includes the former. leave the former
in source that obviously uses specific bits of it (for completeness.)
 1.16  13-Jul-2001  perseant branches: 1.16.4;
Merge the short-lived perseant-lfsv2 branch into the trunk.

Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default. Changes for the v2 layout include:

- Segments of non-PO2 size and arbitrary block offset, so these can be
matched to convenient physical characteristics of the partition (e.g.,
stripe or track size and offset).

- Address by fragment instead of by disk sector, paving the way for
non-512-byte-sector devices. In theory fragments can be as large
as you like, though in reality they must be smaller than MAXBSIZE in size.

- Use serial number and filesystem identifier to ensure that roll-forward
doesn't get old data and think it's new. Roll-forward is enabled for
v2 filesystems, though not for v1 filesystems by default.

- The inode free list is now a tailq, paving the way for undelete (undelete
is not yet implemented, but can be without further non-backwards-compatible
changes to disk structures).

- Inode atime information is kept in the Ifile, instead of on the inode;
that is, the inode is never written *just* because atime was changed.
Because of this the inodes remain near the file data on the disk, rather
than wandering all over as the disk is read repeatedly. This speeds up
repeated reads by a small but noticeable amount.

Other changes of note include:

- The ifile written by newfs_lfs can now be of arbitrary length, it is no
longer restricted to a single indirect block.

- Fixed an old bug where ctime was changed every time a vnode was created.
I need to look more closely to make sure that the times are only updated
during write(2) and friends, not after-the-fact during a segment write,
and certainly not by the cleaner.
 1.15  04-Feb-2001  christos branches: 1.15.2; 1.15.4; 1.15.6;
don't include lfs_extern.h; ufs/inode.h does too.
 1.14  25-Nov-2000  perseant Use u_int32_t instead of u_long to compute LFS checksums, since the
checksum is stored in a u_int32_t.
 1.13  09-Sep-2000  perseant Various bug-fixes to LFS, to wit:


Kernel:

* Add runtime quantity lfs_ravail, the number of disk-blocks reserved
for writing. Writes to the filesystem first reserve a maximum amount
of blocks before their write is allowed to proceed; after the blocks
are allocated the reserved total is reduced by a corresponding amount.

If the lfs_reserve function cannot immediately reserve the requested
number of blocks, the inode is unlocked, and the thread sleeps until
the cleaner has made enough space available for the blocks to be
reserved. In this way large files can be written to the filesystem
(or, smaller files can be written to a nearly-full but thoroughly
clean filesystem) and the cleaner can still function properly.

* Remove explicit switching on dlfs_minfreeseg from the kernel code; it
is now merely a fs-creation parameter used to compute dlfs_avail and
dlfs_bfree (and used by fsck_lfs(8) to check their accuracy). Its
former role is better assumed by a properly computed dlfs_avail.

* Bounds-check inode numbers submitted through lfs_bmapv and lfs_markv.
This prevents a panic, but, if the cleaner is feeding the filesystem
the wrong data, you are still in a world of hurt.

* Cleanup: remove explicit references of DEV_BSIZE in favor of
btodb()/dbtob().

lfs_cleanerd:

* Make -n mean "send N segments' blocks through a single call to
lfs_markv". Previously it had meant "clean N segments though N calls
to lfs_markv, before looking again to see if more need to be cleaned".
The new behavior gives better packing of direct data on disk with as
little metadata as possible, largely alleviating the problem that the
cleaner can consume more disk through inefficient use of metadata than
it frees by moving dirty data away from clean "holes" to produce
entirely clean segments.

* Make -b mean "read as many segments as necessary to write N segments
of dirty data back to disk", rather than its former meaning of "read
as many segments as necessary to free N segments worth of space". The
new meaning, combined with the new -n behavior described above,
further aids in cleaning storage efficiency as entire segments can be
written at once, using as few blocks as possible for segment summaries
and inode blocks.

* Make the cleaner take note of segments which could not be cleaned due
to error, and not attempt to clean them until they are entirely free
of dirty blocks. This prevents the case in which a cleanerd running
with -n 1 and without -b (formerly the default) would spin trying
repeatedly to clean a corrupt segment, while the remaining space
filled and deadlocked the filesystem.

* Update the lfs_cleanerd manual page to describe all the options,
including the changes mentioned here (in particular, the -b and -n
flags were previously undocumented).

fsck_lfs:

* Check, and optionally fix, lfs_avail (to an exact figure) and
lfs_bfree (within a margin of error) in pass 5.

newfs_lfs:

* Reduce the default dlfs_minfreeseg to 1/20 of the total segments.

* Add a warning if the sgs disklabel field is 16 (the default for FFS'
cpg, but not usually desirable for LFS' sgs: 5--8 is a better range).

* Change the calculation of lfs_avail and lfs_bfree, corresponding to
the kernel changes mentioned above.

mount_lfs:

* Add -N and -b options to pass corresponding -n and -b options to
lfs_cleanerd.

* Default to calling lfs_cleanerd with "-b -n 4".


[All of these changes were largely tested in the 1.5 branch, with the
idea that they (along with previous un-pulled-up work) could be applied
to the branch while it was still in ALPHA2; however my test system has
experienced corruption on another filesystem (/dev/console has gone
missing :^), and, while I believe this unrelated to the LFS changes, I
cannot with good conscience request that the changes be pulled up.]
 1.12  30-Mar-2000  augustss branches: 1.12.4;
Remove register declarations.
 1.11  25-Mar-1999  perseant branches: 1.11.8;
Change lfs_sb_cksum to use offsetof() instead of an inlined version.

Fix lfs_vref/lfs_vunredf to ignore VXLOCKed vnodes that are also being
flushed.

Improve the debugging messages somewhat.
 1.10  10-Mar-1999  perseant New sources should leave the LFS in a more-or-less working state. Changes
include:

- DIROP segregation is enabled, and greater care is taken
to make sure that a checkpoint completes. Fsck is not
needed to remount the filesystem.
- Several checks to make sure that the LFS subsystem does not
overuse various resources (memory, in particular).
- The cleaner routines, lfs_markv in particular, are completely
rewritten. A buffer overflow is removed. Greater care is taken
to ensure that inodes come from where lfs_cleanerd say they come
from (so we know nothing has changed since lfs_bmapv was called).
- Fragment allocation is fixed, so that writes beyond end-of-file
do the right thing.
 1.9  11-Sep-1998  pk PR#6032: define fixed sized on-disk superblock structure.
 1.8  01-Mar-1998  fvdl Merge with Lite2 + local changes
 1.7  15-Sep-1997  lukem prototype lfs_cksum ifndef KERNEL
 1.6  16-Feb-1996  christos branches: 1.6.12;
Protect include in lfs_cksum.c so that it can be used by userland programs.
 1.5  09-Feb-1996  christos lfs prototypes
 1.4  14-Dec-1994  mycroft Sync with CSRG.
 1.3  20-Sep-1994  cgd c syntax
 1.2  29-Jun-1994  cgd New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD'
 1.1  08-Jun-1994  mycroft branches: 1.1.1;
Update to 4.4-Lite fs code, with local changes.
 1.1.1.2  01-Mar-1998  fvdl Import 4.4BSD-Lite2
 1.1.1.1  01-Mar-1998  fvdl Import 4.4BSD-Lite for reference
 1.6.12.1  22-Sep-1997  thorpej Update marc-pcmcia branch from trunk.
 1.11.8.3  11-Feb-2001  bouyer Sync with HEAD.
 1.11.8.2  08-Dec-2000  bouyer Sync with HEAD.
 1.11.8.1  20-Nov-2000  bouyer Update thorpej_scsipi to -current as of a month ago
 1.12.4.1  14-Sep-2000  perseant Pull up recent LFS kernel changes (approved by thorpej):

ufs/ufs/inode.h, 1.20--1.22 (add i_lfs_effnblks extension ;
make ITIMES aware of LFS_ITIMES;
_LKM protection so userland progs
compile)
ufs/ufs/ufs_vnops.c, 1.69, 1.71 (remove IN_ADIROP;
use ITIMES instead of FFS_ITIMES)
ufs/ufs/ufs_readwrite.c, 1.27 (use lfs_reserve in lfs_write)
ufs/lfs/lfs.h, 1.26--1.32 (define LFS_EST_* macros ;
change MIN_FREE_SEGS to lfs_minfreesegs ;
add avail and bfree to CLEANERINFO ;
change lfs_uinodes to signed ;
change lfs_dmeta to signed ;
add whitespace to line up structure
members ;
explicit cast to int32_t in LFS_EST_*
macros)
ufs/lfs/lfs_alloc.c, back out 1.34.2.3 (pullups of 1.39, 1.40);
then pull up 1.38 (clean up on error)
1.39--1.43 (restore fvdl's ufs_hashlock fix ;
restore fvdl's ufs_hashlock fix ;
set i_lfs_effnblks ;
use UINO macros ;
add comments and fix long lines)
ufs/lfs/lfs_balloc.c, 1.19 (don't succeed halfway)
1.21--1.25 (use i_lfs_effnblks ;
fix i_lfs_effnblks computation and
quieten ;
fix i_ffs_blocks in unwritten fragment ;
remove useless debugging check ;
add comments and (c) 2000)
ufs/lfs/lfs_bio.c, 1.24--1.30 (cleanup and make lfs_flush_fs take
"struct lfs *" instead of "struct
mount *" ;
use lfs_minfreeseg instead of
MIN_FREE_SEGS ;
use UINO macros, and copy bfree/avail
to CLEANERINFO ;
add lfs_reserve function ;
1.28--1.30 fix printf formatting)
ufs/lfs/lfs_cksum.c, 1.13 (add (c) 2000)
ufs/lfs/lfs_debug.c, 1.11 (use btodb instead of DEV_BSIZE)
ufs/lfs/lfs_extern.h, 1.18, 1.20--1.21 (function prototype changes)
ufs/lfs/lfs_inode.c, 1.38 (rewrite lfs_truncate from
ffs_truncate)
1.40--1.44 (count written and unwritten blocks
seperately ;
use disk block units instead of bytes ;
remove unnecessary "mod" variable ;
correct B_DELWRI to avoid bawrite panic ;
use lfs_reserve)
ufs/lfs/lfs_segment.c, 1.52-1.59 (use lfs_dmeta to note used summaries ;
check for UNWRITTEN in indirect blocks ;
more debugging stuff inside #ifdef
DEBUG_LFS ;
use LK_CANRECURSE ;
don't drop dirty indirect blocks ;
use UINO macros ;
don't hose the free list ;
use btodb() instead of DEV_BSIZE ;
make it compile again (oops))
ufs/lfs/lfs_subr.c, 1.16--1.17 (check for locked inodes before
changing ;
use btodb() instead of DEV_BSIZE, (c)
2000)
ufs/lfs/lfs_syscalls.c, back out 1.41.4.2 (fvdl's ufs_hashlock fix);
then pull up 1.43 (use lfs_dmeta)
1.44--1.45 (restore fvdl's ufs_hashlock fix)
1.46--1.47 (fix lfs_avail leakage from sblock
segments ;
use UINO macros)
1.49 (bounds-check inode numbers in
lfs_markv)
ufs/lfs/lfs_vfsops.c, 1.53 (use LFS_EST_* macros in lfs_statfs)
1.56--1.58 (initialize lfs_minfreeseg, lfs_effnblk ;
initialize lfs_uinodes ;
initialize lfs_ravail)
ufs/lfs/lfs_vnops.c, 1.40 (remove VDIROP from removed files)
1.42--1.44 (move SET_ENDOP below the removal of
VDIROP ;
use UINO macros and add lfs_itimes
function ;
use lfs_reserve in dirops)
 1.15.6.3  23-Jun-2002  jdolecek catch up with -current on kqueue branch
 1.15.6.2  10-Jan-2002  thorpej Sync kqueue branch with -current.
 1.15.6.1  03-Aug-2001  lukem update to -current
 1.15.4.1  29-Jun-2001  perseant Get rid of __P(), protoizing where it had not already been done
 1.15.2.4  20-Jun-2002  nathanw Catch up to -current.
 1.15.2.3  08-Jan-2002  nathanw Catch up to -current.
 1.15.2.2  14-Nov-2001  nathanw Catch up to -current.
 1.15.2.1  24-Aug-2001  nathanw Catch up with -current.
 1.16.4.1  12-Nov-2001  thorpej Sync the thorpej-mips-cache branch with -current.
 1.19.10.1  20-Jun-2002  lukem Pull up revision 1.20 (requested by perseant in ticket #325):
For synchronous writes, keep separate i/o counters for each write, so
processes don't have to wait for one another to finish (e.g., nfsd seems
to be a little happier now, though I haven't measured the difference).
Synchronous checkpoints, however, must always wait for all i/o to finish.
Take the contents of the callback functions and have them run in thread
context instead (aiodoned thread). lfs_iocount no longer has to be
protected in splbio(), and quite a bit less of the segment construction
loop needs to be in splbio() as well.
If lfs_markv is handed a block that is not the correct size according to
the inode, refuse to process it. (Formerly it was extended to the "correct"
size.) This is possibly more prone to deadlock, but less prone to corruption.
lfs_segclean now outright refuses to clean segments that appear to have live
bytes in them. Again this may be more prone to deadlock but avoids
corruption.
Replace ufsspec_close and ufsfifo_close with LFS equivalents; this means
that no UFS functions need to know about LFS_ITIMES any more. Remove
the reference from ufs/inode.h.
Tested on i386, test-compiled on alpha.
 1.19.8.1  20-Jun-2002  gehenna catch up with -current.
 1.22.2.4  04-Mar-2005  skrll Sync with HEAD.

Hi Perry!
 1.22.2.3  21-Sep-2004  skrll Fix the sync with head I botched.
 1.22.2.2  18-Sep-2004  skrll Sync with HEAD.
 1.22.2.1  03-Aug-2004  skrll Sync with HEAD
 1.24.10.1  19-Mar-2005  yamt sync with head. xen and whitespace. xen part is not finished.
 1.24.8.1  29-Apr-2005  kent sync with -current
 1.26.74.1  16-May-2008  yamt sync with head.
 1.26.72.1  18-May-2008  yamt sync with head.
 1.26.70.1  02-Jun-2008  mjf Sync with HEAD.
 1.27.44.2  03-Dec-2017  jdolecek update from HEAD
 1.27.44.1  23-Jun-2013  tls resync from head
 1.27.34.1  22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.29.10.1  22-Sep-2015  skrll Sync with HEAD

RSS XML Feed