bsd.own.mk: rename GCC_NO_* to CC_WNO_*

Rename compiler-warning-disable variables from
GCC_NO_warning
to
CC_WNO_warning
where warning is the full warning name as used by the compiler.

GCC_NO_IMPLICIT_FALLTHRU is CC_WNO_IMPLICIT_FALLTHROUGH

Using the convention CC_compilerflag, where compilerflag
is based on the full compiler flag name.

introduce some common variables for use in GCC warning disables:

GCC_NO_FORMAT_TRUNCATION -Wno-format-truncation (GCC 7/8)
GCC_NO_STRINGOP_TRUNCATION -Wno-stringop-truncation (GCC 8)
GCC_NO_STRINGOP_OVERFLOW -Wno-stringop-overflow (GCC 8)
GCC_NO_CAST_FUNCTION_TYPE -Wno-cast-function-type (GCC 8)

use these to turn off warnings for most GCC-8 complaints. many
of these are false positives, most of the real bugs are already
commited, or are yet to come.


we plan to introduce versions of (some?) of these that use the
"-Wno-error=" form, which still displays the warnings but does
not make it an error, and all of the above will be re-considered
as either being "fix me" (warning still displayed) or "warning
is wrong."

branches: 1.16.46;
Use RUMPPRG.

ok Greg Oster

Update RUMP_ACTION to use rumpclient.

The server must of course have some disks configured. Let's say
we have this simple server with disks as a few sparse host files:

main()
{
rump_init();
rump_pub_etfs_register("/disk1", "./disk1.img", RUMP_ETFS_BLK);
rump_pub_etfs_register("/disk2", "./disk2.img", RUMP_ETFS_BLK);
rump_pub_etfs_register("/disk3", "./disk3.img", RUMP_ETFS_BLK);
rump_pub_etfs_register("/disk4", "./disk4.img", RUMP_ETFS_BLK);
pause();
}

And we run the server:

mainbus0 (root)
Kernelized RAIDframe activated
/disk1: hostpath ./disk1.img (97 GB)
/disk2: hostpath ./disk2.img (97 GB)
/disk3: hostpath ./disk3.img (97 GB)
/disk4: hostpath ./disk4.img (97 GB)

We can then configure the raid against the server:

> ./raidctl -c theraid.conf raid0

And lo, we have evidence of a level1 raid in the server dmesg:

raid0: RAID Level 1
raid0: Components: /disk1 /disk2 /disk3 /disk4
raid0: Total Sectors: 409599744 (199999 MB)

yea, i initialized it already in a previous run:

> ./raidctl -S raid0
Reconstruction is 100% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.

close rump kernel file descriptors properly

Support RUMP_ACTION, i.e. compile-time switch to make this execute
rump syscalls to configure raidframe in a rump kernel.

Move WARNS=3 to the Makefile.inc, and add a little const to the remaining
programs that did not compile before.

Kill __P(), use ANSI function declarations; WARNS=3.

WARNS=2 is the default defined in sbin/Makefile.inc. (thanks wiz)

Remove cruft. We no longer need to look into sys/dev/raidframe to
find the needed include files.

Move parts of a RAID type mapping table from rf_layout.c to rf_configure.c.
This means we don't need to compile or link with rf_layout.c here.

Build with WARNS=2.

Move two functions from rf_strutils.c into rf_configure.c, removing
the need for rf_strutils.c.

Move COPTS to CPPFLAGS. Nuke some no-longer needed compile options.

Add missing NetBSD RCS ID's! Thanks to SAITOH Masanobu (msaitoh@netbsd.org)
for pointing this out.

no need for CWARNFLAGS; ../Makefile.inc sets WARNS=1

Allow CFLAGS to be set in /etc/mk.conf

RAIDframe, version 1.1, from the Parallel Data Laboratory at
Carnegie Mellon University. Full RAID implementation, including
levels 0, 1, 4, 5, 6, parity logging, and a few other goodies.
Ported to NetBSD by Greg Oster.

raidctl is our userland configuration tool for RAIDframe.

Exterminate a bug I created in 2009.

branches: 1.1.2;
Use RUMPPRG.

ok Greg Oster

We no longer need the deprecated copyback functionality now that
incorporating a used spare is automatic.

Copyback has always been an issue, as to do a copyback all IO to
the array had to be suspended, and so was very, very unlikely to
have been used in anything resembling a production system.

Implement command-line configuration of simple RAID sets with raidctl
based on the usage pattern:

raidctl <device> create <level> <component1> <component2> ...

For example,

raidctl raid0 create mirror absent /dev/wd1e

will create a RAID level 1 (mirror) set with an absent first component
and /dev/wd1e as the second component. The resulting RAID device will
be marked as auto-configurable, will have a serial number set (based
on the current time), and parity will be initialized. Reasonable
performance values are automatically used by default for other parameters
normally specified in the configuration file.

Also: Only print out Autoconfig status if being verbose.

Implement hot removal of spares and components. From manu@.

Implement a long desired feature of automatically incorporating
a used spare into the array after a reconstruct.

Given the configuration:
Components:
/dev/wd0e: failed
/dev/wd1e: optimal
/dev/wd2e: optimal
Spares:
/dev/wd3e: spare

Running 'raidctl -F /dev/wd0e raid0' will now result in the
following configuration after a successful rebuild:
Components:
/dev/wd3e: optimal
/dev/wd1e: optimal
/dev/wd2e: optimal
No spares.

Thanks to manu@ for the development of the initial set of changes
which allowed the changes to automatically incorporate a used spare
to come to fruition. Thanks also to manu@ for useful discussions
about and additional testing of these changes.

branches: 1.79.2;
Implement "raidctl -t config-file"

This does the same config file parse that -c/-C do, but only
that (hence no raidframe device is needed, or accepted).

Any syntax errors in the config file will be reported, nothing
else happens.

Support on-demand re-scanning all devices to look for
autoconfig RAID sets. raidctl now supports looking
for autoconfig RAID sets with a new '-L' flag.

Avoid using Pq.

In this case, the parentheses were wrong when a line was extended.

move the basic history i wrote from raidctl(8) to raid(4), and
fix the x86 (thanks simonb) and sparc bootable dates, a couple
of spelling errors.

update the list of platforms supporting loading kernels from raid.
expand the HISTORY section to include notable RF moments in netbsd.
bump date.

branches: 1.74.10;
implement 32-bit compat support for raidframe.

convert several raidframe ioctls to be bitsize idempotent so that
they work the same in 32 and 64 bit worlds, allowing netbsd32 to
configure and query raid properly. remove useless 'row' in a few
places. add COMPAT_80 and put the old ioctls there.

raidframeio.h:
RAIDFRAME_TEST_ACC
- remove, unused
RAIDFRAME_GET_COMPONENT_LABEL
- convert to label not pointer to label
RAIDFRAME_CHECK_RECON_STATUS_EXT
RAIDFRAME_CHECK_PARITYREWRITE_STATUS_EXT
RAIDFRAME_CHECK_COPYBACK_STATUS_EXT
- convert to progress info not pointer to info
RAIDFRAME_GET_INFO
- version entirely.
raidframevar.h:
- rf_recon_req{} has row, flags and raidPtr removed (they're
not a useful part of this interface.)
- RF_Config_s{} and RF_DeviceConfig_s{} have numRow/rows removed.
- RF_RaidDisk_s{} is re-ordered slightly to fix alignment
padding - the actual data was already OK.
- InstallSpareTable() loses row argument

rf_compat32.c has code for RF_Config_s{} in 32 bit mode, used
by RAIDFRAME_CONFIGURE and RAIDFRAME_GET_INFO32.

rf_compat80.c has code for rf_recon_req{}, RF_RaidDisk_s{} and
RF_DeviceConfig_s{} to handle RAIDFRAME_FAIL_DISK,
RAIDFRAME_GET_COMPONENT_LABEL, RAIDFRAME_CHECK_RECON_STATUS_EXT,
RAIDFRAME_CHECK_PARITYREWRITE_STATUS_EXT,
RAIDFRAME_CHECK_COPYBACK_STATUS_EXT, RAIDFRAME_GET_INFO.

move several of the per-ioctl code blocks into separate functions.

add rf_recon_req_internal{} to replace old usage of global
rf_recon_req{} that had unused void * in the structure, ruining
it's 32/64 bit ABI.

add missing case for RAIDFRAME_GET_INFO50.

adjust raid tests to use the new .conf format, and add a case to
test the old method as well.

raidctl:
deal with lack of 'row' members in a couple of places.
fail request no longer takes row.
handle "START array" sections with just "numCol numSpare", ie
no "numRow" specified. for now, generate old-style configuration
but update raidctl.8 to specify the new style (keeping reference
to the old style.)

note that: RF_ComponentLabel_s::{row,num_rows} and
RF_SingleComponent_s::row are obsolete but not removed yet.

support NAME=<wedge name> syntax for disks and spares

Remove workaround for ancient HTML generation code.

branches: 1.71.8;
Use standard sort order for options.

Add -U to usage.

Access to the SET_LAST_UNIT ioctl.

Bump date for previous. Add whitespace.
Fix some mandoc warnings.

Clarify that what was previously known as -A root is now -A forceroot, not
-A softroot and that -A root can still be used for historical reasons.

branches: 1.67.4;
Add the ability to "softroot" mount (i.e. mount root only when the raid
set contains the boot device), as opposed to "hardroot" (the previous
default which forces the raid to be root no matter what).

Belatedly note root on RAIDframe support for sandpoint.

Note that NetBSD/amd64 can boot of RAID volumes.

branches: 1.64.2;
Remove unexpected newline between Em macro and text.

branches: 1.63.2; 1.63.4; 1.63.8; 1.63.10;
New sentence, new line.

Document the need for zeroing out the first 64 blocks of a replacement
component in a failed RAID set in order to avoid potentially configuring
RAId 1 sets with erroneous values taken from random extent data in the
replacement component partitions.

+ Fatal errors due to uninitialized components are ignored.
for -C. For dillo@

Consistently use "START disks" in examples. Bump date.

Remove trailing whitespace.

Bump date; also fix typo pointed out by snj@.

Finally commit the RAIDframe parity map Summer Of Code project.

Drastically reduces the amount of time spent rewriting parity after an
unclean shutdown by keeping better track of which regions might have had
outstanding writes. Enabled by default; can be disabled on a per-set
basis, or tuned, with the new raidctl(8) commands.

Discussed on tech-kern@ to a general air of approval; exhortations to
commit from mrg@, christos@, and others.

Thanks to Google for their sponsorship, oster@ for mentoring the
project, assorted developers for trying very hard to break it, and
probably more I'm forgetting.

branches: 1.56.2; 1.56.8;
Drop trailing whitespace.

Bump date (before wizd).

Add description of some of the things 'raidctl -u' doesn't do.

We have the notion of an 'absent' disk, so use that in another example.

branches: 1.52.2;
Move TNF licenses to 2 clause form

branches: 1.51.4; 1.51.6;
Fix a minor typo: wd0h was repeated, change second instance to wd1h as is likely intended.

branches: 1.50.2; 1.50.4; 1.50.8;
Talk a bit more about how the size of a RAID set is determined, and
of RF_PROTECTED_SECTORS. Requested by (and with comments from and
thanks to) Christoph (bad@).

Bump date for previous.

Add a few more lines about the 'Parity status:' output of 'raidctl -s'.
This should address PR#29540.

Give Dd the full month names as argument.

Greg Oster just reminded me, I left out "vax".

Sync with raid(4) regarding the architectures that support booting directly
from RAID 1, using language supplied by Anders Dinson in PR bin/29158, with
some additions by myself.

Use Dq instead of quotes; bump date for previous.

Update docs to reflect the support for the word "absent" in the "disks"
section of RAID config files.

Add commas in enumeration; bump date for previous.

Add i386 to the list of systems that support booting from raid1.
RAID(4) also needs changing - if someone knows where the source is....

Bump date for last.

Swapping to RAID 5 is happy now. Remove indications to the contrary.

branches: 1.38.2;
Note in the description of hot-adding components that a component label is
not required for the component. Thanks to David Maxwell for the
documentation suggestion.

Bump date for previous.

Add a couple little warnings about swapping to RAID 5 sets.
(it currently doesn't work.)

.Nm does not need a dummy argument ("") before punctuation or
for correct formatting of the SYNOPSIS any longer.

Remove trailing space.

Catch up on some documentation stuff:
- add a note about adding 'swapoff=YES' to /etc/rc.conf
- add a note about not using raid0[cd] for a filesystem.
- filesystems -> file systems

Use more markup.

New sentence, new line. By Robert Elz with minimal fixes.

Remove duplicate 'to'. From Ryo HAYASAKA in bin/17562.

branches: 1.29.2;
Generate <>& symbolically. I'm avoiding .../dist/... directories for now.

Drop trailing whitespace.

Add a description of how to possibly recover a RAID set in the
event of a multiple disk failure.

Don't abuse -unfilled for -literal; sort sections, sort SEE ALSO, drop
lots of unnecessary .Pps.

Whitespace nits

* add -G, which lists the configuration of the given raid set in the
same configuration format that -c and -C use.
this is useful if you're using autoconfig and you've misplaced the
/etc/raidXXX.conf files
* "filesystem" -> "file system", and other man page cleanups.

Drop arguments of .Os.

- cleanup wording and add additional comments on such things as
"component1" and "raidctl -A yes"
- add a note about how to build a RAID set with a limited number of disks
(thanks to Simon Burge for suggestions)
- improve layout of 'raidctl -i' discussion (thanks to Hubert Feyrer)
- add a (small) section on Performance Tuning

Clarify a few things about parity. Add more documentation about RAID on RAID,
and root on RAID.

Add a .Sq for single quotes, instead of doing it by hand.
Fixes a formatting problem noted by Hubert Feyrer (Thanks Hubert!).

branches: 1.19.2;
Slight wording improvement on the 'raidctl -R' case.

Fix up some single quotes that messed up a line. (Thanks Wiz!)

branches: 1.17.2;
Update for 'options RAID_AUTOCONFIG'.

Add a few more examples, and a summary.

More updates, spelling fixes, etc.

First revamp of the docs. This is still work-in-progress. More to come
shortly.

Document the new autoconfiguration switches, and update the Usage: line.
More doc changes expected soon.

it's -> its

- add '-v' (Verbose) option, which, most notably, provides
a spiffy lukemftp-inspired progress bar for parity re-writing,
reconstructs, copybacks, et al.

- make 'raidctl -P' wait until the parity has been updated before exiting
(it waited previously, but only because the parity re-write ioctl wouldn't
exit until the parity was successfully re-written)

Typos (from OpenBSD)

branches: 1.9.4;
Add two more options to raidctl:
-p check (and return) the status of the parity
-P check the status of the parity, and rebuild if necessary

Addresses PR#7494

Remove blank lines.

Fix minor formatting error.

Fix minor formatting error.

Update for recent changes: component labels, clean bits, adding hot
spares, and rebuilding components in-place. Re-arrange the
letters/options to make more sense (and make better use of the alphabet).

fix typo

Add missing NetBSD RCS ID's! Thanks to SAITOH Masanobu (msaitoh@netbsd.org)
for pointing this out.

Fix tpyo.

Oops. Forgot the man-page for 'raidctl' when the rest of 'raidctl' went in.

raidctl: fix <ctype.h> usage, remove redundant break statements

raidctl: trim trailing whitespace

raidctl: remove unused and undocumented option '-B'

branches: 1.83.2;
fix various typos in comments and log messages.

We no longer need the deprecated copyback functionality now that
incorporating a used spare is automatic.

Copyback has always been an issue, as to do a copyback all IO to
the array had to be suspended, and so was very, very unlikely to
have been used in anything resembling a production system.

Remove a couple of unneeded comments.

Implement command-line configuration of simple RAID sets with raidctl
based on the usage pattern:

raidctl <device> create <level> <component1> <component2> ...

For example,

raidctl raid0 create mirror absent /dev/wd1e

will create a RAID level 1 (mirror) set with an absent first component
and /dev/wd1e as the second component. The resulting RAID device will
be marked as auto-configurable, will have a serial number set (based
on the current time), and parity will be initialized. Reasonable
performance values are automatically used by default for other parameters
normally specified in the configuration file.

Also: Only print out Autoconfig status if being verbose.

Implement hot removal of spares and components. From manu@.

Implement a long desired feature of automatically incorporating
a used spare into the array after a reconstruct.

Given the configuration:
Components:
/dev/wd0e: failed
/dev/wd1e: optimal
/dev/wd2e: optimal
Spares:
/dev/wd3e: spare

Running 'raidctl -F /dev/wd0e raid0' will now result in the
following configuration after a successful rebuild:
Components:
/dev/wd3e: optimal
/dev/wd1e: optimal
/dev/wd2e: optimal
No spares.

Thanks to manu@ for the development of the initial set of changes
which allowed the changes to automatically incorporate a used spare
to come to fruition. Thanks also to manu@ for useful discussions
about and additional testing of these changes.

branches: 1.78.2;
Implement "raidctl -t config-file"

This does the same config file parse that -c/-C do, but only
that (hence no raidframe device is needed, or accepted).

Any syntax errors in the config file will be reported, nothing
else happens.

In the previous (and some earlier) version(s) of raidctl.c
the following comment appeared:

/*
* After NetBSD 9, convert this to not output the numRow's value,
* which is no longer required or ever used.
*/

We are after NetBSD 9 (well after). The change requested in that
comment is made here, and the comment is thus removed.

A couple of places in rf_configure.c where a value for the "rows"
parameter was output in an error message (always simply as the
constant 0) have also been updated (those messages will no longer
include "row 0", which they always said previously). One of them
was also slightly reworded to be clearer what problem it was
experiencing (when it said 'unable to get device file' it meant
it was unable to locate the name for the device in the config file,
not that it was found, and there was some other problem with it).

Reorder the getopts() switch () (slightly) to sort the options. NFCI.

KNF (whitespace & 80 column limits) - NFCI.

This is the first of a series of 5 commits in this
directory, all coming within minutes or now.

Support on-demand re-scanning all devices to look for
autoconfig RAID sets. raidctl now supports looking
for autoconfig RAID sets with a new '-L' flag.

Move case 'l' to be in sorted order. No functional change.

Allow components to be specified by wedge name.

Need MIN in userland. Also some more signed/unsigned clashes.

nspares is now unsigned. Validate and use as positive integer.

Correct printed IOCTL name that was incorrect since rev 1.5.

- add the string length as an explicit parameter to get_time_string()
- remove casts when the same type is used on both sides
- expand hours_buffer[] to fit the range of hours in an 'int'
- add a work around for the sprintf() truncation checker that fails
to detect that 'minutes' and 'seconds' have a small range

branches: 1.67.2;
Follow the ioctl arg changes of RAIDFRAME_GET_COMPONENT_LABEL and
RAIDFRAME_CHECK_*_STATUS_EXT.

This should fix strange raidctl -s outputs reported in
current-users ML.

branches: 1.66.2;
implement 32-bit compat support for raidframe.

convert several raidframe ioctls to be bitsize idempotent so that
they work the same in 32 and 64 bit worlds, allowing netbsd32 to
configure and query raid properly. remove useless 'row' in a few
places. add COMPAT_80 and put the old ioctls there.

raidframeio.h:
RAIDFRAME_TEST_ACC
- remove, unused
RAIDFRAME_GET_COMPONENT_LABEL
- convert to label not pointer to label
RAIDFRAME_CHECK_RECON_STATUS_EXT
RAIDFRAME_CHECK_PARITYREWRITE_STATUS_EXT
RAIDFRAME_CHECK_COPYBACK_STATUS_EXT
- convert to progress info not pointer to info
RAIDFRAME_GET_INFO
- version entirely.
raidframevar.h:
- rf_recon_req{} has row, flags and raidPtr removed (they're
not a useful part of this interface.)
- RF_Config_s{} and RF_DeviceConfig_s{} have numRow/rows removed.
- RF_RaidDisk_s{} is re-ordered slightly to fix alignment
padding - the actual data was already OK.
- InstallSpareTable() loses row argument

rf_compat32.c has code for RF_Config_s{} in 32 bit mode, used
by RAIDFRAME_CONFIGURE and RAIDFRAME_GET_INFO32.

rf_compat80.c has code for rf_recon_req{}, RF_RaidDisk_s{} and
RF_DeviceConfig_s{} to handle RAIDFRAME_FAIL_DISK,
RAIDFRAME_GET_COMPONENT_LABEL, RAIDFRAME_CHECK_RECON_STATUS_EXT,
RAIDFRAME_CHECK_PARITYREWRITE_STATUS_EXT,
RAIDFRAME_CHECK_COPYBACK_STATUS_EXT, RAIDFRAME_GET_INFO.

move several of the per-ioctl code blocks into separate functions.

add rf_recon_req_internal{} to replace old usage of global
rf_recon_req{} that had unused void * in the structure, ruining
it's 32/64 bit ABI.

add missing case for RAIDFRAME_GET_INFO50.

adjust raid tests to use the new .conf format, and add a case to
test the old method as well.

raidctl:
deal with lack of 'row' members in a couple of places.
fail request no longer takes row.
handle "START array" sections with just "numCol numSpare", ie
no "numRow" specified. for now, generate old-style configuration
but update raidctl.8 to specify the new style (keeping reference
to the old style.)

note that: RF_ComponentLabel_s::{row,num_rows} and
RF_SingleComponent_s::row are obsolete but not removed yet.

Use standard sort order for options.

Add -U to usage.

Access to the SET_LAST_UNIT ioctl.

Rename argument of rf_output_devname() from devname to name to avoid a
warning about shadowing a global symbol when compiled by buildrump.sh.
Discussed with mrg.

convert "component*" into "absent" for "START disks" part of the
output from "raidctl -G". now this actually works when fed back
into raidctl -[cC].

Compare correct length string for force option to -A

&stars[offset] -> stars+offset. It's shorter!

Coincidentally, the change also works around a gcc 5.1 bug which causes
a segmentation fault when trying to compile the longer version (guess
the compiler got exhausted, or something).
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66345

use strtou

Better sanity check numbers given to raidctl(8)

Replace atoi(3) by strtol(3), and check that numbers are valid,
positive, and in int32_t range. The previous lack of check could
silently lead to the same serial being set to all RAID volumes
for instance because given numbers were bigger than INT_MAX. The
consequence is in an awful mess when RAIDframe would mix volumes...

branches: 1.57.4;
Add the ability to "softroot" mount (i.e. mount root only when the raid
set contains the boot device), as opposed to "hardroot" (the previous
default which forces the raid to be root no matter what).

fix unused variable warnings.

branches: 1.55.2; 1.55.4; 1.55.8;
PR/45456: Tetsuya Isaki: Don't mix stdio and write.

print the serial number as an unsigned number.

Use __dead

Define accessors for number of blocks and partition size in the
component label and use them where appropriate. Disscussed on tech-kern.

Exterminate a bug I created in 2009.

branches: 1.50.2;
Use RUMPPRG.

ok Greg Oster

Update RUMP_ACTION to use rumpclient.

The server must of course have some disks configured. Let's say
we have this simple server with disks as a few sparse host files:

main()
{
rump_init();
rump_pub_etfs_register("/disk1", "./disk1.img", RUMP_ETFS_BLK);
rump_pub_etfs_register("/disk2", "./disk2.img", RUMP_ETFS_BLK);
rump_pub_etfs_register("/disk3", "./disk3.img", RUMP_ETFS_BLK);
rump_pub_etfs_register("/disk4", "./disk4.img", RUMP_ETFS_BLK);
pause();
}

And we run the server:

mainbus0 (root)
Kernelized RAIDframe activated
/disk1: hostpath ./disk1.img (97 GB)
/disk2: hostpath ./disk2.img (97 GB)
/disk3: hostpath ./disk3.img (97 GB)
/disk4: hostpath ./disk4.img (97 GB)

We can then configure the raid against the server:

> ./raidctl -c theraid.conf raid0

And lo, we have evidence of a level1 raid in the server dmesg:

raid0: RAID Level 1
raid0: Components: /disk1 /disk2 /disk3 /disk4
raid0: Total Sectors: 409599744 (199999 MB)

yea, i initialized it already in a previous run:

> ./raidctl -S raid0
Reconstruction is 100% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.

Give a more polite message for `raidctl -m` on a non-parity RAID set.

fix sign-compare issue

Exclude parity map regions that don't actually exist from the dirty region count
in `raidctl -m`. Makes for less confusing output during `raidctl -i`.

use warn/err appropriately.

error message: \n\n -> \n

Slight change to the wording of the parity map info: the parity is
"marked clean" after however much inactivity; it is *actually* clean
as soon as the component disks all do their thing (on the order of ms,
usually), just the same as before.

The bikeshed is now less of a taupe and more of an ecru.

Finally commit the RAIDframe parity map Summer Of Code project.

Drastically reduces the amount of time spent rewriting parity after an
unclean shutdown by keeping better track of which regions might have had
outstanding writes. Enabled by default; can be disabled on a per-set
basis, or tuned, with the new raidctl(8) commands.

Discussed on tech-kern@ to a general air of approval; exhortations to
commit from mrg@, christos@, and others.

Thanks to Google for their sponsorship, oster@ for mentoring the
project, assorted developers for trying very hard to break it, and
probably more I'm forgetting.

Support RUMP_ACTION, i.e. compile-time switch to make this execute
rump syscalls to configure raidframe in a rump kernel.

Use correct format to print the "numBlocks" element in a RAIDframe
component label. raidctl(8) should now print the correct number of
blocks for RAID sets larger than 1TB.

Patch supplied by Bernhard Moellemann in PR bin/40479.

branches: 1.39.4;
Remove clause 3 and 4 from TNF licenses

branches: 1.38.4; 1.38.16; 1.38.20; 1.38.22;
appease gcc -Wuninitialized

Cleanup the meter code a bit -- a bunch of stuff wasn't even being used.
Fix a few overflow issues. (Thanks to beefy and pooka, among others)

branches: 1.36.2;
Kill __P(), use ANSI function declarations; WARNS=3.

As suggested by Paul Ripke, have raidctl use raw devices by default.

Don't compare argc to NULL.

Add NetBSD RCS Ids.

correct questionable use of strncpy().

Cleanup and update usage().

Open the raid device read-only for the commands that don't need
write access. With the default set up this allows users in group
operator to query the status of the raid devices.

Fixes PR admin/15840.

rf_configure.c
- remove dependence on stuff in sys/dev/raidframe
- now rely on <dev/raidframe/*>
- bring in some needed prototypes for local functions
- nuke RF_ASSERT's.
- drag in some needed RF_ERRORMSG's

raidctl.c
- rely on <dev/raidframe/*>
- welcome to the New Way of doing RAIDframe #includes.

(No functional changes.)

Since this is the only place it's used, nuke RF_DEV2RAIDID and
replace it with DISKUNIT.

* add -G, which lists the configuration of the given raid set in the
same configuration format that -c and -C use.
this is useful if you're using autoconfig and you've misplaced the
/etc/raidXXX.conf files
* "filesystem" -> "file system", and other man page cleanups.

convert to use getprogname()

- use opendisk(3) instead of homegrown solution
- separate items in -s output with commas; much easier to read
- add maxOutstanding (aka queue len) to -s output
- sort #includes (per knf)

It is impolite to poll in a tight loop. (Thanks to Christos for noting
the problem.)

branches: 1.23.2;
In certain cases (e.g. where it makes sense) read/display the
component label for spare disks too.

Make sure we handle the 'no progress' case properly.
Thanks to Manuel Bouyer for noting the problem (and supplying a patch!).

Obtaining component labels from spares is not supported yet, but add
some code that makes some progress in that direction.

Cleanup ETA calculation stuff.

Have raidctl use the fine-grained progress info. ETA calculation stuff
needs to be cleaned up.

branches: 1.18.2;
Build with WARNS=2.

ioctl(2) commands are u_long, so do_meter() must take the same.

Don't declare 'extern opt*' getopt variables.

When displaying component labels, indicate whether or not the set is
marked as being the one that should be used for the root partition.

Reorganize a few things. Make the general status ('-s') a bit more verbose
by including component labels, and less verbose by removing some other stuff.

Document the new autoconfiguration switches, and update the Usage: line.
More doc changes expected soon.

- support for autoconfig setup and / on RAID setup
- new parts are still work-in-progress (no docs yet either)

Nuke a debugging printf that I forgot to remove before last commit.

- add '-v' (Verbose) option, which, most notably, provides
a spiffy lukemftp-inspired progress bar for parity re-writing,
reconstructs, copybacks, et al.

- make 'raidctl -P' wait until the parity has been updated before exiting
(it waited previously, but only because the parity re-write ioctl wouldn't
exit until the parity was successfully re-written)

rf_MakeConfig succeeds only if it returns 0, not if it returns
values 0 or greater.

Display the parity status as part of the status ('-s') display.

Add two more options to raidctl:
-p check (and return) the status of the parity
-P check the status of the parity, and rebuild if necessary

Addresses PR#7494

Update for recent changes: component labels, clean bits, adding hot
spares, and rebuilding components in-place. Re-arrange the
letters/options to make more sense (and make better use of the alphabet).

Clean things up a bit. Teach raidctl a little about component labels
and hot-adding of spares. New code is there, but not enabled (yet).

Add missing NetBSD RCS ID's! Thanks to SAITOH Masanobu (msaitoh@netbsd.org)
for pointing this out.

Take care of a few minor things due to the major RAIDframe cleanup.

Again, no functionality changes.

recover from errno.h changes

RAIDframe, version 1.1, from the Parallel Data Laboratory at
Carnegie Mellon University. Full RAID implementation, including
levels 0, 1, 4, 5, 6, parity logging, and a few other goodies.
Ported to NetBSD by Greg Oster.

raidctl is our userland configuration tool for RAIDframe.

need <sys/stat.h> for fstat()

branches: 1.2.26;
Exterminate a bug I created in 2009.

branches: 1.1.2;
Use RUMPPRG.

ok Greg Oster

Exterminate a bug I created in 2009.

branches: 1.1.2;
Use RUMPPRG.

ok Greg Oster

PR bin/56917

If getfsspecname() fails that will usually mean that a NAME=wedge or
ROOT.x partition is unabailable. raidframe specified unavailable
partitions as "absent" so in this case, pass "absent" rather than the
unaltered NAME= or ROOT.x string, which the kernel has no clue what
do do with, and doesn't configure the raid at all.

Fix some config file parsing.

First, and what got me started on this set of cleanups, the queue
length in the "queue" section (START queue) is limited to what will
fit in a char without losing accuracy (I tried setting it to 200,
rather than the more common (universal?) 100 and found that the
value configured into the array was -56 instead.

Why the value needs to be passed through a char variable I have no
idea (it is an int in the filesystem raidframe headers) - but that's
the way it is done, and changing it would be an ABI change I believe
(and so need versioning to alter) and that isn't worth it for this
(or not now, IMO).

Instead check that the value in the char is the same value as was
read from the config file, and complain if not. Those of you with
unsigned chars will be able to have queue lengths up to 255, the
rest of us are limited to 127.

While looking at that, I noticed some code that obviously fails to
understand that scanf("%s") will never return a string containing
spaces, and proceeded to attempt to remove trailing spaces from the
result ... amusingly, after having used the result for its intended
purpose (non existent trailing spaces unremoved), after which that
buffer was never used again. That code is now gone (but for now,
just #if 0'd rather than actually deleted - it should be cleaned up
sometime).

Then I saw some other issues with how the config was parsed - a
simple (unbounded) scanf("%s") into a buffer, which hypothetically
might not be large enough (not a security issue really, raidctl has
no special privs, and it isn't likely that root could easily be
tricked into running it on a bogus config file - or not without
looking first anyway, and a huge long string would rather stand
out). Bound the string length to something reasonable, and
assert() that the buffer is big enough to contain it.

Lastly, in the event of one particular detected error in the
config file, the code would write a warning, but then just go
ahead and use the bad data (or nothing perhaps) anyway - a
failure of logic flow (unlikely to have ever happened, everyone
seems to simply copy the sample config from the man page, and
make minor adjustments as needed).

If any of these changes make any difference to anyone (except
me with my attempt to make longer queues - for no particularly
well thought out reason), I'd be very surprised.

In the previous (and some earlier) version(s) of raidctl.c
the following comment appeared:

/*
* After NetBSD 9, convert this to not output the numRow's value,
* which is no longer required or ever used.
*/

We are after NetBSD 9 (well after). The change requested in that
comment is made here, and the comment is thus removed.

A couple of places in rf_configure.c where a value for the "rows"
parameter was output in an error message (always simply as the
constant 0) have also been updated (those messages will no longer
include "row 0", which they always said previously). One of them
was also slightly reworded to be clearer what problem it was
experiencing (when it said 'unable to get device file' it meant
it was unable to locate the name for the device in the config file,
not that it was found, and there was some other problem with it).

avoid trying to printf() a NULL as %s. fixes likely bug.

implement 32-bit compat support for raidframe.

convert several raidframe ioctls to be bitsize idempotent so that
they work the same in 32 and 64 bit worlds, allowing netbsd32 to
configure and query raid properly. remove useless 'row' in a few
places. add COMPAT_80 and put the old ioctls there.

raidframeio.h:
RAIDFRAME_TEST_ACC
- remove, unused
RAIDFRAME_GET_COMPONENT_LABEL
- convert to label not pointer to label
RAIDFRAME_CHECK_RECON_STATUS_EXT
RAIDFRAME_CHECK_PARITYREWRITE_STATUS_EXT
RAIDFRAME_CHECK_COPYBACK_STATUS_EXT
- convert to progress info not pointer to info
RAIDFRAME_GET_INFO
- version entirely.
raidframevar.h:
- rf_recon_req{} has row, flags and raidPtr removed (they're
not a useful part of this interface.)
- RF_Config_s{} and RF_DeviceConfig_s{} have numRow/rows removed.
- RF_RaidDisk_s{} is re-ordered slightly to fix alignment
padding - the actual data was already OK.
- InstallSpareTable() loses row argument

rf_compat32.c has code for RF_Config_s{} in 32 bit mode, used
by RAIDFRAME_CONFIGURE and RAIDFRAME_GET_INFO32.

rf_compat80.c has code for rf_recon_req{}, RF_RaidDisk_s{} and
RF_DeviceConfig_s{} to handle RAIDFRAME_FAIL_DISK,
RAIDFRAME_GET_COMPONENT_LABEL, RAIDFRAME_CHECK_RECON_STATUS_EXT,
RAIDFRAME_CHECK_PARITYREWRITE_STATUS_EXT,
RAIDFRAME_CHECK_COPYBACK_STATUS_EXT, RAIDFRAME_GET_INFO.

move several of the per-ioctl code blocks into separate functions.

add rf_recon_req_internal{} to replace old usage of global
rf_recon_req{} that had unused void * in the structure, ruining
it's 32/64 bit ABI.

add missing case for RAIDFRAME_GET_INFO50.

adjust raid tests to use the new .conf format, and add a case to
test the old method as well.

raidctl:
deal with lack of 'row' members in a couple of places.
fail request no longer takes row.
handle "START array" sections with just "numCol numSpare", ie
no "numRow" specified. for now, generate old-style configuration
but update raidctl.8 to specify the new style (keeping reference
to the old style.)

note that: RF_ComponentLabel_s::{row,num_rows} and
RF_SingleComponent_s::row are obsolete but not removed yet.

Several more cleanups:
1. Don't force use of "for" when "while" works better.
2. No need to check c != '\0' when we also check (c == ' ' || c == '\t')
3. Use the size of the buffer we're using, rather than a different one
(not really a concern, they're the same size)
4. Don't use fscanf() to read file data, use fgets() & sscanf().
5. After using a pointer as a char *, validate alignment before switching
to int * (can only fail if kernel #define gets set stupidly) Or #6...
6. Validate sparemap file name isn't too long for assigned space.
7. recognise that strlen() returns size_t - don't shove it into an int.
8. On out of mem, be more clear which allocation failed in warning msg.

ATF tests all pass. But I don't think they use sparemap files.

Avoid needless pointer calisthenics: &foo[0] -> foo

With char bug[SIZE] using sizeof(bug[0]) is kind of boring, use
sizeof(bug) instead...

White space and comment formatting. NFC.

stop using magic constants
wrap long lines
use warn{,x}
make static
knf

support NAME=<wedge name> syntax for disks and spares

branches: 1.26.8;
PR/50921: David Binderman: Fix memory leak

use warn/err appropriately.

fix sign-compare issue

branches: 1.23.30;
close fp in several error paths, closing Coverity CIDs 1690, 1692.

"go" by go.

Kill __P(), use ANSI function declarations; WARNS=3.

Cleanup a bit by nuking RF_Malloc macro, and add in some error
checking for the case where we can't allocate memory. (Not like
anyone uses this part of the code anyway.)

Strip off trailing \n's and spaces from configuration file lines.
Makes RAIDframe more forgiving if you accidentally add a space after
things like device names. Fixed at long last after a reminder over
the weekend.

How about we use the 'len' parameter to rf_get_next_nonblank_line()
instead of assuming all buffers are 256 bytes.

use bouded string ops

Add NetBSD RCS Ids.

sscanf overrun

rf_configure.c
- remove dependence on stuff in sys/dev/raidframe
- now rely on <dev/raidframe/*>
- bring in some needed prototypes for local functions
- nuke RF_ASSERT's.
- drag in some needed RF_ERRORMSG's

raidctl.c
- rely on <dev/raidframe/*>
- welcome to the New Way of doing RAIDframe #includes.

(No functional changes.)

remove redundant prototypes.

Move parts of a RAID type mapping table from rf_layout.c to rf_configure.c.
This means we don't need to compile or link with rf_layout.c here.

Check return value of fgets. Approved by Greg Oster.

Format string cleanups by Bill Sommerfeld.

branches: 1.10.4;
Apply indent.

Build with WARNS=2.

No need to include rf_sys.h.

Move two functions from rf_strutils.c into rf_configure.c, removing
the need for rf_strutils.c.

branches: 1.6.2;
Nuke unused code.

Add missing NetBSD RCS ID's! Thanks to SAITOH Masanobu (msaitoh@netbsd.org)
for pointing this out.

Take care of a few minor things due to the major RAIDframe cleanup.

Again, no functionality changes.

recover from errno.h changes

fix compile errors on the alpha.

RAIDframe, version 1.1, from the Parallel Data Laboratory at
Carnegie Mellon University. Full RAID implementation, including
levels 0, 1, 4, 5, 6, parity logging, and a few other goodies.
Ported to NetBSD by Greg Oster.

raidctl is our userland configuration tool for RAIDframe.

stop using magic constants
wrap long lines
use warn{,x}
make static
knf

branches: 1.1.92;
Add a pruned-down version of rf_configure.h. First step at disentangling
raidctl files from the hairy .h file mess.

Never used, and certainly no longer needed.

RAIDframe, version 1.1, from the Parallel Data Laboratory at
Carnegie Mellon University. Full RAID implementation, including
levels 0, 1, 4, 5, 6, parity logging, and a few other goodies.
Ported to NetBSD by Greg Oster.

raidctl is our userland configuration tool for RAIDframe.

Never used, and certainly no longer needed.

RAIDframe, version 1.1, from the Parallel Data Laboratory at
Carnegie Mellon University. Full RAID implementation, including
levels 0, 1, 4, 5, 6, parity logging, and a few other goodies.
Ported to NetBSD by Greg Oster.

raidctl is our userland configuration tool for RAIDframe.