Fix vndcompress restart failure fallback when input is a pipe.

Defer seeking the *input* image, or winding it forward, until we are
certain we all ready in the cloop2 output, because when the input
image is a pipe, we don't get a chance to seek back to the beginning
and start from the top instead of restarting.

If restart does fail, don't try to seek the input image back to the
beginning unless we had already tried to seek or wind it forward.

Add some automatic tests for this and related cases.

XXX pullup to netbsd-7, netbsd-6

branches: 1.13.4;
Change vndcompress to use a default window size of 512.

For vnduncompress on nonseekable input, the window size is as large
as it needs to be by default, as before. Not clear that this is the
right choice -- by default vnduncompress on nonseekable input will
just use unbounded memory unsolicited.

Rename block size option from `-s' to `-b'.

Makes more sense and makes it consistent with other utilities such as
pax and pigz. This vndcompress has never gone out in a release, so
changing the name of the option shouldn't cause too many problems...

Fix up ulimited vndcompress tests and explain what's up with them.

Add some simple automatic tests for window sizes.

Add comment explaining why piperestart.cl2part is allowed to fail.

Add WARNS=5.

Use write-to-temporary/rename-to-permanent pattern in Makefile.

Abstract handling of the cloop2 offset table.

Preparation for converting it to use a fixed-size window.

Move vndcompress utilities to utils.c.

Remove redundant WARNS=5.

Rewrite vndcompress to support SIGINFO and restart after interrupt.

Make it generally more robust in the process.

No objection (or comment) on tech-userlevel.

ok christos

branches: 1.2.6; 1.2.8; 1.2.12;
Enable WARNS=4 by default for usr.bin, except for:
awk bdes checknr compile_et error gss hxtool kgetcred kinit
klist ldd less lex locale login m4 man menuc mk_cmds
mklocale msgc openssl rpcgen rpcinfo sdiff spell ssh
string2key telnet tn3270 verify_krb5_conf xlint

branches: 1.1.1;
Initial revision

Clarify compile-time and run-time arithmetic safety assertions.

This is an experiment with a handful of macros for writing the
checks, most of which are compile-time:

MUL_OK(t, a, b) Does a*b avoid overflow in type t?
ADD_OK(t, a, b) Does a + b avoid overflow in type t?
TOOMANY(t, x, b, m) Are there more than m b-element blocks in x in type t?
(I.e., does ceiling(x/b) > m?)

Addenda that might make sense but are not needed here:

MUL(t, a, b, &p) Set p = a*b and return 0, or return ERANGE if overflow.
ADD(t, a, b, &s) Set s = a+b and return 0, or return ERANGE if overflow.

Example:

uint32_t a = ..., b = ..., y = ..., z = ..., x, w;

/* input validation */
error = MUL(size_t, a, b, &x);
if (error)
fail;
if (TOOMANY(uint32_t, x, BLKSIZ, MAX_NBLK))
fail;
y = HOWMANY(x, BLKSIZ);
if (z > Z_MAX)
fail;
...
/* internal computation */
__CTASSERT(MUL_OK(uint32_t, Z_MAX, MAX_NBLK));
w = z*y;

Obvious shortcomings:

1. Nothing checks your ctassert matches your subsequent arithmetic.
(Maybe we could have BOUNDED_MUL(t, x, xmax, y, ymax) with a
ctassert inside.)

2. Nothing flows the bounds needed by the arithmetic you use back
into candidate definitions of X_MAX/Y_MAX.

But at least the reviewer's job is only to make sure that (a) the
MUL_OK matches the *, and (b) the bounds in the assertion match the
bounds on the inputs -- in particular, the reviewer need not derive
the bounds from the context, only confirm they are supported by the
paths to it.

This is not meant to be a general-purpose proof assistant, or even a
special-purpose one like gfverif <http://gfverif.cryptojedi.org/>.
Rather, it is an experiment in adding a modicum of compile-time
verification with a simple C API change.

This also is not intended to serve as trapping arithmetic on
overflow. The goal here is to enable writing the program with
explicit checks on input and compile-time annotations on computation
to gain confident that overflow won't happen in the computation.

Emphasize that MAX_WINDOW_SIZE is bounded by the maximum uint32_t.

Since we store window sizes in uint32_t, the maximum had better fit
in uint32_t!

branches: 1.6.4; 1.6.8; 1.6.12; 1.6.16;
Change vndcompress to use a default window size of 512.

For vnduncompress on nonseekable input, the window size is as large
as it needs to be by default, as before. Not clear that this is the
right choice -- by default vnduncompress on nonseekable input will
just use unbounded memory unsolicited.

Rename block size option from `-s' to `-b'.

Makes more sense and makes it consistent with other utilities such as
pax and pigz. This vndcompress has never gone out in a release, so
changing the name of the option shouldn't cause too many problems...

Add option -w to vnd(un)compress to specify the window size.

Add some leading zero digits to the flags. Cosmetic change only.

Implement machinery for fixed-size windows into the offset table.

branches: 1.1.2;
Rewrite vndcompress to support SIGINFO and restart after interrupt.

Make it generally more robust in the process.

No objection (or comment) on tech-userlevel.

ok christos

branches: 1.3.4; 1.3.8;
Rename block size option from `-s' to `-b'.

Makes more sense and makes it consistent with other utilities such as
pax and pigz. This vndcompress has never gone out in a release, so
changing the name of the option shouldn't cause too many problems...

Add option -w to vnd(un)compress to specify the window size.

branches: 1.1.2;
Rewrite vndcompress to support SIGINFO and restart after interrupt.

Make it generally more robust in the process.

No objection (or comment) on tech-userlevel.

ok christos

Clarify compile-time and run-time arithmetic safety assertions.

This is an experiment with a handful of macros for writing the
checks, most of which are compile-time:

MUL_OK(t, a, b) Does a*b avoid overflow in type t?
ADD_OK(t, a, b) Does a + b avoid overflow in type t?
TOOMANY(t, x, b, m) Are there more than m b-element blocks in x in type t?
(I.e., does ceiling(x/b) > m?)

Addenda that might make sense but are not needed here:

MUL(t, a, b, &p) Set p = a*b and return 0, or return ERANGE if overflow.
ADD(t, a, b, &s) Set s = a+b and return 0, or return ERANGE if overflow.

Example:

uint32_t a = ..., b = ..., y = ..., z = ..., x, w;

/* input validation */
error = MUL(size_t, a, b, &x);
if (error)
fail;
if (TOOMANY(uint32_t, x, BLKSIZ, MAX_NBLK))
fail;
y = HOWMANY(x, BLKSIZ);
if (z > Z_MAX)
fail;
...
/* internal computation */
__CTASSERT(MUL_OK(uint32_t, Z_MAX, MAX_NBLK));
w = z*y;

Obvious shortcomings:

1. Nothing checks your ctassert matches your subsequent arithmetic.
(Maybe we could have BOUNDED_MUL(t, x, xmax, y, ymax) with a
ctassert inside.)

2. Nothing flows the bounds needed by the arithmetic you use back
into candidate definitions of X_MAX/Y_MAX.

But at least the reviewer's job is only to make sure that (a) the
MUL_OK matches the *, and (b) the bounds in the assertion match the
bounds on the inputs -- in particular, the reviewer need not derive
the bounds from the context, only confirm they are supported by the
paths to it.

This is not meant to be a general-purpose proof assistant, or even a
special-purpose one like gfverif <http://gfverif.cryptojedi.org/>.
Rather, it is an experiment in adding a modicum of compile-time
verification with a simple C API change.

This also is not intended to serve as trapping arithmetic on
overflow. The goal here is to enable writing the program with
explicit checks on input and compile-time annotations on computation
to gain confident that overflow won't happen in the computation.

Justify the last unjustified assertion here.

Sprinkle a few more assertions to help along the way.

(Actually, it was justified; I just hadn't made explicit the relation
to the value of fdpos that all two callers specify.)

branches: 1.13.4; 1.13.8; 1.13.10; 1.13.14; 1.13.18;
Get SIZE_MAX and OFF_MAX straight...

Factor out an offtab_compute_window_position routine.

Fix some more integer overflow/truncation issues.

Arithmetic in C is hard. Let's go shopping!

Mark offtab_bug[x] as dead.

Fix $NetBSD$ tag.

Simplify vndcompress offtab_compute_window_size.

Fix typo in comment.

Remove silly comment in offtab_reset_write.

Split guard in offtab_write_window into offtab_maybe_write_window.

Seek if necessary at end of offtab_reset_read.

Fixes vnduncompress with a small window, and makes offtab_reset_read
symmetric with offtab_reset_write.

Judicious (and justified) casts to avoid signed/unsigned comparisons.

Implement machinery for fixed-size windows into the offset table.

Abstract handling of the cloop2 offset table.

Preparation for converting it to use a fixed-size window.

Justify the last unjustified assertion here.

Sprinkle a few more assertions to help along the way.

(Actually, it was justified; I just hadn't made explicit the relation
to the value of fdpos that all two callers specify.)

branches: 1.2.4; 1.2.8; 1.2.10; 1.2.14; 1.2.18;
Implement machinery for fixed-size windows into the offset table.

Abstract handling of the cloop2 offset table.

Preparation for converting it to use a fixed-size window.

Simplify.

branches: 1.5.2; 1.5.4;
__diagused, not __unused -- used in an assert.

branches: 1.4.4; 1.4.8; 1.4.10;
Judicious (and justified) casts to avoid signed/unsigned comparisons.

Implement machinery for fixed-size windows into the offset table.

Move block_signals/restore_sigmask to utils.c

Move vndcompress utilities to utils.c.

branches: 1.3.4; 1.3.8; 1.3.10;
Implement machinery for fixed-size windows into the offset table.

Move block_signals/restore_sigmask to utils.c

Move vndcompress utilities to utils.c.

Since vnconfig(8) was renamed (many years ago), update some cross-refs
in vndcompress(1) man page.

branches: 1.14.26;
Bump date on vndcompress(1) man page.

Fix vndcompress man page to reflect default window size.

Document the new vndcompress -w option and nuke BUGS section.

Perhaps vndcompress and vnduncompress ought by default to choose a
limited window size (say, 8192 entries, i.e. 64k bytes, the default
MAXPHYS), and vnduncompress should fall back to an unlimited window
only if the input is nonseekable.

Reflect rename of `-s' to `-b' in the vndcompress man page.

Add -l option to synopsis for vndcompress(1) man page.

Sort. Add EXIT STATUS section.

Rewrite vndcompress to support SIGINFO and restart after interrupt.

Make it generally more robust in the process.

No objection (or comment) on tech-userlevel.

ok christos

branches: 1.7.2;
Improve wording, fix typo, bump date.
From Bug Hunting.

branches: 1.6.34; 1.6.36;
Make example consistent (fix typo in file names).

Fix typo ("program").

Fix typo reported by Mathieu in PR 32846.

Fix old name in .Dt tag (CLCONFIG -> VNDCOMPRESS)

branches: 1.2.2;
Various improvements.

branches: 1.1.1;
Initial revision

Clarify compile-time and run-time arithmetic safety assertions.

This is an experiment with a handful of macros for writing the
checks, most of which are compile-time:

MUL_OK(t, a, b) Does a*b avoid overflow in type t?
ADD_OK(t, a, b) Does a + b avoid overflow in type t?
TOOMANY(t, x, b, m) Are there more than m b-element blocks in x in type t?
(I.e., does ceiling(x/b) > m?)

Addenda that might make sense but are not needed here:

MUL(t, a, b, &p) Set p = a*b and return 0, or return ERANGE if overflow.
ADD(t, a, b, &s) Set s = a+b and return 0, or return ERANGE if overflow.

Example:

uint32_t a = ..., b = ..., y = ..., z = ..., x, w;

/* input validation */
error = MUL(size_t, a, b, &x);
if (error)
fail;
if (TOOMANY(uint32_t, x, BLKSIZ, MAX_NBLK))
fail;
y = HOWMANY(x, BLKSIZ);
if (z > Z_MAX)
fail;
...
/* internal computation */
__CTASSERT(MUL_OK(uint32_t, Z_MAX, MAX_NBLK));
w = z*y;

Obvious shortcomings:

1. Nothing checks your ctassert matches your subsequent arithmetic.
(Maybe we could have BOUNDED_MUL(t, x, xmax, y, ymax) with a
ctassert inside.)

2. Nothing flows the bounds needed by the arithmetic you use back
into candidate definitions of X_MAX/Y_MAX.

But at least the reviewer's job is only to make sure that (a) the
MUL_OK matches the *, and (b) the bounds in the assertion match the
bounds on the inputs -- in particular, the reviewer need not derive
the bounds from the context, only confirm they are supported by the
paths to it.

This is not meant to be a general-purpose proof assistant, or even a
special-purpose one like gfverif <http://gfverif.cryptojedi.org/>.
Rather, it is an experiment in adding a modicum of compile-time
verification with a simple C API change.

This also is not intended to serve as trapping arithmetic on
overflow. The goal here is to enable writing the program with
explicit checks on input and compile-time annotations on computation
to gain confident that overflow won't happen in the computation.

Omit needless XXX comment.

Justify the last unjustified assertion here.

Sprinkle a few more assertions to help along the way.

(Actually, it was justified; I just hadn't made explicit the relation
to the value of fdpos that all two callers specify.)

branches: 1.26.2;
need <sys/stat.h>

branches: 1.25.2;
Fix vndcompress restart failure fallback when input is a pipe.

Defer seeking the *input* image, or winding it forward, until we are
certain we all ready in the cloop2 output, because when the input
image is a pipe, we don't get a chance to seek back to the beginning
and start from the top instead of restarting.

If restart does fail, don't try to seek the input image back to the
beginning unless we had already tried to seek or wind it forward.

Add some automatic tests for this and related cases.

XXX pullup to netbsd-7, netbsd-6

branches: 1.24.4;
Fix some more integer overflow/truncation issues.

Arithmetic in C is hard. Let's go shopping!

CID 1164169: integer overflow

Change vndcompress to use a default window size of 512.

For vnduncompress on nonseekable input, the window size is as large
as it needs to be by default, as before. Not clear that this is the
right choice -- by default vnduncompress on nonseekable input will
just use unbounded memory unsolicited.

Rename block size option from `-s' to `-b'.

Makes more sense and makes it consistent with other utilities such as
pax and pigz. This vndcompress has never gone out in a release, so
changing the name of the option shouldn't cause too many problems...

Window size is now an option; remove XXX comment to the contrary.

Add option -w to vnd(un)compress to specify the window size.

Implement machinery for fixed-size windows into the offset table.

Write offsets in hexadecimal, not decimal.

Move block_signals/restore_sigmask to utils.c

Abstract handling of the cloop2 offset table.

Preparation for converting it to use a fixed-size window.

Move vndcompress utilities to utils.c.

Make partial read/write error messages more consistent in vndcompress.

Add __printflike to vsnprintf_ss.

'unsigned long' prints with %lu, not %zu.

Fix sign-compare in compress_blocks.

Not sure why my builds didn't reveal this one -- they revealed
several others during development.

__printflike for vwarnx_ss, __dead for err_ss and errx_ss.

Rewrite vndcompress to support SIGINFO and restart after interrupt.

Make it generally more robust in the process.

No objection (or comment) on tech-userlevel.

ok christos

branches: 1.7.2; 1.7.4; 1.7.8;
Use static and __dead

Fix another sign-compare issue

fix sign-compare issues

branches: 1.4.12;
Take a small step toward making vndcompress into a host tool:
remove #include <arpa/inet.h> and use bswap32() instead of ntohl().

branches: 1.3.2; 1.3.14;
The return type from getopt(3) is int, not char. Besides, char is
not guaranteed to be signed, so comparison with -1 will cause a
warning (turned error) for some of our ports (e.g. our arm ports).
Fix this by making the 'ch' variable an int instead of a char.

Sync usage with man page.

branches: 1.1.1;
Initial revision

Rewrite vndcompress to support SIGINFO and restart after interrupt.

Make it generally more robust in the process.

No objection (or comment) on tech-userlevel.

ok christos

branches: 1.3.2; 1.3.4; 1.3.8;
Use static and __dead

Take a small step toward making vndcompress into a host tool:
remove #include <arpa/inet.h> and use bswap32() instead of ntohl().

branches: 1.1.1;
Initial revision

Clarify compile-time and run-time arithmetic safety assertions.

This is an experiment with a handful of macros for writing the
checks, most of which are compile-time:

MUL_OK(t, a, b) Does a*b avoid overflow in type t?
ADD_OK(t, a, b) Does a + b avoid overflow in type t?
TOOMANY(t, x, b, m) Are there more than m b-element blocks in x in type t?
(I.e., does ceiling(x/b) > m?)

Addenda that might make sense but are not needed here:

MUL(t, a, b, &p) Set p = a*b and return 0, or return ERANGE if overflow.
ADD(t, a, b, &s) Set s = a+b and return 0, or return ERANGE if overflow.

Example:

uint32_t a = ..., b = ..., y = ..., z = ..., x, w;

/* input validation */
error = MUL(size_t, a, b, &x);
if (error)
fail;
if (TOOMANY(uint32_t, x, BLKSIZ, MAX_NBLK))
fail;
y = HOWMANY(x, BLKSIZ);
if (z > Z_MAX)
fail;
...
/* internal computation */
__CTASSERT(MUL_OK(uint32_t, Z_MAX, MAX_NBLK));
w = z*y;

Obvious shortcomings:

1. Nothing checks your ctassert matches your subsequent arithmetic.
(Maybe we could have BOUNDED_MUL(t, x, xmax, y, ymax) with a
ctassert inside.)

2. Nothing flows the bounds needed by the arithmetic you use back
into candidate definitions of X_MAX/Y_MAX.

But at least the reviewer's job is only to make sure that (a) the
MUL_OK matches the *, and (b) the bounds in the assertion match the
bounds on the inputs -- in particular, the reviewer need not derive
the bounds from the context, only confirm they are supported by the
paths to it.

This is not meant to be a general-purpose proof assistant, or even a
special-purpose one like gfverif <http://gfverif.cryptojedi.org/>.
Rather, it is an experiment in adding a modicum of compile-time
verification with a simple C API change.

This also is not intended to serve as trapping arithmetic on
overflow. The goal here is to enable writing the program with
explicit checks on input and compile-time annotations on computation
to gain confident that overflow won't happen in the computation.

Omit needless XXX comment.

Justify the last unjustified assertion here.

Sprinkle a few more assertions to help along the way.

(Actually, it was justified; I just hadn't made explicit the relation
to the value of fdpos that all two callers specify.)

branches: 1.11.4; 1.11.8; 1.11.12; 1.11.16;
Fix some more integer overflow/truncation issues.

Arithmetic in C is hard. Let's go shopping!

Change vndcompress to use a default window size of 512.

For vnduncompress on nonseekable input, the window size is as large
as it needs to be by default, as before. Not clear that this is the
right choice -- by default vnduncompress on nonseekable input will
just use unbounded memory unsolicited.

Move err1 & errx1 to the end of vnduncompress.c; add __printflike.

Add option -w to vnd(un)compress to specify the window size.

Implement machinery for fixed-size windows into the offset table.

Write offsets in hexadecimal, not decimal.

Abstract handling of the cloop2 offset table.

Preparation for converting it to use a fixed-size window.

Use read_block instead of read in vnduncompress.

Fail if malloc can't allocate offset table.

branches: 1.2.2;
Make partial read/write error messages more consistent in vndcompress.

Rewrite vndcompress to support SIGINFO and restart after interrupt.

Make it generally more robust in the process.

No objection (or comment) on tech-userlevel.

ok christos