merge ktrace-lwp.

branches: 1.2.4;
comment typo

Install cryptodev.h into /usr/include/crypto/ for userspace applications.

opencrypto: Switch from legacy rijndael API to new aes API.

While here, apply various rijndael->aes renames, reduce the size
of aesxcbc_ctx by 480 bytes, and convert some malloc->kmem.

Leave in the symbol enc_xform_rijndael128 for now, though, so this
doesn't break any kernel ABI.

From Alexander Nasonov:
- Make constants static: Shrinks code and data size.
- Avoid overflow in limit calculation.
- Use uint8_t instead of u_char to match types
While here:
- Remove unnecessary casts
- s/u_int8_t/uint8_t/g

branches: 1.1.2; 1.1.6; 1.1.18; 1.1.36; 1.1.40;
copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

opencrypto: Switch from legacy rijndael API to new aes API.

While here, apply various rijndael->aes renames, reduce the size
of aesxcbc_ctx by 480 bytes, and convert some malloc->kmem.

Leave in the symbol enc_xform_rijndael128 for now, though, so this
doesn't break any kernel ABI.

branches: 1.1.2; 1.1.6;
copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

Use the Blowfish in crypto/blowfish, which has hooks for using tuned
assembly for the transform.

In Blowfish_initstate(), make the initstate static, otherwise the
compiler will emit code to first copy it onto the stack before
copying it into the destination context structure. With this change,
it will only be copied once.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

Use the Blowfish in crypto/blowfish, which has hooks for using tuned
assembly for the transform.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

Move the opencrypto CAST-128 implementation to crypto/cast128, removing
the old one. Rename the functions/structures from cast_* to cast128_*.
Adapt the KAME IPsec to use the new CAST-128 code, which has a simpler
API and smaller footprint.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

Move the opencrypto CAST-128 implementation to crypto/cast128, removing
the old one. Rename the functions/structures from cast_* to cast128_*.
Adapt the KAME IPsec to use the new CAST-128 code, which has a simpler
API and smaller footprint.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

Move the opencrypto CAST-128 implementation to crypto/cast128, removing
the old one. Rename the functions/structures from cast_* to cast128_*.
Adapt the KAME IPsec to use the new CAST-128 code, which has a simpler
API and smaller footprint.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

branches: 1.2.2; 1.2.6;
Remove left-over #includes

Split the module glue out from the rest of opencrypto/ocryptodev to
make rump happy.

Rump doesn't have compat modules (the compat code is included in the
relevant librump*.so), so there's no module compat_50 listed in
link_set_modules, and thus ocryptodev's MODULE(...) can't "require"
it.

This fixes the problem of "built-in module compat_50 not found" when
starting up rump_allserver (or rump_server with -l rumpdev_opencrypto).

XXX This does not resolve the long-standing "crypto: unable to
XXX register devsw, error 17" message noted at line 78 of
XXX sys/rump/dev/lib/libopencrypto/opencrypto_component.c

Rename min/max -> uimin/uimax for better honesty.

These functions are defined on unsigned int. The generic name
min/max should not silently truncate to 32 bits on 64-bit systems.
This is purely a name change -- no functional change intended.

HOWEVER! Some subsystems have

#define min(a, b) ((a) < (b) ? (a) : (b))
#define max(a, b) ((a) > (b) ? (a) : (b))

even though our standard name for that is MIN/MAX. Although these
may invite multiple evaluation bugs, these do _not_ cause integer
truncation.

To avoid `fixing' these cases, I first changed the name in libkern,
and then compile-tested every file where min/max occurred in order to
confirm that it failed -- and thus confirm that nothing shadowed
min/max -- before changing it.

I have left a handful of bootloaders that are too annoying to
compile-test, and some dead code:

cobalt ews4800mips hp300 hppa ia64 luna68k vax
acorn32/if_ie.c (not included in any kernels)
macppc/if_gm.c (superseded by gem(4))

It should be easy to fix the fallout once identified -- this way of
doing things fails safe, and the goal here, after all, is to _avoid_
silent integer truncations, not introduce them.

Maybe one day we can reintroduce min/max as type-generic things that
never silently truncate. But we should avoid doing that for a while,
so that existing code has a chance to be detected by the compiler for
conversion to uimin/uimax without changing the semantics until we can
properly audit it all. (Who knows, maybe in some cases integer
truncation is actually intended!)

branches: 1.8.54; 1.8.56;
Don't panic, just truncate, if the iov is too short in a COPYBACK.
This case can be triggered from userland cryptodev if the buffer
for decompressed data is too small.
(It would look cleaner if the lengths would be passed explicitely
everywhere, but that would thwart the abstraction done by COPYDATA/COPYBACK
which allows to treat mbufs and iovs the same way.)

branches: 1.7.4; 1.7.6; 1.7.8;
ANSIfy another 1261 function definitions.
The only ones left in sys are beyond by sed script!
(or in sys/dist or sys/external)
Mostly they have function pointer parameters.

branches: 1.6.10; 1.6.18; 1.6.24;
This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

branches: 1.5.16; 1.5.22;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.4.26;
merge ktrace-lwp.

branches: 1.3.4;
nuke trailing whitespace

branches: 1.2.4; 1.2.10; 1.2.12;
make the panic messages match reality

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

opencrypto(9): Fix missing initialization in error branch.

Reported-by: syzbot+8c519140cac567be1ee1@syzkaller.appspotmail.com

opencrypto: Assert session id is valid in crypto_freesession.

This gives us the opportunity to detect usage mistakes like
use-after-free.

Exception: Continue to silently ignore sid=0.

opencrypto: crypto_dispatch never fails now. Make it return void.

Same with crypto_kdispatch.

opencrypto: Assert driver process routine returns 0 or ERESTART.

No other errors are allowed -- other errors must be transmitted by
crypto_done. All drivers in tree (sun8i_crypto, glxsb, via_padlock,
mvcesa, mvxpsec, hifn, qat, ubsec, cryptosoft) have been audited for
this.

opencrypto: Rip out EAGAIN logic when unregistering crypto drivers.

I'm pretty sure this never worked reliably based on code inspection,
and it's unlikely to have ever been tested because it only applies
when unregistering a driver -- but we have no crypto drivers for
removable devices, so it would only apply if we went out of our way
to trigger detach with drvctl.

Instead, just make the operation fail with ENODEV, and remove all the
callback logic to resubmit the request on EAGAIN. (Maybe this should
be ENXIO, but crypto_kdispatch already does ENODEV.)

opencrypto: Assert nonnull callback up front in crypto_dispatch.

Same with crypto_kdispatch.

Convert some dead branches downstream to assertions too.

opencrypto: Make crypto_freesession return void.

No callers use the return value. It is not sensible to allow this to
fail.

opencrypto: Make freesession callback return void.

No functional change intended: all drivers already return zero
unconditionally.

opencrypto: Assert crp_desc and crp_buf are nonnull.

- crypto_getreq ensures crp_desc is nonnull.
- Caller is responsible for setting crp_buf.

opencrypto: Assert num>0 in crypto_getreq, num=1 in crypto_kgetreq.

- For crypto_getreq this makes downstream reasoning easier: on
success, crp_desc is guaranteed to be nonnull.

- For crypto_kgetreq, this was already assumed, just silently
ignored and not checked by anything.

opencrypto: Nix CRYPTO_F_DONE.

Nothing uses it any more.

opencrypto: Make sid=0 always invalid, but OK to free.

Previously, crypto_newsession could sometimes return 0 as the
driver-specific part of the session id, and 0 as the hid, for sid=0.
But netipsec assumes that it is always safe to free sid=0 from
zero-initialized memory even if crypto_newsession has never
succeeded. So it was up to every driver in tree to gracefully handle
sid=0, if it happened to get assigned hid=0. And, as long as the
freesession callback was expected to just return an error code when
given a bogus session id, that worked out fine...because nothing ever
used the error code.

That was a terrible fragile system that should never have been
invented. Instead, let's just ensure that valid session ids are
nonzero, and make crypto_freesession with sid=0 be a no-op.

opencrypto: Assert !cpu_intr_p() on dispatch and invoke.

These should only ever have been potentially called from hard
interrupt context by CRYPTO_F_CBIMM callbacks (CBIMM = call back
immediately). CRYPTO_F_CBIMM is no more, so there is no more need to
allow this case of call from hard interrupt context.

opencrypto: Nix CRYPTO_F_USER, CRYPTO_F_CBIMM, CRYPTO_F_CBIFSYNC.

CRYPTO_F_USER is no longer needed. It was introduced in 2008 by
darran@ in crypto.c 1.30, cryptodev.c 1.45 in an attempt to avoid
double-free between the issuing thread and asynchronous callback.
But the `fix' didn't work. In 2017, knakahara@ fixed it properly in
cryptodev.c 1.87 by distinguishing `the crypto operation has
completed' (CRYPTO_F_DONE) from `the callback is done touching the
crp object' (CRYPTO_F_DQRETQ, now renamed to CRYPTODEV_F_RET).

CRYPTO_F_CBIMM formerly served to invoke the callback synchronously
from the driver's interrupt completion routine, to reduce contention
on what was once a single cryptoret thread. Now, there is a per-CPU
queue and softint for much cheaper processing, so there is less
motivation for this in the first place. So let's remove the
complicated logic. This means the callbacks never run in hard
interrupt context, which means we don't need to worry about recursion
into crypto_dispatch in hard interrupt context.

opencrypto: Factor setting CRYPTO_F_DONE out of branches.

This had been done in 1.30 when the locking was different. No need
any more. No functional change intended.

fix typo in CRK_ALGORITHM_MIN definition to match CRK_ALGORITHM_MAX one.
while here fix few typos in comments.

fix typos in asymmetry, asymmetric(al), symmetrical.

Revert previous change to use SYSCTL_SETUP since it breaks on macppc.

For some reason, the crypto module fails to link, and this results in
opencrypto sysctl failures.

Should resolve PR kern/55154

Use the module subsystem's ability to process SYSCTL_SETUP() entries to
automate installation of sysctl nodes.

Note that there are still a number of device and pseudo-device modules
that create entries tied to individual device units, rather than to the
module itself. These are not changed.

softint_disestablish does xc_barrier(0) for us already.

Switch opencrypto to percpu_create.

Can't sleep for allocation in percpu_foreach.

branches: 1.110.2;
xc_barrier - convenience function to xc_broadcast() a nop.

Make the intent more clear and also avoid a bunch of (xcfunc_t)nullop
casts that gcc 8 -Wcast-function-type is not happy about.

in many device attach paths, allocate memory with KM_SLEEP instead of KM_NOSLEEP
and remove code to handle failures that can no longer happen.

relinguish our lock while we are autoloading.

Try to load swcrypto if we we did not find any software drivers.

branches: 1.106.2;
Remove duplicate ;

branches: 1.105.2;
Fix PR kern/52910. Reported and implemented a patch by Sevan Janiyan, thanks.

Committed debugging logs by mistake, sorry. Revert cryoto.c:r.1.103 and ip6_flow.c:r.1.37.

Fix PR kern/52910. Reported and implemented a patch by Sevan Janiyan, thanks.

use PR_NOWAIT.

fix opencrypto(9) part of PR kern/52515

percpu data use pointers to TAILQ instead of TAILQ itself.

use pool_cache(9) instead of pool(9) as they can be called concurrently.

use kmem_alloc KPI instead of malloc KPI.

pack crypto_drivers variables to struct and add cacheline_aligned qualifier.

refactor: remove glue macros for FreeBSD code.

divide crp_ret_{,k}q by CPU to avoid reordering.

update locking note later.

make crp_{,k}q percpu to scale crypto_dispatch().

update locking note later.

fix typo

fix panic when using ecryption devices attached earlier than ipi_sysinit().

pointed out and tested by martin@n.o, thanks.

branches: 1.92.2;
make cryptoret() context softint to balance dequeuing crypto_ret_q with enqueuing it.

simplify mutex_enter/exit(crypto_q_mtx), and fix missing exit.

Reduce crypto_ret_q_mtx lock regions.

crypto.c does not access the members of crp when the crp is in crp_q or
crp_ret_q. Furthermore, crp_q and crp_ret_q are protected by each mutex,
so the members of crp is not shared. That means crp_flags is not required
mutex in crypto.c.

support multiple encryption drivers (port from FreeBSD).

refactor crypto_newsession() like FreeBSD.

must release cap->cc_lock before calling cap->cc_newsession() because of spinlock.

Put back crypto_checkdriver(); use it when we need to make sure that we
get back a cryptocap that has been initialized.

- acquire lock
- use c99 loop indexes
- initialize featp

apply the same fix as crypto.c:r1.83 for crypto_dispatch to crypto_kdispatch.

avoid crp_q reordering as hardware interrupts.

crypto_{,k}invoke() can be called with holding crp_q_mtx now.

restructure locks(2/2): crypto_q_mtx can be adaptive now.

restructure locks(1/2): make relation between lock and data explicit.

+ crypto_drv_mtx protects
- whole crypto_drivers
+ crypto_drivers[i].cc_lock (new) protects
- crypto_drivers[i] itself
- member of crypto_drivers[i]
+ crypto_q_mtx protects
- crp_q
- crp_kq
+ crypto_ret_q_mtx protects
- crp_ret_q
- crp_ret_kq
- crypto_exit_flag

I will add locking note later.

fix reading crp_q without holding crypto_q_mtx

use crypto_checkdriver_uninit() when it may touch uninitialized crypto_drivers.

branches: 1.78.2;
fix: crypto_unregister didn't work.

strictly use crypto_checkdriver(i) instead of using crypto_drivers[i] directly.

add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

make the default values of q_maxlen build parameters.

implement crypto_ret_q limitation. original code is implemented by hsuenaga@IIJ.

add crypto_ret_{,k}q length sysctl entries and statistics codes.

initialize sysctl in the same way regardless of module or not.

decrease the priority of batch crp even if there are more than one batch crp.

refactor cryptointr(), no functional changes.

fix cryptointr() can process unexpected request.

If migrate crp is linked after batch crp, "submit" is already set to
the batch crp. So, cryptointr() can process the batch crp instead of
the target migrate crp.

refactor crypto_kdispatch() in a similar way as crypto_dispatch().

refactor crypto_dispatch (3/3): do "blocked" operation previously

refactor crypto_dispatch (2/3): divide migrate operation

refactor crypto_dispatch (1/3): divide batch operation

Processing batch operation at first, crypto_q_mtx's lock region is reduced
because crp does not require crypto_q_mtx.

opencrypto: cleanup debug messages.

fix: crypto_drivers[hid].cc_process() could be called even if it was null.

If a crypto driver is unregistered before calling cyrptointr(),
the crypto_drivers[hid].cc_process is null in spite of the hid
is less than crypto_drivers_num.

reffered to FreeBSD code.

unify implementation of crypto_unregister() and crypto_unregister_all()

refactor crypto_unregister()

- separate logic to crypto_unregister_locked()
- refactor cryptocap cleanup condition

use macro instead of immediate value

add some assertion. tested by ATF net/ipsec/ and crypto/.

branches: 1.58.2;
When crypto request is deferred processing, opencrypto should not return error.

When the crypto device blocks a crypto request, opnecrypto enqueues the request,
that is, the request is just deferred and no error occurs.
The pseudo error causes problems, e.g. ipsec can send wrong ICMP host unreach.

contributed by hsuenaga@IIJ, thanks.

separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

reduce crypto_q_mtx lock regions.

refactor crypto_unblock(). No functional change.

the processing said "ghastly hacks" is unnecessary now.

fix build failure ALL

fix processes accessing /dev/crypto stall when over three processes run with a hardware encryption driver

The process has stalled at cv_wait(&crp->crp_cv) because cryptodev_cb()
is not called as cryptoret() kthread keep waiting at cv_wait(&cryptoret_cv).
Previous opencrypto implementation assumes the thread from cryptodev.c
does all processing in the same context, so skips enqueueing and sending
cryptoret_cv. However, the context can be switched, e.g. when we use
a hardware encryption driver.

And add debug messages.

fix missing mutex_exit() in crypto_destroy().

crypto_destroy() is called only in error case or unloading module.

fix: remove unmatched mutex_exit/enter. must be forgetting to remove at crypto.c:r1.41

make some locks and condvar static

branches: 1.48.2; 1.48.4;
KNF. Remove extra spaces. No functional change.

fix the build

Re-work the module init and destroy code to allow it to be unloaded and
then reloaded.

Should fix PR kern/49842

branches: 1.45.6;
Ensure that the top level sysctl nodes (kern, vfs, net, ...) exist before
the sysctl link sets are processed, and remove redundancy.

Shaves >13kB off of an amd64 GENERIC, not to mention >1k duplicate
lines of code.

Clean-up module initialization

When using modular opencrypto, make sure the sysctl variables get
created.

Modularize the opencrypto components and link to the build

branches: 1.41.2; 1.41.12; 1.41.16;
-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.40.2;
split the "crypto_mtx" spinlock into 3: one spinlock each for
the incoming and outgoing request queues (which can be dealt with
by hardware accelerators) and an adaptive lock for "all the rest"
(mostly driver configuration, but also some unrelated stuff in
cryptodev.c which should be revisited)
The latter one seems to be uneeded at many places, but for now I've
done simple replacements only, except minor fixes (where
softint_schedule() was called without the lock held)

As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

make the crypto softint MPSAFE -- I see no reason not to do it, and
it didn't cause trouble for me. (It doesn't give additional parallelization
as things look now, just saves the acquisition of KERNEL_LOCK.)

PR/44470: Dr. Wolfgang Stukenbrock: opencrypto kernel implementation may pass
outdated argument to worker

branches: 1.36.2; 1.36.4;
Keep condvar wmesg within 8 char limit.

Consistently use a single CRYPTO_SESID2HID-like macro.
Improve CRYPTO_DEBUG printing a bit:
print pointers with %p
print unsigned with %u rather than %d
use CRYPTO_SESID2LID instead of just casting to uint32_t

branches: 1.34.2; 1.34.4;
Remove extra whitespace added by a stupid tool.
XXX: more in src/sys/arch

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

bcopy -> memcpy

bzero -> memset

branches: 1.30.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.29.2; 1.29.4;
In crypto_freereq, destroy explicitly the condvar

ok by tls@

branches: 1.28.2; 1.28.6;
Remove clause 3 and 4 from TNF licenses

branches: 1.27.2; 1.27.4;
Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

branches: 1.26.6;
cryptoret: avoid a sleep/wakeup race.

The kthread is now MPSAFE.

Some locking fixes (double-release mutex in softintr wakeup case, which I
hadn't tested) and an uninitialized field in cse which Darran Hunt
found. Some more debugging printfs.

Turn on MPSAFE for the kthread. We're not sure it's safe for the softint
yet. Gives a little performance kick for swcrypto with many requests on
MP systems.

Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

branches: 1.21.4;
Use the softint API.

branches: 1.20.6; 1.20.8; 1.20.10;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

branches: 1.19.2;
Pass an ipl argument to pool_init/POOL_INIT to be used when initializing
the pool's lock.

branches: 1.18.2; 1.18.6;
crypto_init does not call crypto_init0 only once, because the marker
created with ONCE_DECL() is local. This results in reinitializing
the driver list when crypto_get_driverid() (and leaks memory). Fix
this by making the marker static.

Fixes PR/35412.

Ack freza@.

branches: 1.17.2;
__unused removal on arguments; approved by core.

avoid yet another GCC uninitialised warning error that only comes
up with -O3.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.14.6; 1.14.8;
merge FreeBSD timecounters from branch simonb-timecounters
- struct timeval time is gone
time.tv_sec -> time_second
- struct timeval mono_time is gone
mono_time.tv_sec -> time_uptime
- access to time via
{get,}{micro,nano,bin}time()
get* versions are fast but less precise
- support NTP nanokernel implementation (NTP API 4)
- further reading:
Timecounter Paper: http://phk.freebsd.dk/pubs/timecounter.pdf
NTP Nanokernel: http://www.eecis.udel.edu/~mills/ntp/html/kern.html

branches: 1.13.6;
Add the 3 missing sysctl we are supposed to export.

branches: 1.12.2; 1.12.4; 1.12.6;
- tweak RUN_ONCE api to allow init_func returns an error.
- physio: handle failure of workqueue_create.

branches: 1.11.2;
- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.10.4; 1.10.10;
nuke trailing whitespace

branches: 1.9.2; 1.9.6; 1.9.8;
Change control knob for /dev/crypto to only allow requests which
will be hardware-accelerated. Avoids copyin()/copyout() overhead and
spending exceessive tie inside the kernel.

Pullup after: 24 hours, or confirmation by Jason Thorpe that this is the
consensus tech-kern agreed upon last summer.

branches: 1.8.2;
Split opencrypto configuration into an attribute, usable by inkernel
clients, and a pseudo-device for userspace access.

The attribute is named `opencrypto'. The pseudo-device is renamed to
"crypto", which has a dependency on "opencrypto". The sys/conf/majors
entry and pseudo-device attach entrypoint are updated to match the
new pseudo-device name.

Fast IPsec (sys/netipsec/files.ipsec) now lists a dependency on the
"opencrypto" attribute. Drivers for crypto accelerators (ubsec,
hifn775x) also pull in opencrypto, as providers of opencrypto transforms.

Fix typo.

Clean up userlevel access to software kernel transforms, in preparation
for using /dev/crypto for OpenSSL:

1. Add comments explaining crypto_devallowsoft, explaining the
OpenBSD-style three-way logic actully implemented in crypto_newsession().

2. Pass crypto_devallowsoft as the final argument to crypto_newsession(),
instead of a constant 0 value.

3. Set the default value of crypto_devallowsoft to 1, to allow
/dev/crypto access only for hardware-supported transforms.

Items 1-3 may be revised to match the FreeBSD two-way logic, if the
consensus is that there's no point to forcing software transforms.
But as a first step, let the description match what the code actually does.

GC unused variables usercrypto, userasmcrypto, cryptodevallowsoft from
cryptodev.c, in favour of variables crypto_usercrypto, crypto_userasmcrypto,
crypto_devallowsoft, which are used as well as defined in crypto.c.

Fix a genuine uninitialised variable.

Implement nanouptime as a function to avoid 'dereferencing type-punned pointer'.
No token after #endif

Remove bogus include which snuck into previous commit (from comparison
to Quentin Garnier's suggested patch).

Move the initialization of the crypto framework from the userland
pseudo-device to init_main(), so the framework is ready for
registration requests at autoconfiguration time.

Thanks to Quentin Garnier for confirming the change was required, and
for testing a similar fix.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

crypto_mbuf.c is unused; remove it.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

opencrypto: Fix typo that somehow snuck past our resident typo-hunter.

crypto_userasymcrypto (as in asymmetric), not crypto_userasmcrypto
(as in assembly).

branches: 1.125.10;
fix misspellings of 'available' and nearby typos

opencrypto: Prune dead code now that crypto_dispatch never fails.

opencrypto: crypto_dispatch never fails now. Make it return void.

Same with crypto_kdispatch.

opencrypto: Rip out EAGAIN logic when unregistering crypto drivers.

I'm pretty sure this never worked reliably based on code inspection,
and it's unlikely to have ever been tested because it only applies
when unregistering a driver -- but we have no crypto drivers for
removable devices, so it would only apply if we went out of our way
to trigger detach with drvctl.

Instead, just make the operation fail with ENODEV, and remove all the
callback logic to resubmit the request on EAGAIN. (Maybe this should
be ENXIO, but crypto_kdispatch already does ENODEV.)

crypto(4): Nix dead code now that crypto_freesession never fails.

opencrypto: Make crypto_freesession return void.

No callers use the return value. It is not sensible to allow this to
fail.

crypto(4): crypto_freesession should never fail here.

It can only fail if we pass it an invalid sid, which the logic to
maintain the user sessions should not do. So kassert error=0 here.

crypto(4): Refuse crypto operations with nothing in them earlier.

This way we avoid passing 0 to crypto_getreq -- makes it easier to
reason about everything downstream.

opencrypto: Make crp_callback, krp_callback return void.

Nothing uses the return values inside opencrypto, so let's stop
making users return them.

crypto(4): Fix possible use-after-free in race around detach.

This is extremely unlikely because I don't think we have any drivers
for removable crypto decelerators^Waccelerators...but if we were to
sprout one, and someone ran crypto_dispatch concurrently with
crypto_unregister, cryptodev_cb/mcb would enter with crp->crp_etype =
EAGAIN and with CRYPTO_F_DONE set in crp->crp_flags. In this case,
cryptodev_cb/mcb would issue crypto_dispatch but -- since nothing
clears CRYPTO_F_DONE -- it would _also_ consider the request done and
notify the ioctl thread of that.

With this change, we return early if crypto_dispatch succeeds. No
need to consult CRYPTO_F_DONE: if the callback is invoked it's done,
and if we try to redispatch it on EAGAIN but crypto_dispatch fails,
it's done. (Soon we'll get rid of the possibility of crypto_dispatch
failing synchronously, but not just yet.)

XXX This path could really use some testing!

crypto(4): Fix set-but-unused variable warning.

This deliberately ignores the error code returned by crypto_dispatch,
but that error code is fundamentally incoherent and the issue will be
mooted by subsequent changes to make it return void and always pass
the error through the callback, as well as subsequent changes to rip
out the EAGAIN logic anyway.

crypto(4): Don't signal the condvar for multi-operation completion.

The condvar may be destroyed by the time we got here, and nothing
waits on it anyway -- instead the caller is expected to select/poll
for completion in userland.

The bug was already here, but the recent change to eliminate
CRYPTO_F_CBIMM made it happen more often by causing the callback to
_always_ be run asynchronously instead of sometimes being run
synchronously.

opencrypto: Nix CRYPTO_F_USER, CRYPTO_F_CBIMM, CRYPTO_F_CBIFSYNC.

CRYPTO_F_USER is no longer needed. It was introduced in 2008 by
darran@ in crypto.c 1.30, cryptodev.c 1.45 in an attempt to avoid
double-free between the issuing thread and asynchronous callback.
But the `fix' didn't work. In 2017, knakahara@ fixed it properly in
cryptodev.c 1.87 by distinguishing `the crypto operation has
completed' (CRYPTO_F_DONE) from `the callback is done touching the
crp object' (CRYPTO_F_DQRETQ, now renamed to CRYPTODEV_F_RET).

CRYPTO_F_CBIMM formerly served to invoke the callback synchronously
from the driver's interrupt completion routine, to reduce contention
on what was once a single cryptoret thread. Now, there is a per-CPU
queue and softint for much cheaper processing, so there is less
motivation for this in the first place. So let's remove the
complicated logic. This means the callbacks never run in hard
interrupt context, which means we don't need to worry about recursion
into crypto_dispatch in hard interrupt context.

crypto(4): Simplify error test in cryptodev_op.

No functional change intended.

crypto(4): Narrow scope of cryptodev_mtx to cover wait.

No functional change intended -- this only removes an unnecessary
lock/unlock cycle in the error case.

crypto(4): Nix long-dead code and comments.

crypto(4): Use IPL_NONE, not IPL_NET, for /dev/crypto pools.

These are used (pool_get/put) only from thread context, never from
interrupt or even soft interrupt context.

opencrypto(9): Omit needless casts around callbacks.

Just declare the right types to begin with. No functional change
intended.

For device modules that provide both auto-config and /dev/xxx
interfaces, make sure that initialization and destruction
follow the proper sequence. This is triggered by the recent
changes to the devsw stuff; per riastradh@ the required call
sequence is:

devsw_attach()
config_init_component() or config_cf*_attach()
...
config_fini_component() or config_cf*_detach()
devsw_detach()

While here, add a few missing calls to some of the detach
routines.

Testing of these changes has been limited to:
1. compile without build break
2. no related test failures from atf
3. modload/modunload work as well as
before.

No functional device testing done, since I don't have any
of these devices. Let me know of any damage I might cause
here!

XXX Some of the modules affected by this commit are already
XXX broken; see kern/56772. This commit does not break
any additional modules (as far as I know).

Rename enc_xform_rijndael128 -> enc_xform_aes.

Update netipsec dependency.

slightly change and fix the semantics of pool_set*wat(), pool_sethardlimit()
and pool_prime() (and their pool_cache_* counterparts):

- the pool_set*wat() APIs are supposed to specify thresholds for the count of
free items in the pool before pool pages are automatically allocated or freed
during pool_get() / pool_put(), whereas pool_sethardlimit() and pool_prime()
are supposed to specify minimum and maximum numbers of total items
in the pool (both free and allocated). these were somewhat conflated
in the existing code, so separate them as they were intended.

- change pool_prime() to take an absolute number of items to preallocate
rather than an increment over whatever was done before, and wait for
any memory allocations to succeed. since pool_prime() can no longer fail
after this, change its return value to void and adjust all callers.

- pool_setlowat() is documented as not immediately attempting to allocate
any memory, but it was changed some time ago to immediately try to allocate
up to the lowat level, so just fix the manpage to describe the current
behaviour.

- add a pool_cache_prime() to complete the API set.

branches: 1.104.4;
If we get an error from devsw_attach(), don't destroy the error value,
since we need to return it to our caller. While we're here, improve
the value of the debug message by actually printing the error value.

Initialize the session variable to an impossible session to prevent compiler
warnings.

branches: 1.102.2;
crypto(4): accept CRYPTO_SHA2_384_HMAC and CRYPTO_SHA2_512_HMAC.

don't always panic when modunload crypto (int the pool destroy code, because
the pools are busy). XXX: this is still racy; we need to prevent creating
more sessions while destroying.

Rename the MODULE_*_HOOK() macros to MODULE_HOOK_*() as briefly
discussed on irc.

NFCI intended.

Ride the earlier kernel bump - it;s getting crowded.

Merge the [pgoyette-compat] branch

branches: 1.98.2; 1.98.4;
Typos.

add fo_name so we can identify the fileops in a simple way.

check results of pool_prime.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

sanitize in CIOCNCRYPTM and initialize comp_alg in CIOCNGSESSION

sanitize count used for kmem_alloc size.

Hmm, who uses CIOCNGSESSION, CIOCNFSESSION, CIOCNCRYPTM or CIOCNFKEYM?

branches: 1.92.2;
rename crypto_mtx to cryptodev_mtx

It is used by cryptodev.c and ocryptodev.c only.

add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

opencrypto: cleanup debug messages.

branches: 1.89.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

the processing said "ghastly hacks" is unnecessary now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

fix processes accessing /dev/crypto stall when over three processes run with a hardware encryption driver

The process has stalled at cv_wait(&crp->crp_cv) because cryptodev_cb()
is not called as cryptoret() kthread keep waiting at cv_wait(&cryptoret_cv).
Previous opencrypto implementation assumes the thread from cryptodev.c
does all processing in the same context, so skips enqueueing and sending
cryptoret_cv. However, the context can be switched, e.g. when we use
a hardware encryption driver.

And add debug messages.

branches: 1.85.2; 1.85.4;
KNF. Remove extra spaces. No functional change.

include "ioconf.h" to get the 'void <driver>attach(int count);' prototype.

Trivial printf format changes and typo fix

branches: 1.82.2;
Return ENOSPC instead of ENOMEM when there is no room in the buffer to
store results. ENOMEM in this subsystem means we cannot allocate more
requests or internal buffers for xforms.

Try not to use f_data, use f_fcrypt to get a correctly typed pointer.

At least crypto_mtx needs initialisation here. Spotted during PR/49065
investigation.

Add d_discard to all struct cdevsw instances I could find.

All have been set to "nodiscard"; some should get a real implementation.

branches: 1.78.2;
Change (mostly mechanically) every cdevsw/bdevsw I can find to use
designated initializers.

I have not built every extant kernel so I have probably broken at
least one build; however I've also found and fixed some wrong
cdevsw/bdevsw entries so even if so I think we come out ahead.

Undo previous - it still needs a lot more work.

For now, we'll use the hand-crafted cf* structures and directly
call all the config routines.

Replace home-grown config with standardized calls to
config_{init,fini}_component()

As requested by mrg@, since there is still a small window during which
the in-module ref-counting can fail, completely disable auto-unload.

Implement in-module ref-counting, and do not allow auto-unload if there
are existing references.

Note that manual unloading is not prevented.

OK christos@

XXX Also note that there is still a small window where the ref-count can
XXX be decremented, and then the process/thread preempted. If auto-unload
XXX happens before that thread can return from the module's code, bad
XXX things (tm) could happen.

knf: Blank line even if no variable declarations.

bail out unloading for now

When crypto(4) is built-in, crypto_modcmd() doesn't need to handle all
the auto-config stuff.

While here, ensure that we depend on opencrypto.

Modularize the opencrypto components and link to the build

Fix return value of cryptodev_msessionfin.

branches: 1.68.2; 1.68.12; 1.68.16;
Fix memset usage.

-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.66.2;
allow testing of GCM/GMAC code from userland

fix building of a linked list if multiple algorithms are requested
in a session -- this just didn't work

catch some corner cases of user input

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

-remove references to crypto/arc4/arc4.* -- the code isn't used
anywhere afaics
(The confusion comes probably from use of arc4random() at various places,
but this lives in libkern and doesn't share code with the former.)
-g/c non-implementation of arc4 encryption in swcrypto(4)
-remove special casing of ARC4 in crypto(4) -- the point is that it
doesn't use an IV, and this fact is made explicit by the new "ivsize"
property of xforms

If symmetric encryption is done from userland crypto(4) and no IV
is specified, the kernel gets one from the random generator. Make sure it
is copied out to the user, otherwise the result is quite useless.

being here, export camellia-cbc through crypto(4) to allow userland tests

add an AES-CTR xform, from OpenBSD

-in the descriptor for encryption xforms, split the "blocksize" field
into "blocksize" and "IV size"
-add an "reinit" function pointer which, if set, means that the xform
does its IV handling itself and doesn't want the default CBC handling
by the framework (poor name, but left that way to avoid unecessary
differences)
This syncs with Open/FreeBSD, purpose is to allow non-CBC transforms.
Refer to ivsize instead of blocksize where appropriate.
(At this point, blocksize and ivsize are identical.)

split the "crypto_mtx" spinlock into 3: one spinlock each for
the incoming and outgoing request queues (which can be dealt with
by hardware accelerators) and an adaptive lock for "all the rest"
(mostly driver configuration, but also some unrelated stuff in
cryptodev.c which should be revisited)
The latter one seems to be uneeded at many places, but for now I've
done simple replacements only, except minor fixes (where
softint_schedule() was called without the lock held)

As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

make the compatibility code conditional on COMPAT_50

more "const"

branches: 1.53.2; 1.53.4;
Consistently use a single CRYPTO_SESID2HID-like macro.
Improve CRYPTO_DEBUG printing a bit:
print pointers with %p
print unsigned with %u rather than %d
use CRYPTO_SESID2LID instead of just casting to uint32_t

branches: 1.52.2; 1.52.4;
Add missing "break" for CRYPTO_CAST_CBC, and some assorted comment fixes.
openssl(1) checks for CAST (and others) on ~every startup.

If a multithreaded app closes an fd while another thread is blocked in
read/write/accept, then the expectation is that the blocked thread will
exit and the close complete.
Since only one fd is affected, but many fd can refer to the same file,
the close code can only request the fs code unblock with ERESTART.
Fixed for pipes and sockets, ERESTART will only be generated after such
a close - so there should be no change for other programs.
Also rename fo_abort() to fo_restart() (this used to be fo_drain()).
Fixes PR/26567

Rename fo_drain() to fo_abort(), 'drain' is used to mean 'wait for output
do drain' in many places, whereas fo_drain() was called in order to force
blocking read()/write() etc calls to return to userspace so that a close()
call from a different thread can complete.
In the sockets code comment out the broken code in the inner function,
it was being called from compat code.

Fix locking as Andy explained. Also fill in uid and gid like sys_pipe did.

Fix PR/37878 and PR/37550: Provide stat(2) for all devices and don't use
fbadop_stat.

Add fileops::fo_drain(), to be called from fd_close() when there is more
than one active reference to a file descriptor. It should dislodge threads
sleeping while holding a reference to the descriptor. Implemented only for
sockets but should be extended to pipes, fifos, etc.

Fixes the case of a multithreaded process doing something like the
following, which would have hung until the process got a signal.

thr0 accept(fd, ...)
thr1 close(fd)

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.45.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.44.4; 1.44.6; 1.44.8;
Coverity CID 5021: Check pointers before using.

Coverity CID 5027: Remove impossible test.

KNF, whitespace, b* -> mem*. No functional change.

branches: 1.41.2;
Make various bits of debug code compile again.

Remove clause 3 and 4 from TNF licenses

branches: 1.39.2;
As suggested by rmind, do not check return status of KM_SLEEP/PR_WAITOK
allocations. A little hair-raising but it does make the code easier to
read.

branches: 1.38.2;
Protect selrecord/selnotify calls with crypto_mtx; few misc changes.

fix 64-bit b0rkenness.

Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

branches: 1.34.6;
Some locking fixes (double-release mutex in softintr wakeup case, which I
hadn't tested) and an uninitialized field in cse which Darran Hunt
found. Some more debugging printfs.

Turn on MPSAFE for the kthread. We're not sure it's safe for the softint
yet. Gives a little performance kick for swcrypto with many requests on
MP systems.

Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

Fix accidental checkin inverting the sense of cryptodev_allowsoft, which
is crazy but has always documented.

Make /dev/crypto properly cloning. Leave CRIOGET in place but note that
it is deprecated, no longer required, and will be removed in a future
release of NetBSD.

Dramatically reduce the size of the session structure by removing an
IOV_MAX array of iovecs where only the first was use. Saves an 8k
bzero on each session creation.

Convert fixed-size allocations in cryptodev.c to pools.

Some minor opencrypto fixes, one with a major performance impact for
OpenSSL:

1) Fix extremely misleading text in crypto.4 manual page so it does not
appear to claim that a new cloned file descriptor is required for every
session.

2) Fix severe performance problem (and fd leak!) in openssl cryptodev
engine resulting from misunderstanding probably caused by said manual
page text.

3) Check for session-ID wraparound in kernel cryptodev provider. Also,
start allocating sessions at 1, not 0 -- this will be necessary when
we add ioctls for the creation of multiple sessions at once, so we
can tell which if any creations failed.

Add constants for modular arithmetic operations other than exponentiation -- there's hardware out there which can do them.

branches: 1.26.16; 1.26.22; 1.26.28;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.25.4;
__unused removal on arguments; approved by core.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.23.2; 1.23.4;
fix incomplete initializer

Use the LWP cached credentials where sane.

integrate kauth.

Coverity CID 1083: Avoid possible NULL pointer deref.

don't use MALLOC with a non-constant size; use malloc instead.

branches: 1.18.2; 1.18.4;
sprinkle DPRINTF()...

branches: 1.17.2;
merge yamt-uio_vmspace branch.

- use vmspace rather than proc or lwp where appropriate.
the latter is more natural to specify an address space.
(and less likely to be abused for random purposes.)
- fix a swdmover race.

branches: 1.16.2; 1.16.4; 1.16.6;
merge ktrace-lwp.

- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.14.6;
No change. Forced commit to record commit message for previous revision, viz:

Fix vulnerability to a denial-of-service attack which passes a
length-0 crypto op. Check for zero length and return EINVAL, taken from:

http://cvsweb.FreeBSD.org/src/sys/opencrypto/cryptodev.c.diff?r1=1.25&r2=1.26

Original FreeBSD log mesage:

Modified files:
sys/opencrypto cryptodev.c
Log:
Fix bogus check. It was possible to panic the kernel by giving 0 length.
This is actually a local DoS, as every user can use /dev/crypto if there
is crypto hardware in the system and cryptodev.ko is loaded (or compiled
into the kernel).

Reported by: Mike Tancsa <mike@sentex.net>


thanks to Sam Leffler for passing on a heads-up about this issue.

*** empty log message ***

branches: 1.12.12;
Cloning cleanup:
1. make fileops const
2. add 2 new negative errno's to `officially' support the cloning hack:
- EDUPFD (used to overload ENODEV)
- EMOVEFD (used to overload ENXIO)
3. Created an fdclone() function to encapsulate the operations needed for
EMOVEFD, and made all cloners use it.
4. Centralize the local noop/badop fileops functions to:
fnullop_fcntl, fnullop_poll, fnullop_kqfilter, fbadop_stat

There's no need to pass a proc value when using UIO_SYSSPACE with
vn_rdwr(9) and uiomove(9).

OK'd by Jason Thorpe

branches: 1.10.4;
Wrap noisy pointless message about denied userspace requests with
`#ifdef CRYPTO_DEBUG', per Jason Thorpe's suggestion.

Clean up userlevel access to software kernel transforms, in preparation
for using /dev/crypto for OpenSSL:

1. Add comments explaining crypto_devallowsoft, explaining the
OpenBSD-style three-way logic actully implemented in crypto_newsession().

2. Pass crypto_devallowsoft as the final argument to crypto_newsession(),
instead of a constant 0 value.

3. Set the default value of crypto_devallowsoft to 1, to allow
/dev/crypto access only for hardware-supported transforms.

Items 1-3 may be revised to match the FreeBSD two-way logic, if the
consensus is that there's no point to forcing software transforms.
But as a first step, let the description match what the code actually does.

GC unused variables usercrypto, userasmcrypto, cryptodevallowsoft from
cryptodev.c, in favour of variables crypto_usercrypto, crypto_userasmcrypto,
crypto_devallowsoft, which are used as well as defined in crypto.c.

Remove '#ifdef notdef' around userspace ioctl() requests for
pure (non-HMAC) MD5 and SHA1.

Remove a bunch of unnecessary includes.

It's bad form to use the <opencrypto/rmd160.h> header file while
using the crypto/ripemd160/rmd160.c implementation. Remove the
opencrypto-local copies of these files entirely.

on netbsd, major # for /dev/crypto depends on arch

Pull up `done' flag for crypto operations from FreeBSD. FreeBSD deltas:
cryptodev.c: 1.4.2.3 -> 1.4.2.4
cryptodev.h: 1.4.2.4 -> 1.4.2.5

Garbage-collect references to OpenBSD-only <dev/rndvar.h>.

Remove vestiges of OpenBSD <sys/md5k.h> header.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

opencrypto/cryptodev.h: Fix includes.

- Move sys/condvar.h under #ifdef _KERNEL.
- Add some other necessary includes and forward declarations.
- Sort.

branches: 1.50.4;
opencrypto: crypto_dispatch never fails now. Make it return void.

Same with crypto_kdispatch.

opencrypto: Rip out EAGAIN logic when unregistering crypto drivers.

I'm pretty sure this never worked reliably based on code inspection,
and it's unlikely to have ever been tested because it only applies
when unregistering a driver -- but we have no crypto drivers for
removable devices, so it would only apply if we went out of our way
to trigger detach with drvctl.

Instead, just make the operation fail with ENODEV, and remove all the
callback logic to resubmit the request on EAGAIN. (Maybe this should
be ENXIO, but crypto_kdispatch already does ENODEV.)

opencrypto: Make crypto_freesession return void.

No callers use the return value. It is not sensible to allow this to
fail.

opencrypto: Make freesession callback return void.

No functional change intended: all drivers already return zero
unconditionally.

opencrypto: Make crp_callback, krp_callback return void.

Nothing uses the return values inside opencrypto, so let's stop
making users return them.

opencrypto: Nix CRYPTO_F_DONE.

Nothing uses it any more.

opencrypto: Make sid=0 always invalid, but OK to free.

Previously, crypto_newsession could sometimes return 0 as the
driver-specific part of the session id, and 0 as the hid, for sid=0.
But netipsec assumes that it is always safe to free sid=0 from
zero-initialized memory even if crypto_newsession has never
succeeded. So it was up to every driver in tree to gracefully handle
sid=0, if it happened to get assigned hid=0. And, as long as the
freesession callback was expected to just return an error code when
given a bogus session id, that worked out fine...because nothing ever
used the error code.

That was a terrible fragile system that should never have been
invented. Instead, let's just ensure that valid session ids are
nonzero, and make crypto_freesession with sid=0 be a no-op.

opencrypto: Nix CRYPTO_F_USER, CRYPTO_F_CBIMM, CRYPTO_F_CBIFSYNC.

CRYPTO_F_USER is no longer needed. It was introduced in 2008 by
darran@ in crypto.c 1.30, cryptodev.c 1.45 in an attempt to avoid
double-free between the issuing thread and asynchronous callback.
But the `fix' didn't work. In 2017, knakahara@ fixed it properly in
cryptodev.c 1.87 by distinguishing `the crypto operation has
completed' (CRYPTO_F_DONE) from `the callback is done touching the
crp object' (CRYPTO_F_DQRETQ, now renamed to CRYPTODEV_F_RET).

CRYPTO_F_CBIMM formerly served to invoke the callback synchronously
from the driver's interrupt completion routine, to reduce contention
on what was once a single cryptoret thread. Now, there is a per-CPU
queue and softint for much cheaper processing, so there is less
motivation for this in the first place. So let's remove the
complicated logic. This means the callbacks never run in hard
interrupt context, which means we don't need to worry about recursion
into crypto_dispatch in hard interrupt context.

fix typo in CRK_ALGORITHM_MIN definition to match CRK_ALGORITHM_MAX one.
while here fix few typos in comments.

fix typos in asymmetry, asymmetric(al), symmetrical.

HMAC-SHA-512 has 32 bytes MAC.

branches: 1.39.4;
update locking notes of opencrypto(9)

branches: 1.38.2;
make cryptoret() context softint to balance dequeuing crypto_ret_q with enqueuing it.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

add locking notes.

restructure locks(1/2): make relation between lock and data explicit.

+ crypto_drv_mtx protects
- whole crypto_drivers
+ crypto_drivers[i].cc_lock (new) protects
- crypto_drivers[i] itself
- member of crypto_drivers[i]
+ crypto_q_mtx protects
- crp_q
- crp_kq
+ crypto_ret_q_mtx protects
- crp_ret_q
- crp_ret_kq
- crypto_exit_flag

I will add locking note later.

branches: 1.34.2;
add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

remove obsoleted declarations.

opencrypto: cleanup debug messages.

branches: 1.31.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

add sysctl to select software/hardware encryption driver. can enable CRYPTO_DEBUG.

branches: 1.28.2; 1.28.4;
KNF. Remove extra spaces. No functional change.

Needs sys/time.h for struct timespec.

Re-work the module init and destroy code to allow it to be unloaded and
then reloaded.

Should fix PR kern/49842

branches: 1.25.12; 1.25.30;
-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.24.2;
pull in AES-GCM/GMAC support from OpenBSD
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

add an AES-CTR xform, from OpenBSD

remove redundant declarations

make camellia-cbc known to the opencrypto framework

make the use of SHA2-HMAC by FAST_IPSEC compliant to current standards:
-RFC2104 says that the block size of the hash algorithm must be used
for key/ipad/opad calculations. While formerly all ciphers used a block
length of 64, SHA384 and SHA512 use 128 bytes. So we can't use the
HMAC_BLOCK_LEN constant anymore. Add a new field to "struct auth_hash"
for the per-cipher blocksize.
-Due to this, there can't be a single "CRYPTO_SHA2_HMAC" external name
anymore. Replace this by 3 for the 3 different keysizes.
This was done by Open/FreeBSD before.
-Also fix the number of authenticator bits used tor ESP and AH to
conform to RFC4868, and remove uses of AH_HMAC_HASHLEN which did
assume a fixed authenticator size of 12 bytes.

FAST_IPSEC will not interoperate with KAME IPSEC anymore if sha2 is used,
because the latter doesn't implement these standards. It should
interoperate with at least modern Free/OpenBSD now.
(I've only tested with NetBSD-current/FAST_IPSEC on both ends.)

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

branches: 1.17.4; 1.17.6; 1.17.8;
It is best to explicitly include <sys/condvar.h> if we're going to
use kcondvar_t.

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.15.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.14.6; 1.14.8; 1.14.10;
Remove clause 3 and 4 from TNF licenses

branches: 1.13.2; 1.13.4;
Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

branches: 1.12.6;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

Add constants for modular arithmetic operations other than exponentiation -- there's hardware out there which can do them.

branches: 1.8.16; 1.8.22; 1.8.28;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.7.26;
- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.6.4; 1.6.10;
nuke trailing whitespace

branches: 1.5.4; 1.5.10; 1.5.12;
Pull in FreeBSD sys/opencrypto/cryptodev.h, revision 1.2.2.5 -> 1.2.2.6.
Sam Leffler's FreeBSD commit message was
``to eliminate context switch when returning results from the
software crypto driver''
but the patch also contains the CRYPTO_SESID*() macros used in newer
ubsec and hifn drivers.

Pull up `done' flag for crypto operations from FreeBSD. FreeBSD deltas:
cryptodev.c: 1.4.2.3 -> 1.4.2.4
cryptodev.h: 1.4.2.4 -> 1.4.2.5

Move the initialization of the crypto framework from the userland
pseudo-device to init_main(), so the framework is ready for
registration requests at autoconfiguration time.

Thanks to Quentin Garnier for confirming the change was required, and
for testing a similar fix.

Cleanup traces of previous standalone m_apply()/m_getptr().

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

rename crypto_mtx to cryptodev_mtx

It is used by cryptodev.c and ocryptodev.c only.

separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

branches: 1.1.2; 1.1.4; 1.1.8; 1.1.20; 1.1.38; 1.1.42;
make the compatibility code conditional on COMPAT_50

Correct a typo in rev 1.59, which results in a crash if the number of
sessions exceeds CRYPTO_SW_SESSIONS = 32.

branches: 1.64.4; 1.64.10;
opencrypto: Make freesession callback return void.

No functional change intended: all drivers already return zero
unconditionally.

cryptosoft(4): Prune dead branches. Assert session id validity.

cryptosoft(4): Rip out nonsense to quietly ignore sid=0.

This is no longer necessary because crypto_freesession no longer
calls into the driver for session ids that were never allocated in
the first place.

Fix ATF failures, sorry.

refactor: reduce access to swcr_sessions[i] directly

refactor: reduce changing swcr_sesnum

use kmem_{z,}alloc() instead of malloc()

branches: 1.57.4;
Fix kmem_free size in recent malloc->kmem conversion.

Should address this bracket report that has my name all over it:

https://mail-index.netbsd.org/current-users/2020/07/04/msg039059.html

opencrypto: Switch from legacy rijndael API to new aes API.

While here, apply various rijndael->aes renames, reduce the size
of aesxcbc_ctx by 480 bytes, and convert some malloc->kmem.

Leave in the symbol enc_xform_rijndael128 for now, though, so this
doesn't break any kernel ABI.

swcrypto(4): Simplify iv generation logic with cprng_fast.

add (void *) intermediate casts to elide gcc function cast warnings. This
is the simplest solution; choices:
- add pragmas, complex and ugly (need to be gcc-specific)
- add -Wno to COPTS. Needs to be done in many makefiles because of rump
- add intermediate functions: slows down things

Disable unloading until we keep track of references

branches: 1.52.6;
fix cryptosoft.c:r1.51 mistake. swcrypto_attach() must not be called from module_init_class().

swcrypto_attach() will call softint_establish(), it must be called after cpus
attached. module_init_class() is too early to call softint_establish().

branches: 1.51.2;
swcrypto0 was initialized twice. Fix like pseudo network interfaces.

ok by pgoyette@n.o.

opencrypto: cleanup debug messages.

branches: 1.49.2;
Remove duplicate assignment.
We assign the same value unconditionally just before.

from clang static analyzer

XXX surrounding code seems fishy

Fix usage of MD5Final/SHA1Final

Passing NULL as the digest parameter is wrong.

branches: 1.47.2; 1.47.4;
include "ioconf.h" to get the 'void <driver>attach(int count);' prototype.

branches: 1.46.4;
If we register with pmf on attach, deregister on detach.

register with pmf.

branches: 1.44.2;
Modularize the opencrypto components and link to the build

Remove unused variable

branches: 1.42.2;
Replace consttime_bcmp/explicit_bzero by consttime_memequal/explicit_memset.

consttime_memequal is the same as the old consttime_bcmp.
explicit_memset is to memset as explicit_bzero was to bcmp.

Passes amd64 release and i386/ALL, but I'm sure I missed some spots,
so please let me know.

fix compilation

branches: 1.40.2;
Add "consttime_bcmp" and "explicit_bzero" functions for both kernel
abd userland, as proposed on tech-security, with explicit_bzero using
a volatile function pointer as suggested by Alan Barrett.
Both do what the name says. For userland, both are prefixed by "__"
to keep them out of the user namespace.
Change some memset/memcmp uses to the new functions where it makes
sense -- these are just some examples, more to come.

Remove arc4random() and arc4randbytes() from the kernel API. Replace
arc4random() hacks in rump with stubs that call the host arc4random() to
get numbers that are hopefully actually random (arc4random() keyed with
stack junk is not). This should fix some of the currently failing anita
tests -- we should no longer generate duplicate "random" MAC addresses in
the test environment.

branches: 1.38.2;
use a simple counter as IV for AES-GMAC as suggested in RFC4543

branches: 1.37.2;
pull in AES-GCM/GMAC support from OpenBSD
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

move the "context size" struct member (which is a pure software
implementation thing) from the abstract xform descriptor to
the cryptosoft implementation part -- for sanity, and now clients
of opencrypto don't depend on headers of cipher implementations anymore

Change the way the IV is generated for AES-CTR: use a simple counter
instead of arc4random(). AES-CTR is sensitive against IV recurrence
(with the same key / nonce), and a random number doesn't give that
guarantee.
This needs a little API change in cryptosoft -- I've suggested it to
Open/FreeBSD, might change it depending on feedback.
Thanks to Steven Bellovin for hints.

add an AES-CTR xform, from OpenBSD

-in the descriptor for encryption xforms, split the "blocksize" field
into "blocksize" and "IV size"
-add an "reinit" function pointer which, if set, means that the xform
does its IV handling itself and doesn't want the default CBC handling
by the framework (poor name, but left that way to avoid unecessary
differences)
This syncs with Open/FreeBSD, purpose is to allow non-CBC transforms.
Refer to ivsize instead of blocksize where appropriate.
(At this point, blocksize and ivsize are identical.)

fix a logics bug (which has been here from the beginning) which made
that only 96 random bits were used for IV generation,
this caused eg that the last 4 bytes of the IV in ESP/AES-CBC
were constant, leaking kernel memory
affects FAST_IPSEC only

support camellia-cbc by swcrypt

make the use of SHA2-HMAC by FAST_IPSEC compliant to current standards:
-RFC2104 says that the block size of the hash algorithm must be used
for key/ipad/opad calculations. While formerly all ciphers used a block
length of 64, SHA384 and SHA512 use 128 bytes. So we can't use the
HMAC_BLOCK_LEN constant anymore. Add a new field to "struct auth_hash"
for the per-cipher blocksize.
-Due to this, there can't be a single "CRYPTO_SHA2_HMAC" external name
anymore. Replace this by 3 for the 3 different keysizes.
This was done by Open/FreeBSD before.
-Also fix the number of authenticator bits used tor ESP and AH to
conform to RFC4868, and remove uses of AH_HMAC_HASHLEN which did
assume a fixed authenticator size of 12 bytes.

FAST_IPSEC will not interoperate with KAME IPSEC anymore if sha2 is used,
because the latter doesn't implement these standards. It should
interoperate with at least modern Free/OpenBSD now.
(I've only tested with NetBSD-current/FAST_IPSEC on both ends.)

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

Don't store temporary values in the opencrypto session data struct which
can be shared by multiple threads -- pass them on the stack instead.
Add some "const" to document this. (One _could_ use the session struct
for temporary stuff with proper locking, but it seems unnecessary here.)
Also remove the unused SW_crc member in the session struct.
From Wolfgang Stukenbrock per PR kern/44472.

branches: 1.26.2; 1.26.4;
Consistently use a single CRYPTO_SESID2HID-like macro.
Improve CRYPTO_DEBUG printing a bit:
print pointers with %p
print unsigned with %u rather than %d
use CRYPTO_SESID2LID instead of just casting to uint32_t

branches: 1.25.2; 1.25.4;
Remove extra whitespace added by a stupid tool.
XXX: more in src/sys/arch

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

bcopy -> memcpy

bzero -> memset

branches: 1.21.2;
kill MALLOC and FREE macros.

branches: 1.20.6; 1.20.10; 1.20.18; 1.20.20; 1.20.26;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

Add CRYPTO_*_HMAC_96 defines -- missed this file in previous commit.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

branches: 1.17.16; 1.17.22;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.16.2;
Add an opencrypto provider for the AES xcrypt instructions found on VIA
C5P and later cores (also known as 'ACE', which is part of the VIA PadLock
security engine). Ported from OpenBSD.

Reviewed on tech-crypto and port-i386, no objections to commiting this.

__unused removal on arguments; approved by core.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.13.8; 1.13.10;
malloc data the size the pointer points to, not the size of a pointer.
Maybe we get away with this (at least on 32bit archs) because the structure
is 24 bytes and I bet the minimum allocation size is 32.
Fixed coverty CIDs 2732 and 2733

don't use MALLOC with a non-constant size; use malloc instead.

branches: 1.11.4; 1.11.6; 1.11.8; 1.11.10; 1.11.12;
swcr -> swcrypto

- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.9.4; 1.9.10;
nuke trailing whitespace

branches: 1.8.4; 1.8.10; 1.8.12;
Some const poisoning.

Remove a bunch of unnecessary includes.

It's bad form to use the <opencrypto/rmd160.h> header file while
using the crypto/ripemd160/rmd160.c implementation. Remove the
opencrypto-local copies of these files entirely.

Garbage-collect references to OpenBSD-only <dev/rndvar.h>.

Remove vestiges of OpenBSD <sys/md5k.h> header.

Cleanup traces of previous standalone m_apply()/m_getptr().

Fix authentication hashes requested via /dev/crypto. The handler for
userland hashes case was partly omitted in the OpenBSD -> FreeBSD port.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

Don't store temporary values in the opencrypto session data struct which
can be shared by multiple threads -- pass them on the stack instead.
Add some "const" to document this. (One _could_ use the session struct
for temporary stuff with proper locking, but it seems unnecessary here.)
Also remove the unused SW_crc member in the session struct.
From Wolfgang Stukenbrock per PR kern/44472.

branches: 1.6.4; 1.6.6; 1.6.8;
Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.5.40; 1.5.50; 1.5.52; 1.5.56; 1.5.60;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.4.2;
Add an opencrypto provider for the AES xcrypt instructions found on VIA
C5P and later cores (also known as 'ACE', which is part of the VIA PadLock
security engine). Ported from OpenBSD.

Reviewed on tech-crypto and port-i386, no objections to commiting this.

- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.2.4; 1.2.18; 1.2.26;
Some const poisoning.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

Rename enc_xform_rijndael128 -> enc_xform_aes.

Update netipsec dependency.

opencrypto: Switch from legacy rijndael API to new aes API.

While here, apply various rijndael->aes renames, reduce the size
of aesxcbc_ctx by 480 bytes, and convert some malloc->kmem.

Leave in the symbol enc_xform_rijndael128 for now, though, so this
doesn't break any kernel ABI.

add (void *) intermediate casts to elide gcc function cast warnings. This
is the simplest solution; choices:
- add pragmas, complex and ugly (need to be gcc-specific)
- add -Wno to COPTS. Needs to be done in many makefiles because of rump
- add intermediate functions: slows down things

branches: 1.27.20;
simplify, no functional change

fix compilation

branches: 1.25.8;
Remove arc4random() and arc4randbytes() from the kernel API. Replace
arc4random() hacks in rump with stubs that call the host arc4random() to
get numbers that are hopefully actually random (arc4random() keyed with
stack junk is not). This should fix some of the currently failing anita
tests -- we should no longer generate duplicate "random" MAC addresses in
the test environment.

branches: 1.24.2;
use a simple counter as IV for AES-GMAC as suggested in RFC4543

branches: 1.23.2;
pull in AES-GCM/GMAC support from OpenBSD
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

move the "context size" struct member (which is a pure software
implementation thing) from the abstract xform descriptor to
the cryptosoft implementation part -- for sanity, and now clients
of opencrypto don't depend on headers of cipher implementations anymore

Change the way the IV is generated for AES-CTR: use a simple counter
instead of arc4random(). AES-CTR is sensitive against IV recurrence
(with the same key / nonce), and a random number doesn't give that
guarantee.
This needs a little API change in cryptosoft -- I've suggested it to
Open/FreeBSD, might change it depending on feedback.
Thanks to Steven Bellovin for hints.

-remove references to crypto/arc4/arc4.* -- the code isn't used
anywhere afaics
(The confusion comes probably from use of arc4random() at various places,
but this lives in libkern and doesn't share code with the former.)
-g/c non-implementation of arc4 encryption in swcrypto(4)
-remove special casing of ARC4 in crypto(4) -- the point is that it
doesn't use an IV, and this fact is made explicit by the new "ivsize"
property of xforms

add an AES-CTR xform, from OpenBSD

-in the descriptor for encryption xforms, split the "blocksize" field
into "blocksize" and "IV size"
-add an "reinit" function pointer which, if set, means that the xform
does its IV handling itself and doesn't want the default CBC handling
by the framework (poor name, but left that way to avoid unecessary
differences)
This syncs with Open/FreeBSD, purpose is to allow non-CBC transforms.
Refer to ivsize instead of blocksize where appropriate.
(At this point, blocksize and ivsize are identical.)

check key size on initialization -- othwise the rijndael code
can fail silently

support camellia-cbc by swcrypt

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

more "const"

branches: 1.12.4; 1.12.6; 1.12.8;
Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

bzero -> memset

Change about 4500 of the K&R function definitions to ANSI ones.
There are still about 1600 left, but they have ',' or /* ... */
in the actual variable definitions - which my awk script doesn't handle.
There are also many that need () -> (void).
(The script does handle misordered arguments.)

branches: 1.9.2;
kill MALLOC and FREE macros.

branches: 1.8.6; 1.8.10; 1.8.18; 1.8.20; 1.8.26;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

don't allocate AES keys with M_WAITOK!

branches: 1.5.16; 1.5.22;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.4.4;
__unused removal on arguments; approved by core.

Merge kernel and userland rmd160 and sha2 implementation.
XXX: We still install rmd160.h and sha2.h in /usr/include/crypto, unlike
the other hash functions which get installed in /usr/include for compatibility.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.1.2; 1.1.4; 1.1.22; 1.1.26; 1.1.28;
- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

opencrypto: cleanup debug messages.

branches: 1.22.8;
Trivial printf format changes and typo fix

branches: 1.21.12; 1.21.30;
avoid some uninitialised warning issues GCC 4.5 complains about.

-start to make the GZIP code similar to DEFLATE: make error handling
work the same way, grow output buffer exponentially and kill
reallocation of metadata
-minor cleanup, make definitions private which are implementation
details of deflate.gzip

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

redo result buffer allocation, to avoid dynamic allocations:
-use exponentially growing buffer sizes instead of just linear extension
-drop the dynamic allocation of buffer metadata introduced in rev.1.8 --
if the initial array is not sufficient something is wrong
-apply some (arbitrary, heuristic) limit so that compressed data
which extract into insane amounts of constant data don't kill the system
This addresses PR kern/36864 by Wolfgang Stukenbrock. Some tuning
might be useful, but hopefully this is an improvement already.

address the obvious byteorder and alignment problems in gzip size/crc
tail, should fix PR kern/44210 by Wolfgang Stukenbrock
being here, fix a bug in crc calculation of decompressed data, and
actually verify the crc

The decompressor in sys/net/zlib.c has a bug: It returns Z_BUF_ERROR after
a successful decompression in rare cases. A necessary but not sufficient
condition seems to be that the decompressed data end exactly at the end
of an allocated output buffer. (I can reproduce this reliably with
a userland program built against kernel zlib. Userland libz is much
newer and not affected.)
Since kernel zlib is based on an old version and heavily modified, I don't
dare to touch it. So catch this case in the wrapper.
Being here, reorder deflate/inflate error handling and add comments
to make understandable what is tested and why.

-avoid allocation of an extra result buffer and data copy in case
the DEFLATE complssion/decompression result is within a single
buffer already
-simplify bookkeeping of allocated buffers (and don't waste the
last member of the metadata array)
from Wolfgang Stukenbrock per PR kern/36865 (with some cleanup
of error handling by me)
The Gzip compression case can be improved too, but for now I've applied
the buffer bookkeeping changes.

tested with IP4 IPCOMP

whan compressing, set the Z_FINISH flag to zlib to tell that
the data chunk is the final one, which makes that zlib issues the
proper termination marker
(KAME IPSEC does this, but doesn't check eagerly in the receive
path, so the missing termination didn't cause problems so far)
closes my PR kern/44539
being here, replace the Z_PARTIAL_FLUSH flag which is marked
deprecated by zlib by Z_SYNC_FLUSH in the decompression path
(tested with IPv4 IPCOMP on i386)

branches: 1.13.4; 1.13.6; 1.13.8;
Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

Change about 4500 of the K&R function definitions to ANSI ones.
There are still about 1600 left, but they have ',' or /* ... */
in the actual variable definitions - which my awk script doesn't handle.
There are also many that need () -> (void).
(The script does handle misordered arguments.)

branches: 1.11.8; 1.11.10; 1.11.14; 1.11.18;
Back out previous. It broke the build.

Move zlib out of net/ and into kern/. It would probably be better to use
the reachover Makefiles and libz, but this is already here and it works.

branches: 1.9.24;
Fix a stupid bogus bcopy -> memcpy conversion.
I will write it 1000 times : when I convert bcopy to memcpy, don't forget to
swap first and secund args.

branches: 1.8.6; 1.8.8;
Use dynamic array instead of an static array to decompress. It lets us to
decompress any data, whatever is the radio decompressed data / compressed
data.

It fixes the last issues with fast_ipsec and ipcomp.

While here, bzero -> memset, bcopy -> memcpy, FREE -> free

Reviewed a long time ago by sam@

branches: 1.7.2; 1.7.8; 1.7.10; 1.7.16;
__unused removal on arguments; approved by core.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.5.8; 1.5.10;
don't use MALLOC with a non-constant size; use malloc instead.

branches: 1.4.4; 1.4.6; 1.4.8; 1.4.10; 1.4.12;
merge ktrace-lwp.

branches: 1.3.4;
nuke trailing whitespace

branches: 1.2.4; 1.2.10; 1.2.12;
Tidy up the namespace of this a little.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

-start to make the GZIP code similar to DEFLATE: make error handling
work the same way, grow output buffer exponentially and kill
reallocation of metadata
-minor cleanup, make definitions private which are implementation
details of deflate.gzip

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

-avoid allocation of an extra result buffer and data copy in case
the DEFLATE complssion/decompression result is within a single
buffer already
-simplify bookkeeping of allocated buffers (and don't waste the
last member of the metadata array)
from Wolfgang Stukenbrock per PR kern/36865 (with some cleanup
of error handling by me)
The Gzip compression case can be improved too, but for now I've applied
the buffer bookkeeping changes.

tested with IP4 IPCOMP

branches: 1.6.4; 1.6.6; 1.6.8;
Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.5.8; 1.5.10; 1.5.14; 1.5.18;
Back out previous. It broke the build.

Move zlib out of net/ and into kern/. It would probably be better to use
the reachover Makefiles and libz, but this is already here and it works.

branches: 1.3.74;
merge ktrace-lwp.

nuke trailing whitespace

branches: 1.1.4; 1.1.10; 1.1.12;
Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

These are unused; remove them.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

These are unused; remove them.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

opencrypto: Switch from legacy rijndael API to new aes API.

While here, apply various rijndael->aes renames, reduce the size
of aesxcbc_ctx by 480 bytes, and convert some malloc->kmem.

Leave in the symbol enc_xform_rijndael128 for now, though, so this
doesn't break any kernel ABI.

Make crypto/rijindael optional again as cprng_strong does no longer
depend on it. Dependency is explicitly declared in files.foo if a
component requires it.

branches: 1.28.4;
Split the module glue out from the rest of opencrypto/ocryptodev to
make rump happy.

Rump doesn't have compat modules (the compat code is included in the
relevant librump*.so), so there's no module compat_50 listed in
link_set_modules, and thus ocryptodev's MODULE(...) can't "require"
it.

This fixes the problem of "built-in module compat_50 not found" when
starting up rump_allserver (or rump_server with -l rumpdev_opencrypto).

XXX This does not resolve the long-standing "crypto: unable to
XXX register devsw, error 17" message noted at line 78 of
XXX sys/rump/dev/lib/libopencrypto/opencrypto_component.c

branches: 1.27.6;
Merge the [pgoyette-compat] branch

branches: 1.26.8; 1.26.10;
make the default values of q_maxlen build parameters.

branches: 1.25.8; 1.25.26;
First step of random number subsystem rework described in
<20111022023242.BA26F14A158@mail.netbsd.org>. This change includes
the following:

An initial cleanup and minor reorganization of the entropy pool
code in sys/dev/rnd.c and sys/dev/rndpool.c. Several bugs are
fixed. Some effort is made to accumulate entropy more quickly at
boot time.

A generic interface, "rndsink", is added, for stream generators to
request that they be re-keyed with good quality entropy from the pool
as soon as it is available.

The arc4random()/arc4randbytes() implementation in libkern is
adjusted to use the rndsink interface for rekeying, which helps
address the problem of low-quality keys at boot time.

An implementation of the FIPS 140-2 statistical tests for random
number generator quality is provided (libkern/rngtest.c). This
is based on Greg Rose's implementation from Qualcomm.

A new random stream generator, nist_ctr_drbg, is provided. It is
based on an implementation of the NIST SP800-90 CTR_DRBG by
Henric Jungheim. This generator users AES in a modified counter
mode to generate a backtracking-resistant random stream.

An abstraction layer, "cprng", is provided for in-kernel consumers
of randomness. The arc4random/arc4randbytes API is deprecated for
in-kernel use. It is replaced by "cprng_strong". The current
cprng_fast implementation wraps the existing arc4random
implementation. The current cprng_strong implementation wraps the
new CTR_DRBG implementation. Both interfaces are rekeyed from
the entropy pool automatically at intervals justifiable from best
current cryptographic practice.

In some quick tests, cprng_fast() is about the same speed as
the old arc4randbytes(), and cprng_strong() is about 20% faster
than rnd_extract_data(). Performance is expected to improve.

The AES code in src/crypto/rijndael is no longer an optional
kernel component, as it is required by cprng_strong, which is
not an optional kernel component.

The entropy pool output is subjected to the rngtest tests at
startup time; if it fails, the system will reboot. There is
approximately a 3/10000 chance of a false positive from these
tests. Entropy pool _input_ from hardware random numbers is
subjected to the rngtest tests at attach time, as well as the
FIPS continuous-output test, to detect bad or stuck hardware
RNGs; if any are detected, they are detached, but the system
continues to run.

A problem with rndctl(8) is fixed -- datastructures with
pointers in arrays are no longer passed to userspace (this
was not a security problem, but rather a major issue for
compat32). A new kernel will require a new rndctl.

The sysctl kern.arandom() and kern.urandom() nodes are hooked
up to the new generators, but the /dev/*random pseudodevices
are not, yet.

Manual pages for the new kernel interfaces are forthcoming.

branches: 1.24.4;
pull in AES-GCM/GMAC support from OpenBSD
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

support camellia-cbc by swcrypt

make the compatibility code conditional on COMPAT_50

branches: 1.20.4; 1.20.6; 1.20.8;
Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.19.10; 1.19.18; 1.19.20; 1.19.24; 1.19.28;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.