Home | History | Annotate | Download | only in netipsec
History log of /src/sys/netipsec/xform_esp.c
RevisionDateAuthorComments
 1.107  05-Jul-2024  rin sys: Drop redundant NULL check before m_freem(9)

m_freem(9) safely has accepted NULL argument at least since 4.2BSD:
https://www.tuhs.org/cgi-bin/utree.pl?file=4.2BSD/usr/src/sys/sys/uipc_mbuf.c

Compile-tested on amd64/ALL.

Suggested by knakahara@
 1.106  25-May-2022  ozaki-r ipsec: don't assert for the format of incoming packets
 1.105  22-May-2022  riastradh opencrypto: crypto_dispatch never fails now. Make it return void.

Same with crypto_kdispatch.
 1.104  22-May-2022  riastradh opencrypto: Rip out EAGAIN logic when unregistering crypto drivers.

I'm pretty sure this never worked reliably based on code inspection,
and it's unlikely to have ever been tested because it only applies
when unregistering a driver -- but we have no crypto drivers for
removable devices, so it would only apply if we went out of our way
to trigger detach with drvctl.

Instead, just make the operation fail with ENODEV, and remove all the
callback logic to resubmit the request on EAGAIN. (Maybe this should
be ENXIO, but crypto_kdispatch already does ENODEV.)
 1.103  22-May-2022  riastradh netipsec: Nothing uses xf_zeroize return value. Nix it.
 1.102  22-May-2022  riastradh opencrypto: Make crp_callback, krp_callback return void.

Nothing uses the return values inside opencrypto, so let's stop
making users return them.
 1.101  05-Oct-2020  knakahara Make sequence number of esp header MP-safe for IPsec Tx side. reviewed by ozaki-r@n.o

In IPsec Tx side, one Security Association can be used by multiple CPUs.
On the other hand, in IPsec Rx side, one Security Association is used
by only one CPU.

XXX pullup-{8,9}
 1.100  30-Jun-2020  riastradh Rename enc_xform_rijndael128 -> enc_xform_aes.

Update netipsec dependency.
 1.99  01-Nov-2019  knakahara Fix ipsecif(4) IPV6_MINMTU does not work correctly.
 1.98  12-Jun-2019  christos branches: 1.98.2;
make DPRINTF use varyadic cpp macros, and merge with IPSECLOG.
 1.97  27-Jan-2019  pgoyette Merge the [pgoyette-compat] branch
 1.96  31-May-2018  maxv branches: 1.96.2;
Add a comment and a KASSERT. I remember wondering whether this check was a
problem, since ARC4 has a blocksize of one. Normally ARC4 can't be used in
IPsec.
 1.95  31-May-2018  maxv style
 1.94  30-May-2018  maxv Introduce ah_authsiz, which computes the length of the ICV only. Use it in
esp_hdrsiz, and clarify.

Until now we were using ah_hdrsiz, and were relying on the fact that the
size of the AH header happens to be equal to that of the ESP trailer.

Now the size of the ESP trailer is added manually. This also fixes one
branch in esp_hdrsiz: we always append an ESP trailer, so it must always
be taken into account, and not just when an ICV is here.
 1.93  30-May-2018  maxv Apply the previous change in esp_input too, same as esp_output.
 1.92  30-May-2018  maxv Remove dead code, 'espx' is never NULL and dereferenced earlier, so no need
to NULL-check all the time.
 1.91  30-May-2018  maxv Simplify the padding computation. Until now 'padlen' contained the ESP
Trailer (two bytes), and we were doing minus two all the time.

Declare 'tlen', which contains padlen+ESP_Trailer+ICV, and use 'struct
esptail' instead of hardcoding the construction of the trailer. 'padlen'
now indicates only the length of the padding, so no need to do -2.
 1.90  30-May-2018  maxv Rename padding -> padlen, pad -> tail, and clarify.
 1.89  18-May-2018  maxv IP6_EXTHDR_GET -> M_REGION_GET, no functional change.
 1.88  13-May-2018  maxv Remove unused calls to nat_t_ports_get.
 1.87  11-May-2018  maxv ENOBUFS -> EACCES when updating the replay counter.
 1.86  07-May-2018  maxv Remove unused 'mp' argument from all the xf_output functions. Also clean
up xform.h a bit.
 1.85  01-May-2018  maxv When IP6_EXTHDR_GET fails, return ENOBUFS, and don't log an error (HDROPS
is not supposed to be used here).
 1.84  01-May-2018  maxv When the replay check fails, return EACCES instead of ENOBUFS.
 1.83  01-May-2018  maxv Remove double include, opencrypto/xform.h is already included in
netipsec/xform.h.
 1.82  28-Apr-2018  maxv Remove IPSEC_SPLASSERT_SOFTNET, it has always been a no-op.
 1.81  19-Apr-2018  maxv Remove extra long file paths from the headers.
 1.80  19-Apr-2018  maxv Style, and remove meaningless XXX.
 1.79  16-Feb-2018  maxv branches: 1.79.2;
Add [ah/esp/ipcomp]_enable sysctls, and remove the FreeBSD #ifdefs.
Discussed with ozaki-r@.
 1.78  16-Feb-2018  maxv Remove some more FreeBSD sysctl declarations that already have NetBSD
counterparts. Discussed with ozaki-r@.
 1.77  15-Feb-2018  maxv Style a bit, and if we don't know the pad-filling policy use
SADB_X_EXT_PZERO by default.

There doesn't seem to be a sanity check in the keysock API to make sure
this place is never reached, and it's better to fill in with zeros than
not filling in at all (and leaking uninitialized mbuf data).
 1.76  15-Feb-2018  ozaki-r Don't relook up an SP/SA in opencrpyto callbacks

We don't need to do so because we have a reference to it. And also
relooking-up one there may return an sp/sav that has different
parameters from an original one.
 1.75  14-Feb-2018  ozaki-r Dedup common codes in error paths (NFCI)
 1.74  14-Feb-2018  ozaki-r Fix mbuf leaks on error paths

Pointed out by maxv@
 1.73  24-Jan-2018  maxv Add missing NULL check. Normally that's not triggerable remotely, since we
are guaranteed that 8 bytes are valid at mbuf+skip.
 1.72  03-Oct-2017  ozaki-r Constify isr at many places (NFC)
 1.71  10-Aug-2017  ozaki-r Use pool_cache(9) instead of pool(9) for tdb_crypto objects

The change improves network throughput especially on multi-core systems.
 1.70  09-Aug-2017  ozaki-r MP-ify SAD (savlist)

localcount(9) is used to protect savlist of sah. The basic design is
similar to MP-ifications of SPD and SAD sahlist. Please read the
locking notes of SAD for more details.
 1.69  03-Aug-2017  ozaki-r Introduce KEY_SA_UNREF and replace KEY_FREESAV with it where sav will never be actually freed in the future

KEY_SA_UNREF is still key_freesav so no functional change for now.

This change reduces diff of further changes.
 1.68  02-Aug-2017  ozaki-r Make IPsec SPD MP-safe

We use localcount(9), not psref(9), to make the sptree and secpolicy (SP)
entries MP-safe because SPs need to be referenced over opencrypto
processing that executes a callback in a different context.

SPs on sockets aren't managed by the sptree and can be destroyed in softint.
localcount_drain cannot be used in softint so we delay the destruction of
such SPs to a thread context. To do so, a list to manage such SPs is added
(key_socksplist) and key_timehandler_spd deletes dead SPs in the list.

For more details please read the locking notes in key.c.

Proposed on tech-kern@ and tech-net@
 1.67  27-Jul-2017  ozaki-r Don't acquire global locks for IPsec if NET_MPSAFE

Note that the change is just to make testing easy and IPsec isn't MP-safe yet.
 1.66  20-Jul-2017  ozaki-r Use pool to allocate tdb_crypto

For ESP and AH, we need to allocate an extra variable space in addition
to struct tdb_crypto. The fixed size of pool items may be larger than
an actual requisite size of a buffer, but still the performance
improvement by replacing malloc with pool wins.
 1.65  19-Jul-2017  ozaki-r Hold a reference to an SP during opencrypto processing

An SP has a list of isr (ipsecrequest) that represents a sequence
of IPsec encryption/authentication processing. One isr corresponds
to one opencrypto processing. The lifetime of an isr follows its SP.

We pass an isr to a callback function of opencrypto to continue
to a next encryption/authentication processing. However nobody
guaranteed that the isr wasn't freed, i.e., its SP wasn't destroyed.

In order to avoid such unexpected destruction of isr, hold a reference
to its SP during opencrypto processing.
 1.64  19-Jul-2017  ozaki-r Don't bother the case of crp->crp_buf == NULL in callbacks
 1.63  19-Jul-2017  ozaki-r Don't release sav if calling crypto_dispatch again
 1.62  14-Jul-2017  ozaki-r Prepare to stop using isr->sav

isr is a shared resource and using isr->sav as a temporal storage
for each packet processing is racy. And also having a reference from
isr to sav makes the lifetime of sav non-deterministic; such a reference
is removed when a packet is processed and isr->sav is overwritten by
new one. Let's have a sav locally for each packet processing instead of
using shared isr->sav.

However this change doesn't stop using isr->sav yet because there are
some users of isr->sav. isr->sav will be removed after the users find
a way to not use isr->sav.
 1.61  14-Jul-2017  ozaki-r Pass sav directly to opencrypto callback

In a callback, use a passed sav as-is by default and look up a sav
only if the passed sav is dead.
 1.60  13-Jul-2017  ozaki-r Fix header size calculation of esp where sav is NULL
 1.59  10-Jul-2017  ozaki-r Use explicit_memset to surely zero-clear key_auth and key_enc
 1.58  07-Jul-2017  ozaki-r Rename key_alloc* functions (NFC)

We shouldn't use the term "alloc" for functions that just look up
data and actually don't allocate memory.
 1.57  05-Jul-2017  ozaki-r Remove codes for PACKET_TAG_IPSEC_IN_CRYPTO_DONE

It seems that PACKET_TAG_IPSEC_IN_CRYPTO_DONE is for network adapters
that have IPsec accelerators; a driver sets the mtag to a packet
when its device has already encrypted the packet.

Unfortunately no driver implements such offload features for long
years and seems unlikely to implement them soon. (Note that neither
FreeBSD nor Linux doesn't have such drivers.) Let's remove related
(unused) codes and simplify the IPsec code.
 1.56  29-Jun-2017  ozaki-r Apply C99-style struct initialization to xformsw
 1.55  11-May-2017  ryo branches: 1.55.2;
Make ipsec_address() and ipsec_logsastr() mpsafe.
 1.54  19-Apr-2017  ozaki-r branches: 1.54.2;
Retire ipsec_osdep.h

We don't need to care other OSes (FreeBSD) anymore.

Some macros are alive in ipsec_private.h.
 1.53  18-Apr-2017  ozaki-r Convert IPSEC_ASSERT to KASSERT or KASSERTMSG

IPSEC_ASSERT just discarded specified message...
 1.52  18-Apr-2017  ozaki-r Remove __FreeBSD__ and __NetBSD__ switches

No functional changes (except for a debug printf).

Note that there remain some __FreeBSD__ for sysctl knobs which counerparts
to NetBSD don't exist. And ipsec_osdep.h isn't touched yet; tidying it up
requires actual code changes.
 1.51  15-Apr-2017  christos cosmetic fixes:
- __func__ in printfs
- no space after sizeof
- eliminate useless casts
- u_intX_t -> uintX_t
 1.50  13-Apr-2017  christos Redo the statistics through an indirection array and put the definitions
of the arrays in pfkeyv2.h so that they are next to the index definitions.
Remove "bogus" comment about compressing the statistics which is now fixed.
 1.49  13-Apr-2017  ozaki-r Fix that ah_algorithm_lookup and esp_algorithm_lookup don't handle some algorithms

Unrelated upper limit values, AH_ALG_MAX and ESP_ALG_MAX, prevented some
algorithms from being looked up.
 1.48  10-Apr-2017  christos PR/52150: Ryota Ozaki: ipsec: kernel panic on adding a key with an invalid
length.
 1.47  06-Apr-2017  ozaki-r Prepare netipsec for rump-ification

- Include "opt_*.h" only if _KERNEL_OPT is defined
- Allow encapinit to be called twice (by ifinit and ipe4_attach)
- ifinit didn't call encapinit if IPSEC is enabled (ipe4_attach called
it instead), however, on a rump kernel ipe4_attach may not be called
even if IPSEC is enabled. So we need to allow ifinit to call it anyway
- Setup sysctls in ipsec_attach explicitly instead of using SYSCTL_SETUP
- Call ip6flow_invalidate_all in key_spdadd only if in6_present
- It's possible that a rump kernel loads the ipsec library but not
the inet6 library
 1.46  30-Mar-2015  ozaki-r branches: 1.46.2; 1.46.4;
Tidy up opt_ipsec.h inclusions

Some inclusions of opt_ipsec.h were for IPSEC_NAT_T and are now unnecessary.
Add inclusions to some C files for IPSEC_DEBUG.
 1.45  03-Nov-2013  mrg branches: 1.45.4; 1.45.6; 1.45.8; 1.45.12;
- apply some __diagused
- remove unused variables
- move some variables inside their relevant use #ifdef
 1.44  28-Aug-2013  riastradh Fix sense of consttime_memequal and update all callers.

Now it returns true (nonzero) to mean equal and false (zero) to mean
inequal, as the name suggests.

As promised on tech-userlevel back in June:

https://mail-index.netbsd.org/tech-userlevel/2013/06/24/msg007843.html
 1.43  24-Jun-2013  riastradh branches: 1.43.2;
Replace consttime_bcmp/explicit_bzero by consttime_memequal/explicit_memset.

consttime_memequal is the same as the old consttime_bcmp.
explicit_memset is to memset as explicit_bzero was to bcmp.

Passes amd64 release and i386/ALL, but I'm sure I missed some spots,
so please let me know.
 1.42  04-Jun-2013  christos PR/47886: Dr. Wolfgang Stukenbrock: IPSEC_NAT_T enabled kernels may access
outdated pointers and pass ESP data to UPD-sockets.
While here, simplify the code and remove the IPSEC_NAT_T option; always
compile nat-traversal in so that it does not bitrot.
 1.41  30-Aug-2012  drochner branches: 1.41.2;
Add "consttime_bcmp" and "explicit_bzero" functions for both kernel
abd userland, as proposed on tech-security, with explicit_bzero using
a volatile function pointer as suggested by Alan Barrett.
Both do what the name says. For userland, both are prefixed by "__"
to keep them out of the user namespace.
Change some memset/memcmp uses to the new functions where it makes
sense -- these are just some examples, more to come.
 1.40  25-Jan-2012  drochner branches: 1.40.2; 1.40.6; 1.40.8;
Make sure the mbufs in the input path (only the parts which we are going
to modify in the AH case) are writable/non-shared.
This addresses PR kern/33162 by Jeff Rizzo, and replaces the insufficient
patch from that time by a radical solution.
(The PR's problem had been worked around by rev.1.3 of xennetback_xenbus.c,
so it needs a network driver modification to reproduce it.)
Being here, clarify a bit of ipcomp -- uncompression is done in-place,
the header must be removed explicitly.
 1.39  31-Aug-2011  plunky branches: 1.39.2; 1.39.6;
NULL does not need a cast
 1.38  26-May-2011  drochner pull in AES-GCM/GMAC support from OpenBSD
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.
 1.37  23-May-2011  drochner g/c remainders of IV handling in pfkey code -- this is done in
opencrypto now
 1.36  23-May-2011  drochner allow ESP to use AES-CTR
(pfkey and userland tool support is already there because it has been
in KAME IPSEC all the time)
tested against KAME IPSEC
 1.35  23-May-2011  drochner -in the descriptor for encryption xforms, split the "blocksize" field
into "blocksize" and "IV size"
-add an "reinit" function pointer which, if set, means that the xform
does its IV handling itself and doesn't want the default CBC handling
by the framework (poor name, but left that way to avoid unecessary
differences)
This syncs with Open/FreeBSD, purpose is to allow non-CBC transforms.
Refer to ivsize instead of blocksize where appropriate.
(At this point, blocksize and ivsize are identical.)
 1.34  06-May-2011  drochner As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.
 1.33  05-May-2011  drochner fix C&P botch in diagnostic printfs
 1.32  05-May-2011  drochner support camellia-cbc as ESP cipher
 1.31  27-Mar-2011  spz fix compiling with IPSEC_DEBUG:
it's authsize not authlen in struct auth_hash
 1.30  25-Feb-2011  drochner make the use of SHA2-HMAC by FAST_IPSEC compliant to current standards:
-RFC2104 says that the block size of the hash algorithm must be used
for key/ipad/opad calculations. While formerly all ciphers used a block
length of 64, SHA384 and SHA512 use 128 bytes. So we can't use the
HMAC_BLOCK_LEN constant anymore. Add a new field to "struct auth_hash"
for the per-cipher blocksize.
-Due to this, there can't be a single "CRYPTO_SHA2_HMAC" external name
anymore. Replace this by 3 for the 3 different keysizes.
This was done by Open/FreeBSD before.
-Also fix the number of authenticator bits used tor ESP and AH to
conform to RFC4868, and remove uses of AH_HMAC_HASHLEN which did
assume a fixed authenticator size of 12 bytes.

FAST_IPSEC will not interoperate with KAME IPSEC anymore if sha2 is used,
because the latter doesn't implement these standards. It should
interoperate with at least modern Free/OpenBSD now.
(I've only tested with NetBSD-current/FAST_IPSEC on both ends.)
 1.29  19-Feb-2011  degroote Fix a missing const in FAST_IPSEC && IPSEC_DEBUG
 1.28  18-Feb-2011  drochner more "const"
 1.27  18-Feb-2011  drochner sprinkle some "const", documenting that the SA is not supposed to
change during an xform operation
 1.26  14-Feb-2011  drochner one more botched statistics counter (could increment semi-random locations)
 1.25  14-Feb-2011  drochner fix output bytecount statcounter
 1.24  14-Feb-2011  drochner change locking order, to make sure the cpu is at splsoftnet()
before the softnet_lock (adaptive) mutex is acquired, from
Wolfgang Stukenbrock, should fix a recursive lock panic
 1.23  10-Feb-2011  drochner -in opencrypto callbacks (which run in a kernel thread), pull softnet_lock
everywhere splsoftnet() was used before, to fix MP concurrency problems
-pull KERNEL_LOCK where ip(6)_output() is called, as this is what
the network stack (unfortunately) expects, in particular to avoid
races for packets in the interface send queues
From Wolfgang Stukenbrock per PR kern/44418, with the application
of KERNEL_LOCK to what I think are the essential points, tested
on a dual-core i386.
 1.22  20-Mar-2009  cegger branches: 1.22.4; 1.22.6; 1.22.8;
Correct bungled bcopy() -> memcpy() conversion
 1.21  18-Mar-2009  cegger bcopy -> memcpy
 1.20  18-Mar-2009  cegger bzero -> memset
 1.19  18-Mar-2009  cegger bcmp -> memcmp
 1.18  23-Apr-2008  thorpej branches: 1.18.2; 1.18.10; 1.18.16;
Make IPSEC and FAST_IPSEC stats per-cpu. Use <net/net_stats.h> and
netstat_sysctl().
 1.17  04-Feb-2008  tls branches: 1.17.6; 1.17.8;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.
 1.16  27-Jun-2007  degroote branches: 1.16.8; 1.16.14;
Add support for options IPSEC_NAT_T (RFC 3947 and 3948) for fast_ipsec(4).

No objection on tech-net@
 1.15  04-Mar-2007  degroote branches: 1.15.2; 1.15.4;
Remove useless cast
Use NULL instead of (void*) 0
 1.14  04-Mar-2007  degroote Fix fallout from caddr_t changes
 1.13  04-Mar-2007  christos Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.
 1.12  16-Nov-2006  christos branches: 1.12.4;
__unused removal on arguments; approved by core.
 1.11  13-Oct-2006  christos more __unused
 1.10  28-Apr-2006  pavel branches: 1.10.8; 1.10.10;
The esp_input_cb function used m_copyback, which fails if the mbuf is
read-only. This can actually happen if the packet was received by the
xennet interface, see PR kern/33162. Change it to m_copyback_cow.

AH and IPCOMP probably need similar fixes.

Requested by Jeff Rizzo, tested on Xen with -current by him.
 1.9  11-Apr-2006  rpaulo Add two new sysctls protected under IPSEC_DEBUG:

net.inet.ipsec.test_replay - When set to 1, IPsec will send packets with
the same sequence number. This allows to verify if the other side
has proper replay attacks detection.

net.inet.ipsec.test_integrity - When set 1, IPsec will send packets with
corrupted HMAC. This allows to verify if the other side properly
detects modified packets.

(a message will be printed indicating when these sysctls changed)

By Pawel Jakub Dawidek <pjd@FreeBSD.org>.
Discussed with Christos Zoulas and Jonathan Stone.
 1.8  23-Mar-2006  rpaulo FreeBSD SA-06:11 and CVE-2006-0905: update the replay sequence number
or else the anti-reply technique won't work as expected.
 1.7  11-Dec-2005  christos branches: 1.7.4; 1.7.6; 1.7.8; 1.7.10; 1.7.12;
merge ktrace-lwp.
 1.6  27-May-2005  seanb branches: 1.6.2;
- Discrepency between malloc / free types with init vector (see free
in netipsec/key.c).
- Reviewed by christos.
 1.5  17-Mar-2004  jonathan branches: 1.5.2; 1.5.4; 1.5.8; 1.5.16; 1.5.18; 1.5.20;
sys/netinet6/ip6_ecn.h is reportedly a FreeBSD-ism; NetBSD has
prototypes for the IPv6 ECN ingress/egress functions in sys/netinet/ip_ecn.h,
inside an #ifdef INET6 wrapper. So, wrap sys/netipsec ocurrences of
#include <netinet6/ip6_ecn.h>
in #ifdef __FreeBSD__/#endif, until both camps can agree on this
teensy little piece of namespace. Affects:
ipsec_output.c xform_ah.c xform_esp.c xform_ipip.c
 1.4  06-Oct-2003  tls Reversion of "netkey merge", part 2 (replacement of removed files in the
repository by christos was part 1). netipsec should now be back as it
was on 2003-09-11, with some very minor changes:

1) Some residual platform-dependent code was moved from ipsec.h to
ipsec_osdep.h; without this, IPSEC_ASSERT() was multiply defined. ipsec.h
now includes ipsec_osdep.h

2) itojun's renaming of netipsec/files.ipsec to netipsec/files.netipsec has
been left in place (it's arguable which name is less confusing but the
rename is pretty harmless).

3) Some #endif TOKEN has been replaced by #endif /* TOKEN */; #endif TOKEN
is invalid and GCC 3 won't compile it.

An i386 kernel with "options FAST_IPSEC" and "options OPENCRYPTO" now
gets through "make depend" but fails to build with errors in ip_input.c.
But it's better than it was (thank heaven for small favors).
 1.3  12-Sep-2003  itojun merge netipsec/key* into netkey/key*. no need for both.
change confusing filename
 1.2  20-Aug-2003  jonathan opt_inet6.h is FreeBSD-specific, so wrap it with #ifdef __FreeBSD__/#endif.
 1.1  13-Aug-2003  jonathan Initial import of Sam Leffler's `Fast-IPsec' from FreeBSD 4.
Fast-IPsec is a rework of the OpenBSD and KAME IPsec code, using the
OpenCryptoFramework (and thus hardware crypto accelerators) and
numerous detailed performance improvements.

This import is (aside from SPL-level names) the FreeBSD source,
imported ``as-is'' as a historical snapshot, for future maintenance
and comparison against the FreeBSD source. For now, several minor
kernel-API differences are hidden by macros a shim file, ipsec_osdep.h,
which (aside from SPL names) can be targeted at either NetBSD or FreeBSD.
 1.5.20.1  28-Mar-2006  riz Pull up following revision(s) (requested by rpaulo in ticket #1222):
sys/netipsec/xform_esp.c: revision 1.8
FreeBSD SA-06:11 and CVE-2006-0905: update the replay sequence number
or else the anti-reply technique won't work as expected.
 1.5.18.1  30-Mar-2006  riz Pull up following revision(s) (requested by rpaulo in ticket #10384):
sys/netipsec/xform_esp.c: revision 1.8
FreeBSD SA-06:11 and CVE-2006-0905: update the replay sequence number
or else the anti-reply technique won't work as expected.
 1.5.16.1  28-Mar-2006  riz Pull up following revision(s) (requested by rpaulo in ticket #1222):
sys/netipsec/xform_esp.c: revision 1.8
FreeBSD SA-06:11 and CVE-2006-0905: update the replay sequence number
or else the anti-reply technique won't work as expected.
 1.5.8.1  30-Mar-2006  riz Pull up following revision(s) (requested by rpaulo in ticket #10384):
sys/netipsec/xform_esp.c: revision 1.8
FreeBSD SA-06:11 and CVE-2006-0905: update the replay sequence number
or else the anti-reply technique won't work as expected.
 1.5.4.5  10-Nov-2005  skrll Sync with HEAD. Here we go again...
 1.5.4.4  21-Sep-2004  skrll Fix the sync with head I botched.
 1.5.4.3  18-Sep-2004  skrll Sync with HEAD.
 1.5.4.2  03-Aug-2004  skrll Sync with HEAD
 1.5.4.1  17-Mar-2004  skrll file xform_esp.c was added on branch ktrace-lwp on 2004-08-03 10:55:29 +0000
 1.5.2.1  30-Mar-2006  riz Pull up following revision(s) (requested by rpaulo in ticket #10384):
sys/netipsec/xform_esp.c: revision 1.8
FreeBSD SA-06:11 and CVE-2006-0905: update the replay sequence number
or else the anti-reply technique won't work as expected.
 1.6.2.4  04-Feb-2008  yamt sync with head.
 1.6.2.3  03-Sep-2007  yamt sync with head.
 1.6.2.2  30-Dec-2006  yamt sync with head.
 1.6.2.1  21-Jun-2006  yamt sync with head.
 1.7.12.2  24-May-2006  tron Merge 2006-05-24 NetBSD-current into the "peter-altq" branch.
 1.7.12.1  28-Mar-2006  tron Merge 2006-03-28 NetBSD-current into the "peter-altq" branch.
 1.7.10.2  11-May-2006  elad sync with head
 1.7.10.1  19-Apr-2006  elad sync with head.
 1.7.8.2  24-May-2006  yamt sync with head.
 1.7.8.1  01-Apr-2006  yamt sync with head.
 1.7.6.2  01-Jun-2006  kardel Sync with head.
 1.7.6.1  22-Apr-2006  simonb Sync with head.
 1.7.4.1  09-Sep-2006  rpaulo sync with head
 1.10.10.2  10-Dec-2006  yamt sync with head.
 1.10.10.1  22-Oct-2006  yamt sync with head
 1.10.8.1  18-Nov-2006  ad Sync with head.
 1.12.4.1  12-Mar-2007  rmind Sync with HEAD.
 1.15.4.1  11-Jul-2007  mjf Sync with head.
 1.15.2.1  15-Jul-2007  ad Sync with head.
 1.16.14.1  18-Feb-2008  mjf Sync with HEAD.
 1.16.8.1  23-Mar-2008  matt sync with HEAD
 1.17.8.1  18-May-2008  yamt sync with head.
 1.17.6.1  02-Jun-2008  mjf Sync with HEAD.
 1.18.16.1  13-May-2009  jym Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.
 1.18.10.1  28-Apr-2009  skrll Sync with HEAD.
 1.18.2.1  04-May-2009  yamt sync with head.
 1.22.8.2  05-Mar-2011  bouyer Sync with HEAD
 1.22.8.1  17-Feb-2011  bouyer Sync with HEAD
 1.22.6.1  06-Jun-2011  jruoho Sync with HEAD.
 1.22.4.3  31-May-2011  rmind sync with head
 1.22.4.2  21-Apr-2011  rmind sync with head
 1.22.4.1  05-Mar-2011  rmind sync with head
 1.39.6.1  18-Feb-2012  mrg merge to -current.
 1.39.2.3  22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.39.2.2  30-Oct-2012  yamt sync with head
 1.39.2.1  17-Apr-2012  yamt sync with head
 1.40.8.1  13-Mar-2018  snj Pull up following revision(s) (requested by maxv in ticket #1532):
sys/netipsec/xform_ah.c: 1.77 via patch
sys/netipsec/xform_esp.c: 1.73 via patch
sys/netipsec/xform_ipip.c: 1.56-1.57 via patch
Reinforce and clarify.
--
Add missing NULL check. Normally that's not triggerable remotely, since we
are guaranteed that 8 bytes are valid at mbuf+skip.
--
Fix use-after-free. There is a path where the mbuf gets pulled up without
a proper mtod afterwards:
218 ipo = mtod(m, struct ip *);
281 m = m_pullup(m, hlen);
232 ipo->ip_src.s_addr
Found by Mootja.
Meanwhile it seems to me that 'ipo' should be set to NULL if the inner
packet is IPv6, but I'll revisit that later.
--
As I said in my last commit in this file, ipo should be set to NULL;
otherwise the 'local address spoofing' check below is always wrong on
IPv6.
 1.40.6.1  13-Mar-2018  snj Pull up following revision(s) (requested by maxv in ticket #1532):
sys/netipsec/xform_ah.c: 1.77 via patch
sys/netipsec/xform_esp.c: 1.73 via patch
sys/netipsec/xform_ipip.c: 1.56-1.57 via patch
Reinforce and clarify.
--
Add missing NULL check. Normally that's not triggerable remotely, since we
are guaranteed that 8 bytes are valid at mbuf+skip.
--
Fix use-after-free. There is a path where the mbuf gets pulled up without
a proper mtod afterwards:
218 ipo = mtod(m, struct ip *);
281 m = m_pullup(m, hlen);
232 ipo->ip_src.s_addr
Found by Mootja.
Meanwhile it seems to me that 'ipo' should be set to NULL if the inner
packet is IPv6, but I'll revisit that later.
--
As I said in my last commit in this file, ipo should be set to NULL;
otherwise the 'local address spoofing' check below is always wrong on
IPv6.
 1.40.2.1  13-Mar-2018  snj Pull up following revision(s) (requested by maxv in ticket #1532):
sys/netipsec/xform_ah.c: 1.77 via patch
sys/netipsec/xform_esp.c: 1.73 via patch
sys/netipsec/xform_ipip.c: 1.56-1.57 via patch
Reinforce and clarify.
--
Add missing NULL check. Normally that's not triggerable remotely, since we
are guaranteed that 8 bytes are valid at mbuf+skip.
--
Fix use-after-free. There is a path where the mbuf gets pulled up without
a proper mtod afterwards:
218 ipo = mtod(m, struct ip *);
281 m = m_pullup(m, hlen);
232 ipo->ip_src.s_addr
Found by Mootja.
Meanwhile it seems to me that 'ipo' should be set to NULL if the inner
packet is IPv6, but I'll revisit that later.
--
As I said in my last commit in this file, ipo should be set to NULL;
otherwise the 'local address spoofing' check below is always wrong on
IPv6.
 1.41.2.3  03-Dec-2017  jdolecek update from HEAD
 1.41.2.2  20-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.41.2.1  23-Jun-2013  tls resync from head
 1.43.2.2  18-May-2014  rmind sync with head
 1.43.2.1  28-Aug-2013  rmind sync with head
 1.45.12.1  15-Feb-2018  martin Pull up following revision(s) (requested by maxv in ticket #1569):
sys/netipsec/xform_ah.c: revision 1.77, 1.81 (via patch)
sys/netipsec/xform_esp.c: revision 1.73 (via patch)
sys/netipsec/xform_ipip.c: revision 1.56, 1.57 (via patch)

Fix use-after-free. There is a path where the mbuf gets pulled up without
a proper mtod afterwards:

218 ipo = mtod(m, struct ip *);
281 m = m_pullup(m, hlen);
232 ipo->ip_src.s_addr

Found by Mootja.

Meanwhile it seems to me that 'ipo' should be set to NULL if the inner
packet is IPv6, but I'll revisit that later.

Reinforce and clarify.

Add missing NULL check. Normally that's not triggerable remotely, since we
are guaranteed that 8 bytes are valid at mbuf+skip.

As I said in my last commit in this file, ipo should be set to NULL;
otherwise the 'local address spoofing' check below is always wrong on
IPv6.

Make sure the Authentication Header fits the mbuf chain, otherwise panic.
 1.45.8.1  15-Feb-2018  martin Pull up following revision(s) (requested by maxv in ticket #1569):
sys/netipsec/xform_ah.c: revision 1.77, 1.81 (via patch)
sys/netipsec/xform_esp.c: revision 1.73 (via patch)
sys/netipsec/xform_ipip.c: revision 1.56, 1.57 (via patch)

Fix use-after-free. There is a path where the mbuf gets pulled up without
a proper mtod afterwards:

218 ipo = mtod(m, struct ip *);
281 m = m_pullup(m, hlen);
232 ipo->ip_src.s_addr

Found by Mootja.

Meanwhile it seems to me that 'ipo' should be set to NULL if the inner
packet is IPv6, but I'll revisit that later.

Reinforce and clarify.

Add missing NULL check. Normally that's not triggerable remotely, since we
are guaranteed that 8 bytes are valid at mbuf+skip.

As I said in my last commit in this file, ipo should be set to NULL;
otherwise the 'local address spoofing' check below is always wrong on
IPv6.

Make sure the Authentication Header fits the mbuf chain, otherwise panic.
 1.45.6.2  28-Aug-2017  skrll Sync with HEAD
 1.45.6.1  06-Apr-2015  skrll Sync with HEAD
 1.45.4.1  15-Feb-2018  martin Pull up following revision(s) (requested by maxv in ticket #1569):
sys/netipsec/xform_ah.c: revision 1.77, 1.81 (via patch)
sys/netipsec/xform_esp.c: revision 1.73 (via patch)
sys/netipsec/xform_ipip.c: revision 1.56, 1.57 (via patch)

Fix use-after-free. There is a path where the mbuf gets pulled up without
a proper mtod afterwards:

218 ipo = mtod(m, struct ip *);
281 m = m_pullup(m, hlen);
232 ipo->ip_src.s_addr

Found by Mootja.

Meanwhile it seems to me that 'ipo' should be set to NULL if the inner
packet is IPv6, but I'll revisit that later.

Reinforce and clarify.

Add missing NULL check. Normally that's not triggerable remotely, since we
are guaranteed that 8 bytes are valid at mbuf+skip.

As I said in my last commit in this file, ipo should be set to NULL;
otherwise the 'local address spoofing' check below is always wrong on
IPv6.

Make sure the Authentication Header fits the mbuf chain, otherwise panic.
 1.46.4.1  21-Apr-2017  bouyer Sync with HEAD
 1.46.2.1  26-Apr-2017  pgoyette Sync with HEAD
 1.54.2.1  19-May-2017  pgoyette Resolve conflicts from previous merge (all resulting from $NetBSD
keywork expansion)
 1.55.2.4  08-Oct-2020  martin Pull up following revision(s) (requested by knakahara in ticket #1612):

sys/netipsec/xform_esp.c: revision 1.101

Make sequence number of esp header MP-safe for IPsec Tx side. reviewed by ozaki-r@n.o

In IPsec Tx side, one Security Association can be used by multiple CPUs.
On the other hand, in IPsec Rx side, one Security Association is used
by only one CPU.

XXX pullup-{8,9}
 1.55.2.3  30-Mar-2018  martin Pull up following revision(s) (requested by maxv in ticket #670):

sys/netipsec/xform_esp.c: revision 1.73

Add missing NULL check. Normally that's not triggerable remotely, since we
are guaranteed that 8 bytes are valid at mbuf+skip.
 1.55.2.2  26-Feb-2018  martin Pull up following revision(s) (requested by ozaki-r in ticket #587):
sys/netipsec/xform_ipcomp.c: revision 1.54-1.56
sys/netipsec/xform_ah.c: revision 1.78,1.79(patch),1.82-1.84
sys/netipsec/xform_esp.c: revision 1.74-1.76

Fix mbuf leaks on error paths

Dedup common codes in error paths (NFCI)

Don't relook up an SP/SA in opencrpyto callbacks
We don't need to do so because we have a reference to it. And also
relooking-up one there may return an sp/sav that has different
parameters from an original one.

Fix kernel panic (assertion failure) on receiving an IPv6 packet with large options
If an IPv6 packet has large options, a necessary space for evacuation can
exceed the expected size (ah_pool_item_size). Give up using the pool_cache
if it happens.

Style.

Commonalize error paths (NFC)

Fix buffer overflow on sending an IPv6 packet with large options
If an IPv6 packet has large options, a necessary space for evacuation can
exceed the expected size (ah_pool_item_size). Give up using the pool_cache
if it happens.
Pointed out by maxv@
 1.55.2.1  21-Oct-2017  snj Pull up following revision(s) (requested by ozaki-r in ticket #300):
crypto/dist/ipsec-tools/src/setkey/parse.y: 1.19
crypto/dist/ipsec-tools/src/setkey/token.l: 1.20
distrib/sets/lists/tests/mi: 1.754, 1.757, 1.759
doc/TODO.smpnet: 1.12-1.13
sys/net/pfkeyv2.h: 1.32
sys/net/raw_cb.c: 1.23-1.24, 1.28
sys/net/raw_cb.h: 1.28
sys/net/raw_usrreq.c: 1.57-1.58
sys/net/rtsock.c: 1.228-1.229
sys/netinet/in_proto.c: 1.125
sys/netinet/ip_input.c: 1.359-1.361
sys/netinet/tcp_input.c: 1.359-1.360
sys/netinet/tcp_output.c: 1.197
sys/netinet/tcp_var.h: 1.178
sys/netinet6/icmp6.c: 1.213
sys/netinet6/in6_proto.c: 1.119
sys/netinet6/ip6_forward.c: 1.88
sys/netinet6/ip6_input.c: 1.181-1.182
sys/netinet6/ip6_output.c: 1.193
sys/netinet6/ip6protosw.h: 1.26
sys/netipsec/ipsec.c: 1.100-1.122
sys/netipsec/ipsec.h: 1.51-1.61
sys/netipsec/ipsec6.h: 1.18-1.20
sys/netipsec/ipsec_input.c: 1.44-1.51
sys/netipsec/ipsec_netbsd.c: 1.41-1.45
sys/netipsec/ipsec_output.c: 1.49-1.64
sys/netipsec/ipsec_private.h: 1.5
sys/netipsec/key.c: 1.164-1.234
sys/netipsec/key.h: 1.20-1.32
sys/netipsec/key_debug.c: 1.18-1.21
sys/netipsec/key_debug.h: 1.9
sys/netipsec/keydb.h: 1.16-1.20
sys/netipsec/keysock.c: 1.59-1.62
sys/netipsec/keysock.h: 1.10
sys/netipsec/xform.h: 1.9-1.12
sys/netipsec/xform_ah.c: 1.55-1.74
sys/netipsec/xform_esp.c: 1.56-1.72
sys/netipsec/xform_ipcomp.c: 1.39-1.53
sys/netipsec/xform_ipip.c: 1.50-1.54
sys/netipsec/xform_tcp.c: 1.12-1.16
sys/rump/librump/rumpkern/Makefile.rumpkern: 1.170
sys/rump/librump/rumpnet/net_stub.c: 1.27
sys/sys/protosw.h: 1.67-1.68
tests/net/carp/t_basic.sh: 1.7
tests/net/if_gif/t_gif.sh: 1.11
tests/net/if_l2tp/t_l2tp.sh: 1.3
tests/net/ipsec/Makefile: 1.7-1.9
tests/net/ipsec/algorithms.sh: 1.5
tests/net/ipsec/common.sh: 1.4-1.6
tests/net/ipsec/t_ipsec_ah_keys.sh: 1.2
tests/net/ipsec/t_ipsec_esp_keys.sh: 1.2
tests/net/ipsec/t_ipsec_gif.sh: 1.6-1.7
tests/net/ipsec/t_ipsec_l2tp.sh: 1.6-1.7
tests/net/ipsec/t_ipsec_misc.sh: 1.8-1.18
tests/net/ipsec/t_ipsec_sockopt.sh: 1.1-1.2
tests/net/ipsec/t_ipsec_tcp.sh: 1.1-1.2
tests/net/ipsec/t_ipsec_transport.sh: 1.5-1.6
tests/net/ipsec/t_ipsec_tunnel.sh: 1.9
tests/net/ipsec/t_ipsec_tunnel_ipcomp.sh: 1.1-1.2
tests/net/ipsec/t_ipsec_tunnel_odd.sh: 1.3
tests/net/mcast/t_mcast.sh: 1.6
tests/net/net/t_ipaddress.sh: 1.11
tests/net/net_common.sh: 1.20
tests/net/npf/t_npf.sh: 1.3
tests/net/route/t_flags.sh: 1.20
tests/net/route/t_flags6.sh: 1.16
usr.bin/netstat/fast_ipsec.c: 1.22
Do m_pullup before mtod

It may fix panicks of some tests on anita/sparc and anita/GuruPlug.
---
KNF
---
Enable DEBUG for babylon5
---
Apply C99-style struct initialization to xformsw
---
Tweak outputs of netstat -s for IPsec

- Get rid of "Fast"
- Use ipsec and ipsec6 for titles to clarify protocol
- Indent outputs of sub protocols

Original outputs were organized like this:

(Fast) IPsec:
IPsec ah:
IPsec esp:
IPsec ipip:
IPsec ipcomp:
(Fast) IPsec:
IPsec ah:
IPsec esp:
IPsec ipip:
IPsec ipcomp:

New outputs are organized like this:

ipsec:
ah:
esp:
ipip:
ipcomp:
ipsec6:
ah:
esp:
ipip:
ipcomp:
---
Add test cases for IPComp
---
Simplify IPSEC_OSTAT macro (NFC)
---
KNF; replace leading whitespaces with hard tabs
---
Introduce and use SADB_SASTATE_USABLE_P
---
KNF
---
Add update command for testing

Updating an SA (SADB_UPDATE) requires that a process issuing
SADB_UPDATE is the same as a process issued SADB_ADD (or SADB_GETSPI).
This means that update command must be used with add command in a
configuration of setkey. This usage is normally meaningless but
useful for testing (and debugging) purposes.
---
Add test cases for updating SA/SP

The tests require newly-added udpate command of setkey.
---
PR/52346: Frank Kardel: Fix checksumming for NAT-T
See XXX for improvements.
---
Remove codes for PACKET_TAG_IPSEC_IN_CRYPTO_DONE

It seems that PACKET_TAG_IPSEC_IN_CRYPTO_DONE is for network adapters
that have IPsec accelerators; a driver sets the mtag to a packet
when its device has already encrypted the packet.

Unfortunately no driver implements such offload features for long
years and seems unlikely to implement them soon. (Note that neither
FreeBSD nor Linux doesn't have such drivers.) Let's remove related
(unused) codes and simplify the IPsec code.
---
Fix usages of sadb_msg_errno
---
Avoid updating sav directly

On SADB_UPDATE a target sav was updated directly, which was unsafe.
Instead allocate another sav, copy variables of the old sav to
the new one and replace the old one with the new one.
---
Simplify; we can assume sav->tdb_xform cannot be NULL while it's valid
---
Rename key_alloc* functions (NFC)

We shouldn't use the term "alloc" for functions that just look up
data and actually don't allocate memory.
---
Use explicit_memset to surely zero-clear key_auth and key_enc
---
Make sure to clear keys on error paths of key_setsaval
---
Add missing KEY_FREESAV
---
Make sure a sav is inserted to a sah list after its initialization completes
---
Remove unnecessary zero-clearing codes from key_setsaval

key_setsaval is now used only for a newly-allocated sav. (It was
used to reset variables of an existing sav.)
---
Correct wrong assumption of sav->refcnt in key_delsah

A sav in a list is basically not to be sav->refcnt == 0. And also
KEY_FREESAV assumes sav->refcnt > 0.
---
Let key_getsavbyspi take a reference of a returning sav
---
Use time_mono_to_wall (NFC)
---
Separate sending message routine (NFC)
---
Simplify; remove unnecessary zero-clears

key_freesaval is used only when a target sav is being destroyed.
---
Omit NULL checks for sav->lft_c

sav->lft_c can be NULL only when initializing or destroying sav.
---
Omit unnecessary NULL checks for sav->sah
---
Omit unnecessary check of sav->state

key_allocsa_policy picks a sav of either MATURE or DYING so we
don't need to check its state again.
---
Simplify; omit unnecessary saidx passing

- ipsec_nextisr returns a saidx but no caller uses it
- key_checkrequest is passed a saidx but it can be gotton by
another argument (isr)
---
Fix splx isn't called on some error paths
---
Fix header size calculation of esp where sav is NULL
---
Fix header size calculation of ah in the case sav is NULL

This fix was also needed for esp.
---
Pass sav directly to opencrypto callback

In a callback, use a passed sav as-is by default and look up a sav
only if the passed sav is dead.
---
Avoid examining freshness of sav on packet processing

If a sav list is sorted (by lft_c->sadb_lifetime_addtime) in advance,
we don't need to examine each sav and also don't need to delete one
on the fly and send up a message. Fortunately every sav lists are sorted
as we need.

Added key_validate_savlist validates that each sav list is surely sorted
(run only if DEBUG because it's not cheap).
---
Add test cases for SAs with different SPIs
---
Prepare to stop using isr->sav

isr is a shared resource and using isr->sav as a temporal storage
for each packet processing is racy. And also having a reference from
isr to sav makes the lifetime of sav non-deterministic; such a reference
is removed when a packet is processed and isr->sav is overwritten by
new one. Let's have a sav locally for each packet processing instead of
using shared isr->sav.

However this change doesn't stop using isr->sav yet because there are
some users of isr->sav. isr->sav will be removed after the users find
a way to not use isr->sav.
---
Fix wrong argument handling
---
fix printf format.
---
Don't validate sav lists of LARVAL or DEAD states

We don't sort the lists so the validation will always fail.

Fix PR kern/52405
---
Make sure to sort the list when changing the state by key_sa_chgstate
---
Rename key_allocsa_policy to key_lookup_sa_bysaidx
---
Separate test files
---
Calculate ah_max_authsize on initialization as well as esp_max_ivlen
---
Remove m_tag_find(PACKET_TAG_IPSEC_PENDING_TDB) because nobody sets the tag
---
Restore a comment removed in previous

The comment is valid for the below code.
---
Make tests more stable

sleep command seems to wait longer than expected on anita so
use polling to wait for a state change.
---
Add tests that explicitly delete SAs instead of waiting for expirations
---
Remove invalid M_AUTHIPDGM check on ESP isr->sav

M_AUTHIPDGM flag is set to a mbuf in ah_input_cb. An sav of ESP can
have AH authentication as sav->tdb_authalgxform. However, in that
case esp_input and esp_input_cb are used to do ESP decryption and
AH authentication and M_AUTHIPDGM never be set to a mbuf. So
checking M_AUTHIPDGM of a mbuf on isr->sav of ESP is meaningless.
---
Look up sav instead of relying on unstable sp->req->sav

This code is executed only in an error path so an additional lookup
doesn't matter.
---
Correct a comment
---
Don't release sav if calling crypto_dispatch again
---
Remove extra KEY_FREESAV from ipsec_process_done

It should be done by the caller.
---
Don't bother the case of crp->crp_buf == NULL in callbacks
---
Hold a reference to an SP during opencrypto processing

An SP has a list of isr (ipsecrequest) that represents a sequence
of IPsec encryption/authentication processing. One isr corresponds
to one opencrypto processing. The lifetime of an isr follows its SP.

We pass an isr to a callback function of opencrypto to continue
to a next encryption/authentication processing. However nobody
guaranteed that the isr wasn't freed, i.e., its SP wasn't destroyed.

In order to avoid such unexpected destruction of isr, hold a reference
to its SP during opencrypto processing.
---
Don't make SAs expired on tests that delete SAs explicitly
---
Fix a debug message
---
Dedup error paths (NFC)
---
Use pool to allocate tdb_crypto

For ESP and AH, we need to allocate an extra variable space in addition
to struct tdb_crypto. The fixed size of pool items may be larger than
an actual requisite size of a buffer, but still the performance
improvement by replacing malloc with pool wins.
---
Don't use unstable isr->sav for header size calculations

We may need to optimize to not look up sav here for users that
don't need to know an exact size of headers (e.g., TCP segmemt size
caclulation).
---
Don't use sp->req->sav when handling NAT-T ESP fragmentation

In order to do this we need to look up a sav however an additional
look-up degrades performance. A sav is later looked up in
ipsec4_process_packet so delay the fragmentation check until then
to avoid an extra look-up.
---
Don't use key_lookup_sp that depends on unstable sp->req->sav

It provided a fast look-up of SP. We will provide an alternative
method in the future (after basic MP-ification finishes).
---
Stop setting isr->sav on looking up sav in key_checkrequest
---
Remove ipsecrequest#sav
---
Stop setting mtag of PACKET_TAG_IPSEC_IN_DONE because there is no users anymore
---
Skip ipsec_spi_*_*_preferred_new_timeout when running on qemu

Probably due to PR 43997
---
Add localcount to rump kernels
---
Remove unused macro
---
Fix key_getcomb_setlifetime

The fix adjusts a soft limit to be 80% of a corresponding hard limit.

I'm not sure the fix is really correct though, at least the original
code is wrong. A passed comb is zero-cleared before calling
key_getcomb_setlifetime, so
comb->sadb_comb_soft_addtime = comb->sadb_comb_soft_addtime * 80 / 100;
is meaningless.
---
Provide and apply key_sp_refcnt (NFC)

It simplifies further changes.
---
Fix indentation

Pointed out by knakahara@
---
Use pslist(9) for sptree
---
Don't acquire global locks for IPsec if NET_MPSAFE

Note that the change is just to make testing easy and IPsec isn't MP-safe yet.
---
Let PF_KEY socks hold their own lock instead of softnet_lock

Operations on SAD and SPD are executed via PF_KEY socks. The operations
include deletions of SAs and SPs that will use synchronization mechanisms
such as pserialize_perform to wait for references to SAs and SPs to be
released. It is known that using such mechanisms with holding softnet_lock
causes a dead lock. We should avoid the situation.
---
Make IPsec SPD MP-safe

We use localcount(9), not psref(9), to make the sptree and secpolicy (SP)
entries MP-safe because SPs need to be referenced over opencrypto
processing that executes a callback in a different context.

SPs on sockets aren't managed by the sptree and can be destroyed in softint.
localcount_drain cannot be used in softint so we delay the destruction of
such SPs to a thread context. To do so, a list to manage such SPs is added
(key_socksplist) and key_timehandler_spd deletes dead SPs in the list.

For more details please read the locking notes in key.c.

Proposed on tech-kern@ and tech-net@
---
Fix updating ipsec_used

- key_update_used wasn't called in key_api_spddelete2 and key_api_spdflush
- key_update_used wasn't called if an SP had been added/deleted but
a reply to userland failed
---
Fix updating ipsec_used; turn on when SPs on sockets are added
---
Add missing IPsec policy checks to icmp6_rip6_input

icmp6_rip6_input is quite similar to rip6_input and the same checks exist
in rip6_input.
---
Add test cases for setsockopt(IP_IPSEC_POLICY)
---
Don't use KEY_NEWSP for dummy SP entries

By the change KEY_NEWSP is now not called from softint anymore
and we can use kmem_zalloc with KM_SLEEP for KEY_NEWSP.
---
Comment out unused functions
---
Add test cases that there are SPs but no relevant SAs
---
Don't allow sav->lft_c to be NULL

lft_c of an sav that was created by SADB_GETSPI could be NULL.
---
Clean up clunky eval strings

- Remove unnecessary \ at EOL
- This allows to omit ; too
- Remove unnecessary quotes for arguments of atf_set
- Don't expand $DEBUG in eval
- We expect it's expanded on execution

Suggested by kre@
---
Remove unnecessary KEY_FREESAV in an error path

sav should be freed (unreferenced) by the caller.
---
Use pslist(9) for sahtree
---
Use pslist(9) for sah->savtree
---
Rename local variable newsah to sah

It may not be new.
---
MP-ify SAD slightly

- Introduce key_sa_mtx and use it for some list operations
- Use pserialize for some list iterations
---
Introduce KEY_SA_UNREF and replace KEY_FREESAV with it where sav will never be actually freed in the future

KEY_SA_UNREF is still key_freesav so no functional change for now.

This change reduces diff of further changes.
---
Remove out-of-date log output

Pointed out by riastradh@
---
Use KDASSERT instead of KASSERT for mutex_ownable

Because mutex_ownable is too heavy to run in a fast path
even for DIAGNOSTIC + LOCKDEBUG.

Suggested by riastradh@
---
Assemble global lists and related locks into cache lines (NFCI)

Also rename variable names from *tree to *list because they are
just lists, not trees.

Suggested by riastradh@
---
Move locking notes
---
Update the locking notes

- Add locking order
- Add locking notes for misc lists such as reglist
- Mention pserialize, key_sp_ref and key_sp_unref on SP operations

Requested by riastradh@
---
Describe constraints of key_sp_ref and key_sp_unref

Requested by riastradh@
---
Hold key_sad.lock on SAVLIST_WRITER_INSERT_TAIL
---
Add __read_mostly to key_psz

Suggested by riastradh@
---
Tweak wording (pserialize critical section => pserialize read section)

Suggested by riastradh@
---
Add missing mutex_exit
---
Fix setkey -D -P outputs

The outputs were tweaked (by me), but I forgot updating libipsec
in my local ATF environment...
---
MP-ify SAD (key_sad.sahlist and sah entries)

localcount(9) is used to protect key_sad.sahlist and sah entries
as well as SPD (and will be used for SAD sav).

Please read the locking notes of SAD for more details.
---
Introduce key_sa_refcnt and replace sav->refcnt with it (NFC)
---
Destroy sav only in the loop for DEAD sav
---
Fix KASSERT(solocked(sb->sb_so)) failure in sbappendaddr that is called eventually from key_sendup_mbuf

If key_sendup_mbuf isn't passed a socket, the assertion fails.
Originally in this case sb->sb_so was softnet_lock and callers
held softnet_lock so the assertion was magically satisfied.
Now sb->sb_so is key_so_mtx and also softnet_lock isn't always
held by callers so the assertion can fail.

Fix it by holding key_so_mtx if key_sendup_mbuf isn't passed a socket.

Reported by knakahara@
Tested by knakahara@ and ozaki-r@
---
Fix locking notes of SAD
---
Fix deadlock between key_sendup_mbuf called from key_acquire and localcount_drain

If we call key_sendup_mbuf from key_acquire that is called on packet
processing, a deadlock can happen like this:
- At key_acquire, a reference to an SP (and an SA) is held
- key_sendup_mbuf will try to take key_so_mtx
- Some other thread may try to localcount_drain to the SP with
holding key_so_mtx in say key_api_spdflush
- In this case localcount_drain never return because key_sendup_mbuf
that has stuck on key_so_mtx never release a reference to the SP

Fix the deadlock by deferring key_sendup_mbuf to the timer
(key_timehandler).
---
Fix that prev isn't cleared on retry
---
Limit the number of mbufs queued for deferred key_sendup_mbuf

It's easy to be queued hundreds of mbufs on the list under heavy
network load.
---
MP-ify SAD (savlist)

localcount(9) is used to protect savlist of sah. The basic design is
similar to MP-ifications of SPD and SAD sahlist. Please read the
locking notes of SAD for more details.
---
Simplify ipsec_reinject_ipstack (NFC)
---
Add per-CPU rtcache to ipsec_reinject_ipstack

It reduces route lookups and also reduces rtcache lock contentions
when NET_MPSAFE is enabled.
---
Use pool_cache(9) instead of pool(9) for tdb_crypto objects

The change improves network throughput especially on multi-core systems.
---
Update

ipsec(4), opencrypto(9) and vlan(4) are now MP-safe.
---
Write known issues on scalability
---
Share a global dummy SP between PCBs

It's never be changed so it can be pre-allocated and shared safely between PCBs.
---
Fix race condition on the rawcb list shared by rtsock and keysock

keysock now protects itself by its own mutex, which means that
the rawcb list is protected by two different mutexes (keysock's one
and softnet_lock for rtsock), of course it's useless.

Fix the situation by having a discrete rawcb list for each.
---
Use a dedicated mutex for rt_rawcb instead of softnet_lock if NET_MPSAFE
---
fix localcount leak in sav. fixed by ozaki-r@n.o.

I commit on behalf of him.
---
remove unnecessary comment.
---
Fix deadlock between pserialize_perform and localcount_drain

A typical ussage of localcount_drain looks like this:

mutex_enter(&mtx);
item = remove_from_list();
pserialize_perform(psz);
localcount_drain(&item->localcount, &cv, &mtx);
mutex_exit(&mtx);

This sequence can cause a deadlock which happens for example on the following
situation:

- Thread A calls localcount_drain which calls xc_broadcast after releasing
a specified mutex
- Thread B enters the sequence and calls pserialize_perform with holding
the mutex while pserialize_perform also calls xc_broadcast
- Thread C (xc_thread) that calls an xcall callback of localcount_drain tries
to hold the mutex

xc_broadcast of thread B doesn't start until xc_broadcast of thread A
finishes, which is a feature of xcall(9). This means that pserialize_perform
never complete until xc_broadcast of thread A finishes. On the other hand,
thread C that is a callee of xc_broadcast of thread A sticks on the mutex.
Finally the threads block each other (A blocks B, B blocks C and C blocks A).

A possible fix is to serialize executions of the above sequence by another
mutex, but adding another mutex makes the code complex, so fix the deadlock
by another way; the fix is to release the mutex before pserialize_perform
and instead use a condvar to prevent pserialize_perform from being called
simultaneously.

Note that the deadlock has happened only if NET_MPSAFE is enabled.
---
Add missing ifdef NET_MPSAFE
---
Take softnet_lock on pr_input properly if NET_MPSAFE

Currently softnet_lock is taken unnecessarily in some cases, e.g.,
icmp_input and encap4_input from ip_input, or not taken even if needed,
e.g., udp_input and tcp_input from ipsec4_common_input_cb. Fix them.

NFC if NET_MPSAFE is disabled (default).
---
- sanitize key debugging so that we don't print extra newlines or unassociated
debugging messages.
- remove unused functions and make internal ones static
- print information in one line per message
---
humanize printing of ip addresses
---
cast reduction, NFC.
---
Fix typo in comment
---
Pull out ipsec_fill_saidx_bymbuf (NFC)
---
Don't abuse key_checkrequest just for looking up sav

It does more than expected for example key_acquire.
---
Fix SP is broken on transport mode

isr->saidx was modified accidentally in ipsec_nextisr.

Reported by christos@
Helped investigations by christos@ and knakahara@
---
Constify isr at many places (NFC)
---
Include socketvar.h for softnet_lock
---
Fix buffer length for ipsec_logsastr
 1.79.2.4  25-Jun-2018  pgoyette Sync with HEAD
 1.79.2.3  21-May-2018  pgoyette Sync with HEAD
 1.79.2.2  02-May-2018  pgoyette Synch with HEAD
 1.79.2.1  22-Apr-2018  pgoyette Sync with HEAD
 1.96.2.2  13-Apr-2020  martin Mostly merge changes from HEAD upto 20200411
 1.96.2.1  10-Jun-2019  christos Sync with HEAD
 1.98.2.1  08-Oct-2020  martin Pull up following revision(s) (requested by knakahara in ticket #1103):

sys/netipsec/xform_esp.c: revision 1.101

Make sequence number of esp header MP-safe for IPsec Tx side. reviewed by ozaki-r@n.o

In IPsec Tx side, one Security Association can be used by multiple CPUs.
On the other hand, in IPsec Rx side, one Security Association is used
by only one CPU.

XXX pullup-{8,9}

RSS XML Feed