xref: /src/doc/TODO.smpnet

$NetBSD: TODO.smpnet,v 1.51 2025/06/17 02:00:25 ozaki-r Exp $

MP-safe components
==================

They work without the big kernel lock (KERNEL_LOCK), i.e., with NET_MPSAFE
kernel option.  Some components scale up and some don't.

 - Device drivers
   - aq(4)
   - awge(4)
   - bcmgenet(4)
   - bge(4)
   - ena(4)
   - iavf(4)
   - ixg(4)
   - ixl(4)
   - ixv(4)
   - mcx(4)
   - rge(4)
   - se(4)
   - sunxi_emac(4)
   - vioif(4)
   - vmx(4)
   - wm(4)
   - xennet(4)
   - usbnet(4) based adapters:
     - axe(4)
     - axen(4)
     - cdce(4)
     - cue(4)
     - kue(4)
     - mos(4)
     - mue(4)
     - smsc(4)
     - udav(4)
     - upl(4)
     - ure(4)
     - url(4)
     - urndis(4)
 - Layer 2
   - Ethernet (if_ethersubr.c)
   - bridge(4)
     - STP
   - Fast forward (ipflow)
 - Layer 3
   - All except for items in the below section
 - Interfaces
   - canloop(4)
   - gif(4)
   - ipsecif(4)
   - l2tp(4)
   - lagg(4)
   - pppoe(4)
     - if_spppsubr.c
   - tap(4)
   - tun(4)
   - vether(4)
   - vlan(4)
 - Packet filters
   - npf(7)
   - ipf(4)
 - Others
   - bpf(4)
   - ipsec(4)
   - opencrypto(9)
   - pfil(9)

Non MP-safe components and kernel options
=========================================

The components and options aren't MP-safe, i.e., requires the big kernel lock,
yet.  Some of them can be used safely even if NET_MPSAFE is enabled because
they're still protected by the big kernel lock.  The others aren't protected and
so unsafe, e.g, they may crash the kernel.

Protected ones
--------------

 - Device drivers
   - Most drivers other than ones listed in the above section
 - Layer 4
   - DCCP
   - SCTP
   - TCP
   - UDP

Unprotected ones
----------------

 - Layer 2
   - ARCNET (if_arcsubr.c)
   - IEEE 1394 (if_ieee1394subr.c)
   - IEEE 802.11 (ieee80211(4))
 - Layer 3
   - IPSELSRC
   - MROUTING
   - PIM
   - MPLS (mpls(4))
   - IPv6 address selection policy
 - Interfaces
   - agr(4)
   - carp(4)
   - faith(4)
   - gre(4)
   - ppp(4)
   - sl(4)
   - stf(4)
   - if_srt
 - Packet filters
   - pf(4)
 - Others
   - AppleTalk (sys/netatalk/)
   - Bluetooth (sys/netbt/)
   - altq(4)
   - kttcp(4)
   - NFS

Know issues
===========

NOMPSAFE
--------

We use "NOMPSAFE" as a mark that indicates that the code around it isn't MP-safe
yet.  We use it in comments and also use as part of function names, for example
m_get_rcvif_NOMPSAFE.  Let's use "NOMPSAFE" to make it easy to find non-MP-safe
codes by grep.

bpf
---

MP-ification of bpf requires all of bpf_mtap* are called in normal LWP context
or softint context, i.e., not in hardware interrupt context.  For Tx, all
bpf_mtap satisfy the requirement.  For Rx, most of bpf_mtap are called in softint.
Unfortunately some bpf_mtap on Rx are still called in hardware interrupt context.

This is the list of the functions that have such bpf_mtap:

 - sca_frame_process() @ sys/dev/ic/hd64570.c

Ideally we should make the functions run in softint somehow, but we don't have
actual devices, no time (or interest/love) to work on the task, so instead we
provide a deferred bpf_mtap mechanism that forcibly runs bpf_mtap in softint
context.  It's a workaround and once the functions run in softint, we should use
the original bpf_mtap again.

if_mcast_op() - SIOCADDMULTI/SIOCDELMULTI
-----------------------------------------
Helper function is called to add or remove multicast addresses for
interface.  When called via ioctl it takes IFNET_LOCK(), when called
via sosetopt() it doesn't.

Various network drivers can't assert IFNET_LOCKED() in their if_ioctl
because of this. Generally drivers still take care to splnet() even
with NET_MPSAFE before calling ether_ioctl(), but they do not take
KERNEL_LOCK(), so this is actually unsafe.

Lingering obsolete variables
-----------------------------

Some obsolete global variables and member variables of structures remain to
avoid breaking old userland programs which directly access such variables via
kvm(3).

The following programs still use kvm(3) to get some information related to
the network stack.

 - netstat(1)
 - vmstat(1)
 - fstat(1)

netstat(1) accesses ifnet_list, the head of a list of interface objects
(struct ifnet), and traverses each object through ifnet#if_list member variable.
ifnet_list and ifnet#if_list is obsoleted by ifnet_pslist and
ifnet#if_pslist_entry respectively. netstat also accesses the IP address list
of an interface through ifnet#if_addrlist. struct ifaddr, struct in_ifaddr
and struct in6_ifaddr are accessed and the following obsolete member variables
are stuck: ifaddr#ifa_list, in_ifaddr#ia_hash, in_ifaddr#ia_list,
in6_ifaddr#ia_next and in6_ifaddr#_ia6_multiaddrs. Note that netstat already
implements alternative methods to fetch the above information via sysctl(3).

vmstat(1) shows statistics of hash tables created by hashinit(9) in the kernel.
The statistic information is retrieved via kvm(3). The global variables
in_ifaddrhash and in_ifaddrhashtbl, which are for a hash table of IPv4
addresses and obsoleted by in_ifaddrhash_pslist and in_ifaddrhashtbl_pslist,
are kept for this purpose. We should provide a means to fetch statistics of
hash tables via sysctl(3).

fstat(1) shows information of bpf instances. Each bpf instance (struct bpf) is
obtained via kvm(3). bpf_d#_bd_next, bpf_d#_bd_filter and bpf_d#_bd_list
member variables are obsolete but remain. ifnet#if_xname is also accessed
via struct bpf_if and obsolete ifnet#if_list is required to remain to not change
the offset of ifnet#if_xname. The statistic counters (bpf#bd_rcount,
bpf#bd_dcount and bpf#bd_ccount) are also victims of this restriction; for
scalability the statistic counters should be per-CPU and we should stop using
atomic operations for them however we have to remain the counters and atomic
operations.

Scalability
-----------

 - Per-CPU rtcaches (used in say IP forwarding) aren't scalable on multiple
   flows per CPU
 - ipsec(4) isn't scalable on the number of SA/SP; the cost of a look-up
   is O(n)
 - opencrypto(9)'s crypto_newsession()/crypto_freesession() aren't scalable
   as they are serialized by one mutex

ALTQ
----

If ALTQ is enabled in the kernel, it enforces to use just one Tx queue (if_snd)
for packet transmissions, resulting in serializing all Tx packet processing on
the queue.  We should probably design and implement an alternative queuing
mechanism that deals with multi-core systems at the first place, not making the
existing ALTQ MP-safe because it's just annoying.

Using kernel modules
--------------------

Please note that if you enable NET_MPSAFE in your kernel, and you use and
loadable kernel modules (including compat_xx modules or individual network
interface if_xxx device driver modules), you will need to build custom
modules.  For each module you will need to add the following line to its
Makefile:

	CPPFLAGS+=	NET_MPSAFE

Failure to do this may result in unpredictable behavior.

IPv4 address initialization atomicity
-------------------------------------

An IPv4 address is referenced by several data structures: an associated
interface, its local route, a connected route (if necessary), the global list,
the global hash table, etc.  These data structures are not updated atomically,
i.e., there can be inconsistent states on an IPv4 address in the kernel during
the initialization of an IPv4 address.

One known failure of the issue is that incoming packets destinating to an
initializing address can loop in the network stack in a short period of time.
The address initialization creates an local route first and then registers an
initializing address to the global hash table that is used to decide if an
incoming packet destinates to the host by checking the destination of the packet
is registered to the hash table.  So, if the host allows forwarding, an incoming
packet can match on a local route of an initializing address at ip_output while
it fails the to-self check described above at ip_input.  Because a matched local
route points a loopback interface as its destination interface, an incoming
packet sends to the network stack (ip_input) again, which results in looping.
The loop stops once an initializing address is registered to the hash table.

One solution of the issue is to reorder the address initialization instructions,
first register an address to the hash table then create its routes.  Another
solution is to use the routing table for the to-self check instead of using the
global hash table, like IPv6.

if_flags
--------

To avoid data race on if_flags it should be protected by a lock (currently it's
IFNET_LOCK).  Thus, if_flags should not be accessed on packet processing to
avoid performance degradation by lock contentions.  Traditionally IFF_RUNNING,
IFF_UP and IFF_OACTIVE flags of if_flags are checked on packet processing.  If
you make a driver MP-safe you must remove such checks.

Drivers should not touch IFF_ALLMULTI.  They are tempted to do so when updating
hardware multicast filters on SIOCADDMULTI/SIOCDELMULTI.  Instead, they should
use the ETHER_F_ALLMULTI bit in struct ethercom::ec_flags, under ETHER_LOCK.
ether_ioctl takes care of presenting IFF_ALLMULTI according to the current state
of ETHER_F_ALLMULTI when queried with SIOCGIFFLAGS.

Also IFF_PROMISC is checked in ether_input and we should get rid of it somehow.

Too many kpreempt_disable/kpreempt_enable
-----------------------------------------

Packet counters in the network stack such as if_statadd() and ip_statinc() are
implemented with percpu(9) to avoid atomic operations.  The implementation seems
good for scalability, however, it introduces another issue.  Since percpu(9)
requires kpreempt_{dis,en}able() for each per-cpu operation, we have to call
them for each packet counting.  An observation shows that
kpreempt_{dis,en}able()s over 10 times are called for each packet on forwarding.
For better performance on a single flow, we should reduce per-packet operations
as much as possible.

One possible solution for the issue is to make the whole network stack
non-preemptive so that we don't need to kpreempt_{dis,en}able()s for each packet
counting.