TODO.smpnet revision 1.25 
1$NetBSD: TODO.smpnet,v 1.25 2018/09/23 13:48:16 maxv Exp $
2
3MP-safe components
4==================
5
6They work without the big kernel lock (KERNEL_LOCK), i.e., with NET_MPSAFE
7kernel option.  Some components scale up and some don't.
8
9 - Device drivers
10   - vioif(4)
11   - vmx(4)
12   - wm(4)
13   - ixg(4)
14   - ixv(4)
15 - Layer 2
16   - Ethernet (if_ethersubr.c)
17   - bridge(4)
18     - STP
19   - Fast forward (ipflow)
20 - Layer 3
21   - All except for items in the below section
22 - Interfaces
23   - gif(4)
24   - ipsecif(4)
25   - l2tp(4)
26   - pppoe(4)
27     - if_spppsubr.c
28   - tun(4)
29   - vlan(4)
30 - Packet filters
31   - npf(7)
32 - Others
33   - bpf(4)
34   - ipsec(4)
35   - opencrypto(9)
36   - pfil(9)
37
38Non MP-safe components and kernel options
39=========================================
40
41The components and options aren't MP-safe, i.e., requires the big kernel lock,
42yet.  Some of them can be used safely even if NET_MPSAFE is enabled because
43they're still protected by the big kernel lock.  The others aren't protected and
44so unsafe, e.g, they may crash the kernel.
45
46Protected ones
47--------------
48
49 - Device drivers
50   - Most drivers other than ones listed in the above section
51 - Layer 4
52   - DCCP
53   - SCTP
54   - TCP
55   - UDP
56
57Unprotected ones
58----------------
59
60 - Layer 2
61   - ARCNET (if_arcsubr.c)
62   - BRIDGE_IPF
63   - FDDI (if_fddisubr.c)
64   - HIPPI (if_hippisubr.c)
65   - IEEE 1394 (if_ieee1394subr.c)
66   - IEEE 802.11 (ieee80211(4))
67   - Token ring (if_tokensubr.c)
68 - Layer 3
69   - IPSELSRC
70   - MROUTING
71   - PIM
72   - MPLS (mpls(4))
73   - IPv6 address selection policy
74 - Interfaces
75   - agr(4)
76   - carp(4)
77   - faith(4)
78   - gre(4)
79   - ppp(4)
80   - sl(4)
81   - stf(4)
82   - strip(4)
83   - if_srt
84   - tap(4)
85 - Packet filters
86   - ipf(4)
87   - pf(4)
88 - Others
89   - AppleTalk (sys/netatalk/)
90   - Bluetooth (sys/netbt/)
91   - altq(4)
92   - CIFS (sys/netsmb/)
93   - kttcp(4)
94   - NFS
95
96Know issues
97===========
98
99NOMPSAFE
100--------
101
102We use "NOMPSAFE" as a mark that indicates that the code around it isn't MP-safe
103yet.  We use it in comments and also use as part of function names, for example
104m_get_rcvif_NOMPSAFE.  Let's use "NOMPSAFE" to make it easy to find non-MP-safe
105codes by grep.
106
107bpf
108---
109
110MP-ification of bpf requires all of bpf_mtap* are called in normal LWP context
111or softint context, i.e., not in hardware interrupt context.  For Tx, all
112bpf_mtap satisfy the requrement.  For Rx, most of bpf_mtap are called in softint.
113Unfortunately some bpf_mtap on Rx are still called in hardware interrupt context.
114
115This is the list of the functions that have such bpf_mtap:
116
117 - sca_frame_process() @ sys/dev/ic/hd64570.c
118 - rxintr_cleanup() @ sys/dev/pci/if_lmc.c
119
120Ideally we should make the functions run in softint somehow, but we don't have
121actual devices, no time (or interest/love) to work on the task, so instead we
122provide a deferred bpf_mtap mechanism that forcibly runs bpf_mtap in softint
123context.  It's a workaround and once the functions run in softint, we should use
124the original bpf_mtap again.
125
126Lingering obsolete variables
127-----------------------------
128
129Some obsolete global variables and member variables of structures remain to
130avoid breaking old userland programs which directly access such variables via
131kvm(3).
132
133The following programs still use kvm(3) to get some information related to
134the network stack.
135
136 - netstat(1)
137 - vmstat(1)
138 - fstat(1)
139
140netstat(1) accesses ifnet_list, the head of a list of interface objects
141(struct ifnet), and traverses each object through ifnet#if_list member variable.
142ifnet_list and ifnet#if_list is obsoleted by ifnet_pslist and
143ifnet#if_pslist_entry respectively. netstat also accesses the IP address list
144of an interface throught ifnet#if_addrlist. struct ifaddr, struct in_ifaddr
145and struct in6_ifaddr are accessed and the following obsolete member variables
146are stuck: ifaddr#ifa_list, in_ifaddr#ia_hash, in_ifaddr#ia_list,
147in6_ifaddr#ia_next and in6_ifaddr#_ia6_multiaddrs. Note that netstat already
148implements alternative methods to fetch the above information via sysctl(3).
149
150vmstat(1) shows statistics of hash tables created by hashinit(9) in the kernel.
151The statistic information is retrieved via kvm(3). The global variables
152in_ifaddrhash and in_ifaddrhashtbl, which are for a hash table of IPv4
153addresses and obsoleted by in_ifaddrhash_pslist and in_ifaddrhashtbl_pslist,
154are kept for this purpose. We should provide a means to fetch statistics of
155hash tables via sysctl(3).
156
157fstat(1) shows information of bpf instances. Each bpf instance (struct bpf) is
158obtained via kvm(3). bpf_d#_bd_next, bpf_d#_bd_filter and bpf_d#_bd_list
159member variables are obsolete but remain. ifnet#if_xname is also accessed
160via struct bpf_if and obsolete ifnet#if_list is required to remain to not change
161the offset of ifnet#if_xname. The statistic counters (bpf#bd_rcount,
162bpf#bd_dcount and bpf#bd_ccount) are also victims of this restriction; for
163scalability the statistic counters should be per-CPU and we should stop using
164atomic operations for them however we have to remain the counters and atomic
165operations.
166
167Scalability
168-----------
169
170 - Per-CPU rtcaches (used in say IP forwarding) aren't scalable on multiple
171   flows per CPU
172 - ipsec(4) isn't scalable on the number of SA/SP; the cost of a look-up
173   is O(n)
174 - opencrypto(9)'s crypto_newsession()/crypto_freesession() aren't scalable
175   as they are serialized by one mutex
176
177ec_multi* of ethercom
178---------------------
179
180ec_multiaddrs and ec_multicnt of struct ethercom and items listed in
181ec_multiaddrs must be protected by ec_lock.  The core of ethernet subsystem is
182already MP-safe, however, device drivers that use the data should also be fixed.
183A typical change should be to protect manipulations of the data via ETHER_*
184macros such as ETHER_FIRST_MULTI by ETHER_LOCK and ETHER_UNLOCK.
185
186ALTQ
187----
188
189If ALTQ is enabled in the kernel, it enforces to use just one Tx queue (if_snd)
190for packet transmissions, resulting in serializing all Tx packet processing on
191the queue.  We should probably design and implement an alternative queuing
192mechanism that deals with multi-core systems at the first place, not making the
193existing ALTQ MP-safe because it's just annoying.
194