Remove KERN.ioconf, ksyms does not really need it.

generate ioconf.h for pseudo-device attach prototype

Rump is living up to its name

- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

branches: 1.1.2;
On the one thousand and ninth day rump's mainbus was moved from
rumpdev to rumpkern, liberating all rumpnet users from the need to
-lrumpdev -lrumpvfs just because a loopback interface is mandatory.

Rename rumpdev/autoconf.c to rumpkern/rump_autoconf.c to avoid
accidentally picking up e.g. sys/arch/amd64/amd64/autoconf.c through
make's .PATH.
Move rumpdev/MAINBUS.ioconf to rumpkern.

Install rump libraries and utilities to the base system and remove the
private non-installed build infrastructure from sys/rump.

breakdown of commit:
* install relevant headers into /usr/include/rump
* build sys/rump/librump/rumpuser and sys/rump/librump/rumpkern
from src/lib and install as librumpuser and librump, respectively
+ this retains the ability to test a librump build with just the
kernel sources at hand
* move sys/rump/fs/lib/libukfs and sys/rump/fs/lib/libp2k to src/lib
for general consumption, they are not kernel-space dwellers anyway
* build and install sys/rump/fs/lib/lib$fs as librumpfs_$fs
* add chapter 3 manual pages for rump, rumpuser, ukfs and p2k
* build and install userspace kernel file system daemons if MKPUFFS=yes
is spexified
* retire fsconsole for now, it will make a comeback with an actually
implemented version shortly

blkset.S is needed on vax by some macros, so include it in the build.

You must untypo what you have typoed: rumpnode_if.h -> rumpvnode_if.h

vfs_subr2 has lost its will to live. vfs_subr was originally split
into two parts so that some of the routines could be used by rump.
Now that rump uses both vfs_subr and vfs_subr2 and there is no
reason to keep two files lying around, re-unite them.

branches: 1.39.2;
Add dependencies on the input files + Makefile for the auto-generated
files rumpdefs.h, rumpnode_if.h, and rumpvnode_if.c. This is so that
an UPDATE build has a chance of succeeding.

Discussed with pooka@, and he wasn't totally against adding this...

Fixes to build rump utilities as host binaries on Linux by removing
sys namespace pollution which has crept in.

Submitted in private mail by takemura, domain ca2.so-net.ne.jp

Don't build rb.c anymore since it's in libc now.

Don't compile kern_lock for rump any more, it's no longer required.
Allows us to get rid of the incorrect _RUMPKERNEL ifdefs outside sys/rump.

branches: 1.35.2;
Make it build again.

Specify RUMP_WITHOUT_THREADS only in one place.

branches: 1.33.2;
Pull in the kernel module stuff so that rump can set up built-in modules.

branches: 1.32.2; 1.32.4;
Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

Compile rump_syscalls, use them in a few places.

Backup some fixes for recent breakage from local tree. Also some
other improvements such as exporting the real kernel namei and
using that in ukfs instead of the homegrown heap'o hacks namei.
"etcetc".

branches: 1.29.2; 1.29.6;
compile genfs_vfsops.c (hi dh2 ;)

Use vfs_subr.c from sys/kern. This brings differences in the vnode
life cycle between rump and a real kernel to a minimum.

Include vfs_getcwd.c too. It isn't used currently, but it's linked.

Use namei() etc. from kernel sources instead of a reimplementation.
To accommodate, give the rootvnode its own vnode op vector with a
simple lookup operation. This is used for looking up the file
system's device vnode instead of doing that directly in a homesmoked
namei().

Add the ability to run puffs in userspace. This means that puffs
can now be developed in userspace using puffs for development
(I hate emulators, they are annoyingly clumsy).

To e.g. mount psshfs using puffs-on-puffs, run fs/bin/syspuffs/syspuffs
with the regular mount_psshfs command line as an argument:

golem> ./syspuffs /usr/sbin/mount_psshfs ftp.netbsd.org:/pub /puffs

This will make the mount appear as usual, with the exception that the
requests will be passed through puffs both in the kernel and userspace:

ftp.netbsd.org:/pub on /puffs type puffs|p2k|puffs|psshfs

Build intr.c.

pull in atomic ops from vmlocking2

branches: 1.22.2; 1.22.6;
Temporarily make 'atomic_add_int()' a non-atomic add so the build suceeds.
At some point the functions will (presumably) make their way into either
libc or libutil.

branches: 1.21.2;
Include bufq priority implementations also and link librump with
--whole-archive since the bufq priorities use link sets.

Make it possible to run rumps multithreaded. This brings real
locking and makes it possible to run file systems which create
threads. It also makes rump file system behaviour better match
file system behaviour in the kernel.

branches: 1.19.2;
RUMUSEROBJDIR -> RUMPUSEROBJDIR (those cold-blooded old times ...)

branches: 1.18.2;
Compile most genfs routines directly out of the kernel sources.
Move special I/O routines to genfs_io.c to mirror kernel naming.

branches: 1.17.2;
create generated files before running makedepend

no need to include param.c twice in the source list

branches: 1.15.2; 1.15.4;
split implemented lock stuff out of lock_stub.c into lock.c

Don't play rename & typecast games with kauth_cred_t, but rather
declare the type in rump.h only if necessary with the help of ifdef
magic.

Implement credentials. Access control is now similar to if the file
system were run in the kernel.

branches: 1.12.2;
Hide NetBSD kernel headers completely from ukfs. This includes creating
accessors for:
* struct mount & VFS ops
* struct uio
* struct vnode
* struct vattr

and some namespace games for:
* namei flags
* VOPs
* enum vtype

Also, split rump services into two categories: library private and public
(rump_private.h and rump.h, respectively).

As a result, it is now possible to compile and use the NetBSD kernel
file systems on Linux (and probably other systems too with very
little work), although the makefiles need a bit of work to make it
a pleasureable experience.

Namespace management: do not call VOP_FOO directly from ukfs or
p2k, but rather make the calls go through librump. This avoids
having to include NetBSD kernel headers in userspace programs.
Stay tuned for some more mods of the same sort ...

jump through rename hoops (hi ad!)

branches: 1.9.2;
Kill handrolled buffercache and use vfs_bio from the kernel. This is
mostly to get the flag jungle in sync with the kernel.

compile in clock_subr.c

Use supermarket variety vfs_vnops.c instead of homecooked routines
(that's actually a good thing here).

compile in libkern __assert.c, argument order differs from libc model

Play catchup with ufs/ffs changes: compile subr_specificdata.c
and emulate kmem_foo() to support the prior.

* compile subr_time.c from kern/
* some pmap stubs

* compile param.c
* move opts to a separate directory

flip order of includes to get NETBSDSRCDIR. from agc

branches: 1.1.2;
Introduce RUMPs - Runnable Userspace Meta-Programs

/sys/rump contains programs which run unmodified kernel code in an
emulated userspace environment. The kernel environment is provided
by librump. Currently supported are a number of file systems,
which by using puffs integrate seamlessly into the system and
provide a similar user experience to if the code was running as
part of the kernel. Potential future rumpification targets include
for example parts of the networking stack and some device drivers.

This work was supported by Google Summer of Code 2007.

kern: Move some purely arithmetic routines to subr_time_arith.c.

Preparation for testing and fixing:

PR kern/58922: itimer(9): arithmetic overflow
PR kern/58925: itimer(9) responds erratically to clock wound back
PR kern/58926: itimer(9) integer overflow in overrun counting
PR kern/58927: itimer(9): overrun accounting is broken

branches: 1.191.6;
bsd.own.mk: rename GCC_NO_* to CC_WNO_*

Rename compiler-warning-disable variables from
GCC_NO_warning
to
CC_WNO_warning
where warning is the full warning name as used by the compiler.

GCC_NO_IMPLICIT_FALLTHRU is CC_WNO_IMPLICIT_FALLTHROUGH

Using the convention CC_compilerflag, where compilerflag
is based on the full compiler flag name.

rump: Move vnode_if.c from rumpkern to rumpvfs.

This has become increasingly less of a `fully dynamic interface', and
the need for it in the rest of sys/kern/ has been obviated, so let's
put it where it belongs in rumpvfs.

secmodel_extensions: Split vfs part into separate .c file.

This way we can provide weak rumpkern stubs that don't require
rumpvfs for things that are relevant to vfs, but if you do link
rumpvfs then you get the same logic in secmodel extensions.

include the extensions secmodel

Sort includes, nix trailing whitespace, fix comment.

Move address hashing from init_main.c to kern_sysctl.c.

This way rump gets it automatically. Make sure blake2s is in
librumpkern.so, not just in librumpkern_crypto.so, for this to work.

New system call getrandom() compatible with Linux and others.

Three ways to call:

getrandom(p, n, 0) Blocks at boot until full entropy.
Returns up to n bytes at p; guarantees
up to 256 bytes even if interrupted
after blocking. getrandom(0,0,0)
serves as an entropy barrier: return
only after system has full entropy.

getrandom(p, n, GRND_INSECURE) Never blocks. Guarantees up to 256
bytes even if interrupted. Equivalent
to /dev/urandom. Safe only after
successful getrandom(...,0),
getrandom(...,GRND_RANDOM), or read
from /dev/random.

getrandom(p, n, GRND_RANDOM) May block at any time. Returns up to n
bytes at p, but no guarantees about how
many -- may return as short as 1 byte.
Equivalent to /dev/random. Legacy.
Provided only for source compatibility
with Linux.

Can also use flags|GRND_NONBLOCK to fail with EWOULDBLOCK/EAGAIN
without producing any output instead of blocking.

- The combination GRND_INSECURE|GRND_NONBLOCK is the same as
GRND_INSECURE, since GRND_INSECURE never blocks anyway.

- The combinations GRND_INSECURE|GRND_RANDOM and
GRND_INSECURE|GRND_RANDOM|GRND_NONBLOCK are nonsensical and fail
with EINVAL.

As proposed on tech-userlevel, tech-crypto, tech-security, and
tech-kern, and subsequently adopted by core (minus the getentropy part
of the proposal, because other operating systems and participants in
the discussion couldn't come to an agreement about getentropy and
blocking semantics):

https://mail-index.netbsd.org/tech-userlevel/2020/05/02/msg012333.html

Rewrite cprng_fast in terms of new ChaCha API.

Rewrite entropy subsystem.

Primary goals:

1. Use cryptography primitives designed and vetted by cryptographers.
2. Be honest about entropy estimation.
3. Propagate full entropy as soon as possible.
4. Simplify the APIs.
5. Reduce overhead of rnd_add_data and cprng_strong.
6. Reduce side channels of HWRNG data and human input sources.
7. Improve visibility of operation with sysctl and event counters.

Caveat: rngtest is no longer used generically for RND_TYPE_RNG
rndsources. Hardware RNG devices should have hardware-specific
health tests. For example, checking for two repeated 256-bit outputs
works to detect AMD's 2019 RDRAND bug. Not all hardware RNGs are
necessarily designed to produce exactly uniform output.

ENTROPY POOL

- A Keccak sponge, with test vectors, replaces the old LFSR/SHA-1
kludge as the cryptographic primitive.

- `Entropy depletion' is available for testing purposes with a sysctl
knob kern.entropy.depletion; otherwise it is disabled, and once the
system reaches full entropy it is assumed to stay there as far as
modern cryptography is concerned.

- No `entropy estimation' based on sample values. Such `entropy
estimation' is a contradiction in terms, dishonest to users, and a
potential source of side channels. It is the responsibility of the
driver author to study the entropy of the process that generates
the samples.

- Per-CPU gathering pools avoid contention on a global queue.

- Entropy is occasionally consolidated into global pool -- as soon as
it's ready, if we've never reached full entropy, and with a rate
limit afterward. Operators can force consolidation now by running
sysctl -w kern.entropy.consolidate=1.

- rndsink(9) API has been replaced by an epoch counter which changes
whenever entropy is consolidated into the global pool.
. Usage: Cache entropy_epoch() when you seed. If entropy_epoch()
has changed when you're about to use whatever you seeded, reseed.
. Epoch is never zero, so initialize cache to 0 if you want to reseed
on first use.
. Epoch is -1 iff we have never reached full entropy -- in other
words, the old rnd_initial_entropy is (entropy_epoch() != -1) --
but it is better if you check for changes rather than for -1, so
that if the system estimated its own entropy incorrectly, entropy
consolidation has the opportunity to prevent future compromise.

- Sysctls and event counters provide operator visibility into what's
happening:
. kern.entropy.needed - bits of entropy short of full entropy
. kern.entropy.pending - bits known to be pending in per-CPU pools,
can be consolidated with sysctl -w kern.entropy.consolidate=1
. kern.entropy.epoch - number of times consolidation has happened,
never 0, and -1 iff we have never reached full entropy

CPRNG_STRONG

- A cprng_strong instance is now a collection of per-CPU NIST
Hash_DRBGs. There are only two in the system: user_cprng for
/dev/urandom and sysctl kern.?random, and kern_cprng for kernel
users which may need to operate in interrupt context up to IPL_VM.

(Calling cprng_strong in interrupt context does not strike me as a
particularly good idea, so I added an event counter to see whether
anything actually does.)

- Event counters provide operator visibility into when reseeding
happens.

INTEL RDRAND/RDSEED, VIA C3 RNG (CPU_RNG)

- Unwired for now; will be rewired in a subsequent commit.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

branches: 1.181.2;
Split subr_cpu.c out of kern_cpu.c, to contain routines shared with rump.

- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.

Eliminate per-hook duplication of common code as suggested by
(and with major contributions from) riastradh@

Welcome to 9.99.23

introduce some common variables for use in GCC warning disables:

GCC_NO_FORMAT_TRUNCATION -Wno-format-truncation (GCC 7/8)
GCC_NO_STRINGOP_TRUNCATION -Wno-stringop-truncation (GCC 8)
GCC_NO_STRINGOP_OVERFLOW -Wno-stringop-overflow (GCC 8)
GCC_NO_CAST_FUNCTION_TYPE -Wno-cast-function-type (GCC 8)

use these to turn off warnings for most GCC-8 complaints. many
of these are false positives, most of the real bugs are already
commited, or are yet to come.


we plan to introduce versions of (some?) of these that use the
"-Wno-error=" form, which still displays the warnings but does
not make it an error, and all of the above will be re-considered
as either being "fix me" (warning still displayed) or "warning
is wrong."

Switch from NIST CTR_DRBG with AES to NIST Hash_DRBG with SHA-256.

Benefits:

- larger seeds -- a 128-bit key alone is not enough for `128-bit security'
- better resistance to timing side channels than AES
- a better-understood security story (https://eprint.iacr.org/2018/349)
- no loss in compliance with US government standards that nobody ever
got fired for choosing, at least in the US-dominated western world
- no dirty endianness tricks
- self-tests

Drawbacks:

- performance hit: throughput is reduced to about 1/3 in naive measurements
=> possible to mitigate by using hardware SHA-256 instructions
=> all you really need is 32 bytes to seed a userland PRNG anyway
=> if we just used ChaCha this would go away...

XXX pullup-7
XXX pullup-8
XXX pullup-9

branches: 1.175.2;
On the one thousand and ninth day rump's mainbus was moved from
rumpdev to rumpkern, liberating all rumpnet users from the need to
-lrumpdev -lrumpvfs just because a loopback interface is mandatory.

Rename rumpdev/autoconf.c to rumpkern/rump_autoconf.c to avoid
accidentally picking up e.g. sys/arch/amd64/amd64/autoconf.c through
make's .PATH.
Move rumpdev/MAINBUS.ioconf to rumpkern.

Merge the [pgoyette-compat] branch

Add threadpool(9), an abstraction that provides shared pools of kernel
threads running at specific priorities, with support for unbound pools
and per-cpu pools.

Written by riastradh@, and based on the May 2014 draft, with a few changes
by me:
- Working on the assumption that a relative few priorities will actually
be used, reduce the memory footprint by using linked lists, rather than
2 large (and mostly empty) tables. The performance impact is essentially
nil, since these lists are consulted only when pools are created (and
destroyed, for DIAGNOSTIC checks), and the lists will have at most 225
entries.
- Make threadpool job object, which the caller must allocate storage for,
really opaque.
- Use typedefs for the threadpool types, to reduce the verbosity of the
API somewhat.
- Fix a bunch of pool / worker thread / job object lifecycle bugs.

Also include an ATF unit test, written by me, that exercises the basics
of the API by loading a kernel module that exposes several sysctls that
allow the ATF test script to create and destroy threadpools, schedule a
basic job, and verify that it ran.

And thus NetBSD 8.99.29 has arrived.

Import thmap -- a concurrent trie-hash map, combining the elements of
hashing and radix trie. It supports lock-free lookups and concurrent
inserts/deletes. It is designed to be optimal as a general purpose
*concurrent* associative array.

Upstream: https://github.com/rmind/thmap
Discussed on tech-kern@

retire kern_xxx.c. long live kern_xxx.c.

split it into kern_reboot.c and kern_scdebug.c. while here,
add my copyright to kern_scdebug.c as it was largely rewritten
for kernhist support.

branches: 1.170.2; 1.170.4;
Add localcount to rump kernels

branches: 1.169.4; 1.169.6;
adjust flag.

branches: 1.168.2;
need kern_ssp.c for a full SSP build.

branches: 1.167.2;
Add psref to rump kernel

nuke a few missed -Ifactiondir CPPFLAGSitions.

Add a COMMENT describing what each component roughly does.

"make describe" prints the comment.

Requested/inspired by Vincent Schwarzer on rumpkernel-users

Use the more widely accepted version of alphabetical order.

Allow rumpkernel to use rw_obj_*

Remove KERN.ioconf, ksyms does not really need it.

generate ioconf.h for pseudo-device attach prototype

Remove unreal allocators, unconditionally use subr_{kmem,pool}.

Will, with other work, allow to tighten the memory allocation hypercall
specification to page-granularity allocations in the future.

Rename RUMP_COMPAT to RUMP_NBCOMBAT to better signify what the
variable does.

g/c the never-used and never-useful hyperstubs.c

Build compat code only when specified by RUMP_COMPAT

Include kern_clock.c in rump kernels.

Fix rump build: rndpseudo_50.c now needed by kernel, not rnd device.

default newvers.sh parameters to reproducible build

Move sysproxy support into a separate component, rumpkern_sysproxy,
instead of it being always provided by the rump kernel base. This
move accomplishes two things:

1) it is no longer necessary to provide sysproxy hypercall stubs for
platforms which do not want to use sysproxy
2) it is easier to reason about the security aspects, since configurations
not linking the sysproxy component simply do not support remote
system calls

discussed on rumpkernel-users

Put all sysproxy routines to their own C module, sysproxy.c

Remove shlib_version files and just use Makefile SHLIB_MAJOR/MINOR,
with the default provided by Makefile.rump (they're all 0.0 anyway)

branches: 1.150.2;
Move rump kernel man pages from various sources to sys/rump

namely:
* src/lib is used only when building for POSIX'y platforms, but
the man pages have their use for all platforms
* rumpuser.3 is a function of the rump kernel, not one of the of
the POSIX'y implementation hosted in src/lib/librumpuser

no functional change

Add MKCOMPAT support for aarch64 (COMPAT_MACHINE_CPU)

branches: 1.148.2; 1.148.4; 1.148.8;
Merge tls-earlyentropy branch into HEAD.

librump: include pcq(9) interface.

Eliminate weak symbols from rump kernel syscall handlers, part 7:

Build component constructors which establish syscalls at boottime.

gardenize rump.c: move data structure helper routines to accessors.c

Move the etfs linkage from rumpvfs to rumpkern, and replace the
weak alias show with an honest pointer indirection.

No client-visible change. (apart from this version working e.g.
on musl w/ dlopen)

branches: 1.143.2;
Add compat 50 time syscalls, needed by rump sys_linux.

Put nanosleep() and folks in librump instead of maintaining them in
the separate rumpkern_time component. Keeping time-related routines
elsewhere lead to some illogical behavior if you didn't think of linking
in rumpkern_time (hands up everyone who checks the return value of
nanosleep()).

Add warnings if rumpkern_time is linked or used. I'll remove it in a
month or two instead of now since it was part of a buildrump.sh snapshot
and it's nicer if trying to use it gives a warning instead of an error
in the next snapshot.

"everything should be as modular as possible, but no more modular than that"

Use uniprocessor-optimized locking in RUMP_LOCKS_UP=yes (default: no)

Allow multiple "rumpcomp_user" source modules to be specified by
introducing RUMPCOMP_USER_SRCS. Make RUMPCOMP_USER issue a deprecation
warning, but for compat make it set RUMPCOMP_USER_SRCS=rumpcomp_user.c
for now.

Move the "is arch capable of loading native kernel modules into
rump kernel" clauses from bsd.own.mk to Makefile.rump. Also,
add a rump_nativeabi_p() call to determine if rump kernel is
compiled with native ABI support.

Use the new FEAT_LDREX to replace ARMV6/ARMV7

Use same uvm_swap_shutdown() stub for !vmswap kernels and rump kernels.

Rototill a bit, and attempt to disguise it as non-gratuitous.

Add arch/generic and move non-x86 files from rumpkern/ there. Also,
move files from arch/i386 to arch/x86, and make both i386 and x86_64
use those.

This clarifies the situation with what is MD vs. MI code.

renames:
rumpcpu_generic,kobj_stubs,pmap_stubs => arch/generic/rump_generic_$x
arch/i386/* => arch/x86/rump_x86_$x

(for those who forget, x86 requires MD code because rump kernels
use the same ABI as kernel modules)

Use subr_cprng.c instead of stub implementation. Rijndael migrates from
rumpkern_crypto to rumpkern due to it being mandatory for cprng.

Make ktrace a compile-time option

Support ktrace for rump kernels.

Requested by Justin Cormack on rumpkernel-users.

Add an initial console device and open fd's 0/1/2 for initproc. This is
again useful in standalone-type environments such as Xen, where all
printf/etc calls go through the rump kernel.

+ don't rename rump_syscalls.*o
+ support RUMP_KERNEL_IS_LIBC

Teach this about ARMV7

Coldfire uses atomic_cas_generic.c

branches: 1.128.2; 1.128.4;
Rework rndsink(9) abstraction and adapt arc4random(9) and cprng(9).

rndsink(9):
- Simplify API.
- Simplify locking scheme.
- Add a man page.
- Avoid races in destruction.
- Avoid races in requesting entropy now and scheduling entropy later.

Periodic distribution of entropy to sinks reduces the need for the
last one, but this way we don't need to rely on periodic distribution
(e.g., in a future tickless NetBSD).

rndsinks_lock should probably eventually merge with the rndpool lock,
but we'll put that off for now.

cprng(9):
- Make struct cprng_strong opaque.
- Move rndpseudo.c parts that futz with cprng guts to subr_cprng.c.
- Fix kevent locking. (Is kevent locking documented anywhere?)
- Stub out rump cprng further until we can rumpify rndsink instead.
- Strip code to grovel through struct cprng_strong in fstat.

Actually, there's no point in unconditionally compiling in weak stubs
which will never be used in the NetBSD build. Comment hyperstubs.c
out from SRCS, but retain the source module as documentation.

weak stubs for optional hypercalls

* treat kern_malloc.c as an unreal allocator (it's so lightweight)
* get rid of the rumpuser_realloc() hypercall
* pass size to rumpuser_free()

Allow Makefile.rump to append to SRCS.

Use kern_malloc.c instead of the relegated allocators in memalloc.c.
Previously this didn't make sense due to the use of kmem_map, but the
new malloc is more dynamic and puts sense into using it.

Always include subr_vmem.c, even with RUMP_UNREAL_ALLOCATORS=yes
(previously it was just missing in that case).

Record wchan to unreal pool_init() to avoid memory leak warning.

Take into account armv6 hacks from common/lib/libc/arch/arm/atomic to
allow this to build with -march=armv6k

Add references to ${_NETBSD_VERSION_DEPENDS} for files that
need to be re-created when the NetBSD version changes. They
will also be re-created when any build settings are changed.

branches: 1.119.2;
Make it possible to select between real and unreal allocators from
make. Plus some gratuitous renaming.

rumpkern:
- Add subr_kcpuset.c and subr_pserialize.c modules.
- Add kcpuset_{running,attached} for RUMP env.

G/C kern_malloc_stdtype.c

P1003_1B_SEMAPHORE is no longer optional.

branches: 1.115.2;
Entropy-pool implementation move and cleanup.

1) Move core entropy-pool code and source/sink/sample management code
to sys/kern from sys/dev.

2) Remove use of NRND as test for presence of entropy-pool code throughout
source tree.

3) Remove use of RND_ENABLED in device drivers as microoptimization to
avoid expensive operations on disabled entropy sources; make the
rnd_add calls do this directly so all callers benefit.

4) Fix bug in recent rnd_add_data()/rnd_add_uint32() changes that might
have lead to slight entropy overestimation for some sources.

5) Add new source types for environmental sensors, power sensors, VM
system events, and skew between clocks, with a sample implementation
for each.

ok releng to go in before the branch due to the difficulty of later
pullup (widespread #ifdef removal and moved files). Tested with release
builds on amd64 and evbarm and live testing on amd64.

Implement the register/deregister/evaluation API for secmodel(9). It
allows registration of callbacks that can be used later for
cross-secmodel "safe" communication.

When a secmodel wishes to know a property maintained by another
secmodel, it has to submit a request to it so the other secmodel can
proceed to evaluating the request. This is done through the
secmodel_eval(9) call; example:

bool isroot;
error = secmodel_eval("org.netbsd.secmodel.suser", "is-root",
cred, &isroot);
if (error == 0 && !isroot)
result = KAUTH_RESULT_DENY;

This one asks the suser module if the credentials are assumed to be root
when evaluated by suser module. If the module is present, it will
respond. If absent, the call will return an error.

Args and command are arbitrarily defined; it's up to the secmodel(9) to
document what it expects.

Typical example is securelevel testing: when someone wants to know
whether securelevel is raised above a certain level or not, the caller
has to request this property to the secmodel_securelevel(9) module.
Given that securelevel module may be absent from system's context (thus
making access to the global "securelevel" variable impossible or
unsafe), this API can cope with this absence and return an error.

We are using secmodel_eval(9) to implement a secmodel_extensions(9)
module, which plugs with the bsd44, suser and securelevel secmodels
to provide the logic behind curtain, usermount and user_set_cpu_affinity
modes, without adding hooks to traditional secmodels. This solves a
real issue with the current secmodel(9) code, as usermount or
user_set_cpu_affinity are not really tied to secmodel_suser(9).

The secmodel_eval(9) is also used to restrict security.models settings
when securelevel is above 0, through the "is-securelevel-above"
evaluation:
- curtain can be enabled any time, but cannot be disabled if
securelevel is above 0.
- usermount/user_set_cpu_affinity can be disabled any time, but cannot
be enabled if securelevel is above 0.

Regarding sysctl(7) entries:
curtain and usermount are now found under security.models.extensions
tree. The security.curtain and vfs.generic.usermount are still
accessible for backwards compat.

Documentation is incoming, I am proof-reading my writings.

Written by elad@, reviewed and tested (anita test + interact for rights
tests) by me. ok elad@.

See also
http://mail-index.netbsd.org/tech-security/2011/11/29/msg000422.html

XXX might consider va0 mapping too.

XXX Having a secmodel(9) specific printf (like aprint_*) for reporting
secmodel(9) errors might be a good idea, but I am not sure on how
to design such a function right now.

branches: 1.113.2;
Revert "stopcap fix" for rump by christos, which causes build failure on
most non-x86 ports and seems unnecessary. (caused by wrong rump_namei.h?)

Add subr_open_disk.c for getdiskinfo(). Once we get rid of getdiskinfo,
this will not be needed.

First step of random number subsystem rework described in
<20111022023242.BA26F14A158@mail.netbsd.org>. This change includes
the following:

An initial cleanup and minor reorganization of the entropy pool
code in sys/dev/rnd.c and sys/dev/rndpool.c. Several bugs are
fixed. Some effort is made to accumulate entropy more quickly at
boot time.

A generic interface, "rndsink", is added, for stream generators to
request that they be re-keyed with good quality entropy from the pool
as soon as it is available.

The arc4random()/arc4randbytes() implementation in libkern is
adjusted to use the rndsink interface for rekeying, which helps
address the problem of low-quality keys at boot time.

An implementation of the FIPS 140-2 statistical tests for random
number generator quality is provided (libkern/rngtest.c). This
is based on Greg Rose's implementation from Qualcomm.

A new random stream generator, nist_ctr_drbg, is provided. It is
based on an implementation of the NIST SP800-90 CTR_DRBG by
Henric Jungheim. This generator users AES in a modified counter
mode to generate a backtracking-resistant random stream.

An abstraction layer, "cprng", is provided for in-kernel consumers
of randomness. The arc4random/arc4randbytes API is deprecated for
in-kernel use. It is replaced by "cprng_strong". The current
cprng_fast implementation wraps the existing arc4random
implementation. The current cprng_strong implementation wraps the
new CTR_DRBG implementation. Both interfaces are rekeyed from
the entropy pool automatically at intervals justifiable from best
current cryptographic practice.

In some quick tests, cprng_fast() is about the same speed as
the old arc4randbytes(), and cprng_strong() is about 20% faster
than rnd_extract_data(). Performance is expected to improve.

The AES code in src/crypto/rijndael is no longer an optional
kernel component, as it is required by cprng_strong, which is
not an optional kernel component.

The entropy pool output is subjected to the rngtest tests at
startup time; if it fails, the system will reboot. There is
approximately a 3/10000 chance of a false positive from these
tests. Entropy pool _input_ from hardware random numbers is
subjected to the rngtest tests at attach time, as well as the
FIPS continuous-output test, to detect bad or stuck hardware
RNGs; if any are detected, they are detached, but the system
continues to run.

A problem with rndctl(8) is fixed -- datastructures with
pointers in arrays are no longer passed to userspace (this
was not a security problem, but rather a major issue for
compat32). A new kernel will require a new rndctl.

The sysctl kern.arandom() and kern.urandom() nodes are hooked
up to the new generators, but the /dev/*random pseudodevices
are not, yet.

Manual pages for the new kernel interfaces are forthcoming.

branches: 1.110.2;
include uvm_object.c in the rump kernel for the new uvm_obj* functions.
don't build the uvm_object.c uvm_object_printit() for _RUMPKERNEL. (XXX)
add empty panic() stubs for uvm_loanbreak() and ubc_purge().

fixes some more 5.99.53 rump build issues.

branches: 1.109.2;
Spell --fatal-warnings with two hyphens

Include bsd.own.mk before making decisions based on mk.conf.

use compat code from sys/compat/common

branches: 1.106.2;
Support LOCKDEBUG. To use it, compile sys/rump with RUMP_LOCKDEBUG=yes.

requested by martin (sparc64 gdb cannot reliably produce a stack trace)

Add SMP support for all architectures.

tested on sparc64 by martin

Support MKREPRO

Duh, it's x86_64, not amd64. This should make the races which
require SMP trigger in the amd64/qemu runs again.

rename atomic_cas_up to rump_atomic_cas_up to avoid collisions

Encode smp-capability into the makefile so that it can be used to
avoid potential screwups.

Add a lockless uniprocessor version of atomic_cas_generic.c, which
is currently used by all the archs that previously used cas_generic.

Realize the >1yo comment above rump_reboot and retire them to make
room for sys_reboot.

"i build dead files". ok, so let's not.

Use standard uvm aobj pager. Most of the kernel aobj pager complexity
comes from swap handling, but that is included only with VMSWAP.

Implement rump_lwproc: the new lwp/proc management routines for
rump. These move the management of the pid/lwpid space from the
application into the kernel, make code more robust, and make it
possible to attach multiple lwp's to non-proc0 processes.

Include kern_prot.c for setuid etc.

Use one line per sys/kern source module. no functional change.

Add the new kern_cfglock.c to rump.

* move stat syscalls to newstyle compat
* implement compat for pollts

Reinstate the blanket pmap.h for archs which do not conform to the
kernel ABI (i.e. not i386 or amd64). Due to the "half function,
half macro, all noodles" nature of pmap.h, it's too entangling and
too brittle to keep up with an ifdeffy MI implementation.

Fix previous in emul.c -- only numbers are operands for cpp comparisons.
Apparently non-numbers logically produce arch-dependent behaviour.

Use kern_proc.c instead of a collection of stubs. But what we
really wanted from this commit was the support for proc_specificdata.

TODO: make creating a new process actually use kern_proc and
maybe even add an interface which starts a process with
"any pid you don't like"

Make vers.c depend on sys/param.h too, to ensure that this file is
regenerated for on kernel version bump. Avoids __NetBSD_Version__ and
osrelease out of sync problem for mkupdate builds.

ok from pooka@.

Whoops, default to MP locking.

Add uniprocessor versions of mutex/rw/cv. They work only on virtual
unicpu configurations (i.e. RUMP_NCPU==1), but are massively faster
than the multiprocessor versions since the fast path does not have
to perform any cache coherent operations. _Applications_ with
lock-happy kernel paths, i.e. _not_ lock microbenchmarks, measure
up to tens of percents speedup on my Core2 Duo. Every globally
atomic state required by normal locks/atomic ops implies a hideous
speed penalty even for the fast path.

While this requires a unicpu configuration, it should be noted that
we are talking about a virtual unicpu configuration. The host can
have as many processors as it desires, and the speed benefit of
virtual unicpu is still there. It's pretty obvious that in terms
of scalability simple workload partitioning and replication into
multiple kernels wins hands down over complicated locking or
locklessing algorithms which depend on globally atomic state.

Move routines related to kernel locking and scheduling from
locks.c to klock.c.

No functional change.

Actually, push defining _RUMPKERNEL down to libkern, since it's
not needed elsewhere.

Limit visibility of _RUMPKERNEL to prevent abuse.

Include devsw_conv0 from an i386 kernel compilation (no, we don't
care about the arch as long as all the devices we care about are
present). The file should be autogenerated, but that requires some
more changes to config(1).

Implement kobj_renamespace() for rump. Support for a few archs is
missing, but that doesn't really matter, since they are living in
their own "everything is a macro" happyland and don't support the
native kernel ABI anyway.

Move sys_module from vfs to kern -- while modules cannot be loaded,
there's not forbidden about querying the list of (builtin) modules
even when running without vfs.

support kern_resource

Move all signal-related from emul.c to signals.c. Additionally,
define a few alternate signal models for the rump kernel, including
ones where signals are ignored or sent to host processes.

Use kern_syscall.c instead of homegrown syscall dis/establishment routines.

Include kern_tc and use a timecounter driver instead of homerolled
kern_tc implementation.

support lwp specificdata

branches: 1.74.2;
Globally define -Wno-pointer-sign, as it has become a pointless
exercise of "add it to every Makefile individually".

XXX: should autosynchronize with the rest of the kernel buildflags
in sys/conf/Makefile.kern.inc.

branches: 1.73.2;
Include newly-created subr_device.c and remove few special case
device accessor copypastes. This makes it possible to link static
binaries which use -lrumpdev.

include kern_hook.c

Use subr_percpu.c instead of homegrown implementation. ...except
when using malloc(3)-relegated allocators (happens in production
e.g. on Linux), since subr_percpu.c uses vmem and i don't want to
reimplement vmem.

update to newnewvers.sh usage

Generate vers.c and include it in the kernel component.

Make librump play with mips nicely. Define ARCH_ELFSIZE for mips to be 32.
This works for N64 kernels because objcopy them to be 32bit to the bootloaders
can handle them.

rename LD32DIR to MLIBDIR.

Include cpu crosscall support (instead of stubbing it out).

Now that Makefile.rump was changed and everything gets built in
update builds too, flip the allocator define to prefer the kernel
pool/kmem instead of malloc(3). Use malloc(3) only if
RUMP_USE_UNREAL_ALLOCATORS is defined.

include sys_pipe.c

Enable kernel kmem/vmem/pool/pool_cache by default again instead
of malloc(3) allocators.

misc_stub and emul have been the same thing for a looong time now,
so just move the few remaining routines in misc_stub to emul.

Give the kthread->pthread interface emulation its own module.

Pull all relegating memory allocators under a common roof in memalloc.c

move copy-related routines to their own module

Use std. uiomove() & friends.

Use kern_mutex_obj.c directly instead of copypasting code.

move module to SRCS where it logically belongs. no functional change.

Actually, put uvm_readahead into rumpkern, since while it's
technically vfs stuff, sys_descrip depends on it and readahead
itself uses only the pager interface.

treat sun2 like the other losing platforms.

Include sys_select.c for proper select()/poll() support.

Add initial work on a rump virtual cpu scheduler. This is necessary
for kernel code which has been written to avoid MP contention by
using cpu-local storage (most prominently, select and pool_cache).

Instead of always assuming rump_cpu, the scheduler must now be run
(and unrun) on all entry points into rump. Likewise, rumpuser
unruns and re-runs the scheduler around each potentially blocking
operation. As an optimization, I modified some locking primitives
to try to get the lock without blocking before releasing the cpu.

Also, ltsleep was modified to assume that it is never called without
the biglock held and made to use the biglock as the sleep interlock.
Otherwise there is just too much drama with deadlocks. If some
kernel code wants to call ltsleep without the biglock, then, *snif*,
it's no longer supported and rump and should be modified to support
newstyle locks anyway.

Adjust rump sources for external/internal interfaces.
No functional change.

First part of secmodel cleanup and other misc. changes:

- Separate the suser part of the bsd44 secmodel into its own secmodel
and directory, pending even more cleanups. For revision history
purposes, the original location of the files was

src/sys/secmodel/bsd44/secmodel_bsd44_suser.c
src/sys/secmodel/bsd44/suser.h

- Add a man-page for secmodel_suser(9) and update the one for
secmodel_bsd44(9).

- Add a "secmodel" module class and use it. Userland program and
documentation updated.

- Manage secmodel count (nsecmodels) through the module framework.
This eliminates the need for secmodel_{,de}register() calls in
secmodel code.

- Prepare for secmodel modularization by adding relevant module bits.
The secmodels don't allow auto unload. The bsd44 secmodel depends
on the suser and securelevel secmodels. The overlay secmodel depends
on the bsd44 secmodel. As the module class is only cosmetic, and to
prevent ambiguity, the bsd44 and overlay secmodels are prefixed with
"secmodel_".

- Adapt the overlay secmodel to recent changes (mainly vnode scope).

- Stop using link-sets for the sysctl node(s) creation.

- Keep sysctl variables under nodes of their relevant secmodels. In
other words, don't create duplicates for the suser/securelevel
secmodels under the bsd44 secmodel, as the latter is merely used
for "grouping".

- For the suser and securelevel secmodels, "advertise presence" in
relevant sysctl nodes (sysctl.security.models.{suser,securelevel}).

- Get rid of the LKM preprocessor stuff.

- As secmodels are now modules, there's no need for an explicit call
to secmodel_start(); it's handled by the module framework. That
said, the module framework was adjusted to properly load secmodels
early during system startup.

- Adapt rump to changes: Instead of using empty stubs for securelevel,
simply use the suser secmodel. Also replace secmodel_start() with a
call to secmodel_suser_start().

- 5.99.20.

Testing was done on i386 ("release" build). Spearated module_init()
changes were tested on sparc and sparc64 as well by martin@ (thanks!).

Mailing list reference:

http://mail-index.netbsd.org/tech-kern/2009/09/25/msg006135.html

Include humanize and extent support in rumpkern.

include init_sysctl_base.c

Run rump_dev_init() where available.

include syscalls.c

branches: 1.45.2;
Do not include securelevel, it includes too many dependencies on
vfs in its current form.

Add proof-of-concept code for enabling system calls to rump virtual
kernels running in other processes on the same machine or on an
entirely different host. I wrote this a while ago and am now
committing it mainly to avoid losing it. It works, but could do
with a little tuning here and there.

What this will hopefully eventually buy us is the ability to use
standard userland tools to configure rump kernels, e.g. ifconfig(8)
and route(8) could be used to configure the networking stack provided
by a rump kernel. Also some distributed OS implications may apply.

fun fact: a system call which just does copyin/copyout takes >1000x
longer when made over the LAN as compared to when made on the same
machine.

When I switched to real kauth, I forgot to include a secmodel.
Fix this oversight by including bsd44. Makes permissions for p2k
work again.

include subr_evcnt

Support kqueue in the rump virtual kernel.

branches: 1.40.2;
Turn of real allocators and fall back to malloc(3) for the time
being. Since we have many threads but pretend to have only one
cpu, the pool code runs into concurrency trouble for cpu-private
data.

Add a compile-time option to use kmem/vmem/pools from the kernel
sources instead of homegrown allocators. Default to "on", even
though they appear to be a few percent slower at least on short
jobs (e.g. untar to tmpfs).

Change the use of formally undocumented features, which have now been
made to fail. Specifically, change
.ifdef(SYMBOL) -> .ifdef SYMBOL or .if defined(SYMBOL),
and corresponding for .ifndef.

Also correct one error in lib/libm/Makefile (.ifdef (${MKCOMPLEX} != "no")?!?).

Do rump kernel symbol protection for vax. All archs support it now.

* Add adapted version of the userspace atomic_cas ops for platforms
lacking special instructions. We always use the spinlocked version
(could use RAS on UP NetBSD host, but it's not portable).
* Add platform-based symbol quirk tables for selectively not renaming
toolchain symbols for some platforms. Although, this should really
depend on the (toolchain,platform)-tuple and not just the platform.

This allows arm, hppa, mips and sh3 to build succesfully with an
isolated rump kernel namespace. ... now, one arch remains: vax.
you must compile vax. then, only then will you MI be.

Remove vax MD source which is now brought in automatically via libkern.

adjust LD32DIR comment. per discussion with mrg

Rename malloc() to kern_malloc() to avoid name conflict with libc.
Now it is possible to use the kernel linker with rump.

Assemble with _LOCORE.

Include libkern contents in librump.

Add kludge to allow amd64 compat to build. This is not a proper
fix which most likely requires some compat lib build infra toggle.

Include kernel printf routines in rump instead of relying on the
magic libc symbol. This also allows to bid farewell to subr_prf2.c
and merge the contents back to subr_prf.c. The host kernel bridging
is now done via rumpuser_putchar().

Purge multiple kernel opt files.

Define MODULAR for rump core components. This enables module
loading via the kernel module framework (instead of dlopen()).
For now it only works on amd64 and i386, but for the rest it should
just be a matter of including the relevant kobj_machdep.c modules
from the kernel sources.

Include rb.c instead of relying on it being in libc.

-I${RUMPTOP}/librump/rumpkern so that build from src/lib works.

noted by Geoff Wing on current-users

Switch i386 away from rump/include/machine. This means that rump
on i386 now uses the native kernel ABI. This in turn means that
rump modules and kernel modules are binary equivalent and can be
used interchangeably.

include subr_devsw in rumpkern

include snprintb

.PATH maintenance

Split vfs out of rumpkern into rumpvfs. Non-fs rumps no longer
include the file system code. File system rumps explicitly need
to include rumpvfs from now on.

cwd is logically a vfs concept, so take it out from the bosom of
kern_descrip and into vfs_cwd. No functional change.

Move rump public headers to include/rump

branches: 1.17.2; 1.17.4;
Use ${TOOL_SED} instead if plain sed in Makefiles.

branches: 1.16.2;
Add support bits necessary for rumpnet functionality.

Move uidinfo to its own module in kern_uidinfo.c and include in rump.
No functional change to uidinfo.

Add a simple percpu implementation (which isn't actually percpu at
all, since we don't currently have the notion of "real" cpu in
rump...but that's beyond the point).

Support callouts and call callout_hardclock() from the timer
interrupt thread.

The sleepq implementation required for callouts is horrible, kludged
only for callouts, and generally unacceptable. It needs revisiting,
but I'm not sure yet should rump or kern_timeout be improved. It's
almost untested as of now, but committing this will give me some
maneuvering space while letting application compile.

add kern_rate, subr_iostat and subr_once

Reorganize SRCS+= into smaller chunks to make adding new files
easier. No functional change.

No point in having our private atomic ops, just use the ones now
available in libc.

Switch to std kern_auth.

branches: 1.8.2;
Help out reinoud a bit with the challenge of adding vfs_dirhash.c here

Move global malloc types from kern_malloc into a separate module.

branches: 1.6.2;
Add support for using real kmem/vmem. Don't enable it by default,
though, since it a) is a lot of unnecessary indirection in rump
b) requires callouts which are so far unimplemented.

sort sys/kern SRCS alphabetically.

support real sysctls

Merge the simonb-wapbl branch. From the original branch commit:

Add Wasabi System's WAPBL (Write Ahead Physical Block Logging)
journaling code. Originally written by Darrin B. Jewell while
at Wasabi and updated to -current by Antti Kantee, Andy Doran,
Greg Oster and Simon Burge.

OK'd by core@, releng@.

branches: 1.2.2;
Fix race during creation of rumpdefs.h, rumpvnode_if.h, and rumpvnode_if.c.
Patch from pooka@ with tweak from me.

Approved by: pooka

Install rump libraries and utilities to the base system and remove the
private non-installed build infrastructure from sys/rump.

breakdown of commit:
* install relevant headers into /usr/include/rump
* build sys/rump/librump/rumpuser and sys/rump/librump/rumpkern
from src/lib and install as librumpuser and librump, respectively
+ this retains the ability to test a librump build with just the
kernel sources at hand
* move sys/rump/fs/lib/libukfs and sys/rump/fs/lib/libp2k to src/lib
for general consumption, they are not kernel-space dwellers anyway
* build and install sys/rump/fs/lib/lib$fs as librumpfs_$fs
* add chapter 3 manual pages for rump, rumpuser, ukfs and p2k
* build and install userspace kernel file system daemons if MKPUFFS=yes
is spexified
* retire fsconsole for now, it will make a comeback with an actually
implemented version shortly

Put the kernelside rump kernel headers into <rump-sys> instead of
sprinkling them around the faction directories. Avoids having
to add a CPPFLAGS (or several) to pretty much every component
Makefile.

Leave compat headers around in the old locations.

The commit changes some autogenerated files, but I'll fix the
generators shortly and regen.

Use vmspace of calling [rump kernel] process instead of sysspace.

branches: 1.1.2; 1.1.4; 1.1.6; 1.1.10; 1.1.12;
gardenize rump.c: move data structure helper routines to accessors.c

No point in having our private atomic ops, just use the ones now
available in libc.

Install rump libraries and utilities to the base system and remove the
private non-installed build infrastructure from sys/rump.

breakdown of commit:
* install relevant headers into /usr/include/rump
* build sys/rump/librump/rumpuser and sys/rump/librump/rumpkern
from src/lib and install as librumpuser and librump, respectively
+ this retains the ability to test a librump build with just the
kernel sources at hand
* move sys/rump/fs/lib/libukfs and sys/rump/fs/lib/libp2k to src/lib
for general consumption, they are not kernel-space dwellers anyway
* build and install sys/rump/fs/lib/lib$fs as librumpfs_$fs
* add chapter 3 manual pages for rump, rumpuser, ukfs and p2k
* build and install userspace kernel file system daemons if MKPUFFS=yes
is spexified
* retire fsconsole for now, it will make a comeback with an actually
implemented version shortly

branches: 1.4.2;
Don't rely on <sys/mutex.h> being implicitly included, because on
some ports (mvme68k for example) that's not happening.

branches: 1.3.2; 1.3.4;
Remove clause 3 and 4 from TNF licenses

branches: 1.2.2; 1.2.4; 1.2.6; 1.2.8; 1.2.14; 1.2.16; 1.2.18;
pull in atomic ops from vmlocking2

branches: 1.1.2;
file atomic.c was initially added on branch vmlocking2.

provide generic cas for _LP64

branches: 1.2.76;
I got a report about this not compiling on mips64. While I could
not repeat the problem and while this should not be used at all on
mips64, the real fix is a little diablo in the details. So hack
around it for now (yes, it results in only half of the locks being
used on 64bit platforms. no biggie for a hack).

branches: 1.1.2; 1.1.4; 1.1.8;
* Add adapted version of the userspace atomic_cas ops for platforms
lacking special instructions. We always use the spinlocked version
(could use RAS on UP NetBSD host, but it's not portable).
* Add platform-based symbol quirk tables for selectively not renaming
toolchain symbols for some platforms. Although, this should really
depend on the (toolchain,platform)-tuple and not just the platform.

This allows arm, hppa, mips and sh3 to build succesfully with an
isolated rump kernel namespace. ... now, one arch remains: vax.
you must compile vax. then, only then will you MI be.

rename atomic_cas_up to rump_atomic_cas_up to avoid collisions

Add a lockless uniprocessor version of atomic_cas_generic.c, which
is currently used by all the archs that previously used cas_generic.

Switch to std kern_auth.

Install rump libraries and utilities to the base system and remove the
private non-installed build infrastructure from sys/rump.

breakdown of commit:
* install relevant headers into /usr/include/rump
* build sys/rump/librump/rumpuser and sys/rump/librump/rumpkern
from src/lib and install as librumpuser and librump, respectively
+ this retains the ability to test a librump build with just the
kernel sources at hand
* move sys/rump/fs/lib/libukfs and sys/rump/fs/lib/libp2k to src/lib
for general consumption, they are not kernel-space dwellers anyway
* build and install sys/rump/fs/lib/lib$fs as librumpfs_$fs
* add chapter 3 manual pages for rump, rumpuser, ukfs and p2k
* build and install userspace kernel file system daemons if MKPUFFS=yes
is spexified
* retire fsconsole for now, it will make a comeback with an actually
implemented version shortly

branches: 1.9.2;
Fixes to build rump utilities as host binaries on Linux by removing
sys namespace pollution which has crept in.

Submitted in private mail by takemura, domain ca2.so-net.ne.jp

Allow KAUTH_SYSTEM_MKNOD in kauth_authorize_generic() to make it
possible to execute sys_mknod().

from Arnaud Ysmal

branches: 1.7.4; 1.7.6; 1.7.8;
Backup some fixes for recent breakage from local tree. Also some
other improvements such as exporting the real kernel namei and
using that in ukfs instead of the homegrown heap'o hacks namei.
"etcetc".

branches: 1.6.2; 1.6.6;
Use namei() etc. from kernel sources instead of a reimplementation.
To accommodate, give the rootvnode its own vnode op vector with a
simple lookup operation. This is used for looking up the file
system's device vnode instead of doing that directly in a homesmoked
namei().

Theoretically this is supposed to be interchangeable with real
kernel code. Use kmem_alloc/free instead of some wily homerolled
rump interfaces for memory allocation.

Add the ability to run puffs in userspace. This means that puffs
can now be developed in userspace using puffs for development
(I hate emulators, they are annoyingly clumsy).

To e.g. mount psshfs using puffs-on-puffs, run fs/bin/syspuffs/syspuffs
with the regular mount_psshfs command line as an argument:

golem> ./syspuffs /usr/sbin/mount_psshfs ftp.netbsd.org:/pub /puffs

This will make the mount appear as usual, with the exception that the
requests will be passed through puffs both in the kernel and userspace:

ftp.netbsd.org:/pub on /puffs type puffs|p2k|puffs|psshfs

branches: 1.3.4;
Remove cn_lwp from struct componentname. curlwp should be used
from on. The NDINIT() macro no longer takes the lwp parameter and
associates the credentials of the calling thread with the namei
structure.

branches: 1.2.2; 1.2.4; 1.2.6; 1.2.10; 1.2.14; 1.2.16; 1.2.18;
Don't play rename & typecast games with kauth_cred_t, but rather
declare the type in rump.h only if necessary with the help of ifdef
magic.

Implement credentials. Access control is now similar to if the file
system were run in the kernel.

Kill handrolled buffercache and use vfs_bio from the kernel. This is
mostly to get the flag jungle in sync with the kernel.

fool some KASSERTs

Add some stubs for lfs.

branches: 1.1.2;
Introduce RUMPs - Runnable Userspace Meta-Programs

/sys/rump contains programs which run unmodified kernel code in an
emulated userspace environment. The kernel environment is provided
by librump. Currently supported are a number of file systems,
which by using puffs integrate seamlessly into the system and
provide a similar user experience to if the code was running as
part of the kernel. Potential future rumpification targets include
for example parts of the networking stack and some device drivers.

This work was supported by Google Summer of Code 2007.

use compat code from sys/compat/common

branches: 1.2.2; 1.2.4; 1.2.6;
Add select to list on compat syscalls. Makes some code work for
me with a nb5 userland without having to compile with -g ;)

* move stat syscalls to newstyle compat
* implement compat for pollts

- Eliminate the global "boottime" variable, which was being accessed
without any synchronization against changes by e.g. clock_settime().
- Replace with new getbinboottime() / getnanoboottime() / getmicroboottime()
functions (naming mirrors that of other time access functions in kern_tc.c).
It returns the (maybe-converted) value of timebasebin, which also tracks
our estimate of when the system was booted (i.e. the legacy "boottime" was
redundant).

XXX There needs to be a lockless synchronization mechanism for reading
timebasebin, but this is a problem in kern_tc.c that pre-existed these
"boottime" changes. At least now the problem is centralized in one location.

Rename min/max -> uimin/uimax for better honesty.

These functions are defined on unsigned int. The generic name
min/max should not silently truncate to 32 bits on 64-bit systems.
This is purely a name change -- no functional change intended.

HOWEVER! Some subsystems have

#define min(a, b) ((a) < (b) ? (a) : (b))
#define max(a, b) ((a) > (b) ? (a) : (b))

even though our standard name for that is MIN/MAX. Although these
may invite multiple evaluation bugs, these do _not_ cause integer
truncation.

To avoid `fixing' these cases, I first changed the name in libkern,
and then compile-tested every file where min/max occurred in order to
confirm that it failed -- and thus confirm that nothing shadowed
min/max -- before changing it.

I have left a handful of bootloaders that are too annoying to
compile-test, and some dead code:

cobalt ews4800mips hp300 hppa ia64 luna68k vax
acorn32/if_ie.c (not included in any kernels)
macppc/if_gm.c (superseded by gem(4))

It should be easy to fix the fallout once identified -- this way of
doing things fails safe, and the goal here, after all, is to _avoid_
silent integer truncations, not introduce them.

Maybe one day we can reintroduce min/max as type-generic things that
never silently truncate. But we should avoid doing that for a while,
so that existing code has a chance to be detected by the compiler for
conversion to uimin/uimax without changing the semantics until we can
properly audit it all. (Who knows, maybe in some cases integer
truncation is actually intended!)

branches: 1.7.2; 1.7.4;
add fo_name so we can identify the fileops in a simple way.

Put the kernelside rump kernel headers into <rump-sys> instead of
sprinkling them around the faction directories. Avoids having
to add a CPPFLAGS (or several) to pretty much every component
Makefile.

Leave compat headers around in the old locations.

The commit changes some autogenerated files, but I'll fix the
generators shortly and regen.

Implement fo_poll so that rump_sys_poll(stdout) works
more or less as expected.

from Martin Lucina <martin@lucina.net> via rumpkernel-users

branches: 1.4.2;
No longer create a special process context to fork remote clients off
of, simply always rfork off of proc1 closing all descriptors, and have
the rump kernel open 0/1/2 if the parent process is "1".

Fixes tests/rump/rumpkernel/t_sp, which was failing since the
abovementioned special process change due to attempting to deliver a
signal to the special process and the special process was not equipped
to handle one.

branches: 1.3.4; 1.3.6; 1.3.8; 1.3.10;
Fool isatty() into thinking we're a tty.

fix -Wuninitialized

Add an initial console device and open fd's 0/1/2 for initproc. This is
again useful in standalone-type environments such as Xen, where all
printf/etc calls go through the rump kernel.

Use subr_cprng.c instead of stub implementation. Rijndael migrates from
rumpkern_crypto to rumpkern due to it being mandatory for cprng.

Remove now-needless rump cprng_strong_deplete stub.

These rump stubs need to go away at some point...

branches: 1.7.2;
Rework rndsink(9) abstraction and adapt arc4random(9) and cprng(9).

rndsink(9):
- Simplify API.
- Simplify locking scheme.
- Add a man page.
- Avoid races in destruction.
- Avoid races in requesting entropy now and scheduling entropy later.

Periodic distribution of entropy to sinks reduces the need for the
last one, but this way we don't need to rely on periodic distribution
(e.g., in a future tickless NetBSD).

rndsinks_lock should probably eventually merge with the rndpool lock,
but we'll put that off for now.

cprng(9):
- Make struct cprng_strong opaque.
- Move rndpseudo.c parts that futz with cprng guts to subr_cprng.c.
- Fix kevent locking. (Is kevent locking documented anywhere?)
- Stub out rump cprng further until we can rumpify rndsink instead.
- Strip code to grovel through struct cprng_strong in fstat.

Make hypercall calling conventions consistent: iff a hypercall can fail,
it returns an int containing the error value.

make the random hypercall more generic

branches: 1.4.4; 1.4.8;

Separate /dev/random pseudodevice implemenation from kernel entropy pool
implementation. Rewrite pseudodevice code to use cprng_strong(9).

The new pseudodevice is cloning, so each caller gets bits from a stream
generated with its own key. Users of /dev/urandom get their generators
keyed on a "best effort" basis -- the kernel will rekey generators
whenever the entropy pool hits the high water mark -- while users of
/dev/random get their generators rekeyed every time key-length bits
are output.

The underlying cprng_strong API can use AES-256 or AES-128, but we use
AES-128 because of concerns about related-key attacks on AES-256. This
improves performance (and reduces entropy pool depletion) significantly
for users of /dev/urandom but does cause users of /dev/random to rekey
twice as often.

Also fixes various bugs (including some missing locking and a reseed-counter
overflow in the CTR_DRBG code) found while testing this.

For long reads, this generator is approximately 20 times as fast as the
old generator (dd with bs=64K yields 53MB/sec on 2Ghz Core2 instead of
2.5MB/sec) and also uses a separate mutex per instance so concurrency
is greatly improved. For reads of typical key sizes for modern
cryptosystems (16-32 bytes) performance is about the same as the old
code: a little better for 32 bytes, a little worse for 16 bytes.

branches: 1.3.2;
Remove arc4random() and arc4randbytes() from the kernel API. Replace
arc4random() hacks in rump with stubs that call the host arc4random() to
get numbers that are hopefully actually random (arc4random() keyed with
stack junk is not). This should fix some of the currently failing anita
tests -- we should no longer generate duplicate "random" MAC addresses in
the test environment.

No need to include MD <machine/cpu_counter.h> here.

First step of random number subsystem rework described in
<20111022023242.BA26F14A158@mail.netbsd.org>. This change includes
the following:

An initial cleanup and minor reorganization of the entropy pool
code in sys/dev/rnd.c and sys/dev/rndpool.c. Several bugs are
fixed. Some effort is made to accumulate entropy more quickly at
boot time.

A generic interface, "rndsink", is added, for stream generators to
request that they be re-keyed with good quality entropy from the pool
as soon as it is available.

The arc4random()/arc4randbytes() implementation in libkern is
adjusted to use the rndsink interface for rekeying, which helps
address the problem of low-quality keys at boot time.

An implementation of the FIPS 140-2 statistical tests for random
number generator quality is provided (libkern/rngtest.c). This
is based on Greg Rose's implementation from Qualcomm.

A new random stream generator, nist_ctr_drbg, is provided. It is
based on an implementation of the NIST SP800-90 CTR_DRBG by
Henric Jungheim. This generator users AES in a modified counter
mode to generate a backtracking-resistant random stream.

An abstraction layer, "cprng", is provided for in-kernel consumers
of randomness. The arc4random/arc4randbytes API is deprecated for
in-kernel use. It is replaced by "cprng_strong". The current
cprng_fast implementation wraps the existing arc4random
implementation. The current cprng_strong implementation wraps the
new CTR_DRBG implementation. Both interfaces are rekeyed from
the entropy pool automatically at intervals justifiable from best
current cryptographic practice.

In some quick tests, cprng_fast() is about the same speed as
the old arc4randbytes(), and cprng_strong() is about 20% faster
than rnd_extract_data(). Performance is expected to improve.

The AES code in src/crypto/rijndael is no longer an optional
kernel component, as it is required by cprng_strong, which is
not an optional kernel component.

The entropy pool output is subjected to the rngtest tests at
startup time; if it fails, the system will reboot. There is
approximately a 3/10000 chance of a false positive from these
tests. Entropy pool _input_ from hardware random numbers is
subjected to the rngtest tests at attach time, as well as the
FIPS continuous-output test, to detect bad or stuck hardware
RNGs; if any are detected, they are detached, but the system
continues to run.

A problem with rndctl(8) is fixed -- datastructures with
pointers in arrays are no longer passed to userspace (this
was not a security problem, but rather a major issue for
compat32). A new kernel will require a new rndctl.

The sysctl kern.arandom() and kern.urandom() nodes are hooked
up to the new generators, but the /dev/*random pseudodevices
are not, yet.

Manual pages for the new kernel interfaces are forthcoming.

Remove uscanner(4) driver

This exists for compatibility with a Linux interface which was apparently
deprecated in Linux 2.6. There are various mailing list threads going
back to 2004 where the usefulness of this driver is discussed, but
the conclusion is that scanner software has all moved to using ugen(4)
instead, and enabling this driver will not help you scan things.

branches: 1.10.16;
Remove the satlink driver. It was disabled everywhere, had no man page and
no use either. Spotted by thorpej in PR/21345, ok christos.

Remove compat_svr4 and compat_svr4_32, as discussed on tech-kern@ recently,
but also as discussed several times in the past.

branches: 1.8.14; 1.8.16;
Don't call the drvctl module's initialization code during rump component
initialization. Instead, temporarily attach the drvctl's cdevsw to
determine its device c-major, create the /dev/drvctl node using that
c-major, and then detach. This leaves things in a state where normal
module initialization can run.

Since we're now creating the /dev/drvctl device node correctly, we don't
need to create it again. So mark the device as DEVNODE_DONTBOTHER in
the devsw_conv0 conversion table.

This bug was introduced more than a year ago (src/sys/kern/kern_drvctl.c
rev 1.40), but was silently ignored except when running a rump_server
built with LOCKDEBUG.

branches: 1.7.14; 1.7.32; 1.7.36;
Don't autogenerate a large number of unnecessary device nodes, just
slows bootstrap.

support /dev/zero

branches: 1.5.2; 1.5.4;
"regen": /dev/md*

branches: 1.4.2; 1.4.4;
Borrow /dev/mem major from i386 for /dev/null and autogenerate node.

add /dev/putter to "devfs" (forgot to commit this earlier)

regen for vnd (or at least something like that since this file
still isn't really generated, just copypasted)

branches: 1.1.2;
Include devsw_conv0 from an i386 kernel compilation (no, we don't
care about the arch as long as all the devices we care about are
present). The file should be autogenerated, but that requires some
more changes to config(1).

initialize {sys,max}_{cb}devsws next to what they refer to

e.g. {sys,max}_cdevsws next to the definition of cdevsw.

branches: 1.201.6;
rump: Sprinkle sys/syncobj.h here too.

Fix root search in RAID 1 sets

We use the wedge information given by bootstrap, where the kernel was
found. This requires src/sys/arch/i386/stand/i386/lib/biosdisk.c 1.59
to work in all cases.

rumpkern: Provide stub pnbuf_cache weak alias.

Needed only by sys_module.c handle_modctl_load, which won't work if
there's no rumpvfs linked in anyway.

rumpkern: Provide fs_filtops weak alias.

We should introduce a link set for sys_kfilters, but this works
around the problem for now -- librump's dependency on librumpvfs
symbols.

nkmempages should be size_t

branches: 1.196.20;
Rewrite entropy subsystem.

Primary goals:

1. Use cryptography primitives designed and vetted by cryptographers.
2. Be honest about entropy estimation.
3. Propagate full entropy as soon as possible.
4. Simplify the APIs.
5. Reduce overhead of rnd_add_data and cprng_strong.
6. Reduce side channels of HWRNG data and human input sources.
7. Improve visibility of operation with sysctl and event counters.

Caveat: rngtest is no longer used generically for RND_TYPE_RNG
rndsources. Hardware RNG devices should have hardware-specific
health tests. For example, checking for two repeated 256-bit outputs
works to detect AMD's 2019 RDRAND bug. Not all hardware RNGs are
necessarily designed to produce exactly uniform output.

ENTROPY POOL

- A Keccak sponge, with test vectors, replaces the old LFSR/SHA-1
kludge as the cryptographic primitive.

- `Entropy depletion' is available for testing purposes with a sysctl
knob kern.entropy.depletion; otherwise it is disabled, and once the
system reaches full entropy it is assumed to stay there as far as
modern cryptography is concerned.

- No `entropy estimation' based on sample values. Such `entropy
estimation' is a contradiction in terms, dishonest to users, and a
potential source of side channels. It is the responsibility of the
driver author to study the entropy of the process that generates
the samples.

- Per-CPU gathering pools avoid contention on a global queue.

- Entropy is occasionally consolidated into global pool -- as soon as
it's ready, if we've never reached full entropy, and with a rate
limit afterward. Operators can force consolidation now by running
sysctl -w kern.entropy.consolidate=1.

- rndsink(9) API has been replaced by an epoch counter which changes
whenever entropy is consolidated into the global pool.
. Usage: Cache entropy_epoch() when you seed. If entropy_epoch()
has changed when you're about to use whatever you seeded, reseed.
. Epoch is never zero, so initialize cache to 0 if you want to reseed
on first use.
. Epoch is -1 iff we have never reached full entropy -- in other
words, the old rnd_initial_entropy is (entropy_epoch() != -1) --
but it is better if you check for changes rather than for -1, so
that if the system estimated its own entropy incorrectly, entropy
consolidation has the opportunity to prevent future compromise.

- Sysctls and event counters provide operator visibility into what's
happening:
. kern.entropy.needed - bits of entropy short of full entropy
. kern.entropy.pending - bits known to be pending in per-CPU pools,
can be consolidated with sysctl -w kern.entropy.consolidate=1
. kern.entropy.epoch - number of times consolidation has happened,
never 0, and -1 iff we have never reached full entropy

CPRNG_STRONG

- A cprng_strong instance is now a collection of per-CPU NIST
Hash_DRBGs. There are only two in the system: user_cprng for
/dev/urandom and sysctl kern.?random, and kern_cprng for kernel
users which may need to operate in interrupt context up to IPL_VM.

(Calling cprng_strong in interrupt context does not strike me as a
particularly good idea, so I added an event counter to see whether
anything actually does.)

- Event counters provide operator visibility into when reseeding
happens.

INTEL RDRAND/RDSEED, VIA C3 RNG (CPU_RNG)

- Unwired for now; will be rewired in a subsequent commit.

- Eliminate the global "boottime" variable, which was being accessed
without any synchronization against changes by e.g. clock_settime().
- Replace with new getbinboottime() / getnanoboottime() / getmicroboottime()
functions (naming mirrors that of other time access functions in kern_tc.c).
It returns the (maybe-converted) value of timebasebin, which also tracks
our estimate of when the system was booted (i.e. the legacy "boottime" was
redundant).

XXX There needs to be a lockless synchronization mechanism for reading
timebasebin, but this is a problem in kern_tc.c that pre-existed these
"boottime" changes. At least now the problem is centralized in one location.

Add shutting_down variable for rump.

- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

Provide a weak alias for vnode_to_path to be used unless librumpvfs is present.

branches: 1.191.2;

Apply a patch from hannken@ which adds a weak alias for rump_getcwd_common()
allowing -lrump to be used without -lrumpvfs.

This is an alternate fix to the earluer one which added -lrumvfs to
many rump based tests (and the rump server) which might be undone soon.

This also fixes the sun2 build.

Rumpkernel has its own thread deallocation. Add missing fstrans_lwp_dtor()
to lwproc_freelwp().

PR bin/50350: rump/rumpkern/t_sp/stress_{long,short} fail on Core 2 Quad

no more need for get_expose_address() here.

add get_expose_address()

Bring back three state file system suspension:

NORMAL -> SUSPENDING -> SUSPENDED

and add operation fstrans_start_lazy() that only blocks while SUSPENDED.

Change vndthread() support operation handle_with_rdwr() to bracket
its file system operations by fstrans_start_lazy() and fstrans_done().

PR kern/53624 (dom0 freeze on domU exit)

Allow syscall_establish() to install new syscalls when the existing
entry-point is either sys_nomodule or sys_nosys. Update the
makesyscalls.sh script to create a const array of bits to allow
syscall_disestablish() to properly restore the original entry-point.
Update all the initializers of struct emul to initialize the pointer
to the bit array struct emul.

XXX Regen of all files created by makesyscalls.sh will come soon,
XXX followed by a kernel version bump (since struct emul is being
XXX modified).

This commit should address PR kern/45781 and also removes the need
for the work-around for that PR in file

sys/arch/usermode/modules/syscallemu/syscallemu.c

branches: 1.185.2; 1.185.4;
Add missing inclusion of pserialize.h (fix build)

Implement debugging feature for pserialize(9)

The debugging feature detects violations of pserialize constraints.
It causes a panic:
- if a context switch happens in a read section, or
- if a sleepable function is called in a read section.

The feature is enabled only if LOCKDEBUG is on.

Discussed on tech-kern@

added booted_method

Operations fstrans_start() and fstrans_start_nowait() now always
use FSTRANS_SHARED as lock type so remove the lock type argument.

File system state FSTRANS_SUSPENDING is now unused so remove it.

Regen vnode_if files.

Ride 8.99.1 less than a hour ago.

branches: 1.181.6;
Add weak aliases for _fstrans_start() and fstrans_done().

branches: 1.180.2;
physmem should be of type psize_t

Also, use PRIxPSIZE when printf(9)ing physmem.

branches: 1.179.2;
Put the kernelside rump kernel headers into <rump-sys> instead of
sprinkling them around the faction directories. Avoids having
to add a CPPFLAGS (or several) to pretty much every component
Makefile.

Leave compat headers around in the old locations.

The commit changes some autogenerated files, but I'll fix the
generators shortly and regen.

Fix build with RUMP_LOCKDEBUG=yes

put lwp/proc stuff into the same source module (emul.c -> lwproc.c)

massively reduce header pollution from times prehistoric

boottime is a timespec, not timeval

Now that the table of auto-loadable syscalls is per-emulation, make sure
that the rump-kernel has its own list. Otherwise, missing syscalls won't
trigger a module auto-load.

This commit finishes the work to get tests/lib/librumphijack/t_tcpip
nfs_autoload test case working again after 16 months of failures! (see
PR bin/49153).

add cpu_getmodel()

Since the rump kernel does not know when the container it's running in
actually halts, print "halted" in the hypercall.

Include kern_clock.c in rump kernels.

Give remote clients struct pmap storage. Although the pmap is unused,
that way we can sure that the pmap macro framework doesn't access all
the wrong places.

Put all sysproxy routines to their own C module, sysproxy.c

branches: 1.168.2;
delay() can't use rumpuser_clock_sleep(), because that may unschedule().

Instead, busyloop with calls to rumpuser_clock_gettime(). Fixing delay()
properly should addressed in hypercall rev.18 by introducing a nowrap
variant/flag for rumpuser_clock_sleep().

Attempt to make sure that DELAY()/delay() is available in all MD names.

Add missing __diagused in rump code

cpu_reboot() is more of an emul.c thing than a rump.c thing

branches: 1.164.2;
Put the syncobjs in emul.c instead of locks.c so that they are
present for both locks.c and locks_up.c alike.

+ resettodr

lwp0 needs l_proc set for panic->cpu_reboot to dump core the right way

Use subr_cprng.c instead of stub implementation. Rijndael migrates from
rumpkern_crypto to rumpkern due to it being mandatory for cprng.

Translate return values for emulations, e.g. Linux. For ports without
__HAVE_MINIMAL_EMUL, we simply look up the values from p->p_emul->e_errno.
For ports which cannot afford to keep an extra pointer per emul structure
around, we hope there is __HAVE_SYSCALL_INTERN support and thread the
errno values through p_emuldata. Notably, we cannot alter the syscall
method like most ports do with syscall_intern, since they do it via
p_mdproc, so MI code is not possible there.

Support ktrace for rump kernels.

Requested by Justin Cormack on rumpkernel-users.

branches: 1.158.4;
Flip parameter order in the clock hypercalls to make them consistent
with everything else wrt in/out parameters.

Make hypercall calling conventions consistent: iff a hypercall can fail,
it returns an int containing the error value.

Improve the time-related hypercalls so that's it's possible to
sleep until an absolute time on the host's monotonic clock (should
something like that be supported).

calculate vnode cache size based on the resource it gets allocated from
this stops setting kern.maxvnodes to high so it exhausts available space in kmem

http://mail-index.netbsd.org/tech-kern/2013/03/08/msg015095.html

set e_nsysent

Stopgap fix to make rump cooperate with pserialize, may be revisited later.
Patch from pooka, ok: rmind. No related regressions in a complete atf test
run (which works again with this, even on non x86 SMP machines).

Provide xc_send_ipi() routine in RUMP, which is required for high-priority
xcall(9) mechanism. It is emulated using low-priority xcall(9).

split device_t/softc for all remaining drivers.
replace "struct device *" with "device_t".
use device_xname(), device_unit(), etc.

branches: 1.150.4; 1.150.10; 1.150.14;
Update copyright statements.

no functional change.

Since "physmem" is largely unused except for zfs wanting to know
if it has enough memory available, just pick a number which makes
zfs happy.

We *could* use a MIN() of available host mem and rump_memlimit,
though ...

branches: 1.148.2; 1.148.4;
Support LOCKDEBUG. To use it, compile sys/rump with RUMP_LOCKDEBUG=yes.

requested by martin (sparc64 gdb cannot reliably produce a stack trace)

Realize the >1yo comment above rump_reboot and retire them to make
room for sys_reboot.

Make common kernel module binaries work on both sun3 and sun3x.
Tested on 3/160 (on TME) and (real) 3/80.

XXX: module files can be loaded only on single user?

Implement rump_lwproc: the new lwp/proc management routines for
rump. These move the management of the pid/lwpid space from the
application into the kernel, make code more robust, and make it
possible to attach multiple lwp's to non-proc0 processes.

As normal, fix breakage from untested commits by rmind.

Remove comment which wasn't supposed to get committed.

Make some stubby tty routines used by printf weak aliases, so that
the full ones will be used if we have the upcoming tty component
loaded.

Fix previous in emul.c -- only numbers are operands for cpp comparisons.
Apparently non-numbers logically produce arch-dependent behaviour.

deal with sun3's module disability

add rest of sparc pagesize variables

Use kern_proc.c instead of a collection of stubs. But what we
really wanted from this commit was the support for proc_specificdata.

TODO: make creating a new process actually use kern_proc and
maybe even add an interface which starts a process with
"any pid you don't like"

adjust comment in previous.
XXX: should make that (and physmem) mean something here

remove unnecessary #ifdef

Now that we have a big devsw_conv0, need at least as many entries
in devsw as the highest device number in the conv table. Do a
"good for a hundred years" guess now, fix properly later.

Include devsw_conv0 from an i386 kernel compilation (no, we don't
care about the arch as long as all the devices we care about are
present). The file should be autogenerated, but that requires some
more changes to config(1).

Make initial devsw a little more believable. Especially adjust
max_sys_devsw, since it was previously a bit too much, given that
the correct value is 0.

remove stuff which now comes via std. kernel source modules

support kern_resource

Move all signal-related from emul.c to signals.c. Additionally,
define a few alternate signal models for the rump kernel, including
ones where signals are ignored or sent to host processes.

no \n in panic

One emul is enough and since we need emul_netbsd, retire emul_rump.

Move scheduling related routines from emul.c to scheduler.c

Use kern_syscall.c instead of homegrown syscall dis/establishment routines.

Include kern_tc and use a timecounter driver instead of homerolled
kern_tc implementation.

We don't have a real rootdev (by default at least), so set it to NODEV.

branches: 1.123.2;
add signal stubs required by mfs

Remove separate mb_map. The nmbclusters is computed at boot time based
on the amount of physical memory and limited by NMBCLUSTERS if present.
Architectures without direct mapping also limit it based on the kmem_map
size, which is used as backing store. On i386 and ARM, the maximum KVA
used for mbuf clusters is limited to 64MB by default.

The old default limits and limits based on GATEWAY have been removed.
key_registered_sb_max is hard-wired to a value derived from 2048
clusters.

branches: 1.121.2;
Include newly-created subr_device.c and remove few special case
device accessor copypastes. This makes it possible to link static
binaries which use -lrumpdev.

include kern_hook.c

Use subr_percpu.c instead of homegrown implementation. ...except
when using malloc(3)-relegated allocators (happens in production
e.g. on Linux), since subr_percpu.c uses vmem and i don't want to
reimplement vmem.

Minimize unnecessary differences in rump.

Add a few symbols used by the tty code.

Include kern_pmf.c in rumpdev.

Expand PMF_FN_* macros.

allow rump to build

Generate vers.c and include it in the kernel component.

Decide it's not worth the fuss to have rumpfs as a module and just
hardcode attach into rump_vfs_init(). Saves us from a lot of
pingpong init bouncing from one component to another to get the order
right.

Include cpu crosscall support (instead of stubbing it out).

For rumpfs, do mountroot instead of the bunch of homegrown hacks
currently there. Still needs a little massage to get the kernel
interfaces right and avoid copypaste especially from main().

Also, move it a bit more into the direction of a real file system
(finally!) by giving it a vfsops. Most ops are still unimplemented,
though.

misc_stub and emul have been the same thing for a looong time now,
so just move the few remaining routines in misc_stub to emul.

Give the kthread->pthread interface emulation its own module.

Pull all relegating memory allocators under a common roof in memalloc.c

move copy-related routines to their own module

Use std. uiomove() & friends.

Sync rump with kernel changes.

Include sys_select.c for proper select()/poll() support.

Give lwp usage some much-needed love: stop treating lwp0 as the
all-sink and make sure each separate thread in rump has its own
lwp. Happy-go-lucky callers will get scheduled a temporary lwp
on entry, while true lwp connoisseurs may request a stable lwp
for their purposes. Some more love may be required later down the
road, but for now different threads will stepping on each others
toes.

Add initial work on a rump virtual cpu scheduler. This is necessary
for kernel code which has been written to avoid MP contention by
using cpu-local storage (most prominently, select and pool_cache).

Instead of always assuming rump_cpu, the scheduler must now be run
(and unrun) on all entry points into rump. Likewise, rumpuser
unruns and re-runs the scheduler around each potentially blocking
operation. As an optimization, I modified some locking primitives
to try to get the lock without blocking before releasing the cpu.

Also, ltsleep was modified to assume that it is never called without
the biglock held and made to use the biglock as the sleep interlock.
Otherwise there is just too much drama with deadlocks. If some
kernel code wants to call ltsleep without the biglock, then, *snif*,
it's no longer supported and rump and should be modified to support
newstyle locks anyway.

"rumppriv" goes back to "rump" per internal interface naming change.

Adjust rump sources for external/internal interfaces.
No functional change.

Provide an interface for reboot.

Need proc_uidmatch

remove a no longer necessary hack

default to AB_SILENT

provide pmf stubs

move hppa cpp kludge to the end of the file

add a few global symbols required by kernel code

realloc works much better if it actually uses realloc instead of malloc

Nuke some bss stuff which is coming from vfs_bio via rumpvfs.

Don't create unpgc thread if RUMP_THREADS is 0. The benefit is the
usual: possible to run programs using AF_LOCAL without threads.

Add proof-of-concept code for enabling system calls to rump virtual
kernels running in other processes on the same machine or on an
entirely different host. I wrote this a while ago and am now
committing it mainly to avoid losing it. It works, but could do
with a little tuning here and there.

What this will hopefully eventually buy us is the ability to use
standard userland tools to configure rump kernels, e.g. ifconfig(8)
and route(8) could be used to configure the networking stack provided
by a rump kernel. Also some distributed OS implications may apply.

fun fact: a system call which just does copyin/copyout takes >1000x
longer when made over the LAN as compared to when made on the same
machine.

Fix getnano/microuptime to report actual uptime.

Catch up with recent changes to sys/proc.h (Hello, rmind!)

we want microuptime too.

need getnanouptime not getmicrouptime

one more timeval -> timespec line

Catch up with kernel changes

Support kqueue in the rump virtual kernel.

Ansify function definitions w/o arguments. Generated with sed.

decouple rumpuser gettime from struct timeval

Make rumpuser stat and nanosleep independent of the host system
stat and timespec definitions.

branches: 1.77.2;
Make the clock interrupt tick based on absolute time instead of
relative time. This prevents drifting. Also, keep track of time
within userspace, so we do not have to make a syscall to get the
clock value. This is approximately 7 times cheaper, but on the
negative side is limited to the clock interrupt frequency.

Implement block I/O as a real block driver instead of a hacked copy
of specfs. That was easier years ago when rump didn't support
devices, but brings no gain now. This allows us to include the
real specfs in rump.

Shove in some hppa love to help with spcopy.S (why does that need
to be in libkern?).

Protect against nfssilly in RUMP_THREADS=0

merge christos-time_t

provide kern_realloc

Rename malloc() to kern_malloc() to avoid name conflict with libc.
Now it is possible to use the kernel linker with rump.

Include libkern contents in librump.

Sauce with some kludges:
* revert rev 1.68 of emul.c: delay() is hopelessly MD and needs
whipping (not the fun kind)
* #undef DELAY in cpu.h

This will hopefully fix build for all archs and give me time to
hop through our 46521 fine archs and edit the headers.

Export delay, _delay and delay_func symbols to attempt to appease
the macro mania set over all architectures.

Include kernel printf routines in rump instead of relying on the
magic libc symbol. This also allows to bid farewell to subr_prf2.c
and merge the contents back to subr_prf.c. The host kernel bridging
is now done via rumpuser_putchar().

Define MODULAR for rump core components. This enables module
loading via the kernel module framework (instead of dlopen()).
For now it only works on amd64 and i386, but for the rest it should
just be a matter of including the relevant kobj_machdep.c modules
from the kernel sources.

properly handle kthread_exit()

include subr_devsw in rumpkern

branches: 1.63.2;
undo turn malloc flags to enums.
Requested by christos.

Keep malloc flag defines in kernel section.

Turn malloc flags into an enum. Fixes build breakage.

__KERNEL_RCSID

If available (__NetBSD__), use pthread_setname_np() to set the
thread name for kthread_create().

If the arch uses __BSWAP_RENAME, provide non-namespaced kernel
symbols which just call the libc symbols. Allows to get rid of
rump machine/bswap.h.

XXX:
1) this is unlikely to work on non-NetBSD
==>
2) should look into including libkern in librump

Make non-mpsafe threads KERNEL_LOCK() before starting execution.

Remove the whole magic ubc window thingie. All file systems use
ubc_uiomove() now, so we can hook ourselves there.

Add a few symbols required by nfsd

Add more verbose commentage to the module thread non-creation.

Ignore the module unload thread.

branches: 1.53.2; 1.53.4; 1.53.8;
Add some more trivial emulations.

Provide psignal() so that we don't get linked against psignal(3).
As there is no real process model currently, just "trap" the signals
appropriately.

Support callouts and call callout_hardclock() from the timer
interrupt thread.

The sleepq implementation required for callouts is horrible, kludged
only for callouts, and generally unacceptable. It needs revisiting,
but I'm not sure yet should rump or kern_timeout be improved. It's
almost untested as of now, but committing this will give me some
maneuvering space while letting application compile.

Remove rumpuser_yield(). Not only doesn't it really make sense
here, some kind soul made it completely empty.

Switch to std kern_auth.

Move global malloc types from kern_malloc into a separate module.

Make it possible to control starting of threads per env variable
instead of only at compile-time.

Add support for using real kmem/vmem. Don't enable it by default,
though, since it a) is a lot of unnecessary indirection in rump
b) requires callouts which are so far unimplemented.

support real sysctls

Install rump libraries and utilities to the base system and remove the
private non-installed build infrastructure from sys/rump.

breakdown of commit:
* install relevant headers into /usr/include/rump
* build sys/rump/librump/rumpuser and sys/rump/librump/rumpkern
from src/lib and install as librumpuser and librump, respectively
+ this retains the ability to test a librump build with just the
kernel sources at hand
* move sys/rump/fs/lib/libukfs and sys/rump/fs/lib/libp2k to src/lib
for general consumption, they are not kernel-space dwellers anyway
* build and install sys/rump/fs/lib/lib$fs as librumpfs_$fs
* add chapter 3 manual pages for rump, rumpuser, ukfs and p2k
* build and install userspace kernel file system daemons if MKPUFFS=yes
is spexified
* retire fsconsole for now, it will make a comeback with an actually
implemented version shortly

Repeat after me: do not pollute sys with #ifdef _RUMPKERNEL

emulate vlog()

branches: 1.41.2;
Don't compile kern_lock for rump any more, it's no longer required.
Allows us to get rid of the incorrect _RUMPKERNEL ifdefs outside sys/rump.

Make kpause() use nanosleep() instead of usleep(). Fixes >=1s sleeps
to actually sleep a bit too.

from Arnaud Ysmal

branches: 1.39.2;
Also fake the namecache g/c thread in kthread_create() so that file
systems can again be run without threads (and hence gdb can be used
on them).

branches: 1.38.2; 1.38.4;
Catch up with process locking changes.

branches: 1.37.2;
Add a device_xname() stub.

- for some ports, especially for ones without pmap_growkernel,
buf_memcalc is used by bootstrap as well. fix NULL dereference for them.
- limit kva usage for each cache to 20% of vm_map. XXX a bit arbitrary.
- add a comment.

Adapt to sel* changes

when calculating some cache sizes, consider the amount of available kva.
PR/33185.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

Add lwp_unsleep from kern/kern_lwp.c so that this builds. ok ad@

Support multiple file systems within one process with ukfs by using
a "chroot" for each file system.

Backup some fixes for recent breakage from local tree. Also some
other improvements such as exporting the real kernel namei and
using that in ukfs instead of the homegrown heap'o hacks namei.
"etcetc".

branches: 1.29.2; 1.29.6;
Implement yield().

Add dummy hardclock_ticks.

Use vfs_subr.c from sys/kern. This brings differences in the vnode
life cycle between rump and a real kernel to a minimum.

Use namei() etc. from kernel sources instead of a reimplementation.
To accommodate, give the rootvnode its own vnode op vector with a
simple lookup operation. This is used for looking up the file
system's device vnode instead of doing that directly in a homesmoked
namei().

Don't use PAGE_SIZE when faking physmem, as it's not always available
at compile-time. We could init it runtime, but it's just a random
number anyway.

Default physmem was too tight, increase to 256megs from 0 bytes.

Until debugging threaded programs in NetBSD is fixed, supply the
cpp option RUMP_WITHOUT_THREADS as a workaround. If defined, it
makes rump itself operate single-threaded and prevents kthread_create()
from working.

Now that __HAVE_TIMECOUNTER and __HAVE_GENERIC_TODR are invariants,
remove the conditionals and the code associated with the undef case.

Print "panic: " before panicstr when panicking.

Add the ability to run puffs in userspace. This means that puffs
can now be developed in userspace using puffs for development
(I hate emulators, they are annoyingly clumsy).

To e.g. mount psshfs using puffs-on-puffs, run fs/bin/syspuffs/syspuffs
with the regular mount_psshfs command line as an argument:

golem> ./syspuffs /usr/sbin/mount_psshfs ftp.netbsd.org:/pub /puffs

This will make the mount appear as usual, with the exception that the
requests will be passed through puffs both in the kernel and userspace:

ftp.netbsd.org:/pub on /puffs type puffs|p2k|puffs|psshfs

branches: 1.19.6;
fix build (hi rmind!)

branches: 1.18.2;
* sprinkle some locking into the vm code
* avoid extra insert+search+remove step in file systems using
ubc_uiomove() instead of standard uiomove()

Make it possible to run rumps multithreaded. This brings real
locking and makes it possible to run file systems which create
threads. It also makes rump file system behaviour better match
file system behaviour in the kernel.

branches: 1.16.2;
"flags * (M_CANFAIL | M_NOWAIT)" is probably not correct. Use & instead.
And while committing, add other random cruft I've needed recently.

machine/{bus,cpu,intr}.h -> sys/{bus,cpu,intr}.h

branches: 1.14.2;
Don't whine about wakeup() not being implemented every time it's
called. Nobody cares unless ltsleep() is called (and it panics).

branches: 1.13.2;
If !_HAVE_TIMECOUNTER, initialize 'time' instead of 'time_second.'

Provide errno from rumpuser_{gettimeofday,close}() to be consistent.

branches: 1.11.2; 1.11.4;
few panicky functions

branches: 1.10.2;
Hide NetBSD kernel headers completely from ukfs. This includes creating
accessors for:
* struct mount & VFS ops
* struct uio
* struct vnode
* struct vattr

and some namespace games for:
* namei flags
* VOPs
* enum vtype

Also, split rump services into two categories: library private and public
(rump_private.h and rump.h, respectively).

As a result, it is now possible to compile and use the NetBSD kernel
file systems on Linux (and probably other systems too with very
little work), although the makefiles need a bit of work to make it
a pleasureable experience.

Wrap malloc() so that we catch the kernel arguments (namely M_ZERO)
properly. It's fairly amusing that this wasn't noticed until now.

branches: 1.8.2;
Kill handrolled buffercache and use vfs_bio from the kernel. This is
mostly to get the flag jungle in sync with the kernel.

Use supermarket variety vfs_vnops.c instead of homecooked routines
(that's actually a good thing here).

compile in libkern __assert.c, argument order differs from libc model

Add some stubs for lfs.

Provide rumpmachine bswap.h, which makes bswapxx() call rumpuser
directly instead of relying on a symbol in rumpkern. I would like
to make it call the libc symbol directly, but I don't currently know
how to make it do that MI.

Makes hfs work (on i386), as it avoids endless recursion in bswap64().
Thanks to dillo for the image!

* compile param.c
* move opts to a separate directory

add microtime

branches: 1.1.2;
Introduce RUMPs - Runnable Userspace Meta-Programs

/sys/rump contains programs which run unmodified kernel code in an
emulated userspace environment. The kernel environment is provided
by librump. Currently supported are a number of file systems,
which by using puffs integrate seamlessly into the system and
provide a similar user experience to if the code was running as
part of the kernel. Potential future rumpification targets include
for example parts of the networking stack and some device drivers.

This work was supported by Google Summer of Code 2007.

Put the kernelside rump kernel headers into <rump-sys> instead of
sprinkling them around the faction directories. Avoids having
to add a CPPFLAGS (or several) to pretty much every component
Makefile.

Leave compat headers around in the old locations.

The commit changes some autogenerated files, but I'll fix the
generators shortly and regen.

branches: 1.1.2; 1.1.4; 1.1.6; 1.1.10; 1.1.12;
Move the etfs linkage from rumpvfs to rumpkern, and replace the
weak alias show with an honest pointer indirection.

No client-visible change. (apart from this version working e.g.
on musl w/ dlopen)

Split vfs out of rumpkern into rumpvfs. Non-fs rumps no longer
include the file system code. File system rumps explicitly need
to include rumpvfs from now on.

branches: 1.6.2;
Have a COW with fscow_run(). Fixes problem of allocating files
on ffs with indirect blocks.

found by simonb

branches: 1.5.6; 1.5.10; 1.5.12; 1.5.14; 1.5.16;
Add the ability to run puffs in userspace. This means that puffs
can now be developed in userspace using puffs for development
(I hate emulators, they are annoyingly clumsy).

To e.g. mount psshfs using puffs-on-puffs, run fs/bin/syspuffs/syspuffs
with the regular mount_psshfs command line as an argument:

golem> ./syspuffs /usr/sbin/mount_psshfs ftp.netbsd.org:/pub /puffs

This will make the mount appear as usual, with the exception that the
requests will be passed through puffs both in the kernel and userspace:

ftp.netbsd.org:/pub on /puffs type puffs|p2k|puffs|psshfs

Merge vmlocking2 to head.

branches: 1.3.2; 1.3.6;
Make it compile again.

branches: 1.2.4; 1.2.6;
Update the file system copy-on-write handler.

- Instead of hooking the handler on the specdev of a mounted file system
hook directly on the `struct mount'.

- Rename from `vn_cow_*' to `fscow_*' and move to `kern/vfs_trans.c'. Use
`mount_*specific' instead of clobbering `struct mount' or `struct specinfo'.

- Replace the hand-made reader/writer lock with a krwlock.

- Keep `vn_cow_*' functions and mark as obsolete.

- Welcome to NetBSD 4.99.32 - `struct specinfo' changed size.

Reviewed by: Jason Thorpe <thorpej@netbsd.org>

branches: 1.1.2; 1.1.4; 1.1.6; 1.1.8; 1.1.10; 1.1.12;
Introduce RUMPs - Runnable Userspace Meta-Programs

/sys/rump contains programs which run unmodified kernel code in an
emulated userspace environment. The kernel environment is provided
by librump. Currently supported are a number of file systems,
which by using puffs integrate seamlessly into the system and
provide a similar user experience to if the code was running as
part of the kernel. Potential future rumpification targets include
for example parts of the networking stack and some device drivers.

This work was supported by Google Summer of Code 2007.

Compile most genfs routines directly out of the kernel sources.
Move special I/O routines to genfs_io.c to mirror kernel naming.

Merge from vmlocking:

- Split vnode::v_flag into three fields, depending on field locking.
- simple_lock -> kmutex in a few places.
- Fix some simple locking problems.

branches: 1.18.2; 1.18.4;
* fill struct buf a little better
* hop, skip & jump to make uvm_pageratop work

branches: 1.17.2;
g/c comment invalidated by previous

* in getpages, make sure we always align our buffer size to the page size
* in putpages, make sure we don't try attempt to push data in a page
after eof. also, skip blocks which bmap can't locate (see comment
in code).

the effect: file systems with fs_bsize < PAGE_SIZE now work r/w

branches: 1.15.2;
Hide NetBSD kernel headers completely from ukfs. This includes creating
accessors for:
* struct mount & VFS ops
* struct uio
* struct vnode
* struct vattr

and some namespace games for:
* namei flags
* VOPs
* enum vtype

Also, split rump services into two categories: library private and public
(rump_private.h and rump.h, respectively).

As a result, it is now possible to compile and use the NetBSD kernel
file systems on Linux (and probably other systems too with very
little work), although the makefiles need a bit of work to make it
a pleasureable experience.

branches: 1.14.2;
Kill handrolled buffercache and use vfs_bio from the kernel. This is
mostly to get the flag jungle in sync with the kernel.

Use supermarket variety vfs_vnops.c instead of homecooked routines
(that's actually a good thing here).

* move rump_vopwrite_fault() into history - we now support the file
system faulting in pages if it does e.g. fragment reallocation
* get rid of rumpvm_findpage() and always use uvm_pagelookup()
* determine a vnode's cleanness by flagging it as being on the work
list if we "take" a write fault and removing it from the worklist
once pages are flushed. There is no work list here, but at least
there is symmetry with the kernel.

Fix it a bit & wait for the dust to settle. Also, enable UBC by default.

ffs in userspace on top of puffs/p2k/rump is now stable enough to
host a make -j4 kernel build (well, at least my kernel build ...
but, yes, I am currently running that kernel on my desktop)

* fix symlink
* actually call a couple of VOPs

deal with holes in getpages()

fool some KASSERTs

Add some stubs for lfs.

* adjust device block number based on if we're putting pages to a
fs block boudary or not
* round writes to DEV_BSIZE, just to be kosher

remove the allocstorage param from makepage - not needed

track dirty vm objects

offset orgies, part n+1. this is still not right, but at least it's better

branches: 1.2.2;
Instead of going for a PAGE_SIZE nuke, do getpages() in a more
sniper style. Makes efs (& other file systems with bsize < PAGE_SIZE)
work better.

Introduce RUMPs - Runnable Userspace Meta-Programs

/sys/rump contains programs which run unmodified kernel code in an
emulated userspace environment. The kernel environment is provided
by librump. Currently supported are a number of file systems,
which by using puffs integrate seamlessly into the system and
provide a similar user experience to if the code was running as
part of the kernel. Potential future rumpification targets include
for example parts of the networking stack and some device drivers.

This work was supported by Google Summer of Code 2007.

Split vfs out of rumpkern into rumpvfs. Non-fs rumps no longer
include the file system code. File system rumps explicitly need
to include rumpvfs from now on.