apply -Wno-dangling-pointer to callcontext.c.

puffs_cc_getcc() uses a mask against a stack variable to find the
struct puffs_cc stashed below the stack, triggering the dangling
pointer problem.

libpuffs: ignore lint warning about nested extern for now

This should be properly fixed by someone familiar with puffs.

lint: warn about extern declarations outside headers

https://mail-index.netbsd.org/tech-userlevel/2023/03/15/msg013727.html

Allow calling puffs_mount(3) before puffs_daemon(3)

puffs_daemon(3) creates a pipe before forking, and the parent process
waits for the child to either complete puffs_mount(3) or fail. If a
user calls puffs_daemon(3) after puffs_mount(3), the function
deadlocks. While this error-reporting functionality is really a nice
thing to have, deadlocking is not great. If the filesystem has already
been mounted, puffs_mount(3) should just daemonize the process and
return.

This became an issue because fuse_daemonize(3) in FUSE API had no such
requirement and some FUSE filesystems in the wild suffered deadlocks.

Define _KERNTYPES for things that need it.

These directories default to WARNS?=5

branches: 1.24.6;
Remove PUFFS_WITH_THREADS ifdef garbage, just use libc weak symbols.

Remove suspension support from here too.

Part 3/n of extensive changes to message handling:

Get rid of the original puffs_req(3) framework and use puffs_framebuf(3)
instead for file system requests. It has the advantage of being
suitable for transporting a distributed message passing protocol
and therefore us being able to run the file system server on any
host.

Ok, puffs is not quite here yet: libpuffs needs to grow request
routing support and the message contents need to be munged into a
host independent format. Saying which format would be telling,
but it might begin with an X, end in an L and have the 13th character
in the middle. Keep an eye out for the sequels: Parts 3+m/n.

Add some very initial support for a threading worker model as an
alternative to the (vastly superior ;) continuation model. This
is very preliminary stuff and not compiled by default (which it
even won't do without some other patches I cannot commit yet).

The raison d'commit of the patch is a snippet which ensures proper
in-order dispatching of all operations, including those which don't
require a response. Previously many of them would be dispatched
simultaneosly, e.g. fsync and reclaim on the same node, which
obviously isn't all that nice for correct operation.

branches: 1.20.4; 1.20.6;
get rid of DBG=

Add new Makefile knob, USE_FORT, which extends USE_SSP by turning on the
FORTIFY_SOURCE feature of libssp, thus checking the size of arguments to
various string and memory copy and set functions (as well as a few system
calls and other miscellany) where known at function entry. RedHat has
evidently built all "core system packages" with this option for some time.

This option should be used at the top of Makefiles (or Makefile.inc where
this is used for subdirectories) but after any setting of LIB.

This is only useful for userland code, and cannot be used in libc or in
any code which includes the libc internals, because it overrides certain
libc functions with macros. Some effort has been made to make USE_FORT=yes
work correctly for a full-system build by having the bsd.sys.mk logic
disable the feature where it should not be used (libc, libssp iteself,
the kernel) but no attempt has been made to build the entire system with
USE_FORT and doing so will doubtless expose numerous bugs and misfeatures.

Adjust the system build so that all programs and libraries that are setuid,
directly handle network data (including serial comm data), perform
authentication, or appear likely to have (or have a history of having)
data-driven bugs (e.g. file(1)) are built with USE_FORT=yes by default,
with the exception of libc, which cannot use USE_FORT and thus uses
only USE_SSP by default. Tested on i386 with no ill results; USE_FORT=no
per-directory or in a system build will disable if desired.

move puffs request dispatching code into it's own module.
no functional change

add stubby version of puffs_framebuf manual page. proper one
forthcoming to an internet near you Real Soon Now(tm)

Add puffs "frame buffers", which are an abstraction of the buffering
and event handling mechanisms required in file servers with blocking
I/O backends. puffs_framebuf is built on the concept of puffs_cc
and uses those to multiplex execution where needed.

File systems are required to implement three methods:
* read frame
* write frame
* compare if frame is a response to the given one

Memory management is provided by puffs_framebuf, but the file
systems must still, of course, interpret the protocol and do e.g.
byte order conversion.

As always, puffs_framebuf is work in progress. Current users are
mount_psshfs and mount_9p.

Move file system and node callbacks into their own manual page.
Improve markup and descriptions a bit (not completely finished yet).

document credentials & access control methods

document puffs_fs_suspend()

add interfaces for suspending the file system

document request handling routines

Bring the documentation slightly more into sync with the current
reality. It's still really a far cry from proper documentation,
though.

Checkpoint some more work in progress: for the benefit those file
system backends which operate purely based on paths, push out more
path management into the library and make path management more
abstract: enable a file system to define a bunch of path management
callbacks, which are used by the framework. Management of normal
/this/is/a/path type paths is provided by the library.

Add preliminary code for a nullfs layer, which can be used to mount
a directory hierarchy to another point, just like with the kernel
nullfs. This is not really a layering scheme yet, but it should
evolve into one. Currently it can just be used to do 1:1 mapping.

lib interface for invalidation routines

checkpoint some experimental work-in-progress, namely:

Add support for having multiple outstanding operations. This is done
by exposing enough interfaces so that it is convenient to have the
main event loop in the implementation itself and by providing a
continuation framework for convinient blocking and rescheduling.

works fine, but will undergo further cleanup & development

outline & export code for interfacing with kernel GET/PUT requests

initial documentation for the puffs library interface

pass lint; fix install.

add the .include <bsd.own.mk>, so .d and .o files don't end up in the CWD.
remove INCDIR=/usr/include, as my leenooks system doesn't have puffs .h files.

add libpuffs - the userspace programming interface for puffs

This work was initially started and completed for Google SoC 2005
and tweaked to work a bit better in the past few weeks. While
being far from complete, it is functional enough to be able and
stable to host a fairly general-purpose in-memory file system in
userspace. Even so, puffs should be considered experimental and
no binary compatibility for interfaces or crash-freedom or zero
security implications should be relied upon just yet.

The GSoC project was mentored by William Studenmund and the final
review for the code was done by Christos.

branches: 1.3.16;
stuff

pid is gone

quick TODO-list. very incomplete

branches: 1.1.2;
file Version.map was initially added on branch agc-symver.

revert previous now that MAP_ALIGNED has been fixed.

correct previous cast.

cast for lint

branches: 1.27.38; 1.27.40;
Deal with __MACHINE_STACK_GROWS_UP machines.

constify the other psize while I'm here.

Thanks to Christos for looking.

branches: 1.26.2;
puffs_cc_yield: fix a cc leak. PR/44679

puffs__cc_destroy: clear PCC_HASCALLER. the caller info is not
necessarily relevant to the next life of the cc. this also fixes
the failure of the assertion in the next line.

branches: 1.24.2;
typo in a comment

clear cached puffs_cc's in puffs_exit()

* make sure we come out of mainloop in the main context. otherwise in
certain cases the server would jump to hyperspace when unmounted
* sprinkle some debug prints

branches: 1.21.6;
Don't add asserts between testing and commit - especially not
if you're going to write "assert(foo = 0)"

Implement lazy contexts for file systems using puffs_mainloop().
Previously each request was executed on its own callcontext and
switched to every time the request was being processed. Now requests
borrow the mainloop context and switch only if/when they yield.
This takes two context switches away from a file system request
bringing down the typical amounts 2->0 (e.g. dtfs) and 4->2 (e.g.
psshfs).

The interfaces for manually executing requests changed a bit:
puffs_dispatch_create() and puffs_dispatch_exec() must now be used.
They are not tested, as nothing in-tree wants them and I doubt
anyone else is really interested in them either.

Also do some misc code cleanup related to execution contexts. The
"work-in-progress checkpoint" committed over a year ago was starting
to look slightly weed-infested.

Fix fakecc case.

Noticed by Pavel Jirout on current-users

Cache a number of execution contexts instead of re-initializing
them every time. Speeds up pure in-memory file systems such as
sysctlfs or dtfs quite a bit. For actual I/O-workhorses the result
is of course less tasty.

Use PROT_NONE page to protect bottom of stack from overrun

* nuke puffs_cc_get{specific,usermount} for good
* move prototypes for puffs_docc and puffs_dopufbuf into the
public header, as they are should be exposed

Part 3/n of extensive changes to message handling:

Get rid of the original puffs_req(3) framework and use puffs_framebuf(3)
instead for file system requests. It has the advantage of being
suitable for transporting a distributed message passing protocol
and therefore us being able to run the file system server on any
host.

Ok, puffs is not quite here yet: libpuffs needs to grow request
routing support and the message contents need to be munged into a
host independent format. Saying which format would be telling,
but it might begin with an X, end in an L and have the 13th character
in the middle. Keep an eye out for the sequels: Parts 3+m/n.

* allocate pcc as a slice from the stack instead of allocating it
separately
* provide puffs_cc_getcc()

This is in preparation for the removal of you-should-guess-what as
an argument to routines here and there and everywhere.

Fix a few bugs and memleaks with the threading code.

Add some very initial support for a threading worker model as an
alternative to the (vastly superior ;) continuation model. This
is very preliminary stuff and not compiled by default (which it
even won't do without some other patches I cannot commit yet).

The raison d'commit of the patch is a snippet which ensures proper
in-order dispatching of all operations, including those which don't
require a response. Previously many of them would be dispatched
simultaneosly, e.g. fsync and reclaim on the same node, which
obviously isn't all that nice for correct operation.

mmap stacks, makes it possible to align them

Add puffs_cc_schedule() which marks a pcc runnable and will pass
execution to it when in the main loop the next time.

Always provide caller information from the kernel based on curlwp.
(but don't deprecate the old puffs_cid interface just yet)

Part 1/n of some pretty extensive changes to how the kernel module
interacts with the userspace file server:

* since the kernel-user communication is not purely request-response
anymore (hasn't been since 2006), try to rename some "request" to
"message". more similar mangling will take place in the future.

* completely rework how messages are allocated. previously most of
them were borrowed from the stack (originally *all* of them),
but now always allocate dynamically. this makes the structure
of the code much cleaner. also makes it possible to fix a
locking order violation. it enables plenty of future enhancements.

* start generalizing the transport interface to be independent of puffs

* move transport interface to read/write instead of ioctl. the
old one had legacy design problems, and besides, ioctl's suck.
implement a very generic version for now; this will be
worked on later hopefully some day reaching "highly optimized".

* implement libpuffs support behind existing library request
interfaces. this will change eventually (I hate those interfaces)

branches: 1.7.4;
Move puffs to a two clause license where it already isn't so. And
as agc pointed out, even files with the third clause were already
effectively two clause because of a slight bug in the language...

Step 3 and currently final step of a more general purpose event loop:
support removal and addition of i/o file descriptors on the fly.

* detect closed file descriptors
* automatically free waiters of a dead file descriptor
* give the file server the possibility to specify a callback which
notifies of a dead file descriptor
* move loop function to be a property of the mainloop instead of
framebuf (doesn't change effective behaviour)
* add the possibility to configure a timespec parameter which
attempts to call the loop function periodically
* move the event loop functions from the puffs_framebuf namespace
to puffs_framev to differential between pure memory management
functions

assert that yield/continue is done only with a real continuation

add puffs_cc_getspecific(pcc), which is shorthand for:
puffs_getspecific(puffs_cc_getusermount(pcc))

#include <ucontext.h>

* get rid of the mount callback; it's no great surprise to the
server that it needs to mount the file system backend if it wants
to call mount
* provide some options for getmntopts(), assume that callers will parse
command line (or fstab) args
* reorganize the puffs_cc interface just a bit, preparing for a bigger
revamp later

checkpoint some experimental work-in-progress, namely:

Add support for having multiple outstanding operations. This is done
by exposing enough interfaces so that it is convenient to have the
main event loop in the implementation itself and by providing a
continuation framework for convinient blocking and rescheduling.

works fine, but will undergo further cleanup & development

Use puffs_access() return value in the VA_UTIMES_NULL case.

branches: 1.15.6; 1.15.8;
In getgroups(), copy only up to the number of supplementary groups
that actually exist.

branches: 1.14.12; 1.14.18;
Now that "l" is gone both as an argument to operations and from
componentname, remove all vestiges of puffs_cid.

* Change type of boolean functions to, well, boolean instead of int.
* add puffs_cid_isequal() which can be used to test if the two caller
id's refer to the same process/lwp.

branches: 1.12.4;
Instead of supplying a plain pid, supply an abstract struct puffs_cid *,
which can currently be used to query the pid and lwpid.

make puffs_cred an opaque type

Move puffs to a two clause license where it already isn't so. And
as agc pointed out, even files with the third clause were already
effectively two clause because of a slight bug in the language...

allow kernel creds in addition to superuser creds when checking for access

unify the location of *pcr in puffs_access_*

copy ngroups * sizeof(gid_t) groups instead of just ngroups

* return -1 and set errno instead of returning errno directly
* introduce puffs_cred_isregular() which checks if the cred structure
has regular uid/gid (i.e. "person" credentials and not kernel creds)

add routiens which check against ufs semantics for permission to
chown, chmod and change a/mtime

implement counterpart of vaccess()

fix a lint warning.

pass lint; fix install.

add libpuffs - the userspace programming interface for puffs

This work was initially started and completed for Google SoC 2005
and tweaked to work a bit better in the past few weeks. While
being far from complete, it is functional enough to be able and
stable to host a fairly general-purpose in-memory file system in
userspace. Even so, puffs should be considered experimental and
no binary compatibility for interfaces or crash-freedom or zero
security implications should be relied upon just yet.

The GSoC project was mentored by William Studenmund and the final
review for the code was done by Christos.

pathconf needs to return EINVAL when the variable is invalid
or cannot be associated with a file. This also needs to be true
when the node doesn't implement the pathconf function at all.

branches: 1.49.6;
cast to the proper enums for lint

libpuffs support for fallocate and fdiscard operations

Add an open2 method, like open but with an additionnal oflags used
by the filesystem to send back information about the file.
This is used to implement PUFFS_OPEN_IO_DIRECT by which the filesystem
tells the kernel that read/write should bypass the page cache.

branches: 1.46.4;
revert previous, will turn off globally.

XXX: gcc initialization (mc68k gcc-4.5.x gcc-4.8.x)

Fix regression that has been introduced when the lookup/reclaim race
condition was addressed in libpuffs by counting lookups.

The fix assumes that cookies map to struct puffs_cookie, which has not
been documented as a requirement for filesystems using libpuffs. As an
example, we got burnt by this assumption in libp2k (kern/46734), and
we fixed bit by actually mapping libp2k cookies to struct puffs_node.

It is unlikely, but there may be third party filesystems that use cookies
unmapped to struct puffs_node, and they were left broken for now.

- we introduce a puffs_init() flag PUFFS_FLAG_PNCOOKIE that let filesystems
inform libpuffs that they map cookies to struct puffs_node. Is that flag
is used, the lookup/reclaim race condition fix is enabled. We enable the
flag for libp2k.

- filesystems that use puffs_pn_new() obviouslty use struct puffs_node
and gain PUFFS_FLAG_PNCOOKIE automatically even if they did not specify
it in puffs_init(). This include all our PUFFS filesystem in-tree except
libp2k.

- for filesystems not willing to use struct puffs_node, we introduce a
reclaim2 vnop, which is reclaim with an additionnal lookup count argument.
This vnop let the filesystem implement the lookup/reclaim race fix on
its own.

Fix race condition between (create|mknod|mkdir|symlino) and reclaim, just
like we did it between lookup and reclaim.

- Improve PUFFS_KFLAG_CACHE_FS_TTL by reclaiming older inactive nodes.

The normal kernel behavior is to retain inactive nodes in the freelist
until it runs out of vnodes. This has some merit for local filesystems,
where the cost of an allocation is about the same as the cost of a
lookup. But that situation is not true for distributed filesystems.
On the other hand, keeping inactive nodes for a long time hold memory
in the file server process, and when the kernel runs out of vnodes, it
produce reclaim avalanches that increase lattency for other operations.

We do not reclaim inactive vnodes immediatly either, as they may be
looked up again shortly. Instead we introduce a grace time and we
reclaim nodes that have been inactive beyond the grace time.

- Fix lookup/reclaim race condition.

The above improvement undercovered a race condition between lookup and
reclaim. If we reclaimed a vnode associated with a userland cookie while
a lookup returning that same cookiewas inprogress, then the kernel ends
up with a vnode associated with a cookie that has been reclaimed in
userland. Next operation on the cookie will crash (or at least confuse)
the filesystem.

We fix this by introducing a lookup count in kernel and userland. On
reclaim, the kernel sends the count, which enable userland to detect
situation where it initiated a lookup that is not completed in kernel.
In such a situation, the reclaim must be ignored, as the node is about
to be looked up again.

Pass the FAF flag to handlers:
- setattr_ttl is updated to add a flag argument. Since it was not present in
a previous release, we can change its API
- write2 is introduced, this is write with an extra flag for FAF.
- fsync already has the FAF information in a flag and needs no change
- for other operations, FAF is unconditional

- When using PUFFS_KFLAG_CACHE_FS_TTL, do not use puffs_node to carry
attribute and TTL fora newly created node. Instead extend puffs_newinfo
and add puffs_newinfo_setva() and puffs_newinfo_setttl()
- Remove node_mk_common_final in libperfuse. It used to set uid/gid for
a newly created vnode but has been made redundant along time ago since
uid and gid are properly set in FUSE header.
- In libperfuse, check for corner case where opc = 0 on INACTIVE and RECLAIM (how is it possible? Check for it to avoid a crash anyway)
- In libperfuse, make sure we unlimit RLIMIT_AS and RLIMIT_DATA so that
we do notrun out of memory because the kernel is lazy at reclaiming vnodes.
- In libperfuse, cleanup style of perfuse_destroy_pn()

Add PUFFS_KFLAG_CACHE_FS_TTL flag to puffs_init(3) to use name and
attribute cache with filesystem provided TTL.
lookup, create, mknod, mkdir, symlink, getattr and setattr messages
have been extended so that attributes and their TTL can be provided
by the filesytem. lookup, create, mknod, mkdir, and symlink messages
are also extended so that the filesystem can provide name TTL.
The filesystem updates attributes and TTL using
puffs_pn_getvap(3), puffs_pn_getvattl(3), and puffs_pn_getcnttl(3)

branches: 1.38.2;
Rollback previous change as it breaks autmates tests

Remove files specified in pcn.

When remove files using name from pnode, another link on this file
can be unlinked. E.g. "touch 1; ln 1 2; rm 2" will remove file named
"1". Thus puffs_null_node_remove should remove directory entry which
name is provided by pcn (as said in puffs_ops.3). Caller should
provide appropriately initialized pcn.

From Evgeniy Ivanov <lolkaantimat@gmail.com>

branches: 1.36.2;
Add a flag to VOP_LISTEXTATTR(9) so that the vnode interface can tell the
filesystem in which format extended attribute shall be listed.

There are currently two formats:
- NUL-terminated strings, used for listxattr(2), this is the default.
- one byte length-pprefixed, non NUL-terminated strings, used for
extattr_list_file(2), which is obtanined by setting the
EXTATTR_LIST_PREFIXLEN flag to VOP_LISTEXTATTR(9)

This approach avoid the need for converting the list back and forth, except
in libperfuse, since FUSE uses NUL-terminated strings, and the kernel may
have requested EXTATTR_LIST_PREFIXLEN.

Remove PUFFS_WITH_THREADS ifdef garbage, just use libc weak symbols.

* support extended attributes
* bump major due to structure growth
* add some spare space
* remove ABI sillyness

Dispatch puffs_node_abortop().

Note: We use the storage of puffs_cache_write from puffs_ops for
this purpose. It's not issued by the kernel and hence currently
unused, and this saves us from the trouble of bumping the lib major
version.

branches: 1.32.4; 1.32.10;
Replace void * by puffs_cookie_t where appropriate. No functional change.

branches: 1.31.4;
Reserve enough memory for ops which return more than they receive.

branches: 1.30.2;
Implement lazy contexts for file systems using puffs_mainloop().
Previously each request was executed on its own callcontext and
switched to every time the request was being processed. Now requests
borrow the mainloop context and switch only if/when they yield.
This takes two context switches away from a file system request
bringing down the typical amounts 2->0 (e.g. dtfs) and 4->2 (e.g.
psshfs).

The interfaces for manually executing requests changed a bit:
puffs_dispatch_create() and puffs_dispatch_exec() must now be used.
They are not tested, as nothing in-tree wants them and I doubt
anyone else is really interested in them either.

Also do some misc code cleanup related to execution contexts. The
"work-in-progress checkpoint" committed over a year ago was starting
to look slightly weed-infested.

Comment out the puffs_executor hacks for now, they have races.
The issue should be solved in another fashion anyway.

Cache a number of execution contexts instead of re-initializing
them every time. Speeds up pure in-memory file systems such as
sysctlfs or dtfs quite a bit. For actual I/O-workhorses the result
is of course less tasty.

* nuke puffs_cc_get{specific,usermount} for good
* move prototypes for puffs_docc and puffs_dopufbuf into the
public header, as they are should be exposed

Now that "l" is gone both as an argument to operations and from
componentname, remove all vestiges of puffs_cid.

Part 3/n of extensive changes to message handling:

Get rid of the original puffs_req(3) framework and use puffs_framebuf(3)
instead for file system requests. It has the advantage of being
suitable for transporting a distributed message passing protocol
and therefore us being able to run the file system server on any
host.

Ok, puffs is not quite here yet: libpuffs needs to grow request
routing support and the message contents need to be munged into a
host independent format. Saying which format would be telling,
but it might begin with an X, end in an L and have the 13th character
in the middle. Keep an eye out for the sequels: Parts 3+m/n.

Rototill.

Ok, ok, a few more words about it: stop holding puffs_cc as a holy
value and passing it around to almost every possible place (popquiz:
which kernel variable does this remind you of?). Instead, pass
the natural choice, puffs_usermount, and fetch puffs_cc via
puffs_cc_getcc() only in routines which actually need it. This
not only simplifies code, but (thanks to the introduction of
puffs_cc_getcc()) enables constructs which weren't previously sanely
possible, say layering as a curious example.

There's still a little to do on this front, but this was the major
fs interface blast.

* allocate pcc as a slice from the stack instead of allocating it
separately
* provide puffs_cc_getcc()

This is in preparation for the removal of you-should-guess-what as
an argument to routines here and there and everywhere.

Remove "puffs_cid" from the puffs interface following l-removal
from the kernel vfs interfaces. puffs_cc_getcaller(pcc) can be
used now should the same information be desired.

Fix a few bugs and memleaks with the threading code.

lock massage, although I don't think there's a happy ending here

Sprinkle some hackish locks here and there, just enough to allow
libp2k & rump to run multithreaded.

Add possibility to set pre- and post callbacks which will be called
for all operations before or after executing the specific callback
(the api is not be final yet, though).

Add some very initial support for a threading worker model as an
alternative to the (vastly superior ;) continuation model. This
is very preliminary stuff and not compiled by default (which it
even won't do without some other patches I cannot commit yet).

The raison d'commit of the patch is a snippet which ensures proper
in-order dispatching of all operations, including those which don't
require a response. Previously many of them would be dispatched
simultaneosly, e.g. fsync and reclaim on the same node, which
obviously isn't all that nice for correct operation.

Always provide caller information from the kernel based on curlwp.
(but don't deprecate the old puffs_cid interface just yet)

Part 1/n of some pretty extensive changes to how the kernel module
interacts with the userspace file server:

* since the kernel-user communication is not purely request-response
anymore (hasn't been since 2006), try to rename some "request" to
"message". more similar mangling will take place in the future.

* completely rework how messages are allocated. previously most of
them were borrowed from the stack (originally *all* of them),
but now always allocate dynamically. this makes the structure
of the code much cleaner. also makes it possible to fix a
locking order violation. it enables plenty of future enhancements.

* start generalizing the transport interface to be independent of puffs

* move transport interface to read/write instead of ioctl. the
old one had legacy design problems, and besides, ioctl's suck.
implement a very generic version for now; this will be
worked on later hopefully some day reaching "highly optimized".

* implement libpuffs support behind existing library request
interfaces. this will change eventually (I hate those interfaces)

g/c #if 0'd code which isn't likely to get used any time soon

Pass reason string to pu_errnotify. Print it in the default
version before aborting.

Add error notifications, which are used to deliver errors from the
kernel to the file server for silly things the file server did,
e.g. attempting to create a file with size VSIZENOTSET. The file
server can handle these as it chooses, but the default action is
for it to throw its hands in the air and sing "goodbye, cruel world,
it's over, walk on by".

branches: 1.11.4; 1.11.6;
mmap fflags -> prot

Get rid of the "int *refs" parameter to inactive: the same can be
accomplished now with puffs_setbacks.

Instead of requesting various pieces of information from node-creating
methods using "please fill me"-pointers, pass "struct puffs_newinfo *"
and provide methods to fill in new node info.

Instead of supplying a plain pid, supply an abstract struct puffs_cid *,
which can currently be used to query the pid and lwpid.

make puffs_cred an opaque type

Move puffs to a two clause license where it already isn't so. And
as agc pointed out, even files with the third clause were already
effectively two clause because of a slight bug in the language...

default refcount to 1 in inactive

dispatch PUFFS_VN_POLL

Step 3 and currently final step of a more general purpose event loop:
support removal and addition of i/o file descriptors on the fly.

* detect closed file descriptors
* automatically free waiters of a dead file descriptor
* give the file server the possibility to specify a callback which
notifies of a dead file descriptor
* move loop function to be a property of the mainloop instead of
framebuf (doesn't change effective behaviour)
* add the possibility to configure a timespec parameter which
attempts to call the loop function periodically
* move the event loop functions from the puffs_framebuf namespace
to puffs_framev to differential between pure memory management
functions

copyright adjustment

move puffs request dispatching code into it's own module.
no functional change

Replace void * by puffs_cookie_t where appropriate. No functional change.

branches: 1.15.8;
free flushbuffer after use

Send a response message for flush operations from the kernel instead
of abusing the return value of write(2).

* split the putter header into a kernel version and a userland version
+ install latter to /usr/include/dev/putter
* remove last dependencies to puffs from putter, it's completely
independent now

Part 1/n of some pretty extensive changes to how the kernel module
interacts with the userspace file server:

* since the kernel-user communication is not purely request-response
anymore (hasn't been since 2006), try to rename some "request" to
"message". more similar mangling will take place in the future.

* completely rework how messages are allocated. previously most of
them were borrowed from the stack (originally *all* of them),
but now always allocate dynamically. this makes the structure
of the code much cleaner. also makes it possible to fix a
locking order violation. it enables plenty of future enhancements.

* start generalizing the transport interface to be independent of puffs

* move transport interface to read/write instead of ioctl. the
old one had legacy design problems, and besides, ioctl's suck.
implement a very generic version for now; this will be
worked on later hopefully some day reaching "highly optimized".

* implement libpuffs support behind existing library request
interfaces. this will change eventually (I hate those interfaces)

branches: 1.11.4; 1.11.6;
Don't float puffs_kargs needlessly around after mount, saves
a couple k of memory (and there was much rejoycing!)

Move puffs to a two clause license where it already isn't so. And
as agc pointed out, even files with the third clause were already
effectively two clause because of a slight bug in the language...

Instead of keeping on adding parameters to puffs_mount(), make it
only take the bare essentials, which currently means removing
"maxreqlen" from the argument list (all current callers I'm aware
of set it as 0 anyway). Introduce puffs_init(), which provides a
context for setting various parameters and puffs_domount(), which
can be used to mount the file system. Keep puffs_mount() as a
shortcut for the above two for simple file systems.

Bump development ABI version to 13. After all, it's Friday the 13th.
Watch out! Bad things can happen on Friday the 13th. --No carrier--

Hide struct puffs_usermount from the rest of the world and provide
accessors for interesting data in it. Namely, you can now get
pu->pu_privdata with puffs_getspecific(), pu->pu_pn_root with
puffs_set/getroot() and pu->pu_maxreqlen with puffs_getmaxreqlen().

support page cache flushing

use range invalidation op also for invalidating entire page cache of a node

implement puffs_inval_pagecache_node_range()

g/c unused debug variable

initial support for cacheops

rename name cache invalidation interface to be in sync with the
kernel operation names, i.e. contain "namecache" instead of just "name"

lib interface for invalidation routines

lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.

puffs(3): Replace realloc(x * y) with reallocarr

remove now unnecessary casts.

Adjust to prototype.

Only add a write filter for sockets. It is not supported for vnodes, or
for 1/2 closed fifos (which we both have now).

branches: 1.32.22;
Update old-style definitions to ANSI, remove a couple of register
definitions along the way. Fixed gcc 4.1 build (thank you vax)

Fix error reporting in puffs_framev_enqueue_waitevent and callers.

ok christos

branches: 1.30.6;
Add convenience routine puffs_unmountonsignal(), which does exactly that.

Fix double free when i/o descriptor is violently closed. Bug was
introduced in rev 1.27.

fix provided by Taylor R Campbell in lib/39353

branches: 1.28.6;
Fix a weirdosity in the framebuf gotfb interface: hand buffer
ownership to the called routine instead of keeping it. This means
that callees must now destroy the buffer once they are done with it.

Implement lazy contexts for file systems using puffs_mainloop().
Previously each request was executed on its own callcontext and
switched to every time the request was being processed. Now requests
borrow the mainloop context and switch only if/when they yield.
This takes two context switches away from a file system request
bringing down the typical amounts 2->0 (e.g. dtfs) and 4->2 (e.g.
psshfs).

The interfaces for manually executing requests changed a bit:
puffs_dispatch_create() and puffs_dispatch_exec() must now be used.
They are not tested, as nothing in-tree wants them and I doubt
anyone else is really interested in them either.

Also do some misc code cleanup related to execution contexts. The
"work-in-progress checkpoint" committed over a year ago was starting
to look slightly weed-infested.

* nuke puffs_cc_get{specific,usermount} for good
* move prototypes for puffs_docc and puffs_dopufbuf into the
public header, as they are should be exposed

Part 3/n of extensive changes to message handling:

Get rid of the original puffs_req(3) framework and use puffs_framebuf(3)
instead for file system requests. It has the advantage of being
suitable for transporting a distributed message passing protocol
and therefore us being able to run the file system server on any
host.

Ok, puffs is not quite here yet: libpuffs needs to grow request
routing support and the message contents need to be munged into a
host independent format. Saying which format would be telling,
but it might begin with an X, end in an L and have the 13th character
in the middle. Keep an eye out for the sequels: Parts 3+m/n.

realloc() len+incr, not offset+incr. Makes space reservation
actually work properly.

document puffs_framev_framebuf_ccpromote()

Give cmpframe() the ability to shortcircuit the comparison loop by
signalling back that the buffer under examination is not a response
at all.

Add gotframe callback, which is called if an incoming frame matches
no outstanding request.

branches: 1.20.2;
Pass usermount to errnotify, it might not be available there through
other structures. CID 4496

branches: 1.19.4;
Return -1 and errno from puffs_framev_enqueue_waitevent instead of
a direct error value.

simplify send loop

Add puffs_framev_enqueue_waitevent, which waits for read/write/error
events from an i/o descriptor. It yields until an event happens
and continues execution when the event loop notices the event has
happened.

g/c leftover

make notifyfunction optional and do not call removeonclose() if
none is given.

Make puffs_framebuf better suited from environments with multiple sources:

* in addition add/remove, allow enable/disable, which can be used
to control events for descriptors without having to remove all the
data associated with them
* add directsend/receive, which can be used to pass the same buffer
from the caller to read/writeframe and back again
* add flags to enqueue functions and allow urgent buffers to be
processed as the next PDU

Introduce puffs_framev_framebuf_ccpromote, which can be used to
turn a reference to puffs_framebuf in the file system from a
cb/justsend operation to a cc wait, should the file system find
itself desiring the result.

Change the signature of puffs_respcmp to return -1,0,1 for smaller,
equal, larger, respectively instead of 0/1 for non/equal. This
will allow sorting the buffers for faster matching in libpuffs.
While here, change the name from respcmp to framecmp, as that better
reflects the purpose.

NOTE! there is no obvious way to make compilation fail for file
systems which may already be using this feature (although I don't
think there are any outside our tree, as the feature is two weeks
old). Nevertheless, non-updated file systems will fail very quickly.

Allocate buf in 4k chunks instead of 64k chunks. If we are seriously
reclaiming and the network/server is slow, we might have thousands
of buffers allocated at the same time causing the process to run
out of vm space. Rate limiting the number of outstanding ops would
be a nicer choice, but that requires more complex changes.

Add error parameter to puffs_framev_cb to signal if there is a
result of if the function should just do resource cleanup.

switch the arbitrary default errno value for network i/o from
ECONNABORTED to ECONNRESET, which is at least a little bit better

Step 3 and currently final step of a more general purpose event loop:
support removal and addition of i/o file descriptors on the fly.

* detect closed file descriptors
* automatically free waiters of a dead file descriptor
* give the file server the possibility to specify a callback which
notifies of a dead file descriptor
* move loop function to be a property of the mainloop instead of
framebuf (doesn't change effective behaviour)
* add the possibility to configure a timespec parameter which
attempts to call the loop function periodically
* move the event loop functions from the puffs_framebuf namespace
to puffs_framev to differential between pure memory management
functions

use size_t instead of int to calm over-eager lint

Step 2 of general-purpose eventloop: get rid of puffs_framebuf_eventloop().
puffs_mainloop() now handles that functionality as well.

Step 1 in moving towards a general-purpose eventloop: allow the
framebuf event loop to take n i/o fd's as parameters and also allow
dynamic add/remove of fd's. (not tested except for one fd, but more
changes coming soon)

Remove the puffs_usermount argument from puffs_req_handle(). It can
be found from the other arguments and this way puffs_req_handle() is
more symmetric with the other puffs_req() routines.

implement the conveniently-already-prototyped puffs_framebuf_remaining(),
which is just shorthand for tellsize() - telloff()

Remove the user-specified argument for the framebuf loop callback.
It can be generated through other means if it is desired.

Add puffs "frame buffers", which are an abstraction of the buffering
and event handling mechanisms required in file servers with blocking
I/O backends. puffs_framebuf is built on the concept of puffs_cc
and uses those to multiplex execution where needed.

File systems are required to implement three methods:
* read frame
* write frame
* compare if frame is a response to the given one

Memory management is provided by puffs_framebuf, but the file
systems must still, of course, interpret the protocol and do e.g.
byte order conversion.

As always, puffs_framebuf is work in progress. Current users are
mount_psshfs and mount_9p.

Add missing double p and d for stopped and overriden accordingly.
Fix few more typos along the way, mainly in copy-pasted comments.

puffs_null_node_fsync: don't leak directory handle

Directory handles returned by opendir must be closed by closedir.
Also directory(3) says we mustn't close(2) descriptors returned by dirfd(3)

Restore binary compatibility by using the statvfs90 structure internally.

branches: 1.33.40;
Rollback previous change as it breaks autmates tests

Remove files specified in pcn.

When remove files using name from pnode, another link on this file
can be unlinked. E.g. "touch 1; ln 1 2; rm 2" will remove file named
"1". Thus puffs_null_node_remove should remove directory entry which
name is provided by pcn (as said in puffs_ops.3). Caller should
provide appropriately initialized pcn.

From Evgeniy Ivanov <lolkaantimat@gmail.com>

Remove pnode of overwritten file.

When puffs_null_node_rename() overwrites existing file, its pnode
must�be removed, because src pnode already represents this file.

From Evgeniy Ivanov <lolkaantimat@gmail.com>

branches: 1.30.2;
From Evgeniy Ivanov: puffs_node_readdir() should set eofflag to one, when
end-of-directory is reached.

Commit fix for misc/45029.
Approved by pooka@

include some headers we use

Cast a value to time_t (not unsigned) when comparing to timespec tv_sec

support:
a) readdir cookies and eofflag
b) file handle routines.

File handle routines are REALLY ugly and are not stable, i.e.
the file handle will be lost if the file server goes byebye.
They probably should be redone to not use getfh(), but for nullfs
I wanted to make them the same as when the file system is accessed
without the null layer. Well, turns out it doesn't pass anyway
since the kernel adds the fsid tags for fhandle_t. Anyway, it's
useful for some limited hacking purposes, so I'll commit this
since it works .. kinda.

Replace void * by puffs_cookie_t where appropriate. No functional change.

branches: 1.24.8;
Rototill.

Ok, ok, a few more words about it: stop holding puffs_cc as a holy
value and passing it around to almost every possible place (popquiz:
which kernel variable does this remind you of?). Instead, pass
the natural choice, puffs_usermount, and fetch puffs_cc via
puffs_cc_getcc() only in routines which actually need it. This
not only simplifies code, but (thanks to the introduction of
puffs_cc_getcc()) enables constructs which weren't previously sanely
possible, say layering as a curious example.

There's still a little to do on this front, but this was the major
fs interface blast.

Remove "puffs_cid" from the puffs interface following l-removal
from the kernel vfs interfaces. puffs_cc_getcaller(pcc) can be
used now should the same information be desired.

Call puffs_genfs_reclaim directly instead of through an extra jumpgate.

branches: 1.21.4;
Instead of requesting various pieces of information from node-creating
methods using "please fill me"-pointers, pass "struct puffs_newinfo *"
and provide methods to fill in new node info.

Instead of supplying a plain pid, supply an abstract struct puffs_cid *,
which can currently be used to query the pid and lwpid.

fold in common code for node creation. no functional change.

Since we locate in-memory nodes based on the inode number, stat a
file always after creation to cache the inode number given by the
backend file system. Otherwise we would not find a newly created
node from incore and create another one. In practise this was
pretty well hidden by the kernel name cache.

Introduce puffs_null_setops(), which initializes the ops vector with
puffs nullfs ops.

Introduce puffs_pn_remove() to signal that a node has been deleted
and destroyed, but not yet reclaimed. This prevents puffs_pn_nodewalk()
from returning stale entries. Make nullfs use this (some file
systems are a bit too happy with recycling inode numbers).

don't return errno from a function which returns an fd - there's
a "little" room for confusion. instead, return -1 and let the
caller grab errno.

Move puffs to a two clause license where it already isn't so. And
as agc pointed out, even files with the third clause were already
effectively two clause because of a slight bug in the language...

Hide struct puffs_usermount from the rest of the world and provide
accessors for interesting data in it. Namely, you can now get
pu->pu_privdata with puffs_getspecific(), pu->pu_pn_root with
puffs_set/getroot() and pu->pu_maxreqlen with puffs_getmaxreqlen().

* support nodetofh and fhtonode callbacks
* use new signature of node_readdir

va->va_mode doesn't contain the full argument to the mknod() system call,
so introduce puffs_addvtype2mode() and use that in null.c to generate the
proper syscall argument

return errno instead of -1 if file is not present in lookup

Create in-memory nodes only per unique inode, not unique pathname.
Solves problem with e.g. cache coherency of hardlinked files.

Readlink should place the length of the link (without terminating
nul) in the length field. Make it so.

use puffs_path_walkcmp() instead of a homegrown comparison routine

* get rid of the holy kernel-given ISDOTDOT, and instead define a
macro which does strcmp against ".." and (the untranslated)
componentname
* make PUFFS_FLAG_BUILDPATH build paths also if dotdot is the case,
and adapt the regular path objects to this
* make nullfs lookup readable because we can now get rid of dotdot
processing there

If we can't set the attributes when making the directory, use rmdir to
remove the directory, not unlink.

Checkpoint some more work in progress: for the benefit those file
system backends which operate purely based on paths, push out more
path management into the library and make path management more
abstract: enable a file system to define a bunch of path management
callbacks, which are used by the framework. Management of normal
/this/is/a/path type paths is provided by the library.

implement fsync

* do the l-variations of syscalls, since some nodes are symlinks
* truncate only regular files to set size
* do the chmod()-dance for cache flush to now write-protected files
until I can think of a nicer way to solve this

Add preliminary code for a nullfs layer, which can be used to mount
a directory hierarchy to another point, just like with the kernel
nullfs. This is not really a layering scheme yet, but it should
evolve into one. Currently it can just be used to do 1:1 mapping.

cast to the proper enums for lint

libpuffs support for fallocate and fdiscard operations

branches: 1.36.10;
Add __printflike attribution to use vprintf and friends with an argument
as format string.

branches: 1.35.6;
print to stderr instead of stdout

Export the opid -> opstr tables (for outside debugging use).

adjust for protocol changes

Don't try to print the names of messages we don't know about.

* fix typo
* fix "that's not supposed to be there" causing off-by-one in array
* make compile regardless of dev_t size

Be slightly more forgiving about unknown operation types.

Dump contents of struct vattr -- i never remember which members
the kernel passes when, so this helps tapping that info.

* make indentation a little less ridiculous
* note that interfaces provided by <puffsdump.h> are not stable and
are to be used only for debug builds
* remove a few unimplemented/used dump interfaces

dump readdir offset

* dump op-specific return info before the generic one
* dump readdir response values

branches: 1.25.2;
fix for dev_t 64 bits.

Replace void * by puffs_cookie_t where appropriate. No functional change.

branches: 1.23.6;
For operations with a target (remove, rmdir, link), print the
target cookie.

* dump r/w return resid
* fix RV dump order to be more logical

print newnode cookie value for operations which create nodes

add flushops

print mode in open

dump lookup return values

Sprinkle some hackish locks here and there, just enough to allow
libp2k & rump to run multithreaded.

dump pid/lwp of caller

Part 1/n of some pretty extensive changes to how the kernel module
interacts with the userspace file server:

* since the kernel-user communication is not purely request-response
anymore (hasn't been since 2006), try to rename some "request" to
"message". more similar mangling will take place in the future.

* completely rework how messages are allocated. previously most of
them were borrowed from the stack (originally *all* of them),
but now always allocate dynamically. this makes the structure
of the code much cleaner. also makes it possible to fix a
locking order violation. it enables plenty of future enhancements.

* start generalizing the transport interface to be independent of puffs

* move transport interface to read/write instead of ioctl. the
old one had legacy design problems, and besides, ioctl's suck.
implement a very generic version for now; this will be
worked on later hopefully some day reaching "highly optimized".

* implement libpuffs support behind existing library request
interfaces. this will change eventually (I hate those interfaces)

support error notifications

branches: 1.13.2;
Dump lookup/componentname specific info in opdump.
(sigh, I guess I guess I'll never do opdump properly at this rate)

dump further info for read/write

print time elapsed between calls

Move puffs to a two clause license where it already isn't so. And
as agc pointed out, even files with the third clause were already
effectively two clause because of a slight bug in the language...

initial support for cacheops

dump return values for operations

add interfaces for suspending the file system

adapt to kernel change:
Allow multiple requests to be transferred in each GET/PUTOP

* support FAF (Fire-And-Forget) class operations
* adapt to new lookup

dump correct name for vfs operations

use portable prints

pass lint; fix install.

add libpuffs - the userspace programming interface for puffs

This work was initially started and completed for Google SoC 2005
and tweaked to work a bit better in the past few weeks. While
being far from complete, it is functional enough to be able and
stable to host a fairly general-purpose in-memory file system in
userspace. Even so, puffs should be considered experimental and
no binary compatibility for interfaces or crash-freedom or zero
security implications should be relied upon just yet.

The GSoC project was mentored by William Studenmund and the final
review for the code was done by Christos.

Replace void * by puffs_cookie_t where appropriate. No functional change.

branches: 1.7.12;
Move puffs to a two clause license where it already isn't so. And
as agc pointed out, even files with the third clause were already
effectively two clause because of a slight bug in the language...

Introduce PUFFS_FLAG_HASHPATH, which for PUFFS_FLAG_BUILDPATH file servers
creates a hash of the node's path into po->po_hash to avoid a full pathcmp
every time.

in case rootpath is "/", make dotdot lookup in "/foo" build
a path of "/", not ""

Introduce puffs_path_walkcmp(), which is to be called from nodewalk
and compares the path of the node against the given pathobject.
Also make comparison method take a flag to indicate if it should
check if the second path is a true prefix of the first.

plus some namespace cleanup

* get rid of the holy kernel-given ISDOTDOT, and instead define a
macro which does strcmp against ".." and (the untranslated)
componentname
* make PUFFS_FLAG_BUILDPATH build paths also if dotdot is the case,
and adapt the regular path objects to this
* make nullfs lookup readable because we can now get rid of dotdot
processing there

make it possible for the root path to be "/" and the next path to
be "/foo" instead of "//foo"

Checkpoint some more work in progress: for the benefit those file
system backends which operate purely based on paths, push out more
path management into the library and make path management more
abstract: enable a file system to define a bunch of path management
callbacks, which are used by the framework. Management of normal
/this/is/a/path type paths is provided by the library.

Fix regression that has been introduced when the lookup/reclaim race
condition was addressed in libpuffs by counting lookups.

The fix assumes that cookies map to struct puffs_cookie, which has not
been documented as a requirement for filesystems using libpuffs. As an
example, we got burnt by this assumption in libp2k (kern/46734), and
we fixed bit by actually mapping libp2k cookies to struct puffs_node.

It is unlikely, but there may be third party filesystems that use cookies
unmapped to struct puffs_node, and they were left broken for now.

- we introduce a puffs_init() flag PUFFS_FLAG_PNCOOKIE that let filesystems
inform libpuffs that they map cookies to struct puffs_node. Is that flag
is used, the lookup/reclaim race condition fix is enabled. We enable the
flag for libp2k.

- filesystems that use puffs_pn_new() obviouslty use struct puffs_node
and gain PUFFS_FLAG_PNCOOKIE automatically even if they did not specify
it in puffs_init(). This include all our PUFFS filesystem in-tree except
libp2k.

- for filesystems not willing to use struct puffs_node, we introduce a
reclaim2 vnop, which is reclaim with an additionnal lookup count argument.
This vnop let the filesystem implement the lookup/reclaim race fix on
its own.

- When using PUFFS_KFLAG_CACHE_FS_TTL, do not use puffs_node to carry
attribute and TTL fora newly created node. Instead extend puffs_newinfo
and add puffs_newinfo_setva() and puffs_newinfo_setttl()
- Remove node_mk_common_final in libperfuse. It used to set uid/gid for
a newly created vnode but has been made redundant along time ago since
uid and gid are properly set in FUSE header.
- In libperfuse, check for corner case where opc = 0 on INACTIVE and RECLAIM (how is it possible? Check for it to avoid a crash anyway)
- In libperfuse, make sure we unlimit RLIMIT_AS and RLIMIT_DATA so that
we do notrun out of memory because the kernel is lazy at reclaiming vnodes.
- In libperfuse, cleanup style of perfuse_destroy_pn()

Add PUFFS_KFLAG_CACHE_FS_TTL flag to puffs_init(3) to use name and
attribute cache with filesystem provided TTL.
lookup, create, mknod, mkdir, symlink, getattr and setattr messages
have been extended so that attributes and their TTL can be provided
by the filesytem. lookup, create, mknod, mkdir, and symlink messages
are also extended so that the filesystem can provide name TTL.
The filesystem updates attributes and TTL using
puffs_pn_getvap(3), puffs_pn_getvattl(3), and puffs_pn_getcnttl(3)

branches: 1.10.2; 1.10.20;
Replace void * by puffs_cookie_t where appropriate. No functional change.

branches: 1.9.10;
Add accessors for struct puffs_node to prepare for removal from
exported headers.

Instead of requesting various pieces of information from node-creating
methods using "please fill me"-pointers, pass "struct puffs_newinfo *"
and provide methods to fill in new node info.

Introduce puffs_pn_remove() to signal that a node has been deleted
and destroyed, but not yet reclaimed. This prevents puffs_pn_nodewalk()
from returning stale entries. Make nullfs use this (some file
systems are a bit too happy with recycling inode numbers).

Move puffs to a two clause license where it already isn't so. And
as agc pointed out, even files with the third clause were already
effectively two clause because of a slight bug in the language...

Hide struct puffs_usermount from the rest of the world and provide
accessors for interesting data in it. Namely, you can now get
pu->pu_privdata with puffs_getspecific(), pu->pu_pn_root with
puffs_set/getroot() and pu->pu_maxreqlen with puffs_getmaxreqlen().

Remove check against null parameter along with the XXX comment wondering
why the check was there. CID 4408

Checkpoint some more work in progress: for the benefit those file
system backends which operate purely based on paths, push out more
path management into the library and make path management more
abstract: enable a file system to define a bunch of path management
callbacks, which are used by the framework. Management of normal
/this/is/a/path type paths is provided by the library.

Add preliminary code for a nullfs layer, which can be used to mount
a directory hierarchy to another point, just like with the kernel
nullfs. This is not really a layering scheme yet, but it should
evolve into one. Currently it can just be used to do 1:1 mapping.

checkpoint some experimental work-in-progress, namely:

Add support for having multiple outstanding operations. This is done
by exposing enough interfaces so that it is convenient to have the
main event loop in the implementation itself and by providing a
continuation framework for convinient blocking and rescheduling.

works fine, but will undergo further cleanup & development

Allow calling puffs_mount(3) before puffs_daemon(3)

puffs_daemon(3) creates a pipe before forking, and the parent process
waits for the child to either complete puffs_mount(3) or fail. If a
user calls puffs_daemon(3) after puffs_mount(3), the function
deadlocks. While this error-reporting functionality is really a nice
thing to have, deadlocking is not great. If the filesystem has already
been mounted, puffs_mount(3) should just daemonize the process and
return.

This became an issue because fuse_daemonize(3) in FUSE API had no such
requirement and some FUSE filesystems in the wild suffered deadlocks.

Use size_t in place of voff_t in puffs_setrootinfo(3)

Chuck Silvers pointed out that voff_t was also supposed to be
kernel-only. The correct type to use in userland would be off_t, but
since changing vsize_t to either voff_t or off_t is an ABI change on
32-bit platforms, we use size_t knowing that it is technically
incorrect.

Avoid using register_t or vsize_t in <puffs.h>

<puffs.h> is a user-space header, and should not use any of
kernel-only types. It's not reasonable to require user-land
filesystems to #define _KERNTYPES.

s/conjuction/conjunction/

branches: 1.63.6;
Remove workaround for ancient HTML generation code.

there is no pflags, just flags.

Bump date for previous.
filesystem -> file system

Add PUFFS_KFLAG_NOFLUSH_META to prevent sending metadata flush to FUSE

FUSE filesystems do not expect to get metadata updates for [amc]time
and size, they updates the value on their own after operations.

The PUFFS PUFFS_KFLAG_NOFLUSH_META option prevents regular metadata cache
flushes to the filesystem , and libperfuse uses it to match Linux FUSE
behavior.

While there, fix a bug in SETATTR: do not update kernel metadata cache
from SETATTR reply when the request is asynchronous, as we do not have
the reply yet.

branches: 1.59.4;
Merge riastradh-drm2 to HEAD.

Use Mt for email addresses.

branches: 1.57.2; 1.57.4;
Bump date for previous.

Fix regression that has been introduced when the lookup/reclaim race
condition was addressed in libpuffs by counting lookups.

The fix assumes that cookies map to struct puffs_cookie, which has not
been documented as a requirement for filesystems using libpuffs. As an
example, we got burnt by this assumption in libp2k (kern/46734), and
we fixed bit by actually mapping libp2k cookies to struct puffs_node.

It is unlikely, but there may be third party filesystems that use cookies
unmapped to struct puffs_node, and they were left broken for now.

- we introduce a puffs_init() flag PUFFS_FLAG_PNCOOKIE that let filesystems
inform libpuffs that they map cookies to struct puffs_node. Is that flag
is used, the lookup/reclaim race condition fix is enabled. We enable the
flag for libp2k.

- filesystems that use puffs_pn_new() obviouslty use struct puffs_node
and gain PUFFS_FLAG_PNCOOKIE automatically even if they did not specify
it in puffs_init(). This include all our PUFFS filesystem in-tree except
libp2k.

- for filesystems not willing to use struct puffs_node, we introduce a
reclaim2 vnop, which is reclaim with an additionnal lookup count argument.
This vnop let the filesystem implement the lookup/reclaim race fix on
its own.

Use more markup. New sentence, new line. Bump date for previous.

Add PUFFS_KFLAG_CACHE_DOTDOT so that vnodes hold a reference on their
parent, keeping them active, and allowing to lookup .. without sending
a request to the filesystem.

Enable the featuure for perfused, as this is how FUSE works.

Bump date for previous.
New sentence, new line; remove trailing whitespace; fix typos;
punctuation nits.

- When using PUFFS_KFLAG_CACHE_FS_TTL, do not use puffs_node to carry
attribute and TTL fora newly created node. Instead extend puffs_newinfo
and add puffs_newinfo_setva() and puffs_newinfo_setttl()
- Remove node_mk_common_final in libperfuse. It used to set uid/gid for
a newly created vnode but has been made redundant along time ago since
uid and gid are properly set in FUSE header.
- In libperfuse, check for corner case where opc = 0 on INACTIVE and RECLAIM (how is it possible? Check for it to avoid a crash anyway)
- In libperfuse, make sure we unlimit RLIMIT_AS and RLIMIT_DATA so that
we do notrun out of memory because the kernel is lazy at reclaiming vnodes.
- In libperfuse, cleanup style of perfuse_destroy_pn()

Split file system.
Comma fixes.
Remove dangling "and".
Bump date for previous.

Add PUFFS_KFLAG_CACHE_FS_TTL flag to puffs_init(3) to use name and
attribute cache with filesystem provided TTL.
lookup, create, mknod, mkdir, symlink, getattr and setattr messages
have been extended so that attributes and their TTL can be provided
by the filesytem. lookup, create, mknod, mkdir, and symlink messages
are also extended so that the filesystem can provide name TTL.
The filesystem updates attributes and TTL using
puffs_pn_getvap(3), puffs_pn_getvattl(3), and puffs_pn_getcnttl(3)

branches: 1.49.2;
Remove obsolete puffs_suspend(3) xref.

Documentation fix

Evgeniy Ivanov <lolkaantimat@gmail.com>

branches: 1.47.6;
Add convenience routine puffs_unmountonsignal(), which does exactly that.

Remove the paragraph saying interfaces will change without warning.
Note that a stable version of puffs first appeared in NetBSD 5.0.

branches: 1.45.2;
add:
puffs_setspecific(pu, private)
Can be used to set the specific data after the call to
puffs_init().

note that puffs_daemon() must be called *before* puffs_mount()

fix typographical snafu

branches: 1.42.4;
Add AsiaBSDCon 2008 paper to SEE ALSO.

Replace void * by puffs_cookie_t where appropriate. No functional change.

Remove trailing whitespace.

branches: 1.39.4; 1.39.6;
Implement lazy contexts for file systems using puffs_mainloop().
Previously each request was executed on its own callcontext and
switched to every time the request was being processed. Now requests
borrow the mainloop context and switch only if/when they yield.
This takes two context switches away from a file system request
bringing down the typical amounts 2->0 (e.g. dtfs) and 4->2 (e.g.
psshfs).

The interfaces for manually executing requests changed a bit:
puffs_dispatch_create() and puffs_dispatch_exec() must now be used.
They are not tested, as nothing in-tree wants them and I doubt
anyone else is really interested in them either.

Also do some misc code cleanup related to execution contexts. The
"work-in-progress checkpoint" committed over a year ago was starting
to look slightly weed-infested.

Introduce PUFFS_STACKSIZE_MIN to avoid overloading 0 and getting
a warning from the library when really wanting to say "just the
minimum, please".

document puffs_dopufbuf

Part 3/n of extensive changes to message handling:

Get rid of the original puffs_req(3) framework and use puffs_framebuf(3)
instead for file system requests. It has the advantage of being
suitable for transporting a distributed message passing protocol
and therefore us being able to run the file system server on any
host.

Ok, puffs is not quite here yet: libpuffs needs to grow request
routing support and the message contents need to be munged into a
host independent format. Saying which format would be telling,
but it might begin with an X, end in an L and have the 13th character
in the middle. Keep an eye out for the sequels: Parts 3+m/n.

Fix a typo that has migrated across files.

Typo fixes.

Introduce puffs_daemon(), which works like daemon(3) except that it
also synchronizes with puffs_mount() and does not return (exit) in the
parent process until the file system has been mounted. This makes
it possible to reliably run e.g. mount_foo jippi /kai ; cd /kai/ee

Open the kernel descriptor as part of mount(), not init(). Then
it doesn't matter if someone fork()s or does other tricks between
init() and mount() (and besides, now it's where it logically should be).

Pull the daemonizing code out of the library mainloop into the file
servers. Calling daemon() (i.e. fork()ing) inside a library can
cause nice surprises for e.g. threaded programs. As discussed with
Greg Oster & others.

SEE ALSO paper

Add refuse paper to SEE ALSO

xref refuse, don't bump date

branches: 1.27.4; 1.27.6;
update

document PUFFS_KFLAG_LOOKUP_FULLPNBUF

document KFLAG_NOCACHE{_PAGE,_NAME,}

reflect today's mount changes

Step 3 and currently final step of a more general purpose event loop:
support removal and addition of i/o file descriptors on the fly.

* detect closed file descriptors
* automatically free waiters of a dead file descriptor
* give the file server the possibility to specify a callback which
notifies of a dead file descriptor
* move loop function to be a property of the mainloop instead of
framebuf (doesn't change effective behaviour)
* add the possibility to configure a timespec parameter which
attempts to call the loop function periodically
* move the event loop functions from the puffs_framebuf namespace
to puffs_framev to differential between pure memory management
functions

List all current ways to write a file system. Also, add a reference
to puffs_req(3) for the manual operation case.

document PUFFS_KFLAG_WTCACHE and PUFFS_KFLAG_IAONDEMAND

Introduce PUFFS_FLAG_HASHPATH, which for PUFFS_FLAG_BUILDPATH file servers
creates a hash of the node's path into po->po_hash to avoid a full pathcmp
every time.

Move file system and node callbacks into their own manual page.
Improve markup and descriptions a bit (not completely finished yet).

document new interfaces

document accessor routines added when making struct puffs_usermount
implementation private

document credentials & access control methods

match puffs_node_access prototype with reality

add puffs paper presented at AsiaBSDCon 2007 to "SEE ALSO"

Nm the manual puffs instead of libpuffs

document puffs_fs_suspend()

xref puffs_req

document puffs_getstate() and puffs_setstacksize()

Sort SEE ALSO.

Bring the documentation slightly more into sync with the current
reality. It's still really a far cry from proper documentation,
though.

sauce catchup

update with reality

* document open, close and access
* nomenclature improvements

Some mdoc and grammar nits.

add note to BUGS about the questionable nature of puffs_node

sync with change to lookup

initial documentation for the puffs library interface

lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.

Allow calling puffs_mount(3) before puffs_daemon(3)

puffs_daemon(3) creates a pipe before forking, and the parent process
waits for the child to either complete puffs_mount(3) or fail. If a
user calls puffs_daemon(3) after puffs_mount(3), the function
deadlocks. While this error-reporting functionality is really a nice
thing to have, deadlocking is not great. If the filesystem has already
been mounted, puffs_mount(3) should just daemonize the process and
return.

This became an issue because fuse_daemonize(3) in FUSE API had no such
requirement and some FUSE filesystems in the wild suffered deadlocks.

Use size_t in place of voff_t in puffs_setrootinfo(3)

Chuck Silvers pointed out that voff_t was also supposed to be
kernel-only. The correct type to use in userland would be off_t, but
since changing vsize_t to either voff_t or off_t is an ABI change on
32-bit platforms, we use size_t knowing that it is technically
incorrect.

Avoid using register_t or vsize_t in <puffs.h>

<puffs.h> is a user-space header, and should not use any of
kernel-only types. It's not reasonable to require user-land
filesystems to #define _KERNTYPES.

puffs(3): Replace realloc(x * y) with reallocarr

use a switch.

branches: 1.123.2; 1.123.4;
Typos.

fix error messages (use __func__, quotes)
also differentiate between canonical and non-relative.

Adjust to prototype.

fix error messages

Do not warn about relative path because of trailing slash

libpuffs calls realpath() to obtain an absolute path to use for mounting.
If the obtained path is different from the one given by the caller, a
warning is issued. This included the situation where the path passed by
the caller just have trailing slashes, a situation where we just want them
to be striped without a warning.

libpuffs support for fallocate and fdiscard operations

branches: 1.117.18;
fix crashes caused by using the results of getcontext()
after the caller returns.

branches: 1.116.4;
Call advlock method if supplied

Remove PUFFS_WITH_THREADS ifdef garbage, just use libc weak symbols.

unwhine lint

stat mount target path only if we're actually goint to use it.
This avoids errors from the comfd case when the mount target does
not exist in our local namespace.

set time_t compat where necessary

When using the top secret PUFFS_COMFD model, write out size of the
puffs kernel args before the contents. This allows parties which
have no clue about the contents to proxy the protocol easily.

Reenable pathconf now that the retval argument is register_t.

Actually, disable pathconf for a while longer: the retval needs a
little finetuning (not that everyone is jumping right to using it,
but fixing will have to wait until tomorrow, and who knows what
i'll forget during the night and how long it will eventually take).

signal pathconf in the vector of supported ops

* support extended attributes
* bump major due to structure growth
* add some spare space
* remove ABI sillyness

remove three years out-of-date comment

Add convenience routine puffs_unmountonsignal(), which does exactly that.

Use typecast instead of ULL for a less whiny lint on LP64.

Rewrite puffs_exit() in terms of the PUFFSOP_UNMOUNT kernel request.
Now server exit is a lot quite cleaner than previously.

Set high bit on server->kernel requests to better differentiate
between the direction of the request.

Add an error handler which syslogs but does not abort() and make it
the default. Most servers just shovel bits from their backend to
the kernel without checking the contents and dumping core in case
the backend gives garbage is a bit harsh.

Make it possible to supply parameters to puffs_init() later (but
before puffs_mount()).

Dispatch puffs_node_abortop().

Note: We use the storage of puffs_cache_write from puffs_ops for
this purpose. It's not issued by the kernel and hence currently
unused, and this saves us from the trouble of bumping the lib major
version.

fix some sign comparison warnings

Actually we need 'long int' to avoif a stupid 'truncating long to int'
warning.

Change type of 'n' to 'int' so that it can be printed with %d.
The domain of the value is [-1..sizeof (int)] so a type of 'int' is fine.

Close write-end of daemonpipe in reader to get EOF if the writer
goes awol.

Add puffs_cancel(), which allows to back down after puffs_init()
and before puffs_mount().
(this should actually be integrated with puffs_exit(), but that
routine is currently so non-sensible that I didn't bother now)

add:
puffs_setspecific(pu, private)
Can be used to set the specific data after the call to
puffs_init().

branches: 1.92.4; 1.92.8;
Replace void * by puffs_cookie_t where appropriate. No functional change.

clear cached puffs_cc's in puffs_exit()

* make sure we come out of mainloop in the main context. otherwise in
certain cases the server would jump to hyperspace when unmounted
* sprinkle some debug prints

branches: 1.89.6;
Implement lazy contexts for file systems using puffs_mainloop().
Previously each request was executed on its own callcontext and
switched to every time the request was being processed. Now requests
borrow the mainloop context and switch only if/when they yield.
This takes two context switches away from a file system request
bringing down the typical amounts 2->0 (e.g. dtfs) and 4->2 (e.g.
psshfs).

The interfaces for manually executing requests changed a bit:
puffs_dispatch_create() and puffs_dispatch_exec() must now be used.
They are not tested, as nothing in-tree wants them and I doubt
anyone else is really interested in them either.

Also do some misc code cleanup related to execution contexts. The
"work-in-progress checkpoint" committed over a year ago was starting
to look slightly weed-infested.

Cache a number of execution contexts instead of re-initializing
them every time. Speeds up pure in-memory file systems such as
sysctlfs or dtfs quite a bit. For actual I/O-workhorses the result
is of course less tasty.

Use PROT_NONE page to protect bottom of stack from overrun

Introduce PUFFS_STACKSIZE_MIN to avoid overloading 0 and getting
a warning from the library when really wanting to say "just the
minimum, please".

liberate gas-huffin crack-puffin' assert

fix pasto

Also pass fromname in the "distributed" case.

Provide some transparent "support" for "distributed" file systems.
If the env variable PUFFS_COMFD is set, the descriptor value
contained in it is used for communication instead of opening
/dev/puffs and doing mount(2).

This feature is obviously very undocumented and should not be used
without adult supervision.

Add a public interface for setting a cookie mapper.

Send a response message for flush operations from the kernel instead
of abusing the return value of write(2).

Part 3/n of extensive changes to message handling:

Get rid of the original puffs_req(3) framework and use puffs_framebuf(3)
instead for file system requests. It has the advantage of being
suitable for transporting a distributed message passing protocol
and therefore us being able to run the file system server on any
host.

Ok, puffs is not quite here yet: libpuffs needs to grow request
routing support and the message contents need to be munged into a
host independent format. Saying which format would be telling,
but it might begin with an X, end in an L and have the 13th character
in the middle. Keep an eye out for the sequels: Parts 3+m/n.

* allocate pcc as a slice from the stack instead of allocating it
separately
* provide puffs_cc_getcc()

This is in preparation for the removal of you-should-guess-what as
an argument to routines here and there and everywhere.

print more info when receiving an abort

puffs_mount():
* initialize values better
* always go through "fail" to see if puffs_daemon() needs to be
woken up
* open device r/w instead of r/o

Introduce puffs_daemon(), which works like daemon(3) except that it
also synchronizes with puffs_mount() and does not return (exit) in the
parent process until the file system has been mounted. This makes
it possible to reliably run e.g. mount_foo jippi /kai ; cd /kai/ee

Open the kernel descriptor as part of mount(), not init(). Then
it doesn't matter if someone fork()s or does other tricks between
init() and mount() (and besides, now it's where it logically should be).

Pull the daemonizing code out of the library mainloop into the file
servers. Calling daemon() (i.e. fork()ing) inside a library can
cause nice surprises for e.g. threaded programs. As discussed with
Greg Oster & others.

Sprinkle some hackish locks here and there, just enough to allow
libp2k & rump to run multithreaded.

Add possibility to set pre- and post callbacks which will be called
for all operations before or after executing the specific callback
(the api is not be final yet, though).

Add some very initial support for a threading worker model as an
alternative to the (vastly superior ;) continuation model. This
is very preliminary stuff and not compiled by default (which it
even won't do without some other patches I cannot commit yet).

The raison d'commit of the patch is a snippet which ensures proper
in-order dispatching of all operations, including those which don't
require a response. Previously many of them would be dispatched
simultaneosly, e.g. fsync and reclaim on the same node, which
obviously isn't all that nice for correct operation.

mmap stacks, makes it possible to align them

Make pu_haskq a flag instead of its own variable.

Add puffs_cc_schedule() which marks a pcc runnable and will pass
execution to it when in the main loop the next time.

Part 1/n of some pretty extensive changes to how the kernel module
interacts with the userspace file server:

* since the kernel-user communication is not purely request-response
anymore (hasn't been since 2006), try to rename some "request" to
"message". more similar mangling will take place in the future.

* completely rework how messages are allocated. previously most of
them were borrowed from the stack (originally *all* of them),
but now always allocate dynamically. this makes the structure
of the code much cleaner. also makes it possible to fix a
locking order violation. it enables plenty of future enhancements.

* start generalizing the transport interface to be independent of puffs

* move transport interface to read/write instead of ioctl. the
old one had legacy design problems, and besides, ioctl's suck.
implement a very generic version for now; this will be
worked on later hopefully some day reaching "highly optimized".

* implement libpuffs support behind existing library request
interfaces. this will change eventually (I hate those interfaces)

Pass reason string to pu_errnotify. Print it in the default
version before aborting.

silence lint. also noticed by xtraeme

Add error notifications, which are used to deliver errors from the
kernel to the file server for silly things the file server did,
e.g. attempting to create a file with size VSIZENOTSET. The file
server can handle these as it chooses, but the default action is
for it to throw its hands in the air and sing "goodbye, cruel world,
it's over, walk on by".

branches: 1.62.2;
run realpath() on the mountpath

branches: 1.61.4;
Add puffs_framev_enqueue_waitevent, which waits for read/write/error
events from an i/o descriptor. It yields until an event happens
and continues execution when the event loop notices the event has
happened.

Don't float puffs_kargs needlessly around after mount, saves
a couple k of memory (and there was much rejoycing!)

Use ioctl(PUFFSREQSIZEOP) instead of mount(MNT_GETARGS) to fetch
the maximum request size after mount. Calling mount(MNT_GETARGS)
from the file server is currently not kosher, as it vrele()s the
root vnode, potentially causing an inactive, which the file server
cannot handle while it itself is inactive in the kernel (deadlock).

* add mntfromname parameter to puffs_init()
* set it in file servers

Add new 'data_len' parameter to mount system call.
Add an additional call with MNT_GETARGS to retrieve the modified
information instead of relying on the kernel code abusing the mount
interface during mount.

Make puffs_framebuf better suited from environments with multiple sources:

* in addition add/remove, allow enable/disable, which can be used
to control events for descriptors without having to remove all the
data associated with them
* add directsend/receive, which can be used to pass the same buffer
from the caller to read/writeframe and back again
* add flags to enqueue functions and allow urgent buffers to be
processed as the next PDU

* reorder calls to get results processed even if our I/O descriptors
go back to zero
* don't give up on the read-side of an fd if there is still unreceived data

Get rid of the "int *refs" parameter to inactive: the same can be
accomplished now with puffs_setbacks.

print a specific warning if /dev/puffs cannot be opened in init

Move puffs to a two clause license where it already isn't so. And
as agc pointed out, even files with the third clause were already
effectively two clause because of a slight bug in the language...

remember to put the results of the final operations (including unmount)
to the kernel in mainloop

* stuff results back to kernel before calling kevent - the file server
loop function might generate some results. and this is still "after"
event handling (except for the first call, but I'm not too keen on
optimizing for that)
* don't be such a baby about EINTR from kevent(). if we get it, suck
it up and continue instead of quitting

Reflect kernel changes for supplying root node information in mount
instead of puffs_start(). Get completely rid of puffs_start(), as
everything it used to do is now handled by the mount routine.
Introduce an optional pre-mount call puffs_setrootinfo() for setting
non-default root node information. As the old puffs_mount() is
now virtually useless, say byebye to it and rename the old
puffs_domount() to puffs_mount(), but add a root cookie parameter
to compensate for the late puffs_start().

switch the arbitrary default errno value for network i/o from
ECONNABORTED to ECONNRESET, which is at least a little bit better

memset freshly allocated puffs_usermount to 0

Step 3 and currently final step of a more general purpose event loop:
support removal and addition of i/o file descriptors on the fly.

* detect closed file descriptors
* automatically free waiters of a dead file descriptor
* give the file server the possibility to specify a callback which
notifies of a dead file descriptor
* move loop function to be a property of the mainloop instead of
framebuf (doesn't change effective behaviour)
* add the possibility to configure a timespec parameter which
attempts to call the loop function periodically
* move the event loop functions from the puffs_framebuf namespace
to puffs_framev to differential between pure memory management
functions

move puffs request dispatching code into it's own module.
no functional change

Step 2 of general-purpose eventloop: get rid of puffs_framebuf_eventloop().
puffs_mainloop() now handles that functionality as well.

Add some kludges to be able to execute requests on the regular program
stack instead of the continuation stack. This is for lib/36011, where
pthread gets confused since we aren't running on the regular stack.
I'm not really sure which direction to go to with this quite yet, so
make the hack hard to enable on purpose. The whole request dispatch
code needs cleaning anyway.

Remove the puffs_usermount argument from puffs_req_handle(). It can
be found from the other arguments and this way puffs_req_handle() is
more symmetric with the other puffs_req() routines.

Return 0 from puffs_mainloop() if no error occurs.
pointed out by Phil Nelson

Add puffs_setback(), which can be used to set setbacks for nodes before
the call returns to the kernel. Currently it can be called only from
open(), mmap(), remove() and rmdir(). (But that might change)

assert that the file server doesn't try to return more cookies than
it was supposed to

alloc to set flags when setting fhsize

some routines for setting mount parameters

Instead of keeping on adding parameters to puffs_mount(), make it
only take the bare essentials, which currently means removing
"maxreqlen" from the argument list (all current callers I'm aware
of set it as 0 anyway). Introduce puffs_init(), which provides a
context for setting various parameters and puffs_domount(), which
can be used to mount the file system. Keep puffs_mount() as a
shortcut for the above two for simple file systems.

Bump development ABI version to 13. After all, it's Friday the 13th.
Watch out! Bad things can happen on Friday the 13th. --No carrier--

Hide struct puffs_usermount from the rest of the world and provide
accessors for interesting data in it. Namely, you can now get
pu->pu_privdata with puffs_getspecific(), pu->pu_pn_root with
puffs_set/getroot() and pu->pu_maxreqlen with puffs_getmaxreqlen().

* support nodetofh and fhtonode callbacks
* use new signature of node_readdir

initial support for cacheops

Until someone can sort non-priviledged mounts and kauth, force
MNT_NOSUID | MNT_NODEV for mounts with geteuid() != 0. I'm tired
of typing them on the command line every time I test a file system.

dump return values for operations

Introduce puffs_path_walkcmp(), which is to be called from nodewalk
and compares the path of the node against the given pathobject.
Also make comparison method take a flag to indicate if it should
check if the second path is a true prefix of the first.

plus some namespace cleanup

* get rid of the holy kernel-given ISDOTDOT, and instead define a
macro which does strcmp against ".." and (the untranslated)
componentname
* make PUFFS_FLAG_BUILDPATH build paths also if dotdot is the case,
and adapt the regular path objects to this
* make nullfs lookup readable because we can now get rid of dotdot
processing there

add interfaces for suspending the file system

Be consistent with the rest of the framework about the argument
order of puffs_docc and puffs_dopreq

be consistent in naming request handlers

revoke revoke

Checkpoint some more work in progress: for the benefit those file
system backends which operate purely based on paths, push out more
path management into the library and make path management more
abstract: enable a file system to define a bunch of path management
callbacks, which are used by the framework. Management of normal
/this/is/a/path type paths is provided by the library.

For directory renames we need to rename all the child nodes for
the given directory if the file system wants paths (PUFFS_FLAG_BUILDPATH).
Do this by walking the nodelist and adjusting the path prefix of
each matching node.

* don't chdir in possible daemon() call for the benefit of file systems
which specify a relative path as the root
* if (buildpath), build link source full pathname for the duration of
the operation

* get rid of the mount callback; it's no great surprise to the
server that it needs to mount the file system backend if it wants
to call mount
* provide some options for getmntopts(), assume that callers will parse
command line (or fstab) args
* reorganize the puffs_cc interface just a bit, preparing for a bigger
revamp later

* inform kernel of our version
* homegrown lib ABI check (to avoid bumping major a ludicrillion times
before reaching stability)

checkpoint some experimental work-in-progress, namely:

Add support for having multiple outstanding operations. This is done
by exposing enough interfaces so that it is convenient to have the
main event loop in the implementation itself and by providing a
continuation framework for convinient blocking and rescheduling.

works fine, but will undergo further cleanup & development

CID4360: don't leak `buf' upon return
ok pooka@

outline & export code for interfacing with kernel GET/PUT requests

g/c out-of-date comment

support mmap

cleanup generated stuff from comments

rototill the operation callbacks: unify the ops under struct puffs_ops
and namespace them

adapt to kernel change:
Allow multiple requests to be transferred in each GET/PUTOP

PUFFSFLAG -> PUFFS_FLAG to be consistent with the kernel

build & send implemented operations list to kernel

summon daemon(3) in mainloop unless the nodaemon flag is given

* call statvfs() at mount() time, pass that info along
* start is gone

shuffle flags a bit

* catch up with kernel changes
* better error handling when mounting

* support FAF (Fire-And-Forget) class operations
* adapt to new lookup

pass rdev for specfs support

handle PUFFS_VN_INACTIVE

pass lint; fix install.

add libpuffs - the userspace programming interface for puffs

This work was initially started and completed for Google SoC 2005
and tweaked to work a bit better in the past few weeks. While
being far from complete, it is functional enough to be able and
stable to host a fairly general-purpose in-memory file system in
userspace. Even so, puffs should be considered experimental and
no binary compatibility for interfaces or crash-freedom or zero
security implications should be relied upon just yet.

The GSoC project was mentored by William Studenmund and the final
review for the code was done by Christos.

branches: 1.1.4;
libpuffs: Add expected symbols list.

NOTE: This exports a symbol pu_lock which is only mentioned in
libpuffs/puffs_priv.h which looks very internal. Intentional?

PR lib/58838: shared libraries in base should all have expsym lists

Use size_t in place of voff_t in puffs_setrootinfo(3)

Chuck Silvers pointed out that voff_t was also supposed to be
kernel-only. The correct type to use in userland would be off_t, but
since changing vsize_t to either voff_t or off_t is an ABI change on
32-bit platforms, we use size_t knowing that it is technically
incorrect.

Avoid using register_t or vsize_t in <puffs.h>

<puffs.h> is a user-space header, and should not use any of
kernel-only types. It's not reasonable to require user-land
filesystems to #define _KERNTYPES.

Restore binary compatibility by using the statvfs90 structure internally.

branches: 1.127.16;
consistency fixes.

libpuffs support for fallocate and fdiscard operations

Add an open2 method, like open but with an additionnal oflags used
by the filesystem to send back information about the file.
This is used to implement PUFFS_OPEN_IO_DIRECT by which the filesystem
tells the kernel that read/write should bypass the page cache.

branches: 1.124.10;
Fix regression that has been introduced when the lookup/reclaim race
condition was addressed in libpuffs by counting lookups.

The fix assumes that cookies map to struct puffs_cookie, which has not
been documented as a requirement for filesystems using libpuffs. As an
example, we got burnt by this assumption in libp2k (kern/46734), and
we fixed bit by actually mapping libp2k cookies to struct puffs_node.

It is unlikely, but there may be third party filesystems that use cookies
unmapped to struct puffs_node, and they were left broken for now.

- we introduce a puffs_init() flag PUFFS_FLAG_PNCOOKIE that let filesystems
inform libpuffs that they map cookies to struct puffs_node. Is that flag
is used, the lookup/reclaim race condition fix is enabled. We enable the
flag for libp2k.

- filesystems that use puffs_pn_new() obviouslty use struct puffs_node
and gain PUFFS_FLAG_PNCOOKIE automatically even if they did not specify
it in puffs_init(). This include all our PUFFS filesystem in-tree except
libp2k.

- for filesystems not willing to use struct puffs_node, we introduce a
reclaim2 vnop, which is reclaim with an additionnal lookup count argument.
This vnop let the filesystem implement the lookup/reclaim race fix on
its own.

- Improve PUFFS_KFLAG_CACHE_FS_TTL by reclaiming older inactive nodes.

The normal kernel behavior is to retain inactive nodes in the freelist
until it runs out of vnodes. This has some merit for local filesystems,
where the cost of an allocation is about the same as the cost of a
lookup. But that situation is not true for distributed filesystems.
On the other hand, keeping inactive nodes for a long time hold memory
in the file server process, and when the kernel runs out of vnodes, it
produce reclaim avalanches that increase lattency for other operations.

We do not reclaim inactive vnodes immediatly either, as they may be
looked up again shortly. Instead we introduce a grace time and we
reclaim nodes that have been inactive beyond the grace time.

- Fix lookup/reclaim race condition.

The above improvement undercovered a race condition between lookup and
reclaim. If we reclaimed a vnode associated with a userland cookie while
a lookup returning that same cookiewas inprogress, then the kernel ends
up with a vnode associated with a cookie that has been reclaimed in
userland. Next operation on the cookie will crash (or at least confuse)
the filesystem.

We fix this by introducing a lookup count in kernel and userland. On
reclaim, the kernel sends the count, which enable userland to detect
situation where it initiated a lookup that is not completed in kernel.
In such a situation, the reclaim must be ignored, as the node is about
to be looked up again.

Pass the FAF flag to handlers:
- setattr_ttl is updated to add a flag argument. Since it was not present in
a previous release, we can change its API
- write2 is introduced, this is write with an extra flag for FAF.
- fsync already has the FAF information in a flag and needs no change
- for other operations, FAF is unconditional

- When using PUFFS_KFLAG_CACHE_FS_TTL, do not use puffs_node to carry
attribute and TTL fora newly created node. Instead extend puffs_newinfo
and add puffs_newinfo_setva() and puffs_newinfo_setttl()
- Remove node_mk_common_final in libperfuse. It used to set uid/gid for
a newly created vnode but has been made redundant along time ago since
uid and gid are properly set in FUSE header.
- In libperfuse, check for corner case where opc = 0 on INACTIVE and RECLAIM (how is it possible? Check for it to avoid a crash anyway)
- In libperfuse, make sure we unlimit RLIMIT_AS and RLIMIT_DATA so that
we do notrun out of memory because the kernel is lazy at reclaiming vnodes.
- In libperfuse, cleanup style of perfuse_destroy_pn()

Add PUFFS_KFLAG_CACHE_FS_TTL flag to puffs_init(3) to use name and
attribute cache with filesystem provided TTL.
lookup, create, mknod, mkdir, symlink, getattr and setattr messages
have been extended so that attributes and their TTL can be provided
by the filesytem. lookup, create, mknod, mkdir, and symlink messages
are also extended so that the filesystem can provide name TTL.
The filesystem updates attributes and TTL using
puffs_pn_getvap(3), puffs_pn_getvattl(3), and puffs_pn_getcnttl(3)

branches: 1.119.2; 1.119.4;
Use __dead

Add a flag to VOP_LISTEXTATTR(9) so that the vnode interface can tell the
filesystem in which format extended attribute shall be listed.

There are currently two formats:
- NUL-terminated strings, used for listxattr(2), this is the default.
- one byte length-pprefixed, non NUL-terminated strings, used for
extattr_list_file(2), which is obtanined by setting the
EXTATTR_LIST_PREFIXLEN flag to VOP_LISTEXTATTR(9)

This approach avoid the need for converting the list back and forth, except
in libperfuse, since FUSE uses NUL-terminated strings, and the kernel may
have requested EXTATTR_LIST_PREFIXLEN.

Make retval argument for pathconf a register_t to match VOP_PATHCONF.
This makes the size the same on 64bit archs. Don't bother bumping
any version, since you'd have explicitly had to jump through some
hoops to use pathconf before.

add option string for no attribute cache
(foreseeing the odd event I might actually implement one some day)

* support extended attributes
* bump major due to structure growth
* add some spare space
* remove ABI sillyness

Add convenience routine puffs_unmountonsignal(), which does exactly that.

Add an error handler which syslogs but does not abort() and make it
the default. Most servers just shovel bits from their backend to
the kernel without checking the contents and dumping core in case
the backend gives garbage is a bit harsh.

Make it possible to supply parameters to puffs_init() later (but
before puffs_mount()).

Dispatch puffs_node_abortop().

Note: We use the storage of puffs_cache_write from puffs_ops for
this purpose. It's not issued by the kernel and hence currently
unused, and this saves us from the trouble of bumping the lib major
version.

Add puffs_cancel(), which allows to back down after puffs_init()
and before puffs_mount().
(this should actually be integrated with puffs_exit(), but that
routine is currently so non-sensible that I didn't bother now)

add:
puffs_setspecific(pu, private)
Can be used to set the specific data after the call to
puffs_init().

branches: 1.108.4; 1.108.8;
Replace void * by puffs_cookie_t where appropriate. No functional change.

branches: 1.107.6;
Fix a weirdosity in the framebuf gotfb interface: hand buffer
ownership to the called routine instead of keeping it. This means
that callees must now destroy the buffer once they are done with it.

Bump version for lazy contexts (probably not necessary, but it's
been a long time since I bumbed the version ;)

Implement lazy contexts for file systems using puffs_mainloop().
Previously each request was executed on its own callcontext and
switched to every time the request was being processed. Now requests
borrow the mainloop context and switch only if/when they yield.
This takes two context switches away from a file system request
bringing down the typical amounts 2->0 (e.g. dtfs) and 4->2 (e.g.
psshfs).

The interfaces for manually executing requests changed a bit:
puffs_dispatch_create() and puffs_dispatch_exec() must now be used.
They are not tested, as nothing in-tree wants them and I doubt
anyone else is really interested in them either.

Also do some misc code cleanup related to execution contexts. The
"work-in-progress checkpoint" committed over a year ago was starting
to look slightly weed-infested.

Introduce PUFFS_STACKSIZE_MIN to avoid overloading 0 and getting
a warning from the library when really wanting to say "just the
minimum, please".

Add a public interface for setting a cookie mapper.

Add generic getattr routine.

* nuke puffs_cc_get{specific,usermount} for good
* move prototypes for puffs_docc and puffs_dopufbuf into the
public header, as they are should be exposed

nuke #define whose purpose has ceased to be

Now that "l" is gone both as an argument to operations and from
componentname, remove all vestiges of puffs_cid.

Part 3/n of extensive changes to message handling:

Get rid of the original puffs_req(3) framework and use puffs_framebuf(3)
instead for file system requests. It has the advantage of being
suitable for transporting a distributed message passing protocol
and therefore us being able to run the file system server on any
host.

Ok, puffs is not quite here yet: libpuffs needs to grow request
routing support and the message contents need to be munged into a
host independent format. Saying which format would be telling,
but it might begin with an X, end in an L and have the 13th character
in the middle. Keep an eye out for the sequels: Parts 3+m/n.

Rototill.

Ok, ok, a few more words about it: stop holding puffs_cc as a holy
value and passing it around to almost every possible place (popquiz:
which kernel variable does this remind you of?). Instead, pass
the natural choice, puffs_usermount, and fetch puffs_cc via
puffs_cc_getcc() only in routines which actually need it. This
not only simplifies code, but (thanks to the introduction of
puffs_cc_getcc()) enables constructs which weren't previously sanely
possible, say layering as a curious example.

There's still a little to do on this front, but this was the major
fs interface blast.

* allocate pcc as a slice from the stack instead of allocating it
separately
* provide puffs_cc_getcc()

This is in preparation for the removal of you-should-guess-what as
an argument to routines here and there and everywhere.

Remove "puffs_cid" from the puffs interface following l-removal
from the kernel vfs interfaces. puffs_cc_getcaller(pcc) can be
used now should the same information be desired.

Introduce puffs_daemon(), which works like daemon(3) except that it
also synchronizes with puffs_mount() and does not return (exit) in the
parent process until the file system has been mounted. This makes
it possible to reliably run e.g. mount_foo jippi /kai ; cd /kai/ee

remember to bump version for puffs_mainloop() change

Pull the daemonizing code out of the library mainloop into the file
servers. Calling daemon() (i.e. fork()ing) inside a library can
cause nice surprises for e.g. threaded programs. As discussed with
Greg Oster & others.

Add possibility to set pre- and post callbacks which will be called
for all operations before or after executing the specific callback
(the api is not be final yet, though).

mmap stacks, makes it possible to align them

Add puffs_cc_schedule() which marks a pcc runnable and will pass
execution to it when in the main loop the next time.

Always provide caller information from the kernel based on curlwp.
(but don't deprecate the old puffs_cid interface just yet)

* Change type of boolean functions to, well, boolean instead of int.
* add puffs_cid_isequal() which can be used to test if the two caller
id's refer to the same process/lwp.

Part 1/n of some pretty extensive changes to how the kernel module
interacts with the userspace file server:

* since the kernel-user communication is not purely request-response
anymore (hasn't been since 2006), try to rename some "request" to
"message". more similar mangling will take place in the future.

* completely rework how messages are allocated. previously most of
them were borrowed from the stack (originally *all* of them),
but now always allocate dynamically. this makes the structure
of the code much cleaner. also makes it possible to fix a
locking order violation. it enables plenty of future enhancements.

* start generalizing the transport interface to be independent of puffs

* move transport interface to read/write instead of ioctl. the
old one had legacy design problems, and besides, ioctl's suck.
implement a very generic version for now; this will be
worked on later hopefully some day reaching "highly optimized".

* implement libpuffs support behind existing library request
interfaces. this will change eventually (I hate those interfaces)

Pass reason string to pu_errnotify. Print it in the default
version before aborting.

Add error notifications, which are used to deliver errors from the
kernel to the file server for silly things the file server did,
e.g. attempting to create a file with size VSIZENOTSET. The file
server can handle these as it chooses, but the default action is
for it to throw its hands in the air and sing "goodbye, cruel world,
it's over, walk on by".

Give cmpframe() the ability to shortcircuit the comparison loop by
signalling back that the buffer under examination is not a response
at all.

Add gotframe callback, which is called if an incoming frame matches
no outstanding request.

branches: 1.81.2;
Mimic namei structure changes for puffs. bump both kernel & lib version.

puffs_cc_create/destroy aren't supposed to be exported to the user

Add accessors for struct puffs_node to prepare for removal from
exported headers.

Nuke PUFFSLOOKUP_FOO and move to NAMEI_FOO

branches: 1.77.4;
mmap fflags -> prot

Add puffs_framev_enqueue_waitevent, which waits for read/write/error
events from an i/o descriptor. It yields until an event happens
and continues execution when the event loop notices the event has
happened.

* add mntfromname parameter to puffs_init()
* set it in file servers

add IO_NDELAY

g/c leftover

make notifyfunction optional and do not call removeonclose() if
none is given.

Make puffs_framebuf better suited from environments with multiple sources:

* in addition add/remove, allow enable/disable, which can be used
to control events for descriptors without having to remove all the
data associated with them
* add directsend/receive, which can be used to pass the same buffer
from the caller to read/writeframe and back again
* add flags to enqueue functions and allow urgent buffers to be
processed as the next PDU

Get rid of the "int *refs" parameter to inactive: the same can be
accomplished now with puffs_setbacks.

Instead of requesting various pieces of information from node-creating
methods using "please fill me"-pointers, pass "struct puffs_newinfo *"
and provide methods to fill in new node info.

Instead of supplying a plain pid, supply an abstract struct puffs_cid *,
which can currently be used to query the pid and lwpid.

make puffs_cred an opaque type

Actually, keep PUFFS_KFLAG_NOCACHE and -o cache around as shorthand
to neither page- nor namecache.

PUFFS_KFLAG_NOCACHE became two, so introduce the command line options
-o nonamecache and -o nopagecache and adapt file systems where necessary.

Introduce puffs_null_setops(), which initializes the ops vector with
puffs nullfs ops.

Introduce puffs_pn_remove() to signal that a node has been deleted
and destroyed, but not yet reclaimed. This prevents puffs_pn_nodewalk()
from returning stale entries. Make nullfs use this (some file
systems are a bit too happy with recycling inode numbers).

Move puffs to a two clause license where it already isn't so. And
as agc pointed out, even files with the third clause were already
effectively two clause because of a slight bug in the language...

Introduce puffs_framev_framebuf_ccpromote, which can be used to
turn a reference to puffs_framebuf in the file system from a
cb/justsend operation to a cc wait, should the file system find
itself desiring the result.

Change the signature of puffs_respcmp to return -1,0,1 for smaller,
equal, larger, respectively instead of 0/1 for non/equal. This
will allow sorting the buffers for faster matching in libpuffs.
While here, change the name from respcmp to framecmp, as that better
reflects the purpose.

NOTE! there is no obvious way to make compilation fail for file
systems which may already be using this feature (although I don't
think there are any outside our tree, as the feature is two weeks
old). Nevertheless, non-updated file systems will fail very quickly.

dispatch PUFFS_VN_POLL

Reflect kernel changes for supplying root node information in mount
instead of puffs_start(). Get completely rid of puffs_start(), as
everything it used to do is now handled by the mount routine.
Introduce an optional pre-mount call puffs_setrootinfo() for setting
non-default root node information. As the old puffs_mount() is
now virtually useless, say byebye to it and rename the old
puffs_domount() to puffs_mount(), but add a root cookie parameter
to compensate for the late puffs_start().

Add error parameter to puffs_framev_cb to signal if there is a
result of if the function should just do resource cleanup.

Step 3 and currently final step of a more general purpose event loop:
support removal and addition of i/o file descriptors on the fly.

* detect closed file descriptors
* automatically free waiters of a dead file descriptor
* give the file server the possibility to specify a callback which
notifies of a dead file descriptor
* move loop function to be a property of the mainloop instead of
framebuf (doesn't change effective behaviour)
* add the possibility to configure a timespec parameter which
attempts to call the loop function periodically
* move the event loop functions from the puffs_framebuf namespace
to puffs_framev to differential between pure memory management
functions

Step 2 of general-purpose eventloop: get rid of puffs_framebuf_eventloop().
puffs_mainloop() now handles that functionality as well.

Step 1 in moving towards a general-purpose eventloop: allow the
framebuf event loop to take n i/o fd's as parameters and also allow
dynamic add/remove of fd's. (not tested except for one fd, but more
changes coming soon)

Remove the puffs_usermount argument from puffs_req_handle(). It can
be found from the other arguments and this way puffs_req_handle() is
more symmetric with the other puffs_req() routines.

Add puffs_setback(), which can be used to set setbacks for nodes before
the call returns to the kernel. Currently it can be called only from
open(), mmap(), remove() and rmdir(). (But that might change)

Remove the user-specified argument for the framebuf loop callback.
It can be generated through other means if it is desired.

Add puffs "frame buffers", which are an abstraction of the buffering
and event handling mechanisms required in file servers with blocking
I/O backends. puffs_framebuf is built on the concept of puffs_cc
and uses those to multiplex execution where needed.

File systems are required to implement three methods:
* read frame
* write frame
* compare if frame is a response to the given one

Memory management is provided by puffs_framebuf, but the file
systems must still, of course, interpret the protocol and do e.g.
byte order conversion.

As always, puffs_framebuf is work in progress. Current users are
mount_psshfs and mount_9p.

Introduce PUFFS_FLAG_HASHPATH, which for PUFFS_FLAG_BUILDPATH file servers
creates a hash of the node's path into po->po_hash to avoid a full pathcmp
every time.

use more accurate flag masking

add puffs_cc_getspecific(pcc), which is shorthand for:
puffs_getspecific(puffs_cc_getusermount(pcc))

Introduce PUFFS_STORE_DCOOKIE, which stores a directory cookie for a
given offset iff the directory read in question wants cookies.

alloc to set flags when setting fhsize

some routines for setting mount parameters

Instead of keeping on adding parameters to puffs_mount(), make it
only take the bare essentials, which currently means removing
"maxreqlen" from the argument list (all current callers I'm aware
of set it as 0 anyway). Introduce puffs_init(), which provides a
context for setting various parameters and puffs_domount(), which
can be used to mount the file system. Keep puffs_mount() as a
shortcut for the above two for simple file systems.

Bump development ABI version to 13. After all, it's Friday the 13th.
Watch out! Bad things can happen on Friday the 13th. --No carrier--

Hide struct puffs_usermount from the rest of the world and provide
accessors for interesting data in it. Namely, you can now get
pu->pu_privdata with puffs_getspecific(), pu->pu_pn_root with
puffs_set/getroot() and pu->pu_maxreqlen with puffs_getmaxreqlen().

* support nodetofh and fhtonode callbacks
* use new signature of node_readdir

support page cache flushing

implement puffs_inval_pagecache_node_range()

unify the location of *pcr in puffs_access_*

* return -1 and set errno instead of returning errno directly
* introduce puffs_cred_isregular() which checks if the cred structure
has regular uid/gid (i.e. "person" credentials and not kernel creds)

remove variable names from header

add routiens which check against ufs semantics for permission to
chown, chmod and change a/mtime

implement counterpart of vaccess()

initial support for cacheops

va->va_mode doesn't contain the full argument to the mknod() system call,
so introduce puffs_addvtype2mode() and use that in null.c to generate the
proper syscall argument