devel/lloadd/design.md

TODO:
- [ ] keep a global op in-flight counter? (might need locking)
- [-] scheduling (who does what, more than one select thread? How does the proxy
      work get distributed between threads?)
- [ ] managing timeouts?
- [X] outline locking policy: seems like there might be a lock inversion in the
      design looming: when working with op, might need a lock on both client and
      upstream but depending on where we started, we might want to start with
      locking one, then other
- [ ] how to deal with the balancer running out of fds? Especially when we hit
      the limit, then lose an upstream connection and accept() a client, we
      wouldn't be able to initiate a new one. A bit of a DoS... But probably not
      a concern for Ericsson
- [ ] non-Linux? No idea how anything other than poll works (moot if building a
      libevent/libuv-based load balancer since they take care of that, except
      edge-triggered I/O?)
- [-] rootDSE? Controls and exops might have different semantics and need
      binding to the same upstream connection.
- [ ] Just piggybacking on OpenLDAP as a module? Would still need some updates
      in the core and the module/subsystem would be a very invasive one. On the
      other hand, allows to expose live configuration and monitoring over LDAP
      over the current slapd listeners without re-inventing the wheel.


Expecting to handle only LDAPv3

terms:
  server - configured target
  upstream - a single connection to a server
  client - an incoming connection

To maintain fairness `G( requested => ( F( progressed | failed ) ) )`, use
queues and put timeouts in

Runtime organisation
------
- main thread with its own event base handling signals
- one thread (later possibly more) listening on the rendezvous sockets, handing
  the new sockets to worker threads
- n worker threads dealing with client and server I/O (dispatching actual work
  to the thread pool most likely)
- a thread pool to handle actual work

Operational behaviour
------

- client read -> upstream write:
  - client read:
    - if TLS_SETUP, keep processing, set state back when finished and note that
      we're under TLS
    - ber_get_next(), if we don't have a tag, finished (unless we have true
      edge-triggered I/O, also put the fd back into the ones we're waiting for)
    - peek at op tag:
      - unbind:
        - with a single lock, mark all pending ops in upstreams abandoned, clear
          client link (would it be fast enough if we remove them from upstream
          map instead?)
        - locked per op:
          - remove op from upstream map
          - check upstream is not write-suspended, if it is ...
          - try to write the abandon op to upstream, suspend upstream if not
            fully sent
          - remove op from client map (how if we're in avl_apply?, another pass?)
        - would be nice if we could wipe the complete client map then, otherwise
          we need to queue it to have it freed when all abandons get passed onto
          the upstream (just dropping them might put extra strain on upstreams,
          will probably have a queue on each client/upstream anyway, not just a
          single Ber)
      - bind:
        - check mechanism is not EXTERNAL (or implement it)
        - abandon existing ops (see unbind)
        - set state to BINDING, put DN into authzid
        - pick upstream, create PDU and sent
      - abandon:
        - find op, mark for abandon, send to appropriate upstream
      - Exop:
        - check not BINDING (unless it's a cancel?)
        - check OID:
          - STARTTLS:
            - check we don't have TLS yet
            - abandon all
            - set state to TLS_SETUP
            - send the hello
          - VC(?):
            - similar to bind except for the abandons/state change
      - other:
        - check not BINDING
        - pick an upstream
        - create a PDU, send (marking upstream suspended if not written in full)
      - check if should read again (keep a counter of number of times to read
        off a connection in a single pass so that we maintain fairness)
      - if read enough requests and can still read, re-queue ourselves (if we
        don't have true edge-triggered I/O, we can just register the fd again)
  - upstream write (only when suspended):
    - flush the current BER
    - there shouldn't be anything else?
- upstream read -> client write:
  - upstream read:
    - ber_get_next(), if we don't have a tag, finished (unless we have true
      edge-triggered I/O, also put the fd back into the ones we're waiting for)
    - when we get it, peek at msgid, resolve client connection, lock, check:
      - if unsolicited, handle as close (and mark connection closing)
      - if op is abandoned or does not exist, drop PDU and op, update counters
      - if client backlogged, suspend upstream, register callback to unsuspend
        (on progress when writing to client or abandon from client (connection
        death, abandon proper, ...))
    - reconstruct final PDU, write BER to client, if did not write fully,
      suspend client
    - if a final response, decrement operation counts on upstream and client
    - check if should read again (keep a counter of number of responses to read
      off a connection in a single pass so that we don't starve any?)
  - client write ready (only checked for when suspended):
    - write the rest of pending BER if any
    - on successful write, pick all pending ops that need failure response, push
      to client (are there any controls that need to be present in response even
      in the case of failure?, what to do with them?)
    - on successfully flushing them, walk through suspended upstreams, picking
      the pending PDU (unsuspending the upstream) and writing, if PDU flushed
      successfully, pick next upstream
    - if we successfully flushed all suspended upstreams, unsuspend client
      (and disable the write callback)
- upstream close/error:
  - look up pending ops, try to write to clients, mark clients suspended that
    have ops that need responses (another queue associated with client to speed
    up?)
  - schedule a new connection open
- client close/error:
  - same as unbind
- client inactive (no pending ops and nothing happened in x seconds)
  - might just send notice of disconnection and close
- op timeout handling:
  - mark for abandon
  - send abandon
  - send timeLimitExceeded/adminLimitExceeded to client

Picking an upstream:
- while there is a level available:
  - pick a random ordering of upstreams based on weights
  - while there is an upstream in the level:
    - check number of ops in-flight (this is where we lock the upstream map)
    - find the least busy connection (and check if a new connection should be
      opened)
    - try to lock for socket write, if available (no BER queued) we have our
      upstream

PDU processing:
- request (have an upstream selected):
  - get new msgid from upstream
  - create an Op structure (actually, with the need for freelist lock, we can
    make it a cache for freed operation structures, avoiding some malloc
    traffic, to reset, we need slap_sl_mem_create( ,,, 1 ))
  - check proxyauthz is not present? or just let upstream reject it if there are
    two?
  - add own controls at the end:
    - construct proxyauthz from authzid
    - construct session tracking from remote IP, own name, authzid
  - send over
  - insert Op into client and upstream maps
- response/intermediate/entry:
  - look up Op in upstream's map
  - write old msgid, rest of the response can go unchanged
  - if a response, remove Op from all maps (client and upstream)

Managing upstreams:
- async connect up to min_connections (is there a point in having a connection
  count range if we can't use it when needed since all of the below is async?)
- when connected, set up TLS (if requested)
- when done, send a bind
- go for the bind interaction
- when done, add it to the upstream's connection list
- (if a connection is suspended or connections are over 75 % op limit, schedule
  creating a new connection setup unless connection limit has been hit)

Managing timeouts:
- two options:
  - maintain a separate locked priority queue to give a perfect ordering to when
    each operation is to time out, would need to maintain yet another place
    where operations can be found.
    - the locking protocol for disposing of the operation would need to be
      adjusted and might become even more complicated, might do the alternative
      initially and then attempt this if it helps performance
  - just do a sweep over all clients (that mutex is less contended) every so
    often. With many in-flight operations might be a lot of wasted work.
    - we still need to sweep over all clients to check if they should be killed
      anyway

Dispatcher thread (2^n of them, fd x is handled by thread no x % (2^n)):
- poll on all registered fds
- remove each fd that's ready from the registered list and schedule the work
- work threads can put their fd back in if they deem necessary (=not suspended)
- this works as a poor man's edge-triggered polling, with enough workers, should
  we do proper edge triggered I/O? What about non-Linux?

Listener thread:
- slapd has just one, which then reassigns the sockets to separate I/O
  threads

Threading:
- if using slap_sl_malloc, how much perf do we gain? To allocate a context per
  op, we should have a dedicated parent context so that when we free it, we can
  use that exclusively. The parent context's parent would be the main thread's
  context. This implies a lot of slap_sl_mem_setctx/slap_sl_mem_create( ,,, 0 )
  and making sure an op does not allocate/free things from two threads at the
  same time (might need an Op mutex after all? Not such a huge cost if we
  routinely reuse Op structures)

Locking policy:
- read mutexes are unnecessary, we only have one thread receiving data from the
  connection - the one started from the dispatcher
- two reference counters of operation structures (an op is accessible from
  client and upstream map, each counter is consistent when thread has a lock on
  corresponding map), when decreasing the counter to zero, start freeing
  procedure
- place to mark disposal finished for each side, consistency enforced by holding
  the freelist lock when reading/manipulating
- when op is created, we already have a write lock on upstream socket and map,
  start writing, insert to upstream map with upstream refcount 1, unlock, lock
  client, insert (client refcount 0), unlock, lock upstream, decrement refcount
  (triggers a test if we need to drop it now), unlock upstream, done
- when upstream processes a PDU, locks its map, increments counter, (potentially
  removes if it's a response), unlocks, locks client's map, write mutex (this
  order?) and full client mutex (if a bind response)
- when client side wants to work with a PDU (abandon, (un)bind), locks its map,
  increase refcount, unlocks, locks upstream map, write mutex, sends or queues
  abandon, unlocks write mutex, initiates freeing procedure from upstream side
  (or if having to remember we've already increased client-side refcount, mark
  for deletion, lose upstream lock, lock client, decref, either triggering
  deletion from client or mark for it)
- if we have operation lock, we can simplify a bit (no need for three-stage
  locking above)

Shutdown:
- stop accept() thread(s) - potentially add a channel to hand these listening
  sockets over for zero-downtime restart
- if very gentle, mark connections as closing, start timeout and:
  - when a new non-abandon PDU comes in from client - return LDAP_UNAVAILABLE
  - when receiving a PDU from upstream, send over to client, if no ops pending,
    send unsolicited response and close (RFC4511 suggests unsolicited response
    is the last PDU coming from the upstream and libldap agrees, so we can't
    send it for a socket we want to shut down more gracefully)
- gentle (or very gentle timed out):
  - set timeout
  - mark all ops as abandoned
  - send unbind to all upstreams
  - send unsolicited to all clients
- imminent (or gentle timed out):
  - async close all connections?
  - exit()

RootDSE:
- default option is not to care and if a control/exop has special restrictions,
  it is the admin's job to flag it as such in the load-balancer's config
- another is not to care about the search request but check each search entry
  being passed back, check DN and if it's a rootDSE, filter the list of
  controls/exops/sasl mechs (external!) that are supported
- last one is to check all search requests for the DN/scope and synthesise the
  response locally - probably not (would need to configure the complete list of
  controls, exops, sasl mechs, naming contexts in the balancer)

Potential red flags:
- we suspend upstreams, if we ever suspend clients we need to be sure we can't
  create dependency cycles
  - is this an issue when only suspending the read side of each? Because even if
    we stop reading from everything, we should eventually flush data to those we
    can still talk to, as upstreams are flushed, we can start sending new
    requests from live clients (those that are suspended are due to their own
    inability to accept data)
  - we might need to suspend a client if there is a reason to choose a
    particular upstream (multi-request operation - bind, VC, PR, TXN, ...)
    - a SASL bind, but that means there are no outstanding ops to receive
      it holds that !suspended(client) \or !suspended(upstream), so they
      cannot participate in a cycle
    - VC - multiple binds at the same time - !!! more analysis needed
    - PR - should only be able to have one per connection (that's a problem
      for later, maybe even needs a dedicated upstream connection)
    - TXN - ??? probably same situation as PR
  - or if we have a queue for pending Bers on the server, we not need to suspend
    clients, upstream is only chosen if the queue is free or there is a reason
    to send it to that particular upstream (multi-stage bind/VC, PR, ...), but
    that still makes it possible for a client to exhaust all our memory by
    sending requests (VC or other ones bound to a slow upstream or by not
    reading the responses at all)