xref: /src/sys/dev/dm/doc/locking.txt

				Device-mapper Locking architecture

Overview

There are 2 users in device-mapper driver 
      a) Users who uses disk drives 
      b) Users who uses ioctl management interface

Management is done by dm_dev_*_ioctl and dm_table_*_ioctl routines. There are 
two major structures used in these routines/device-mapper. 

Table entry:

typedef struct dm_table_entry {
        struct dm_dev *dm_dev;          /* backlink */
        uint64_t start;
        uint64_t length;

        struct dm_target *target;      /* Link to table target. */
        void *target_config;           /* Target specific data. */
        SLIST_ENTRY(dm_table_entry) next;
} dm_table_entry_t;

This structure stores every target part of dm device. Every device can have
more than one target mapping entries stored in a list. This structure describe
mapping between logical/physical blocks in dm device. 

start  length target block device offset
0 	   102400 linear /dev/wd1a     384
102400 204800 linear /dev/wd2a     384
204800 409600 linear /dev/wd3a     384

Every device has at least two tables ACTIVE and INACTIVE. Only ACTIVE table is 
used during IO. Every IO operation on dm device have to walk through dm_table_entries list. 

Device entry:

typedef struct dm_dev {
        char name[DM_NAME_LEN];
        char uuid[DM_UUID_LEN];

        int minor;
        uint32_t flags; /* store communication protocol flags */

        kmutex_t dev_mtx; /* mutex for generall device lock */
        kcondvar_t dev_cv; /* cv for ioctl synchronisation */

        uint32_t event_nr;
        uint32_t ref_cnt;

        uint32_t dev_type;

        dm_table_head_t table_head;

        struct dm_dev_head upcalls;

        struct disklabel *dk_label;    /* Disklabel for this table. */

        TAILQ_ENTRY(dm_dev) next_upcall; /* LIST of mirrored, snapshoted devices. */

        TAILQ_ENTRY(dm_dev) next_devlist; /* Major device list. */
} dm_dev_t;

Every device created in dm device-mapper is represented with this structure. 
All devices are stored in a list. Every ioctl routine have to work with this 
structure.

	Locking in dm driver
	
Locking must be done in two ways. Synchronisation between ioctl routines and 
between IO operations and ioctl. Table entries are read during IO and during some ioctl routines. There are only few routines which manipulates table lists.

Read access to table list:

dmsize 
dmstrategy
dm_dev_status_ioctl
dm_table_info_ioctl
dm_table_deps_ioctl
dm_disk_ioctl 		-> DIOCCACHESYNC ioctl 

Write access to table list:
dm_dev_remove_ioctl        -> remove device from list, this routine have to 		 
							  remove all tables.
dm_dev_resume_ioctl		   -> Switch tables on suspended device, switch INACTIVE 
							  and ACTIVE tables.
dm_table_clear_ioctl  	   -> Remove INACTIVE table from table list.


Synchronisation between readers and writers in table list

I moved everything needed for table synchronisation to struct dm_table_head.

typedef struct dm_table_head {
        /* Current active table is selected with this. */
        int cur_active_table;
        struct dm_table tables[2];

        kmutex_t   table_mtx;
        kcondvar_t table_cv; /*IO waiting cv */

        uint32_t io_cnt;
} dm_table_head_t;

dm_table_head_t is used as entry for every dm_table synchronisation routine.

Because every table user have to get list to table list head I have implemented
these routines to manage access to table lists. 

/*                                                                                                            
 * Destroy all table data. This function can run when there are no                                            
 * readers on table lists.                                                                                    
 */
int dm_table_destroy(dm_table_head_t *, uint8_t);

/*                                                                                                            
 * Return length of active table in device.                                                                   
 */
uint64_t dm_table_size(dm_table_head_t *);

/*                                                                                                            
 * Return current active table to caller, increment io_cnt reference counter.                                 
 */
struct dm_table * dm_table_get_entry(dm_table_head_t *, uint8_t);

/*                                                                                                            
 * Return > 0 if table is at least one table entry (returns number of entries)                                
 * and return 0 if there is not. Target count returned from this function                                     
 * doesn't need to be true when userspace user receive it (after return                                       
 * there can be dm_dev_resume_ioctl), therfore this isonly informative.                                       
 */
int dm_table_get_target_count(dm_table_head_t *, uint8_t);

/*                                                                                                            
 * Decrement io reference counter and wake up all callers, with table_head cv.                                
 */
void dm_table_release(dm_table_head_t *, uint8_t s);

/*                                                                                                            
 * Switch table from inactive to active mode. Have to wait until io_cnt is 0.                                 
 */
void dm_table_switch_tables(dm_table_head_t *);

/*                                                                                                            
 * Initialize table_head structures, I'm trying to keep this structure as                                     
 * opaque as possible.                                                                                        
 */
void dm_table_head_init(dm_table_head_t *);

/*                                                                                                            
 * Destroy all variables in table_head                                                                        
 */
void dm_table_head_destroy(dm_table_head_t *);

Internal table synchronisation protocol

Readers:
dm_table_size
dm_table_get_target_count
dm_table_get_target_count

Readers with hold reference counter:
dm_table_get_entry
dm_table_release

Writer:
dm_table_destroy
dm_table_switch_tables

For managing synchronisation to table lists I use these routines. Every reader 
uses dm_table_busy routine to hold reference counter during work and dm_table_unbusy for reference counter release. Every writer have to wait while 
is reference counter 0 and only then it can work with device. It will sleep on 
head->table_cv while there are other readers. dm_table_get_entry is specific in that it will return table with hold reference counter. After dm_table_get_entry 
every caller must call dm_table_release when it doesn't want to work with it. 

/*                                                                                                            
 * Function to increment table user reference counter. Return id                                              
 * of table_id table.                                                                                         
 * DM_TABLE_ACTIVE will return active table id.                                                               
 * DM_TABLE_INACTIVE will return inactive table id.                                                           
 */
static int
dm_table_busy(dm_table_head_t *head, uint8_t table_id)
{
        uint8_t id;

        id = 0;

        mutex_enter(&head->table_mtx);

        if (table_id == DM_TABLE_ACTIVE)
                id = head->cur_active_table;
        else
                id = 1 - head->cur_active_table;

        head->io_cnt++;

        mutex_exit(&head->table_mtx);
        return id;
}

/*                                                                                                            
 * Function release table lock and eventually wakeup all waiters.                                             
 */
static void
dm_table_unbusy(dm_table_head_t *head)
{
        KASSERT(head->io_cnt != 0);

        mutex_enter(&head->table_mtx);

        if (--head->io_cnt == 0)
                cv_broadcast(&head->table_cv);

        mutex_exit(&head->table_mtx);
}

Device-mapper betwwen ioctl device synchronisation 


Every ioctl user have to find dm_device with name, uuid, minor number. 
For this dm_dev_lookup is used. This routine returns device with hold reference 
counter. 

void
dm_dev_busy(dm_dev_t *dmv)
{
        mutex_enter(&dmv->dev_mtx);
        dmv->ref_cnt++;
        mutex_exit(&dmv->dev_mtx);
}

void
dm_dev_unbusy(dm_dev_t *dmv)
{
        KASSERT(dmv->ref_cnt != 0);

        mutex_enter(&dmv->dev_mtx);
        if (--dmv->ref_cnt == 0)
                cv_broadcast(&dmv->dev_cv);
        mutex_exit(&dmv->dev_mtx);
}

Before returning from ioctl routine must release reference counter with 
dm_dev_unbusy.

dm_dev_remove_ioctl routine have to remove dm_dev from global device list,
and wait until all ioctl users from dm_dev are gone.