Home | History | Annotate | Download | only in ata
History log of /src/sys/dev/ata/ata_wdc.c
RevisionDateAuthorComments
 1.120  05-Oct-2021  rin PR kern/56403

Fix kernel freeze for wdc(4) variants with ATAC_CAP_NOIRQ:

(1) Change ata_xfer_ops:c_poll from void to int function. When it returns
ATAPOLL_AGAIN, let ata_xfer_start() iterate itself again.

(2) Let wdc_ata_bio_poll() return ATAPOLL_AGAIN until ATA_ITSDONE is
achieved.

A similar change has been made for mvsata(4) (see mvsata_bio_poll()),
and no functional changes for other devices.

This is how the drivers worked before jdolecek-ncq branch was merged.

Note that this changes are less likely to cause infinite recursion:

(1) wdc_ata_bio_intr() called from wdc_ata_bio_poll() asserts ATA_ITSDONE
in its error handling paths via wdc_ata_bio_done().

(2) Return value from c_start (= wdc_ata_bio_start()) is checked in
ata_xfer_start().

Therefore, errors encountered in ata_xfer_ops:c_poll and c_start routines
terminate the recursion for wdc(4). The situation is similar for mvsata(4).

Still, there is a possibility where ata_xfer_start() takes long time to
finish a normal operation. This can result in a delayed response for lower
priority interrupts. But, I've never observed such a situation, even when
heavy thrashing takes place for swap partition in wd(4).

"Go ahead" by jdolecek@.
 1.119  25-Dec-2020  skrll Use designated initializers for struct ata_bustype
 1.118  21-May-2020  jdolecek branches: 1.118.2;
stop timeout handler while scheduling another part of partial I/O,
to avoid race between the timeout and I/O submission; the I/O
submission can sleep with xfer while waiting for the controller to
be ready once it gets to thread context, and timeout might cause
the xfer to be freed, leading to crashes due to use-after-free

this fixes another type of crashes with slow devices under QEMU reported
by Paul Ripke - thanks a lot with extensive debugging help
 1.117  19-May-2020  jdolecek only start the timeout machinery once the I/O is completely setup
and successful, particularly after PIO write is finished

fixes crashes in case the setup is so slow that timeout is triggered
e.g. while still waiting in wdc_wait_for_unbusy() or shortly after, without
drive actually having chance to complete the I/O, as seen in some
configuration under QEMU by Paul Ripke
 1.116  15-May-2020  jdolecek whitespace (bad indent)
 1.115  13-Apr-2020  jdolecek fix use-after-free for ata xfer on bio submission found by KASAN

driver ata_bio hooks read parts of the xfer after ata_exec_xfer()
call in order to determine return value, change so that the hook
doesn't return any value - callers do not care already,
as all I/O requests are asynchronous

this problem was uncovered by recent change for wd(4) to not hold
wd mutex during ata_bio call, the interrupt for the xfer might
thus actually fire immediately

adjust also ata_exec_command driver hooks similarily - remove all
completion and waiting logic from drivers, upper layer ata code
using AT_WAIT/AT_POLL changed to call ata_wait_cmd() itself

PR kern/55169 by Nick Hudson
 1.114  04-Apr-2020  jdolecek branches: 1.114.2;
fix deadlock in wdcwait() when xfer timeout happens while the atabus
thread sleeps in wdcwait() - check current lwp rather than relying
on global ATACH_TH_RUN channel flag

should fix the hang part of the problem reported in
http://mail-index.netbsd.org/netbsd-users/2020/03/12/msg024249.html

thanks to Paul Ripke for providing extensive debugging info
 1.113  12-Nov-2018  jdolecek branches: 1.113.4;
hold channel lock during whole ata_dmaerr()/ata_downgrade_mode() -
according to code inspection this is safe, none of the set_modes
hooks execute anything which would be taking the lock

adresses PR kern/53714 by Andreas Gustafsson
 1.112  22-Oct-2018  jdolecek Merge jdolecek-ncqfixes branch

- ata_xfer's are dynamicall allocated as needed using a pool, no longer
limited to number of possible openings supported by controller; dump
and recovery paths use dedicated pre-allocated storage
- moved callouts and condvars from ata_xfer to queue or channel, so that
ata_xfer does not need special initialization
- slot allocation now done when xfer is being activated, uncoupled
from memory allocation; active slots are no longer tracked by controller
code
- channel and drive reset is done always via the atabus thread, and
now executes with channel locked the whole time
- NCQ recovery moved to shared function, and run via the thread also
- added some workarounds for buggy error recovery AHCI emulation in QEMU
and Parallels

designed to primarily fix kern/52614, but might also help with kern/47041
and kern/53183
 1.111  03-Sep-2018  riastradh Rename min/max -> uimin/uimax for better honesty.

These functions are defined on unsigned int. The generic name
min/max should not silently truncate to 32 bits on 64-bit systems.
This is purely a name change -- no functional change intended.

HOWEVER! Some subsystems have

#define min(a, b) ((a) < (b) ? (a) : (b))
#define max(a, b) ((a) > (b) ? (a) : (b))

even though our standard name for that is MIN/MAX. Although these
may invite multiple evaluation bugs, these do _not_ cause integer
truncation.

To avoid `fixing' these cases, I first changed the name in libkern,
and then compile-tested every file where min/max occurred in order to
confirm that it failed -- and thus confirm that nothing shadowed
min/max -- before changing it.

I have left a handful of bootloaders that are too annoying to
compile-test, and some dead code:

cobalt ews4800mips hp300 hppa ia64 luna68k vax
acorn32/if_ie.c (not included in any kernels)
macppc/if_gm.c (superseded by gem(4))

It should be easy to fix the fallout once identified -- this way of
doing things fails safe, and the goal here, after all, is to _avoid_
silent integer truncations, not introduce them.

Maybe one day we can reintroduce min/max as type-generic things that
never silently truncate. But we should avoid doing that for a while,
so that existing code has a chance to be detected by the compiler for
conversion to uimin/uimax without changing the semantics until we can
properly audit it all. (Who knows, maybe in some cases integer
truncation is actually intended!)
 1.110  01-Jun-2018  macallan branches: 1.110.2; 1.110.4;
add a flag to start DMA before issuing commands - needed to work around a bug
in some SATA chips which get confused if the disk responds too fast
Mostly for K2 SATA / svwsata found in G5 Macs
adapted from OpenBSD
 1.109  17-Oct-2017  jdolecek branches: 1.109.2;
reintroduce ATACH_IRQ_WAIT flag for attachments using wdcintr(), only
process the interrupt when the flag is set - this fixes spurious interrupt
during post-reset drive setup in wdc_ata_bio_start(), and wdc_atapi_start()

while those functions set WDCTL_IDS, this seems to be ignored by certain
(maybe all) PCI-IDE controllers; usually the implicit KERNEL_LOCK() would
prevent the interrupt anyway, but not when the start routine is started
from the atabus thread, which doesn't take it

fixes 'panic: wdc_ata_bio_intr: bad state' reported on current-users
by Chavdar Ivanov
 1.108  15-Oct-2017  jdolecek revert the logic for wdc_wait_for_unbusy() in wdc_ata_bio_intr() to pre-NCQ,
wdcintr() and wdctimeout() need that; follows same change
in atapi_wdc.c rev. 1.128
 1.107  08-Oct-2017  christos Re-use the err label more so it is not unused for drivers that don't support
dma.
 1.106  07-Oct-2017  jdolecek Merge support for SATA NCQ (Native Command Queueing) from jdolecek-ncq branch

ATA subsystem was changed to support several outstanding commands, and use
NCQ xfers if supported by both the controller and the disk, including NCQ
error recovery. Set NCQ high priority for BPRIO_TIMECRITICAL xfers
if supported. Added FUA support.

Done some work towards MP-safe, all ATA code tsleep()/wakeup() replaced
by condvars, and switched most code from spl* to mutexes (separate
wd(4) and ata channel lock).

Introduced new option WD_CHAOS_MONKEY to facilitate testing of error
handling, fixed several uncovered issues. Also fixed several problems
with kernel dump to wd(4) disk.

Tested with ahcisata(4), mvsata(4), siisata(4), piixide(4) on amd64,
with and without port multiplier, both disk and ATAPI devices; other
drivers and archs mechanically adjusted and compile-tested. NCQ is
supported for ahcisata(4) and siisata(4) for any controller, for
mvsata(4) only Gen IIe ones for now. Also enabled ATAPI support in
mvsata(4).

Thanks to Matt Thomas for initial ATA infrastructure patch, and
Jonathan A.Kollasch for siisata(4) NCQ changes and general testing.

Also fixes PR kern/43169 (wd(4)); and PR kern/11811, PR kern/47041,
PR kern/51979 (kernel dump)
 1.105  02-Jan-2015  christos branches: 1.105.6;
We have three sets of DTYPE_ constants in the kernel:
altq Drop Type
disklabel Disk Type
file Descriptor Type
(not to mention constants that contain the string DTYPE).
Let's make them two, by changing the disklabel one to be DisK TYPE since the
other disklabel constants seem to do that. Not many userland programs use
these constants (and the ones that they do are mostly in ifdefs). They will
be fixed shortly.
 1.104  10-Sep-2014  matt branches: 1.104.2;
g/u_int[0-9]*_t/ s/u_int/uint/g
 1.103  03-Feb-2013  jakllsch Add argument to wdccommandext() to allow the entire contents of the
device/head register to be specified. Needed for upcoming port multipler
support in mvsata(4).
 1.102  19-Nov-2012  rkujawa Introduce WDC_CAPABILITY_NO_AUXCTL flag. For lame controllers that don't
have aux control registers (driver coming soon).
 1.101  31-Jul-2012  bouyer branches: 1.101.2;
Apply back changes that were reverted on Jul 24 and Jul 26 (general ata/wdc
cleanup and SATA PMP support), now that I'm back to fix the fallouts.
 1.100  26-Jul-2012  jakllsch Revert, with intention of restoring in a less invasive way, the SATA Port
Multiplier code.

ok christos@
 1.99  24-Jul-2012  jakllsch Revert dsl@'s changes of Sun, 15 Jul 2012 10:55:35 +0000 and
Sun, 15 Jul 2012 10:56:50 +0000, excepting the kernel version bump.
First step in reverting regressions to ata(4) subsystem during the addition of
port multiplier support.
 1.98  15-Jul-2012  dsl Some namespace protection (and add greppablity).
Prefix the DRIVE_ and DRIVET_ constants from atavar.h with ATA_.
Don't use an enum for drive_type - you don't know how big it will be.
Move driver_type to avoid implicit structure padding (esp on arm).
This change is purely lexical and mechanical.

Update to 6.99.9 - this wasn't done when the SATA PMP changes
were made - I'm sure they warranted a bump.
 1.97  02-Jul-2012  bouyer Add sata Port MultiPlier (PMP) support to the ata bus layer,
as described in
http://mail-index.netbsd.org/tech-kern/2012/06/23/msg013442.html
PMP support in integrated to the atabus layer.
struct ata_channel's ch_drive[] is not dynamically allocated, and ch_ndrive
(renamed to ch_ndrives) closely reflects the size of the ch_drive[] array.
Add helper functions atabus_alloc_drives() and atabus_free_drives()
to manage ch_drive[]/ch_ndrives.
Add wdc_maxdrives to struct wdc_softc so that bus front-end can specify
how much drive they really support (master/slave or single).
ata_reset_drive() callback gains a uint32_t *sigp argument which,
when not NULL, will contain the signature of the device being reset.
While there, some cosmetic changes:
- added a drive_type enum to ata_drive_datas, and stop encoding the
probed drive type in drive_flags (we were out of drive flags anyway).
- rename DRIVE_ATAPIST to DRIVE_ATAPIDSCW to better reflect what this
really is
- remove ata_channel->ata_drives, it's redundant with the pointer in
ata_drive_datas
- factor out the interpretation of SATA signatures in sata_interpet_sig()

propagate these changes to the ATA HBA drivers, and add support for PMP
to ahcisata(4) and siisata(4).

Thanks to:
- Protocase (http://www.protocase.com/) which provided a system
with lots of controllers, SATA PMP and drive slots
- Conservation Genomics Laboratory, Department of Biology, New Mexico State
University for hosting the above system
- Brook Milligan, who set up remote access and has been very responsive
when SATA cable move was needed
 1.96  09-Jan-2012  jakllsch Instead of assume that 'features' is 0 in wdccommandext(),
pass it as an argument, as is done for wdccommand().
 1.95  28-Aug-2011  christos branches: 1.95.2; 1.95.6;
make this compile.
 1.94  27-Aug-2011  bouyer The loongon2f+cs5526+jmicron PATA->SATA bridge cause an interresting issue:
1) because the CS5536 is not associated with a x86 CPU, interrupts are not
ack'ed as it expects so interrupts cannot configured as edge-triggered
(as is expected for a PCIIDE in compat mode)
2) the PATA->SATA bridge ignores the WDC_IDS (interrupt disable bit) so
the PATA IRQ line gets asserted when resetting or running some polled
commands. It also wrongly asserts IRQ when the (nonexistent) slave
device is selected
2) wouldn't be an issue with edge-triggered interrupt because we would
get a spurious interrupt and continue operation, a new interrupt only shows
up when the PATA IRQ line goes low and high again. But because of 1),
we get an unclearable interrupt instead, and the system loops on the
interrupt handler.

To workaround this, introduce a WDC_NO_IDS compile option which runs
all polled commands (including reset) at splbio() and without sleeps,
so that the controller's interrupt is effectively disabled and
won't be reenabled before the interrupt can be cleared.

The conditions triggering this problem are speficic enough to handle
this via a compile-time option; no need for a run-time (e.g. a
config(9), device property or callback to disable interrupts) solution.
 1.93  28-Mar-2010  snj Spell "enough" properly.
 1.92  19-Oct-2009  bouyer branches: 1.92.2; 1.92.4;
Remove closes 3 & 4 from my licence. Lots of thanks to Soren Jacobsen
for the booring work !
 1.91  16-Nov-2008  bouyer cpu_intr_p() doesn't account for software interrupts (e.g. callouts) so
we can't use it here. Rssurect ATACH_TH_RUN, backing out
src/sys/dev/ata/ata.c 1.101
src/sys/dev/ata/ata_wdc.c 1.90
src/sys/dev/ata/atavar.h 1.77
src/sys/dev/ic/wdc.c 1.255
src/sys/dev/scsipi/atapi_wdc.c 1.108
Should fix kern/39927 and kern/39725.
 1.90  02-Oct-2008  bouyer branches: 1.90.2; 1.90.4;
Kill ATACH_TH_RUN and use cpu_intr_p() instead.
 1.89  28-Apr-2008  martin branches: 1.89.2; 1.89.6;
Remove clause 3 and 4 from TNF licenses
 1.88  18-Mar-2008  cube branches: 1.88.2; 1.88.4;
Split device_t and softc for ATA devices, as well as wd(4). Other
cosmetic changes where appropriate.
 1.87  19-Oct-2007  ad branches: 1.87.12; 1.87.16;
machine/{bus,cpu,intr}.h -> sys/{bus,cpu,intr}.h
 1.86  08-Apr-2007  dyoung branches: 1.86.8; 1.86.10; 1.86.14;
Include opt_ata.h for ATADEBUG definition instead of #defining it
unconditionally.

Make this compile when ATADEBUG is not #defined.
 1.85  30-Sep-2006  itohy branches: 1.85.4; 1.85.8; 1.85.10;
A little effort against kernel bloat....
Exclude ATA DMA support if no ATA DMA capable drivers are compiled in.
 1.84  07-Sep-2006  itohy branches: 1.84.2; 1.84.4;
Add PIOBM (busmastering transfer using ATA PIO mode) support.
The PIOBM is used by only one driver (will be added later,
stay tuned) and intruduce an attribute "ata_piobm" so that
it will be conditionally compiled in.
The "ata_dma" (busmastering transfer using ATA DMA mode) and
"ata_udma" (busmastering transfer using ATA Ultra DMA mode)
attributes are also added for consistency, but unused for now.
 1.83  11-Dec-2005  christos branches: 1.83.4; 1.83.8;
merge ktrace-lwp.
 1.82  06-Jul-2005  thorpej branches: 1.82.2;
- When starting an ATA or ATAPI transfer, handle the case where (*dma_init)()
returns EINVAL, indicating that DMA cannot be done for this transfer.
Fall back to PIO in this case.
- Add a geodeide_dma_init() routine that checks to make sure that transfers
start on a 16 byte boundary, returning EINVAL if not. Works around a chip
bug that causes a hard system hang.

Problem reported and patch tested by Erik Fair.
 1.81  07-Jun-2005  peter Fix typos.
 1.80  29-May-2005  christos - remove casts [to avoid volatile]
- sprinkle const
- avoid shadowed variables
 1.79  27-Feb-2005  perry branches: 1.79.2;
nuke trailing whitespace
 1.78  16-Jan-2005  bouyer branches: 1.78.2;
Also call ata_dmaerr() on DMA errors (as reported by the underlying controller
driver). We want to downgrade to PIO mode if DMA isn't functionnal.
 1.77  08-Dec-2004  bouyer branches: 1.77.2;
Move ata_dmaerr() so that it's called only when there really has been an error
(ata_bio->r_error doesn't containt anything meaningfull if no error occured).
Fix kern/28255 by Izumi Tsutsui, but with a different patch.
 1.76  28-Oct-2004  yamt move buffer queue related stuffs from buf.h to their own header, bufq.h.
 1.75  21-Aug-2004  thorpej - Move to48() to atareg.h and rename it atacmd_to48().
- Add an atacmd_tosataq() that converts a 32-bit command to a Native
SATA Queued command.
 1.74  20-Aug-2004  thorpej - Add an (*ata_reset_channel)() member to ata_bustype.
- Add an ata_reset_channel() function that performs the common parts
of resetting an ATA channel, which uses the (*ata_reset_channel)()
callback to do the heavy lifting. Adjust callers to use ata_reset_channel()
instead of wdc_reset_channel().

This removes the last wdc-specific code from ata.c!
 1.73  20-Aug-2004  thorpej Move most of wdc_softc into a new atac_softc structure that contains
info common to all types of ATA controllers.
 1.72  14-Aug-2004  thorpej - Split the register handles out of struct wdc_channel into a separate
wdc_regs structure, and array of which (indexed per channel) is pointed
to by struct wdc_softc.
- Move the resulting wdc_channel structure to atavar.h and rename it to
ata_channel. Rename the corresponding flags.
- Add a "ch_ndrive" member to struct ata_channel, which indicates the
maximum number of drives that can be present on the channel. For now,
this is always 2. Add an ATA_MAXDRIVES constant that places an upper
limit on this value, also currently 2.
 1.71  13-Aug-2004  thorpej WDCDEBUG -> ATADEBUG.
 1.70  13-Aug-2004  thorpej Don't bother with bits that tell of the presence of optional callbacks;
just check the function pointers for NULL.
 1.69  13-Aug-2004  thorpej Move wdcstart() to ata.c and rename it to atastart().
 1.68  13-Aug-2004  thorpej Move wdc_exec_xfer() to ata.c and rename it ata_exec_xfer().
 1.67  12-Aug-2004  thorpej Move wdc_addref() and wdc_delref() to ata.c and rename them to
ata_*.
 1.66  12-Aug-2004  thorpej Move wdc_kill_pending() to ata.c and rename it ata_kill_pending().
 1.65  12-Aug-2004  thorpej - Move wdc_xfer_pool, wdc_get_xfer(), wdc_free_xfer() to ata.c, and rename
to ata_*.
- Use a static initializer for the ata_xfer_pool.
 1.64  12-Aug-2004  thorpej Rename some constants:
WDC_COMPLETE -> ATACMD_COMPLETE
WDC_QUEUED -> ATACMD_QUEUED
WDC_TRY_AGAIN -> ATACMD_TRY_AGAIN
 1.63  11-Aug-2004  mycroft Make datain_pio and dataout_pio function pointers in wdc_softc, which can
be overridden by the backend if desired. Add experimental code to wdc_pcmcia
to use this in memory-mapped mode, disabled by default.
 1.62  11-Aug-2004  mycroft Add two helper functions -- wdc_datain_pio() and wdc_dataout_pio() -- which
encapsulate the logic for the various methods of transferring data. Use these
throughout.
 1.61  04-Aug-2004  bouyer Change wdc_kill_pending() to take a struct ata_drive_datas * as argument,
and kill only pending requests for this drive.
Implement a DRIVE_WAITDRAIN flag, which will cause the active command to
be killed once complete.
Other minor fixes.
Now it's possible to detach a ATA or ATAPI device from ioctl even when
a dd on the raw char partition is running.
 1.60  04-Aug-2004  bouyer Add
struct ata_xfer *active_xfer
to ata_queue. Now the active xfer isn't the head of the queue any more,
this makes a few things easier (this will also help for tagged queuing
support).
Remove the WDCF_ACTIVE flag, test active_xfer != NULL instead.
clean up wdc_free_xfer() and kill_xfer().
Clean up wdc_reset_channel(), and make it issue a ATAPI_SOFT_RESET if the
active command is ATAPI.
In wdc_atapi_get_params(), use AT_WAIT | AT_POLL for ATAPI_SOFT_RESET,
so that we'll use tsleep() instead of delay().
In wdc_atapi_start(), call wdc_dmawait() at the right place.
 1.59  01-Aug-2004  bouyer Implement an atabus control device, and define some ATA bus control
IOCTLS. Implement ATABUSIORESET, which will reset the given ATA bus.
 1.58  31-Jul-2004  bouyer Implement asynchronous channel reset.
Use this to reset the channel before doing a dump, instead of the hack in
wdc_exec_xfer() based on C_POLL. This hack was causing problems on
controllers with a shared queue, because we now can have C_POLL set during
concurent channels probes (problem found and analysed on sparc64 by
Martin Husemann).
This should even make core dumps marginally more reliable on ATA drives.
 1.57  22-Jun-2004  mycroft Update copyright.
 1.56  22-Jun-2004  mycroft Do not force ATA_SINGLE on devices that don't support READ/WRITE MULTI. We
just have to take an interrupt for each sector.
Tested with one laptop disk (which normally runs in DMA mode and was forced
to single-sector transfers) and 3 CF cards. Increases the performance of
the CF cards substantially (760KB/s->1240KB/s in one case, 410KB/s->750KB/s
in the other two cases).
 1.55  01-Jun-2004  mycroft Only downgrade modes due to an actual CRC error. Downgrading on other errors
is anti-social -- especially given that there's no way to upgrade again short
of rebooting.

Also, downgrade UDMA modes more slowly. It's entirely possible that they're
using an 80-wire cable, but it's just too long for the higher modes, or there
is minor crosstalk.
 1.54  08-May-2004  bouyer Add a delay(10) after re-enabling interrupts in the control register.
Some controllers/drives (e.g. SataLink 3114 with WD Raptor) require
it. Should fix kern/23808 by Chris Gilbert, patch suplied by Chris Gilbert
on tech-kern, extended to all places enabling interrupts by me.
 1.53  02-Mar-2004  fvdl branches: 1.53.2;
During a control operation (like a reset), wait 10us after selecting
the drive. This fixes (or rather, works around) a timing problem
with WD Raptor drives attached to a Sil3114 SATA controller.

Should fix PR 23808, it fixes the same problem for me.
 1.52  03-Jan-2004  thorpej More structure member namespace cleanup: thread -> ch_thread
 1.51  03-Jan-2004  thorpej More wdc_channel structure member namespace cleanup:
- channel -> ch_channel
- wdc -> ch_wdc
 1.50  03-Jan-2004  thorpej Rename "struct channel_softc" to "struct wdc_channel".
 1.49  01-Jan-2004  thorpej Prepend "wdc_" to wait_for_drq, wait_for_unbusy, and wait_for_ready.
 1.48  01-Jan-2004  thorpej Rename:
- wdc_xfer to ata_xfer
- channel_queue to ata_queue
and move them to <dev/ata/atavar.h> so they can be used by non-wdc ATA
controllers. Clean up the member names of these structures while at it.
 1.47  14-Dec-2003  thorpej No need to include wdvar.h anymore.
 1.46  14-Dec-2003  thorpej Instead of calling wddone() directly, set a (*drv_done)() callback in
the ata_drive_datas structure and invoke that.
 1.45  14-Dec-2003  thorpej Liberally apply static.
 1.44  14-Dec-2003  thorpej De-__P and use ANSI-style function decls.
 1.43  27-Nov-2003  fvdl There are some cards that map the ATA control and IDE DMA registers
in a different fashion. Individually, they have the same functionality,
but their layout is different. An example of such a chipset is
the Promise 203xx.

To be able to deal with this, transform the cmd and dma bus_space handles
into an array of handles, each seperately created with bus_space_subregion.
The code generated by using the extra indirection shouldn't change much,
since the extra indirection is negated by having the offset calculation
already done in bus_space_subregion. E.g.

bus_space_write_4(tag, handle, offset, value)

becomes

bus_space_write_4(tag, handles[offset], 0, value)

Reviewed by Manuel Bouyer. Tested on wdc_isa, wdc_pcmcia, viaide, piixide (i386)
and on cmdide (sparc64).
 1.42  29-Oct-2003  bouyer freese->freeze, as pointed out by Frederick Bruck.
 1.41  08-Oct-2003  bouyer Make the ATA mid-layer appears as atabus, as proposed in
http://mail-index.netbsd.org/tech-kern/2003/09/25/0006.html
This adds a device (atabus) between IDE controllers and wd or atapibus, to
have each ATA channel show up in the device tree. Later there will be atabus
devices in /dev, so that we can do IOCTL on them.
Each atabus has its own kernel thread, to handle operations that needs polling,
e.g. reset and others.

Device probing on each bus it defered to the atabus thread creation.
This allows to do the reset and basic device probes in parallel, which reduce
boot time on systems with several pciide controllers.
 1.40  05-Oct-2003  bouyer Remove references to University of California from my copyright notices.
 1.39  28-Apr-2003  nakayama branches: 1.39.2;
Make DMA mode works on Promise Ultra66/100 with 48-bit LBA drives.
Ok'ed by bouyer in tech-kern@netbsd.org.
 1.38  27-Sep-2002  provos remove trailing \n in panic(). approved perry.
 1.37  26-Jul-2002  wiz enouth -> enough.
 1.36  23-Apr-2002  bouyer branches: 1.36.2; 1.36.4;
More copyright fixes, pointed out by Thomas. Thanks !
 1.35  14-Jan-2002  christos add the dma command ext translations. pointed out by manuel.
 1.34  13-Jan-2002  christos Add LBA48 support based on patches from Keisuke YOSHIDA. KNF a bit while
I am there.
 1.33  03-Dec-2001  bouyer Update my copyrigth.
 1.32  03-Dec-2001  bouyer Allow wd to attach to something else than wdc/pciide (like USB<->ATA bridges):
- move some functions from ata.c to ata_wdc.c or wdc.c.
- add callbacks to struct ata_bustype so that wd.c doesn't call directly
functions from the lower level driver.
 1.31  13-Nov-2001  lukem add RCSID
 1.30  13-Jun-2001  bjh21 branches: 1.30.2;
Add explicit support for IDE and SCSI adaptors which don't support interrupts.
On such adaptors, all transfers are done in polling mode.

OK'ed by Manuel on tech-kern.
 1.29  18-Apr-2001  jdolecek fix typo in comment - s/splio()/splbio()/
 1.28  12-Jun-2000  bouyer branches: 1.28.4;
Add a callback (*irqack), for controllers that need special action to ack
the interrupt once it has been ack'd on the drive.
 1.27  15-May-2000  bouyer branches: 1.27.2;
Sync my copyrigth notice
 1.26  01-Apr-2000  bouyer - DMA code cleanup: pciide_dma_finish() doesn't stop/unload the current DMA op
if an IRQ was not detected, unless the force flag was given. Use this to
detect if the IRQ was for us (closer to shared IRQ for controllers which
don't have their own IRQ handler in pciide.c) and to poll for DMA xfer.
Also makes the timeout recovery code simpler.
- ATAPI cleanup: don't call controller-specific functions from atapiconf.c
(wdc_*), so that it's possible to attach an atapibus to something else
than a wdc/pciide (Hi Lennart :).
Overload struct scsi_adapter with struct atapi_adapter, defined
as struct scsi_adapter + atapi-specific callbacks. scsipi_link still points
to an scsi_adapter, atapi code casts it to atapi_adapter if needed.
Move atapi_softc to atapiconf.h so that it can be used by the underlying
controller code (e.g. atapi_wdc.c).
Add an atapi-specific callback *atapi_probedev(), which probe a drive
in a controller-specific way, allocate the sc_link and fills in the
ataparams if needed. It then calls atapi_probedev() (from atapiconf.c)
to do the generic initialisations and attach the device.
- While I'm there merge and centralise the state definitions in atavar.h.
It should now be possible to use a common ata/atapi routine to set the
drive's modes (will do later).
 1.25  25-Mar-2000  bouyer Remove unneeded assignement to nblks
 1.24  23-Mar-2000  thorpej New callout mechanism with two major improvements over the old
timeout()/untimeout() API:
- Clients supply callout handle storage, thus eliminating problems of
resource allocation.
- Insertion and removal of callouts is constant time, important as
this facility is used quite a lot in the kernel.

The old timeout()/untimeout() API has been removed from the kernel.
 1.23  17-Jan-2000  bouyer Improve the downgrade logic:
- If UDMA 2 is failing try UDMA 1 first, it helps in some cases
- downgrade if we get an error in the first 4000 xfers, or if we get
4 errors in 4000 xfers if the first 4000 went without troubles.

While I'm there commit a local change I have since some time to get my CD
probed: issue a "blanck" IDENTIFY before the one used to detect slave ghosts,
with my drive the first IDENTIFY following a controller reset fails with an
aborted command ...
 1.22  20-Oct-1999  enami Cancel active transfers on aic/wdc detach.
Also makes LS-120 drive works for me again.
 1.21  09-Aug-1999  bouyer branches: 1.21.2; 1.21.4; 1.21.6;
Don't use C_NEEDDONE, it's deprecated.
 1.20  06-Aug-1999  bouyer - Add some debug printf (WDCPROBE) in _wdcreset_wait(), I've needed these
2 times in the past
- Set up timeout per xfer instead of per interrupt. This helps with
PIO transfer (we would call timeout()/untimout() several times for a
transfer).
- If we missed an interrupt for a PIO transfer, reset and restart it
immedialy, don't try to recover and continue. If we missed an interrupt we
may have lost a read/write cycle on the IDE bus. If this happens 1) we
corrupt data and 2) we enter an interrupt loop at the end of the xfer, as
the drive has some more data to read/write, but the host thinks the xfer is
done.
This last change fix the (or at last some of the) 'lookup after lost interrupt'
some peoples have been experiencing.
 1.19  01-Apr-1999  bouyer branches: 1.19.2;
- change the interrupt routines to take a 3rd arguments, set to 1 if we
are called from the interrupt or timeout handler, 0 otherwise.
- use this to know if we can busy-wait for wait_for_unbusy or wait_for_ready
This fixes a bug where CDs withot the DRQ_INTR capability would not busy-wait
for the CMDOUT phase.
While I'm there change 2 delay() to DELAY() for consistency, and
garbage-collect some old code from wdcintr() which has been ifdef'd out
for some time now.
 1.18  25-Mar-1999  bouyer Avoid busy-waiting when possible if we are not polling (real IRQ or timeout
callback). Shared PCI IRQ should now work (but still untested).
 1.17  17-Mar-1999  bouyer Revert to a 10s timeout, 1s is too low for drives in sleep mode.
 1.16  07-Mar-1999  bouyer Keep track of CRC errors in Ultra-DMA mode. If we noticed a CRC error and we
need to downgrade, downgrade to PIO, as it has been shown if we got CRC errors
in Ultra-DMA mode, we will have silent data corruption in multiword DMA mode
(isn't IDE wonderfull ? :).
Set timeout to 1s for "normal" ata I/O, to minimise the effects of missed
interrupts.
 1.15  21-Feb-1999  hubertf Only define WDCDEBUG if not already define (e.g. by kernel option)
 1.14  08-Feb-1999  bouyer There's no ATA draft where it is required for the drive to set DRDY | DSC when
the disk is ready to transfer data, and in ATA-5 the DSC has been obsoleted.
So only wait for DRQ to transfer data. This can be made conditional on the
ATA version if it's proven to break with some drives (worked with all the
drives I have access to).
While I'm there correct a few typos.
 1.13  29-Jan-1999  bouyer Cosmetic changes, from Soren S. Jorvan.
 1.12  09-Jan-1999  thorpej Make this compile again after void * arithmetic changes.
 1.11  08-Jan-1999  augustss Avoid arithmetic on `void *' since that's not ANSI C.
 1.10  16-Dec-1998  bouyer Keep track of DMA errors, and downgrade the transert mode (UDMA ->DMA,
DMA->PIO) in case of 2 consecutive errors. Don't downgrade if the
PIO/DMA/UDMA modes were forced by a config flag.
 1.9  21-Nov-1998  drochner in wdc_softc: access the per-channel data via a pointer array instead of
an array of fixed-sized channel_softc elements. This way IDE controllers
which more than 1 channel (pciide) can extend the channel data easily
for private needs.
To avoid the double dereference at runtime, change the argument of
wdcstart() to the channel data pointer instead of the array index.
 1.8  20-Nov-1998  thorpej Implement controller reference counting on ATA devices.
 1.7  11-Nov-1998  bouyer - clearify the boot messages (features supported vs features used). Thanks to
Havard Eidnes for his complains about this :)
- fix some typo in comments
- hoppefully better detection of drives reporting bogus PIO modes.
 1.6  13-Oct-1998  bouyer Add a new drive flag, DRIVE_MODE, set when the drive properly reported its
PIO/DMA modes. Don't try a SET_FEATURE if it didn't report its mode.
 1.5  13-Oct-1998  bouyer In debug message, always print controller:channel:drive
 1.4  13-Oct-1998  bouyer Properly separate DEBUF_XFERS and DEBUG_FUNCS: DEBUF_XFERS is for data
transfers (may be called often), where DEBUG_FUNCS is for setup functions
(not used for normal contitions).
 1.3  13-Oct-1998  thorpej Default to "no debugging messages" (still patchable).
 1.2  12-Oct-1998  bouyer Merge bouyer-ide
 1.1  04-Jun-1998  bouyer branches: 1.1.2;
file ata_wdc.c was initially added on branch bouyer-ide.
 1.1.2.15  05-Oct-1998  bouyer Rename WDC_CAPABILITY_PIO to WDC_CAPABILITY_MODE, as it's used to tell wether
the controller's driver can tell which timing mode it uses. Check this
before setting DMA modes too.
This allow the generic DMA code to work again.
 1.1.2.14  04-Oct-1998  bouyer Remove an unneeded delay().
 1.1.2.13  04-Oct-1998  bouyer New flag for wdc_xfer: C_DMA, set when a cmd will use DMA. This reduce some
if() in ata_wdc.c and atapi_wdc.c from 3 tests to one.
 1.1.2.12  02-Oct-1998  bouyer Cleanup a few error message, remove some dead code. Re-add a reset at end of
attach, unless we have WDC_NO_EXTRA_RESETS.
Add support for data32iot/data32ioh from -current: each controller pass
WDC_CAPABILITY_DATA32 if they can do 32bit, and WDC_CAPABILITY_DATA16 if they
can do 16bit. For controller that support both, the usual autodetect
mechanism is still used.
 1.1.2.11  20-Sep-1998  bouyer Init up xfer->drive in wdc_ata_bio(), or bad things will happen for drive 1.
No need to init xfer->channel, it's done in wdc_exec_xfer().
 1.1.2.10  20-Sep-1998  bouyer - always call config_found() from wdcattach(), instead of printing our own
the "not configured" message ourselve. When no atapibus is configured,
use a fake ata_atapi_attach, else call wdc_atapibus_attach(). This way we
don't have to include the whole atapi support in an ata-only config.
- the dma_* function pointers take a 'int flags' as last argument, instead
of an 'int read'. The 2 valid flags are: WDC_DMA_READ (to select read/write)
and WDC_DMA_POLL, to signal interrupt-less mode (for core dumps).
- Reworked wdcprobe() so look more like the old one. A status of 0xff is
interpreted as "no drive" (freebsd does this); this this speed up
the probe for non-IDE machines (a reset timeout has been bumped to 31s,
to match the specs). The probe set ups the drive flags to either ATA or
ATAPI (depend on register signature) or OLD (if register signature don't
match, but a working drive appears to be there). Later the ATA code
will look at the OLD flag to decide wether there is no drive, or an old
(ST506) one if WDC_IDENTIFY fails.
- For known PCI chips, don't use the wdcprobe() heuristic to find wether
a channel is enabled/disabled; use the chip's specific registers for this.
- Clear the command queue when we have a polled command. This allows dumps to
work even if a command was being handled by the controller at panic time.
- Increase the number of available DMA segments by one, as user requests may
not be aligned on a page boundary.
 1.1.2.9  21-Aug-1998  bouyer Sync with -current
 1.1.2.8  13-Aug-1998  bouyer - sync with HEAD
- better error detection and report for wd drives
- better wddump() support
- wdcwait() now takes a timout argument, so that we can honnor the
timeout argument of scsipi cmds.
 1.1.2.7  23-Jun-1998  leo Add necessary cruft to handle swapped byte-lanes on the data-register. It
basically boils down to the fact that there are 2 new options:
WDC_CAPABILITY_{ATA,ATAPI}_NOSTREAM with which you are able to control
the usage bus_space_read_multi_X() vs. bus_space_read_multi_stream_X().
 1.1.2.6  09-Jun-1998  bouyer If an error occured while setting up the drive, reset the drive's state
to 0.
 1.1.2.5  09-Jun-1998  bouyer - sync with trunck
- don't try to set the drive's PIO mode if we don't know how to set it on the
controller. This seems to confuse some 'intelligent' ISA controllers.
 1.1.2.4  05-Jun-1998  bouyer _DMA -> _UDMA, + cosmetic changes.
Ultra DMA still doesn't work.
 1.1.2.3  05-Jun-1998  bouyer - call wdc_ata_bio_done() only once.
- better error recovery: if an error occured, retry the transfer in
single sector PIO mode.
 1.1.2.2  05-Jun-1998  bouyer A few bux fixes (untested):
- If the PIO mode of the controller is unknown, put the drive in the
hightest mode it supports. A drive in a faster mode than the controller
should work.
- Create DMA maps even for unknown controllers in pciide.
 1.1.2.1  04-Jun-1998  bouyer Commit changes to the IDE system in a branch. This allows a better separation
between higth-level and low-level (i.e. registers read/write) and generalize
the queue for all commands. This also add supports for IDE DMA.
 1.19.2.3  07-Jul-2000  he Apply patch (requested by bouyer):
Add support for the following PCIIDE controllers:
o AMD 756
o CMD PCI0648 and PCI0649
o Hightpoint HPT366
o OPTi 82c621 (and a few of its derivatives)
o Promise Ultra/33 and Ultra/66
o Intel 82801 (ICH/ICH0)
Also fix PR#10437 (detect more ATAPI devices).
 1.19.2.2  23-Jan-2000  he Pull up revision 1.23 (requested by bouyer):
Improve the downgrade logic to work better in some cases (especially
for Acer Labs M5229 controllers with Ultra-DMA/66 disks).
 1.19.2.1  25-Aug-1999  he Pull up revision 1.19->1.20:
Add some debug printfs, change timeout handling to be more efficient
and less error-prone, and fix the "lockup after lost interrupt"
problem some users have been reporting. (bouyer)
 1.21.6.1  27-Dec-1999  wrstuden Pull up to last week's -current.
 1.21.4.1  15-Nov-1999  fvdl Sync with -current
 1.21.2.3  21-Apr-2001  bouyer Sync with HEAD
 1.21.2.2  20-Nov-2000  bouyer Update thorpej_scsipi to -current as of a month ago
A i386 GENERIC kernel compiles without the siop, ahc and bha drivers
(will be updated later). i386 IDE/ATAPI and ncr work, as well as
sparc/esp_sbus. alpha should work as well (untested yet).
siop, ahc and bha will be updated once I've updated the branch to current
-current, as well as machine-dependant code.
 1.21.2.1  20-Oct-1999  thorpej Sync w/ trunk.
 1.27.2.1  22-Jun-2000  minoura Sync w/ netbsd-1-5-base.
 1.28.4.7  18-Oct-2002  nathanw Catch up to -current.
 1.28.4.6  01-Aug-2002  nathanw Catch up to -current.
 1.28.4.5  20-Jun-2002  nathanw Catch up to -current.
 1.28.4.4  28-Feb-2002  nathanw Catch up to -current.
 1.28.4.3  08-Jan-2002  nathanw Catch up to -current.
 1.28.4.2  14-Nov-2001  nathanw Catch up to -current.
 1.28.4.1  21-Jun-2001  nathanw Catch up to -current.
 1.30.2.5  10-Oct-2002  jdolecek sync kqueue with -current; this includes merge of gehenna-devsw branch,
merge of i386 MP branch, and part of autoconf rototil work
 1.30.2.4  06-Sep-2002  jdolecek sync kqueue branch with HEAD
 1.30.2.3  23-Jun-2002  jdolecek catch up with -current on kqueue branch
 1.30.2.2  11-Feb-2002  jdolecek Sync w/ -current.
 1.30.2.1  10-Jan-2002  thorpej Sync kqueue branch with -current.
 1.36.4.1  28-Apr-2003  tron Pull up revision 1.39 (requested by nakayama in ticket #1275):
Make DMA mode works on Promise Ultra66/100 with 48-bit LBA drives.
Ok'ed by bouyer in tech-kern@netbsd.org.
 1.36.2.1  29-Aug-2002  gehenna catch up with -current.
 1.39.2.10  10-Nov-2005  skrll Sync with HEAD. Here we go again...
 1.39.2.9  04-Mar-2005  skrll Sync with HEAD.

Hi Perry!
 1.39.2.8  17-Jan-2005  skrll Sync with HEAD.
 1.39.2.7  18-Dec-2004  skrll Sync with HEAD.
 1.39.2.6  02-Nov-2004  skrll Sync with HEAD.
 1.39.2.5  21-Sep-2004  skrll Fix the sync with head I botched.
 1.39.2.4  18-Sep-2004  skrll Sync with HEAD.
 1.39.2.3  25-Aug-2004  skrll Sync with HEAD.
 1.39.2.2  12-Aug-2004  skrll Sync with HEAD.
 1.39.2.1  03-Aug-2004  skrll Sync with HEAD
 1.53.2.3  11-Aug-2004  jmc branches: 1.53.2.3.2;
Pullup rev 1.58 (requested by bouyer in ticket #733)

Implement asynchronous channel reset.
Use this to reset the channel before doing a dump, instead of the hack in
wdc_exec_xfer() based on C_POLL. This hack was causing problems on
controllers with a shared queue, because we now can have C_POLL set during
concurent channels probes
 1.53.2.2  02-Jul-2004  he Pull up revisions 1.56-1.57 (requested by mycroft in ticket #572):
Several fixes mostly related to USB:
o Don't force ATA_SINGLE on devices which don't support
READ/WRITE MULTI. Increases performance with several
CF cards substantially.
o Update copyright.
 1.53.2.1  29-May-2004  tron Pull up revision 1.54 (requested by bouyer in ticket #396):
Add a delay(10) after re-enabling interrupts in the control register.
Some controllers/drives (e.g. SataLink 3114 with WD Raptor) require
it. Should fix kern/23808 by Chris Gilbert, patch suplied by Chris Gilbert
on tech-kern, extended to all places enabling interrupts by me.
 1.53.2.3.2.4  18-Jul-2005  riz Pull up revision 1.82 (requested by thorpej in ticket #2072):
- When starting an ATA or ATAPI transfer, handle the case where
(*dma_init)()
returns EINVAL, indicating that DMA cannot be done for this
transfer.
Fall back to PIO in this case.
- Add a geodeide_dma_init() routine that checks to make sure that
transfers
start on a 16 byte boundary, returning EINVAL if not. Works
around a chip
bug that causes a hard system hang.
Problem reported and patch tested by Erik Fair.
 1.53.2.3.2.3  16-Mar-2005  tron Pull up revision 1.78 (requested by bouyer in ticket #1325):
Also call ata_dmaerr() on DMA errors (as reported by the underlying controller
driver). We want to downgrade to PIO mode if DMA isn't functionnal.
 1.53.2.3.2.2  16-Mar-2005  tron Pull up revision 1.77 (requested by bouyer in ticket #1324):
Move ata_dmaerr() so that it's called only when there really has been an error
(ata_bio->r_error doesn't containt anything meaningfull if no error occured).
Fix kern/28255 by Izumi Tsutsui, but with a different patch.
 1.53.2.3.2.1  16-Mar-2005  jmc Pullup rev 1.55 (requested by mycroft in ticket #1018)

Only downgrade modes due to an actual CRC error. Downgrading on other errors
is anti-social -- especially given that there's no way to upgrade again short
of rebooting.

Also, downgrade UDMA modes more slowly. It's entirely possible that they're
using an 80-wire cable, but it's just too long for the higher modes, or there
is minor crosstalk.
 1.77.2.1  29-Apr-2005  kent sync with -current
 1.78.2.1  19-Mar-2005  yamt sync with head. xen and whitespace. xen part is not finished.
 1.79.2.1  06-Jul-2005  tron Pull up revision 1.82 (requested by thorpej in ticket #553):
- When starting an ATA or ATAPI transfer, handle the case where (*dma_init)()
returns EINVAL, indicating that DMA cannot be done for this transfer.
Fall back to PIO in this case.
- Add a geodeide_dma_init() routine that checks to make sure that transfers
start on a 16 byte boundary, returning EINVAL if not. Works around a chip
bug that causes a hard system hang.
Problem reported and patch tested by Erik Fair.
 1.82.2.4  24-Mar-2008  yamt sync with head.
 1.82.2.3  27-Oct-2007  yamt sync with head.
 1.82.2.2  03-Sep-2007  yamt sync with head.
 1.82.2.1  30-Dec-2006  yamt sync with head.
 1.83.8.1  14-Sep-2006  yamt sync with head.
 1.83.4.1  09-Sep-2006  rpaulo sync with head
 1.84.4.1  22-Oct-2006  yamt sync with head
 1.84.2.1  18-Nov-2006  ad Sync with head.
 1.85.10.1  11-Jul-2007  mjf Sync with head.
 1.85.8.2  23-Oct-2007  ad Sync with head.
 1.85.8.1  10-Apr-2007  ad Sync with head.
 1.85.4.1  15-Apr-2007  yamt sync with head.
 1.86.14.1  25-Oct-2007  bouyer Sync with HEAD.
 1.86.10.2  23-Mar-2008  matt sync with HEAD
 1.86.10.1  06-Nov-2007  matt sync with HEAD
 1.86.8.1  26-Oct-2007  joerg Sync with HEAD.

Follow the merge of pmap.c on i386 and amd64 and move
pmap_init_tmp_pgtbl into arch/x86/x86/pmap.c. Modify the ACPI wakeup
code to restore CR4 before jumping back into kernel space as the large
page option might cover that.
 1.87.16.4  17-Jan-2009  mjf Sync with HEAD.
 1.87.16.3  05-Oct-2008  mjf Sync with HEAD.
 1.87.16.2  02-Jun-2008  mjf Sync with HEAD.
 1.87.16.1  03-Apr-2008  mjf Sync with HEAD.
 1.87.12.1  24-Mar-2008  keiichi sync with head.
 1.88.4.3  11-Aug-2010  yamt sync with head.
 1.88.4.2  11-Mar-2010  yamt sync with head
 1.88.4.1  16-May-2008  yamt sync with head.
 1.88.2.1  18-May-2008  yamt sync with head.
 1.89.6.2  13-Dec-2008  haad Update haad-dm branch to haad-dm-base2.
 1.89.6.1  19-Oct-2008  haad Sync with HEAD.
 1.89.2.1  10-Oct-2008  skrll Sync with HEAD.
 1.90.4.1  20-Nov-2008  snj Pull up following revision(s) (requested by bouyer in ticket #67):
sys/dev/ic/wdc.c: revision 1.256
sys/dev/ata/ata.c: revision 1.102
sys/dev/scsipi/atapi_wdc.c: revision 1.109
sys/dev/ata/atavar.h: revision 1.78
sys/dev/ata/ata_wdc.c: revision 1.91
cpu_intr_p() doesn't account for software interrupts (e.g. callouts) so
we can't use it here. Rssurect ATACH_TH_RUN, backing out
src/sys/dev/ata/ata.c 1.101
src/sys/dev/ata/ata_wdc.c 1.90
src/sys/dev/ata/atavar.h 1.77
src/sys/dev/ic/wdc.c 1.255
src/sys/dev/scsipi/atapi_wdc.c 1.108
Should fix kern/39927 and kern/39725.
 1.90.2.1  19-Jan-2009  skrll Sync with HEAD.
 1.92.4.1  30-May-2010  rmind sync with head
 1.92.2.1  30-Apr-2010  uebayasi Sync with HEAD.
 1.95.6.1  18-Feb-2012  mrg merge to -current.
 1.95.2.4  22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.95.2.3  16-Jan-2013  yamt sync with (a bit old) head
 1.95.2.2  30-Oct-2012  yamt sync with head
 1.95.2.1  17-Apr-2012  yamt sync with head
 1.101.2.2  03-Dec-2017  jdolecek update from HEAD
 1.101.2.1  25-Feb-2013  tls resync with head
 1.104.2.1  06-Apr-2015  skrll Sync with HEAD
 1.105.6.12  30-Sep-2017  jdolecek must drop channel lock before calling ata_dmaerr() to avoid 'locking against
myself' in case of consecutive errors from ata_reset_channel() called via
ata_downgrade_mode()
 1.105.6.11  26-Sep-2017  jdolecek make compile without NATA_DMA
 1.105.6.10  21-Sep-2017  jdolecek add missing ata_channel_unlock() in bailout path of wdc_ata_bio_intr()
 1.105.6.9  10-Sep-2017  jdolecek refactor code so that xfer c_start() hook is called with channel mutex held,
and hence the controller submit code no longer relies on spl

tested all the affected drivers - wdc (via piixide), ahci, mvsata, siisata,
both disk and atapi I/O
 1.105.6.8  12-Aug-2017  jdolecek convert the atabus thread to use the channel lock and a condvar, adjust
code which sets the relevant channel flags to take the lock while doing so
 1.105.6.7  12-Aug-2017  jdolecek remove all logic around ATACH_IRQ_WAIT and channel-global ch_error/ch_status,
so that there is less hidden state shared by commands; primary intent is
to make the NCQ and non-NCQ paths more similar, and remove possibility
of incorrect handling for the NCQ commands

tested both disk and ATAPI - piixide(4) on QEMU, and siisata(4),
ahcisata(4), mvsata(4) on real hw
 1.105.6.6  27-Jun-2017  jdolecek attend error paths, more strict asserts and code consistency

- atastart() and ata_kill_pending() now KASSERT() that all xfers on queue
have same channel
- inactive xfers are killed via new reason KILL_GONE_INACTIVE, controller
code must not call any resource deactivation in that case
- c_intr() must call ata_waitdrain_xfer_check() as first thing, and must not
further touch any xfer structures on exit path; any resource cleanup
is supposed to be done in c_kill_xfer()
- c_kill_xfer() should never call atastart()
- ata_waitdrain_check() removed, replaced by ata_waitdrain_xfer_check()
- ATA_DRIVE_WAITDRAIN handling converted to use condvar
- removed unused ata_c callback
 1.105.6.5  20-Jun-2017  jdolecek remove ata queue downsizing - every device, attached to the same channel,
uses slots according to it's own limits

wdc code changed to expect maximum one active xfer, and not check number
of openings in the channel; this is to facilitate using wdc functions
for e.g. handling of atapi commands for drivers which support both ATAPI
and NCQ
 1.105.6.4  16-Jun-2017  jdolecek adjust reset channel and dump paths
- channel reset now always kills active transfer, even on dump path, but
now doesn't touch the queued waiting transfers; also kill_xfer hook is
always called, so that HBA can free any private xfer resources and thus
the dump request has chance to work
- kill_xfer routines now always call ata_deactivate_xfer(); added KASSERT()s
to ata_free_xfer() to expect deactivated xfer
- when called during channel reset before dump, ata_kill_active() drops
any queued waiting transfers without processing
- do not (re)queue any transfers in wddone() when dumping
- kill AT_RST_NOCMD flag

This should also hopefully fix the 'polled command has been queued' panic
as reported in:
PR kern/11811 by John Hawkinson
PR kern/47041 by Taylor R Campbell
PR kern/51979 by Martin Husemann

dump tested working with piixide(4) and ahci(4). mvsata(4) dump times out,
but otherwise tested working, will be fixed separately. siisata(4) mechanically
changed and not tested.
 1.105.6.3  19-Apr-2017  jdolecek adjust ata code to support more than one active command, including the
timeout handling, add support for NCQ commands

move probe for NCQ and number of tags to middle layer, negotiate mutual
support between drive and controller

implement NCQ support in ahci(4)
 1.105.6.2  15-Apr-2017  jdolecek make ata_xfer's allocated as part of ata_queue and make it include ata_bio;
they are pre-allocated on attach and ata_get_xfer() now never sleep, drop the
pool

modify wd(4) to file the bio requests using the xfers and hence
make it possible to have more than one active I/O request in flight;
ata_bio callback doesn't need to allocate any memory any more,
require it to never return ATACMD_TRY_AGAIN

move lp, badsect, multi from ata_bio to ata_drive_datas, as they are per-drive,
not per transfer

drop unused drv atac_claim_hw/atac_free_hw hooks, and also drop again ata_bio
c_hwslot
 1.105.6.1  10-Apr-2017  jdolecek ATA infrastructure improvements to eventually support more outstanding
commands

patch by Matt Thomas
 1.109.2.3  26-Nov-2018  pgoyette Sync with HEAD, resolve a couple of conflicts
 1.109.2.2  06-Sep-2018  pgoyette Sync with HEAD

Resolve a couple of conflicts (result of the uimin/uimax changes)
 1.109.2.1  25-Jun-2018  pgoyette Sync with HEAD
 1.110.4.4  15-Oct-2018  jdolecek change the SATA/NCQ recovery to run in the atabus thread
 1.110.4.3  17-Sep-2018  jdolecek convert from malloc()/free() to kmem_zalloc()/kmem_free()
 1.110.4.2  17-Sep-2018  jdolecek move low-level protocol handlers hooks from ata_xfer to separate struct,
initialized statically

primarily to reduce ata_xfer struct size, but also improves readibility,
and enforces consistency
 1.110.4.1  31-Aug-2018  jdolecek refactor ata_xfer to be just dumb structure; move all callouts/condvars out

retry callout to wd(4); reset callout and the active/cmd finish condvars
to channel queue; change code using the condvars so it works if there
are multiple waiters

simplify the async wait code for cmds, replace ata_wait_xfer()/ata_wake_xfer()
with ata_wait_cmd()

fix the callout_invoking/ack race handling code for timeouts to
actually have chance to work; change mvsata(4) to use generic timeout func

towards resolution of kern/52614
 1.110.2.3  21-Apr-2020  martin Sync with HEAD
 1.110.2.2  08-Apr-2020  martin Merge changes from current as of 20200406
 1.110.2.1  10-Jun-2019  christos Sync with HEAD
 1.113.4.1  30-Dec-2022  martin Pull up following revision(s) (requested by tsutsui in ticket #1557):

sys/dev/ic/ahcisata_core.c: revision 1.83
sys/dev/ic/ahcisata_core.c: revision 1.102
sys/dev/ata/ata.c: revision 1.164
sys/dev/ata/ata_wdc.c: revision 1.115
sys/dev/ata/ata_recovery.c: revision 1.4
sys/dev/ic/siisata.c: revision 1.42
sys/dev/ic/wdc.c: revision 1.308
sys/dev/ic/mvsata.c: revision 1.56
sys/dev/scsipi/atapi_wdc.c: revision 1.138
sys/dev/ic/siisata.c: revision 1.49
sys/dev/ata/atavar.h: revision 1.105
sys/dev/ata/wd.c: revision 1.460
sys/dev/ata/ata.c: revision 1.155
sys/dev/ata/wd.c: revision 1.462
sys/dev/ata/atavar.h: revision 1.109
sys/dev/ata/satapmp_subr.c: revision 1.16
sys/dev/ic/wdc.c: revision 1.299
sys/dev/ic/ahcisata_core.c: revision 1.93
sys/dev/ata/ata_wdc.c: revision 1.120
sys/dev/ic/wdcvar.h: revision 1.100
sys/dev/scsipi/atapi_wdc.c: revision 1.141
sys/dev/ic/mvsata.c: revision 1.61
sys/dev/usb/umass_isdata.c (apply patch)

drop wd lock in wdstart1() before calling the ata_bio hook; when called
from ata thread context, that can still need to sleep for wdc attachments
in wdcwait()

fix use-after-free for ata xfer on bio submission found by KASAN
driver ata_bio hooks read parts of the xfer after ata_exec_xfer()
call in order to determine return value, change so that the hook
doesn't return any value - callers do not care already,
as all I/O requests are asynchronous

this problem was uncovered by recent change for wd(4) to not hold
wd mutex during ata_bio call, the interrupt for the xfer might
thus actually fire immediately

adjust also ata_exec_command driver hooks similarily - remove all
completion and waiting logic from drivers, upper layer ata code
using AT_WAIT/AT_POLL changed to call ata_wait_cmd() itself
PR kern/55169 by Nick Hudson

Function declaration formating whitespace consistency. NFCI.

PR kern/56403
Fix kernel freeze for wdc(4) variants with ATAC_CAP_NOIRQ:

(1) Change ata_xfer_ops:c_poll from void to int function. When it returns
ATAPOLL_AGAIN, let ata_xfer_start() iterate itself again.
(2) Let wdc_ata_bio_poll() return ATAPOLL_AGAIN until ATA_ITSDONE is
achieved.

A similar change has been made for mvsata(4) (see mvsata_bio_poll()),
and no functional changes for other devices.

This is how the drivers worked before jdolecek-ncq branch was merged.
Note that this changes are less likely to cause infinite recursion:

(1) wdc_ata_bio_intr() called from wdc_ata_bio_poll() asserts ATA_ITSDONE
in its error handling paths via wdc_ata_bio_done().
(2) Return value from c_start (= wdc_ata_bio_start()) is checked in
ata_xfer_start().

Therefore, errors encountered in ata_xfer_ops:c_poll and c_start routines
terminate the recursion for wdc(4). The situation is similar for mvsata(4).

Still, there is a possibility where ata_xfer_start() takes long time to
finish a normal operation. This can result in a delayed response for lower
priority interrupts. But, I've never observed such a situation, even when
heavy thrashing takes place for swap partition in wd(4).
"Go ahead" by jdolecek@.
 1.114.2.1  20-Apr-2020  bouyer Sync with HEAD
 1.118.2.1  03-Jan-2021  thorpej Sync w/ HEAD.

RSS XML Feed