Home | History | Annotate | Download | only in arm32
History log of /src/sys/arch/arm/arm32/bus_dma.c
RevisionDateAuthorComments
 1.150  26-Sep-2025  skrll arm: Adjust barriers issued in bus_dmamap_sync for the coherent case.

PR/59654

This change adjusts the memory barriers issued by bus_dmamap_sync for
the coherent case. In the non-coherent case the CPU cache operations
are expected to provide any, and all required barriers.

These barriers are emitted after bouncing for PREWRITE and before
bouncing for POSTREAD.

A new indrection is introduced to deal with the differencs in barrier
(dsb) options between Arm V8 and V7, the lack of options in Arm V6,
and the need to use cpu_drain_writebuf for all other earlier versions.

The Arm V8 Net change is:

op old new
 1.149  08-Mar-2025  jmcneill arm: Support BUS_DMA_NOCACHE
 1.148  10-Dec-2024  msaitoh arm/bus_dma: Sprinkle error check with __predict_{true,false}.

Sprinkle error check in bus_dmamap_load*() and bus_dmamap_sync() with
__predict_{true,false} to improve performance.
 1.147  20-Oct-2024  skrll Revert previous - I misread the code.
 1.146  20-Oct-2024  skrll G/C an unused event counter.
 1.145  20-Oct-2024  skrll Sync with the RISC-V version. Mostly

Use PRI{u,x}{BUSSIZE,VADDR,PADDR}
 1.144  28-Jul-2023  rin branches: 1.144.6;
arm/bus_dma: Fix misleading comment:

- * For a virtually-indexed write-back cache, we need
- * to do the following things:
+ * For a write-back cache, we need to do the following things:

This comment was written long ago when we supported only virtually-
indexed caches for arm. But subsequent comments are valid also for
physically-indexed cache.
 1.143  03-Apr-2023  skrll Use PRI{u,x}{BUSSIZE,VADDR,PADDR} etc
 1.142  03-Apr-2023  skrll More KNF
 1.141  03-Apr-2023  skrll Some KNF
 1.140  03-Apr-2023  skrll spaces -> TAB
 1.139  25-Feb-2023  skrll Don't refer to physical_{start,end} here and just pass minimum (0UL)
address and maximum (~0UL) physical address. The values are passed
through to uvm_pglistalloc which knows about the RAM ranges.
 1.138  11-Oct-2022  andvar fix typos in log messages s/bus_dmamem_create/bus_dmamap_create/ and
s/bus_dmamem_load/bus_dmamap_load/.
Inspired by recent similar fixes in OpenBSD.
 1.137  26-Jul-2022  andvar s/functin/function/ in copy pasted comment.
 1.136  24-May-2022  andvar s/entrace/entrance/ in copy pasted comment.
 1.135  06-Feb-2022  mrg if we have to bounce a buffer, clear our anything already setup in
the map before trying again.

riastradh@ noticed that a map had two types of mbuf types, and this
seems to avoid it as seen on eqos(4) on quartz64.

add a couple more event counts for bus dma events.

ok @skrll @raistradh @jmcneill


XXX: pullup-8 and pullup-9 probably.
 1.134  20-Dec-2021  skrll _bus_dmatag_subregion is always EOPNOTSUPP for !_ARM32_NEED_BUS_DMA_BOUNCE
No need to check {min,max}_addr. Compiler did the right thing, but...
 1.133  30-Aug-2021  jmcneill Add storage for an iommu cookie in bus_dmamap_t and store untranslated CPU
address in bus_dma_segment_t.
 1.132  07-Feb-2021  skrll Check for programming errors in _bus_dmamap_load_buffer
 1.131  07-Feb-2021  skrll pretty print
 1.130  07-Feb-2021  skrll Consistently use %#x, etc
 1.129  22-Dec-2020  skrll Count bus_dmamap_sync events for coherent ranges and non-coherent ranges
 1.128  20-Dec-2020  jmcneill style fix
 1.127  19-Dec-2020  jmcneill No need to allocate bounce buffers if we have a single range that covers
all of physical memory.
 1.126  04-Dec-2020  skrll Use __func__ in panic calls (and misc style changes)
 1.125  04-Dec-2020  skrll Update stats in the coherent case. Reported by jmcneill.
 1.124  24-Oct-2020  skrll branches: 1.124.2;
Fix the logic around "parent is a subset and can be re-used" in
bus_dmatag_subregion. Thanks to mrg@ for spotting the problem.
 1.123  08-Sep-2020  skrll A few bus_dmatag_subregion fixes

- return EOPNOTSUPP if min_addr isn't less than max_addr
- fix the subset check to ensure that all the ranges in the parent tag are
within the {min,max}_addr range. If so we can just continue to use the
parent tag.
- when building the new ranges read the parent tag range rather than un-
initialised memory.
- remove the max_addr != 0xffffffff check - the overflow should be handled
by the unsigned arithmetic for arm32.
- add a KASSERT
- add comments
 1.122  20-Jun-2020  skrll KNF #includes

Remove some unnecessary ones while I'm here.
 1.121  14-Mar-2020  ad - Hide the details of SPCF_SHOULDYIELD and related behind a couple of small
functions: preempt_point() and preempt_needed().

- preempt(): if the LWP has exceeded its timeslice in kernel, strip it of
any priority boost gained earlier from blocking.
 1.120  22-Feb-2020  skrll Fix previous for _ARM32_NEED_BUS_DMA_BOUNCE
 1.119  21-Feb-2020  maya Make sure to convey a failure to the calling function

Likely fix for PR kern/54999, but that was a one-off panic.
 1.118  05-Nov-2019  jmcneill branches: 1.118.2;
Do not try to use direct map for prefetchable mappings
 1.117  05-Nov-2019  jmcneill bus_dmamem_map: honour BUS_DMA_PREFETCHABLE hint
 1.116  24-Aug-2019  jmcneill bus_dmamap_load_raw: support coherent mappings
bus_dmamap_sync: support syncing "raw" buffer types
 1.115  14-Jun-2019  skrll branches: 1.115.2;
Simplify the _ARM32_NEED_BUS_DMA_BOUNCE #ifdefs and rely on compiler
optimisation of the bouncing = false case.

Drain the write buf (aka DSB) in more cases

Catch all CPUs that support speculation. (thunderx isn't CPU_CORTEX)
 1.114  08-Jun-2019  skrll Fix comment
 1.113  14-Sep-2018  skrll Fix BUS_DMASYNC_PREREAD|BUS_DMASYNC_PREWRITE for aarch64.

The readonly mbuf optimisation is for the old arm pmap.
 1.112  03-Sep-2018  riastradh Rename min/max -> uimin/uimax for better honesty.

These functions are defined on unsigned int. The generic name
min/max should not silently truncate to 32 bits on 64-bit systems.
This is purely a name change -- no functional change intended.

HOWEVER! Some subsystems have

#define min(a, b) ((a) < (b) ? (a) : (b))
#define max(a, b) ((a) > (b) ? (a) : (b))

even though our standard name for that is MIN/MAX. Although these
may invite multiple evaluation bugs, these do _not_ cause integer
truncation.

To avoid `fixing' these cases, I first changed the name in libkern,
and then compile-tested every file where min/max occurred in order to
confirm that it failed -- and thus confirm that nothing shadowed
min/max -- before changing it.

I have left a handful of bootloaders that are too annoying to
compile-test, and some dead code:

cobalt ews4800mips hp300 hppa ia64 luna68k vax
acorn32/if_ie.c (not included in any kernels)
macppc/if_gm.c (superseded by gem(4))

It should be easy to fix the fallout once identified -- this way of
doing things fails safe, and the goal here, after all, is to _avoid_
silent integer truncations, not introduce them.

Maybe one day we can reintroduce min/max as type-generic things that
never silently truncate. But we should avoid doing that for a while,
so that existing code has a chance to be detected by the compiler for
conversion to uimin/uimax without changing the semantics until we can
properly audit it all. (Who knows, maybe in some cases integer
truncation is actually intended!)
 1.111  17-Jul-2018  christos use unintmax_t instead of u_quad_t.
 1.110  17-Jul-2018  martin Revert previous and cast to u_quad_t instead (t is for ptrdiff_t and off_t
does not match that on all arm)
 1.109  16-Jul-2018  christos fix incorrect printf format (off_t needs t)
 1.108  27-Apr-2018  maxv branches: 1.108.2;
M_CLUSTER -> M_EXT_CLUSTER
 1.107  01-Apr-2018  ryo Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)
 1.106  04-Mar-2018  skrll branches: 1.106.2;
Less ifdefs -> more readable.

NFC intended
 1.105  03-Mar-2018  skrll Style
 1.104  03-Mar-2018  skrll Rely on the cache operations to perform the necessary barriers.
 1.103  03-Mar-2018  skrll Wrap some long lines
 1.102  03-Mar-2018  skrll Remove unnecessary includes
 1.101  29-Dec-2017  skrll More KNF
 1.100  29-Dec-2017  skrll KNF
cvS: ----------------------------------------------------------------------
 1.99  03-Nov-2017  skrll Fix a comment
 1.98  05-Jan-2017  msaitoh branches: 1.98.8;
Update the dmamp argument only when the allocation succeeded.
 1.97  23-Dec-2016  cherry "Make NetBSD great again!"

Introduce uvm_hotplug(9) to the kernel.

Many thanks, in no particular order to:

TNF, for funding the project.

Chuck Silvers - for multiple API reviews and feedback.
Nick Hudson - for testing on multiple architectures and bugfix patches.
Everyone who helped with boot testing.

KeK (http://www.kek.org.in) for hosting the primary developers.
 1.96  05-Nov-2016  skrll Also return early from _bus_dmamap_sync if we're not bouncing and there
are no {pre,post} ops

Now the KASSERT doesn't fire on cats
 1.95  18-Jun-2016  skrll branches: 1.95.2;
Trailing whitespace.
 1.94  18-Oct-2015  jmcneill implement _bus_dmamap_load_raw, ok skrll@
 1.93  24-Aug-2015  matt Fix mbuf dmasyncing for extended MMU kernels.
 1.92  21-Aug-2015  matt Assert we aren't trying to sync a read-only mbuf as PREREAD.
 1.91  25-Feb-2015  joerg Improve inline asm around dsb/dmb/isb:
- always use volatile and mark them as memory barrier
- use the common version from locore.h in all places not included from
userland
 1.90  12-Feb-2015  joerg Kill redundant semicolons.
 1.89  09-Nov-2014  skrll branches: 1.89.2;
Post a dmb before invalidating the cache in the post-{read,write}
operations to ensure that any/all cachelines brought in via speculation
are really flushed.
 1.88  18-Oct-2014  snj src is too big these days to tolerate superfluous apostrophes. It's
"its", people!
 1.87  08-Sep-2014  matt With PMAP_NEED_ALLOC_POOLPAGE, before using pmap_map_poolpage make sure
the page comes from the same freelist that pmap_map_poolpage uses.
 1.86  10-Apr-2014  matt branches: 1.86.2;
Deal with non-4KB pages properly.
 1.85  06-Apr-2014  skrll Remove a comment that became largely redundant with the PMAP_NOCACHE
change.
 1.84  02-Apr-2014  matt branches: 1.84.2;
Deal with bus_dmamem_unmap of direct mapped KVA.
 1.83  24-Mar-2014  christos use cpu_{g,s}etmodel
 1.82  26-Feb-2014  skrll Fix off by one in _bus_dma_{paddr_inrange,_to_paddr} when checking the
dma range.
 1.81  26-Feb-2014  matt Use kmem instead of malloc.
Fix various corner cases with bounce buffers.
Use PMAP_NOCACHE instead of manipulating PTEs directly.
 1.80  18-Feb-2013  matt branches: 1.80.2;
Disable when using extended MMU support.
 1.79  16-Feb-2013  matt Use KASSERT* in preference to #ifdef DIAGNOSTIC / panic
 1.78  15-Feb-2013  matt Actually use the calculated uboundary.
 1.77  14-Feb-2013  matt Make sync counters only count when caches ops are performed.
 1.76  14-Feb-2013  matt Rework counters.
Get rid of a badly done goto.
Properly deal with boundary in bus_dmamem_alloc_range.
 1.75  14-Feb-2013  matt fix writebuf draining fix.
 1.74  13-Feb-2013  matt bounce buffer fixes + convert some kassert -> kassertmsg
 1.73  04-Feb-2013  macallan support BUS_DMA_PREFETCHABLE in bus_dmamem_mmap()
 1.72  28-Jan-2013  skrll Move and tweak an #if 0 printf
 1.71  27-Jan-2013  matt Get rid of _BUS_DMAMAP_MEM_XLATE
 1.70  27-Jan-2013  matt Add a _BUS_DMAMAP_NOALLOC which tells bus_dmamem_alloc to skip that
dmarange when allocating memory.
Add a second dmarange to bcm23xx obio to allow it to map coherently mapped
memory.
 1.69  27-Jan-2013  matt Make _BUS_DMAMAP_MEM_XLATE supplied by the caller.
 1.68  27-Jan-2013  matt Add a flag to make bus_dmamem_map use the bus/sys transation table when
mapping bus addresses. Make bcm2835 obio use it.
 1.67  16-Jan-2013  matt On Cortex, speculative loads can cache lines to be populated after then they've
been invalidated for a DMA read. So after the DMA read we have to reinvalidate
them again. We have to both invalidates since the former prevents dirty lines
overwriting just DMAed data.
 1.66  23-Oct-2012  skrll Correct inverted boolean logic for the coherent flag introduced in r1.61

Fixes slow nfs seen by msaitoh@
 1.65  22-Oct-2012  matt Only call cpu_{,s}dcache_inv_range if len > 0
 1.64  21-Oct-2012  matt Use right flag name for bus_dmamem_map (fixes usb on BCM53xx).
 1.63  19-Oct-2012  matt Make IS_BOUNCING a map flag and use it to simplify code and to avoid calling
the sync routines if (COHERENT|IS_BOUNCING) == COHERENT. (this eeks out a
little bit more performance).
 1.62  19-Oct-2012  matt Make bus_dmamem_map aware of coherent dma regions. Also, take advantage
of the use/existance of PMAP_MAP_POOLPAGE to avoid redundant mappings.
 1.61  17-Oct-2012  matt Add per-segment and per-ragne flag (to store _BUS_DMAMAP_COHERENT).
Use the per-range flag to set the per-segment flag.
This allows bus_dma to skip flushing for known coherent memory regions.
 1.60  06-Oct-2012  matt Fix a cut&paste error with supersections.
 1.59  22-Sep-2012  matt Add busaddr to paddr routine and use it.
cleanup the PREREAD sync case.
 1.58  18-Sep-2012  matt Add bounce buffer support for ARM bus_dma(9). Add macros to help initialize
bus_dma_tag structures.
 1.57  11-Sep-2012  matt branches: 1.57.2;
Rework dmasync routines so that secondary caches can be flushed as well
as the primary caches.
 1.56  02-Sep-2012  matt Supersections are on armv6 too.
 1.55  02-Sep-2012  matt Make bus_dma understand supersections.
 1.54  01-Jul-2011  dyoung branches: 1.54.2; 1.54.10;
#include <sys/bus.h> instead of <machine/bus.h>.
 1.53  06-Nov-2010  uebayasi Machine dependent code is considered as part of UVM. Include
internal API header.
 1.52  04-Jun-2008  ad branches: 1.52.16; 1.52.18; 1.52.20;
vm_page: put TAILQ_ENTRY into a union with LIST_ENTRY, so we can use both.
 1.51  28-Apr-2008  martin branches: 1.51.2;
Remove clause 3 and 4 from TNF licenses
 1.50  04-Mar-2007  christos branches: 1.50.42; 1.50.44; 1.50.46;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.
 1.49  10-Apr-2006  simonb branches: 1.49.14;
Unwrap a not-too-long-line.
 1.48  01-Mar-2006  yamt branches: 1.48.2; 1.48.4; 1.48.6;
merge yamt-uio_vmspace branch.

- use vmspace rather than proc or lwp where appropriate.
the latter is more natural to specify an address space.
(and less likely to be abused for random purposes.)
- fix a swdmover race.
 1.47  24-Dec-2005  perry branches: 1.47.2; 1.47.4; 1.47.6;
Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.
 1.46  11-Dec-2005  christos merge ktrace-lwp.
 1.45  24-Nov-2005  yamt bus_dmamem_map: honour BUS_DMA_NOWAIT. noted by Manuel Bouyer.
bus_space_map: always do NOWAIT allocation as it used to be before yamt-km.

we have too many copies!
 1.44  01-Apr-2005  yamt branches: 1.44.2; 1.44.8;
merge yamt-km branch.
- don't use managed mappings/backing objects for wired memory allocations.
save some resources like pv_entry. also fix (most of) PR/27030.
- simplify kernel memory management API.
- simplify pmap bootstrap of some ports.
- some related cleanups.
 1.43  09-Mar-2005  matt Add a dm_maxsegsz public member to bus_dmamap_t. This allows a user of the API
to select the maximum segment size for each bus_dmamap_load (up to the maxsegsz
supplied to bus_dmamap_create). dm_maxsegsz is reset to the value supplied to
bus_dmamap_create when the dmamap is unloaded.
 1.42  02-Jan-2005  chris branches: 1.42.2; 1.42.4;
Remove direct references to TAILQ internal structures.

No functional change, just tidying up code.
 1.41  28-Nov-2004  thorpej Sync with i386 bus_dma.
 1.40  06-Oct-2004  scw Apply patch provided by Hiroki Nakano in PR kern/27159 to fix
compilation with DEBUG_DMA defined.
 1.39  19-Jun-2004  thorpej Merge x86/bus_dma.c 1.11 by yamt:

Simplify bus_dma internal "load" functions by eliminating the "first"
variable, using seg == -1 instead.
 1.38  30-Oct-2003  scw In bus_dmamap_sync(), we can no longer rely on the data cache being
flushed on every context switch as an indicator that a mapping is
not resident in the cache.

Instead, used the per-pmap flag maintained by the cpu_switch/pmap code.
 1.37  29-Oct-2003  mycroft If something goes wrong, we want to return ENOMEM, not 0!
 1.36  26-Oct-2003  chris Fix up some unitialised variables.
 1.35  06-Sep-2003  rearnsha Make sure _ARM32_BUS_DMA_PRIVATE is defined before we pull in any
include files to avoid problems with the rats nest of dependencies.
 1.34  24-Jul-2003  briggs Clear ARM32_DMAMAP_COHERENT in _bus_dmamap_load_buffer for userspace mappings.
 1.33  15-Jul-2003  lukem __KERNEL_RCSID()
 1.32  29-Jun-2003  fvdl branches: 1.32.2;
Back out the lwp/ktrace changes. They contained a lot of colateral damage,
and need to be examined and discussed more.
 1.31  28-Jun-2003  darrenr Pass lwp pointers throughtout the kernel, as required, so that the lwpid can
be inserted into ktrace records. The general change has been to replace
"struct proc *" with "struct lwp *" in various function prototypes, pass
the lwp through and use l_proc to get the process pointer when needed.

Bump the kernel rev up to 1.6V
 1.30  21-May-2003  thorpej Remove #ifdefs supporting the old pmap, switching fully to the new.
 1.29  18-Apr-2003  scw Add the generic arm32 bits of the new pmap, contributed by Wasabi Systems.

Some features of the new pmap are:

- It allows L1 descriptor tables to be shared efficiently between
multiple processes. A typical "maxusers 32" kernel, where NPROC is set
to 532, requires 35 L1s. A "maxusers 2" kernel runs quite happily
with just 4 L1s. This completely solves the problem of running out
of contiguous physical memory for allocating new L1s at runtime on a
busy system.

- Much improved cache/TLB management "smarts". This change ripples
out to encompass the low-level context switch code, which is also
much smarter about when to flush the cache/TLB, and when not to.

- Faster allocation of L2 page tables and associated metadata thanks,
in part, to the pool_cache enhancements recently contributed to
NetBSD by Wasabi Systems.

- Faster VM space teardown due to accurate referenced tracking of L2
page tables.

- Better/faster cache-alias tracking.

The new pmap is enabled by adding options ARM32_PMAP_NEW to the kernel
config file, and making the necessary changes to the port-specific
initarm() function. Several ports have already been converted and will
be committed shortly.
 1.28  09-Apr-2003  thorpej Use cached physical addresses for mbufs and clusters to save having
to extract the physical address from the virtual.

On the ARM, also use the "read-only at MMU" indication to avoid a
redundant cache clean operation.

Other platforms should use these two as examples of how to use these
new pool/mbuf features to improve network performance. Note this requires
a platform to provide a working POOL_VTOPHYS().

Part 3 in a series of simple patches contributed by Wasabi Systems
to improve network performance.
 1.27  01-Apr-2003  thorpej Use PAGE_SIZE rather than NBPG.
 1.26  27-Mar-2003  mycroft Remove references to variables that aren't used here.
 1.25  03-Feb-2003  chris bus dma memory is allocated as M_DMAMAP so free it as M_DMAMAP, not DEVBUF.
 1.24  02-Feb-2003  wiz guarantee, not guarentee. Idea from miod@openbsd.
 1.23  17-Jan-2003  thorpej Merge the nathanw_sa branch.
 1.22  25-Sep-2002  thorpej Don't include <sys/map.h>.
 1.21  22-Aug-2002  thorpej * Add PTE_SYNC() and PTE_SYNC_RANGE() macros. These don't actually do
anything yet.
* Use PTE_SYNC() and PTE_SYNC_RANGE() in some obvious places, i.e.
where vtopte() is used.
 1.20  17-Aug-2002  briggs Do not trim 'offset' from 'len' in _bus_dmamap_sync_linear().
 1.19  17-Aug-2002  briggs Inline bus_dma_inrange() and bus_dmamap_sync_*().
 1.18  14-Aug-2002  thorpej When doing PREREAD sync operations, if the start and end addresses
of the range are aligned to a cacheline boundary, when do a dcache-inv
operation, rather than a dcache-wbinv operation.

XXX It could be a little smarter (align using wbinv, inv, then finish
up using wbinv), but even this simple change is good for a nearly 40%
improvement in my test case on XScale.
 1.17  14-Aug-2002  thorpej * Add an ARM32_DMAMAP_COHERENT flag to indicate that a loaded DMA
map contains "coherent" (non-cached in ARM-land) mappings.
* Set ARM32_DMAMAP_COHERENT in the map at the start of a load operation,
and clear it in _bus_dmamap_load_buffer() if we encounter any cacheable
mappings.
* In _bus_dmamap_sync(), if the map is marked COHERENT, skip any cache
flushing.
 1.16  14-Aug-2002  thorpej Whe making a mapping "coherent", clear *ALL* the cache bits, not
just L2_B and L2_C.
 1.15  31-Jul-2002  thorpej Overhaul how DMA ranges work in the ARM bus_dma implementation.

A new "arm32_dma_range" structure now describes a DMA window, with
a system address base, bus address base, and length. In addition to
providing info about which memory regions are legal for DMA, the new
structure provides address translation support, as well.

As before, if a tag does not list any ranges, then all addresses are
considered valid, and no DMA address translation is performed.

This allows us to remove a large chunk of code which was duplicated and
tweaked slightly (to do the address translation) from the stock ARM
bus_dma in the XScale IOP and ARM Integrator ports.

Test compiled on all ARM platforms, test booted on Intel IQ80321 and Shark.
 1.14  28-Jul-2002  thorpej Change the way that DMA map syncs are done. Instead of remembering
the virtual address for each DMA segment, just cache a pointer to the
original buffer/buftype used to load the DMA map, and use that. This
lets us shrink the bus_dma_segment_t down from 12 bytes to 8, and the
cache flushing is also more efficient.

Tested on an i80321 -- changes to others are mechanical.
 1.13  28-Jun-2002  briggs Remove complaint: bus_dmamap_destroy() called for map with valid
mappings bus_dma(9) states: "In the event that the DMA handle contains
a valid mapping, the mapping will be unloaded via the same mechanism
used by bus_dmamap_unload()." And some drivers do mean to skip the
unload step.
 1.12  02-Jun-2002  drochner move initialization of the "struct pglist" returned by uvm_pglistalloc()
from the calling code into uvm_pglistalloc() itself for consistency
and easier error handling
 1.11  10-Apr-2002  thorpej branches: 1.11.2; 1.11.4;
vm_offset_t -> vaddr_t,paddr_t
 1.10  05-Apr-2002  thorpej * Rewrite the 32-bit ARM pte.h based on the ARM architecture manual.
Significant cleanup, here, including better PTE bit names.
* Add XScale PTE extensions (ECC enable, write-allocate cache mode).
* Mechanical changes everywhere else to update for new pte.h. While
doing this, two bugs (as a result of typos) were fixed in

arm/arm32/bus_dma.c
evbarm/integrator/int_bus_dma.c
 1.9  24-Mar-2002  thorpej * arm_byte_to_page() -> arm_btop()
* arm_page_to_byte() -> arm_ptob()
 1.8  25-Jan-2002  thorpej Overhaul bus_dmamap_sync for the ARM:
* Track which process (XXX really, vmspace) owns the mapping. When
we sync the map, if the mapping doesn't belong to the kernel or to
the current process (XXX really, vmspace), then no cache fobbing
is necessary, since the cache is Wb-Inv'd on context switch (XXX need
to revisit this when we support FCSE).
* Be smarter about which cache operation we do when sync'ing the map:
- PREREAD -- Invalidate D$ (XXX right now, we actually do Wb-Inv)
- PREWRITE -- Write-back D$ (note, we do NOT invalidate here)
- PREREAD|PREWRITE -- Wb-Inv D$

More work is needed here. In particular, a version for CPUs
with write-through caches should be provided, to eliminate
the write-back steps (which are noops on such CPUs, but skipping
two branches would be nice).
 1.7  25-Jan-2002  thorpej ANSI'ify function decls.
 1.6  25-Jan-2002  thorpej Overhaul of the ARM cache code. This is mostly a simplification
pass. Rather than providing a whole slew of cache operations that
aren't ever used, distill them down to some useful primitives:

icache_sync_all Synchronize I-cache
icache_sync_range Synchronize I-cache range

dcache_wbinv_all Write-back and Invalidate D-cache
dcache_wbinv_range Write-back and Invalidate D-cache range
dcache_inv_range Invalidate D-cache range
dcache_wb_range Write-back D-cache range

idcache_wbinv_all Write-back and Invalidate D-cache,
Invalidate I-cache
idcache_wbinv_range Write-back and Invalidate D-cache,
Invalidate I-cache range

Note: This does not yet include an overhaul of the actual asm files
that implement the primitives. Instead, we've provided a safe default
for each CPU type, and the individual CPU types can now be optimized
one at a time.
 1.5  26-Nov-2001  thorpej Don't include <machine/psl.h> directly.
 1.4  23-Nov-2001  thorpej Use <arm/cpufunc.h>, not <machine/cpufunc.h>.
 1.3  12-Sep-2001  rearnsha branches: 1.3.4;
Fix warning errors for printing size_t types when compiling with
DEBUG_DMA (uses casts due to a.out and ELF having different base
types for size_t).
 1.2  10-Sep-2001  chris Update pmap_update to now take the updated pmap as an argument.
This will allow improvements to the pmaps so that they can more easily defer expensive operations, eg tlb/cache flush, til the last possible moment.

Currently this is a no-op on most platforms, so they should see no difference.

Reviewed by Jason.
 1.1  28-Jul-2001  chris branches: 1.1.2; 1.1.4;
Move the generic arm32 files into arm/arm32 from arm32/arm32, tested kernel builds on cats and riscpc.
 1.1.4.1  01-Oct-2001  fvdl Catch up with -current.
 1.1.2.8  10-Oct-2002  jdolecek sync kqueue with -current; this includes merge of gehenna-devsw branch,
merge of i386 MP branch, and part of autoconf rototil work
 1.1.2.7  06-Sep-2002  jdolecek sync kqueue branch with HEAD
 1.1.2.6  23-Jun-2002  jdolecek catch up with -current on kqueue branch
 1.1.2.5  11-Feb-2002  jdolecek Sync w/ -current.
 1.1.2.4  10-Jan-2002  thorpej Sync kqueue branch with -current.
 1.1.2.3  13-Sep-2001  thorpej Update the kqueue branch to HEAD.
 1.1.2.2  03-Aug-2001  lukem update to -current
 1.1.2.1  28-Jul-2001  lukem file bus_dma.c was added on branch kqueue on 2001-08-03 04:10:57 +0000
 1.3.4.12  18-Oct-2002  nathanw Catch up to -current.
 1.3.4.11  27-Aug-2002  thorpej Sync with -current.
 1.3.4.10  19-Aug-2002  thorpej Partial (ARM only) sync with trunk -- significant performance improvements
for XScale-based systems.
 1.3.4.9  01-Aug-2002  nathanw Catch up to -current.
 1.3.4.8  24-Jun-2002  nathanw Curproc->curlwp renaming.

Change uses of "curproc->l_proc" back to "curproc", which is more like the
original use. Bare uses of "curproc" are now "curlwp".

"curproc" is now #defined in proc.h as ((curlwp) ? (curlwp)->l_proc) : NULL)
so that it is always safe to reference curproc (*de*referencing curproc
is another story, but that's always been true).
 1.3.4.7  20-Jun-2002  nathanw Catch up to -current.
 1.3.4.6  17-Apr-2002  nathanw Catch up to -current.
 1.3.4.5  11-Apr-2002  thorpej Make this compile again.
XXX Need to re-think this.
 1.3.4.4  01-Apr-2002  nathanw Catch up to -current.
(CVS: It's not just a program. It's an adventure!)
 1.3.4.3  28-Feb-2002  nathanw Catch up to -current.
 1.3.4.2  08-Jan-2002  nathanw Catch up to -current.
 1.3.4.1  12-Sep-2001  nathanw file bus_dma.c was added on branch nathanw_sa on 2002-01-08 00:23:07 +0000
 1.11.4.2  07-Dec-2002  he Pull up revision 1.16 (requested by thorpej in ticket #678):
When making a mapping ``coherent'', clear *all* the cacche
bits, not just L2_B and L2_C.
 1.11.4.1  07-Dec-2002  he Pull up revisions 1.13-1.14 (requested by thorpej in ticket #568):
Remove warning from bus_dmamap_destroy().
Change the way DMA map syncs are done. Instead of remembering
the virtual address for each DMA segment, just cache a
pointer to the original buffer/buftype used to load the
DMA map, and use that. Shrinks bus_dma_segment_t from 12
to 8 bytes, and the cache flushing is also more efficient.
 1.11.2.3  30-Aug-2002  gehenna catch up with -current.
 1.11.2.2  16-Jul-2002  gehenna catch up with -current.
 1.11.2.1  14-Jul-2002  gehenna catch up with -current.
 1.32.2.10  11-Dec-2005  christos Sync with head.
 1.32.2.9  01-Apr-2005  skrll Sync with HEAD.
 1.32.2.8  19-Feb-2005  skrll Don't deref a NULL struct lwp *. This shouldn't happen, but...
 1.32.2.7  17-Jan-2005  skrll Sync with HEAD.
 1.32.2.6  18-Dec-2004  skrll Sync with HEAD.
 1.32.2.5  19-Oct-2004  skrll Sync with HEAD
 1.32.2.4  21-Sep-2004  skrll Fix the sync with head I botched.
 1.32.2.3  18-Sep-2004  skrll Sync with HEAD.
 1.32.2.2  03-Aug-2004  skrll Sync with HEAD
 1.32.2.1  02-Jul-2003  darrenr Apply the aborted ktrace-lwp changes to a specific branch. This is just for
others to review, I'm concerned that patch fuziness may have resulted in some
errant code being generated but I'll look at that later by comparing the diff
from the base to the branch with the file I attempt to apply to it. This will,
at the very least, put the changes in a better context for others to review
them and attempt to tinker with removing passing of 'struct lwp' through
the kernel.
 1.42.4.2  19-Mar-2005  yamt sync with head. xen and whitespace. xen part is not finished.
 1.42.4.1  28-Jan-2005  yamt convert arch/arm to new apis.
 1.42.2.1  29-Apr-2005  kent sync with -current
 1.44.8.1  29-Nov-2005  yamt sync with head.
 1.44.2.2  03-Sep-2007  yamt sync with head.
 1.44.2.1  21-Jun-2006  yamt sync with head.
 1.47.6.1  22-Apr-2006  simonb Sync with head.
 1.47.4.1  09-Sep-2006  rpaulo sync with head
 1.47.2.2  18-Feb-2006  yamt keep vmspace rather than proc in bus_dmamap_t.
 1.47.2.1  18-Feb-2006  yamt adapt bus_dma implementations.
 1.48.6.1  24-May-2006  tron Merge 2006-05-24 NetBSD-current into the "peter-altq" branch.
 1.48.4.1  19-Apr-2006  elad sync with head - hopefully this will work
 1.48.2.1  11-Apr-2006  yamt sync with head
 1.49.14.1  12-Mar-2007  rmind Sync with HEAD.
 1.50.46.2  04-May-2009  yamt sync with head.
 1.50.46.1  16-May-2008  yamt sync with head.
 1.50.44.2  17-Jun-2008  yamt sync with head.
 1.50.44.1  18-May-2008  yamt sync with head.
 1.50.42.2  05-Jun-2008  mjf Sync with HEAD.

Also fix build.
 1.50.42.1  02-Jun-2008  mjf Sync with HEAD.
 1.51.2.1  23-Jun-2008  wrstuden Sync w/ -current. 34 merge conflicts to follow.
 1.52.20.1  05-Mar-2011  rmind sync with head
 1.52.18.1  08-Jul-2010  uebayasi Fix build.
 1.52.16.1  15-Feb-2014  matt Merge armv7 support from HEAD, specifically support for the BCM5301X
and BCM56340 evbarm kernels.
 1.54.10.6  15-Feb-2013  matt Actually use the calculated uboundary
 1.54.10.5  14-Feb-2013  matt Sync with HEAD.
 1.54.10.4  14-Feb-2013  matt Sync with HEAD
 1.54.10.3  14-Feb-2013  matt sync with HEAD
 1.54.10.2  16-Jan-2013  matt Pullup from HEAD:
On Cortex, speculative loads can cache lines to be populated after then they've
been invalidated for a DMA read. So after the DMA read we have to reinvalidate
them again. We have to both invalidates since the former prevents dirty lines
overwriting just DMAed data.
 1.54.10.1  28-Nov-2012  matt Merge improved arm support (especially Cortex) from HEAD
including OMAP and BCM53xx support.
 1.54.2.3  22-May-2014  yamt sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")
 1.54.2.2  23-Jan-2013  yamt sync with head
 1.54.2.1  30-Oct-2012  yamt sync with head
 1.57.2.4  03-Dec-2017  jdolecek update from HEAD
 1.57.2.3  20-Aug-2014  tls Rebase to HEAD as of a few days ago.
 1.57.2.2  25-Feb-2013  tls resync with head
 1.57.2.1  20-Nov-2012  tls Resync to 2012-11-19 00:00:00 UTC
 1.80.2.1  18-May-2014  rmind sync with head
 1.84.2.1  10-Aug-2014  tls Rebase.
 1.86.2.3  27-May-2015  msaitoh Pull up following revision(s) (requested by skrll in ticket #805):
sys/arch/arm/include/arm32/pmap.h: revision 1.138
sys/arch/arm/arm/cpufunc.c: revision 1.151
sys/arch/arm/arm32/bus_dma.c: revision 1.90
sys/arch/arm/broadcom/bcm53xx_pax.c: revision 1.14
sys/arch/arm/arm32/bus_dma.c: revision 1.91
sys/arch/arm/samsung/exynos_space.c: revision 1.2
sys/arch/arm/arm32/db_machdep.c: revision 1.23
sys/arch/arm/allwinner/awin_space.c: revision 1.4
sys/arch/arm/include/rwlock.h: revision 1.9
sys/arch/arm/amlogic/amlogic_space.c: revision 1.2
sys/arch/arm/zynq/zynq_space.c: revision 1.2
sys/arch/arm/broadcom/bcm2835_space.c: revision 1.7
sys/arch/arm/arm32/pmap.c: revision 1.317
sys/arch/arm/include/locore.h: revision 1.19
sys/arch/arm/include/mutex.h: revision 1.20
sys/arch/arm/include/lock.h: revision 1.31
sys/arch/arm/include/lock.h: revision 1.32
sys/arch/arm/broadcom/bcmgen_space.c: revision 1.5
- Kill redundant semicolons.
- Indentation.
- Improve inline asm around dsb/dmb/isb:
- always use volatile and mark them as memory barrier
- use the common version from locore.h in all places not included from
userland
 1.86.2.2  14-Nov-2014  martin Pull up the following revisions, requested by skrll in ticket #222:
src/sys/arch/arm/arm32/locore.S 1.37
src/sys/arch/arm/arm32/bus_dma.c 1.87

Make the initial svcstack is doubleword aligned if EABI.

With PMAP_NEED_ALLOC_POOLPAGE, before using pmap_map_poolpage make sure
the page comes from the same freelist that pmap_map_poolpage uses.
 1.86.2.1  10-Nov-2014  martin Pull up following revision(s) (requested by skrll in ticket #209):
sys/arch/arm/pic/pic.c: revision 1.25
sys/arch/arm/arm/cpufunc_asm_armv7.S: revision 1.21
sys/arch/arm/arm32/pmap.c: revision 1.312
sys/arch/arm/arm32/bus_dma.c: revision 1.89
sys/arch/arm/arm32/pmap.c: revision 1.313
sys/arch/arm/arm32/pmap.c: revision 1.314
sys/arch/arm/arm32/pmap.c: revision 1.315
sys/arch/arm/arm32/pmap.c: revision 1.316
Include opt_multiprocessor.h
When allocing a l1page, if a page isn't available, use uvm_wait to wait
for one to become available. Should fix PR/49364.
Post a dmb before invalidating the cache in the post-{read,write}
operations to ensure that any/all cachelines brought in via speculation
are really flushed.
Ensure all memory operations are complete by before wfi. For example, the
cpu could have just been in uvm_pageidlezero.
In pmap_fault_fixup re-instate the TLB flush for the shared L1 case that
occurs for non-ARM_MMU_EXTENDED kernels.
This fixes rump/rumpkern/t_sp:stress_killer on rpi which is currently
non-ARM_MMU_EXTENDED
Remove an unnecessary flush that sneaked in as part of break-before-make
change.
Remove an XXXNH comment.
Update PTE_SYNC_CURRENT to add a dsb for armv7 - part of the
break-before-make fix.
 1.89.2.5  05-Feb-2017  skrll Sync with HEAD
 1.89.2.4  05-Dec-2016  skrll Sync with HEAD
 1.89.2.3  27-Dec-2015  skrll Sync with HEAD (as of 26th Dec)
 1.89.2.2  22-Sep-2015  skrll Sync with HEAD
 1.89.2.1  06-Apr-2015  skrll Sync with HEAD
 1.95.2.1  07-Jan-2017  pgoyette Sync with HEAD. (Note that most of these changes are simply $NetBSD$
tag issues.)
 1.98.8.1  25-Feb-2020  martin Pull up following revision(s) (requested by skrll in ticket #1510):

sys/arch/arm/arm32/bus_dma.c: revision 1.119
sys/arch/arm/arm32/bus_dma.c: revision 1.120

Make sure to convey a failure to the calling function
Likely fix for PR kern/54999, but that was a one-off panic.
-
Fix previous for non-_ARM32_NEED_BUS_DMA_BOUNCE builds
 1.106.2.5  30-Sep-2018  pgoyette Ssync with HEAD
 1.106.2.4  06-Sep-2018  pgoyette Sync with HEAD

Resolve a couple of conflicts (result of the uimin/uimax changes)
 1.106.2.3  28-Jul-2018  pgoyette Sync with HEAD
 1.106.2.2  02-May-2018  pgoyette Synch with HEAD
 1.106.2.1  07-Apr-2018  pgoyette Sync with HEAD. 77 conflicts resolved - all of them $NetBSD$
 1.108.2.3  13-Apr-2020  martin Mostly merge changes from HEAD upto 20200411
 1.108.2.2  08-Apr-2020  martin Merge changes from current as of 20200406
 1.108.2.1  10-Jun-2019  christos Sync with HEAD
 1.115.2.3  25-Feb-2020  martin Pull up following revision(s) (requested by skrll in ticket #721):

sys/arch/arm/arm32/bus_dma.c: revision 1.119
sys/arch/arm/arm32/bus_dma.c: revision 1.120

Make sure to convey a failure to the calling function
Likely fix for PR kern/54999, but that was a one-off panic.
-
Fix previous for non-_ARM32_NEED_BUS_DMA_BOUNCE builds
 1.115.2.2  06-Nov-2019  martin Pull up following revision(s) (requested by jmcneill in ticket #399):

sys/arch/arm/arm32/bus_dma.c: revision 1.117
sys/arch/arm/arm32/bus_dma.c: revision 1.118
sys/external/bsd/drm2/drm/drm_gem_cma_helper.c: revision 1.8

bus_dmamem_map: honour BUS_DMA_PREFETCHABLE hint

Map buffer objects with BUS_DMA_PREFETCHABLE.

Do not try to use direct map for prefetchable mappings
 1.115.2.1  24-Sep-2019  martin Pull up following revision(s) (requested by jmcneill in ticket #236):

sys/arch/arm/arm32/bus_dma.c: revision 1.116

bus_dmamap_load_raw: support coherent mappings
bus_dmamap_sync: support syncing "raw" buffer types
 1.118.2.1  29-Feb-2020  ad Sync with head.
 1.124.2.3  03-Apr-2021  thorpej Sync with HEAD.
 1.124.2.2  03-Jan-2021  thorpej Sync w/ HEAD.
 1.124.2.1  14-Dec-2020  thorpej Sync w/ HEAD.
 1.144.6.1  02-Aug-2025  perseant Sync with HEAD

RSS XML Feed