Lines Matching refs:sve
281 /* The number of 64-bit elements in an SVE vector. */
718 /* Generic costs for SVE vector operations. */
762 &generic_sve_vector_cost, /* sve */
832 &a64fx_sve_vector_cost, /* sve */
870 nullptr, /* sve */
909 nullptr, /* sve */
946 nullptr, /* sve */
984 nullptr, /* sve */
1021 nullptr, /* sve */
1059 nullptr, /* sve */
1097 nullptr, /* sve */
1134 nullptr, /* sve */
1172 nullptr, /* sve */
2105 &neoversev1_sve_vector_cost, /* sve */
2191 (6 cycles) plus an insertion (2 cycles). Assume a 64-bit SVE gather
2197 and an SVE 32-bit gather, but cost an SVE 32-bit gather as 1 vector
2242 &neoverse512tvb_sve_vector_cost, /* sve */
2355 (8 cycles) plus an insertion (2 cycles). Assume a 64-bit SVE gather
2361 and an SVE 32-bit gather, but cost an SVE 32-bit gather as 1 vector
2430 &neoversen2_sve_vector_cost, /* sve */
2544 (8 cycles) plus an insertion (2 cycles). Assume a 64-bit SVE gather
2550 and an SVE 32-bit gather, but cost an SVE 32-bit gather as 1 vector
2619 &neoversev2_sve_vector_cost, /* sve */
2772 SVE function type",
2795 { "SVE type", 3, 3, false, true, false, true, NULL, NULL },
2796 { "SVE sizeless type", 0, 0, false, true, false, true, NULL, NULL },
2922 /* The preferred condition codes for SVE conditions. */
3244 /* Return the descriptor of the SVE PCS. */
3328 /* Report when we try to do something that requires SVE when SVE is disabled.
3340 error ("this operation requires the SVE ISA extension");
3341 inform (input_location, "you can enable SVE using the command-line"
3441 /* Return an estimate for the number of quadwords in an SVE vector. This is
3442 equivalent to the number of Advanced SIMD vectors in an SVE vector. */
3449 /* Return true if MODE is an SVE predicate mode. */
3468 vector has fewer significant bytes than a full SVE vector. */
3487 /* Partial SVE QI vectors. */
3491 /* Partial SVE HI vectors. */
3494 /* Partial SVE SI vector. */
3496 /* Partial SVE HF vectors. */
3499 /* Partial SVE BF vectors. */
3502 /* Partial SVE SF vector. */
3516 /* x2 SVE vectors. */
3525 /* x3 SVE vectors. */
3534 /* x4 SVE vectors. */
3660 /* Return true if MODE is any form of SVE mode, including predicates,
3668 /* Return true if MODE is an SVE data vector mode; either a single vector
3677 SVE mode MODE, which has vector flags VEC_FLAGS. */
3731 /* Return the SVE vector mode that has NUNITS elements of mode INNER_MODE. */
3776 /* MODE is some form of SVE vector mode. For data modes, return the number
3792 /* Return the SVE predicate mode to use for elements that have
3812 /* Return the SVE predicate mode that should be used to control
3813 SVE mode MODE. */
3834 /* Return the integer element mode associated with SVE mode MODE. */
3858 /* Return the integer vector mode associated with SVE mode MODE.
3878 /* If we're operating on SVE vectors, try to return an SVE mode. */
3887 /* Try to find a full or partial SVE mode with exactly
3926 the else value doesn't matter, since that exactly matches the SVE
4027 SVE vector or predicate registers. */
4053 SVE vector or predicate registers. */
4137 register on non-SVE targets). */
4203 /* The natural size for SVE data modes is one SVE data vector,
4206 /* ??? For now, only do this for variable-width SVE registers.
5123 /* See if there is an svpattern that encodes an SVE predicate of mode
5238 the corresponding SVE predicate mode. Use TARGET for the result
5265 /* Return the assembly string for an SVE prefetch operation with
5421 /* Return true if we can add X using a single SVE INC or DEC instruction. */
5432 /* Return the asm string for adding SVE INC/DEC immediate OFFSET to
5491 /* Return true if X is a valid immediate for an SVE vector INC or DEC
5528 /* Return true if X is a valid immediate for an SVE vector INC or DEC
5537 /* Return the asm template for an SVE vector INC or DEC instruction.
5875 SVE vector register, over and above the minimum size of 128 bits.
6073 /* Duplicate 128-bit Advanced SIMD vector SRC so that it fills an SVE
6134 /* SRC is an SVE CONST_VECTOR that contains N "foreground" values followed
6185 SVE data mode and isn't a legitimate constant. Use TARGET for the
6229 register would be different from its layout in an SVE register,
6264 memory lane N of SVE vector SRC corresponds to architectural
6266 memory lane 0 of SVE vector SRC is in the lsb of VQ_SRC (viewed
6270 N + I * INDEX of the SVE register. */
6513 constant in BUILDER into an SVE predicate register. Return the register
6559 /* Return an SVE predicate register that contains the VNx16BImode
6584 the changes of using SVE DUPM or an Advanced SIMD byte mask. */
6821 /* Emit an SVE predicated move from SRC to DEST. PRED is a predicate
6836 /* Expand a pre-RA SVE data move from SRC to DEST in which at least one
6842 See the comment at the head of aarch64-sve.md for details about the
6863 /* Called only on big-endian targets. See whether an SVE vector move
6865 least one operand is a subreg of an SVE vector that has wider or
6928 /* The optimization handles two single SVE REGs with different element
6961 /* Return the SVE REV[BHW] unspec for reversing quantites of mode MODE
7012 passed in SVE registers. */
7069 " the SVE ISA extension", arg.type);
7071 /* Variadic SVE types are passed by reference. Normal non-variadic
7115 /* Likewise pure scalable types for SVE vector and predicate registers. */
7141 /* Generic vectors that map to full SVE modes with -msve-vector-bits=N
7194 /* Vector types can acquire a partial SVE mode using things like
7198 the associated integer mode, just like they did before SVE
7238 in SVE registers. */
7377 mode in ARG might be the result of replacing partial SVE modes with
7413 /* The PCS says that it is invalid to pass an SVE value to an
7420 error ("SVE type %qT cannot be passed to an unprototyped function",
7440 /* Generic vectors that map to full SVE modes with -msve-vector-bits=N
7445 /* Vector types can acquire a partial SVE mode using things like
7449 the associated integer mode, just like they did before SVE
7556 /* If an argument with an SVE mode needs to be shifted up to the
7681 fatal_error (input_location, "%qE requires the SVE ISA extension",
7685 " the SVE ISA extension", fntype);
7748 /* Don't use the SVE part of the register for __builtin_apply and
7749 __builtin_return. The SVE registers aren't used by the normal PCS,
7751 for SVE types are fundamentally incompatible with the
7860 sorry ("stack probes for SVE frames");
8041 SVE. This emits probes from BASE to BASE - ADJUSTMENT based on a guard size
8205 /* Big-endian SVE frames need a spare predicate register in order
8282 /* If we need to save any SVE vector registers, add them next. */
8470 /* Frame in which all saves are SVE saves:
8473 save SVE registers relative to SP
8481 /* Frame with large area below the saved registers, or with SVE saves,
8487 [save SVE registers relative to SP]
8502 [save SVE registers relative to SP]
8784 /* The caller is going to use ST1D or LD1D to save or restore an SVE
9132 /* If the spare predicate register used by big-endian SVE code
9464 /* Handle the SVE non-constant case first. */
9469 fprintf (dump_file, "Stack clash SVE prologue: ");
9674 | SVE vector registers |
9676 | SVE predicate registers |
9717 - r11: Used by stack clash protection when SVE is enabled, and also
9749 /* Fold the SVE allocation into the initial allocation.
9940 with an SVE component, since we then need both temporary registers
9951 and SVE callee save allocations in the prologue); and
9998 so that we can use P4 as a temporary for big-endian SVE frames. */
10779 /* For SVE, only accept [Rn], [Rn, #offset, MUL VL] and [Rn, Rm, LSL #shift].
10780 The latter is not valid for SVE predicates, and that's rejected through
10891 /* Make "m" use the LD1 offset range for SVE data modes, so
11125 /* A general SVE offset is A * VQ + B. Remove the A component from
11716 /* Handle the SVE single-bit immediates specially, since they have a
12578 /* Use aarch64_sve_reload_mem for SVE memory reloads that cannot use
12579 LDR and STR. See the comment at the head of aarch64-sve.md for
15316 If the operation is an SVE one, PTRUE is a suitable all-true
15588 or SVE version of a vector loop, using the scheme defined by the
15637 SVE code. */
15644 This is only ever different from 1 for SVE. It allows us to consider
15645 what would happen on a 256-bit SVE target even when the -mtune
15646 parameters say that the “likely” SVE length is 128 bits. */
15682 /* If the structure describes SVE code and we have associated issue
15688 return m_issue_info->sve;
15813 /* This loop uses an average operation that is not supported by SVE, but is
15834 - If M_VEC_FLAGS & VEC_ANY_SVE is nonzero then we're costing SVE code. */
15862 /* On some CPUs, SVE and Advanced SIMD provide the same theoretical vector
15863 throughput, such as 4x128 Advanced SIMD vs. 2x256 SVE. In those
15867 than length-agnostic SVE, since the SVE loop would execute an unknown
15932 && costs->sve != NULL)
15933 return costs->sve;
15942 if ((flags & VEC_ANY_SVE) && costs->sve)
15943 return costs->sve;
15965 vector throughput for SVE and Advanced SIMD. */
15971 vectorized for SVE. */
15986 would be known at compile time but the number of SVE iterations
16026 /* Detect whether we're vectorizing for SVE and should apply the unrolling
16030 /* Record the issue information for any SVE WHILE instructions that the
16201 /* Scalar and SVE code can tie the result to any FMLA input (or none,
16202 although that requires a MOVPRFX for SVE). However, Advanced SIMD
16220 /* We are considering implementing STMT_INFO using SVE. If STMT_INFO is an
16221 in-loop reduction that SVE supports directly, return its latency in cycles,
16267 SVE implementation. */
16275 sve_costs = aarch64_tune_params.vec_costs->sve;
16277 /* If the caller is asking for the SVE latency, check for forms of reduction
16278 that only SVE can handle directly. */
16337 sve_costs = aarch64_tune_params.vec_costs->sve;
16351 /* Detect SVE gather loads, which are costed as a single scalar_load
16421 operate on vector type VECTYPE. Adjust the cost as necessary for SVE
16445 but there are no equivalent instructions for SVE. This means that
16446 (all other things being equal) 128-bit SVE needs twice as many load
16450 so it is too simplistic to say that one SVE load or store replaces
16463 An alternative would be to double the cost of any SVE loads and stores
16469 Here we go for a more conservative version: double the costs of SVE
16616 /* Count the predicate operations needed by an SVE comparison. */
16802 /* Do any SVE-specific adjustments to the cost. */
16824 /* If we're applying the SVE vs. Advanced SIMD unrolling heuristic,
16828 as one iteration of the SVE loop. */
16862 /* Return true if (a) we're applying the Advanced SIMD vs. SVE unrolling
16888 /* Subroutine of adjust_body_cost for handling SVE. Use ISSUE_INFO to work out
16889 how fast the SVE code can be issued and compare it to the equivalent value
16912 quickly as the predicate parts of the SVE loop, make the SVE loop
16945 bool sve = m_vec_flags & VEC_ANY_SVE;
16947 an averaging operation that we do not support with SVE and we might use a
16953 if (!sve && !TARGET_SVE2 && m_has_avg)
17047 dump_printf_loc (MSG_NOTE, vect_location, "SVE issue estimate:\n");
17056 doubling in SVE vector length. */
17249 /* If the issue rate of SVE code is limited by predicate operations
17265 " SVE loop is predicate-limited\n");
17518 Accept the valid SVE vector widths allowed by
17809 /* If using Advanced SIMD only for autovectorization disable SVE vector costs
18299 /* 128-bit SVE and Advanced SIMD modes use different register layouts
20182 /* Assign BLKmode to anything that contains multiple SVE predicates.
20482 processing SVE types. */
20484 /* Leave later code to report an error if SVE is disabled. */
20491 /* 64-bit and 128-bit vectors should only acquire an SVE mode if
20646 /* Return the full-width SVE vector mode for element mode MODE, if one
20739 /* Compare an SVE mode SVE_M and an Advanced SIMD mode ASIMD_M
20740 and return whether the SVE mode should be preferred over the
20760 /* If the CPU information does not have an SVE width registered use the
20761 generic poly_int comparison that prefers SVE. If a preference is
20772 /* Preferring SVE means picking it first unless the Advanced SIMD mode
20776 /* Conversely, preferring Advanced SIMD means picking SVE only if SVE
20781 /* In the default case prefer Advanced SIMD over SVE in case of a tie. */
20842 /* Try using N-byte SVE modes only after trying N-byte Advanced SIMD mode.
20849 - If an SVE main loop with N bytes ends up being cheaper than an
20851 the Advanced SIMD version with the SVE one.
20854 than an SVE main loop with N bytes then by default we'll try to
20855 use the SVE loop to vectorize the epilogue instead. */
20876 /* Consider enabling VECT_COMPARE_COSTS for SVE, both so that we
20877 can compare SVE against Advanced SIMD and so that we can compare
20878 multiple SVE vectorization approaches against each other. There's
21048 /* Return true if BASE_OR_STEP is a valid immediate operand for an SVE INDEX
21058 /* Return true if X is a valid immediate for the SVE ADD and SUB instructions
21078 /* Return true if X is a valid immediate for the SVE SQADD and SQSUB
21095 /* Return true if X is a valid immediate operand for an SVE logical
21109 /* Return true if X is a valid immediate for the SVE DUP and CPY
21125 /* Return true if X is a valid immediate operand for an SVE CMP instruction.
21138 /* Return true if X is a valid immediate operand for an SVE FADD or FSUB
21163 /* Return true if X is a valid immediate operand for an SVE FMUL
21281 /* Return true if replicating VAL64 gives a valid immediate for an SVE MOV
21357 /* Return true if X is a valid SVE predicate. If INFO is nonnull, use
21458 /* If all elements in an SVE vector have the same value, we have a free
21466 If not all elements in an SVE vector have the same value, we need the
21784 /* Return true if OP is a valid MEM operand for an SVE LD1R instruction. */
21799 /* Return true if OP is a valid MEM operand for an SVE LD1R{Q,O} instruction
21820 /* Return true if OP is a valid MEM operand for an SVE LD1RQ instruction. */
21828 /* Return true if OP is a valid MEM operand for an SVE LD1RO instruction for
21836 /* Return true if OP is a valid MEM operand for an SVE LDFF1 instruction. */
21853 /* Return true if OP is a valid MEM operand for an SVE LDNF1 instruction. */
21865 /* Return true if OP is a valid MEM operand for an SVE LDR instruction.
21878 /* Return true if OP is a valid address for an SVE PRF[BHWD] instruction,
21944 /* This is only used (and only meaningful) for Advanced SIMD, not SVE. */
21949 alignment of a vector to 128 bits. SVE predicates have an alignment of
21956 direct way we have of identifying real SVE predicate types. */
22613 /* Prepare for an integer SVE multiply-add or multiply-subtract pattern;
23508 into an SVE register. */
23750 /* Expand an SVE vec_perm with the given operands. */
23762 are effectively ignored. SVE TBL instead produces 0 for any
23834 /* We don't need a big-endian lane correction for SVE; see the comment
23835 at the head of aarch64-sve.md for details. */
23925 /* We don't need a big-endian lane correction for SVE; see the comment
23926 at the head of aarch64-sve.md for details. */
23966 /* We don't need a big-endian lane correction for SVE; see the comment
23967 at the head of aarch64-sve.md for details. */
24002 We don't need a big-endian lane correction for SVE; see the comment
24003 at the head of aarch64-sve.md for details. */
24181 /* Try to implement D using an SVE TBL instruction. */
24205 /* Try to implement D using SVE SEL instruction. */
24442 /* Expand an SVE integer comparison using the SVE equivalent of:
24501 /* Emit the SVE equivalent of:
24522 /* Emit the SVE equivalent of:
24540 /* Expand an SVE floating-point comparison using the SVE equivalent of:
24638 /* Expand an SVE vcond pattern with operands OPS. DATA_MODE is the mode
26745 /* Don't allow changes between partial SVE modes and other modes.
26746 The contents of partial SVE modes are distributed evenly across
26751 /* Similarly reject changes between partial SVE modes that have
26765 /* Don't allow changes between SVE modes and other modes that might
26767 divide into 128-bit quantities while SVE modes divide into
26777 /* Don't allow changes between SVE data modes and non-SVE modes.
26778 See the comment at the head of aarch64-sve.md for details. */
26801 /* SVE values are not normally live across a call, so it should be
26831 For cores with a known SVE width all three estimates are the same.
26832 For generic SVE tuning we want to distinguish the maximum estimate from
26835 conservative behavior of auto-vectorizing with SVE when it is a win
26836 even for 128-bit SVE.
26837 When SVE width information is available VAL.coeffs[1] is multiplied by
26918 /* For now, SVE simdclones won't produce illegal simdlen, So only check
26996 /* For now, SVE simdclones won't produce illegal simdlen, So only check
27060 if (!check_attr ("SVE type"))
27062 if (!check_attr ("SVE sizeless type"))
27140 return N_("cannot combine GNU and SVE vectors in a binary operation");
27559 /* Test SVE arithmetic folding. */
Indexes created Wed Jul 15 00:26:21 UTC 2026