[AArch64] Generate zeroing forms of certain SVE2.2 instructions (3/11) #116829
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Member
| @llvm/pr-subscribers-backend-aarch64 Author: Momchil Velikov (momchil-velikov) ChangesSVE2.2 introduces instructions with predicated forms with zeroing of This patch adds support for emitting the zeroing forms of certain Patch is 59.65 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/116829.diff 6 Files Affected:
diff --git a/llvm/lib/Target/AArch64/AArch64InstrInfo.td b/llvm/lib/Target/AArch64/AArch64InstrInfo.td index e4ad27d4bcfc00..f7121373593fbd 100644 --- a/llvm/lib/Target/AArch64/AArch64InstrInfo.td +++ b/llvm/lib/Target/AArch64/AArch64InstrInfo.td @@ -381,6 +381,9 @@ def NoUseScalarIncVL : Predicate<"!Subtarget->useScalarIncVL()">; def UseSVEFPLD1R : Predicate<"!Subtarget->noSVEFPLD1R()">; +def UseUnaryUndefPseudos + : Predicate<"!(Subtarget->isSVEorStreamingSVEAvailable() && (Subtarget->hasSVE2p2() || Subtarget->hasSME2p2()))">; + def AArch64LocalRecover : SDNode<"ISD::LOCAL_RECOVER", SDTypeProfile<1, 1, [SDTCisSameAs<0, 1>, SDTCisInt<1>]>>; diff --git a/llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td b/llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td index 4f146b3ee59e9a..07320d040c0c10 100644 --- a/llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td +++ b/llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td @@ -664,6 +664,14 @@ let Predicates = [HasSVEorSME] in { defm FABS_ZPmZ : sve_int_un_pred_arit_bitwise_fp<0b100, "fabs", AArch64fabs_mt>; defm FNEG_ZPmZ : sve_int_un_pred_arit_bitwise_fp<0b101, "fneg", AArch64fneg_mt>; + let Predicates = [HasSVEorSME, UseUnaryUndefPseudos] in { + defm FABS_ZPmZ : sve_fp_un_pred_arit_hsd<AArch64fabs_mt>; + defm FNEG_ZPmZ : sve_fp_un_pred_arit_hsd<AArch64fneg_mt>; + + defm ABS_ZPmZ : sve_int_un_pred_arit_bhsd<AArch64abs_mt>; + defm NEG_ZPmZ : sve_int_un_pred_arit_bhsd<AArch64neg_mt>; + } + foreach VT = [nxv2bf16, nxv4bf16, nxv8bf16] in { // No dedicated instruction, so just clear the sign bit. def : Pat<(VT (fabs VT:$op)), @@ -4246,21 +4254,21 @@ defm TBLQ_ZZZ : sve2p1_tblq<"tblq", int_aarch64_sve_tblq>; //===----------------------------------------------------------------------===// let Predicates = [HasSVE2p2orSME2p2] in { // SVE Floating-point convert precision, zeroing predicate - defm FCVT_ZPzZ : sve_fp_z2op_p_zd_b_0<"fcvt">; + defm FCVT_ZPzZ : sve_fp_z2op_p_zd_b_0<"fcvt", "int_aarch64_sve_fcvt">; // SVE2p2 floating-point convert precision down (placing odd), zeroing predicate defm FCVTNT_ZPzZ : sve_fp_fcvtntz<"fcvtnt">; def FCVTXNT_ZPzZ_DtoS : sve_fp_fcvt2z<0b0010, "fcvtxnt", ZPR32, ZPR64>; // Placing even - def FCVTX_ZPzZ_DtoS : sve_fp_z2op_p_zd<0b0001010, "fcvtx", ZPR64, ZPR32>; + defm FCVTX_ZPzZ : sve_fp_z2op_p_zd<"fcvtx", int_aarch64_sve_fcvtx_f32f64>; // SVE2p2 floating-point convert precision up, zeroing predicate - defm FCVTLT_ZPzZ : sve_fp_fcvtltz<"fcvtlt">; + defm FCVTLT_ZPzZ : sve_fp_fcvtltz<"fcvtlt", "int_aarch64_sve_fcvtlt">; // SVE2p2 floating-point convert single-to-bf (placing odd), zeroing predicate def BFCVTNT_ZPzZ : sve_fp_fcvt2z<0b1010, "bfcvtnt", ZPR16, ZPR32>; // Placing corresponding - def BFCVT_ZPzZ_StoH : sve_fp_z2op_p_zd<0b1001010, "bfcvt", ZPR32, ZPR16>; + defm BFCVT_ZPzZ_StoH : sve_fp_z2op_p_zd_bfcvt<0b1001010, "bfcvt", int_aarch64_sve_fcvt_bf16f32_v2>; // Floating-point convert to integer, zeroing predicate defm FCVTZS_ZPzZ : sve_fp_z2op_p_zd_d<0b0, "fcvtzs">; @@ -4310,16 +4318,16 @@ let Predicates = [HasSVE2p2orSME2p2] in { defm NOT_ZPzZ : sve_int_un_pred_arit_bitwise_z<0b110, "not">; // floating point - defm FABS_ZPzZ : sve_int_un_pred_arit_bitwise_fp_z<0b100, "fabs">; - defm FNEG_ZPzZ : sve_int_un_pred_arit_bitwise_fp_z<0b101, "fneg">; + defm FABS_ZPzZ : sve_int_un_pred_arit_bitwise_fp_z<0b100, "fabs", AArch64fabs_mt>; + defm FNEG_ZPzZ : sve_int_un_pred_arit_bitwise_fp_z<0b101, "fneg", AArch64fneg_mt>; // SVE2p2 integer unary arithmetic, zeroing predicate defm SXTB_ZPzZ : sve_int_un_pred_arit_h_z<0b000, "sxtb">; defm UXTB_ZPzZ : sve_int_un_pred_arit_h_z<0b001, "uxtb">; defm SXTH_ZPzZ : sve_int_un_pred_arit_w_z<0b010, "sxth">; defm UXTH_ZPzZ : sve_int_un_pred_arit_w_z<0b011, "uxth">; - defm ABS_ZPzZ : sve_int_un_pred_arit_z< 0b110, "abs">; - defm NEG_ZPzZ : sve_int_un_pred_arit_z< 0b111, "neg">; + defm ABS_ZPzZ : sve_int_un_pred_arit_z< 0b110, "abs", AArch64abs_mt>; + defm NEG_ZPzZ : sve_int_un_pred_arit_z< 0b111, "neg", AArch64neg_mt>; def SXTW_ZPzZ_D : sve_int_un_pred_arit_z<0b11, 0b1000, "sxtw", ZPR64>; def UXTW_ZPzZ_D : sve_int_un_pred_arit_z<0b11, 0b1010, "uxtw", ZPR64>; diff --git a/llvm/lib/Target/AArch64/SVEInstrFormats.td b/llvm/lib/Target/AArch64/SVEInstrFormats.td index 6de6aed3b2a816..067b0724b46139 100644 --- a/llvm/lib/Target/AArch64/SVEInstrFormats.td +++ b/llvm/lib/Target/AArch64/SVEInstrFormats.td @@ -482,6 +482,8 @@ let Predicates = [HasSVEorSME] in { //===----------------------------------------------------------------------===// // SVE pattern match helpers. //===----------------------------------------------------------------------===// +def SVEDup0 : ComplexPattern<vAny, 0, "SelectDupZero", []>; +def SVEDup0Undef : ComplexPattern<vAny, 0, "SelectDupZeroOrUndef", []>; class SVE_1_Op_Pat<ValueType vtd, SDPatternOperator op, ValueType vt1, Instruction inst> @@ -502,6 +504,11 @@ multiclass SVE_1_Op_PassthruUndef_Pat<ValueType vtd, SDPatternOperator op, Value (inst $Op3, $Op1, $Op2)>; } +class SVE_1_Op_PassthruUndefZero_Pat<ValueType vtd, SDPatternOperator op, ValueType pg, + ValueType vts, Instruction inst> + : Pat<(vtd (op pg:$Op1, vts:$Op2, (vtd (SVEDup0Undef)))), + (inst $Op1, $Op2)>; + // Used to match FP_ROUND_MERGE_PASSTHRU, which has an additional flag for the // type of rounding. This is matched by timm0_1 in pattern below and ignored. class SVE_1_Op_Passthru_Round_Pat<ValueType vtd, SDPatternOperator op, ValueType pg, @@ -517,8 +524,6 @@ multiclass SVE_1_Op_PassthruUndef_Round_Pat<ValueType vtd, SDPatternOperator op, (inst $Op3, $Op1, $Op2)>; } -def SVEDup0 : ComplexPattern<vAny, 0, "SelectDupZero", []>; - class SVE_1_Op_PassthruZero_Pat<ValueType vtd, SDPatternOperator op, ValueType vt1, ValueType vt2, Instruction inst> : Pat<(vtd (op (vtd (SVEDup0)), vt1:$Op1, vt2:$Op2)), @@ -571,6 +576,11 @@ multiclass SVE_3_Op_Undef_Pat<ValueType vtd, SDPatternOperator op, ValueType vt1 (inst $Op1, $Op2, $Op3)>; } +class SVE_3_Op_UndefZero_Pat<ValueType vtd, SDPatternOperator op, ValueType vt1, + ValueType vt2, ValueType vt3, Instruction inst> + : Pat<(vtd (op (vt1 (SVEDup0Undef)), vt2:$Op1, vt3:$Op2)), + (inst $Op1, $Op2)>; + class SVE_4_Op_Pat<ValueType vtd, SDPatternOperator op, ValueType vt1, ValueType vt2, ValueType vt3, ValueType vt4, Instruction inst> @@ -606,8 +616,6 @@ class SVE_4_Op_Imm_Pat<ValueType vtd, SDPatternOperator op, ValueType vt1, : Pat<(vtd (op vt1:$Op1, vt2:$Op2, vt3:$Op3, (vt4 ImmTy:$Op4))), (inst $Op1, $Op2, $Op3, ImmTy:$Op4)>; -def SVEDup0Undef : ComplexPattern<vAny, 0, "SelectDupZeroOrUndef", []>; - let AddedComplexity = 1 in { class SVE_3_Op_Pat_SelZero<ValueType vtd, SDPatternOperator op, ValueType vt1, ValueType vt2, ValueType vt3, Instruction inst> @@ -2850,9 +2858,12 @@ multiclass sve_fp_fcvtntz<string asm> { def _DtoS : sve_fp_fcvt2z<0b1110, asm, ZPR32, ZPR64>; } -multiclass sve_fp_fcvtltz<string asm> { +multiclass sve_fp_fcvtltz<string asm, string op> { def _HtoS : sve_fp_fcvt2z<0b1001, asm, ZPR32, ZPR16>; def _StoD : sve_fp_fcvt2z<0b1111, asm, ZPR64, ZPR32>; + + def : SVE_3_Op_UndefZero_Pat<nxv4f32, !cast<SDPatternOperator>(op # _f32f16), nxv4f32, nxv4i1, nxv8f16, !cast<Instruction>(NAME # _HtoS)>; + def : SVE_3_Op_UndefZero_Pat<nxv2f64, !cast<SDPatternOperator>(op # _f64f32), nxv2f64, nxv2i1, nxv4f32, !cast<Instruction>(NAME # _StoD)>; } //===----------------------------------------------------------------------===// @@ -3259,6 +3270,12 @@ class sve_fp_z2op_p_zd<bits<7> opc,string asm, RegisterOperand i_zprtype, let mayRaiseFPException = 1; } +multiclass sve_fp_z2op_p_zd<string asm, SDPatternOperator op> { + def _DtoS : sve_fp_z2op_p_zd<0b0001010, asm, ZPR64, ZPR32>; + + def : SVE_3_Op_UndefZero_Pat<nxv4f32, op, nxv4f32, nxv2i1, nxv2f64, !cast<Instruction>(NAME # _DtoS)>; +} + multiclass sve_fp_z2op_p_zd_hsd<bits<5> opc, string asm> { def _H : sve_fp_z2op_p_zd<{ 0b01, opc }, asm, ZPR16, ZPR16>; def _S : sve_fp_z2op_p_zd<{ 0b10, opc }, asm, ZPR32, ZPR32>; @@ -3270,6 +3287,12 @@ multiclass sve_fp_z2op_p_zd_frint<bits<2> opc, string asm> { def _D : sve_fp_z2op_p_zd<{ 0b0010, opc{1}, 1, opc{0} }, asm, ZPR64, ZPR64>; } +multiclass sve_fp_z2op_p_zd_bfcvt<bits<7> opc, string asm, SDPatternOperator op> { + def _StoH : sve_fp_z2op_p_zd<opc, asm, ZPR32, ZPR16>; + + def : SVE_3_Op_UndefZero_Pat<nxv8bf16, op, nxv8bf16, nxv4i1, nxv4f32, !cast<Instruction>(NAME # _StoH)>; +} + multiclass sve_fp_z2op_p_zd_d<bit U, string asm> { def _HtoH : sve_fp_z2op_p_zd<{ 0b011101, U }, asm, ZPR16, ZPR16>; def _HtoS : sve_fp_z2op_p_zd<{ 0b011110, U }, asm, ZPR16, ZPR32>; @@ -3296,13 +3319,20 @@ multiclass sve_fp_z2op_p_zd_d_flogb<string asm> { def _D : sve_fp_z2op_p_zd<0b0011011, asm, ZPR64, ZPR64>; } -multiclass sve_fp_z2op_p_zd_b_0<string asm> { +multiclass sve_fp_z2op_p_zd_b_0<string asm, string op> { def _StoH : sve_fp_z2op_p_zd<0b1001000, asm, ZPR32, ZPR16>; def _HtoS : sve_fp_z2op_p_zd<0b1001001, asm, ZPR16, ZPR32>; def _DtoH : sve_fp_z2op_p_zd<0b1101000, asm, ZPR64, ZPR16>; def _HtoD : sve_fp_z2op_p_zd<0b1101001, asm, ZPR16, ZPR64>; def _DtoS : sve_fp_z2op_p_zd<0b1101010, asm, ZPR64, ZPR32>; def _StoD : sve_fp_z2op_p_zd<0b1101011, asm, ZPR32, ZPR64>; + + def : SVE_3_Op_UndefZero_Pat<nxv8f16, !cast<SDPatternOperator>(op # _f16f32), nxv8f16, nxv4i1, nxv4f32, !cast<Instruction>(NAME # _StoH)>; + def : SVE_3_Op_UndefZero_Pat<nxv8f16, !cast<SDPatternOperator>(op # _f16f64), nxv8f16, nxv2i1, nxv2f64, !cast<Instruction>(NAME # _DtoH)>; + def : SVE_3_Op_UndefZero_Pat<nxv4f32, !cast<SDPatternOperator>(op # _f32f64), nxv4f32, nxv2i1, nxv2f64, !cast<Instruction>(NAME # _DtoS)>; + def : SVE_3_Op_UndefZero_Pat<nxv4f32, !cast<SDPatternOperator>(op # _f32f16), nxv4f32, nxv4i1, nxv8f16, !cast<Instruction>(NAME # _HtoS)>; + def : SVE_3_Op_UndefZero_Pat<nxv2f64, !cast<SDPatternOperator>(op # _f64f16), nxv2f64, nxv2i1, nxv8f16, !cast<Instruction>(NAME # _HtoD)>; + def : SVE_3_Op_UndefZero_Pat<nxv2f64, !cast<SDPatternOperator>(op # _f64f32), nxv2f64, nxv2i1, nxv4f32, !cast<Instruction>(NAME # _StoD)>; } //===----------------------------------------------------------------------===// @@ -4820,23 +4850,18 @@ multiclass sve_int_un_pred_arit<bits<3> opc, string asm, def : SVE_1_Op_Passthru_Pat<nxv8i16, op, nxv8i1, nxv8i16, !cast<Instruction>(NAME # _H)>; def : SVE_1_Op_Passthru_Pat<nxv4i32, op, nxv4i1, nxv4i32, !cast<Instruction>(NAME # _S)>; def : SVE_1_Op_Passthru_Pat<nxv2i64, op, nxv2i1, nxv2i64, !cast<Instruction>(NAME # _D)>; - - def _B_UNDEF : PredOneOpPassthruPseudo<NAME # _B, ZPR8>; - def _H_UNDEF : PredOneOpPassthruPseudo<NAME # _H, ZPR16>; - def _S_UNDEF : PredOneOpPassthruPseudo<NAME # _S, ZPR32>; - def _D_UNDEF : PredOneOpPassthruPseudo<NAME # _D, ZPR64>; - - defm : SVE_1_Op_PassthruUndef_Pat<nxv16i8, op, nxv16i1, nxv16i8, !cast<Pseudo>(NAME # _B_UNDEF)>; - defm : SVE_1_Op_PassthruUndef_Pat<nxv8i16, op, nxv8i1, nxv8i16, !cast<Pseudo>(NAME # _H_UNDEF)>; - defm : SVE_1_Op_PassthruUndef_Pat<nxv4i32, op, nxv4i1, nxv4i32, !cast<Pseudo>(NAME # _S_UNDEF)>; - defm : SVE_1_Op_PassthruUndef_Pat<nxv2i64, op, nxv2i1, nxv2i64, !cast<Pseudo>(NAME # _D_UNDEF)>; } -multiclass sve_int_un_pred_arit_z<bits<3> opc, string asm> { +multiclass sve_int_un_pred_arit_z<bits<3> opc, string asm, SDPatternOperator op> { def _B : sve_int_un_pred_arit_z<0b00, { opc, 0b0 }, asm, ZPR8>; def _H : sve_int_un_pred_arit_z<0b01, { opc, 0b0 }, asm, ZPR16>; def _S : sve_int_un_pred_arit_z<0b10, { opc, 0b0 }, asm, ZPR32>; def _D : sve_int_un_pred_arit_z<0b11, { opc, 0b0 }, asm, ZPR64>; + + def : SVE_1_Op_PassthruUndefZero_Pat<nxv16i8, op, nxv16i1, nxv16i8, !cast<Instruction>(NAME # _B)>; + def : SVE_1_Op_PassthruUndefZero_Pat<nxv8i16, op, nxv8i1, nxv8i16, !cast<Instruction>(NAME # _H)>; + def : SVE_1_Op_PassthruUndefZero_Pat<nxv4i32, op, nxv4i1, nxv4i32, !cast<Instruction>(NAME # _S)>; + def : SVE_1_Op_PassthruUndefZero_Pat<nxv2i64, op, nxv2i1, nxv2i64, !cast<Instruction>(NAME # _D)>; } multiclass sve_int_un_pred_arit_h<bits<3> opc, string asm, @@ -4950,7 +4975,22 @@ multiclass sve_int_un_pred_arit_bitwise_fp<bits<3> opc, string asm, def : SVE_1_Op_Passthru_Pat<nxv4f32, op, nxv4i1, nxv4f32, !cast<Instruction>(NAME # _S)>; def : SVE_1_Op_Passthru_Pat<nxv2f32, op, nxv2i1, nxv2f32, !cast<Instruction>(NAME # _S)>; def : SVE_1_Op_Passthru_Pat<nxv2f64, op, nxv2i1, nxv2f64, !cast<Instruction>(NAME # _D)>; +} + +multiclass sve_int_un_pred_arit_bitwise_fp_z<bits<3> opc, string asm, SDPatternOperator op> { + def _H : sve_int_un_pred_arit_z<0b01, { opc, 0b1 }, asm, ZPR16>; + def _S : sve_int_un_pred_arit_z<0b10, { opc, 0b1 }, asm, ZPR32>; + def _D : sve_int_un_pred_arit_z<0b11, { opc, 0b1 }, asm, ZPR64>; + def : SVE_1_Op_PassthruUndefZero_Pat<nxv8f16, op, nxv8i1, nxv8f16, !cast<Instruction>(NAME # _H)>; + def : SVE_1_Op_PassthruUndefZero_Pat<nxv4f16, op, nxv4i1, nxv4f16, !cast<Instruction>(NAME # _H)>; + def : SVE_1_Op_PassthruUndefZero_Pat<nxv2f16, op, nxv2i1, nxv2f16, !cast<Instruction>(NAME # _H)>; + def : SVE_1_Op_PassthruUndefZero_Pat<nxv4f32, op, nxv4i1, nxv4f32, !cast<Instruction>(NAME # _S)>; + def : SVE_1_Op_PassthruUndefZero_Pat<nxv2f32, op, nxv2i1, nxv2f32, !cast<Instruction>(NAME # _S)>; + def : SVE_1_Op_PassthruUndefZero_Pat<nxv2f64, op, nxv2i1, nxv2f64, !cast<Instruction>(NAME # _D)>; +} + +multiclass sve_fp_un_pred_arit_hsd<SDPatternOperator op> { def _H_UNDEF : PredOneOpPassthruPseudo<NAME # _H, ZPR16>; def _S_UNDEF : PredOneOpPassthruPseudo<NAME # _S, ZPR32>; def _D_UNDEF : PredOneOpPassthruPseudo<NAME # _D, ZPR64>; @@ -4963,10 +5003,16 @@ multiclass sve_int_un_pred_arit_bitwise_fp<bits<3> opc, string asm, defm : SVE_1_Op_PassthruUndef_Pat<nxv2f64, op, nxv2i1, nxv2f64, !cast<Pseudo>(NAME # _D_UNDEF)>; } -multiclass sve_int_un_pred_arit_bitwise_fp_z<bits<3> opc, string asm> { - def _H : sve_int_un_pred_arit_z<0b01, { opc, 0b1 }, asm, ZPR16>; - def _S : sve_int_un_pred_arit_z<0b10, { opc, 0b1 }, asm, ZPR32>; - def _D : sve_int_un_pred_arit_z<0b11, { opc, 0b1 }, asm, ZPR64>; +multiclass sve_int_un_pred_arit_bhsd<SDPatternOperator op> { + def _B_UNDEF : PredOneOpPassthruPseudo<NAME # _B, ZPR8>; + def _H_UNDEF : PredOneOpPassthruPseudo<NAME # _H, ZPR16>; + def _S_UNDEF : PredOneOpPassthruPseudo<NAME # _S, ZPR32>; + def _D_UNDEF : PredOneOpPassthruPseudo<NAME # _D, ZPR64>; + + defm : SVE_1_Op_PassthruUndef_Pat<nxv16i8, op, nxv16i1, nxv16i8, !cast<Pseudo>(NAME # _B_UNDEF)>; + defm : SVE_1_Op_PassthruUndef_Pat<nxv8i16, op, nxv8i1, nxv8i16, !cast<Pseudo>(NAME # _H_UNDEF)>; + defm : SVE_1_Op_PassthruUndef_Pat<nxv4i32, op, nxv4i1, nxv4i32, !cast<Pseudo>(NAME # _S_UNDEF)>; + defm : SVE_1_Op_PassthruUndef_Pat<nxv2i64, op, nxv2i1, nxv2i64, !cast<Pseudo>(NAME # _D_UNDEF)>; } //===----------------------------------------------------------------------===// diff --git a/llvm/test/CodeGen/AArch64/zeroing-forms-abs-neg.ll b/llvm/test/CodeGen/AArch64/zeroing-forms-abs-neg.ll new file mode 100644 index 00000000000000..1caee994220f05 --- /dev/null +++ b/llvm/test/CodeGen/AArch64/zeroing-forms-abs-neg.ll @@ -0,0 +1,666 @@ +; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5 +; RUN: llc -mattr=+sve < %s | FileCheck %s +; RUN: llc -mattr=+sve2p2 < %s | FileCheck %s -check-prefix CHECK-2p2 + +; RUN: llc -mattr=+sme -force-streaming < %s | FileCheck %s +; RUN: llc -mattr=+sme2p2 -force-streaming < %s | FileCheck %s -check-prefix CHECK-2p2 + +target triple = "aarch64-linux" + +define <vscale x 2 x double> @test_svabs_f64_x_1(<vscale x 2 x i1> %pg, <vscale x 2 x double> %x) { +; CHECK-LABEL: test_svabs_f64_x_1: +; CHECK: // %bb.0: // %entry +; CHECK-NEXT: fabs z0.d, p0/m, z0.d +; CHECK-NEXT: ret +; +; CHECK-2p2-LABEL: test_svabs_f64_x_1: +; CHECK-2p2: // %bb.0: // %entry +; CHECK-2p2-NEXT: fabs z0.d, p0/z, z0.d +; CHECK-2p2-NEXT: ret +entry: + %0 = tail call <vscale x 2 x double> @llvm.aarch64.sve.fabs.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x i1> %pg, <vscale x 2 x double> %x) + ret <vscale x 2 x double> %0 +} + +define <vscale x 2 x double> @test_svabs_f64_x_2(<vscale x 2 x i1> %pg, double %z0, <vscale x 2 x double> %x) { +; CHECK-LABEL: test_svabs_f64_x_2: +; CHECK: // %bb.0: // %entry +; CHECK-NEXT: movprfx z0, z1 +; CHECK-NEXT: fabs z0.d, p0/m, z1.d +; CHECK-NEXT: ret +; +; CHECK-2p2-LABEL: test_svabs_f64_x_2: +; CHECK-2p2: // %bb.0: // %entry +; CHECK-2p2-NEXT: fabs z0.d, p0/z, z1.d +; CHECK-2p2-NEXT: ret +entry: + %0 = tail call <vscale x 2 x double> @llvm.aarch64.sve.fabs.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x i1> %pg, <vscale x 2 x double> %x) + ret <vscale x 2 x double> %0 +} + +define <vscale x 2 x double> @test_svabs_f64_z(<vscale x 2 x i1> %pg, double %z0, <vscale x 2 x double> %x) { +; CHECK-LABEL: test_svabs_f64_z: +; CHECK: // %bb.0: // %entry +; CHECK-NEXT: mov z0.d, #0 // =0x0 +; CHECK-NEXT: fabs z0.d, p0/m, z1.d +; CHECK-NEXT: ret +; +; CHECK-2p2-LABEL: test_svabs_f64_z: +; CHECK-2p2: // %bb.0: // %entry +; CHECK-2p2-NEXT: fabs z0.d, p0/z, z1.d +; CHECK-2p2-NEXT: ret +entry: + %0 = tail call <vscale x 2 x double> @llvm.aarch64.sve.fabs.nxv2f64(<vscale x 2 x double> zeroinitializer, <vscale x 2 x i1> %pg, <vscale x 2 x double> %x) + ret <vscale x 2 x double> %0 +} + +define <vscale x 4 x float> @test_svabs_f32_x_1(<vscale x 4 x i1> %pg, <vscale x 4 x float> %x) { +; CHECK-LABEL: test_svabs_f32_x_1: +; CHECK: // %bb.0: // %entry +; CHECK-NEXT: fabs z0.s, p0/m, z0.s +; CHECK-NEXT: ret +; +; CHECK-2p2-LABEL: test_svabs_f32_x_1: +; CHECK-2p2: // %bb.0: // %entry +; CHECK-2p2-NEXT: fabs z0.s, p0/z, z0.s +; CHECK-2p2-NEXT: ret +entry: + %0 = tail call <vscale x 4 x float> @llvm.aarch64.sve.fabs.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x i1> %pg, <vscale x 4 x float> %x) + ret <vscale x 4 x float> %0 +} + +define <vscale x 4 x float> @test_svabs_f32_x_2(<vscale x 4 x i1> %pg, double %z0, <vscale x 4 x float> %x) { +; CHECK-LABEL: test_svabs_f32_x_2: +; CHECK: // %bb.0: // %entry +; CHECK-NEXT: movprfx z0, z1 +; CHECK-NEXT: fabs z0.s, p0/m, z1.s +; CHECK-NEXT: ret +; +; CHECK-2p2-LABEL: test_svabs_f32_x_2: +; CHECK-2p2: // %bb.0: // %entry +; CHECK-2p2-NEXT: fabs z0.s, p0/z, z1.s +; CHECK-2p2-NEXT: ret +entry: + %0 = tail call <vscale x 4 x float> @llvm.aarch64.sve.fabs.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x i1> %pg, <vscale x 4 x float> %x) + ret <vscale x 4 x float> %0 +} + +define <vscale x 4 x float> @test_svabs_f32_z(<vscale x 4 x i1> %pg, double %z0, <vscale x 4 x float> %x) { +; CHECK-LABEL: test_svabs_f32_z: +; CHECK: // %bb.0: // %entry +; CHECK-NEXT: mov z0.s, #0 // =0x0 +; CHECK-NEXT: fabs z0.s, p0/m, z1.s +; CHECK-NEXT: ret +; +; CHECK-2p2-LABEL: test_svabs_f32_z: +; CHECK-2p2: // %bb.0: // %entry +; CHECK-2p2-NEXT: fabs z0.s, p0/z, z1.s +; CHECK-2p2-NEXT: ret +entry: + %0 = tail call <vscale x 4 x float> @llvm.aarch64.sve.fabs.nxv4f32(<vscale x 4 x float> zeroinitializer, <vscale x 4 x i1> %pg, <vscale x 4 x float> %x) + ret <vscale x 4 x float> %0 +} + +define <vscale x 8 x half> @test_svabs_f16_x_1(<vscale x 8 x i1> %pg, <vscale x 8 x half> %x) { +; CHECK-LABEL: test_svabs_f16_x_1: +; CHECK: // %bb.0: // %entry +; CHECK-NEXT: fabs z0.h, p0/m, z0.h +; CHECK-NEXT: ret +; +; CHECK-2p2-LABEL: test_svabs_f16_x_1: +; CHECK-2p2: // %bb.0: // %entry +; CHECK-2p2-NEXT: fabs z0.h, p0/z, z0.h +; CHECK-2p2-NEXT: ret +entry: + %0 = tail call <vscale x 8 x half> @llvm.aarch64.sve.fabs.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x i1> %pg, <vscale x 8 x half> %x) + ret <vscale x 8 x half> %0 +} + +define <vscale x 8 x half> @test_svabs_f16_x_2(<vscale x 8 x i1> %pg, double %z0, <vscale x 8 x half> %x) { +; CHECK-LABEL: test_svabs_f16_x_2: +; CHECK: // %bb.0: // %entry +; CHECK-NEXT: m... [truncated] |
CarolineConcatto approved these changes Nov 27, 2024
momchil-velikov force-pushed the zeroing-03.1-fcvtlt-fcvtx branch from ca7e545 to 9b0d3b2 Compare 
Collaborator
| LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/131/builds/11494 Here is the relevant piece of the build log for the reference |
Collaborator
| LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/53/builds/8800 Here is the relevant piece of the build log for the reference |
Collaborator
This bot is a bit strange because it triggers on changes to LLVM and Flang but doesn't list the |
momchil-velikov added a commit to momchil-velikov/llvm-project that referenced this pull request Dec 20, 2024
… all-true predicate When the predicate of a destructive operation is known to be all-true, for example fabs z0.s, p0/m, z1.s then the entire output register is written and we can use a zeroing (instead of a merging) form of the instruction, for example fabs z0.s, p0/z, z1.s thus eliminate the dependency on the input-output destination register without the need to insert a `movprfx`. This patch complements (and in the case of 2b3266c, fixes a regression) the following: 7f4414b [AArch64] Generate zeroing forms of certain SVE2.2 instructions (4/11) (llvm#116830) 2474cf7 [AArch64] Generate zeroing forms of certain SVE2.2 instructions (3/11) (llvm#116829) 6f285d3 [AArch64] Generate zeroing forms of certain SVE2.2 instructions (2/11) (llvm#116828) 2b3266c [AArch64] Generate zeroing forms of certain SVE2.2 instructions (1/11) (llvm#116259)
momchil-velikov added a commit that referenced this pull request Dec 24, 2024
… all-true predicate (#120595) When the predicate of a destructive operation is known to be all-true, for example fabs z0.s, p0/m, z1.s then the entire output register is written and we can use a zeroing (instead of a merging) form of the instruction, for example fabs z0.s, p0/z, z1.s thus eliminate the dependency on the input-output destination register without the need to insert a `movprfx`. This patch complements (and in the case of 2b3266c, fixes a regression) the following: 7f4414b [AArch64] Generate zeroing forms of certain SVE2.2 instructions (4/11) (#116830) 2474cf7 [AArch64] Generate zeroing forms of certain SVE2.2 instructions (3/11) (#116829) 6f285d3 [AArch64] Generate zeroing forms of certain SVE2.2 instructions (2/11) (#116828) 2b3266c [AArch64] Generate zeroing forms of certain SVE2.2 instructions (1/11) (#116259)
github-actions bot pushed a commit to arm/arm-toolchain that referenced this pull request Jan 10, 2025
…ons with an all-true predicate (#120595) When the predicate of a destructive operation is known to be all-true, for example fabs z0.s, p0/m, z1.s then the entire output register is written and we can use a zeroing (instead of a merging) form of the instruction, for example fabs z0.s, p0/z, z1.s thus eliminate the dependency on the input-output destination register without the need to insert a `movprfx`. This patch complements (and in the case of llvm/llvm-project@2b3266c, fixes a regression) the following: llvm/llvm-project@7f4414b [AArch64] Generate zeroing forms of certain SVE2.2 instructions (4/11) (llvm/llvm-project#116830) llvm/llvm-project@2474cf7 [AArch64] Generate zeroing forms of certain SVE2.2 instructions (3/11) (llvm/llvm-project#116829) llvm/llvm-project@6f285d3 [AArch64] Generate zeroing forms of certain SVE2.2 instructions (2/11) (llvm/llvm-project#116828) llvm/llvm-project@2b3266c [AArch64] Generate zeroing forms of certain SVE2.2 instructions (1/11) (llvm/llvm-project#116259)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
SVE2.2 introduces instructions with predicated forms with zeroing of
the inactive lanes. This allows in some cases to save a
movprfxora
movinstruction when emitting code for_xor_zvariants ofintrinsics.
This patch adds support for emitting the zeroing forms of certain
FCVTLT, andFCVTXinstructions.