Repository navigation
[SPU LLVM] Optimize AVX-512 XFloat conversion - #19244
Conversation
Whatcookie
left a comment
Whatcookie
left a comment
There was a problem hiding this comment.
Maybe a comment could be added clarifying that these paths make use of vpternlog, as they otherwise use no intrinsics. But otherwise lgtm
| const auto smax = splat<u64[4]>(0x47ffffffe0000000).eval(m_ir); | ||
| const auto smin = splat<u64[4]>(0x3810000000000000).eval(m_ir); | ||
| const auto a = m_ir->CreateAnd(d, 0x7fffffffe0000000); | ||
| const auto n = m_ir->CreateICmpUGE(a, smin); |
There was a problem hiding this comment.
Hmm, since the sign bit is cleared, you can use integer comparson on non avx512 path as well. Weird that it got missed.
There was a problem hiding this comment.
what do you have in mind? on AVX2 the current path lowers directly to vcmpnltpd + vminpd, while the integer would need two vpcmpgtq plus a blend
There was a problem hiding this comment.
(V)PCMPGTQ - 64bit signed integer comparison, is available for AVX cpus. Is it generated by llvm? If so, you can use this for all AVX cpus as well by using a signed comparison.
There was a problem hiding this comment.
ok just finished testing this. on avx1 the integer compare gets split into two xmm vpcmpgtqs plus extract/insert, so it takes the path from 7 to 11 instructions
avx2 gets one ymm compare but it’s not really a consistent win. on haswell/broadwell it’s 5 cycle latency vs 3 for vcmppd, and on skylake it has slightly better latency but it’s p5 only with half the throughput
so i think keeping the current avx/avx2 path makes more sense here
Uses vptestmd and vpternlogq to simplify xfloat expansion:
Before:
666: 62 a2 7d 28 35 c7 vpmovzxdq ymm16, xmm23
66c: 62 e1 fd 20 db f1 vpandq ymm22, ymm16, ymm1
672: 62 e1 fd 20 db c2 vpandq ymm16, ymm16, ymm2
678: 62 b1 cd 20 73 f6 20 vpsllq ymm22, ymm22, 0x20
67f: 62 b1 c5 20 73 f0 1d vpsllq ymm23, ymm16, 0x1d
686: 62 f2 fd 20 37 cc vpcmpgtq k1, ymm16, ymm4
68c: 62 a1 c5 20 eb fe vporq ymm23, ymm23, ymm22
692: 62 e1 c5 21 d4 f3 vpaddq ymm22{k1}, ymm23, ymm3
After:
4ad: c4 c2 7d 35 c7 vpmovzxdq ymm0, xmm15
4b2: 62 f2 05 08 27 cc vptestmd k1, xmm15, xmm4
4b8: 62 f1 e5 20 73 f0 20 vpsllq ymm19, ymm0, 0x20
4bf: c5 fd 73 f0 1d vpsllq ymm0, ymm0, 0x1d
4c4: 62 e1 e5 20 db d9 vpandq ymm19, ymm19, ymm1
4ca: 62 f3 e5 20 25 c2 ec vpternlogq ymm0, ymm19, ymm2, 0xec
4d1: 62 e1 fd 29 d4 db vpaddq ymm19{k1}, ymm0, ymm3
It also replaces the fp xfloat clamp with integer compare/min operations:
Before:
605: 62 c2 e5 20 a8 f7 vfmadd213pd ymm22, ymm19, ymm15
60b: 62 c1 cd 20 54 c1 vandpd ymm16, ymm22, ymm9
611: 62 71 cd 20 54 fe vandpd ymm15, ymm22, ymm6
617: 62 d1 fd 20 c2 c8 05 vcmpnltpd k1, ymm16, ymm8
61e: 62 81 fd 20 5d c2 vminpd ymm16, ymm16, ymm26
624: 62 51 fd a1 56 ff vorpd ymm15{k1}{z}, ymm16, ymm15
After:
5c8: 62 22 e5 20 a8 c8 vfmadd213pd ymm25, ymm19, ymm16
5ce: 62 a1 b5 20 54 c6 vandpd ymm16, ymm25, ymm22
5d4: 62 f2 fd 20 37 cf vpcmpgtq k1, ymm16, ymm7
5da: 62 82 fd 20 3b c4 vpminuq ymm16, ymm16, ymm28
5e0: 62 e3 b5 a1 25 c1 f8 vpternlogq ymm16{k1}{z}, ymm25, ymm1, 0xf8