Adding peephole optimization to Clang

Mon, 11 May 2020 10:00:00 +0000

“Adding peephole optimization to GCC” article covers it for GCC and I decided to cover Clang. Clang is basically a front-end on top of LLVM. It parses C/C++ files into an AST and converts them to an intermediate representation - LLVM IR.

High-level architecture for C++ AOT scenario.

opt is a console application, it accepts LLVM IR and optimizes it using a loop of analysis and transformation phases (produces optimized LLVM IR as an output). Then llc (a console app as well) emits machine code/assembly. opt is basically a PassManager with some generic (actually, C/C++-friendly) order of optimization passes. Other languages or JIT-compilers should use their own order of passes. Thus, when we introduce a new optimization in LLVM we improve many programming languages at once! E.g. Rust, Swift, C, C++, C# (Mono and Burst) - all of them use LLVM as a primary back-end. I assume you’re familiar with some LLVM IR basics such as SSA-form, and If you’re not I highly recommend watching/reading these:

Let’s optimize something!

Although it might sound complicated, some of the compilers’ internals are just quite simple programs you can easily build from sources and improve. For instance, local optimizations (aka peepholes) where you don’t really have to care about surroundings, control flow, etc. I decided to make a step-by-step tutorial on how to create your own optimizations for C/C++. A good source of ideas for peepholes are Math formulas from google/books. I decided to check this one:

It’s a pretty basic math formula, let’s check if modern C++ compilers handle it well via godbolt (godbolt.org/z/Nvxs4o):

Ouch, so GCC and MSVC managed to recognize it and optimize to pow(x, a+b) but Clang did not. Let’s fix that! We need to get the LLVM IR clang emits for it first. It can be easily obtained via -emit-llvm argument:

Most of the math.h functions have their built-in LLVM intrinsics. E.g. see my PR for Mono where I tell LLVM our Math.XY() methods should be LLVM-intrinsics in order to get such optimizations for free. We, obviously, need somehow transform the IR like this:

See godbolt.org/z/ZGG2dG.

How to build LLVM

LLVM is now hosted at github (moved recently) and uses cmake as a build system so it takes just a few commands to download it and build. We can ask cmake to generate project files for your favorite IDE, for instance VS on Windows and XCode on macOS.

Windows:

1
2
3
4
git clone git@github.com:llvm/llvm-project.git
mkdir myllvm
cd myllvm
cmake -G "Visual Studio 16" ..\llvm-project\llvm

macOS:

1
2
3
4
git clone git@github.com:llvm/llvm-project.git
mkdir myllvm
cd myllvm
cmake -G Xcode ../llvm-project/llvm

Now open LLVM.xcodeproj or LLVM.sln in myllvm folder and build ALL_BUILD target. It might take a while…

Zhdun or "waiter" (in a non-catering sense)

Once it’s done we need to prepare a test and it’s simple - just copy that IR from godbolt to a file, e.g. test.ll:

1
2
3
4
5
6
7
8
define double @Test(double %x, double %a, double %b) {
  %powa = tail call fast double @llvm.pow.f64(double %x, double %a)
  %powb = tail call fast double @llvm.pow.f64(double %x, double %b)
  %res = fmul fast double %powb, %powa
  ret double %res
}

declare double @llvm.pow.f64(double, double)

I mentioned opt console app several times and actually you can find it in the solution. We are going to use it as an entry-point app where we pass a path to our test and ask InstCombine to run our peepholes. You need to find that opt project and modify its properties like this:

and for XCode (see Edit Scheme):

-O1 means run all the optimizations from O1 set (including -instcombine pass). -S means we prefer a human-readable format.

And that’s it! Click “Run/Debug” and you are able to debug LLVM internals: hit breakpoints, view locals! It’s time to write some code. We want to add a peephole optimization and it means we need InstCombine. Also, we need to optimize multiplication (FMul, F stands for floating-point) so our entry point is visitFMul in InstCombineMulDivRem.cpp. Let me paste the optimization there and explain:

Here we visit a binary node FMul with Op0 and Op1 sub-nodes and we just need to match the pattern we want via match. In our case we want both to be pow intrinsics. It looks super simple and in those InstCombine*.cpp files you can find plenty of similar peephole optimizations. A bit more complicated case is my PR to LLVM: reviews.llvm.org/D79369. Now when we press the Run button we can see the resulting IR:

Voila! IR is optimized! Now let me test it for C# (.NET 5.0 Mono-LLVM-JIT)

1
2
3
4
using System;

double PowAb(double x, double a, double b) =>
	Math.Pow(x, a) * Math.Pow(x, b);

Codegen (MONO_VERBOSE_METHOD=PowAb):

1
2
3
4
Program_PowAb__double_double_double:
   vaddsd %xmm2,%xmm1,%xmm1
   movabs $0x7f9f58caeb00,%rax
   jmpq   *%rax  ; pow

It just works! My commit with these changes can be found here (it also handles FDiv and takes care about Fast Math and OneUse checks).

For reference, here is how the optimization is defined in GCC (in a sort of a DSL):

1
2
3
4
 /* Simplify pow(x,y) * pow(x,z) -> pow(x,y+z). */
 (simplify
  (mult (POW:s @0 @1) (POW:s @0 @2))
   (POW @0 (plus @1 @2)))

Let me know (e.g. on twitter) if you want me to do the same step-by-step tutorial for RyuJIT and C#.

Smart LLVM #1: Optimizing range checks

Sat, 03 Aug 2019 10:00:00 +0000

Sometimes I explore LLVM sources and play with godbolt.org in order to find some interesting optimizations (not only the peephole ones) so I think I’ll post some here in my blog from time to time. Also, if an optimization is simple enough I try to implement it in RuyJIT.
And today I am going to share a nice LLVM trick to optimize some common range checks.
So, let’s say we have a function that checks if a char belongs to a list of reserved chars:
(I actually copy-pasted it from CoreFX)

1
2
3
4
5
6
7
8
9
10
11
12
bool IsReservedCharacter(char character) // uint16_t
{
    return character == ';'
        || character == '/'
        || character == ':'
        || character == '@'
        || character == '&'
        || character == '='
        || character == '+'
        || character == '$'
        || character == ',';
}

Now let’s compare outputs for RuyJIT and LLVM:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
; C# RuyJIT
  movzx    rax, cx
  cmp      eax, 59
  je       SHORT G_IG04
  cmp      eax, 47
  je       SHORT G_IG04
  cmp      eax, 58
  je       SHORT G_IG04
  cmp      eax, 64
  je       SHORT G_IG04
  cmp      eax, 38
  je       SHORT G_IG04
  cmp      eax, 61
  je       SHORT G_IG04
  cmp      eax, 43
  je       SHORT G_IG04
  cmp      eax, 36
  je       SHORT G_IG04 
  cmp      eax, 44
  sete     al
  movzx    rax, al
G_IG03:
  ret      
G_IG04:
  mov      eax, 1
G_IG05:
  ret

1
2
3
4
5
6
7
8
9
10
11
12
13
; LLVM
  add edi, -36
  cmp di, 28
  ja .LBB0_2
  mov al, 1
  movzx ecx, di
  mov edx, 314575237    
  bt rdx, rcx
  jae .LBB0_2
  ret
.LBB0_2:
  xor eax, eax
  ret

As you can see C# generated a pretty simple set of 9 cmp + jumps for each logical OR. LLVM, at the same time, generated something strange with magic numbers and just two branches. Let’s try to convert (disassemble) LLVM’s output to C#:

1
2
3
4
5
6
7
bool IsReservedCharacter(char c)
{
    c = (char)(c - 36);
    if (c > 28)
        return false;
    return ((314575237 >> c) & 1) == 1;
}

so insted of 9 cmp we have add, cmp, shr, and Let me explain the magic constants.
First, we need to convert chars to their ASCII numbers:

1
2
';' '/' ':' '@' '&' '=' '+' '$' ','
59  47  58  64  38  61  43  36  44

The biggest is @ (64) and the smallest is $ (36). So, the range starts from 36 and the length is 64 - 36 = 28. Thus the first if simply ignores all values outside of [36..64] range. Here is how I explained the first two magic numbers. Now it’s 314575237s turn:

Since the range is known and the length is 28 which easily fits into a 32/64bit CPU register we can encode it to a special bit-map (a set of 0 and 1) - a 32/64 bit integer (depending on a platform). Here is how it’s done:

1
2
3
long bitmap = 0;
foreach (char c in new [] { ';', '/', ':', '@', '&', '=', '+', '$', ',' })
    bitmap |= 1L << c - 36;

So, for each char we push (shift) 1 to the left according to c - 36 value (as you remember 36 stands for $ so its index will be zero - on the right)
and our bitmap becomes:

1
2
3
4
5
  
00010010110000000000100110000101 = 314575237
   |  | ||          |  ||    | |
   @  = ;:          /  ,+    & $

Now when we do 314575237 >> (c - 36) we either get 1 (symbol is one of the reserved) or 0 (doesn’t belong to the set)

Let’s benchmark it! I have a random string here and I need to calculate how many symbols are reserved:

1
2
3
4
5
string str = "Some link https://github.com/dotnet/coreclr/issues/12477, some@mail.com.";
int count = 0;
foreach (char c in str)
    if (IsReservedCharacter(c))
        count++;

The results are:

1
2
3
4
|                      Method |      Mean | Ratio |
|---------------------------- |----------:|------:|
| CountReserverCharacters_old |  197.6 ns |  1.43 |
| CountReserverCharacters_new |  138.4 ns |  1.00 |

The improved version is 43% faster! (Core i7 8700K)

Feature request for RuyJIT dotnet/coreclr#12477

LLVM opt: godbolt.org (convert to switch)
LLVM llc: godbolt.org (DAG*)

Peephole optimizations in C++ and C#

Tue, 25 Jun 2019 23:32:00 +0000

“Performance gains due to improvements in compiler optimizations will double
the speed of a program every 18 years” © Proebsting’s Law

When we solve equations, we try to simplify them first, e.g. Y = -(5 - X) can be simplified to just Y = X - 5. In modern compilers it’s called “Peephole Optimizations”. Roughly speaking, compilers search for certain patterns and replace them with corresponding simplified expressions. In this blog post I’ll list some of them which I found in LLVM, GCC and .NET Core (CoreCLR) sources.

Let’s start with simple cases:

1
2
3
4
  X * 1          =>  X
 -X * -Y         =>  X * Y
-(X - Y)         =>  Y - X
  X * Z - Y * Z  =>  Z * (X - Y)

and check the 4th one in C++ and C# compilers:

1
2
3
int Test(int x, int y, int z) {
    return x * z - y * z;       //  =>  z * (x - y)
}

Now let’s take a look at what the compilers output:

1
2
3
4
5
Test(int, int, int):
  mov eax, edi
  sub eax, esi   ; -         
  imul eax, edx  ; *         
  ret

C++ (Clang, GCC, MSVC)

1
2
3
4
5
6
C.Test(Int32, Int32, Int32)  
  mov eax, edx
  imul eax, r9d  ; *
  imul r8d, r9d  ; *
  sub eax, r8d   ; -
  ret

C# (RyuJIT)

All three C++ compilers have just one imul instruction. C# (.NET Core) has two because it has a very limited set of available peephole optimizations and I’ll list some of them later. Be sure to note, the entire InstCombine transformation implementation, where peephole optimizations live, in LLVM takes more than 30K lines of code (+20k LOC in DAGCombiner.cpp). By the way, here is the piece of code in LLVM responsible for the pattern we are inspecting now. GCC has a special DSL which describes all peephole optimizations, and here is the piece of that DSL for our case.

I decided, just for this blog post, to try to implement this optimization for C# in JIT (hold my beer 😛):

Let’s now test my JIT improvement (see EgorBo/coreclr commit for more details) in VS2019 with Disasmo:

lea + imul instead of imul + imul + add

Let’s go back to C++ and trace the optimization in Clang. We need to ask clang to emit LLVM IR for our C++ code via -emit-llvm -g0 flags (see godbolt.org) and then give it to the LLVM optimizer opt via -O2 -print-before-all -print-after-all flags in order to find out what transformation actually removes that extra multiplication from the -O2 set. (see godbolt.org):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
; *** IR Dump Before Combine redundant instructions ***
define dso_local i32 @_Z5Case1iii(i32, i32, i32) {
  %4 = mul nsw i32 %0, %2
  %5 = mul nsw i32 %1, %2
  %6 = sub nsw i32 %4, %5
  ret i32 %6
}

; *** IR Dump After Combine redundant instructions ***
define dso_local i32 @_Z5Case1iii(i32, i32, i32) {
  %4 = sub i32 %0, %1
  %5 = mul i32 %4, %2
  ret i32 %5
}

It's InstCombine!

So it’s InstCombine indeed, we can even use it as the only optimization for our code for tests via -instcombine flag passed to opt:

Let’s go back to the examples. Look what a cute optimization I found in GCC sources:

1
X == C - X  =>  false if C is odd

And that’s true, e.g.: 4 == 8 - 4. Any odd number for C (C usually means a constant/literal) will always be false for the expression:

Foo2(int x) always returns false. LLVM doesn't have this optimization.

Optimizations vs IEEE754

Lots of this type of optimizations work for different data types, e.g. byte, int, unsigned, float. The latter is a bit tricky e.g. you can’t simplify A - B - A to -B for floats/doubles, even (A * B) * C is not equal to A * (B * C) due to the IEEE754 specification. However, C++ compilers have a special flag to let the optimizers be less strict around IEEE754, NaN and other FP corner cases and just apply all of the optimizations - it’s usually called “Fast Math” (-ffast-math for clang and gcc, /fp:fast for MSVC). Btw, here you can find my feature request for .NET Core to introduce the “Fast Math” mode there: dotnet/coreclr#24784).

As you can see, two vsubss were eliminated in the -ffast-math mode:

The C++ optimizers also support math.h functions, e.g.:

1
abs(X) * abs(X)  =>  X * X

The square root is always positive:

1
2
sqrt(X) < Y  => false, if Y is negative.
sqrt(X) < 0  => false

Why should we calculate sqrt(X) if we can just calculate C^2 in compile time instead?:

1
sqrt(X) > C  => X > C * C

More sqrt optimizations:

1
2
3
4
sqrt(X) == sqrt(Y)  => X == Y
sqrt(X) * sqrt(X)   => X
sqrt(X) * sqrt(Y)   => sqrt(X * Y)
logN(sqrt(X))       => 0.5 * logN(X)

or exp:

1
exp(X) * exp(Y)  => exp(X + Y)

And my favorite one:

1
sin(X) / cos(X)  => tan(X)

There are lots of boring bit/bool patterns:

1
2
3
4
5
6
7
8
((a ^ b) | a) -> (a | b)
(a & ~b) | (a ^ b)  -->  a ^ b
((a ^ b) | a) -> (a | b)
(X & ~Y) |^+ (~X & Y) -> X ^ Y
A - (A & B) into ~B & A
X <= Y - 1 equals to X < Y
A < B || A >= B -> true
... hundreds of them ...

Machine-dependent optimizations

Some operations may be faster or slower on different CPUs, e.g.:

1
X / 2  =>  X * 0.5

mulss/mulsd usually have better both latency and throughput than divss/divsd for example, here is the spec for my Intel Haswell CPU:

We can replace / C with * 1/C even in the non-“Fast Math” mode if C is a power of two. Btw, here is my PR for .NET Core for this optimization: dotnet/coreclr#24584.

The same rationale for:

1
X * 2  =>  X + X

test is better than cmp (see my PR dotnet/coreclr#25458 for more details):

1
2
3
4
X >= 1  =>  X > 0
X < 1   =>  X <= 0
X <= -1 =>  X >= 0
X > -1  =>  X >= 0

And what do you think about these ones?:

1
2
3
4
pow(X, 0.5)   =>  sqrt(x)
pow(X, 0.25)  =>  sqrt(sqrt(X))
pow(X, 2)     =>  X * X     ; 1 mul
pow(X, 3)     =>  X * X * X ; 2 mul

How many mul are needed to perform pow(X, 4) or X * X * X * X?

Just 2! Just like for pow(X, 3) and unlike pow(X, 3) we don’t even use the xmm1 register.

Modern CPUs support a special FMA instruction to perform mul and add in just one step without an intermediate rounding operation for mul:

1
X * Y + Z  =>  fmadd(X, Y, Z)

Sometimes compilers are able to replace entire algorithms with just one CPU instruction, e.g.:

Traps for optimizations

We can’t just find patterns & optimize them:

There is a risk to break some code: there are always corner-cases, hidden side-effects. LLVM’s bugzilla contains lots of InstCombine bugs.
An expression or its parts we want to simplify might be used somewhere else.

I borrowed a nice example for the second issue from “Future Directions for Optimizing Compilers” article.
Imagine we have a function:

1
2
3
4
5
int Foo1(int a, int b) {
   int na = -a;
   int nb = -b;
   return na + nb;
}

We need to perform 3 operations here: 0 - a, 0 - b, и na + nb. LLVM simplifies it to just two operations: return -(a + b) - what a smart move, here is the IR:

1
2
3
4
5
define dso_local i32 @_Z4Foo1ii(i32, i32) {
  %3 = add i32 %0, %1 ; a + b
  %4 = sub i32 0, %3  ; 0 - %3
  ret i32 %4
}

Now imagine that we need to store values of na and nb in some global variables, e.g. x and y:

1
2
3
4
5
6
7
8
9
int x, y;

int Foo2(int a, int b) {
   int na = -a;
   int nb = -b;
   x = na;
   y = nb;
   return na + nb;
}

The optimizer still recognizes the pattern and simplifies it by removing redundant (from its point of view) 0 - a and 0 - b operations. But we do need them! We save them to the global variables! Thus, it leads to this:

1
2
3
4
5
6
7
8
9
define dso_local i32 @_Z4Foo2ii(i32, i32) {
  %3 = sub nsw i32 0, %0                    ; 0 - a 
  %4 = sub nsw i32 0, %1                    ; 0 - b
  store i32 %3, i32* @x, align 4, !tbaa !2  
  store i32 %4, i32* @y, align 4, !tbaa !2
  %5 = add i32 %0, %1                       ; a + b
  %6 = sub i32 0, %5                        ; 0 - %5
  ret i32 %6
}

4 math operations instead of 3! The optimizer has just made our code a bit slower. Now let’s see what C# RuyJIT generates for this case:

RuyJIT doesn’t have this optimization so the code contains only 3 operations :-) C# is faster than C++! :p

Do we really need these optimizations?

Well, you never know what the final code will look like after inlining, constant folding, copy propagation, CSE, etc.
Also, both LLVM IR and .NET IL are not tied to a specific programming language and can’t rely on quality of the IR it generates. And you can just run your app/lib with InstCombine pass on and off to measure the performance impact.

What about C#?

As I said earlier, peephole optimizations are very limited in C# at the moment. However, when I say “C#” I mean the most popular C# runtime - CoreCLR with RuyJIT. But there are more, including those, using LLVM as a backend: Mono (see my tweet), Unity Burst and LILLC - these runtimes basically use exactly the same optimizations as clang does. Unity guys are even considering replacing C++ with C# in their internal parts. By the way, since .NET 5 will include Mono as an optional built-in runtime - you will be able to use LLVM power for such cases.

Back to CoreCLR - here are the peephole optimizations I managed to find in the morph.cpp comments (I am sure there are more):

1
2
3
4
5
6
7
8
9
10
*(&X)  =>  X
X % 1  => 0
X / 1  =>  X
X % Y  => X - (X / Y) * Y  // arm
X ^ -1  =>  ~x
X >= 1  =>  X > 0 
X <  1  =>  X <= 0
X + С1 == C2  =>  X == C2 - C1
((X + C1) + C2)  =>  (X + (C1 + C2))
((X + C1) + (Y + C2))  =>  ((X + Y) + (C1 + C2))

There are also some in lowering.cpp (machine-dependent ones) but in general RyuJIT obviously loses to С++ compilers here. RyuJIT just focuses more on different things and has a lot of requirements. The main one is - it should compile fast! it’s called JIT after all. And it does it very well (unlike the C++ compilers - see “Modern” C++ Lamentations). It’s also more important to de-virtualize calls, optimize out boxings, heap allocations (e.g. Object Stack Allocation). However, since RyuJIT is now supporting tiers, who knows maybe there will be a place for peephole optimizations in future in the tier1 or even a separate tier2 ;-). Maybe with some sort of DSL to declare them, just read this article where Prathamesh Kulkarni managed to declare an optimization for GCC in just a few lines of DSL:

1
2
3
4
5
(simplify
 (plus (mult (SIN @0) (SIN @0))
       (mult (COS @0) (COS @0)))
 (if (flag_unsafe_math_optimizations)
  { build_one_cst (TREE_TYPE (@0)); }))

for the following pattern:

1
cos^2(X) + sin^2(X) equals to 1

Hello world