18 September 2023

Some Recent, Rather Informal Language Benchmarks

I have a small set of benchmarks (adapted from some I found online) that I use to track changes in how programming languages perform between releases. They aren’t meant to be some super scientific set of benchmarks, but really just something I can use to gauge how a language is changing over time, and to help make decisions on what to use.

The repo is pretty messy, and some of the wiki information is out of date, but the benchmark code itself is up to date.

Background

I normally use three computers:

• My laptop (Ryzen 3 3200U, 10GB RAM)
• My desktop (Core i9-10850K, 64GB RAM)
• My PineBook Pro (Rockchip RK3399, 4GB RAM)

My laptop is located upstairs and is mainly used for just browsing the web, chatting on Matrix, and listening to music. Occasionally I’ll do some light programming on it as well. Basically it’s not my main driver, but it does get used regularly.

My desktop is where I do all my programming, gaming, music stuff, etc. Basically it’s my main computer. It’s located downstairs and is mainly used in the evening.

My PineBook is located right next to my desktop, and is used kinda like a second monitor. On both it and my desktop, I run a program called Barrier that lets me share the keyboard and mouse over the network. So I’ll often have music or a movie playing on it while I work on my desktop.

I mention all of these because it’s kinda important in understanding how I’m looking at my benchmarks. I basically have an upper low-end laptop, a high end desktop, and something akin to a Chromebook. And on all of them, I use software that I’ve written myself.

Up until recently, I’ve only benchmarked my laptop and desktop, so only x86-64. But I use my major project these days, Benben, on my PineBook a lot. Since I’ve considered doing a similar GUI player in Common Lisp, I figured it was a good idea to add AArch64 to my benchmarks.

Benchmark Results

Results for the Core i9-10850K (desktop):

Geometric Means:
  C      : 1.506488 seconds,   0.000 Joules (from 12 benchmark runs)
  Crystal: 1.680608 seconds,   0.000 Joules (from 12 benchmark runs)
  Sbcl   : 1.509577 seconds,   0.000 Joules (from 12 benchmark runs)
  Pascal : 4.005819 seconds,   0.000 Joules (from 12 benchmark runs)
  Csharp : 3.962749 seconds,   0.000 Joules (from 12 benchmark runs)
  Vala   : 2.930969 seconds,   0.000 Joules (from 12 benchmark runs)
  Go     : 2.048725 seconds,   0.000 Joules (from 12 benchmark runs)

Ryzen 3 3200U (laptop):

Geometric Means:
  C      : 3.113297 seconds,   0.000 Joules (from 12 benchmark runs)
  Crystal: 3.188073 seconds,   0.000 Joules (from 12 benchmark runs)
  Sbcl   : 2.851436 seconds,   0.000 Joules (from 12 benchmark runs)
  Pascal : 6.888914 seconds,   0.000 Joules (from 12 benchmark runs)
  Csharp : 5.884691 seconds,   0.000 Joules (from 12 benchmark runs)
  Vala   : 4.946014 seconds,   0.000 Joules (from 12 benchmark runs)
  Go     : 4.043019 seconds,   0.000 Joules (from 12 benchmark runs)

PineBook Pro (RK3399):

Geometric Means:
  C      : 8.068883 seconds,   0.000 Joules (from 12 benchmark runs)
  Crystal: 9.244011 seconds,   0.000 Joules (from 12 benchmark runs)
  Sbcl   : 10.942852 seconds,   0.000 Joules (from 12 benchmark runs)
  Pascal : 16.739183 seconds,   0.000 Joules (from 12 benchmark runs)
  Vala   : 15.279412 seconds,   0.000 Joules (from 12 benchmark runs)
  Go     : 10.151757 seconds,   0.000 Joules (from 12 benchmark runs)

I didn’t collect energy information this time around, so just ignore the Joules field. SBCL is Steel Bank Common Lisp, my preferred Common Lisp implementation.

Compiler Info

C: GCC 11.2.0 (x86-64) and 13.2.0 (AArch64) Crystal: 1.9.2 SBCL: 2.3.8 FreePascal : 3.2.2 C#: Microsoft Visual C# Compiler (csc, from Mono) 3.9.0-6.21124.20, and Mono 6.12.0.199 for the runtime. Vala: 0.54.6 (x86-64) and 0.56.3 (AArch64) Go: 1.20.5

On my x86-64 machines, LLVM is at 13.0.0. On my PineBook, it’s at 16.0.6.

Thoughts

The languages are ones that I either actively use (Crystal, Common Lisp via SBCL), are interested in (FreePascal), or have used in the past. Overall, the x86-64 results haven’t changed much in the past year. SBCL has become a very tiny bit faster is about the only change I noticed. I don’t expect these results to change dramatically any time soon, either. All of these are still vastly more performant than languages like Python, Ruby, or Lua.

The AArch64 results were mildly surprising. SBCL’s support for AArch64 is still quite new afaik, and is still getting a lot of work done on it, which is why I think it’s a bit slower on my PineBook relative to Crystal and C. It also doesn’t support all of SBCL’s features yet (like “compact instance headers”, and “immobile space” is still glitchy). My guess is that it’ll improve with time.

I was very pleased to see how fast Crystal is on AArch64, though I’m not 100% sure if it’s because of Crystal itself, or differences in the LLVM versions. I’ll know for sure once Slackware 15.1 comes out so I can use the same version on all of my machines (the PineBook is on Slackware-current).

Overall I’d classify the languages as they currently are as such on x86-64:

• C, Crystal, SBCL - Top tier fast.
• Go, Vala - Nearly top tier.
• C#, FreePascal: Mid tier fast.

And on AArch64:

• C, Crystal - Top tier fast.
• SBCL, Go - Nearly top tier.
• Vala, FreePascal: Mid tier fast.

How I’m Using This

So I’ve had an idea floating around since late May or early June about creating a GUI player for VGM files. My initial plan was to do this in Lisp (meaning I’d port my YunoSynth library to Common Lisp) and use the McCLIM toolkit for fun. My initial results after porting just a small number of the emulator cores to Common Lisp show that SBCL is nearly equivalent performance wise to my Crystal code. There’s a few places where it’s slower, but I suspect those spots come from differences in the language that just need to be smoothed over. After all, I don’t access to raw pointers in Lisp. Well ok, I technically do, but I don’t want to use them.

Whatever I end up writing my GUI player in, one thing I have to keep in mind is that I’ll be running it on my PineBook quite often, not just my laptop and desktop. This is the reason I ported my benchmarks to AArch64. What I’m seeing currently is that Crystal may actually be the best language when I need to target both architectures and need to be concerned with runtime performance. It’s top tier in both, it’s not C, and it’s fun to use. So perhaps Crystal would be the better option for a GUI player, especially considering that I already have my YunoSynth library written in it.

On the other hand, Common Lisp has McCLIM :-P And CL-Charms if I go the ncurses route instead. GTK 4 is bloated trash, Qt is trash, GTK2 (my favorite) is sadly going the way of the dinosaur, and GTK3 bindings are an issue. I also really don’t want to do anything with web-based stuff for both technical and philosophical reasons. That leaves me with something like Dear IMGUI or Nuklear on Crystal (note: Nuklear is also available or SBCL). But these have separate issues, namely awkward font handling.

So I dunno. The benchmarks clearly show that Crystal (and C, if you want to use C) are very good choices if you need performance and are targeting either or both platforms. SBCL joins them on x86-64 for the top tier. I’m willing to bet that the performance of SBCL on AArch64 will catch up in time, though.