05 March 2024

Some New Language Benchmarks

I have a small set of benchmarks (adapted from some I found online) that I use to track changes in how programming languages perform between releases. They aren’t meant to be some super scientific set of benchmarks, but really just something I can use to gauge how a language is changing over time, and to help make decisions on what to use.

Benchmarks Source Code

The repo is pretty messy, and some of the wiki information is out of date, but the benchmark code itself is up to date.

Anyway, it’s been a while since I’ve run these - a few SBCL versions have come, and a number of kernel releases - so I’ve run a new batch of benchmarks. These are just for my desktop this time, a Core i9-10850K with 64GB of RAM. As usual, I run each benchmark three times, and there’s four benchmarks (BinaryTrees, Mandelbrot, nBody, SpectralNorm). Then at the end, I take the geometric mean of their runtimes per-language. This is where the “from 12 benchmark runs” comment comes from. A few languages are still missing benchmark implementations, but they’re ones I rarely test, and this is why they don’t have 12 runs. Finally, there’s a 30 second cooldown between each benchmark and each batch to give the CPU time to cool.

So, overall results (ignore “0.000 Joules”, I didn’t measure energy usage this time around):

Geometric Means:
  C      : 1.502583 seconds,   0.000 Joules (from 12 benchmark runs)
  Crystal: 1.678064 seconds,   0.000 Joules (from 12 benchmark runs)
  Sbcl   : 1.496556 seconds,   0.000 Joules (from 12 benchmark runs)
  Ccl    : 9.756819 seconds,   0.000 Joules (from 12 benchmark runs)
  Clisp  : 201.840503 seconds,   0.000 Joules (from 12 benchmark runs)
  Chez   : 8.111610 seconds,   0.000 Joules (from 9 benchmark runs)
  Pascal : 4.000167 seconds,   0.000 Joules (from 12 benchmark runs)
  Csharp : 3.915367 seconds,   0.000 Joules (from 12 benchmark runs)
  Vala   : 2.919772 seconds,   0.000 Joules (from 12 benchmark runs)
  Go     : 2.017984 seconds,   0.000 Joules (from 12 benchmark runs)
  Luajit : 4.913561 seconds,   0.000 Joules (from 6 benchmark runs)
  Lua    : 59.165653 seconds,   0.000 Joules (from 6 benchmark runs)
Overall geometric mean time based on all recorded runs: 5.259032 seconds
Total time of all results: 6255.290004 seconds

And now a sample run (ignore the RAM usage on Pascal, I think that’s wrong):

--== BinaryTrees ==--
C      : == 33.6MB     00:00:01.959569717 ==
Crystal: == 49.0MB     00:00:01.342501578 ==
Sbcl   : == 114MB      00:00:00.582577706 ==
Ccl    : == 74.2MB     00:00:02.259294319 ==
Clisp  : == 47.8MB     00:00:17.011516267 ==
Chez   : == 94.2MB     00:00:00.509951270 ==
Pascal : == 64.1MB     00:00:05.163150693 ==
Csharp : == 76.5MB     00:00:01.054297913 ==
Vala   : == 31.2MB     00:00:04.542070881 ==
Go     : == 26.4MB     00:00:02.074772000 ==
Luajit : == 630MB      00:00:15.381059148 ==
Lua    : == 1.19GB     00:00:55.981471207 ==

--== Mandelbrot ==--
C      : == 0.96MB     00:00:06.085557414 ==
Crystal: == 2.94MB     00:00:06.175731252 ==
Sbcl   : == 13.1MB     00:00:06.558942719 ==
Ccl    : == 15.4MB     00:00:22.772328110 ==
Clisp  : == 8.92MB     00:16:54.677320279 ==
Chez   : == 29.5MB     00:00:56.949132997 ==
Pascal : == 160KB      00:00:10.249364808 ==
Csharp : == 14.4MB     00:00:11.421616128 ==
Vala   : == 2.66MB     00:00:06.090429091 ==
Go     : == 1.1MB      00:00:06.318965715 ==

--== nBody ==--
C      : == 1.57MB     00:00:00.502049789 ==
Crystal: == 3.33MB     00:00:00.782386872 ==
Sbcl   : == 12.6MB     00:00:00.701044850 ==
Ccl    : == 18.2MB     00:00:15.346944018 ==
Clisp  : == 9.05MB     00:03:12.847068220 ==
Pascal : == 160KB      00:00:00.957448552 ==
Csharp : == 15.8MB     00:00:01.580353517 ==
Vala   : == 2.98MB     00:00:02.036945745 ==
Go     : == 1.4MB      00:00:00.746613930 ==

--== SpectralNorm ==--
C      : == 1.23MB     00:00:00.854140688 ==
Crystal: == 3.93MB     00:00:01.204312136 ==
Sbcl   : == 13.2MB     00:00:01.852422723 ==
Ccl    : == 16.6MB     00:00:11.383126295 ==
Clisp  : == 9.29MB     00:08:15.448554378 ==
Chez   : == 29.6MB     00:00:18.181106257 ==
Pascal : == 160KB      00:00:05.034851784 ==
Csharp : == 13.4MB     00:00:12.037564292 ==
Vala   : == 3.05MB     00:00:01.279793423 ==
Go     : == 3.13MB     00:00:01.665142646 ==
Luajit : == 3.33MB     00:00:01.554987574 ==
Lua    : == 4.46MB     00:01:03.079562542 ==

Basically things aren’t changing too much, at least as far as stuff I’m interested in.

And finally, here are the versions for each language:

• GCC 11.2
• Crystal 1.11.2
• Steel Bank Common Lisp 2.4.2
• ClozureCL 1.12
• CLisp 2.49.93+
• Chez Scheme 9.5.1
• FreePascal 3.2.2
• Mono 6.12.0.199
• Vala 0.54.6
• Go 1.21.6
• LuaJIT 2.0.5
• Lua 5.1.5