Performance benchmarks
TeaVM has two sets of benchmarks:
benchmarksmodule measures performance of code produced by TeaVM;compiler-benchmarksmodule measures performance of the TeaVM compiler itself.
Benchmarks of generated code
Benchmarks are ordinary JMH benchmarks.
TeaVM's own tool (tools/perf module) compiles each benchmark with TeaVM and runs it in a browser
(JavaScript, WebAssembly GC) or as a native executable (C).
The same benchmarks can be run with JMH on JVM, so results can be compared.
To run benchmarks:
./gradlew :benchmarks:teavmBenchmark
Reports are written to benchmarks/build/reports/teavm-perf, including a JMH-like text table,
JSON results in JMH format, an HTML report and a Markdown table convenient to paste into pull requests.
Benchmarks are configured by benchmark.* Gradle properties, for example:
./gradlew :benchmarks:teavmBenchmark -Pbenchmark.backends=js,wasm-gc,c -Pbenchmark.compareJvm=true \
-Pbenchmark.include=StringSplit,Pi
Instead of passing them every time, copy benchmarks/gradle-local-template.properties to
benchmarks/gradle-local.properties (ignored by git) and uncomment the properties you need.
The template lists all supported properties with descriptions.
Measuring effect of a change
There are two scripts that run benchmarks twice and produce a report that compares results:
benchmarks/compare-stash.sh [gradle args...]compares uncommitted changes withHEAD;benchmarks/compare-commit.sh <commit> [gradle args...]compares the current working tree with the given commit.
For example:
benchmarks/compare-commit.sh HEAD~1 -Pbenchmark.backends=js,wasm-gc -Pbenchmark.include=String
Note that benchmarks take time, and results are only reliable when nothing else consumes CPU.
Use benchmark.include to select benchmarks that are relevant to your change.
Writing benchmarks
Put benchmarks into benchmarks/src/main/java.
They are written with JMH annotations, but only a subset of JMH is supported
(@Benchmark, @State, @Param, @Setup/@TearDown with trial and iteration levels,
@BenchmarkMode, @Warmup, @Measurement, @Fork, Blackhole and some others).
Keep in mind that the code is compiled by TeaVM, so it can only use APIs supported by TeaVM class library.
See README for the full list of supported features, settings, profiling and other details.
Benchmarks of the compiler
compiler-benchmarks module contains JMH benchmarks that run TeaVM compiler on JVM.
Use them when you change the compiler and want to make sure it does not become slower or consume more memory.
./gradlew :compiler-benchmarks:jmh
Settings are configured in the same way, see compiler-benchmarks/gradle-local-template.properties
for the list of properties, including how to save a baseline and compare with it.