Following some great advice here from Simon, I realized that I had over-engineered things, and that the Task builder methods were a horrible Java8 abstraxction. In Simon's words: "From a usability perspective, this is a bit weird...".
After messing with things a bit more, the original use-case like:
uBench.addTask(Task.buildCheckedIntTask("Legato Java7", () -> getMaximumBeauty(line), expect)); uBench.addTask(Task.buildCheckedIntTask("Legato Java8", () -> getMaximumBeauty8(line), expect)); uBench.addTask(Task.buildCheckedIntTask("Janos Java7", () -> computeMaxBeauty(line), expect)); uBench.addTask(Task.buildCheckedIntTask("Rolfl Java7", () -> beautyMax7(line), expect)); uBench.addTask(Task.buildCheckedIntTask("Rolfl Java8Regex", () -> beautyMaxF(line), expect)); uBench.addTask(Task.buildCheckedIntTask("Rolfl Java8Filter", () -> beautyMax8(line), expect));
Can be drastically simplified if the addTask method takes a Supplier directly (instead of a task), and a separate Predicate to check the results. The same code above, can be expressed as:
uBench.addTask("Legato Java7", () -> getMaximumBeauty(line), g -> g == 1574); uBench.addTask("Legato Java8", () -> getMaximumBeauty8(line), g -> g == 1574); uBench.addTask("Janos Java7", () -> computeMaxBeauty(line), g -> g == 1574); uBench.addTask("Rolfl Java7", () -> beautyMax7(line), g -> g == 1574); uBench.addTask("Rolfl Java8Regex", () -> beautyMaxF(line), g -> g == 1574); uBench.addTask("Rolfl Java8Filter", () -> beautyMax8(line), g -> g == 1574);
(where g is a mnemonic for got). That can in turn be simplified to a single predicate:
Predicate<Integer> check = g -> g == 1574;
and code like:
uBench.addTask("Legato Java7", () -> getMaximumBeauty(line), check); uBench.addTask("Legato Java8", () -> getMaximumBeauty8(line), check); uBench.addTask("Janos Java7", () -> computeMaxBeauty(line), check); uBench.addTask("Rolfl Java7", () -> beautyMax7(line), check); uBench.addTask("Rolfl Java8Regex", () -> beautyMaxF(line), check); uBench.addTask("Rolfl Java8Filter", () -> beautyMax8(line), check);
In addition, to support primitive-type operations (instead of having to auto-box them - which may impact performance), there needs to be an implementation specialized for each of int, double, and long too.
To that end, I have removed the Task class entirely from the public interface, and incorporated it as an internal-only static class. To reduce the code footprint further, I have moved the TaskStats class in as a static nested class as well. This reduces the entire benchmark code to a single Java file, which has a number of advantages when maintaining or distributing the code.
The new usage of the code is (for context, not for review...):
final String line = "This is a test, including punctuation, and other words" + " and numbers like 1, UPPER, and Lower letters"; IntPredicate expect = (g) -> g == 1574; UBench uBench = new UBench("Beautiful"); uBench.addIntTask("Legato Java7", () -> getMaximumBeauty(line), expect); uBench.addIntTask("Legato Java8", () -> getMaximumBeauty8(line), expect); uBench.addIntTask("Janos Java7", () -> computeMaxBeauty(line), expect); uBench.addIntTask("Rolfl Java7", () -> beautyMax7(line), expect); uBench.addIntTask("Rolfl Java8Regex", () -> beautyMaxF(line), expect); uBench.addIntTask("Rolfl Java8Filter", () -> beautyMax8(line), expect); System.out.println("Warming up"); uBench.benchMark(5000).stream().forEach(System.out::println); System.out.println("\n\nReal runs\n\n"); uBench.benchMark(10000).stream().sorted(Comparator.comparing(UBench.Stats::get95thPercentile)) .forEach(System.out::println);
And the UBench code that supports that (the GitHub revision), is:
package net.tuis.ubench; import java.util.Arrays; import java.util.LinkedHashMap; import java.util.List; import java.util.LongSummaryStatistics; import java.util.Map; import java.util.concurrent.TimeUnit; import java.util.function.DoublePredicate; import java.util.function.DoubleSupplier; import java.util.function.IntPredicate; import java.util.function.IntSupplier; import java.util.function.LongPredicate; import java.util.function.LongSupplier; import java.util.function.Predicate; import java.util.function.Supplier; import java.util.stream.Collectors; import java.util.stream.DoubleStream; import java.util.stream.IntStream; import java.util.stream.LongStream; /** * The UBench class encompasses a suite of tasks that are to be compared... * possibly relative to each other. * <p> * Each task can be added to the suite. Once you have the tasks you need, then * all tasks can be benchmarked according to limits given in the run. * * @author rolf * */ public final class UBench { /** * Statistics representing the runs in this task. * <p> * Presents various statistics related to the run times that are useful for * interpreting the run performance. */ public static final class Stats { private static final double NANOxMILLI = 1000000.0; private final long[] results; private final long min; private final long max; private final double average; private final String suit; private final String name; /** * Construct statistics based on the nanosecond times of multiple runs. * * @param name * The name of the task that has been benchmarked * @param results * The nano-second run times of each successful run. */ Stats(String suit, String name, long[] results) { this.suit = suit; this.name = name; this.results = results; LongSummaryStatistics lss = LongStream.of(results).summaryStatistics(); min = lss.getMin(); max = lss.getMax(); average = lss.getAverage(); } /** * Get the raw data the statistics are based off. * * @return the individual test run times (in nanoseconds, and in order * of execution). */ public long[] getRawData() { return Arrays.copyOf(results, results.length); } /** * Summarize the time-progression of the run time for each iteration, in * order of execution (in milliseconds). * <p> * An example helps. If there are 200 results, and a request for 10 * zones, then return 10 double values representing the average time of * the first 20 runs, then the next 20, and so on, until the 10th zone * contains the average time of the last 20 runs. * <p> * This is a good way to see the effects of warm-up times and different * compile levels * * @param zoneCount * @return */ public final double[] getZoneTimesMilli(int zoneCount) { double[] ret = new double[Math.min(zoneCount, results.length)]; int perblock = results.length / ret.length; int overflow = results.length % ret.length; int pos = 0; for (int block = 0; block < ret.length; block++) { int count = perblock + (block < overflow ? 1 : 0); int limit = pos + count; long nanos = 0; while (pos < limit) { nanos += results[pos]; pos++; } ret[block] = (nanos / NANOxMILLI) / count; } return ret; } /** * Compute a log-2-based histogram relative to the fastest run in the * data set. * <p> * This gives a sense of what the general shape of the runs are in terms * of distribution of run times. The histogram is based on the fastest * run. * <p> * By way of an example, the output: <code>100, 50, 10, 1, 0, 1</code> * would suggest that: * <ul> * <li>100 runs were between 1 times and 2 times as slow as the fastest. * <li>50 runs were between 2 and 4 times slower than the fastest. * <li>10 runs were between 4 and 8 times slower * <li>1 run was between 8 and 16 times slower * <li>1 run was between 32 and 64 times slower * * @return */ public final int[] getHistogramByDoublingFactor() { int count = (int) (max / min); int[] histo = new int[Integer.numberOfTrailingZeros(Integer.highestOneBit(count)) + 1]; LongStream.of(results).mapToInt(t -> Integer.numberOfTrailingZeros(Integer.highestOneBit((int) (t / min)))) .forEach(i -> histo[i]++); return histo; } /** * Compute the 95<sup>th</sup> percentile of runtimes (in milliseconds). * <p> * 95% of all runs completed in this time, or faster. * * @return the millisecond time of the 95<sup>th</sup> percentile. */ public final double get95thPercentile() { if (results.length < 100) { return getSlowest(); } long limit = ((results.length + 1) * 95) / 100; return LongStream.of(results).sorted().limit(limit).max().getAsLong() / NANOxMILLI; } /** * Compute the average time of all runs (in milliseconds). * * @return the average time (in milliseconds) */ public final double getAverage() { return average / NANOxMILLI; } /** * Compute the slowest run (in milliseconds). * * @return The slowest run time (in milliseconds). */ public final double getSlowest() { return max / NANOxMILLI; } /** * Compute the fastest run (in milliseconds). * * @return The fastest run time (in milliseconds). */ public final double getFastest() { return min / NANOxMILLI; } @Override public String toString() { return String.format("Task %s -> %s:\n" + " Iterations : %12d\n" + " Fastest : %12.5fms\n" + " Average : %12.5fms\n" + " 95Pctile : %12.5fms\n" + " Slowest : %12.5fms\n" + " TimeBlock : %s\n" + " FactorHisto : %s\n", suit, name, results.length, getFastest(), getAverage(), get95thPercentile(), getSlowest(), formatMillis(getZoneTimesMilli(10)), formatHisto(getHistogramByDoublingFactor())); } private String formatHisto(int[] histogramByXFactor) { return IntStream.of(histogramByXFactor).mapToObj(i -> String.format("%5d", i)) .collect(Collectors.joining(" ")); } private String formatMillis(double[] zoneTimesMilli) { return DoubleStream.of(zoneTimesMilli).mapToObj(d -> String.format("%.5fms", d)) .collect(Collectors.joining(" ")); } public String getSuit() { return suit; } public String getName() { return name; } } private static class NamedTask { private final String name; private final Task task; public NamedTask(String name, Task task) { super(); this.name = name; this.task = task; } public String getName() { return name; } public Task getTask() { return task; } } @FunctionalInterface private interface Task { long time(); } private final Map<String, Task> tasks = new LinkedHashMap<>(); private final String suiteName; public UBench(String suiteName) { this.suiteName = suiteName; } private void putTask(String name, Task t) { synchronized (tasks) { tasks.put(name, t); } } /** * Include a named task (and validator) in to the benchmark. * @param name The name of the task. Only one task with any one name is allowed. * @param task The task to perform * @param check The check of the results from the task. */ public <T> void addTask(String name, Supplier<T> task, Predicate<T> check) { putTask(name, () -> { long start = System.nanoTime(); T result = task.get(); long time = System.nanoTime() - start; if (check != null && !check.test(result)) { throw new IllegalStateException(String.format("Task %s failed Result: %s", name, result)); } return time; }); } /** * Include a named task in to the benchmark. * @param name The name of the task. Only one task with any one name is allowed. * @param task The task to perform */ public <T> void addTask(String name, Supplier<T> task) { addTask(name, task, null); } /** * Include an int-specialized named task (and validator) in to the benchmark. * @param name The name of the task. Only one task with any one name is allowed. * @param task The task to perform * @param check The check of the results from the task. */ public void addIntTask(String name, IntSupplier task, IntPredicate check) { putTask(name, () -> { long start = System.nanoTime(); int result = task.getAsInt(); long time = System.nanoTime() - start; if (check != null && !check.test(result)) { throw new IllegalStateException(String.format("Task %s failed Result: %s", name, result)); } return time; }); } /** * Include an int-specialized named task in to the benchmark. * @param name The name of the task. Only one task with any one name is allowed. * @param task The task to perform */ public void addIntTask(String name, IntSupplier task) { addIntTask(name, task, null); } /** * Include a long-specialized named task (and validator) in to the benchmark. * @param name The name of the task. Only one task with any one name is allowed. * @param task The task to perform * @param check The check of the results from the task. */ public void addLongTask(String name, LongSupplier task, LongPredicate check) { putTask(name, () -> { long start = System.nanoTime(); long result = task.getAsLong(); long time = System.nanoTime() - start; if (check != null && !check.test(result)) { throw new IllegalStateException(String.format("Task %s failed Result: %s", name, result)); } return time; }); } /** * Include a long-specialized named task in to the benchmark. * @param name The name of the task. Only one task with any one name is allowed. * @param task The task to perform */ public void addLongTask(String name, LongSupplier task) { addLongTask(name, task, null); } /** * Include a double-specialized named task (and validator) in to the benchmark. * @param name The name of the task. Only one task with any one name is allowed. * @param task The task to perform * @param check The check of the results from the task. */ public void addDoubleTask(String name, DoubleSupplier task, DoublePredicate check) { putTask(name, () -> { long start = System.nanoTime(); double result = task.getAsDouble(); long time = System.nanoTime() - start; if (check != null && !check.test(result)) { throw new IllegalStateException(String.format("Task %s failed Result: %s", name, result)); } return time; }); } /** * Include a double-specialized named task in to the benchmark. * @param name The name of the task. Only one task with any one name is allowed. * @param task The task to perform */ public void addDoubleTask(String name, DoubleSupplier task) { addDoubleTask(name, task, null); } /** * Benchmark a task until it completes the desired iterations, exceeds the * time limit, or reaches stability, whichever comes first. * * @param iterations * maximum number of iterations to run. * @param minStabilityLen * If this many iterations in a row are all within the * maxVariance, then the benchmark ends. * @param maxVariance * Expressed as a percent from 0.0 to 100.0, and so on * @return the results of all completed tasks. */ public List<Stats> benchMark(final int iterations, final int minStabilityLen, final double maxVariance, final long timeLimit, final TimeUnit timeUnit) { List<NamedTask> mytasks = getTasks(); Stats[] ret = new Stats[mytasks.size()]; int i = 0; for (NamedTask task : mytasks) { ret[i++] = runTask(task, iterations, minStabilityLen, 1 + (maxVariance / 100.0), timeLimit, timeUnit); } return Arrays.asList(ret); } /** * Benchmark all tasks until it they complete the desired elapsed time * * @param iterations * number of iterations to run. * @return the results of all completed tasks. */ public List<Stats> benchMark(final long timeLimit, final TimeUnit timeUnit) { return benchMark(Integer.MAX_VALUE, 0, 100, timeLimit, timeUnit); } /** * Benchmark all tasks until it they complete the desired iteration count * * @param iterations * number of iterations to run. * @return the results of all completed tasks. */ public List<Stats> benchMark(final int iterations) { return benchMark(iterations, 0, 100, 1000, TimeUnit.DAYS); } private List<NamedTask> getTasks() { synchronized (tasks) { return tasks.entrySet().stream().map(e -> new NamedTask(e.getKey(), e.getValue())) .collect(Collectors.toList()); } } private Stats runTask(final NamedTask ntask, final int iterations, final int minStability, final double maxLimit, final long timeLimit, final TimeUnit timeUnit) { long[] results = new long[Math.min(iterations, 10000)]; long[] recents = new long[Math.min(minStability, iterations)]; int rPos = 0; long limit = System.currentTimeMillis() + timeUnit.toMillis(timeLimit); for (int i = 0; i < iterations; i++) { long res = Math.max(ntask.getTask().time(), 1); if (rPos >= results.length) { results = Arrays.copyOf(results, expandTo(results.length)); } if (minStability > 0) { recents[rPos % recents.length] = res; } results[rPos++] = res; if ((timeLimit > 0 && System.currentTimeMillis() >= limit) || (minStability > 0 && rPos >= recents.length && inBounds(recents, maxLimit))) { return new Stats(suiteName, ntask.getName(), Arrays.copyOf(results, rPos)); } } return new Stats(suiteName, ntask.getName(), Arrays.copyOf(results, rPos)); } private int expandTo(int length) { // add 25% + 100 - limit to Integer.Max int toAdd = 100 + (length >> 2); toAdd = Math.min(Integer.MAX_VALUE - length, toAdd); return toAdd + length; } @Override public String toString() { return String.format("%s with tasks: %s", suiteName, tasks.toString()); } /** * Compute whether any of the values in times exceed the given bound, * realtive to the minimum value in times. * * @param times * the times to compute the bounds on * @param bound * the bound is represented as a value like 1.10 for 10% greater * than the minimum * @return true if all values are in bounds. */ private static final boolean inBounds(long[] times, double bound) { long min = times[0]; long max = times[0]; long limit = (long) (min * bound); for (int i = 1; i < times.length; i++) { if (times[i] < min) { min = times[i]; limit = (long) (min * bound); if (max > limit) { return false; } } if (times[i] > max) { max = times[i]; // new max, is it slower than the worst allowed? if (max > limit) { return false; } } } return true; } } Again, I am looking for any and all feedback, but I am particularly interested in usability concerns and API issues.
