org.lenskit.eval.traintest.metrics

Class Metric<X>

java.lang.Object

org.lenskit.eval.traintest.metrics.Metric<X>

Direct Known Subclasses:

PredictMetric, ResultMetric, TopNMetric

public abstract class Metric<X>
extends java.lang.Object

Interface for common aspects of metrics, which are used to measure individual or per-user results. Each metric can measure results or users, and contribute aggregate values to the global aggregate measurement table.

This class should not be extended directly; rather, one of its subclasses should be extended. This hierarchy uses abstract classes instead of interfaces for two reasons: first, to provide useful default behavior, and second, because implementing multiple types of metrics in a single class is likely to be erroneous.

Metrics use contexts to track and accumulate data for experimental conditions. For each experimental condition (algorithm / data set pair), the evaluator will do the following:

1. Create a context for the experiment using createContext(AlgorithmInstance, DataSet, RecommenderEngine).
2. Measure each user or result with the appropriate method.
3. Output the result of each user or result measurement to appropriate output file (if active).
4. Obtain the aggregate results from this metric with getAggregateMeasurements(Object) and add them to the global output file.

The context will generally consist of accumulators for the aggregate results reported by a metric over the entire experimental condition, such as the average of all user measurements. It may also contain additional relevant information, such as anything needed from the algorithm and data set for the measurements, or additional output tables for recording extra data.

Metrics themselves are generally stateless, with all state contained in the context. In this case, there is a single instance of the metric, or an instance per parameterization.

Metrics may be used from multiple threads. LensKit may use multiple threads with the same context.

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	Metric(java.util.List<java.lang.String> labels, java.util.List<java.lang.String> aggLabels)

Method Summary

Modifier and Type	Method and Description
X	createContext(AlgorithmInstance algorithm, DataSet dataSet, RecommenderEngine engine) Create the context for an experimental condition (algorithm/data set pair).
java.util.List<java.lang.String>	getAggregateColumnLabels() Get labels for the aggregate columns output by this metric.
MetricResult	getAggregateMeasurements(X context) Get the aggregate results from an accumulator.
java.util.List<java.lang.String>	getColumnLabels() Get the labels for the per-user or per-result columns output by this metric.
java.util.Set<java.lang.Class<?>>	getRequiredRoots() Get the classes on which this metric depends.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Metric

protected Metric(java.util.List<java.lang.String> labels,
                 java.util.List<java.lang.String> aggLabels)

Method Detail

getAggregateColumnLabels

public java.util.List<java.lang.String> getAggregateColumnLabels()

Get labels for the aggregate columns output by this metric.

Returns:

The labels for this metric’s aggregate output, used as column headers when outputting the results table.

getColumnLabels

public java.util.List<java.lang.String> getColumnLabels()

Get the labels for the per-user or per-result columns output by this metric.

Returns:

The labels for this metric’s output, used as column headers in the appropriate table.

getRequiredRoots

public java.util.Set<java.lang.Class<?>> getRequiredRoots()

Get the classes on which this metric depends. These will be added to the roots of each algorithm configuration.

Returns:

The required roots for this metric.

createContext

@Nullable
public X createContext(AlgorithmInstance algorithm,
                                 DataSet dataSet,
                                 RecommenderEngine engine)

Create the context for an experimental condition (algorithm/data set pair). The default implementation returns null.

Note: Contexts must be thread-safe, in that multiple concurrent calls to the appropriate user-measurement function with the same context must be safe. This can be handled either by the context itself, or by the user-measurement function.

Parameters:

algorithm - The algorithm.

dataSet - The data set.

engine - The LensKit recommender engine, if applicable. This can be null for an external algorithm that does not provide a LensKit recommender.

Returns:

The context. This will be passed to the individual measurement methods. If the metric does need to accumulate any results, this method can return null.

getAggregateMeasurements

@Nonnull
public MetricResult getAggregateMeasurements(X context)

Get the aggregate results from an accumulator. The default implementation returns MetricResult.empty().

Parameters:

context - The context for an experimental condition.

Returns:

The aggregate results from the accumulator.