org.grouplens.lenskit.eval.metrics

Interface Metric<X>

Type Parameters:

X - The type of accumulator used by this metric.

All Superinterfaces:

AutoCloseable, Closeable

All Known Implementing Classes:

AbstractMetric, CoveragePredictMetric, EntropyPredictMetric, FunctionModelMetric, FunctionMultiModelMetric, HLUtilityPredictMetric, IndependentRecallTopNMetric, MAEPredictMetric, MAPTopNMetric, MRRTopNMetric, NDCGPredictMetric, NDCGTopNMetric, OutputPredictMetric, OutputTopNMetric, PrecisionRecallTopNMetric, RMSEPredictMetric, TopNEntropyMetric, TopNLengthMetric, TopNPopularityMetric

public interface Metric<X>
extends Closeable

Base interface for metrics, which are used by evaluations to measure their results. Metrics may have additional resources backing them, such as extra output files, and therefore must be closed.

Metrics use contexts to track and accumulate data for experimental conditions. For each experimental condition (algorithm / data set pair), the evaluator will do the following:

1. Create a context for the experiment using createContext(Attributed, TTDataSet, Recommender).
2. Measure each test user with measureUser(TestUser, Object), passing in the context in which the user should be evaluated.
3. Output the result of each user measurement to the per-user output table (if active).
4. Obtain the aggregate results from this metric with getResults(Object).

The context will general consist of accumulators for the aggregate results reported by a metric over the entire experimental condition, such as the average of all user measurements. It may also contain additional relevant information, such as anything needed from the algorithm and data set for the measurements, or additional output tables for recording extra data.

Metrics themselves are generally stateless, with all state contained in the context. In this case, there is a single instance of the metric, or an instance per parameterization, and the Closeable.close() method is a no-op. Some metrics need state or output resources; such metrics should define a MetricFactory to instantiate them, and clean up and release resources or state in Closeable.close().

Metrics may be used from multiple threads. LensKit does not currently use multiple threads with the same context, but that may change in the future.

AbstractMetric provides a base implementation of this interface that allows user and aggregate measurements to be defined in plan Java objects, so metrics do not need to handle creating table rows themselves.

Since:

0.10 (rewritten in 2.1)

Method Summary

Modifier and Type	Method and Description
X	createContext(Attributed algorithm, TTDataSet dataSet, Recommender recommender) Create the context for an experimental condition (algorithm/data set pair).
List<String>	getColumnLabels() Get labels for the aggregate columns output by this evaluator.
List<Object>	getResults(X context) Get the aggregate results from an accumulator.
List<String>	getUserColumnLabels() Get labels for the per-user columns output by this evaluator.
List<Object>	measureUser(TestUser user, X context) Measure a user in the evaluation.

Methods inherited from interface java.io.Closeable

close

Method Detail

getColumnLabels

List<String> getColumnLabels()

Get labels for the aggregate columns output by this evaluator.

Returns:

The labels for this evaluator's output, used as column headers when outputting the results table.

getUserColumnLabels

List<String> getUserColumnLabels()

Get labels for the per-user columns output by this evaluator.

Returns:

The labels for this evaluator's per-user output, used as column headers when outputting the results table.

See Also:

measureUser(TestUser, Object)

createContext

@Nullable
X createContext(Attributed algorithm,
                          TTDataSet dataSet,
                          Recommender recommender)

Create the context for an experimental condition (algorithm/data set pair).

Parameters:

algorithm - The algorithm.

dataSet - The data set.

recommender - The LensKit recommender, if applicable. This can be null for an external algorithm that does not provide a LensKit recommender.

Returns:

The accumulator. This will be passed to measureUser(TestUser, Object). If the metric does not accumulate any results, this method can return null.

measureUser

@Nonnull
List<Object> measureUser(TestUser user,
                                  X context)

Measure a user in the evaluation.

Parameters:

user - The user to evaluate.

context - The context for the active experimental condition.

Returns:

The table rows resulting from this user's measurement.

getResults

@Nonnull
List<Object> getResults(X context)

Get the aggregate results from an accumulator.

Parameters:

context - The context for an experimental condition.

Returns:

The aggregate results from the accumulator.