org.grouplens.lenskit.eval.traintest

Class TrainTestEvalTask

java.lang.Object

com.google.common.util.concurrent.AbstractFuture<T>

org.grouplens.lenskit.eval.AbstractTask<Table>

org.grouplens.lenskit.eval.traintest.TrainTestEvalTask

All Implemented Interfaces:

ListenableFuture<Table>, Future<Table>, EvalTask<Table>

public class TrainTestEvalTask
extends AbstractTask<Table>

The command that run the algorithmInfo instance and output the prediction result file and the evaluation result file

Constructor Summary

Constructors

Constructor and Description
TrainTestEvalTask()
TrainTestEvalTask(String name)

Method Summary

Modifier and Type	Method and Description
TrainTestEvalTask	addAlgorithm(AlgorithmInstance algorithm)
TrainTestEvalTask	addAlgorithm(Map<String,Object> attrs, String file)
TrainTestEvalTask	addDataset(TTDataSet source)
TrainTestEvalTask	addExternalAlgorithm(ExternalAlgorithm algorithm)
<T> TrainTestEvalTask	addMetric(Class<? extends Metric<T>> metricClass) Add a metric by class.
TrainTestEvalTask	addMetric(List<String> columns, Function<Recommender,List<Object>> metric) Add a metric that takes some metric of an algorithmInfo.
TrainTestEvalTask	addMetric(Metric<?> metric) Add a metric.
TrainTestEvalTask	addMetric(MetricFactory<?> factory) Add a metric by factory.
TrainTestEvalTask	addMultiMetric(File file, List<String> columns, Function<Recommender,List<List<Object>>> metric) Add a metric that may write multiple columns per algorithmInfo.
TrainTestEvalTask	addWritePredictionChannel(Symbol channelSym) Add a channel to be recorded with predict output.
TrainTestEvalTask	addWritePredictionChannel(Symbol channelSym, String label) Deprecated. Use the predictions metric.
boolean	getCacheAllComponents()
File	getComponentCacheDirectory()
org.grouplens.lenskit.eval.traintest.ExperimentSuite	getExperiments()
org.grouplens.lenskit.eval.traintest.MeasurementSuite	getMeasurements()
org.grouplens.lenskit.eval.traintest.ExperimentOutputLayout	getOutputLayout()
org.grouplens.lenskit.eval.traintest.ExperimentOutputs	getOutputs()
File	getTaskGraphFile()
File	getTaskStatusFile()
Table	perform() Run the evaluation on the train test data source files
TrainTestEvalTask	setCacheAllComponents(boolean flag) Control whether all components are cached.
TrainTestEvalTask	setComponentCacheDirectory(File file) Set the component cache directory.
TrainTestEvalTask	setComponentCacheDirectory(String fn) Set the component cache directory by name.
TrainTestEvalTask	setIsolate(boolean iso) Deprecated. Isolate data sets instead.
TrainTestEvalTask	setOutput(File file)
TrainTestEvalTask	setOutput(String fn)
TrainTestEvalTask	setPredictOutput(File file) Deprecated.
TrainTestEvalTask	setPredictOutput(String fn) Deprecated.
TrainTestEvalTask	setRecommendOutput(File file) Deprecated.
TrainTestEvalTask	setRecommendOutput(String fn) Deprecated.
TrainTestEvalTask	setSeparateAlgorithms(boolean sep) Control whether the evaluator separates or combines algorithms.
TrainTestEvalTask	setTaskGraphFile(File f) Set an output file for a description of the task graph.
TrainTestEvalTask	setTaskGraphFile(String f) Set an output file for a description of the task graph.
TrainTestEvalTask	setTaskStatusFile(File f) Set an output file for task status updates.
TrainTestEvalTask	setTaskStatusFile(String f) Set an output file for task status updates.
TrainTestEvalTask	setUserOutput(File file)
TrainTestEvalTask	setUserOutput(String fn)

Methods inherited from class org.grouplens.lenskit.eval.AbstractTask

execute, getName, getProject, setName, setProject

Methods inherited from class com.google.common.util.concurrent.AbstractFuture

addListener, cancel, get, get, interruptTask, isCancelled, isDone, set, setException, wasInterrupted

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.google.common.util.concurrent.ListenableFuture

addListener

Methods inherited from interface java.util.concurrent.Future

cancel, get, get, isCancelled, isDone

Constructor Detail

TrainTestEvalTask

public TrainTestEvalTask()

TrainTestEvalTask

public TrainTestEvalTask(String name)

Method Detail

addDataset

public TrainTestEvalTask addDataset(TTDataSet source)

addAlgorithm

public TrainTestEvalTask addAlgorithm(AlgorithmInstance algorithm)

addAlgorithm

public TrainTestEvalTask addAlgorithm(Map<String,Object> attrs,
                                      String file)
                               throws IOException,
                                      RecommenderConfigurationException

Throws:

IOException

RecommenderConfigurationException

addExternalAlgorithm

public TrainTestEvalTask addExternalAlgorithm(ExternalAlgorithm algorithm)

addMetric

public TrainTestEvalTask addMetric(Metric<?> metric)

Add a metric.

Parameters:

metric - The metric to add.

Returns:

The task (for chaining).

addMetric

public TrainTestEvalTask addMetric(MetricFactory<?> factory)

Add a metric by factory.

Parameters:

factory - The metric factory to add.

Returns:

The task (for chaining).

addMetric

public <T> TrainTestEvalTask addMetric(Class<? extends Metric<T>> metricClass)

Add a metric by class.

Type Parameters:

T - The metric's return type.

Parameters:

metricClass - The metric class.

Returns:

The task (for chaining).

addMultiMetric

public TrainTestEvalTask addMultiMetric(File file,
                                        List<String> columns,
                                        Function<Recommender,List<List<Object>>> metric)

Add a metric that may write multiple columns per algorithmInfo.

Parameters:

file - The output file.

columns - The column headers.

metric - The metric function. It should return a list of table rows, each of which has entries corresponding to each column.

Returns:

The command (for chaining).

addMetric

public TrainTestEvalTask addMetric(List<String> columns,
                                   Function<Recommender,List<Object>> metric)

Add a metric that takes some metric of an algorithmInfo.

Parameters:

columns - The column headers.

metric - The metric function. It should return a list of table rows, each of which has entries corresponding to each column.

Returns:

The command (for chaining).

addWritePredictionChannel

public TrainTestEvalTask addWritePredictionChannel(@Nonnull
                                                   Symbol channelSym)

Add a channel to be recorded with predict output.

Parameters:

channelSym - The channel to output.

Returns:

The command (for chaining)

See Also:

addWritePredictionChannel(org.grouplens.lenskit.symbols.Symbol)

addWritePredictionChannel

@Deprecated
public TrainTestEvalTask addWritePredictionChannel(@Nonnull
                                                               Symbol channelSym,
                                                               @Nullable
                                                               String label)

Deprecated. Use the predictions metric.

Add a channel to be recorded with predict output.

Parameters:

channelSym - The channel to record, if present in the prediction output vector.

label - The column label. If null, the channel symbol's name is used.

Returns:

The command (for chaining).

See Also:

setPredictOutput(File)

setOutput

public TrainTestEvalTask setOutput(File file)

setOutput

public TrainTestEvalTask setOutput(String fn)

setUserOutput

public TrainTestEvalTask setUserOutput(File file)

setUserOutput

public TrainTestEvalTask setUserOutput(String fn)

setPredictOutput

@Deprecated
public TrainTestEvalTask setPredictOutput(File file)

Deprecated.

setPredictOutput

@Deprecated
public TrainTestEvalTask setPredictOutput(String fn)

Deprecated.

setRecommendOutput

@Deprecated
public TrainTestEvalTask setRecommendOutput(File file)

Deprecated.

setRecommendOutput

@Deprecated
public TrainTestEvalTask setRecommendOutput(String fn)

Deprecated.

setComponentCacheDirectory

public TrainTestEvalTask setComponentCacheDirectory(File file)

Set the component cache directory. This directory is used for caching components that might take a lot of memory but are shared between runs. Only meaningful if algorithms are not separated.

Parameters:

file - The cache directory.

Returns:

The task (for chaining)

See Also:

setSeparateAlgorithms(boolean)

setComponentCacheDirectory

public TrainTestEvalTask setComponentCacheDirectory(String fn)

Set the component cache directory by name.

See Also:

setComponentCacheDirectory(java.io.File)

getComponentCacheDirectory

public File getComponentCacheDirectory()

setCacheAllComponents

public TrainTestEvalTask setCacheAllComponents(boolean flag)

Control whether all components are cached. Caching all components will use more disk space, but may allow future evaluations to re-use component instances from prior runs.

Parameters:

flag - true to cache all components.

Returns:

The task (for chaining).

getCacheAllComponents

public boolean getCacheAllComponents()

setIsolate

@Deprecated
public TrainTestEvalTask setIsolate(boolean iso)

Deprecated. Isolate data sets instead.

Control whether the train-test evaluator will isolate data sets. If set to true, then each data set will be run in turn, with no inter-data-set parallelism. This can reduce memory usage for some large runs, keeping the data from only a single data set in memory at a time. Otherwise (the default), individual algorithmInfo/data-set runs may be freely intermingled.

Parameters:

iso - Whether to isolate data sets.

Returns:

The task (for chaining).

setSeparateAlgorithms

public TrainTestEvalTask setSeparateAlgorithms(boolean sep)

Control whether the evaluator separates or combines algorithms. If set to true, each algorithm instance is built without reusing components from other algorithm instances. By default, LensKit will try to reuse common components between algorithm configurations (one downside of this is that it makes build timings meaningless).

Parameters:

sep - If true, separate algorithms.

Returns:

The task (for chaining).

setTaskGraphFile

public TrainTestEvalTask setTaskGraphFile(File f)

Set an output file for a description of the task graph.

Parameters:

f - The output file.

Returns:

The task (for chaining).

setTaskGraphFile

public TrainTestEvalTask setTaskGraphFile(String f)

Set an output file for a description of the task graph.

Parameters:

f - The output file name

Returns:

The task (for chaining).

getTaskGraphFile

public File getTaskGraphFile()

setTaskStatusFile

public TrainTestEvalTask setTaskStatusFile(File f)

Set an output file for task status updates.

Parameters:

f - The output file.

Returns:

The task (for chaining).

setTaskStatusFile

public TrainTestEvalTask setTaskStatusFile(String f)

Set an output file for task status updates.

Parameters:

f - The output file name

Returns:

The task (for chaining).

getTaskStatusFile

public File getTaskStatusFile()

perform

public Table perform()
              throws TaskExecutionException,
                     InterruptedException

Run the evaluation on the train test data source files

Specified by:

perform in class AbstractTask<Table>

Returns:

The summary output table.

Throws:

TaskExecutionException - Failure of the evaluation

InterruptedException

getExperiments

public org.grouplens.lenskit.eval.traintest.ExperimentSuite getExperiments()

getMeasurements

public org.grouplens.lenskit.eval.traintest.MeasurementSuite getMeasurements()

getOutputLayout

public org.grouplens.lenskit.eval.traintest.ExperimentOutputLayout getOutputLayout()

getOutputs

public org.grouplens.lenskit.eval.traintest.ExperimentOutputs getOutputs()