PairedTTester (weka-dev 3.7.6-SNAPSHOT API)

java.lang.Object
- weka.experiment.PairedTTester

All Implemented Interfaces:: Serializable, OptionHandler, RevisionHandler, Tester

Direct Known Subclasses:: PairedCorrectedTTester

public class PairedTTester
extends Object
implements OptionHandler, Tester, RevisionHandler

Calculates T-Test statistics on data stored in a set of instances.

Valid options are:

 -D <index,index2-index4,...>
  Specify list of columns that specify a unique
  dataset.
  First and last are valid indexes. (default none)

 -R <index>
  Set the index of the column containing the run number

 -F <index>
  Set the index of the column containing the fold number

 -G <index1,index2-index4,...>
  Specify list of columns that specify a unique
  'result generator' (eg: classifier name and options).
  First and last are valid indexes. (default none)

 -S <significance level>
  Set the significance level for comparisons (default 0.05)

 -V
  Show standard deviations

 -L
  Produce table comparisons in Latex table format

 -csv
  Produce table comparisons in CSV table format

 -html
  Produce table comparisons in HTML table format

 -significance
  Produce table comparisons with only the significance values

 -gnuplot
  Produce table comparisons output suitable for GNUPlot

Version:: $Revision: 8034 $
Author:: Len Trigg (trigg@cs.waikato.ac.nz)
See Also:: Serialized Form

Constructor Summary

Constructors
Constructor and Description

PairedTTester()

Constructors
Constructor and Description
`PairedTTester()`

Method Summary

Methods
Modifier and Type	Method and Description
`void`	`assign(Tester tester)` retrieves all the settings from the given Tester
`PairedStats`	`calculateStatistics(Instance datasetSpecifier, int resultset1Index, int resultset2Index, int comparisonColumn)` Computes a paired t-test comparison for a specified dataset between two resultsets.
`boolean`	`displayResultset(int index)` Checks whether the resultset with the given index shall be displayed.
`Range`	`getDatasetKeyColumns()` Get the value of DatasetKeyColumns.
`int[]`	`getDisplayedResultsets()` Gets the indices of the the datasets that are displayed (if `null` then all are displayed).
`String`	`getDisplayName()` returns the name of the tester
`int`	`getFoldColumn()` Get the value of FoldColumn.
`Instances`	`getInstances()` Get the value of Instances.
`int`	`getNumDatasets()` Gets the number of datasets in the resultsets
`int`	`getNumResultsets()` Gets the number of resultsets in the data.
`String[]`	`getOptions()` Gets current settings of the PairedTTester.
`ResultMatrix`	`getResultMatrix()` Gets the instance that produces the output.
`Range`	`getResultsetKeyColumns()` Get the value of ResultsetKeyColumns.
`String`	`getResultsetName(int index)` Gets a string descriptive of the specified resultset.
`String`	`getRevision()` Returns the revision string.
`int`	`getRunColumn()` Get the value of RunColumn.
`boolean`	`getShowStdDevs()` Returns true if standard deviations have been requested.
`double`	`getSignificanceLevel()` Get the value of SignificanceLevel.
`int`	`getSortColumn()` Returns the column to sort on, -1 means the default sorting.
`String`	`getSortColumnName()` Returns the name of the column to sort on.
`String`	`getToolTipText()` returns a string that is displayed as tooltip on the "perform test" button in the experimenter
`String`	`header(int comparisonColumn)` Creates a "header" string describing the current resultsets.
`Enumeration`	`listOptions()` Lists options understood by this object.
`static void`	`main(String[] args)` Test the class from the command line.
`String`	`multiResultsetFull(int baseResultset, int comparisonColumn)` Creates a comparison table where a base resultset is compared to the other resultsets.
`String`	`multiResultsetRanking(int comparisonColumn)` returns a ranking of the resultsets
`String`	`multiResultsetSummary(int comparisonColumn)` Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.
`int[][]`	`multiResultsetWins(int comparisonColumn, int[][] nonSigWin)` Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.
`String`	`resultsetKey()` Creates a key that maps resultset numbers to their descriptions.
`void`	`setDatasetKeyColumns(Range newDatasetKeyColumns)` Set the value of DatasetKeyColumns.
`void`	`setDisplayedResultsets(int[] cols)` Sets the indicies of the datasets to display (`null` means all).
`void`	`setFoldColumn(int newFoldColumn)` Set the value of FoldColumn.
`void`	`setInstances(Instances newInstances)` Set the value of Instances.
`void`	`setOptions(String[] options)` Parses a given list of options.
`void`	`setResultMatrix(ResultMatrix matrix)` Sets the matrix to use to produce the output.
`void`	`setResultsetKeyColumns(Range newResultsetKeyColumns)` Set the value of ResultsetKeyColumns.
`void`	`setRunColumn(int newRunColumn)` Set the value of RunColumn.
`void`	`setShowStdDevs(boolean s)` Set whether standard deviations are displayed or not.
`void`	`setSignificanceLevel(double newSignificanceLevel)` Set the value of SignificanceLevel.
`void`	`setSortColumn(int newSortColumn)` Set the column to sort on, -1 means the default sorting.

Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - PairedTTester
```
public PairedTTester()
```
- Method Detail
  - setResultMatrix
```
public void setResultMatrix(ResultMatrix matrix)
```
    Sets the matrix to use to produce the output.
    
    Specified by:
    
    setResultMatrix in interface Tester
    
    Parameters:
    matrix - the instance to use to produce the output
    See Also:
    ResultMatrix
  - getResultMatrix
```
public ResultMatrix getResultMatrix()
```
    Gets the instance that produces the output.
    
    Specified by:
    
    getResultMatrix in interface Tester
    
    Returns:
    the instance to produce the output
  - setShowStdDevs
```
public void setShowStdDevs(boolean s)
```
    Set whether standard deviations are displayed or not.
    
    Specified by:
    
    setShowStdDevs in interface Tester
    
    Parameters:
    s - true if standard deviations are to be displayed
  - getShowStdDevs
```
public boolean getShowStdDevs()
```
    Returns true if standard deviations have been requested.
    
    Specified by:
    
    getShowStdDevs in interface Tester
    
    Returns:
    true if standard deviations are to be displayed.
  - getNumDatasets
```
public int getNumDatasets()
```
    Gets the number of datasets in the resultsets
    
    Specified by:
    
    getNumDatasets in interface Tester
    
    Returns:
    the number of datasets in the resultsets
  - getNumResultsets
```
public int getNumResultsets()
```
    Gets the number of resultsets in the data.
    
    Specified by:
    
    getNumResultsets in interface Tester
    
    Returns:
    the number of resultsets in the data
  - getResultsetName
```
public String getResultsetName(int index)
```
    Gets a string descriptive of the specified resultset.
    
    Specified by:
    
    getResultsetName in interface Tester
    
    Parameters:
    index - the index of the resultset
    
    Returns:
    a descriptive string for the resultset
  - displayResultset
```
public boolean displayResultset(int index)
```
    Checks whether the resultset with the given index shall be displayed.
    
    Specified by:
    
    displayResultset in interface Tester
    
    Parameters:
    index - the index of the resultset to check whether it shall be displayed
    
    Returns:
    whether the specified resultset is displayed
  - calculateStatistics
```
public PairedStats calculateStatistics(Instance datasetSpecifier,
                              int resultset1Index,
                              int resultset2Index,
                              int comparisonColumn)
                                throws Exception
```
    Computes a paired t-test comparison for a specified dataset between two resultsets.
    
    Specified by:
    
    calculateStatistics in interface Tester
    
    Parameters:
    datasetSpecifier - the dataset specifier
    resultset1Index - the index of the first resultset
    resultset2Index - the index of the second resultset
    comparisonColumn - the column containing values to compare
    
    Returns:
    the results of the paired comparison
    
    Throws:
    
    Exception - if an error occurs
  - resultsetKey
```
public String resultsetKey()
```
    Creates a key that maps resultset numbers to their descriptions.
    
    Specified by:
    
    resultsetKey in interface Tester
    
    Returns:
    a value of type 'String'
  - header
```
public String header(int comparisonColumn)
```
    Creates a "header" string describing the current resultsets.
    
    Specified by:
    
    header in interface Tester
    
    Parameters:
    comparisonColumn - a value of type 'int'
    
    Returns:
    a value of type 'String'
  - multiResultsetWins
```
public int[][] multiResultsetWins(int comparisonColumn,
                         int[][] nonSigWin)
                           throws Exception
```
    Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.
    
    Specified by:
    
    multiResultsetWins in interface Tester
    
    Parameters:
    comparisonColumn - the index of the comparison column
    nonSigWin - for storing the non-significant wins
    
    Returns:
    a 2d array where element [i][j] is the number of times resultset j performed significantly better than resultset i.
    
    Throws:
    
    Exception - if an error occurs
  - multiResultsetSummary
```
public String multiResultsetSummary(int comparisonColumn)
                             throws Exception
```
    Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other. The results are summarized in a table.
    
    Specified by:
    
    multiResultsetSummary in interface Tester
    
    Parameters:
    comparisonColumn - the index of the comparison column
    
    Returns:
    the results in a string
    
    Throws:
    
    Exception - if an error occurs
  - multiResultsetRanking
```
public String multiResultsetRanking(int comparisonColumn)
                             throws Exception
```
    returns a ranking of the resultsets
    
    Specified by:
    
    multiResultsetRanking in interface Tester
    
    Parameters:
    comparisonColumn - the column to compare with
    
    Returns:
    the ranking
    
    Throws:
    
    Exception - if something goes wrong
  - multiResultsetFull
```
public String multiResultsetFull(int baseResultset,
                        int comparisonColumn)
                          throws Exception
```
    Creates a comparison table where a base resultset is compared to the other resultsets. Results are presented for every dataset.
    
    Specified by:
    
    multiResultsetFull in interface Tester
    
    Parameters:
    baseResultset - the index of the base resultset
    comparisonColumn - the index of the column to compare over
    
    Returns:
    the comparison table string
    
    Throws:
    
    Exception - if an error occurs
  - listOptions
```
public Enumeration listOptions()
```
    Lists options understood by this object.
    
    Specified by:
    
    listOptions in interface OptionHandler
    
    Returns:
    an enumeration of Options.
  - setOptions
```
public void setOptions(String[] options)
                throws Exception
```
    Parses a given list of options.
    Valid options are:
```
 -D <index,index2-index4,...>
  Specify list of columns that specify a unique
  dataset.
  First and last are valid indexes. (default none)
```
```
 -R <index>
  Set the index of the column containing the run number
```
```
 -F <index>
  Set the index of the column containing the fold number
```
```
 -G <index1,index2-index4,...>
  Specify list of columns that specify a unique
  'result generator' (eg: classifier name and options).
  First and last are valid indexes. (default none)
```
```
 -S <significance level>
  Set the significance level for comparisons (default 0.05)
```
```
 -V
  Show standard deviations
```
```
 -L
  Produce table comparisons in Latex table format
```
```
 -csv
  Produce table comparisons in CSV table format
```
```
 -html
  Produce table comparisons in HTML table format
```
```
 -significance
  Produce table comparisons with only the significance values
```
```
 -gnuplot
  Produce table comparisons output suitable for GNUPlot
```
    Specified by:
    
    setOptions in interface OptionHandler
    
    Parameters:
    options - an array containing options to set.
    
    Throws:
    
    Exception - if invalid options are given
  - getOptions
```
public String[] getOptions()
```
    Gets current settings of the PairedTTester.
    
    Specified by:
    
    getOptions in interface OptionHandler
    
    Returns:
    an array of strings containing current options.
  - getResultsetKeyColumns
```
public Range getResultsetKeyColumns()
```
    Get the value of ResultsetKeyColumns.
    
    Specified by:
    
    getResultsetKeyColumns in interface Tester
    
    Returns:
    Value of ResultsetKeyColumns.
  - setResultsetKeyColumns
```
public void setResultsetKeyColumns(Range newResultsetKeyColumns)
```
    Set the value of ResultsetKeyColumns.
    
    Specified by:
    
    setResultsetKeyColumns in interface Tester
    
    Parameters:
    newResultsetKeyColumns - Value to assign to ResultsetKeyColumns.
  - getDisplayedResultsets
```
public int[] getDisplayedResultsets()
```
    Gets the indices of the the datasets that are displayed (if null then all are displayed). The base is always displayed.
    
    Specified by:
    
    getDisplayedResultsets in interface Tester
    
    Returns:
    the indices of the datasets to display
  - setDisplayedResultsets
```
public void setDisplayedResultsets(int[] cols)
```
    Sets the indicies of the datasets to display (null means all). The base is always displayed.
    
    Specified by:
    
    setDisplayedResultsets in interface Tester
    
    Parameters:
    cols - the indices of the datasets to display
  - getSignificanceLevel
```
public double getSignificanceLevel()
```
    Get the value of SignificanceLevel.
    
    Specified by:
    
    getSignificanceLevel in interface Tester
    
    Returns:
    Value of SignificanceLevel.
  - setSignificanceLevel
```
public void setSignificanceLevel(double newSignificanceLevel)
```
    Set the value of SignificanceLevel.
    
    Specified by:
    
    setSignificanceLevel in interface Tester
    
    Parameters:
    newSignificanceLevel - Value to assign to SignificanceLevel.
  - getDatasetKeyColumns
```
public Range getDatasetKeyColumns()
```
    Get the value of DatasetKeyColumns.
    
    Specified by:
    
    getDatasetKeyColumns in interface Tester
    
    Returns:
    Value of DatasetKeyColumns.
  - setDatasetKeyColumns
```
public void setDatasetKeyColumns(Range newDatasetKeyColumns)
```
    Set the value of DatasetKeyColumns.
    
    Specified by:
    
    setDatasetKeyColumns in interface Tester
    
    Parameters:
    newDatasetKeyColumns - Value to assign to DatasetKeyColumns.
  - getRunColumn
```
public int getRunColumn()
```
    Get the value of RunColumn.
    
    Specified by:
    
    getRunColumn in interface Tester
    
    Returns:
    Value of RunColumn.
  - setRunColumn
```
public void setRunColumn(int newRunColumn)
```
    Set the value of RunColumn.
    
    Specified by:
    
    setRunColumn in interface Tester
    
    Parameters:
    newRunColumn - Value to assign to RunColumn.
  - getFoldColumn
```
public int getFoldColumn()
```
    Get the value of FoldColumn.
    
    Specified by:
    
    getFoldColumn in interface Tester
    
    Returns:
    Value of FoldColumn.
  - setFoldColumn
```
public void setFoldColumn(int newFoldColumn)
```
    Set the value of FoldColumn.
    
    Specified by:
    
    setFoldColumn in interface Tester
    
    Parameters:
    newFoldColumn - Value to assign to FoldColumn.
  - getSortColumnName
```
public String getSortColumnName()
```
    Returns the name of the column to sort on.
    
    Specified by:
    
    getSortColumnName in interface Tester
    
    Returns:
    the name of the column to sort on.
  - getSortColumn
```
public int getSortColumn()
```
    Returns the column to sort on, -1 means the default sorting.
    
    Specified by:
    
    getSortColumn in interface Tester
    
    Returns:
    the column to sort on.
  - setSortColumn
```
public void setSortColumn(int newSortColumn)
```
    Set the column to sort on, -1 means the default sorting.
    
    Specified by:
    
    setSortColumn in interface Tester
    
    Parameters:
    newSortColumn - the new sort column.
  - getInstances
```
public Instances getInstances()
```
    Get the value of Instances.
    
    Specified by:
    
    getInstances in interface Tester
    
    Returns:
    Value of Instances.
  - setInstances
```
public void setInstances(Instances newInstances)
```
    Set the value of Instances.
    
    Specified by:
    
    setInstances in interface Tester
    
    Parameters:
    newInstances - Value to assign to Instances.
  - assign
```
public void assign(Tester tester)
```
    retrieves all the settings from the given Tester
    
    Specified by:
    
    assign in interface Tester
    
    Parameters:
    tester - the Tester to get the settings from
  - getToolTipText
```
public String getToolTipText()
```
    returns a string that is displayed as tooltip on the "perform test" button in the experimenter
    
    Specified by:
    
    getToolTipText in interface Tester
    
    Returns:
    the tool tip
  - getDisplayName
```
public String getDisplayName()
```
    returns the name of the tester
    
    Specified by:
    
    getDisplayName in interface Tester
    
    Returns:
    the display name
  - getRevision
```
public String getRevision()
```
    Returns the revision string.
    
    Specified by:
    
    getRevision in interface RevisionHandler
    
    Returns:
    the revision
  - main
```
public static void main(String[] args)
```
    Test the class from the command line.
    
    Parameters:
    args - contains options for the instance ttests

Class PairedTTester

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

PairedTTester

Method Detail

setResultMatrix

getResultMatrix

setShowStdDevs

getShowStdDevs

getNumDatasets

getNumResultsets

getResultsetName

displayResultset

calculateStatistics

resultsetKey

header

multiResultsetWins

multiResultsetSummary

multiResultsetRanking

multiResultsetFull

listOptions

setOptions

getOptions

getResultsetKeyColumns

setResultsetKeyColumns

getDisplayedResultsets

setDisplayedResultsets

getSignificanceLevel

setSignificanceLevel

getDatasetKeyColumns

setDatasetKeyColumns

getRunColumn

setRunColumn

getFoldColumn

setFoldColumn

getSortColumnName

getSortColumn

setSortColumn

getInstances

setInstances

assign

getToolTipText

getDisplayName

getRevision

main