datanugget - Create, Optimize, and Refine Data Nuggets
Creating, optimizing and refining data nuggets. Data
nuggets reduce a large dataset into a small collection of
nuggets of data, each containing a center (location), weight
(importance), and scale (variability) parameter. Data nugget
centers are selected based on a space-filling maximum-entropy
scheme. Data nugget weights are created by counting the number
observations closest to a given data nugget center. We then say
the data nugget 'contains' these observations and the data
nugget center is recalculated as the mean of these
observations. Data nugget scales are created by calculating the
trace of the covariance matrix of the observations contained
within a data nugget divided by the dimension of the dataset.
The optimal number of data nuggets is determined data-driven
based on the relative second-order differences of propensity
score indices. Data nuggets are refined by 'splitting' data
nuggets which have high scales or elongated shapes (defined as
the ratio of the two largest eigenvalues of the covariance
matrix of the observations contained within the data nugget).
