Divide-and-Conquer Information-Based Optimal Subdata Selection Algorithm

The information-based optimal subdata selection (IBOSS) is a computationally efficient method to select informative data points from large data sets through processing full data by columns. However, when the volume of a data set is too large to be processed in the available memory of a machine, it i...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Journal of statistical theory and practice Ročník 13; číslo 3
Hlavní autor: Wang, HaiYing
Médium: Journal Article
Jazyk:angličtina
Vydáno: Cham Springer International Publishing 01.09.2019
Témata:
Algorithms
Analysis and Advanced Methodologies in the Design of Experiments
Mathematics and Statistics
Original Article
Probability Theory and Stochastic Processes
Statistical Theory and Methods
Statistics
D-optimality
Big data
Subdata
Linear regression
Information matrix
ISSN:1559-8608, 1559-8616
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:The information-based optimal subdata selection (IBOSS) is a computationally efficient method to select informative data points from large data sets through processing full data by columns. However, when the volume of a data set is too large to be processed in the available memory of a machine, it is infeasible to implement the IBOSS procedure. This paper develops a divide-and-conquer IBOSS approach to solving this problem, in which the full data set is divided into smaller partitions to be loaded into the memory and then subsets of data are selected from each partition using the IBOSS algorithm. We derive both finite sample properties and asymptotic properties of the resulting estimator. Asymptotic results show that if the full data set is partitioned randomly and the number of partitions is not very large, then the resultant estimator has the same estimation efficiency as the original IBOSS estimator. We also carry out numerical experiments to evaluate the empirical performance of the proposed method.
ISSN:1559-8608
1559-8616
DOI:10.1007/s42519-019-0048-5