Cardinality-Specified Sparse Regression for Variable Selection : Comparisons Against Lasso and All Possible Regression

Abstract

Variable selection in regression analysis is considered in this paper, with the selection purposed to find the best subset of predictors. The number of all possible subsets is 2p-1 for the full set of p predictors. The best one among 2p-1 subsets can be found through all possible regression (APREG). For avoiding the difficulty that APREG is time-consuming, heuristic variable selection procedures (such as variable increasing/decreasing and stepwise ones) used to be prevalent, but can be considered as useless now, for the following reasons: As described in the first half of this paper, APREG can be performed instantly when p ≤ 13, and we can consider that the variable selection is achieved efficiently by sparse regression procedures even if p > 13. Among those procedures, the one called cardinality-specified regression (CSREG) is focused on and compared with well-known lasso, in this paper. CSREG differs from lasso as follows: The ordinary regression function is minimized subject to the cardinality of regression coefficients being specified in CSREG, while the regression function plus a penalty function is minimized in lasso. The results following from the difference are shown mathematically and numerically in this paper. The mathematical results are summarized as follows: [1] The CSREG solution is scale invariant and equivalent to the APREG counterpart, if the iterative algorithm for CSREG provides the global minimizers; [2] LASSO cannot provide the estimates of coefficients that are same as the APREG counterparts. These results are numerically demonstrated by a real data example and ascertained in a simulation study. In the simulation study, CSREG and lasso are performed for the data sets generated from the true subsets in some conditions. We present the detailed results for how CSREG and lasso differ in variable selection. Additionally, we report and discuss how well the true subsets can be recovered in CSREG and lasso.