Most variable selection methods are based on the comparison of the model containing a variable to the model without this variable. Only the Best Subset selection evaluates models one by one and ranks them according to the scoring value of the individual model.
The statistic for testing the inclusion or exclusion of a single variable in a likelihood-based model (Logistic Regression , Survival Analysis, Bivariate Probit and Time Series Analysis ) is based on the Likelihood Ratio statistics with an approximate chi-square distribution with one degree of freedom. For variable inclusion it is equal to
and for the variable exclusion it is equal to
where L is the maximum likelihood value of the fitted model.
The statistic for testing the inclusion or exclusion of a single variable in the Linear Regression model is based on the Type2SS statistic. For the variable inclusion it is equal to
and for the variable exclusion it is equal to
where SSE it the residual sum of squared errors of the fitted model.
Both statistics estimate the importance of the variable by measuring the change of the scoring value resulting from inserting or removing the variable from the model. In the case of Linear Regression models, each model is evaluated by the residual sum of squared errors (SSE) and in the case of likelihood-based models by the maximum likelihood value of the fitted model.
Regardless of the selected variable selection method, the final model will always contain all of the obligatory variables and no inactive variables.
For the Linear Regression, Logistic Regression, TimeSeries and Survival Analysis models all automatic variable selection methods take into account only the variables selected as active in the current function settings.
Unlike all the other statistical modules, the Bivariate Probit model variable selection refers only to variable preselection for the main bivariate probit model. Preselection means building and selecting variables for two supplementary logistic models independently (one for each equation). Variables which are significant (with respect to the Model Entry Level and Model Leave Level thresholds) become explanatory variables in the bivariate model. Next, the final bivariate model is built without any subsequent variable selection processes.
In this case, all variable selection methods take into account only the variables selected as active or active12 for the first supplementary logistic model (which pre-estimates the default equation parameters), and active2 or active12 for the second supplementary logistic model (which pre-estimates the censoring equation parameters).
A full model is the default case, in which no automatic variable selection is performed. The final model contains all variables selected as active or obligatory in the current function settings and does not contain any inactive variables.
The Forward Selection method starts without any active explanatory variable and with all obligatory variables in the model (and additional constant term only if the Intercept option is selected in the corresponding Algorithm Settings). Variables selected as active are subsequently added to the model according to the decreasing value of the model-dependent Residual Statistic (and the increasing p-value for this statistic). In each step the variable whose Residual Statistic yields the smallest p-value is added. Variables are added as long as the Residual Statistic p-value remains below a specified maximum value (Model Entry Level).
The method terminates when the addition of any of the remaining active variables yields a partial p-value greater than Model Entry Level .
This method has two limitations. Some of the variables never get into the model and hence their importance is never determined. Another limitation is that a variable once included in the model remains there throughout the process, even if it loses its stated significance after the inclusion of some other variables.
The Backward Elimination method starts with all active and obligatory variables included into the model and subsequently eliminates the active variables. For each variable, the value of the model-dependent Residual Statistic (and the p-value for this statistic) are computed. The one whose p-value is greater than Model Leave Level is considered to be the least useful and is therefore excluded from the model.
The process continues until no further active variable can be removed according to the elimination criterion (if the p-values for all attributes included in model are less than value specified in Leave Level option). Variables chosen as obligatory always stay in the model.
The limitation of this method is that model can include highly correlated variables and the estimation of the full model (in the first step) can be impossible.
The Stepwise method is a modification of the forward selection method and begins with the forward selection step. At each step of the selection process, after a new variable is added to the model, the value of the model-dependent Residual Statistic (and the p-value for this statistic) are computed for all variables already in the model. If the largest of these p-values is greater than Model Leave Level the variable is excluded. After the included variables have been examined for exclusion, the excluded variables are re-examined for inclusion.
At each step of the process there can be at most one inclusion (forward step), followed by one or more exclusions (backward step). The method stops if no more variables can be included (with respect to the Model Entry Level criterion) or if the only available variable is the one just excluded. Variables chosen as obligatory always stay in the model.
The Best Subset method searches through all possible subsets of variables in the model. Subsets are processed in the order of increasing magnitude. For each model size (i.e. the number of variables included), the best model (with respect to the model-dependent Scoring Statistic) is reported in the Model Statistics panel of the corresponding model. If the Max Model Size has been specified, the Best Subset method returns the best model of the specified size (refer also to the comments under Max Model Size option). In the other case, it returns the model containing all variables selected as active or obligatory in the current function settings (i.e. the same model as the full option).
The maximum allowed number of active variables for the best subset method is 30, but in practice it is recommended to use few active variables at once (no more than 10), due to the very high computational cost of this method (exponential in the number of active variables).
The progress of the Best Subset selection will be displayed in AdvancedMiner log-viewer component after each 50 newly estimated models, and also on the progress bar located in the bottom right side of window. The process can be terminated, in such case the best models found so far for each model size will be presented in Model Statistics.