Model comparison is discussed from an information theoretic point of view. In particular the posterior predictive entropy is related to the target yielding DIC and modifications thereof. The adequacy of criteria for posterior predictive model comparison is also investigated depending on the comparison to be made. In particular variable selection as a special problem of model choice is formalized in different ways according to whether the comparison is a comparison across models or within an encompassing model and whether a joint or conditional sampling scheme is applied. DIC has been devised for comparisons across models. Its use in variable selection and that of other criteria is illustrated for a simulated data set.
AbstractSample selection arises when the outcome of interest is partially observed in a study. A common challenge is the requirement for exclusion restrictions. That is, some of the covariates affecting missingness mechanism do not affect the outcome. The drive to establish this requirement often leads to the inclusion of irrelevant variables in the model. A suboptimal solution is the use of classical variable selection criteria such as AIC and BIC, and traditional variable selection procedures such as stepwise selection. These methods are unstable when there is limited expert knowledge about the variables to include in the model. To address this, we propose the use of adaptive Lasso for variable selection and parameter estimation in both the selection and outcome submodels simultaneously in the absence of exclusion restrictions. By using the maximum likelihood estimator of the sample selection model, we constructed a loss function similar to the least squares regression problem up to a constant, and minimized its penalized version using an efficient algorithm. We show that the estimator, with proper choice of regularization parameter, is consistent and possesses the oracle properties. The method is compared to Lasso and adaptively weighted $$L_{1}$$
L 1
penalized Two-step method. We applied the methods to the well-known Ambulatory Expenditure Data.
Mención Internacional en el título de doctor ; This thesis has been developed at University Carlos III of Madrid, motivated through a collaboration with the Gregorio Marañón General University Hospital, in Madrid. It is framed within the field of Penalized Linear Models, specifically Variable Selection in Regression, Classification and Survival Models, but it also explores other techniques such as Variable Clustering and Semi-Supervised Learning. In recent years, variable selection techniques based on penalized models have gained considerable importance. With the advance of technologies in the last decade, it has been possible to collect and process huge volumes of data with algorithms of greater computational complexity. However, although it seemed that models that provided simple and interpretable solutions were going to be definitively displaced by more complex ones, they have still proved to be very useful. Indeed, in a practical sense, a model that is capable of filtering important information, easily extrapolated and interpreted by a human, is often more valuable than a more complex model that is incapable of providing any kind of feedback on the underlying problem, even when the latter offers better predictions. This thesis focuses on high dimensional problems, in which the number of variables is of the same order or larger than the sample size. In this type of problems, restrictions that eliminate variables from the model often lead to better performance and interpretability of the results. To adjust linear regression in high dimension the Sparse Group Lasso regularization method has proven to be very efficient. However, in order to use the Sparse Group Lasso in practice, there are two critical aspects on which the solution depends: the correct selection of the regularization parameters, and a prior specification of groups of variables. Very little research has focused on algorithms for the selection of the regularization parameters of the Sparse Group Lasso, and none has explored the issue of the grouping and how to relax this restriction that in practice is an obstacle to using this method. The main objective of this thesis is to propose new methods of variable selection in generalized linear models. This thesis explores the Sparse Group Lasso regularization method, analyzing in detail the correct selection of the regularization parameters, and finally relaxing the problem of group specification by introducing a new variable clustering algorithm based on the Sparse Group Lasso, but much more flexible and that extends it. In a parallel but related line of research, this thesis reveals a connection between penalized linear models and semi-supervised learning. This thesis is structured as a compendium of articles, divided into four chapters. Each chapter has a structure and contents independent from the rest, however, all of them follow a common line. First, variable selection methods based on regularization are introduced, describing the optimization problem that appears and a numerical algorithm to approximate its solution when a term of the objective function is not differentiable. The latter occurs naturally when penalties inducing variable selection are added. A contribution of this work is the iterative Sparse Group Lasso, which is an algorithm to obtain the estimation of the coefficients of the Sparse Group Lasso model, without the need to specify the regularization parameters. It uses coordinate descent for the parameters, while approximating the error function in a validation sample. Moreover, with respect to the traditional Sparse Group Lasso, this new proposal considers a more general penalty, where each group has a flexible weight. A separate chapter presents an extension that uses the iterative Sparse Group Lasso to order the variables in the model according to a defined importance index. The introduction of this index is motivated by problems in which there are a large number of variables, only a few of which are directly related to the response variable. This methodology is applied to genetic data, revealing promising results. A further significant contribution of this thesis is the Group Linear Algorithm with Sparse Principal decomposition, which is also motivated by problems in which only a small number of variables influence the response variable. However, unlike other methodologies, in this case the relevant variables are not necessarily among the observed data. This makes it a potentially powerful method, adaptable to multiple scenarios, which is also, as a side effect, a supervised variable clustering algorithm. Moreover, it can be interpreted as an extension of the Sparse Group Lasso that does not require an initial specification of the groups. From a computational point of view, this paper presents an organized framework for solving problems in which the objective function is a linear combination of a differentiable error term and a penalty. The flexibility of this implementation allows it to be applied to problems in very different contexts, for example, the proposed Generalized Elastic Net for semisupervised learning. Regarding its main objective, this thesis offers a framework for the exploration of generalized interpretable models. In the last chapter, in addition to compiling a summary of the contributions of the thesis, future lines of work in the scope of the thesis are included. ; Esta tesis se ha desarrollado en la Universidad Carlos III de Madrid motivada por una colaboración de investigación con el Hospital General Universitario Gregorio Marañón, en Madrid. Está enmarcada dentro del campo de los Modelos Lineales Penalizados, concretamente Selección de Variables en Modelos de Regresión, Clasificación y Supervivencia, pero también explora otras técnicas como Clustering de Variables y Aprendizaje Semi-Supervisado. En los últimos años, las técnicas de selección de variables basadas en modelos penalizados han cobrado notable importancia. Con el avance de las tecnologías en la última década, se ha conseguido recopilar y tratar enormes volúmenes de datos con algoritmos de una complejidad computacional superior. Sin embargo, aunque parecía que los modelos que aportaban soluciones sencillas e interpretables iban a ser definitivamente desplazados por otros más complejos, han resultado ser todavía muy útiles. De hecho, en un sentido práctico, muchas veces tiene más valor un modelo que sea capaz de filtrar información importante, fácilmente extrapolable e interpretable por un humano, que otro más complejo incapaz de aportar ningún tipo de retroalimentación al problema de fondo, incluso cuando este último ofrezca mejores predicciones. Esta tesis se enfoca en problemas de alta dimensión, en los cuales el número de variables es del mismo orden o superior al tamaño muestral. En este tipo de problemas, restricciones que eliminen variables del modelo a menudo conducen a un mejor desempeño e interpretabilidad de los resultados. Para ajustar regresión lineal en alta dimensión el método de regularización Sparse Group Lasso ha demostrado ser muy eficiente. No obstante, para utilizar en la práctica el Sparse Group Lasso, hay que tener en cuenta dos aspectos fundamentales de los cuales depende la solución, que son la correcta selección de los parámetros de regularización, y una especificación previa de grupos de variables. Muy pocas investigaciones se han centrado en algoritmos para la selección de los parámetros de regularización del Sparse Group Lasso, y ninguna ha explorado el tema de la agrupación y cómo relajar esta restricción que en la práctica constituye una barrera para utilizar este método. El principal objetivo de esta tesis es proponer nuevos métodos de selección de variables en modelos lineales generalizados. Esta tesis explora el método de regularización Sparse Group Lasso, analizando detalladamente la correcta selección de los parámetros de regularización, y finalmente relajando el problema de la especificación de los grupos mediante un nuevo algoritmo de agrupación de variables basado en el Sparse Group Lasso, pero mucho más flexible y que lo extiende. En una línea de investigación paralela, pero relacionada, esta tesis revela una conexión entre los modelos lineales penalizados y el aprendizaje semi-supervisado. Esta tesis está estructurada en formato por compendio de artículos, dividida en cuatro capítulos. Cada capítulo tiene una estructura y contenidos independiente del resto, sin embargo, siguen todos un eje común. Primeramente, se introducen los métodos de selección de variables basados en regularización, describiendo el problema de optimización que aparece y un algoritmo numérico para aproximar su solución cuando una parte de la función objetivo no es diferenciable. Esto último ocurre de manera natural cuando se añaden penalizaciones que inducen selección de variables. Una de las aportaciones de este trabajo es el iterative Sparse Group Lasso, que es un algoritmo para obtener la estimación de los coeficientes del modelo Sparse Group Lasso, sin la necesidad de especificar los parámetros de regularización. Utiliza descenso por coordenadas para los parámetros, mientras aproxima la función de error en una muestra de validación. Además, con respecto al Sparse Group Lasso clásico, esta nueva propuesta considera una penalización más general, donde cada grupo tiene un peso flexible. En otro capítulo se presenta una extensión que utiliza el iterative Sparse Group Lasso para ordenar las variables del modelo según un índice de importancia definido. La introducción de este índice está motivada por problemas en los cuales hay un número elevado de variables, de las cuales solamente unas pocas están relacionadas directamente con la variable respuesta. Esta metodología es aplicada a unos datos genéticos, mostrando resultados prometedores. Otra importante aportación de esta tesis es el Group Linear Algorithm with Sparse Principal decomposition, que está motivado también por problemas en los cuales solamente un número reducido de variables influye en la variable respuesta. Sin embargo, a diferencia de otras metodologías, en este caso las variables influyentes no necesariamente están entre las características observadas. Esto lo convierte en un método muy potente, adaptable a múltiples escenarios, que además, como efecto secundario, es un algoritmo supervisado de agrupación de variables. En un sentido, puede interpretarse como una extensión del Sparse Group Lasso que no requiere una especificación inicial de los grupos. Desde un punto de vista computacional, este trabajo presenta un enfoque organizado para resolver problemas en los cuales la función objetivo es una combinación lineal de un término de error diferenciable y una penalización. La flexibilidad de esta implementación le permite ser aplicada a problemas en contextos muy diferentes, por ejemplo, el Generalized Elastic Net propuesto para aprendizaje semi-supervisado. Con relación a su principal objetivo, esta tesis ofrece un marco para la investigación de modelos generalizados interpretables. En el último capítulo, además de recopilarse un resumen de las aportaciones de la tesis, se incluyen líneas de trabajo futuro en el ámbito de la temática de la tesis. ; Simulations in Sections 3.3 and 3.4 have been carried out in Uranus, a supercomputer cluster located at Universidad Carlos III de Madrid and funded jointly by EU-FEDER funds and by the Spanish Government via the National Projects No. UNC313-4E-2361, No. ENE2009-12213- C03-03, No. ENE2012-33219 and No. ENE2015-68265-P. ; Programa de Doctorado en Ingeniería Matemática por la Universidad Carlos III de Madrid ; Presidente: María Luz Durban Reguera.- Secretario: Alberto José Ferrer Riquelme.- Vocal: Jeff Goldsmith
This dissertation considers the topics of prediction and variable selection for the applied political scientist, particularly in the context of high dimensional data.In Chapter 1, we consider the puzzle of why highly significant variables aren't automatically good predictors. This problem occurs in both simple and complex data. We offer explanations and statistical insights into the puzzle. We suggest shifting the research agenda toward searching for a criterion to locate highly predictive variables rather than highly significant variables. We offer an alternative approach, the Partition Retention (PR) method, which was effective in reducing prediction error from 30% to 8% on a long studied breast cancer data set.In Chapter 2, we propose approaching prediction from a framework grounded in finding the correct prediction rates of variables. While intuitively obvious, not nearly enough attention has been paid to creating a clear theoretical framework for prediction. We present an objective function for prediction rates and consider but ultimately reject an estimator based on the sample analog of the solution due to its inability to distinguish predictive variables from noisy ones, which leads to an inability to estimate it. We offer an alternative solution and demonstrate that the PR's I-score asymptotically approaches this alternative solution. The I-score for a variable set can be written as an asymptotic lower bound for the correct prediction rate. We offer simulations and applications of the I-score on real data.In Chapter 3, I propose a new approach to predicting civil war onsets that emphasizes variable selection. A good variable selection approach should search for variables based on a criterion of predictivity and find variable interactions. I suggest the PR method to conduct variable selection and illustrate with simulations and an application to civil wars data, comparing results with alternative approaches. The PR identifies variable sets, some as large as 5 or 6 variables, to predict war onsets. Using these variable sets to predict boosts correct prediction rates on out of sample data from 78.98% to 98.05%. The application demonstrates gains in prediction rates for political phenomena like civil wars when including a research step for variable selection.
AbstractModels with high‐dimensional covariates arise frequently in economics and other fields. Often, only a few covariates have important effects on the dependent variable. When this happens, the model is said to be sparse. In applications, however, it is not known which covariates are important and which are not. This paper reviews methods for discriminating between important and unimportant covariates with particular attention given to methods that discriminate correctly with probability approaching 1 as the sample size increases. Methods are available for a wide variety of linear, nonlinear, semiparametric and nonparametric models. The performance of some of these methods in finite samples is illustrated through Monte Carlo simulations and an empirical example.
