Suchergebnisse

AbstractWe give a characterization of maximum entropy/minimum relative entropy inference by providing two 'strong entropy concentration' theorems. These theorems unify and generalize Jaynes''concentration phenomenon' and Van Campenhout and Cover's 'conditional limit theorem'. The theorems characterize exactly in what sense a prior distribution Q conditioned on a given constraint and the distribution  minimizing D(P ‖ Q) over all P satisfying the constraint are 'close' to each other. We then apply our theorems to establish the relationship between entropy concentration and a game‐theoretic characterization of maximum entropy inference of Topsøe and others.

AbstractThe no-free-lunch theorems promote a skeptical conclusion that all possible machine learning algorithms equally lack justification. But how could this leave room for a learning theory, that shows that some algorithms are better than others? Drawing parallels to the philosophy of induction, we point out that the no-free-lunch results presuppose a conception of learning algorithms as purely data-driven. On this conception, every algorithm must have an inherent inductive bias, that wants justification. We argue that many standard learning algorithms should rather be understood as model-dependent: in each application they also require for input a model, representing a bias. Generic algorithms themselves, they can be given a model-relative justification.

Abstract
This article comments on the article of Thorn and Schurz in this volume and focuses on, what we call, the problem of parasitic experts. We discuss that both meta- induction and crowd wisdom can be understood as pertaining to absolute reliability rather than comparative optimality, and we suggest that the involvement of reliability will provide a handle on this problem.