Regression vs. Classification
Two numbers, one question each
Every prediction problem begins with the same question: what kind of quantity is being predicted? If it is a number, such as next month's revenue, a patient's blood pressure, or how many minutes a flight will be delayed, the problem is one of regression. If it is a category, such as spam versus not spam, one of four tumor types, or whether a customer will churn, the problem is one of classification. The distinction is tidier in principle than in practice: logistic regression predicts a category, yet it does so by estimating a probability, which is a number, so it sits somewhat awkwardly between the two . Several of the algorithms below make no distinction between the two problem types at all; the underlying idea is the same, and only the final step of reading off an answer differs.
Separately, every learning algorithm below makes its own commitment about how much structure to assume before seeing any data. Some commit to a shape in advance, such as a straight line or a curve of a chosen degree, and thereafter only need to find the best-fitting version of that shape. Additional data sharpens confidence in the fitted numbers, but the shape itself never changes . Others commit to no shape at all, letting the data determine the boundary or curve directly, however complicated that turns out to be. This is powerful, but trustworthy only once enough data exists to support it . The first kind is called parametric, the second non-parametric, and the distinction is independent of regression versus classification: an algorithm may be parametric or not, and may apply to either kind of problem, or both.
More formally, a parametric algorithm assumes the unknown function belongs to a family indexed by a fixed, finite number of parameters, determined before any data are observed: a line has exactly two, regardless of whether n is 20 or 20,000. A non-parametric algorithm makes no such commitment, and its effective complexity grows with the data. Larry Wasserman's textbook on the subject argues that “nonparametric inference” is something of a misnomer: “the basic idea... is to use data to infer an unknown quantity while making as few assumptions as possible. Usually, this means using statistical models that are infinite-dimensional. Indeed, a better name for nonparametric inference might be infinite-dimensional inference” . Finite parameters versus infinite is the precise version of the fixed-shape-versus-no-shape distinction described above.
A controlled experiment
For this module, we fixed the data-generating process for both problems ourselves: the actual mechanism producing every point, noise included, rather than relying on a real dataset whose true mechanism remains unknown even to us . This is what makes the comparison fair. When an algorithm performs poorly here, the reason can be identified precisely rather than debated. The dropdowns below allow a different data-generating process to be selected for either problem. Some are deliberately unfavorable to an algorithm that assumes a simple shape; at least one is deliberately favorable, so that assumption can be seen paying off for once rather than only ever costing something.