Function approximation, formally

Given a target function f, drawn from some space of functions too complicated to write down exactly, and a class of candidate functions you're willing to consider (lines, polynomials, trees, neural networks), find the member of that class that gets as close to f as possible. The idea predates machine learning by a century: the Weierstrass approximation theorem [cite] proved in 1885 that any continuous function on a closed interval can be approximated arbitrarily closely by a polynomial, long before anyone called this “learning.” Statistical learning inherits exactly this problem, with one twist: f itself is never observed, only noisy samples of it [ISLP].

It's not an overstatement to call this the essence of modern AI. Vapnik's foundational theory of learning casts the whole problem as function estimation from data[cite]; the universal approximation theorem is the reason a neural network can represent almost anything at all [cite]; Goodfellow, Bengio, and Courville open the standard deep-learning textbook by defining a feedforward network as, precisely, a function approximator [cite]. Much of it is geometry, too: least squares is nothing more than projecting a vector onto the nearest point of a subspace [cite], and an influential recent research program argues most of deep learning can be organized around exactly that kind of geometric structure [cite].

If you already knew f, there would be nothing left to approximate. Nearly everything else in this course — every model, every method — is really just a different answer to the same question: what happens the moment you don't?

Bringing this to life

Here's the trick this module uses to make that idea concrete. Normally you're the researcher: you stare at noisy data with no real access to the mechanism that produced it. As scientists, we have a formal name for that mechanism, whatever it turns out to be: the data-generating process, or DGP [cite]. This time, you'll start on the other side of it. Imagine you already know the DGP exactly, how tv budget really drives sales, mechanism and all, and your job is just to write that understanding down as a model. That's step one, coming up next.

Then the exercise flips. You'll set that knowledge aside, look only at the noisy data the DGP produces, and try to recover it using nothing but a model and some algebra: the position every real researcher is actually in. Nature already knows; the model has to guess.