Reddit reviews Variational Analysis and Generalized Differentiation I: Basic Theory (Grundlehren der mathematischen Wissenschaften) (v. 1)

I took Terry Rockafellar's special topics course based on the book in 1990 (8 years before the book appeared) and have used material in the book in several research papers.

One way to think of this book is that it is convex analysis (Rockafellar's 1970 classic) with convexity dropped. Another way to think of it is that it is nonsmooth analysis (Frank Clarke's book) updated.

I never figured out why the "variational" (some analogy with calculus of variations, I think), but it is a masterly treatment of optimization theory.

One of the main tools is epiconvergence, the correct notion of convergence of optimization problems.

The book is indeed a lot to digest, but that is because there is a lot there. I think it is one of the great math books of the twentieth century. Rockafellar (1970) is another.

It is not related to functional analysis because, like Rockafellar (1970), it stays with finite dimensional. There is a reason. Epiconvergence can be defined for nonsmooth functions on infinite-dimensional spaces, but doesn't have anywhere near as nice properties. It need not even be topological. But for lower semicontinuous functions on finite-dimensional spaces the topology of epiconvergence is metrizable and compact. Other books (Attouch, 1984) do deal with the infinite-dimensional case (somewhat).

One interesting aspect of the book is its complete analysis of the generalization of subdifferentiation in convex analysis. It turns out that, in general, it splits into two concepts (or four concepts if one considers the dual notions of subgradients and subdifferentials), which merge into one in the so-called "regular" case (which includes convexity and, more generally, Lipschitz) but we also see this same phenomenon when we are not considering functions but just sets (two kinds of tangent vectors and two kinds of normal vectors) that also collapse in the "regular" case, which includes convexity.

So what do we have?

• notions of convergence for set-valued and nonsmooth functions and for sets that are the correct notions for optimization theory.
  
  
• analogs of differentiability for nonsmooth functions and set-valued functions.
  
  
• analogs of first and second order conditions for optimality for nonsmooth functions (analogs of gradient equal to zero and hessian positive definite for local minimum)
  
  And there's more.
  
  There is no prerequisite other than calculus. The whole theory is developed from the beginning. You need to know some measure theory for the last chapter. Baby Rudin (Principles of Mathematical Analysis) would be enough for that.
  
  Other books that cover similar topics are either more advanced or not as complete.
  
  
• Attouch (1984), apparently out of print.
  
  
• Clarke, (1987), apparently also out of print.
  
  
• Aubin and Frankowska (1990)
  
  
• Mordukhovich (2006), two volumes, the Amazon link is just to volume 1.
  
  But the Rockafellar and Wets book is both easier to read and more complete (except for avoiding infinte dimensions) than the others. I have to admit that I haven't really read the Mordukhovich and haven't even bought volume 2 yet.
  
  EDIT: Forgot the "map" question. This is the "calculus" of nonsmooth functions. But only differentiation theory, since classical integration theory handles nonsmooth functions with no problem. So it goes right next to calculus on the "map". Note that this theory was undiscovered until 1964 when Wijsman invented epiconvergence. So this is really new math! Some bits and pieces date earlier (Kuhn-Tucker conditions, tangent cones, Painlevé-Kuratowski set convergence, the analog of epiconvergence for sets), but the subject didn't really get rolling until the 1980's.