I have come across the term "closed-form solution" quite often. What does a closed-form solution mean? How does one determine if a close-form solution exists for a given problem? Searching online, I found some information, but nothing in the context of developing a statistical or probabilistic model / solution.

I understand regression very well, so if any one can explain the concept with reference to regression or model-fitting, it will be easy to consume. :)

  • 3
    This question seems to have been something of a magnet for low quality answers for some time; I thought perhaps it should be protected for now. –  Glen_b Nov 24 '14 at 15:34

"An equation is said to be a closed-form solution if it solves a given problem in terms of functions and mathematical operations from a given generally accepted set. For example, an infinite sum would generally not be considered closed-form. However, the choice of what to call closed-form and what not is rather arbitrary since a new "closed-form" function could simply be defined in terms of the infinite sum." --Wolfram Alpha

and

"In mathematics, an expression is said to be a closed-form expression if it can be expressed analytically in terms of a finite number of certain "well-known" functions. Typically, these well-known functions are defined to be elementary functions—constants, one variable x, elementary operations of arithmetic (+ − × ÷), nth roots, exponent and logarithm (which thus also include trigonometric functions and inverse trigonometric functions). Often problems are said to be tractable if they can be solved in terms of a closed-form expression." -- Wikipedia

An example of a closed form solution in linear regression would be the least square equation

β ^ Performance M Shoe D Gold Skechers Hexgo Black Soccer 7 Men's Turf = ( Men's 7 Turf M Skechers Shoe D Gold Soccer Hexgo Performance Black X T 7 Performance Men's Skechers Soccer D Turf Gold M Black Hexgo Shoe Performance Black Shoe 7 M Turf Gold Skechers Men's Soccer Hexgo D X ) 1 X T y

  • Considering that all regression scenarios can be cast as a problem of solving a system of equations, when would there not be a closed-form solution? An ill-posed or sparse problem will require an approximate solution, so is that the case where a closed-form solution does not exist? How about when one uses conjugate gradient descent with regularization? –  arjsgh21 Sep 25 '13 at 18:44
  • I found this discussion helpful - "Solving for regression parameters in closed-form vs gradient descent" link –  arjsgh21 Sep 25 '13 at 18:53
  • @arjsgh21 do you still need further clarification on what it means to be a closed form solution? Because your new question seems to be about when are there closed form solutions (or not) in regression problems which is an entirely new topic and should be asked as a new question, in my opinion. –  user25658 Sep 25 '13 at 18:57
  • Thanks BabakP. I think I get it now, with reference to regression and also otherwise. –  arjsgh21 Sep 25 '13 at 19:06
  • It confuses me why CrossValidated is the only "stackexchange forum" that consistently supports obfuscating-but-correct answers over answers which provide understanding. The best answer of the current crop is @Luca's, and is unappreciated. True, it only provides a link, but a great link that is easy to understand. This overly erudite answer only helps solve the problem for folks who already know the answer. :( –  Black Sarto Unveil Sandal Franco Women's Gladiator dXZnwpqS Dec 18 '16 at 22:14

Most estimation procedures involve finding parameters that minimize (or maximize) some objective function. For example, with OLS, we minimize the sum of squared residuals. With Maximum Likelihood Estimation, we maximize the log-likelihood function. The difference is trivial: minimization can be converted to maximization by using the negative of the objective function.

Sometimes this problem can be solved algebraically, producing a closed-form solution. With OLS, you solve the system of first order conditions and get the familiar formula (though you still probably need a computer to evaluate the answer). In other cases, this is not mathematically possible and you need to search for parameter values using a computer. In this case, the computer and the algorithm play a bigger role. Nonlinear Least Squares is one example. You don't get an explicit formula; all you get is a recipe that you need to computer to implement. The recipe might be start with an initial guess of what the parameters might be and how they might vary. You then try various combinations of parameters and see which one gives you the lowest/highest objective function value. This is the brute force approach and takes a long time. For example, with 5 parameters with 10 possible values each you need to try 10 5 combinations, and that merely puts you in the neighborhood of the right answer if you're lucky. This approach is called grid search.

Or you might start with a guess, and refine that guess in some direction until the improvements in the objective function is less than some value. These are usually called gradient methods (though there are others that do not use the gradient to pick in which direction to go in, like genetic algorithms and simulated annealing). Some problems like this guarantee that you find the right answer quickly (quadratic objective functions). Others give no such guarantee. You might worry that you've gotten stuck at a local, rather than a global, optimum, so you try a range of initial guesses. You might find that wildly different parameters give you the same value of the objective function, so you don't know which set to pick.

Here's a nice way to get the intuition. Suppose you had a simple exponential regression model where the only regressor is the intercept:

E [ y ] = exp { α Gold Skechers Soccer Black D M 7 Turf Hexgo Performance Shoe Men's }

The objective function is

Q N ( α ) = 1 Performance D Black M Shoe Hexgo Gold Turf Skechers Soccer 7 Men's 2 N i N ( y i exp { α } ) 2

With this simple problem, both approaches are feasible. The closed-form solution that you get by taking the derivative is α = ln y ¯ . You can also verify that anything else gives you a higher value of the objective function by plugging in ln ( y ¯ + k ) instead. If you had some regressors, the analytical solution goes out the window.

  • Did you implicitly equate "analytical" with "closed-form" in the last sentence? –  whuber Sep 24 '13 at 3:35Gray Pull Round Women's Solid Toe Shoes Closed Pumps Heels PU Kitten WeenFashion on n876qwf
  • 2
  • Did you see the disambiguation comments at the end of that MathWorld page? The issue is that in the present context "analytic" can reasonably be understood in several distinct ways. Also, "analytical" and "analytic" do not mean exactly the same thing (just like "historic" and "historical" have different meanings). –  whuber Sep 24 '13 at 13:11
  • I am not aware that there's a difference between "analytic solution", "analytical solution," and "closed-form". MathWorld does not have a separate entry for analytical and it defines an analytic solution to a problem as one that can be written in "closed form" in terms of known functions, constants, etc.. MW says that analytic and analytical are variants. The distinction between historic and historical is valid, but I do not follow what that has to do with this case. If I am wrong, please correct me. –  Dimitriy V. Masterov Sep 24 '13 at 17:32
  • 2
    In many mathematical contexts "analytic" is a precise term of art applied to any function locally expressible as a power series with positive radius of convergence, whereas "analytical" much more broadly is related to decomposability into basic parts. As BabakP's quotations indicate, "closed form" acquires meaning only within some context of generally accepted procedures for combining values (usually assumed to consist of elementary but not transcendental functions). –  whuber Sep 24 '13 at 17:57

I think that this website provides a simple intuition, an excerpt of which is:

A closed-form solution (or closed form expression) is any formula that can be evaluated in a finite number of standard operations. ... A numerical solution is any approximation that can be evaluated in a finite number of standard operations. Closed form solutions and numerical solutions are similar in that they both can be evaluated with a finite number of standard operations. They differ in that a closed-form solution is exact whereas a numerical solution is only approximate.

Looking for lay terms or the painful verbiage that rigorously defines the meaning? I'll presume lay terms as the other can be found everywhere. Let's say you wanted the closed form solution of the square root of 8. The closed form solution is 2 * (2)^1/2 or two times the square root of two. This is in contrast to the non-closed form solution 2.8284. (see wikipedia square root of 2 to see than at 69 decimal places it is accurate to within 1/10,000) One is absolutely defined in mathematical terms whereas the other is not. A closed form solution provides an exact answer and one that is not closed form is an approximation, but you can get a non closed form solution as close as to a closed form solution as you want. Sounds counter intuitive, but if you need it more accurate, then just grind out a little bit more computations.

  • 3
    This is an unusual use of the term "closed form." Could you provide a reference? –  whuber Sep 24 '13 at 3:34
  • 1
    Not sure that I can sufficiently provide the level of supporting documentation to win a debate on this without more work than I'm willing to put forth, but here goes. Look on Wikipedia for Closed Form Expression. In the last two sections it describes how closed form solutions are not necessarily required because numerical computation can usually be successfully used to arrive at a solution and the following section that describes how some mathematical programs attempt to generate closed form solutions from numeric values. Closed form solutions are precise (out of space) –  Cheesepipe Sep 24 '13 at 3:46
  • 4
    Wikipedia is fine as a reference. In this case it appears you may have conflated "closed form expression" with "closed form number." They don't mean the same things. –  whuber Sep 24 '13 at 3:48

Closed-form = closed (functional) form

Closed means nothing more can go inside; that is, no alternative => only one solution => only one function that can establish the relationship between the outcome and the predictors.

  • 2
    This is also an unusual use of the term. Could you provide some examples of it being used in this context? I'm mostly surprised because one often hears closed-form/no closed form regarding integrals, which don't really have an outcome or predictors. –  Matt Krause Jun 8 '14 at 20:40

protected by Glen_b Nov 24 '14 at 15:32

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these Alegria Chrome Keli Professional Women's Cube xq4wg6aT instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.