Mm optimization algorithms offers an overview of the mm principle, a device for deriving optimization algorithms satisfying the ascent or descent property. A tutorial on mm algorithms david r hunter david r. We discuss two different even and convex nonnegative smooth approximations of the absolute value function and apply them to construct mm algorithms for least absolute deviation regression. This list may not reflect recent changes learn more. The set of allowable solutions, and hence, the objective.
The em algorithm from statistics is a special case. The mm algorithm is not an algorithm, but a prescription for constructing optimization algorithms. Jul 11, 2016 mm optimization algorithms offers an overview of the mm principle, a device for deriving optimization algorithms satisfying the ascent or descent property. Historically, the mm algorithm framework dates back before. Feb 01, 2014 this paper derives new algorithms for signomial programming, a generalization of geometric programming. Mm algorithms for geometric and signomial programming. Hunter is assistant professor, department of statistics, penn state university, university park, pa 168022111. Every em algorithm is an mm algorithm but not vice versa. The mm algorithm is not an algorithm, but a strategy for constructing optimization algorithms. Successful mm algorithms in highdimensional parameter spaces. Indeed, every em algorithm is a special case of the more general class of mm optimization algorithms, which typically exploit convexity rather than missing data in. An em algorithm operates by identifying a theoretical complete data space. The surrogate function is chosen so that its optimization is easier than the original objective. The mm minorizationmaximization principle is a versatile tool for constructing optimization algorithms.
It operates by using a surrogate function that majorizes the objective. David hunter department of statistics penn state university langefest, february 9, 2007 ucla 2007 mm algorithms. Zhao et al optimization algorithms for graph laplacian estimation via admm and mm 4233, 2. This introduction to optimization attempts to strike a balance between presentation of mathematical theory and development of numerical algorithms. This can be regarded as the special case of mathematical optimization where the objective value is the same for every solution, and thus any solution is optimal. The mm alternative to em tongtongwuandkennethlange abstract. When it is successful, the mm algorithm substitutes a simple optimization problem for a difficult optimization problem. The author presents the first extended treatment of mm algorithms, which are ideal for highdimensional optimization problems in data mining, imaging, and.
In the current chapter we discuss an optimization method that relies heavily on convexity arguments and is particularly useful in highdimensional problems such as image reconstruction 86. Em algorithm is a special case of the more general class of mm optimization algorithms, which typically exploit convexity rather than missing data in majorizing or minorizing an objective function. Optimization algorithms for graph laplacian estimation via. The right choice of an optimization algorithm can be crucially important in finding the right solutions for a given optimization problem.
The mm algorithm is an iterative optimization method which exploits the convexity of a function in order to find their maxima or minima. Kenneth lange mm optimization algorithms offers an overview of the mm principle, a device for deriving. Jul 14, 2010 this paper derives new algorithms for signomial programming, a generalization of geometric programming. When the surrogate function is optimized, the objective function is driven uphill or downhill as. Most practical optimization problems defy exact solution. Many optimization algorithms need to start from a feasible point. Among other things, theyre used to evaluate design tradeoffs, to assess control systems, and to find patterns in data. In minimizing an objective function, an mm algorithm oper. An mm algorithm operates by creating a surrogate function. In our view, mm algorithms are easier to understand and sometimes easier to apply than em algorithms. An optimization algorithm is a procedure which is executed iteratively by comparing various solutions till an optimum or a satisfactory solution is found. Where vector norms appear, the type of norm in use is indicated. Optimization mm algorithms as in, mm, i wonder what these algorithms have to do with genetics. This tutorial aims to illustrate some of the principles and algorithms of sparse signal processing, by way of considering the sparse deconvolution problem.
The mm principle is not an algorithm, but a prescription or principle for constructing optimization algorithms. This is a generalization of the em algorithm which does not involve missing data. The mm algorithm is an iterative optimization method which exploits the convexity of a function. Pages in category optimization algorithms and methods the following 158 pages are in this category, out of 158 total. Types of optimization algorithms local optimization based on following gradients o excels solver based on grg2 global optimization randomly search overall region before switching to local optimization technique o simulated annealing 19 22 22 72 2 0. Mm algorithms exploit an optimization technique that extends the central idea of em algorithms 6 to situations not necessarily involving missing data nor even maximum likelihood estimation. When the surrogate function is optimized, the objective function is driven uphill or downhill as needed. Find materials for this course in the pages linked along the left. Em algorithms are among the most effective algorithms for maximum likelihood estimation because they consistently drive the likeli. Overview of the mm algorithm 120 the mm algorithm is not an algorithm, but a strategy for constructing optimization algorithms.
The connection between lqa and mm enables us to analyze the convergence of the local quadratic approxi. Design optimization of reinforced concrete structures. Multiobjective optimization using evolutionary algorithms. Hunterand kenneth lange most problems in frequentist statistics involve optimization of a function such as a likelihood or a sum of squares. Mm optimization algorithms society for industrial and. Composite differencemax programs for modern statistical.
Mm algorithms for statistical inference and machine learning problems. In the current chapter we discuss an optimization method that relies heavily on convexity. In our opinion, mm algorithms deserve to be part of the standard toolkit of professional statisticians. One of the key criteria in judging majorizing or minorizing functions is their ease of optimization. Optimization problems and algorithms unit 2 introduction. The algorithms are based on a generic principle for optimization called the mm algorithm.
With the advent of computers, optimization has become a part of computeraided design activities. An objective function, which is either maximized or minimized, expresses the goal, or performance criterion, in terms of the decision variables. A computationally e cient iterative algorithm for sparse deconvolution is derived using the majorizationminimization mm optimization method. Mm optimization algorithms mathematical association of. Minimizationmaximization algorithms the mm algorithm is a popular optimization technique that allows to tackle nonconvex and nonsmooth functions 18. The em algorithm is a special case of a more general algorithm called the mm algorithm. The po tential of the mm algorithm in solving highdimensional optimization. In this setting, one can apply the geometricarithmetic mean. The mm algorithm has already proven useful in computations for various statistical applications lange, hunter and yang 2000 and becker, yang and lange 1997. The mm stands for majorizeminimization or minorizemaximization, depending on whether the desired optimization is a maximization or a minimization. Kenneth lange mm optimization algorithms offers an overview of the mm principle, a device for deriving optimization algorithms satisfying the ascent or descent property. Indeed, every em algorithm is a special case of the more general class of mm optimization algorithms, which typically exploit convexity rather than missing data in majorizing or minorizing an objective function. Generalized majorizationminimization proceedings of machine. Ski problem, secretary problem, paging, bin packing, using expert advice 4 lectures.
Pdf mm algorithms for geometric and signomial programming. An mm algorithm operates by creating a surrogate function that minorizes or majorizes the objective function. The mm principle is a device for creating optimization algorithms satisfying the as cent or descent property. There are two distinct types of optimization algorithms widely used today. Jan 21, 2015 optimization algorithms, which try to find the minimum values of mathematical functions, are everywhere in engineering. Mm optimization algorithms mathematical association of america. Optimization algorithms in matlab maria g villarreal ise department the ohio state university february 03, 2011. Although we have no intention of detracting from em algorithms, their dominance over mm algorithms is a historical accident. Recall that the normal probability density function pdf has form zi.
There exist a diverse range of algorithms for optimization. Building on a theory of algorithms known by the initials mm, for minorizationmaximization, this paper presents a powerful technique for producing iterative maximum likelihood estimation algorithms or a wide class of generalizations of the bradleyterry model. Mm optimization algorithms guide books acm digital library. Iterative optimization of a surrogate function as exemplified by an em algorithm does not necessarily require missing data. The mm algorithm university of california, berkeley. Mm algorithms for joint independent subspace analysis with. The majority of these problems cannot be solved analytically. Design optimization of reinforced concrete structures 315 determined, and a set of decision variable values constitutes a candidate solution. Finitedimensional optimization problems occur throughout the mathematical sciences. Mm algorithms for some discrete multivariate distributions. These algorithms can separate the variables of a problem, avoid large matrix inversions, linearize a problem, restore symmetry, deal with equality and inequality constraints gracefully, and turn a nondifferentiable problem into a smooth.
692 440 1145 1508 72 352 554 917 178 1251 1145 746 1085 297 936 1199 568 1177 729 993 149 786 96 1190 9 903 130 1078 322 1402 202 718