SciPy minimizers and constraints

The Minuit class can call SciPy minimizers implemented in scipy.optimize.minimize as alternatives to the standard Migrad minimizer to minimize the cost function. The SciPy minimizers may perform better or worse on some problems. You can give them a try when you are not satisfied with Migrad.

More importantly, the SciPy minimizers support additional features that Migrad lacks.

  • Migrad does not allow one to use an externally computed hessian matrix.

  • Migrad does not allow one to use additional constraints of the form \(\vec a \leq f(\vec x) \leq \vec b\) in the minimization, where \(\vec x\) is the parameter vector of length \(m\), \(f\) is an arbitrary function \(\mathcal{R}^m \rightarrow \mathcal{R}^k\) and \(\vec a, \vec b\) are vector bounds with length \(k\).

SciPy comes with a variety of minimization algorithms and some of them support these features. The ability to use constraints is interesting for HEP applications. In particular, it allows us to ensure that a pdf as a function of the parameters is always positive. This can be ensured sometimes with suitable limits on the parameters, but not always.

We demonstrate this on a common example of fit of an additive model with a signal and background pdf.

[1]:
from iminuit import Minuit
from iminuit.cost import ExtendedUnbinnedNLL
import numpy as np
from numba_stats import norm, bernstein
import matplotlib.pyplot as plt
from IPython.display import display
import joblib
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File __init__.pxd:942, in numpy.import_array()

RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
Input In [1], in <cell line: 4>()
      2 from iminuit.cost import ExtendedUnbinnedNLL
      3 import numpy as np
----> 4 from numba_stats import norm, bernstein
      5 import matplotlib.pyplot as plt
      6 from IPython.display import display

File ~/python-iminuit/src/python-iminuit/test-env/lib/python3.10/site-packages/numba_stats/norm.py:9, in <module>
      1 """
      2 Normal distribution.
      3
   (...)
      6 scipy.stats.norm: Scipy equivalent.
      7 """
      8 import numpy as np
----> 9 from ._special import erfinv as _erfinv
     10 from ._util import _jit, _trans, _generate_wrappers, _prange
     11 from math import erf as _erf

File ~/python-iminuit/src/python-iminuit/test-env/lib/python3.10/site-packages/numba_stats/_special.py:7, in <module>
      5 from numba.extending import get_cython_function_address
      6 from numba.types import WrapperAddressProtocol, float64
----> 7 import scipy.special.cython_special as cysp
     10 def get(name, signature):
     11     # create new function object with correct signature that numba can call by extracting
     12     # function pointer from scipy.special.cython_special; uses scipy/cython internals
     13     index = 1 if signature.return_type is float64 else 0

File /usr/lib/python3.10/site-packages/scipy/special/__init__.py:649, in <module>
      1 """
      2 ========================================
      3 Special functions (:mod:`scipy.special`)
   (...)
    644
    645 """
    647 from ._sf_error import SpecialFunctionWarning, SpecialFunctionError
--> 649 from . import _ufuncs
    650 from ._ufuncs import *
    652 from . import _basic

File /usr/lib/python3.10/site-packages/scipy/special/_ufuncs.pyx:1, in init scipy.special._ufuncs()

File scipy/special/_ufuncs_extra_code_common.pxi:34, in init scipy.special._ufuncs_cxx()

File __init__.pxd:944, in numpy.import_array()

ImportError: numpy.core.multiarray failed to import

The signal pdf is a Gaussian, the background is modelled with second degree Bernstein polynomials. We perform an extended maximum likelihood fit, where the full density model is given by the sum of the signal and background component.

[2]:
xrange = (0, 1)


def model(x, b0, b1, b2, sig, mu, sigma):
    beta = [b0, b1, b2]
    bint = np.diff(bernstein.integral(xrange, beta, *xrange))
    sint = sig * np.diff(norm.cdf(xrange, mu, sigma))[0]
    return bint + sint, bernstein.density(x, beta, *xrange) + sig * norm.pdf(x, mu, sigma)

In searches for rare decays, it is common to fit models like this to small simulated samples that contain only background, to calculate the distribution of some test statistic (usually the likelihood ratio of S+B and B-only hypotheses). Here, for simplicity, we use the signal amplitude itself as the test statistic.

We run one such fit. The mean and width of the Gaussian are fixed, only the signal amplitude and the background parameters are varied.

[3]:
rng = np.random.default_rng(2)
x = rng.uniform(0, 1, size=35)

cost = ExtendedUnbinnedNLL(x, model)
n = len(x)
m = Minuit(cost, b0=n, b1=n, b2=n, sig=0, mu=0.5, sigma=0.05)
m.print_level = 0
m.limits["b0", "b1", "b2"] = (0, None)
m.fixed["mu", "sigma"] = True
display(m.migrad())

plt.hist(x, bins=50, density=True)
xm = np.linspace(0, 1)
yint, ym = model(xm, *m.values)
plt.plot(xm, ym / yint);
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [3], in <cell line: 10>()
      8 m.limits["b0", "b1", "b2"] = (0, None)
      9 m.fixed["mu", "sigma"] = True
---> 10 display(m.migrad())
     12 plt.hist(x, bins=50, density=True)
     13 xm = np.linspace(0, 1)

File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/minuit.py:691, in Minuit.migrad(self, ncall, iterate)
    689 if self._precision is not None:
    690     migrad.precision = self._precision
--> 691 fm = migrad(ncall, self._tolerance)
    692 if fm.is_valid or fm.has_reached_call_limit:
    693     break

File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/cost.py:470, in Cost.__call__(self, *args)
    455 def __call__(self, *args):
    456     """
    457     Evaluate the cost function.
    458
   (...)
    468     float
    469     """
--> 470     r = self._call(args)
    471     if self.verbose >= 1:
    472         print(args, "->", r)

File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/cost.py:843, in ExtendedUnbinnedNLL._call(self, args)
    841 def _call(self, args):
    842     data = self._masked
--> 843     ns, x = self._model(data, *args)
    844     x = _normalize_model_output(
    845         x, "Model should return numpy array in second position"
    846     )
    847     if self._log:

Input In [2], in model(x, b0, b1, b2, sig, mu, sigma)
      4 def model(x, b0, b1, b2, sig, mu, sigma):
      5     beta = [b0, b1, b2]
----> 6     bint = np.diff(bernstein.integral(xrange, beta, *xrange))
      7     sint = sig * np.diff(norm.cdf(xrange, mu, sigma))[0]
      8     return bint + sint, bernstein.density(x, beta, *xrange) + sig * norm.pdf(x, mu, sigma)

NameError: name 'bernstein' is not defined

In this example, the signal amplitude came out negative. This happens if the background has an underfluctuation where the signal is expected. This is not an issue if the sum of signal and background density is still positive everywhere where we evaluate it. As long as the total density is positive, individual components are allowed to be negative.

There are, however, no principle restrictions in this example that prevent the sum of signal and background from becoming negative for some toy data sets. When that happens, the fit will fail, since the total density cannot mathematically become negative.

If this happens anyway, the fit will fail since taking logarithm of a negative number will cause havoc.

Migrad fit on toys

We apply the fit many times on randomly sampled background-only data to observe this.

[4]:
@joblib.delayed
def compute(itry):
    rng = np.random.default_rng(itry)
    x = rng.uniform(0, 1, size=35)
    cost = ExtendedUnbinnedNLL(x, model)
    m = Minuit(cost, b0=n, b1=n, b2=n, sig=0, mu=0.5, sigma=0.05)
    m.limits["b0", "b1", "b2"] = (0, None)
    m.fixed["mu", "sigma"] = True
    m.migrad()
    return m.values["sig"] if m.valid else np.nan

sigs_migrad = joblib.Parallel(-1)(compute(i) for i in range(200))

print(np.sum(np.isnan(sigs_migrad)), "failed")
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 @joblib.delayed
      2 def compute(itry):
      3     rng = np.random.default_rng(itry)
      4     x = rng.uniform(0, 1, size=35)

NameError: name 'joblib' is not defined
[5]:
nfailed = np.sum(np.isnan(sigs_migrad))
plt.title(f"{nfailed} fits failed ({nfailed / len(sigs_migrad) * 100:.0f} %)")
plt.hist(sigs_migrad, bins=10, range=(-10, 10))
plt.xlabel("sig");
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [5], in <cell line: 1>()
----> 1 nfailed = np.sum(np.isnan(sigs_migrad))
      2 plt.title(f"{nfailed} fits failed ({nfailed / len(sigs_migrad) * 100:.0f} %)")
      3 plt.hist(sigs_migrad, bins=10, range=(-10, 10))

NameError: name 'sigs_migrad' is not defined

The distribution of the signal amplitude looks fairly gaussian which is good, but the fit failed to converge in a few cases due to the problem just described. Simply discarding these cases is not acceptable, it would distort conclusions drawn from the distribution of the test statistic, which is commonly needed to set limits or to compute the p-value for an observed amplitude.

We can repair this by placing a limit on the signal amplitude. This is a suitable solution, although it will bias the signal amplitude and change the shape of the distribution of the test statistic.

An alternative is to perform a constrained minimization, which allows us to directly add a condition to the fit that the model density must be positive at each data point. We merely need to replace the call m.migrad with the call m.scipy and pass the (non-linear) constraint. An appropriate algorithm is automatically selected which performs a constrained minimization. The SciPy minimizers are fully integrated into Minuit, which means that Minuit computes an EDM value for the minimum and parameter uncertainties.

SciPy constrained fit on toys

We run SciPy with the constraint on the same simulated samples on which we ran Migrad before.

[6]:
from scipy.optimize import NonlinearConstraint

@joblib.delayed
def compute(itry):
    rng = np.random.default_rng(itry)
    x = rng.uniform(0, 1, size=35)
    cost = ExtendedUnbinnedNLL(x, model)
    m = Minuit(cost, b0=n, b1=n, b2=n, sig=0, mu=0.5, sigma=0.05)
    m.limits["b0", "b1", "b2"] = (0, None)
    m.fixed["mu", "sigma"] = True
    m.scipy(constraints=NonlinearConstraint(lambda *par: model(x, *par)[1], 0, np.inf))
    return m.values["sig"] if m.valid else np.nan

sigs_constrained = joblib.Parallel(-1)(compute(i) for i in range(200))

print(np.sum(np.isnan(sigs_constrained)), "failed")
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [6], in <cell line: 1>()
----> 1 from scipy.optimize import NonlinearConstraint
      3 @joblib.delayed
      4 def compute(itry):
      5     rng = np.random.default_rng(itry)

File /usr/lib/python3.10/site-packages/scipy/optimize/__init__.py:400, in <module>
      1 """
      2 =====================================================
      3 Optimization and root finding (:mod:`scipy.optimize`)
   (...)
    397
    398 """
--> 400 from ._optimize import *
    401 from ._minimize import *
    402 from ._root import *

File /usr/lib/python3.10/site-packages/scipy/optimize/_optimize.py:33, in <module>
     30 from numpy import (atleast_1d, eye, argmin, zeros, shape, squeeze,
     31                    asarray, sqrt, Inf, asfarray)
     32 import numpy as np
---> 33 from scipy.sparse.linalg import LinearOperator
     34 from ._linesearch import (line_search_wolfe1, line_search_wolfe2,
     35                           line_search_wolfe2 as line_search,
     36                           LineSearchWarning)
     37 from ._numdiff import approx_derivative

File /usr/lib/python3.10/site-packages/scipy/sparse/__init__.py:267, in <module>
    264 import warnings as _warnings
    266 from ._base import *
--> 267 from ._csr import *
    268 from ._csc import *
    269 from ._lil import *

File /usr/lib/python3.10/site-packages/scipy/sparse/_csr.py:10, in <module>
      7 import numpy as np
      9 from ._base import spmatrix
---> 10 from ._sparsetools import (csr_tocsc, csr_tobsr, csr_count_blocks,
     11                            get_csr_submatrix)
     12 from ._sputils import upcast, get_index_dtype
     14 from ._compressed import _cs_matrix

ImportError: numpy.core.multiarray failed to import
[7]:
plt.title(f"{np.sum(np.isnan(sigs_constrained))} constrained fits failed")
plt.hist(sigs_migrad, alpha=0.5, bins=10, range=(-10, 10), label="Migrad")
plt.hist(sigs_constrained, alpha=0.5, bins=10, range=(-10, 10), label=m.fmin.algorithm)
plt.xlabel("sig")
plt.legend();
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [7], in <cell line: 1>()
----> 1 plt.title(f"{np.sum(np.isnan(sigs_constrained))} constrained fits failed")
      2 plt.hist(sigs_migrad, alpha=0.5, bins=10, range=(-10, 10), label="Migrad")
      3 plt.hist(sigs_constrained, alpha=0.5, bins=10, range=(-10, 10), label=m.fmin.algorithm)

NameError: name 'plt' is not defined

There are no failures this time.

For sig > 0, the distributions are identical in this example, as theoretically expected. In practice, there can be small bin migration effects due to finite precision of numerical algorithms. These are not of concern.

Important are the differences for sig < 0, where Migrad did not converge in a few cases and where therefore samples are missing. Those missing samples are recovered in the distribution produced by the constrained fit.

This demonstrates that it is important to not discard failed fits, as this will in general distort the distribution of the test statistic.

Bonus: unconstrained SciPy fit

The issues we describe here are of a principal mathematical nature. We should not expect that an unconstrained minimiser from SciPy does better than Migrad, but let’s test this assumption. The minimiser that SciPy uses when only box constraints are used is the L-BFGS-B method which is roughly comparable to Migrad. Let us see how well this algorithm does on the same toy samples.

[8]:
@joblib.delayed
def compute(itry):
    rng = np.random.default_rng(itry)
    x = rng.uniform(0, 1, size=35)
    cost = ExtendedUnbinnedNLL(x, model)
    m = Minuit(cost, b0=n, b1=n, b2=n, sig=0, mu=0.5, sigma=0.05)
    m.limits["b0", "b1", "b2"] = (0, None)
    m.fixed["mu", "sigma"] = True
    m.scipy()
    return m.values["sig"] if m.valid else np.nan

sigs_bfgs = joblib.Parallel(-1)(compute(i) for i in range(200))

print(np.sum(np.isnan(sigs_bfgs)), "failed")
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [8], in <cell line: 1>()
----> 1 @joblib.delayed
      2 def compute(itry):
      3     rng = np.random.default_rng(itry)
      4     x = rng.uniform(0, 1, size=35)

NameError: name 'joblib' is not defined
[9]:
plt.title(f"{np.sum(np.isnan(sigs_bfgs))} BFGS fits failed")
plt.hist(sigs_migrad, alpha=0.5, bins=10, range=(-10, 10), label="Migrad")
plt.hist(sigs_constrained, alpha=0.5, bins=10, range=(-10, 10), label="SciPy[SLSQS]")
plt.hist(sigs_bfgs, bins=10, range=(-10, 10), fill=False, label="SciPy[L-BFGS-B]")
plt.xlabel("sig")
plt.legend();
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [9], in <cell line: 1>()
----> 1 plt.title(f"{np.sum(np.isnan(sigs_bfgs))} BFGS fits failed")
      2 plt.hist(sigs_migrad, alpha=0.5, bins=10, range=(-10, 10), label="Migrad")
      3 plt.hist(sigs_constrained, alpha=0.5, bins=10, range=(-10, 10), label="SciPy[SLSQS]")

NameError: name 'plt' is not defined

In this example, the BFGS method actually failed much less than Migrad, but it still fails in some cases, while the constrained fit did not fail at all.

Speed comparison

Since constrained fits are so useful, should you use them all the time? Probably not.

Constrained fits are more computationally expensive. Satisfying extra constrains generally slows down convergence. Let’s compare the speed of the three minimisers tested here. We set the strategy to 0, to avoid computing the Hessian matrix automatically, since we want to measure only the time used by the minimiser.

[10]:
m.strategy = 0
[11]:
%timeit -n3 m.reset(); m.migrad()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [11], in <cell line: 1>()
----> 1 get_ipython().run_line_magic('timeit', '-n3 m.reset(); m.migrad()')

File /usr/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2305, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
   2303     kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2304 with self.builtin_trap:
-> 2305     result = fn(*args, **kwargs)
   2306 return result

File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:1166, in ExecutionMagics.timeit(self, line, cell, local_ns)
   1163         if time_number >= 0.2:
   1164             break
-> 1166 all_runs = timer.repeat(repeat, number)
   1167 best = min(all_runs) / number
   1168 worst = max(all_runs) / number

File /usr/lib/python3.10/timeit.py:206, in Timer.repeat(self, repeat, number)
    204 r = []
    205 for i in range(repeat):
--> 206     t = self.timeit(number)
    207     r.append(t)
    208 return r

File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:156, in Timer.timeit(self, number)
    154 gc.disable()
    155 try:
--> 156     timing = self.inner(it, self.timer)
    157 finally:
    158     if gcold:

File <magic-timeit>:1, in inner(_it, _timer)

File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/minuit.py:691, in Minuit.migrad(self, ncall, iterate)
    689 if self._precision is not None:
    690     migrad.precision = self._precision
--> 691 fm = migrad(ncall, self._tolerance)
    692 if fm.is_valid or fm.has_reached_call_limit:
    693     break

File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/cost.py:470, in Cost.__call__(self, *args)
    455 def __call__(self, *args):
    456     """
    457     Evaluate the cost function.
    458
   (...)
    468     float
    469     """
--> 470     r = self._call(args)
    471     if self.verbose >= 1:
    472         print(args, "->", r)

File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/cost.py:843, in ExtendedUnbinnedNLL._call(self, args)
    841 def _call(self, args):
    842     data = self._masked
--> 843     ns, x = self._model(data, *args)
    844     x = _normalize_model_output(
    845         x, "Model should return numpy array in second position"
    846     )
    847     if self._log:

Input In [2], in model(x, b0, b1, b2, sig, mu, sigma)
      4 def model(x, b0, b1, b2, sig, mu, sigma):
      5     beta = [b0, b1, b2]
----> 6     bint = np.diff(bernstein.integral(xrange, beta, *xrange))
      7     sint = sig * np.diff(norm.cdf(xrange, mu, sigma))[0]
      8     return bint + sint, bernstein.density(x, beta, *xrange) + sig * norm.pdf(x, mu, sigma)

NameError: name 'bernstein' is not defined
[12]:
%timeit -n3 m.reset(); m.scipy()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [12], in <cell line: 1>()
----> 1 get_ipython().run_line_magic('timeit', '-n3 m.reset(); m.scipy()')

File /usr/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2305, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
   2303     kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2304 with self.builtin_trap:
-> 2305     result = fn(*args, **kwargs)
   2306 return result

File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:1166, in ExecutionMagics.timeit(self, line, cell, local_ns)
   1163         if time_number >= 0.2:
   1164             break
-> 1166 all_runs = timer.repeat(repeat, number)
   1167 best = min(all_runs) / number
   1168 worst = max(all_runs) / number

File /usr/lib/python3.10/timeit.py:206, in Timer.repeat(self, repeat, number)
    204 r = []
    205 for i in range(repeat):
--> 206     t = self.timeit(number)
    207     r.append(t)
    208 return r

File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:156, in Timer.timeit(self, number)
    154 gc.disable()
    155 try:
--> 156     timing = self.inner(it, self.timer)
    157 finally:
    158     if gcold:

File <magic-timeit>:1, in inner(_it, _timer)

File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/minuit.py:927, in Minuit.scipy(self, method, ncall, hess, hessp, constraints)
    887 """
    888 Minimize with SciPy algorithms.
    889
   (...)
    924 the tolerance :attr:`tol` has no effect on SciPy minimizers.
    925 """
    926 try:
--> 927     from scipy.optimize import (
    928         minimize,
    929         Bounds,
    930         NonlinearConstraint,
    931         LinearConstraint,
    932     )
    933 except ImportError as exc:
    934     exc.msg += "\n\nPlease install scipy to use scipy minimizers in iminuit."

File /usr/lib/python3.10/site-packages/scipy/optimize/__init__.py:400, in <module>
      1 """
      2 =====================================================
      3 Optimization and root finding (:mod:`scipy.optimize`)
   (...)
    397
    398 """
--> 400 from ._optimize import *
    401 from ._minimize import *
    402 from ._root import *

File /usr/lib/python3.10/site-packages/scipy/optimize/_optimize.py:33, in <module>
     30 from numpy import (atleast_1d, eye, argmin, zeros, shape, squeeze,
     31                    asarray, sqrt, Inf, asfarray)
     32 import numpy as np
---> 33 from scipy.sparse.linalg import LinearOperator
     34 from ._linesearch import (line_search_wolfe1, line_search_wolfe2,
     35                           line_search_wolfe2 as line_search,
     36                           LineSearchWarning)
     37 from ._numdiff import approx_derivative

File /usr/lib/python3.10/site-packages/scipy/sparse/__init__.py:267, in <module>
    264 import warnings as _warnings
    266 from ._base import *
--> 267 from ._csr import *
    268 from ._csc import *
    269 from ._lil import *

File /usr/lib/python3.10/site-packages/scipy/sparse/_csr.py:10, in <module>
      7 import numpy as np
      9 from ._base import spmatrix
---> 10 from ._sparsetools import (csr_tocsc, csr_tobsr, csr_count_blocks,
     11                            get_csr_submatrix)
     12 from ._sputils import upcast, get_index_dtype
     14 from ._compressed import _cs_matrix

ImportError: numpy.core.multiarray failed to import

Please install scipy to use scipy minimizers in iminuit.
[13]:
%timeit -n3 m.reset(); m.scipy(constraints=NonlinearConstraint(lambda *par: model(x, *par)[1], 0, np.inf))
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [13], in <cell line: 1>()
----> 1 get_ipython().run_line_magic('timeit', '-n3 m.reset(); m.scipy(constraints=NonlinearConstraint(lambda *par: model(x, *par)[1], 0, np.inf))')

File /usr/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2305, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
   2303     kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2304 with self.builtin_trap:
-> 2305     result = fn(*args, **kwargs)
   2306 return result

File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:1166, in ExecutionMagics.timeit(self, line, cell, local_ns)
   1163         if time_number >= 0.2:
   1164             break
-> 1166 all_runs = timer.repeat(repeat, number)
   1167 best = min(all_runs) / number
   1168 worst = max(all_runs) / number

File /usr/lib/python3.10/timeit.py:206, in Timer.repeat(self, repeat, number)
    204 r = []
    205 for i in range(repeat):
--> 206     t = self.timeit(number)
    207     r.append(t)
    208 return r

File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:156, in Timer.timeit(self, number)
    154 gc.disable()
    155 try:
--> 156     timing = self.inner(it, self.timer)
    157 finally:
    158     if gcold:

File <magic-timeit>:1, in inner(_it, _timer)

NameError: name 'NonlinearConstraint' is not defined

Migrad is the fastest, followed by the L-BFGS-B method. The constrained fit is much slower.

The constrained fit is much slower, since it has to do more work. Why Migrad is faster than L-BFGS-B is not so obvious. There are some general reasons for that, but there may be cases where L-BFGS-B performs better.

Migrad is comparably fast because of its smart stopping criterion. Migrad stops the fit as soon as the improvement of the fitted parameters become small compared to their uncertainties. Migrad was explicitly designed for statistical fits, where the cost function is a log-likelihood or least-squares function. Since it assumes that, it can stops the fit as soon as the parameter improvements become negligible compared to the parameter uncertainty, which is given by the inverse of its internal approximation of the Hessian matrix.

The SciPy minimisers do not expect the cost function to be a log-likelihood or least-squares and thus cannot assume that the Hessian matrix has a special meaning. Instead they tend to optimise until they hit the limits of machine precision. This is the main reason why the L-BFGS-B method is slower. You can also see this in the benchmark section of the documentation.

We can force Migrad to do something similar by setting the tolerance to a tiny value.

[14]:
m.tol = 1e-20
[15]:
%timeit -n3 m.reset(); m.migrad()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [15], in <cell line: 1>()
----> 1 get_ipython().run_line_magic('timeit', '-n3 m.reset(); m.migrad()')

File /usr/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2305, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
   2303     kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2304 with self.builtin_trap:
-> 2305     result = fn(*args, **kwargs)
   2306 return result

File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:1166, in ExecutionMagics.timeit(self, line, cell, local_ns)
   1163         if time_number >= 0.2:
   1164             break
-> 1166 all_runs = timer.repeat(repeat, number)
   1167 best = min(all_runs) / number
   1168 worst = max(all_runs) / number

File /usr/lib/python3.10/timeit.py:206, in Timer.repeat(self, repeat, number)
    204 r = []
    205 for i in range(repeat):
--> 206     t = self.timeit(number)
    207     r.append(t)
    208 return r

File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:156, in Timer.timeit(self, number)
    154 gc.disable()
    155 try:
--> 156     timing = self.inner(it, self.timer)
    157 finally:
    158     if gcold:

File <magic-timeit>:1, in inner(_it, _timer)

File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/minuit.py:691, in Minuit.migrad(self, ncall, iterate)
    689 if self._precision is not None:
    690     migrad.precision = self._precision
--> 691 fm = migrad(ncall, self._tolerance)
    692 if fm.is_valid or fm.has_reached_call_limit:
    693     break

File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/cost.py:470, in Cost.__call__(self, *args)
    455 def __call__(self, *args):
    456     """
    457     Evaluate the cost function.
    458
   (...)
    468     float
    469     """
--> 470     r = self._call(args)
    471     if self.verbose >= 1:
    472         print(args, "->", r)

File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/cost.py:843, in ExtendedUnbinnedNLL._call(self, args)
    841 def _call(self, args):
    842     data = self._masked
--> 843     ns, x = self._model(data, *args)
    844     x = _normalize_model_output(
    845         x, "Model should return numpy array in second position"
    846     )
    847     if self._log:

Input In [2], in model(x, b0, b1, b2, sig, mu, sigma)
      4 def model(x, b0, b1, b2, sig, mu, sigma):
      5     beta = [b0, b1, b2]
----> 6     bint = np.diff(bernstein.integral(xrange, beta, *xrange))
      7     sint = sig * np.diff(norm.cdf(xrange, mu, sigma))[0]
      8     return bint + sint, bernstein.density(x, beta, *xrange) + sig * norm.pdf(x, mu, sigma)

NameError: name 'bernstein' is not defined

Now the runtime of Migrad is closer to L-BFGS-B, but it is still faster in this case.