SciPy minimizers and constraints¶
The Minuit
class can call SciPy minimizers implemented in scipy.optimize.minimize
as alternatives to the standard Migrad minimizer to minimize the cost function. The SciPy minimizers may perform better or worse on some problems. You can give them a try when you are not satisfied with Migrad.
More importantly, the SciPy minimizers support additional features that Migrad lacks.
Migrad does not allow one to use an externally computed hessian matrix.
Migrad does not allow one to use additional constraints of the form \(\vec a \leq f(\vec x) \leq \vec b\) in the minimization, where \(\vec x\) is the parameter vector of length \(m\), \(f\) is an arbitrary function \(\mathcal{R}^m \rightarrow \mathcal{R}^k\) and \(\vec a, \vec b\) are vector bounds with length \(k\).
SciPy comes with a variety of minimization algorithms and some of them support these features. The ability to use constraints is interesting for HEP applications. In particular, it allows us to ensure that a pdf as a function of the parameters is always positive. This can be ensured sometimes with suitable limits on the parameters, but not always.
We demonstrate this on a common example of fit of an additive model with a signal and background pdf.
[1]:
from iminuit import Minuit
from iminuit.cost import ExtendedUnbinnedNLL
import numpy as np
from numba_stats import norm, bernstein
import matplotlib.pyplot as plt
from IPython.display import display
import joblib
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
File __init__.pxd:942, in numpy.import_array()
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf
During handling of the above exception, another exception occurred:
ImportError Traceback (most recent call last)
Input In [1], in <cell line: 4>()
2 from iminuit.cost import ExtendedUnbinnedNLL
3 import numpy as np
----> 4 from numba_stats import norm, bernstein
5 import matplotlib.pyplot as plt
6 from IPython.display import display
File ~/python-iminuit/src/python-iminuit/test-env/lib/python3.10/site-packages/numba_stats/norm.py:9, in <module>
1 """
2 Normal distribution.
3
(...)
6 scipy.stats.norm: Scipy equivalent.
7 """
8 import numpy as np
----> 9 from ._special import erfinv as _erfinv
10 from ._util import _jit, _trans, _generate_wrappers, _prange
11 from math import erf as _erf
File ~/python-iminuit/src/python-iminuit/test-env/lib/python3.10/site-packages/numba_stats/_special.py:7, in <module>
5 from numba.extending import get_cython_function_address
6 from numba.types import WrapperAddressProtocol, float64
----> 7 import scipy.special.cython_special as cysp
10 def get(name, signature):
11 # create new function object with correct signature that numba can call by extracting
12 # function pointer from scipy.special.cython_special; uses scipy/cython internals
13 index = 1 if signature.return_type is float64 else 0
File /usr/lib/python3.10/site-packages/scipy/special/__init__.py:649, in <module>
1 """
2 ========================================
3 Special functions (:mod:`scipy.special`)
(...)
644
645 """
647 from ._sf_error import SpecialFunctionWarning, SpecialFunctionError
--> 649 from . import _ufuncs
650 from ._ufuncs import *
652 from . import _basic
File /usr/lib/python3.10/site-packages/scipy/special/_ufuncs.pyx:1, in init scipy.special._ufuncs()
File scipy/special/_ufuncs_extra_code_common.pxi:34, in init scipy.special._ufuncs_cxx()
File __init__.pxd:944, in numpy.import_array()
ImportError: numpy.core.multiarray failed to import
The signal pdf is a Gaussian, the background is modelled with second degree Bernstein polynomials. We perform an extended maximum likelihood fit, where the full density model is given by the sum of the signal and background component.
[2]:
xrange = (0, 1)
def model(x, b0, b1, b2, sig, mu, sigma):
beta = [b0, b1, b2]
bint = np.diff(bernstein.integral(xrange, beta, *xrange))
sint = sig * np.diff(norm.cdf(xrange, mu, sigma))[0]
return bint + sint, bernstein.density(x, beta, *xrange) + sig * norm.pdf(x, mu, sigma)
In searches for rare decays, it is common to fit models like this to small simulated samples that contain only background, to calculate the distribution of some test statistic (usually the likelihood ratio of S+B and B-only hypotheses). Here, for simplicity, we use the signal amplitude itself as the test statistic.
We run one such fit. The mean and width of the Gaussian are fixed, only the signal amplitude and the background parameters are varied.
[3]:
rng = np.random.default_rng(2)
x = rng.uniform(0, 1, size=35)
cost = ExtendedUnbinnedNLL(x, model)
n = len(x)
m = Minuit(cost, b0=n, b1=n, b2=n, sig=0, mu=0.5, sigma=0.05)
m.print_level = 0
m.limits["b0", "b1", "b2"] = (0, None)
m.fixed["mu", "sigma"] = True
display(m.migrad())
plt.hist(x, bins=50, density=True)
xm = np.linspace(0, 1)
yint, ym = model(xm, *m.values)
plt.plot(xm, ym / yint);
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Input In [3], in <cell line: 10>()
8 m.limits["b0", "b1", "b2"] = (0, None)
9 m.fixed["mu", "sigma"] = True
---> 10 display(m.migrad())
12 plt.hist(x, bins=50, density=True)
13 xm = np.linspace(0, 1)
File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/minuit.py:691, in Minuit.migrad(self, ncall, iterate)
689 if self._precision is not None:
690 migrad.precision = self._precision
--> 691 fm = migrad(ncall, self._tolerance)
692 if fm.is_valid or fm.has_reached_call_limit:
693 break
File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/cost.py:470, in Cost.__call__(self, *args)
455 def __call__(self, *args):
456 """
457 Evaluate the cost function.
458
(...)
468 float
469 """
--> 470 r = self._call(args)
471 if self.verbose >= 1:
472 print(args, "->", r)
File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/cost.py:843, in ExtendedUnbinnedNLL._call(self, args)
841 def _call(self, args):
842 data = self._masked
--> 843 ns, x = self._model(data, *args)
844 x = _normalize_model_output(
845 x, "Model should return numpy array in second position"
846 )
847 if self._log:
Input In [2], in model(x, b0, b1, b2, sig, mu, sigma)
4 def model(x, b0, b1, b2, sig, mu, sigma):
5 beta = [b0, b1, b2]
----> 6 bint = np.diff(bernstein.integral(xrange, beta, *xrange))
7 sint = sig * np.diff(norm.cdf(xrange, mu, sigma))[0]
8 return bint + sint, bernstein.density(x, beta, *xrange) + sig * norm.pdf(x, mu, sigma)
NameError: name 'bernstein' is not defined
In this example, the signal amplitude came out negative. This happens if the background has an underfluctuation where the signal is expected. This is not an issue if the sum of signal and background density is still positive everywhere where we evaluate it. As long as the total density is positive, individual components are allowed to be negative.
There are, however, no principle restrictions in this example that prevent the sum of signal and background from becoming negative for some toy data sets. When that happens, the fit will fail, since the total density cannot mathematically become negative.
If this happens anyway, the fit will fail since taking logarithm of a negative number will cause havoc.
Migrad fit on toys¶
We apply the fit many times on randomly sampled background-only data to observe this.
[4]:
@joblib.delayed
def compute(itry):
rng = np.random.default_rng(itry)
x = rng.uniform(0, 1, size=35)
cost = ExtendedUnbinnedNLL(x, model)
m = Minuit(cost, b0=n, b1=n, b2=n, sig=0, mu=0.5, sigma=0.05)
m.limits["b0", "b1", "b2"] = (0, None)
m.fixed["mu", "sigma"] = True
m.migrad()
return m.values["sig"] if m.valid else np.nan
sigs_migrad = joblib.Parallel(-1)(compute(i) for i in range(200))
print(np.sum(np.isnan(sigs_migrad)), "failed")
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 @joblib.delayed
2 def compute(itry):
3 rng = np.random.default_rng(itry)
4 x = rng.uniform(0, 1, size=35)
NameError: name 'joblib' is not defined
[5]:
nfailed = np.sum(np.isnan(sigs_migrad))
plt.title(f"{nfailed} fits failed ({nfailed / len(sigs_migrad) * 100:.0f} %)")
plt.hist(sigs_migrad, bins=10, range=(-10, 10))
plt.xlabel("sig");
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Input In [5], in <cell line: 1>()
----> 1 nfailed = np.sum(np.isnan(sigs_migrad))
2 plt.title(f"{nfailed} fits failed ({nfailed / len(sigs_migrad) * 100:.0f} %)")
3 plt.hist(sigs_migrad, bins=10, range=(-10, 10))
NameError: name 'sigs_migrad' is not defined
The distribution of the signal amplitude looks fairly gaussian which is good, but the fit failed to converge in a few cases due to the problem just described. Simply discarding these cases is not acceptable, it would distort conclusions drawn from the distribution of the test statistic, which is commonly needed to set limits or to compute the p-value for an observed amplitude.
We can repair this by placing a limit on the signal amplitude. This is a suitable solution, although it will bias the signal amplitude and change the shape of the distribution of the test statistic.
An alternative is to perform a constrained minimization, which allows us to directly add a condition to the fit that the model density must be positive at each data point. We merely need to replace the call m.migrad
with the call m.scipy
and pass the (non-linear) constraint. An appropriate algorithm is automatically selected which performs a constrained minimization. The SciPy minimizers are fully integrated into Minuit, which means that Minuit computes an EDM value for the minimum and
parameter uncertainties.
SciPy constrained fit on toys¶
We run SciPy with the constraint on the same simulated samples on which we ran Migrad before.
[6]:
from scipy.optimize import NonlinearConstraint
@joblib.delayed
def compute(itry):
rng = np.random.default_rng(itry)
x = rng.uniform(0, 1, size=35)
cost = ExtendedUnbinnedNLL(x, model)
m = Minuit(cost, b0=n, b1=n, b2=n, sig=0, mu=0.5, sigma=0.05)
m.limits["b0", "b1", "b2"] = (0, None)
m.fixed["mu", "sigma"] = True
m.scipy(constraints=NonlinearConstraint(lambda *par: model(x, *par)[1], 0, np.inf))
return m.values["sig"] if m.valid else np.nan
sigs_constrained = joblib.Parallel(-1)(compute(i) for i in range(200))
print(np.sum(np.isnan(sigs_constrained)), "failed")
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
Input In [6], in <cell line: 1>()
----> 1 from scipy.optimize import NonlinearConstraint
3 @joblib.delayed
4 def compute(itry):
5 rng = np.random.default_rng(itry)
File /usr/lib/python3.10/site-packages/scipy/optimize/__init__.py:400, in <module>
1 """
2 =====================================================
3 Optimization and root finding (:mod:`scipy.optimize`)
(...)
397
398 """
--> 400 from ._optimize import *
401 from ._minimize import *
402 from ._root import *
File /usr/lib/python3.10/site-packages/scipy/optimize/_optimize.py:33, in <module>
30 from numpy import (atleast_1d, eye, argmin, zeros, shape, squeeze,
31 asarray, sqrt, Inf, asfarray)
32 import numpy as np
---> 33 from scipy.sparse.linalg import LinearOperator
34 from ._linesearch import (line_search_wolfe1, line_search_wolfe2,
35 line_search_wolfe2 as line_search,
36 LineSearchWarning)
37 from ._numdiff import approx_derivative
File /usr/lib/python3.10/site-packages/scipy/sparse/__init__.py:267, in <module>
264 import warnings as _warnings
266 from ._base import *
--> 267 from ._csr import *
268 from ._csc import *
269 from ._lil import *
File /usr/lib/python3.10/site-packages/scipy/sparse/_csr.py:10, in <module>
7 import numpy as np
9 from ._base import spmatrix
---> 10 from ._sparsetools import (csr_tocsc, csr_tobsr, csr_count_blocks,
11 get_csr_submatrix)
12 from ._sputils import upcast, get_index_dtype
14 from ._compressed import _cs_matrix
ImportError: numpy.core.multiarray failed to import
[7]:
plt.title(f"{np.sum(np.isnan(sigs_constrained))} constrained fits failed")
plt.hist(sigs_migrad, alpha=0.5, bins=10, range=(-10, 10), label="Migrad")
plt.hist(sigs_constrained, alpha=0.5, bins=10, range=(-10, 10), label=m.fmin.algorithm)
plt.xlabel("sig")
plt.legend();
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Input In [7], in <cell line: 1>()
----> 1 plt.title(f"{np.sum(np.isnan(sigs_constrained))} constrained fits failed")
2 plt.hist(sigs_migrad, alpha=0.5, bins=10, range=(-10, 10), label="Migrad")
3 plt.hist(sigs_constrained, alpha=0.5, bins=10, range=(-10, 10), label=m.fmin.algorithm)
NameError: name 'plt' is not defined
There are no failures this time.
For sig > 0, the distributions are identical in this example, as theoretically expected. In practice, there can be small bin migration effects due to finite precision of numerical algorithms. These are not of concern.
Important are the differences for sig < 0, where Migrad did not converge in a few cases and where therefore samples are missing. Those missing samples are recovered in the distribution produced by the constrained fit.
This demonstrates that it is important to not discard failed fits, as this will in general distort the distribution of the test statistic.
Bonus: unconstrained SciPy fit¶
The issues we describe here are of a principal mathematical nature. We should not expect that an unconstrained minimiser from SciPy does better than Migrad, but let’s test this assumption. The minimiser that SciPy uses when only box constraints are used is the L-BFGS-B method which is roughly comparable to Migrad. Let us see how well this algorithm does on the same toy samples.
[8]:
@joblib.delayed
def compute(itry):
rng = np.random.default_rng(itry)
x = rng.uniform(0, 1, size=35)
cost = ExtendedUnbinnedNLL(x, model)
m = Minuit(cost, b0=n, b1=n, b2=n, sig=0, mu=0.5, sigma=0.05)
m.limits["b0", "b1", "b2"] = (0, None)
m.fixed["mu", "sigma"] = True
m.scipy()
return m.values["sig"] if m.valid else np.nan
sigs_bfgs = joblib.Parallel(-1)(compute(i) for i in range(200))
print(np.sum(np.isnan(sigs_bfgs)), "failed")
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Input In [8], in <cell line: 1>()
----> 1 @joblib.delayed
2 def compute(itry):
3 rng = np.random.default_rng(itry)
4 x = rng.uniform(0, 1, size=35)
NameError: name 'joblib' is not defined
[9]:
plt.title(f"{np.sum(np.isnan(sigs_bfgs))} BFGS fits failed")
plt.hist(sigs_migrad, alpha=0.5, bins=10, range=(-10, 10), label="Migrad")
plt.hist(sigs_constrained, alpha=0.5, bins=10, range=(-10, 10), label="SciPy[SLSQS]")
plt.hist(sigs_bfgs, bins=10, range=(-10, 10), fill=False, label="SciPy[L-BFGS-B]")
plt.xlabel("sig")
plt.legend();
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Input In [9], in <cell line: 1>()
----> 1 plt.title(f"{np.sum(np.isnan(sigs_bfgs))} BFGS fits failed")
2 plt.hist(sigs_migrad, alpha=0.5, bins=10, range=(-10, 10), label="Migrad")
3 plt.hist(sigs_constrained, alpha=0.5, bins=10, range=(-10, 10), label="SciPy[SLSQS]")
NameError: name 'plt' is not defined
In this example, the BFGS method actually failed much less than Migrad, but it still fails in some cases, while the constrained fit did not fail at all.
Speed comparison¶
Since constrained fits are so useful, should you use them all the time? Probably not.
Constrained fits are more computationally expensive. Satisfying extra constrains generally slows down convergence. Let’s compare the speed of the three minimisers tested here. We set the strategy to 0, to avoid computing the Hessian matrix automatically, since we want to measure only the time used by the minimiser.
[10]:
m.strategy = 0
[11]:
%timeit -n3 m.reset(); m.migrad()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Input In [11], in <cell line: 1>()
----> 1 get_ipython().run_line_magic('timeit', '-n3 m.reset(); m.migrad()')
File /usr/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2305, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
2303 kwargs['local_ns'] = self.get_local_scope(stack_depth)
2304 with self.builtin_trap:
-> 2305 result = fn(*args, **kwargs)
2306 return result
File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:1166, in ExecutionMagics.timeit(self, line, cell, local_ns)
1163 if time_number >= 0.2:
1164 break
-> 1166 all_runs = timer.repeat(repeat, number)
1167 best = min(all_runs) / number
1168 worst = max(all_runs) / number
File /usr/lib/python3.10/timeit.py:206, in Timer.repeat(self, repeat, number)
204 r = []
205 for i in range(repeat):
--> 206 t = self.timeit(number)
207 r.append(t)
208 return r
File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:156, in Timer.timeit(self, number)
154 gc.disable()
155 try:
--> 156 timing = self.inner(it, self.timer)
157 finally:
158 if gcold:
File <magic-timeit>:1, in inner(_it, _timer)
File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/minuit.py:691, in Minuit.migrad(self, ncall, iterate)
689 if self._precision is not None:
690 migrad.precision = self._precision
--> 691 fm = migrad(ncall, self._tolerance)
692 if fm.is_valid or fm.has_reached_call_limit:
693 break
File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/cost.py:470, in Cost.__call__(self, *args)
455 def __call__(self, *args):
456 """
457 Evaluate the cost function.
458
(...)
468 float
469 """
--> 470 r = self._call(args)
471 if self.verbose >= 1:
472 print(args, "->", r)
File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/cost.py:843, in ExtendedUnbinnedNLL._call(self, args)
841 def _call(self, args):
842 data = self._masked
--> 843 ns, x = self._model(data, *args)
844 x = _normalize_model_output(
845 x, "Model should return numpy array in second position"
846 )
847 if self._log:
Input In [2], in model(x, b0, b1, b2, sig, mu, sigma)
4 def model(x, b0, b1, b2, sig, mu, sigma):
5 beta = [b0, b1, b2]
----> 6 bint = np.diff(bernstein.integral(xrange, beta, *xrange))
7 sint = sig * np.diff(norm.cdf(xrange, mu, sigma))[0]
8 return bint + sint, bernstein.density(x, beta, *xrange) + sig * norm.pdf(x, mu, sigma)
NameError: name 'bernstein' is not defined
[12]:
%timeit -n3 m.reset(); m.scipy()
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
Input In [12], in <cell line: 1>()
----> 1 get_ipython().run_line_magic('timeit', '-n3 m.reset(); m.scipy()')
File /usr/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2305, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
2303 kwargs['local_ns'] = self.get_local_scope(stack_depth)
2304 with self.builtin_trap:
-> 2305 result = fn(*args, **kwargs)
2306 return result
File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:1166, in ExecutionMagics.timeit(self, line, cell, local_ns)
1163 if time_number >= 0.2:
1164 break
-> 1166 all_runs = timer.repeat(repeat, number)
1167 best = min(all_runs) / number
1168 worst = max(all_runs) / number
File /usr/lib/python3.10/timeit.py:206, in Timer.repeat(self, repeat, number)
204 r = []
205 for i in range(repeat):
--> 206 t = self.timeit(number)
207 r.append(t)
208 return r
File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:156, in Timer.timeit(self, number)
154 gc.disable()
155 try:
--> 156 timing = self.inner(it, self.timer)
157 finally:
158 if gcold:
File <magic-timeit>:1, in inner(_it, _timer)
File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/minuit.py:927, in Minuit.scipy(self, method, ncall, hess, hessp, constraints)
887 """
888 Minimize with SciPy algorithms.
889
(...)
924 the tolerance :attr:`tol` has no effect on SciPy minimizers.
925 """
926 try:
--> 927 from scipy.optimize import (
928 minimize,
929 Bounds,
930 NonlinearConstraint,
931 LinearConstraint,
932 )
933 except ImportError as exc:
934 exc.msg += "\n\nPlease install scipy to use scipy minimizers in iminuit."
File /usr/lib/python3.10/site-packages/scipy/optimize/__init__.py:400, in <module>
1 """
2 =====================================================
3 Optimization and root finding (:mod:`scipy.optimize`)
(...)
397
398 """
--> 400 from ._optimize import *
401 from ._minimize import *
402 from ._root import *
File /usr/lib/python3.10/site-packages/scipy/optimize/_optimize.py:33, in <module>
30 from numpy import (atleast_1d, eye, argmin, zeros, shape, squeeze,
31 asarray, sqrt, Inf, asfarray)
32 import numpy as np
---> 33 from scipy.sparse.linalg import LinearOperator
34 from ._linesearch import (line_search_wolfe1, line_search_wolfe2,
35 line_search_wolfe2 as line_search,
36 LineSearchWarning)
37 from ._numdiff import approx_derivative
File /usr/lib/python3.10/site-packages/scipy/sparse/__init__.py:267, in <module>
264 import warnings as _warnings
266 from ._base import *
--> 267 from ._csr import *
268 from ._csc import *
269 from ._lil import *
File /usr/lib/python3.10/site-packages/scipy/sparse/_csr.py:10, in <module>
7 import numpy as np
9 from ._base import spmatrix
---> 10 from ._sparsetools import (csr_tocsc, csr_tobsr, csr_count_blocks,
11 get_csr_submatrix)
12 from ._sputils import upcast, get_index_dtype
14 from ._compressed import _cs_matrix
ImportError: numpy.core.multiarray failed to import
Please install scipy to use scipy minimizers in iminuit.
[13]:
%timeit -n3 m.reset(); m.scipy(constraints=NonlinearConstraint(lambda *par: model(x, *par)[1], 0, np.inf))
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Input In [13], in <cell line: 1>()
----> 1 get_ipython().run_line_magic('timeit', '-n3 m.reset(); m.scipy(constraints=NonlinearConstraint(lambda *par: model(x, *par)[1], 0, np.inf))')
File /usr/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2305, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
2303 kwargs['local_ns'] = self.get_local_scope(stack_depth)
2304 with self.builtin_trap:
-> 2305 result = fn(*args, **kwargs)
2306 return result
File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:1166, in ExecutionMagics.timeit(self, line, cell, local_ns)
1163 if time_number >= 0.2:
1164 break
-> 1166 all_runs = timer.repeat(repeat, number)
1167 best = min(all_runs) / number
1168 worst = max(all_runs) / number
File /usr/lib/python3.10/timeit.py:206, in Timer.repeat(self, repeat, number)
204 r = []
205 for i in range(repeat):
--> 206 t = self.timeit(number)
207 r.append(t)
208 return r
File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:156, in Timer.timeit(self, number)
154 gc.disable()
155 try:
--> 156 timing = self.inner(it, self.timer)
157 finally:
158 if gcold:
File <magic-timeit>:1, in inner(_it, _timer)
NameError: name 'NonlinearConstraint' is not defined
Migrad is the fastest, followed by the L-BFGS-B method. The constrained fit is much slower.
The constrained fit is much slower, since it has to do more work. Why Migrad is faster than L-BFGS-B is not so obvious. There are some general reasons for that, but there may be cases where L-BFGS-B performs better.
Migrad is comparably fast because of its smart stopping criterion. Migrad stops the fit as soon as the improvement of the fitted parameters become small compared to their uncertainties. Migrad was explicitly designed for statistical fits, where the cost function is a log-likelihood or least-squares function. Since it assumes that, it can stops the fit as soon as the parameter improvements become negligible compared to the parameter uncertainty, which is given by the inverse of its internal approximation of the Hessian matrix.
The SciPy minimisers do not expect the cost function to be a log-likelihood or least-squares and thus cannot assume that the Hessian matrix has a special meaning. Instead they tend to optimise until they hit the limits of machine precision. This is the main reason why the L-BFGS-B method is slower. You can also see this in the benchmark section of the documentation.
We can force Migrad to do something similar by setting the tolerance to a tiny value.
[14]:
m.tol = 1e-20
[15]:
%timeit -n3 m.reset(); m.migrad()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Input In [15], in <cell line: 1>()
----> 1 get_ipython().run_line_magic('timeit', '-n3 m.reset(); m.migrad()')
File /usr/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2305, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
2303 kwargs['local_ns'] = self.get_local_scope(stack_depth)
2304 with self.builtin_trap:
-> 2305 result = fn(*args, **kwargs)
2306 return result
File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:1166, in ExecutionMagics.timeit(self, line, cell, local_ns)
1163 if time_number >= 0.2:
1164 break
-> 1166 all_runs = timer.repeat(repeat, number)
1167 best = min(all_runs) / number
1168 worst = max(all_runs) / number
File /usr/lib/python3.10/timeit.py:206, in Timer.repeat(self, repeat, number)
204 r = []
205 for i in range(repeat):
--> 206 t = self.timeit(number)
207 r.append(t)
208 return r
File /usr/lib/python3.10/site-packages/IPython/core/magics/execution.py:156, in Timer.timeit(self, number)
154 gc.disable()
155 try:
--> 156 timing = self.inner(it, self.timer)
157 finally:
158 if gcold:
File <magic-timeit>:1, in inner(_it, _timer)
File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/minuit.py:691, in Minuit.migrad(self, ncall, iterate)
689 if self._precision is not None:
690 migrad.precision = self._precision
--> 691 fm = migrad(ncall, self._tolerance)
692 if fm.is_valid or fm.has_reached_call_limit:
693 break
File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/cost.py:470, in Cost.__call__(self, *args)
455 def __call__(self, *args):
456 """
457 Evaluate the cost function.
458
(...)
468 float
469 """
--> 470 r = self._call(args)
471 if self.verbose >= 1:
472 print(args, "->", r)
File ~/python-iminuit/src/python-iminuit/build/lib.linux-x86_64-3.10/iminuit/cost.py:843, in ExtendedUnbinnedNLL._call(self, args)
841 def _call(self, args):
842 data = self._masked
--> 843 ns, x = self._model(data, *args)
844 x = _normalize_model_output(
845 x, "Model should return numpy array in second position"
846 )
847 if self._log:
Input In [2], in model(x, b0, b1, b2, sig, mu, sigma)
4 def model(x, b0, b1, b2, sig, mu, sigma):
5 beta = [b0, b1, b2]
----> 6 bint = np.diff(bernstein.integral(xrange, beta, *xrange))
7 sint = sig * np.diff(norm.cdf(xrange, mu, sigma))[0]
8 return bint + sint, bernstein.density(x, beta, *xrange) + sig * norm.pdf(x, mu, sigma)
NameError: name 'bernstein' is not defined
Now the runtime of Migrad is closer to L-BFGS-B, but it is still faster in this case.