[BUG]: compare_models much slower in version 3.3.0 than in 2.X

[dfpasdel]

pycaret version checks

• I have checked that this issue has not already been reported here.

• I have confirmed this bug exists on the latest version of pycaret.

• I have confirmed this bug exists on the master branch of pycaret (pip install -U git+https://github.com/pycaret/pycaret.git@master).

Issue Description

With pycaret installed in a conda environment (cannot directly use pip due to old machine), I experience much longer training times (in compare_models()) when going from v2.3.10 to v3.3.0. This seems to be amplified when using polynomial features.

v2.3.10, polynomial_degree=1: 22sec
v2.3.10, polynomial_degree=3: 13sec
v3.3.0, polynomial_degree=1: 25sec
v3.3.0, polynomial_degree=3: 308sec

I note that there are some convergence warnings appearing in v.2.3.10. Could it be that convergence is handled in a different way in v3.3.0? Seems also that when increasing the polynomial degree, TheilSen Regressor is by far the slowest of the regressors (see TT) in v3.3.0.

I could in practice exclude the TheilSen Regressor for my real-world data, but the training time is still around 10 times much longer with v3.3.0 than with v2.3.10.

The steps I followed to install v2.3.10 were:

conda create -n env_p36 python=3.6
conda activate env_p36
pip install pycaret==2.3.10

The steps I followed to install v3.3.0 were:

conda create -n env_p311 python=3.11
conda activate env_p311
pip install joblib==1.3.0
pip install pycaret
pip install lightgbm==3.3.5

I had to manual install joblib (#3959) and lightbgm (#3897)

Reproducible Example

import pycaret
try:
    from pycaret import show_versions
    show_versions()
except:
    import sklearn
    import joblib
    import sys
    import lightgbm
    print("python version: ", sys.version)
    print("pycaret version: ", pycaret.__version__)
    print("scikit-learn version: ", sklearn.__version__)
    print("joblib version: ", joblib.__version__)
    print("lightgbm version: ", lightgbm.__version__)
from pycaret.datasets import get_data
from pycaret.regression import *
from packaging.version import Version
import time

data = get_data('insurance', verbose=False)

for polynomial_degree in [1, 3]:
    print("\n# Polynomial degree: {} #".format(polynomial_degree))
    ts = time.time()
    if Version(pycaret.__version__) >= Version("3.0.0"):
        setup(data, target='charges', session_id=123, verbose=False,
              polynomial_features=True if polynomial_degree > 1 else False,
              polynomial_degree=polynomial_degree)
    else:
        setup(data, target='charges', session_id=123, verbose=False,
              polynomial_features=True if polynomial_degree > 1 else False,
              polynomial_degree=polynomial_degree,
              silent=True)
    print("Setup time: {:.03f}sec".format(time.time() - ts))

    ts = time.time()
    best = compare_models(verbose=True if Version(pycaret.__version__) >= Version("3.0.0") else False,
                          turbo=False)
    print("Compare time: {:.03f}sec".format(time.time() - ts))

Expected Behavior

The function compare_models is expected to have comparable execution times for v2.3.10 and v3.3.0

Actual Results

v2.3.10:

# Polynomial degree: 1 #
Setup time: 0.714sec

/home/david_pas/miniconda3/envs/env_p36/lib/python3.6/site-packages/sklearn/linear_model/_huber.py:296: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
  self.n_iter_ = _check_optimize_result("lbfgs", opt_res, self.max_iter)

/home/david_pas/miniconda3/envs/env_p36/lib/python3.6/site-packages/sklearn/neural_network/_multilayer_perceptron.py:585: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (500) reached and the optimization hasn't converged yet.
  % self.max_iter, ConvergenceWarning)

Compare time: 21.640sec

# Polynomial degree: 3 #
Setup time: 1.740sec

/home/david_pas/miniconda3/envs/env_p36/lib/python3.6/site-packages/sklearn/linear_model/_ridge.py:148: LinAlgWarning: Ill-conditioned matrix (rcond=2.10934e-13): result may not be accurate.
  overwrite_a=True).T
/home/david_pas/miniconda3/envs/env_p36/lib/python3.6/site-packages/sklearn/linear_model/_least_angle.py:578: ConvergenceWarning: Regressors in active set degenerate. Dropping a regressor, after 16 iterations, i.e. alpha=9.062e-04, with an active set of 15 regressors, and the smallest cholesky pivot element being 2.220e-16. Reduce max_iter or increase eps parameters.
  ConvergenceWarning)
/home/david_pas/miniconda3/envs/env_p36/lib/python3.6/site-packages/sklearn/linear_model/_ridge.py:190: UserWarning: Singular matrix in solving dual problem. Using least-squares solution instead.
  warnings.warn("Singular matrix in solving dual problem. Using "

Compare time: 13.054sec


==============================================
v3.3.0:

# Polynomial degree: 1 #
Setup time: 8.034sec
                                      Model        MAE           MSE  \         
gbr             Gradient Boosting Regressor  2701.9927  2.354898e+07   
rf                  Random Forest Regressor  2771.4583  2.541650e+07   
lightgbm    Light Gradient Boosting Machine  2992.1828  2.552104e+07   
et                    Extra Trees Regressor  2833.3624  2.842784e+07   
ada                      AdaBoost Regressor  4316.0568  2.922051e+07   
ard       Automatic Relevance Determination  4277.1505  3.827042e+07   
lasso                      Lasso Regression  4303.7697  3.838680e+07   
ridge                      Ridge Regression  4317.6984  3.839644e+07   
lar                  Least Angle Regression  4303.5559  3.838806e+07   
llar           Lasso Least Angle Regression  4303.7694  3.838682e+07   
br                           Bayesian Ridge  4311.2349  3.839195e+07   
kr                             Kernel Ridge  4278.9083  3.847978e+07   
lr                        Linear Regression  4303.5559  3.838806e+07   
tr                       TheilSen Regressor  4127.0396  4.225931e+07   
huber                       Huber Regressor  3463.2216  4.880111e+07   
dt                  Decision Tree Regressor  3383.4916  4.782320e+07   
ransac              Random Sample Consensus  3536.5334  5.686206e+07   
par            Passive Aggressive Regressor  4537.0122  6.734631e+07   
en                              Elastic Net  7372.5238  9.045078e+07   
mlp                           MLP Regressor  9096.3734  1.281837e+08   
omp             Orthogonal Matching Pursuit  9089.9268  1.334394e+08   
knn                   K Neighbors Regressor  8007.1197  1.313800e+08   
dummy                       Dummy Regressor  9192.5418  1.485168e+08   
svm               Support Vector Regression  8394.3603  1.639016e+08   

                RMSE      R2   RMSLE    MAPE  TT (Sec)  
gbr        4832.9682  0.8320  0.4447  0.3137     0.035  
rf         5028.6343  0.8172  0.4690  0.3303     0.052  
lightgbm   5042.0978  0.8149  0.5378  0.3751     0.030  
et         5305.6516  0.7991  0.4877  0.3363     0.045  
ada        5398.4561  0.7903  0.6368  0.7394     0.025  
ard        6166.6403  0.7318  0.6114  0.4383     0.025  
lasso      6176.4824  0.7306  0.5952  0.4434     0.159  
ridge      6177.2329  0.7306  0.5891  0.4459     0.161  
lar        6176.5920  0.7306  0.5949  0.4433     0.019  
llar       6176.4846  0.7306  0.5952  0.4434     0.020  
br         6176.8896  0.7306  0.5910  0.4447     0.020  
kr         6183.0863  0.7306  0.5303  0.4338     0.028  
lr         6176.5920  0.7306  0.5949  0.4433     0.907  
tr         6463.7502  0.7093  0.4888  0.3456     0.172  
huber      6963.9984  0.6544  0.4927  0.2212     0.028  
dt         6895.7016  0.6497  0.5602  0.4013     0.020  
ransac     7482.7024  0.6096  0.5016  0.2140     0.023  
par        8142.7826  0.5422  0.5276  0.3207     0.020  
en         9468.3193  0.3792  0.7342  0.9184     0.136  
mlp       11272.1799  0.1209  0.9120  1.2986     0.158  
omp       11488.4238  0.0884  0.8790  1.1514     0.019  
knn       11424.9843  0.0860  0.8535  0.9231     0.023  
dummy     12132.4733 -0.0175  1.0154  1.5637     0.017  
svm       12709.7341 -0.1094  0.9287  1.0291     0.029  
Compare time: 25.142sec

# Polynomial degree: 3 #
Setup time: 1.866sec
                                      Model            MAE           MSE  \     
gbr             Gradient Boosting Regressor   2.741850e+03  2.459156e+07   
omp             Orthogonal Matching Pursuit   3.172104e+03  2.550629e+07   
br                           Bayesian Ridge   3.163002e+03  2.553838e+07   
ard       Automatic Relevance Determination   3.160258e+03  2.590568e+07   
llar           Lasso Least Angle Regression   3.169176e+03  2.601389e+07   
en                              Elastic Net   3.191751e+03  2.627749e+07   
mlp                           MLP Regressor   3.056204e+03  2.623633e+07   
rf                  Random Forest Regressor   2.786274e+03  2.610734e+07   
lightgbm    Light Gradient Boosting Machine   3.059322e+03  2.606371e+07   
lasso                      Lasso Regression   3.214349e+03  2.706637e+07   
ridge                      Ridge Regression   3.211529e+03  2.709924e+07   
kr                             Kernel Ridge   3.224810e+03  2.716856e+07   
lr                        Linear Regression   3.242496e+03  2.753365e+07   
tr                       TheilSen Regressor   3.291581e+03  2.804094e+07   
et                    Extra Trees Regressor   2.800217e+03  2.852415e+07   
ada                      AdaBoost Regressor   5.765607e+03  4.315071e+07   
ransac              Random Sample Consensus   3.101142e+03  4.889437e+07   
dt                  Decision Tree Regressor   3.209872e+03  4.636549e+07   
huber                       Huber Regressor   3.479201e+03  5.339289e+07   
knn                   K Neighbors Regressor   8.503332e+03  1.392339e+08   
dummy                       Dummy Regressor   9.192542e+03  1.485168e+08   
svm               Support Vector Regression   8.357960e+03  1.637195e+08   
par            Passive Aggressive Regressor   9.823600e+03  2.491201e+08   
lar                  Least Angle Regression  3.748527e+193           inf   

                RMSE      R2     RMSLE           MAPE  TT (Sec)  
gbr        4944.6989  0.8248    0.4469   3.295000e-01     0.124  
omp        5029.2743  0.8211    0.4514   3.547000e-01     0.018  
br         5034.0706  0.8206    0.4476   3.514000e-01     0.021  
ard        5073.1605  0.8177    0.4409   3.367000e-01     1.195  
llar       5082.5255  0.8171    0.4517   3.462000e-01     0.020  
en         5106.6528  0.8153    0.4904   3.465000e-01     0.034  
mlp        5100.6302  0.8151    0.4410   3.154000e-01     0.267  
rf         5096.4530  0.8140    0.4767   3.287000e-01     0.204  
lightgbm   5087.0118  0.8131    0.5240   3.775000e-01     2.679  
lasso      5183.7249  0.8102    0.4716   3.484000e-01     0.031  
ridge      5183.7561  0.8102    0.5222   3.501000e-01     0.021  
kr         5191.1595  0.8096    0.4936   3.515000e-01     0.030  
lr         5228.8457  0.8066    0.5088   3.585000e-01     0.020  
tr         5275.2620  0.8022    0.4979   3.607000e-01    23.060  
et         5321.6815  0.7973    0.4874   3.342000e-01     0.134  
ada        6522.9362  0.7010    0.7902   1.109100e+00     0.058  
ransac     6828.0108  0.6671    0.5243   1.902000e-01     0.092  
dt         6792.5696  0.6649    0.5593   3.840000e-01     0.021  
huber      7274.5182  0.6247    0.4519   1.727000e-01     0.994  
knn       11753.8380  0.0352    0.8867   1.067200e+00     0.026  
dummy     12132.4733 -0.0175    1.0154   1.563700e+00     0.020  
svm       12702.3868 -0.1081    0.9258   1.018600e+00     0.035  
par       12942.3473 -0.5678    0.9606   1.304200e+00     0.019  
lar              inf    -inf  102.6190  5.760759e+189     0.025
Compare time: 308.384sec

Installed Versions

v2.3.10

python version: 3.6.13 |Anaconda, Inc.| (default, Jun 4 2021, 14:25:59) [GCC 7.5.0] pycaret version: 2.3.10 scikit-learn version: 0.23.2 joblib version: 1.0.1 lightgbm version: 3.3.5

v3.3.0

System: python: 3.11.8 (main, Feb 26 2024, 21:39:34) [GCC 11.2.0] executable: /home/david_pas/miniconda3/envs/env_p311/bin/python3 machine: Linux-4.15.0-135-generic-x86_64-with-glibc2.27

PyCaret required dependencies:
pip: 23.3.1
setuptools: 68.2.2
pycaret: 3.3.0
IPython: 8.23.0
ipywidgets: 8.1.2
tqdm: 4.66.2
numpy: 1.26.4
pandas: 2.1.4
jinja2: 3.1.3
scipy: 1.11.4
joblib: 1.3.0
sklearn: 1.4.2
pyod: 1.1.3
imblearn: 0.12.2
category_encoders: 2.6.3
lightgbm: 3.3.5
numba: 0.59.1
requests: 2.31.0
matplotlib: 3.7.5
scikitplot: 0.3.7
yellowbrick: 1.5
plotly: 5.20.0
plotly-resampler: Not installed
kaleido: 0.2.1
schemdraw: 0.15
statsmodels: 0.14.1
sktime: 0.28.0
tbats: 1.1.3
pmdarima: 2.0.4
psutil: 5.9.8
markupsafe: 2.1.5
pickle5: Not installed
cloudpickle: 3.0.0
deprecation: 2.1.0
xxhash: 3.4.1
wurlitzer: 3.0.3

PyCaret optional dependencies:
shap: Not installed
interpret: Not installed
umap: Not installed
ydata_profiling: Not installed
explainerdashboard: Not installed
autoviz: Not installed
fairlearn: Not installed
deepchecks: Not installed
xgboost: Not installed
catboost: Not installed
kmodes: Not installed
mlxtend: Not installed
statsforecast: Not installed
tune_sklearn: Not installed
ray: Not installed
hyperopt: Not installed
optuna: Not installed
skopt: Not installed
mlflow: Not installed
gradio: Not installed
fastapi: Not installed
uvicorn: Not installed
m2cgen: Not installed
evidently: Not installed
fugue: Not installed
streamlit: Not installed
prophet: Not installed