{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use Extended Least Squares\n",
    "\n",
    "Example created by Wilson Rocha Lacerda Junior\n",
    "\n",
    "> **Looking for more details on NARMAX models?**\n",
    "> For comprehensive information on models, methods, and a wide range of examples and benchmarks implemented in SysIdentPy, check out our book:\n",
    "> [*Nonlinear System Identification and Forecasting: Theory and Practice With SysIdentPy*](https://sysidentpy.org/book/0%20-%20Preface/)\n",
    ">\n",
    "> This book provides in-depth guidance to support your work with SysIdentPy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the **Extended Least Squares (ELS)** algorithm, set the `unbiased` parameter to `True` when defining the parameter estimator algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sysidentpy.parameter_estimation import LeastSquares\n",
    "\n",
    "estimator = LeastSquares(unbiased=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The `unbiased` hyperparameter is available in all parameter estimation algorithms, with a default value of `False`.\n",
    "\n",
    "Additionally, the **Extended Least Squares** algorithm is iterative. In **SysIdentPy**, the default number of iterations is set to 20 (`uiter=20`), as studies in the literature indicate that the algorithm typically converges within 10 to 20 iterations. However, you can adjust this value to any number of iterations you prefer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sysidentpy.parameter_estimation import LeastSquares\n",
    "\n",
    "estimator = LeastSquares(unbiased=True, uiter=40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple yet complete code example demonstrating parameter estimation using the **Extended Least Squares (ELS)** algorithm is shown below.\n",
    "\n",
    "*(Simulated data is used for illustrative purposes.)*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import gaussian_kde\n",
    "\n",
    "from sysidentpy.model_structure_selection import FROLS\n",
    "from sysidentpy.basis_function import Polynomial\n",
    "from sysidentpy.parameter_estimation import LeastSquares\n",
    "from sysidentpy.utils.generate_data import get_siso_data\n",
    "\n",
    "x_train, x_valid, y_train, y_valid = get_siso_data(\n",
    "    n=1000, colored_noise=True, sigma=0.2, train_percentage=90\n",
    ")\n",
    "\n",
    "basis_function = Polynomial(degree=2)\n",
    "estimator = LeastSquares(unbiased=True)\n",
    "parameters = np.zeros([3, 50])\n",
    "\n",
    "for i in range(50):\n",
    "    x_train, x_valid, y_train, y_valid = get_siso_data(\n",
    "        n=3000, colored_noise=True, train_percentage=90\n",
    "    )\n",
    "\n",
    "    model = FROLS(\n",
    "        order_selection=False,\n",
    "        n_terms=3,\n",
    "        ylag=2,\n",
    "        xlag=2,\n",
    "        elag=2,\n",
    "        info_criteria=\"aic\",\n",
    "        estimator=estimator,\n",
    "        basis_function=basis_function,\n",
    "    )\n",
    "\n",
    "    model.fit(X=x_train, y=y_train)\n",
    "    parameters[:, i] = model.theta.flatten()\n",
    "\n",
    "plt.figure(figsize=(14, 4))\n",
    "\n",
    "# Compute and plot KDE for each parameter using scipy's gaussian_kde\n",
    "x_grid = np.linspace(np.min(parameters), np.max(parameters), 1000)\n",
    "\n",
    "for i, label in enumerate([\"Parameter 1\", \"Parameter 2\", \"Parameter 3\"]):\n",
    "    kde = gaussian_kde(parameters[i, :])\n",
    "    plt.plot(x_grid, kde(x_grid), label=label)\n",
    "\n",
    "# Plot vertical lines where the real values must lie\n",
    "plt.axvline(x=0.1, color=\"k\", linestyle=\"--\", label=\"Real Value 0.1\")\n",
    "plt.axvline(x=0.2, color=\"k\", linestyle=\"--\", label=\"Real Value 0.2\")\n",
    "plt.axvline(x=0.9, color=\"k\", linestyle=\"--\", label=\"Real Value 0.9\")\n",
    "\n",
    "plt.xlabel(\"Parameter Value\")\n",
    "plt.ylabel(\"Density\")\n",
    "plt.title(\"Kernel Density Estimate of Parameters (Matplotlib only)\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sysidentpyv04",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}