V0.1.6 - Building NARX models using general estimators¶
Example created by Wilson Rocha Lacerda Junior
In this example we will create NARX models using different estimator like GrandientBoostingRegressior, Bayesian Regression, Automatic Relevance Determination (ARD) Regression and Catboost
pip install sysidentpy
Requirement already satisfied: sysidentpy in c:\users\wilso\miniconda3\envs\sysidentpy\lib\site-packages (0.1.6)
Requirement already satisfied: scipy>=1.7.0 in c:\users\wilso\miniconda3\envs\sysidentpy\lib\site-packages (from sysidentpy) (1.7.1)
Requirement already satisfied: matplotlib>=3.3.2 in c:\users\wilso\miniconda3\envs\sysidentpy\lib\site-packages (from sysidentpy) (3.4.3)
Requirement already satisfied: numpy>=1.19.2 in c:\users\wilso\miniconda3\envs\sysidentpy\lib\site-packages (from sysidentpy) (1.20.3)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\wilso\miniconda3\envs\sysidentpy\lib\site-packages (from matplotlib>=3.3.2->sysidentpy) (1.3.2)
Requirement already satisfied: python-dateutil>=2.7 in c:\users\wilso\miniconda3\envs\sysidentpy\lib\site-packages (from matplotlib>=3.3.2->sysidentpy) (2.8.2)
Requirement already satisfied: pyparsing>=2.2.1 in c:\users\wilso\miniconda3\envs\sysidentpy\lib\site-packages (from matplotlib>=3.3.2->sysidentpy) (2.4.7)
Requirement already satisfied: pillow>=6.2.0 in c:\users\wilso\miniconda3\envs\sysidentpy\lib\site-packages (from matplotlib>=3.3.2->sysidentpy) (8.3.2)
Requirement already satisfied: cycler>=0.10 in c:\users\wilso\miniconda3\envs\sysidentpy\lib\site-packages (from matplotlib>=3.3.2->sysidentpy) (0.10.0)
Requirement already satisfied: six in c:\users\wilso\miniconda3\envs\sysidentpy\lib\site-packages (from cycler>=0.10->matplotlib>=3.3.2->sysidentpy) (1.16.0)
Note: you may need to restart the kernel to use updated packages.
import matplotlib.pyplot as plt
from sysidentpy.metrics import mean_squared_error
from sysidentpy.utils.generate_data import get_siso_data
from sysidentpy.general_estimators import NARX
from sklearn.linear_model import BayesianRidge, ARDRegression
from sklearn.ensemble import GradientBoostingRegressor
from catboost import CatBoostRegressor
10-24 13:10:41 - INFO - Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
10-24 13:10:41 - INFO - NumExpr defaulting to 8 threads.
# simulated dataset
x_train, x_valid, y_train, y_valid = get_siso_data(n=10000,
colored_noise=False,
sigma=0.01,
train_percentage=80)
Importance of the NARX architecture¶
To get an idea of the importance of the NARX architecture, lets take a look in the performance of the models without the NARX configuration.
catboost = CatBoostRegressor(iterations=300,
learning_rate=0.1,
depth=6)
gb = GradientBoostingRegressor(loss='quantile', alpha=0.90,
n_estimators=250, max_depth=10,
learning_rate=.1, min_samples_leaf=9,
min_samples_split=9)
def plot_results(yvalid, yhat):
_, ax = plt.subplots(figsize=(14, 8))
ax.plot(y_valid[:200], label='Data', marker='o')
ax.plot(yhat[:200], label='Prediction', marker='*')
ax.set_xlabel("$n$", fontsize=18)
ax.set_ylabel("$y[n]$", fontsize=18)
ax.grid()
ax.legend(fontsize=18)
plt.show()
catboost.fit(x_train, y_train, verbose=False)
plot_results(y_valid, catboost.predict(x_valid))
gb.fit(x_train, y_train)
plot_results(y_valid, gb.predict(x_valid))
C:\Users\wilso\miniconda3\envs\sysidentpy\lib\site-packages\sklearn\utils\validation.py:63: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
return f(*args, **kwargs)
Introducing the NARX configuration using SysIdentPy¶
As you can see, you just need to pass the base estimator you want to the NARX class from SysIdentPy do build the NARX model! You can choose the lags of the input and output variables to build the regressor matrix.
We keep the fit/predict method to make the process straightforward.
NARX with Catboost¶
from sysidentpy.general_estimators import NARX
catboost_narx = NARX(base_estimator=CatBoostRegressor(iterations=300,
learning_rate=0.1,
depth=6),
xlag=2,
ylag=2,
fit_params={'verbose': False}
)
catboost_narx.fit(x_train, y_train)
yhat = catboost_narx.predict(x_valid, y_valid)
print(mean_squared_error(y_valid, yhat))
ee, ex, extras, lam = catboost_narx.residuals(x_valid, y_valid, yhat)
catboost_narx.plot_result(y_valid, yhat, ee, ex, n=200)
10-24 13:10:46 - INFO - Training the model
10-24 13:10:46 - INFO - Creating the regressor matrix
10-24 13:10:46 - INFO - The regressor matrix have 5 features
10-24 13:10:46 - INFO - Done! Model is built!
0.0001582854239779783
NARX with Gradient Boosting¶
from sysidentpy.general_estimators import NARX
gb_narx = NARX(base_estimator=GradientBoostingRegressor(loss='quantile', alpha=0.90,
n_estimators=250, max_depth=10,
learning_rate=.1, min_samples_leaf=9,
min_samples_split=9),
xlag=2,
ylag=2
)
gb_narx.fit(x_train, y_train)
yhat = gb_narx.predict(x_valid, y_valid)
print(mean_squared_error(y_valid, yhat))
ee, ex, extras, lam = gb_narx.residuals(x_valid, y_valid, yhat)
gb_narx.plot_result(y_valid, yhat, ee, ex, n=200)
10-24 13:10:49 - INFO - Training the model
10-24 13:10:49 - INFO - Creating the regressor matrix
10-24 13:10:49 - INFO - The regressor matrix have 5 features
C:\Users\wilso\miniconda3\envs\sysidentpy\lib\site-packages\sklearn\utils\validation.py:63: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
return f(*args, **kwargs)
10-24 13:10:57 - INFO - Done! Model is built!
0.0010076218299222446
NARX with ARD¶
from sysidentpy.general_estimators import NARX
ARD_narx = NARX(base_estimator=ARDRegression(),
xlag=2,
ylag=2
)
ARD_narx.fit(x_train, y_train)
yhat = ARD_narx.predict(x_valid, y_valid)
print(mean_squared_error(y_valid, yhat))
ee, ex, extras, lam = ARD_narx.residuals(x_valid, y_valid, yhat)
ARD_narx.plot_result(y_valid, yhat, ee, ex, n=200)
10-24 13:10:58 - INFO - Training the model
10-24 13:10:58 - INFO - Creating the regressor matrix
10-24 13:10:58 - INFO - The regressor matrix have 5 features
10-24 13:10:58 - INFO - Done! Model is built!
C:\Users\wilso\miniconda3\envs\sysidentpy\lib\site-packages\sklearn\utils\validation.py:63: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
return f(*args, **kwargs)
0.001062559412360286
NARX with Bayesian Rigde¶
from sysidentpy.general_estimators import NARX
BayesianRidge_narx = NARX(base_estimator=BayesianRidge(),
xlag=2,
ylag=2
)
BayesianRidge_narx.fit(x_train, y_train)
yhat = BayesianRidge_narx.predict(x_valid, y_valid)
print(mean_squared_error(y_valid, yhat))
ee, ex, extras, lam = BayesianRidge_narx.residuals(x_valid, y_valid, yhat)
BayesianRidge_narx.plot_result(y_valid, yhat, ee, ex, n=200)
10-24 13:10:59 - INFO - Training the model
10-24 13:10:59 - INFO - Creating the regressor matrix
10-24 13:10:59 - INFO - The regressor matrix have 5 features
10-24 13:10:59 - INFO - Done! Model is built!
C:\Users\wilso\miniconda3\envs\sysidentpy\lib\site-packages\sklearn\utils\validation.py:63: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
return f(*args, **kwargs)
0.0010596798644412848
