Logistic regression is a supervised learning technique applied to classification problems.
In this post, let us explore:
- Logistic Regression model
- Advantages
- Disadvantages
- Example
- Hyperparemeters and Tuning
Logistic Regression model
Use sigmoid function to model the probability of dependent variable being 1 or 0 (binary classification).
Now we will see how to derive the log of odds ratio [p/(1-p)].
As shown in the pictures above, log of odds ratio (logit) is linear function of independent variables.
Advantage
- We can interpret the coefficients (white box model)
We can interpret the results based on the size and sign of the regression coefficients. This is useful to identify the relationship between independent and dependent variables.
Disadvantage
- Requires that each data point be independent of other data points (source)
Example
The data used for demonstrating the logistic regression is from the Titanic dataset. For simplicity I have used only three features (Age, fare and pclass).
#Ravel y=y.values.ravel() from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split #Splitting x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0) #LR model logisticRegr = LogisticRegression() #Fit logisticRegr.fit(x_train, y_train)
The above output shows the default hyperparemeters used in sklearn.
Let us now perform the 5-fold cross-validation and find out the accuracy.
from sklearn.model_selection import cross_val_score
from sklearn.metrics import accuracy_score
#CV scores, 5 fold CV
scores = cross_val_score(logisticRegr, x_train, y_train, cv=5)
#Prediction and accuracy
y_pred = logisticRegr.predict(x_test)
accuracy_test = accuracy_score(y_test, y_pred)
#Print the summary
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
print ("Test Accuracy: %0.2f" % (accuracy_test))# Score
score = logisticRegr.score(x_test, y_test)
print ("Accuracy: %0.2f" % (score))
#Prediction
predictions = logisticRegr.predict(x_test)
predictions
To get the confusion matrix, we can use the following code. We can also visualize the confusion matrix for easier understanding.
from sklearn import metrics
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn import metrics
#confusion matrix
cm = metrics.confusion_matrix(y_test, predictions)
print("Confusion Matrix: \n", cm)
#Visualize the confusion matrix
plt.figure(figsize=(9,9))
sns.heatmap(cm, annot=True, fmt=".0f", linewidths=.5, square = True, cmap = 'Blues_r');
plt.ylabel('Actual Class');
plt.xlabel('Predicted Class');
all_sample_title = "Test Accuracy: %0.2f" % (score)
plt.title(all_sample_title, size = 15);
Hyperparameters
LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
verbose=0, warm_start=False)- C
It is the inverse of regularization strength. Lower the value of C, higher the regularization and hence lower is the chance of overfitting. Default is C=1. - Class weight
By default, sklearn uses ‘balanced’ mode by automatically adjusting the class weights to inverse of frequencies in each class. This is useful while handling class imbalance in dataset. - Fit Intercept
By default, intercept is added to the logistic regression model. - Multiclass
In this we have three options: ovr’, ‘multinomial’, ‘auto’. ‘ovr’ corresponds to One-vs-Rest. Auto selects ‘ovr’ when problem is binary classification, otherwise ‘multinomial’. These are commonly tuned hyperparameters. For more about these read sklearn’s manual.
Tuning the Hyperparameters
#import GridseachCV
from sklearn.model_selection import GridSearchCV
#Instantiate
clf = LogisticRegression()
#Grid
parameter_grid = {'C': [0.01, 0.1, 1, 2, 10, 100], 'penalty': ['l1', 'l2']}
#Gridsearch
gridsearch = GridSearchCV(clf, parameter_grid)
gridsearch.fit(x_train, y_train);
#Get best hyperparameters
gridsearch.best_params_
