In this post, I will walk you through the Logistic Regression algorithm step-by-step.

- We will develop the code for the algorithm from scratch using Python.
- We will run the algorithm on real-world data sets from the UCI Machine Learning Repository.

# Table of Contents

**What is Logistic Regression?****Logistic Regression Algorithm Design****Logistic Regression Algorithm in Python, Coded From Scratch****Logistic Regression Output**

# What is Logistic Regression?

Logistic regression, contrary to the name, is a classification algorithm. Unlike linear regression which outputs a continuous value (e.g. house price) for the prediction, Logistic Regression transforms the output into a probability value (i.e. a number between 0 and 1) using what is known as the logistic sigmoid function. This function is also known as the squashing function since it maps a line — that can run from negative infinity to positive infinity along the y-axis — to a number between 0 and 1.

Here is what the graph of the sigmoid function looks like:

The function is called the sigmoid function because it is s-shaped. Here is what the sigmoid function looks like in mathematical notation:

where:

- h(z) is the predicted probability of a given instance (i.e. example) being in the positive class…that is the class represented as 1 in a data set. For example, in an e-mail classification data set, this would be the probability that a given e-mail instance is spam (If h(z) = 0.73, for example, that would mean that the instance has a 73% chance of being spam).
- 1- h(z) is the probability of an instance being in the negative class, the class represented as 0 (e.g. not spam). h(z) is always a number between 0 and 1. Going back to the example in the bullet point above, this would mean that the instance has a 27% change of being not spam.
- z is the input (e.g. a weighted sum of the attributes of a given instance)
- e is Euler’s number

z is commonly expressed as the
dot product, **w** · **x, **where w is a 1-dimensional vector containing
the weights for each attribute, and x is a vector containing the values of each
attribute for a specific instance of the data set (i.e. example).

Often the dot product, w · x,
is written as matrix multiplication. In that case, z = w^{T}x where T
means transpose of the single dimensional weight vector w. The symbol Ɵ
is often used in place of w.

So substituting **w** · **x
**into the sigmoid equation, we getthe following equation:

where

- w is a 1-dimensional vector containing the weights for each attribute.
- The subscript w on h
_{w }means the attributes x are weighted by the weight vector w. - h
_{w}(x) is the probability (a value between 0 and 1) that an instance is a member of the positive class (i.e. probability an e-mail is spam). - x is a vector containing the values of each attribute for a specific instance of the data set.
- w · x = w
_{0}x_{0}+ w_{1}x_{1}+ w_{2}x_{2}+ …. + w_{d}x_{d }(analogous to the equation of a line y = mx + b from grade school)- d is the number of attributes in the data set

- x
_{0}= 1 by convention, for all instances. This attribute has to be added by the programmer for all instances. It is known formally as the “bias” term.

As is the case for many machine learning algorithms, the starting point for Logistic Regression is to create a trained model. One we have a trained model, we can use it to make predictions on new, unseen instances.

## Training Phase

Creating a trained model entails determining the weight vector w. Once we have the weights, we can make predictions on new unseen examples. All we need are the values of the attributes of those examples (i.e. the x values), and we can weight the x values with the values of w to compute the probabilities h(x) for that example using the sigmoid function.

The rule for making predictions using the sigmoid function is as follows:

- If h
_{w}(x) ≥ 0.5, class = 1 (positive class, e.g. spam) - If h
_{w}(x) < 0.5, class = 0 (negative class, e.g. not spam)

To determine the weights in linear regression, the sum of the squared error was the cost function (where error = actual values – predicted values by the line). The cost function represents how wrong a prediction is. In linear regression, it represents how wrong a line of best fit is on a set of observed training instances. The lower the sum of the squared error, the better a line fits the training data, and, in theory, the better the line will predict new, unseen instances.

Instead, the cost function in Logistic Regression is called cross-entropy. Without getting too detailed into the mathematics and notation of this particular equation, the cross-entropy equation is the one that we want to minimize. Minimizing this equation will yield us a sigmoid curve that best fits the training data and enables us to make the best classification predictions possible for new, unseen test instances. A minimum of the cost function is attained when the gradient of the cost function is close to zero (i.e. the calculated weights stop changing). The formal term for the gradient of the cost function getting close to zero is called convergence.

In order to minimize the cost function, we need to find its gradient (i.e. derivative, slope, etc.) and determine the values for the weight vector w that make its derivative as close to 0 as possible. We cannot just set the gradient to 0 and then enter x-values and calculate the weights directly. Instead, we have to use a method called gradient descent in order to find the weights.

In the gradient descent algorithm for Logistic Regression, we:

- Start off with an empty weight vector (initialized to random values between -0.01 and 0.01). The size of the vector is equal to the number of attributes in the data set.
- Initialize an empty weight change vector initialized to all zeros. The size of the vector is equal to the number of attributes in the data set.
- For each training instance, one at a time.
- a. Make a probability prediction by calculating the weighted sum of the attribute values and running that value through the sigmoid function.
- b. We evaluate the gradient of the cost function by plugging in the actual (i.e. observed) class value and the predicted class value from bullet point 3a above.
- c. The gradient value from 3b gets added to the weight change vector.

- After we finish with the last training instance from 3, we multiply each value in the weight change vector by a learning rate (commonly 0.01).
- The vector from 4 gets added to the empty weight vector to update the weights.
- We then ask two questions
- a. Have the weights continued to change (i.e. is the norm (i.e. magnitude) of the weight change vector less than a certain threshold like 0.001)?
- b. Have we been through the data set less than 10,000 (or whatever we set the maximum iterations to) times?
- c. If the answer is yes to both 6a and 6b, go back to step 2. Otherwise, we return the final weight vector, exiting the algorithm.

The gradient descent pseudocode
for Logistic Regression is provided in Figure 10.6 of *Introduction to Machine Learning* by Ethem Alpaydin (Alpaydin, 2014).

## Testing Phase

Once training is completed, we have the weights and can use these weights, attribute values, and the sigmoid function to make predictions for the set of test instances.

Predictions for a given test instance are made using the aforementioned sigmoid function:

Where the rule for making predictions using the sigmoid function is as follows:

- If h
_{w}(x) ≥ 0.5, class = 1 (positive class, e.g. spam) - If h
_{w}(x) < 0.5, class = 0 (negative class, e.g. not spam)

## Multi-class Logistic Regression

A Multi-class Logistic Regression problem is a twist on the binary Logistic Regression method presented above. Multi-class Logistic Regression can make predictions on both binary and multi-class classification problems.

In order to make predictions for multi-class datasets, we take the training set and create multiple separate binary classification problems (one for each class in the data set). For each of those training sets that we generated, we set the class values for one class to 1 (representing the positive class), and we set all other classes to 0 (i.e. the negative class).

In other words, if there are k classes in a data set, k separate training sets are generated. In each of those k separate training sets, one class is set to 1 and all other classes are set to 0.

In Multi-class Logistic Regression, the training phase entails creating k different weight vectors, one for each class rather than just a single weight vector (which was the case in binary Logistic Regression). Each weight vector will help to predict the probability of an instance being a member of that class. Thus, in the testing phase, when there is an unseen new instance, three different predictions need to be made. This method is called the one-vs-all strategy, sometimes called one-vs-rest.

The rule for making predictions for a given instance are as follows:

- For each new test instance,
- Make k separate probability predictions.

- Pick the class that has the highest probability (i.e. the class that is the most enthusiastic about that instance being a member of its class)

Other multi-class Logistic Regression algorithms include Softmax Regression and the one-vs-one strategy. The one-vs-all strategy was selected due to its popularity as being the default strategy used in practice for many of the well-known machine learning libraries for Python (Rebala, Ravi, & Churiwala, 2019)

## Video

Here is an excellent video on logistic regression that explains the whole process I described above, step-by-step.

# Logistic Regression Algorithm Design

The Logistic Regression algorithm was implemented from scratch. The Breast Cancer, Glass, Iris, Soybean (small), and Vote data sets were preprocessed to meet the input requirements of the algorithms. I used five-fold stratified cross-validation to evaluate the performance of the models.

## Required Data Set Format for Logistic Regression

Columns (0 through N)

- 0: Instance ID
- 1: Attribute 1
- 2: Attribute 2
- 3: Attribute 3
- …
- N: Actual Class

The program then adds two additional columns for the testing set.

- N + 1: Predicted Class
- N + 2: Prediction Correct? (1 if yes, 0 if no)

## Breast Cancer Data Set

This breast cancer data set contains 699 instances, 10 attributes, and a class – malignant or benign (Wolberg, 1992).

### Modification of Attribute Values

The actual class value was changed to “Benign” or “Malignant.”

I transformed the attributes into binary numbers so that the algorithms could process the data properly and efficiently. If attribute value was greater than 5, the value was changed to 1, otherwise it was 0.

### Missing Data

There were 16 missing attribute values, each denoted with a “?”. I chose a random number between 1 and 10 (inclusive) to fill in the data.

## Glass Data Set

This glass data set contains 214 instances, 10 attributes, and 7 classes (German, 1987). The purpose of the data set is to identify the type of glass.

### Modification of Attribute Values

If attribute values were greater than the median of the attribute, value was changed to 1, otherwise it was set to 0.

### Missing Data

There are no missing values in this data set.

## Iris Data Set

This data set contains 3 classes of 50 instances each (150 instances in total), where each class refers to a different type of iris plant (Fisher, 1988).

### Modification of Attribute Values

If attribute values were greater than the median of the attribute, value was changed to 1, otherwise it was set to 0.

### Missing Data

There were no missing attribute values.

## Soybean Data Set (small)

This soybean (small) data set contains 47 instances, 35 attributes, and 4 classes (Michalski, 1980). The purpose of the data set is to determine the disease type.

### Modification of Attribute Values

If attribute values were greater than the median of the attribute, value was changed to 1, otherwise it was set to 0.

### Missing Data

There are no missing values in this data set.

## Vote Data Set

This data set includes votes for each of the U.S. House of Representatives Congressmen (435 instances) on the 16 key votes identified by the Congressional Quarterly Almanac (Schlimmer, 1987). The purpose of the data set is to identify the representative as either a Democrat or Republican.

- 267 Democrats
- 168 Republicans

### Modification of Attribute Values

I did the following modifications:

- Changed all “y” to 1 and all “n” to 0.

### Missing Data

Missing values were denoted as “?”. To fill in those missing values, I chose random number, either 0 (“No”) or 1 (“Yes”).

## Description of Any Tuning Process Applied

Some tuning was performed in this project. The learning rate was set to 0.01 by convention. A higher learning rate (0.5) resulted in poor results for the norm of the gradient (>1).

The stopping criteria for gradient descent was as follows:

- Maximum iterations = 10,000
- Euclidean norm of weight change vector < 0.001

When I tried max iterations at 100, the Euclidean norm of the weight change vector returned high values (> 0.2) which indicated that I needed to set a higher max iterations value in order to have a higher chance of convergence (i.e. weights stop changing) based on the norm stopping criteria.

# Logistic Regression Algorithm in Python, Coded From Scratch

Here are the preprocessed data sets:

Here is the driver code. This is where the main method is located:

import pandas as pd # Import Pandas library import numpy as np # Import Numpy library import five_fold_stratified_cv import logistic_regression # File name: logistic_regression_driver.py # Author: Addison Sears-Collins # Date created: 7/19/2019 # Python version: 3.7 # Description: Driver of the logistic_regression.py program # Required Data Set Format for Disrete Class Values # Columns (0 through N) # 0: Instance ID # 1: Attribute 1 # 2: Attribute 2 # 3: Attribute 3 # ... # N: Actual Class # The logistic_regression.py program then adds 2 additional columns # for the test set. # N + 1: Predicted Class # N + 2: Prediction Correct? (1 if yes, 0 if no) ALGORITHM_NAME = "Logistic Regression" SEPARATOR = "," # Separator for the data set (e.g. "\t" for tab data) def main(): print("Welcome to the " + ALGORITHM_NAME + " Program!") print() # Directory where data set is located data_path = input("Enter the path to your input file: ") #data_path = "iris.txt" # Read the full text file and store records in a Pandas dataframe pd_data_set = pd.read_csv(data_path, sep=SEPARATOR) # Show functioning of the program trace_runs_file = input("Enter the name of your trace runs file: ") #trace_runs_file = "iris_logistic_regression_trace_runs.txt" # Open a new file to save trace runs outfile_tr = open(trace_runs_file,"w") # Testing statistics test_stats_file = input("Enter the name of your test statistics file: ") #test_stats_file = "iris_logistic_regression_test_stats.txt" # Open a test_stats_file outfile_ts = open(test_stats_file,"w") # The number of folds in the cross-validation NO_OF_FOLDS = 5 # Generate the five stratified folds fold0, fold1, fold2, fold3, fold4 = five_fold_stratified_cv.get_five_folds( pd_data_set) training_dataset = None test_dataset = None # Create an empty array of length 5 to store the accuracy_statistics # (classification accuracy) accuracy_statistics = np.zeros(NO_OF_FOLDS) # Run Logistic Regression the designated number of times as indicated by the # number of folds for experiment in range(0, NO_OF_FOLDS): print() print("Running Experiment " + str(experiment + 1) + " ...") print() outfile_tr.write("Running Experiment " + str(experiment + 1) + " ...\n") outfile_tr.write("\n") # Each fold will have a chance to be the test data set if experiment == 0: test_dataset = fold0 training_dataset = pd.concat([ fold1, fold2, fold3, fold4], ignore_index=True, sort=False) elif experiment == 1: test_dataset = fold1 training_dataset = pd.concat([ fold0, fold2, fold3, fold4], ignore_index=True, sort=False) elif experiment == 2: test_dataset = fold2 training_dataset = pd.concat([ fold0, fold1, fold3, fold4], ignore_index=True, sort=False) elif experiment == 3: test_dataset = fold3 training_dataset = pd.concat([ fold0, fold1, fold2, fold4], ignore_index=True, sort=False) else: test_dataset = fold4 training_dataset = pd.concat([ fold0, fold1, fold2, fold3], ignore_index=True, sort=False) accuracy, predictions, weights_for_each_class, no_of_instances_test = ( logistic_regression.logistic_regression(training_dataset,test_dataset)) # Print the trace runs of each experiment print("Accuracy:") print(str(accuracy * 100) + "%") print() print("Classifications:") print(predictions) print() print("Learned Model:") print(weights_for_each_class) print() print("Number of Test Instances:") print(str(no_of_instances_test)) print() outfile_tr.write("Accuracy:") outfile_tr.write(str(accuracy * 100) + "%\n\n") outfile_tr.write("Classifications:\n") outfile_tr.write(str(predictions) + "\n\n") outfile_tr.write("Learned Model:\n") outfile_tr.write(str(weights_for_each_class) + "\n\n") outfile_tr.write("Number of Test Instances:") outfile_tr.write(str(no_of_instances_test) + "\n\n") # Store the accuracy in the accuracy_statistics array accuracy_statistics[experiment] = accuracy outfile_tr.write("Experiments Completed.\n") print("Experiments Completed.\n") # Write to a file outfile_ts.write("----------------------------------------------------------\n") outfile_ts.write(ALGORITHM_NAME + " Summary Statistics\n") outfile_ts.write("----------------------------------------------------------\n") outfile_ts.write("Data Set : " + data_path + "\n") outfile_ts.write("\n") outfile_ts.write("Accuracy Statistics for All 5 Experiments:") outfile_ts.write(np.array2string( accuracy_statistics, precision=2, separator=',', suppress_small=True)) outfile_ts.write("\n") outfile_ts.write("\n") accuracy = np.mean(accuracy_statistics) accuracy *= 100 outfile_ts.write("Classification Accuracy : " + str(accuracy) + "%\n") # Print to the console print() print("----------------------------------------------------------") print(ALGORITHM_NAME + " Summary Statistics") print("----------------------------------------------------------") print("Data Set : " + data_path) print() print() print("Accuracy Statistics for All 5 Experiments:") print(accuracy_statistics) print() print() print("Classification Accuracy : " + str(accuracy) + "%") print() # Close the files outfile_tr.close() outfile_ts.close() main()

Here is the code for logistic regression:

import pandas as pd # Import Pandas library import numpy as np # Import Numpy library # File name: logistic_regression.py # Author: Addison Sears-Collins # Date created: 7/19/2019 # Python version: 3.7 # Description: Multi-class logistic regression using one-vs-all. # Required Data Set Format for Disrete Class Values # Columns (0 through N) # 0: Instance ID # 1: Attribute 1 # 2: Attribute 2 # 3: Attribute 3 # ... # N: Actual Class # This program then adds 2 additional columns for the test set. # N + 1: Predicted Class # N + 2: Prediction Correct? (1 if yes, 0 if no) def sigmoid(z): """ Parameters: z: A real number Returns: 1.0/(1 + np.exp(-z)) """ return 1.0/(1 + np.exp(-z)) def gradient_descent(training_set): """ Gradient descent for logistic regression. Follows method presented in the textbook Introduction to Machine Learning 3rd Edition by Ethem Alpaydin (pg. 252) Parameters: training_set: The training instances as a Numpy array Returns: weights: The vector of weights, commonly called w or THETA """ no_of_columns_training_set = training_set.shape[1] no_of_rows_training_set = training_set.shape[0] # Extract the attributes from the training set. # x is still a 2d array x = training_set[:,:(no_of_columns_training_set - 1)] no_of_attributes = x.shape[1] # Extract the classes from the training set. # actual_class is a 1d array. actual_class = training_set[:,(no_of_columns_training_set - 1)] # Set a learning rate LEARNING_RATE = 0.01 # Set the maximum number of iterations MAX_ITER = 10000 # Set the iteration variable to 0 iter = 0 # Set a flag to determine if we have exceeded the maximum number of # iterations exceeded_max_iter = False # Set the tolerance. When the euclidean norm of the gradient vector # (i.e. magnitude of the changes in the weights) gets below this value, # stop iterating through the while loop GRAD_TOLERANCE = 0.001 norm_of_gradient = None # Set a flag to determine if we have reached the minimum of the # cost (i.e. error) function. converged = False # Create the weights vector with random floats between -0.01 and 0.01 # The number of weights is equal to the number of attributes weights = np.random.uniform(-0.01,0.01,(no_of_attributes)) changes_in_weights = None # Keep running the loop below until convergence on the minimum of the # cost function or we exceed the max number of iterations while(not(converged) and not(exceeded_max_iter)): # Initialize a weight change vector that stores the changes in # the weights at each iteration changes_in_weights = np.zeros(no_of_attributes) # For each training instance for inst in range(0, no_of_rows_training_set): # Calculate weighted sum of the attributes for # this instance output = np.dot(weights, x[inst,:]) # Calculate the sigmoid of the weighted sum # This y is the probability that this instance belongs # to the positive class y = sigmoid(output) # Calculate difference difference = (actual_class[inst] - y) # Multiply the difference by the attribute vector product = np.multiply(x[inst,:], difference) # For each attribute, update the weight changes # i.e. the gradient vector changes_in_weights = np.add(changes_in_weights,product) # Calculate the step size step_size = np.multiply(changes_in_weights, LEARNING_RATE) # Update the weights vector weights = np.add(weights, step_size) # Test to see if we have converged on the minimum of the error # function norm_of_gradient = np.linalg.norm(changes_in_weights) if (norm_of_gradient < GRAD_TOLERANCE): converged = True # Update the number of iterations iter += 1 # If we have exceeded the maximum number of iterations if (iter > MAX_ITER): exceeded_max_iter = True #For debugging purposes #print("Number of Iterations: " + str(iter - 1)) #print("Norm of the gradient: " + str(norm_of_gradient)) #print(changes_in_weights) #print() return weights def logistic_regression(training_set, test_set): """ Multi-class one-vs-all logistic regression Parameters: training_set: The training instances as a Pandas dataframe test_set: The test instances as a Pandas dataframe Returns: accuracy: Classification accuracy as a decimal predictions: Classifications of all the test instances as a Pandas dataframe weights_for_each_class: The weight vectors for each class (one-vs-all) no_of_instances_test: The number of test instances """ # Remove the instance ID column training_set = training_set.drop( training_set.columns[[0]], axis=1) test_set = test_set.drop( test_set.columns[[0]], axis=1) # Make a list of the unique classes list_of_unique_classes = pd.unique(training_set["Actual Class"]) # Replace all the class values with numbers, starting from 0 # in both the test and training sets. for cl in range(0, len(list_of_unique_classes)): training_set["Actual Class"].replace( list_of_unique_classes[cl], cl ,inplace=True) test_set["Actual Class"].replace( list_of_unique_classes[cl], cl ,inplace=True) # Insert a column of 1s in column 0 of both the training # and test sets. This is the bias and helps with gradient # descent. (i.e. X0 = 1 for all instances) training_set.insert(0, "Bias", 1) test_set.insert(0, "Bias", 1) # Convert dataframes to numpy arrays np_training_set = training_set.values np_test_set = test_set.values # Add 2 additional columns to the testing dataframe test_set = test_set.reindex( columns=[*test_set.columns.tolist( ), 'Predicted Class', 'Prediction Correct?']) ############################# Training Phase ############################## no_of_columns_training_set = np_training_set.shape[1] no_of_rows_training_set = np_training_set.shape[0] # Create and store a training set for each unique class # to create separate binary classification # problems trainingsets = [] for cl in range(0, len(list_of_unique_classes)): # Create a copy of the training set temp = np.copy(np_training_set) # This class becomes the positive class 1 # and all other classes become the negative class 0 for row in range(0, no_of_rows_training_set): if (temp[row, (no_of_columns_training_set - 1)]) == cl: temp[row, (no_of_columns_training_set - 1)] = 1 else: temp[row, (no_of_columns_training_set - 1)] = 0 # Add the new training set to the trainingsets list trainingsets.append(temp) # Calculate and store the weights for the training set # of each class. Execute gradient descent on each training set # in order to calculate the weights weights_for_each_class = [] for cl in range(0, len(list_of_unique_classes)): weights_for_this_class = gradient_descent(trainingsets[cl]) weights_for_each_class.append(weights_for_this_class) # Used for debugging #print(weights_for_each_class[0]) #print() #print(weights_for_each_class[1]) #print() #print(weights_for_each_class[2]) ########################### End of Training Phase ######################### ############################# Testing Phase ############################### no_of_columns_test_set = np_test_set.shape[1] no_of_rows_test_set = np_test_set.shape[0] # Extract the attributes from the test set. # x is still a 2d array x = np_test_set[:,:(no_of_columns_test_set - 1)] no_of_attributes = x.shape[1] # Extract the classes from the test set. # actual_class is a 1d array. actual_class = np_test_set[:,(no_of_columns_test_set - 1)] # Go through each row (instance) of the test data for inst in range(0, no_of_rows_test_set): # Create a scorecard that keeps track of the probabilities of this # instance being a part of each class scorecard = [] # Calculate and store the probability for each class in the scorecard for cl in range(0, len(list_of_unique_classes)): # Calculate weighted sum of the attributes for # this instance output = np.dot(weights_for_each_class[cl], x[inst,:]) # Calculate the sigmoid of the weighted sum # This is the probability that this instance belongs # to the positive class this_probability = sigmoid(output) scorecard.append(this_probability) most_likely_class = scorecard.index(max(scorecard)) # Store the value of the most likely class in the "Predicted Class" # column of the test_set data frame test_set.loc[inst, "Predicted Class"] = most_likely_class # Update the 'Prediction Correct?' column of the test_set data frame # 1 if correct, else 0 if test_set.loc[inst, "Actual Class"] == test_set.loc[ inst, "Predicted Class"]: test_set.loc[inst, "Prediction Correct?"] = 1 else: test_set.loc[inst, "Prediction Correct?"] = 0 # accuracy = (total correct predictions)/(total number of predictions) accuracy = (test_set["Prediction Correct?"].sum())/(len(test_set.index)) # Store the revamped dataframe predictions = test_set # Replace all the class values with the name of the class for cl in range(0, len(list_of_unique_classes)): predictions["Actual Class"].replace( cl, list_of_unique_classes[cl] ,inplace=True) predictions["Predicted Class"].replace( cl, list_of_unique_classes[cl] ,inplace=True) # Replace 1 with Yes and 0 with No in the 'Prediction # Correct?' column predictions['Prediction Correct?'] = predictions[ 'Prediction Correct?'].map({1: "Yes", 0: "No"}) # Reformat the weights_for_each_class list of arrays weights_for_each_class = pd.DataFrame(np.row_stack(weights_for_each_class)) # Rename the row names for cl in range(0, len(list_of_unique_classes)): row_name = str(list_of_unique_classes[cl] + " weights") weights_for_each_class.rename(index={cl:row_name}, inplace=True) # Get a list of the names of the attributes training_set_names = list(training_set.columns.values) training_set_names.pop() # Remove 'Actual Class' # Rename the column names for col in range(0, len(training_set_names)): col_name = str(training_set_names[col]) weights_for_each_class.rename(columns={col:col_name}, inplace=True) # Record the number of test instances no_of_instances_test = len(test_set.index) # Return statement return accuracy, predictions, weights_for_each_class, no_of_instances_test

Here is the code for five-fold stratified cross-validation:

import pandas as pd # Import Pandas library import numpy as np # Import Numpy library # File name: five_fold_stratified_cv.py # Author: Addison Sears-Collins # Date created: 7/17/2019 # Python version: 3.7 # Description: Implementation of five-fold stratified cross-validation # Divide the data set into five random groups. Make sure # that the proportion of each class in each group is roughly equal to its # proportion in the entire data set. # Required Data Set Format for Disrete Class Values # Columns (0 through N) # 0: Instance ID # 1: Attribute 1 # 2: Attribute 2 # 3: Attribute 3 # ... # N: Actual Class def get_five_folds(instances): """ Parameters: instances: A Pandas data frame containing the instances Returns: fold0, fold1, fold2, fold3, fold4 Five folds whose class frequency distributions are each representative of the entire original data set (i.e. Five-Fold Stratified Cross Validation) """ # Shuffle the data set randomly instances = instances.sample(frac=1).reset_index(drop=True) # Record the number of columns in the data set no_of_columns = len(instances.columns) # number of columns # Record the number of rows in the data set no_of_rows = len(instances.index) # number of rows # Create five empty folds (i.e. Panda Dataframes: fold0 through fold4) fold0 = pd.DataFrame(columns=(instances.columns)) fold1 = pd.DataFrame(columns=(instances.columns)) fold2 = pd.DataFrame(columns=(instances.columns)) fold3 = pd.DataFrame(columns=(instances.columns)) fold4 = pd.DataFrame(columns=(instances.columns)) # Record the column of the Actual Class actual_class_column = no_of_columns - 1 # Generate an array containing the unique # Actual Class values unique_class_list_df = instances.iloc[:,actual_class_column] unique_class_list_df = unique_class_list_df.sort_values() unique_class_list_np = unique_class_list_df.unique() #Numpy array unique_class_list_df = unique_class_list_df.drop_duplicates()#Pandas df unique_class_list_np_size = unique_class_list_np.size # For each unique class in the unique Actual Class array for unique_class_list_np_idx in range(0, unique_class_list_np_size): # Initialize the counter to 0 counter = 0 # Go through each row of the data set and find instances that # are part of this unique class. Distribute them among one # of five folds for row in range(0, no_of_rows): # If the value of the unique class is equal to the actual # class in the original data set on this row if unique_class_list_np[unique_class_list_np_idx] == ( instances.iloc[row,actual_class_column]): # Allocate instance to fold0 if counter == 0: # Extract data for the new row new_row = instances.iloc[row,:] # Append that entire instance to fold fold0.loc[len(fold0)] = new_row # Increase the counter by 1 counter += 1 # Allocate instance to fold1 elif counter == 1: # Extract data for the new row new_row = instances.iloc[row,:] # Append that entire instance to fold fold1.loc[len(fold1)] = new_row # Increase the counter by 1 counter += 1 # Allocate instance to fold2 elif counter == 2: # Extract data for the new row new_row = instances.iloc[row,:] # Append that entire instance to fold fold2.loc[len(fold2)] = new_row # Increase the counter by 1 counter += 1 # Allocate instance to fold3 elif counter == 3: # Extract data for the new row new_row = instances.iloc[row,:] # Append that entire instance to fold fold3.loc[len(fold3)] = new_row # Increase the counter by 1 counter += 1 # Allocate instance to fold4 else: # Extract data for the new row new_row = instances.iloc[row,:] # Append that entire instance to fold fold4.loc[len(fold4)] = new_row # Reset counter to 0 counter = 0 return fold0, fold1, fold2, fold3, fold4

# Logistic Regression Output

Here are the trace runs:

Here are the results:

Here are the test statistics for each data set:

# Analysis

## Breast Cancer Data Set

I hypothesize that performance was high on this algorithm because of the large number of instances (699 in total). This data set had the highest number of instances out of all the data sets.

These results also suggest that the amount of training data has a direct impact on performance. Higher amounts of data can lead to better learning and better classification accuracy on new, unseen instances.

## Glass Data Set

I hypothesize that the poor performance on the glass data set is due to the high numbers of classes combined with a relatively smaller data set.

## Iris Data Set

Classification accuracy on the iris data set was satisfactory. This data set was small, and more training data would be needed to see if accuracy could be improved by giving the algorithm more data to learn the underlying relationship between the attributes and the flower types.

## Soybean Data Set (small)

I hypothesize that the large numbers of attributes in the soybean data set (35) helped balance the relatively small number of training instances. These results suggest that large numbers of relevant attributes can help a machine learning algorithm create more accurate classifications.

## Vote Data Set

The results show that classification algorithms like Logistic Regression can have outstanding performance on large data sets that are binary classification problems.

# Summary and Conclusions

- Higher amounts of data can lead to better learning and better classification accuracy on new, unseen instances.
- Large numbers of relevant attributes can help a machine learning algorithm create more accurate classifications.
- Classification algorithms like Logistic Regression can achieve excellent classification accuracy on binary classification problems, but performance on multi-class classification algorithms can yield mixed results.

# References

Alpaydin, E. (2014). *Introduction to Machine
Learning.* Cambridge, Massachusetts: The MIT Press.

Fisher, R. (1988, July 01). *Iris Data Set*.
Retrieved from Machine Learning Repository:
https://archive.ics.uci.edu/ml/datasets/iris

German, B. (1987, September 1). *Glass
Identification Data Set*. Retrieved from UCI Machine Learning Repository:
https://archive.ics.uci.edu/ml/datasets/Glass+Identification

Kelleher, J. D., Namee, B., & Arcy, A. (2015). *Fundamentals
of Machine Learning for Predictive Data Analytics.* Cambridge,
Massachusetts: The MIT Press.

Michalski, R. (1980). Learning by being told and
learning from examples: an experimental comparison of the two methodes of
knowledge acquisition in the context of developing an expert system for
soybean disease diagnosis. *International Journal of Policy Analysis and
Information Systems, 4*(2), 125-161.

Rebala, G., Ravi, A., & Churiwala, S. (2019). *An
Introduction to Machine Learning.* Switzerland: Springer.

Schlimmer, J. (1987, 04 27). *Congressional Voting
Records Data Set*. Retrieved from Machine Learning Repository:
https://archive.ics.uci.edu/ml/datasets/Congressional+Voting+Records

Wolberg, W. (1992, 07 15). *Breast Cancer Wisconsin
(Original) Data Set.* Retrieved from Machine Learning Repository:
https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Original%25

Y. Ng, A., & Jordan, M. (2001). On Discriminative
vs. Generative Classifiers: A Comparison of Logistic Regression and Naive
Bayes. *NIPS’01 Proceedings of the 14th International Conference on Neural
Information Processing Systems: Natural and Synthetic *, 841-848.