Determining model preditive strengths on Clevelent heart-disease dataset using different preprocessing techniques

Jakob Serlier

12/11/2019

Predicting the presence of heart-disease within Cleveland dataset

Hello world, I’m Jakob Serlier. This R Markdown file is a part of my final project for the R For Analytics class at Carnegie Mellon University. Let’s dive into it!

The dataset we will be taking a look at is the heart-disease-uci dataset retrieved from Kaggle.com. The reason I chose this dataset is because I saw that it had potential for supervised modelling, which is something I wanted to learn more about in the course of this project. My goal is to:

  1. Explore different supervised learing models in R
  2. Determine the impact of different preprocessing methods on the predictive power of the created models

The analysis of the data will follow the following methodology:

  1. Performing exploratory data analysis
  2. Creating three dataframes which were created using different pre-processing techniques
  3. Testing and running various models using the full dataframe
  4. Running the models for each of the three dataframes multiple times, each time resampling the train and test subset
  5. Reviewing and discuss the results

The three dataframes will be divided as follows:

  1. The full dataset
  2. The full dataset containing various buckets/bins
  3. A subset of the dataset, created by removing the lowest correlating predictors with the target variable.

The following models will be considered:

  1. randomForest’s package RandomForest:“randomForest implements Breiman’s random forest algorithm (based on Breiman and Cutler’s original Fortran code) for classification and regression.”
  2. The base GLM Model: “glm is used to fit generalized linear models, specified by giving a symbolic description of the linear predictor and a description of the error distribution.”
  3. mboost’s GLMBoost: “Gradient boosting for optimizing arbitrary loss functions where component-wise linear models are utilized as base-learners.”
  4. kernlab’s SVM Radial: “Support Vector Machines with Radial Basis Function Kernel”
  5. caTool’s Boosted Logistic Regression: “Train logitboost classification algorithm using decision stumps (one node decision trees) as weak learners.”

All of these models are able to be used in classification problems.

Some assumptions and ‘shortcuts’ have been made during this project in order to keep it within a feasible scope. For the results of each of the model, I have chosen a single indicator variable, such as ‘accuracy’, in order to determine its effectiveness. This of course leaves out a lot of important data which might improve/decrease the effectiveness in a real world scenario (e.g. a confusion matrix). Keep this in mind as I interpret the results.

Exploratory Data Analysis

Let’s start with some simple text/table based EDA:

## [1] FALSE
## [1] 303  14
## 'data.frame':    303 obs. of  14 variables:
##  $ ï..age  : int  63 37 41 56 57 57 56 44 52 57 ...
##  $ sex     : int  1 1 0 1 0 1 0 1 1 1 ...
##  $ cp      : int  3 2 1 1 0 0 1 1 2 2 ...
##  $ trestbps: int  145 130 130 120 120 140 140 120 172 150 ...
##  $ chol    : int  233 250 204 236 354 192 294 263 199 168 ...
##  $ fbs     : int  1 0 0 0 0 0 0 0 1 0 ...
##  $ restecg : int  0 1 0 1 1 1 0 1 1 1 ...
##  $ thalach : int  150 187 172 178 163 148 153 173 162 174 ...
##  $ exang   : int  0 0 0 0 1 0 0 0 0 0 ...
##  $ oldpeak : num  2.3 3.5 1.4 0.8 0.6 0.4 1.3 0 0.5 1.6 ...
##  $ slope   : int  0 0 2 2 2 1 1 2 2 2 ...
##  $ ca      : int  0 0 0 0 0 0 0 0 0 0 ...
##  $ thal    : int  1 2 2 2 2 1 2 3 3 2 ...
##  $ target  : int  1 1 1 1 1 1 1 1 1 1 ...
##     ï..age        sex         cp   trestbps       chol        fbs 
##  9.0821010  0.4660108  1.0320525 17.5381428 51.8307510  0.3561979 
##    restecg    thalach      exang    oldpeak      slope         ca 
##  0.5258596 22.9051611  0.4697945  1.1610750  0.6162261  1.0226064 
##       thal     target 
##  0.6122765  0.4988348

We can see that there are no NA values, which is great, since now we don’t have to bother with imputing data or removing rows.

Dimensions of the dataframe: 303 rows and 14 cols, meaning 1 target variable and 13 possible predictors.

The predictor variables are defined as follows:

  1. age age in years
  2. sex(1 = male; 0 = female)
  3. cp chest pain type
  4. trestbps resting blood pressure (in mm Hg on admission to the hospital)
  5. chol serum cholestoral in mg/dl
  6. fbs(fasting blood sugar > 120 mg/dl) (1 = true; 0 = false)
  7. restecg resting electrocardiographic results
  8. thalach maximum heart rate achieved
  9. exang exercise induced angina (1 = yes; 0 = no)
  10. oldpeakST depression induced by exercise relative to rest
  11. slope the slope of the peak exercise ST segment
  12. ca number of major vessels (0-3) colored by flourosopy
  13. thal3 = normal; 6 = fixed defect; 7 = reversable defect

As we are mostly interested in the predictive power of the variables, we will not spend much time on exploring the real-life implication of these variables. As for the target variables, this is either 1 (positive) or 0 (negative).

In the structure, we can see that we are dealing with 14 numerical (int & num) variables.

Next, let’s rename the age variable to something which won’t mess with Rstudio

Visualizing the data

Correlation plot

By taking a look at the correlation plot we can get an initial idea of which predictors might be interesting to take a look at, especially the ones correlated with the target variable. The best classification and regression models have a low redundancy in predictor variables.

We can see that the variables chol, fbs restecg and trestbps have the lowest correlation, so these will be removed in the third analyzed dataframe.

Feature engineering

My feature engineering will mostly consist of pre-processing the dataset into 3 seperate datasets.

Transforming the target variable:
We will be transforming the target variable and giving it better labels in order to improve later interpretation. We will also be setting the target variable as a factor.

creating the 3 main datasets
As I have discribed in the introduction, we will be creating 3 seperate dataframes which will be analyzed. The first dataframe is the largely unaltered full dataset. Secondly, we will be creating a bucket dataframe, which wil have 3 variables (age, cholesterol levels and thalach) which will be transformed to ordinal variables (put into bins). Lastly, we will create a dataset which excludes the variables with low correlation, chol, fbs restecg and trestbps.

Defining full model test and training set

Next we want to create a training and a test dataset for the full dataframe, as this is the one we will be using in the initial runs of the models.

Running the models (scroll down for the results)

First run of GLM + some interesting variables, results hidden

Running GLM model on the 3 datasets nruns times

Running the basic random forest model on the full dataset, results hidden

Running the RandomForest model on the 3 datasets nruns times

Testing svmRadial model using full dataset and running it on the 3 datasets nrun times

Running LogitBoost on the full model and running it on the 3 dataset nrun times

Running the glmboost model on the full dataset and running the model on the 3 datasets nrun times

compiling the results into variables and into a table

Results of running the models

##                  Logistic Reg.     Boosted glm       RandomForest     
## testing variable Pred. Accuracy    Accuracy          1-Error          
## full model       0.828947368421053 0.825             0.818553817855157
## bucket model     0.822368421052632 0.826315789473684 0.814886709294547
## excluding model  0.821052631578947 0.836842105263158 0.803963868995558
##                  svmRadial         Boosted Log. reg.
## testing variable Accuracy          Accuracy         
## full model       0.830263157894737 0.784210526315789
## bucket model     0.813157894736842 0.798684210526316
## excluding model  0.807894736842105 0.786842105263158

As we can see in the results table, the differences between the means of the indicator variables don’t seem to vary very much. Before we take a look at which model is the best predictory model, we will if the mean values of the indicator variables differ (statistically) significantly. In order to do so, we will run an anova test for the results of the 5 models, taking into account each of the iterations of the model results. The bucket and excluded dataset results will be run against the mean of the full dataset. An alpha value of 0,05 (5%) will be used to evaluate the null-hypothesis. Finally, as an extra measure, the assumption of homogeneity of variance is tested.

Results of ANOVA testing the means

##             Df   Sum Sq   Mean Sq F value Pr(>F)
## bucket       1 0.000224 0.0002244   0.260  0.626
## excl         1 0.001350 0.0013504   1.564  0.251
## Residuals    7 0.006043 0.0008633
##             Df    Sum Sq   Mean Sq F value Pr(>F)
## bucket       1 0.0000024 2.370e-06   0.031  0.864
## excl         1 0.0002321 2.321e-04   3.073  0.123
## Residuals    7 0.0005285 7.550e-05
##             Df   Sum Sq  Mean Sq F value Pr(>F)
## bucket       1 0.000183 0.000183   0.161   0.70
## excl         1 0.001362 0.001362   1.198   0.31
## Residuals    7 0.007960 0.001137
##             Df   Sum Sq  Mean Sq F value Pr(>F)
## bucket       1 0.001172 0.001172   0.478  0.512
## excl         1 0.001813 0.001813   0.739  0.418
## Residuals    7 0.017168 0.002452
##             Df    Sum Sq   Mean Sq F value Pr(>F)
## bucket       1 0.0007531 0.0007531   2.029  0.197
## excl         1 0.0004750 0.0004750   1.280  0.295
## Residuals    7 0.0025980 0.0003711

These results will be discussed in the analysis & discussion section.

Taking a look at the three CARET models

These models are the most promising models, by looking at the plots we will determine the best model.

Analysis & discussion

Let’s start with the three created dataframes. The assumption of homogeneity test was passed. As we can see in the ANOVA analyses, there is not a single significant difference in the predictive power of the models created. This was determined by looking if there is a difference in the mean of the variable displaying the predictive strengths of the model. I find this to be quite surprising, as I had expected there to be significant differences. However, we have to keep in mind that I have only analyzed a single indicator variable, and other properties of the models were kept outside of the scope. In accordance to the results, we can conclude that the removal of the low-correlating variables, and the catigorization of the variables, did not have a significant impact on the predictive power of the resulting models. This concludes the main goal of the analysis.

Next, let’s consider the tested models. We have taken a look at 5 models, and have run them multiple times in order to prevent sampling bias. According to the results plot, the GLMBoost model and svmRadial model both have comparitively high predictive strengths. Out of all models, boosted logistic regression seems to have the least predictive power. My goal in this regard was to become more familiar with these models, which I have. RandomForest is the model which I am most comfortable with, as it is a model which can be used very quickly and easily. Lastly I was surprised by the detail, ease of use and documentation of the base R glm model, and it will be my go-to regression model in the future. The caret library has too many models to count (238 if you do try), and I’m excited to keep learning more about these models in future projects, especially since it has many helper functions which can be used for improving the feature engineering.

After analysing and running the models, I realized that I could’ve tested various models more easily using the trainControl method from the caret package. This is well documented in this article, which I found only after finishing my own analysis. This shows that I still have much to learn while using R, and I look forward to keep getting better at it!

Thank you for taking the time to read through the analysis. I look forward to using my newly learned R skills in future projects!

Jakob Serlier

#Sources for code bits and pieces:
#Age bins source: https://stackoverflow.com/questions/12979456/r-code-to-categorize-age-into-group-bins-breaks
#Valid & train set source: https://stackoverflow.com/questions/17200114/how-to-split-data-into-training-testing-sets-using-sample-function
#Randomforest source: https://cran.r-project.org/web/packages/randomForest/randomForest.pdf
#caret inspiration: https://machinelearningmastery.com/compare-models-and-select-the-best-using-the-caret-r-package/
Logistic regression: #https://stats.idre.ucla.edu/r/dae/logit-regression/
#http://www.utstat.toronto.edu/~brunner/oldclass/appliedf11/handouts/2101f11StepwiseLogisticR.pdf
#https://www.r-bloggers.com/how-to-perform-a-logistic-regression-in-r/

For more of my projects visit https://jakobs.dev.