ISYE 6740 Homework 3 Total 100 points. 1. Basic optimization. (30 points.) Consider a simpli ed logistic regression problem. Given m training samples (xi; yi), i = 1; : : : ;m. The data xi 2 R (note that we only have one feature for each sample), and yi 2 f0; 1g. To t a logistic regression model for classi cation, we solve the following optimization problem, where 2 R is a parameter we aim to nd: max `(); (1) where the log-likelhood function `() = Xm i=1 f???? log(1 + expf????xig) + (yi ???? 1)xig : (a) (10 points) Show step-by-step mathematical derivation for the gradient of the cost function `() in (1) and write a pseudo-code for performing gradient descent to nd the optimizer . This is essentially what the training procedure does. (pseudo-code means you will write down the steps of the algorithm, not necessarily any speci c programming language.) (b) (10 points) Present a stochastic gradient descent algorithm to solve the training of logistic regression problem (1). (c) (10 points) We will show that the training problem in basic logistic regression problem is concave. Derive the Hessian matrix of `() and based on this, show the training problem (1) is concave (note that in this case, since we only have one feature, the Hessian matrix is just a scalar). Explain why the problem can be solved eciently and gradient descent will achieve a unique global optimizer, as we discussed in class. 2. Comparing Bayes, logistic, and KNN classi ers. (30 points) In lectures we learn three dierent classi ers. This question is to implement and compare them. We are suggest use Scikit-learn, which is a commonly-used and powerful Python library with various machine learning tools. But you can also use other similar library in other languages of your choice to perform the tasks. Part One (Divorce classi cation/prediction). (20 points) This dataset is about participants who completed the personal information form and a divorce predic- tors scale. The data is a modi ed version of the publicly available at https://archive.ics.uci.edu/ml/datasets/ Divorce+Predictors+data+set (by injecting noise so you will not replicate the results on uci web- site). There are 170 participants and 54 attributes (or predictor variables) that are all real-valued. The dataset marriage.csv. The last column of the CSV le is label y (1 means divorce”, 0 means no divorce”). Each column is for one feature (predictor variable), and each row is a sample (participant). A detailed explanation for each feature (predictor variable) can be found at the website link above. Our goal is to build a classi er using training data, such that given a test sample, we can classify (or essentially predict) whether its label is 0 (no divorce”) or 1 (divorce”). 1 Build three classi ers using (Naive Bayes, Logistic Regression, KNN). Use the rst 80% data for training and the remaining 20% for testing. If you use scikit-learn you can use train test split to split the dataset. Remark: Please note that, here, for Naive Bayes, this means that we have to estimate the variance for each individual feature from training data. When estimating the variance, if the variance is zero to close to zero (meaning that there is very little variability in the feature), you can set the variance to be a small number, e.g., = 10????3. We do not want to have include zero or nearly variance in Naive Bayes. This tip holds for both Part One and Part Two of this question. (a) (10 points) Report testing accuracy for each of the three classi ers. Comment on their perfor- mance: which performs the best and make a guess why they perform the best in this setting. (b) (10 points) Use the rst two features to train three new classi ers. Plot the data points and decision boundary of each classi er. Comment on the dierence between the decision boundary for the three classi ers. Please clearly represent the data points with dierent labels using dierent colors. Part Two (Handwritten digits classi cation). (10 p…
Looking for solution of this Assignment?
WHY CHOOSE US?
We deliver quality original papers |
Our experts write quality original papers using academic databases.We dont use AI in our work. We refund your money if AI is detected |
Free revisions |
We offer our clients multiple free revisions just to ensure you get what you want. |
Discounted prices |
All our prices are discounted which makes it affordable to you. Use code FIRST15 to get your discount |
100% originality |
We deliver papers that are written from scratch to deliver 100% originality. Our papers are free from plagiarism and NO similarity.We have ZERO TOLERANCE TO USE OF AI |
On-time delivery |
We will deliver your paper on time even on short notice or short deadline, overnight essay or even an urgent essay |