EXAM August 26 - September 1, 2024

Read through the entire exam before you start and think carefully about code-sharing between tasks to save time on execution.

It is a good strategy to attempt to answer all tasks since I will give partial marks. In addition, several of the tasks are interlinked.

Putting random results without motivation is not a good strategy. What I look for in your exams is a clear insight and motivation as to what you do to adress each task. If you do "random" things, or, even worse, "throw everything at the problem" and don't explain why you think this answers the questions, then this results in no marks. So please make sure you clearly state why you use a method/strategy/pipeline.

You should work alone for this exam. The reports will be checked with Urkund. You can discuss coding issues or bugs with other students sparingly, but don't discuss how to intepret figures or tables.

"Can I use ChatGPT?" It's been my experience that, while ChatGPT can be helpful, it is also overconfident in providing answers that are in fact wrong. It takes time to check the output from ChatGPT and, depending on your own skillset, this may result in an overall waste of time rather than a saving time, especially if you want to connect multiple tasks. Your main task is to analyze and interpret the results.

So I would say that you should use ChatGPT with caution and only for simple coding questions like generating plots, proposed functions calls (which may contain bugs), skeleton for coding pipelines, etc. Don't use ChatGPT for generating text.

GOOD LUCK!

• The report should not be longer than 1 page of text + max 1/2 page conclusion/main summary per sub-task (sub-tasks = a,b,c etc).
• You do not have to discuss the methods and assumptions unless this pertains to a result you present - i.e. you don't need to write a "lab report" with introduction, methods, analysis and conclusions. Here, it's only analysis and conclusions that I am interested in. However, you may need to refer to assumptions to explain why your results look like they do.
• Figures and Tables: maximum of 4 figures and 2 tables per sub-task (figures can contain panels but don't go overboard - I need to be able to read axes etc so tiny figures will not be appreciated).
• Make sure you provide captions for the tables and figures - these should clearly state what the figure/table contains as well as a 1-2 conclusion statement about what can be learnt from the figure/table. Refer to the figures and tables in the text!
• MOTIVATE YOUR ANSWERS!
• Don't contradict your results! If something is unexpected say so and explain why you thought you would see something different.
• Be careful about using buzzwords, guessing what I want you to write etc - just go directly from your results.
• Please think about your presentation. Spell check, grammar check. "The arm-length check": meaning, skim through the report as it appears on the screen - does it look nice and neat or is it messy with mis-alignments, cut-off sentences, plots that are tiny etc?
• No cut and paste code mixed with the text.
• No, jupyter notebooks are not accepted as a report. Preferably use latex (overleaf) for neat reports.
• Yes, you can use bulletpoints to present your findings. This could be especially useful for the subtask conclusion.
• Stress your main findings in the 1/2 page subtask conclusion.
• Make sure you put the figures with each subtask after the subtask discussion and before the next subtask so I don't have to scroll back and forth between tasks when I grade.

IMPORTANT

Wherever the exam task contains a question ("Does X differ from Y?", "Is A performing better than B") it is not sufficient to answer "Yes" or "No" - you need to explain how you arrived to this conclusion, which results provide you with these answers, as well as a statement trying explain why the results are what they are. Likewise, when a task is phrased as "Can you do X to get Y?" I, of course, mean for you to attempt to answer this question by doing something, not just hypothesizing about it.

To answer the questions it is for the most part inadequate to perform the task once on the full data set. Make use of re-sampling techniques to support your findings.

Data set

You will use a data set with 1625 observations and 105 features. The data set is contained in a tab-del file called "MyData.txt" posted on canvas. The class labels can be found in the file "MyClass.txt".

For all subtasks, make sure you motivate your answers and provide a take-home message summarizing your main conclusions.

Question 1 (100p)

1a (25p)

Train multiple classifiers (at least 6, of varying degrees of complexity and character (parametric, nonparametric, linear, nonlinear,...) on multiple training-test splits of the data. Compare classification performance.

Pay particular attention to the following;

• imbalance between classes, how you handle this
• the relatively high dimensionality of the data set
• classification accuracy, specificity and sensitivity overall, and at the class level.

Be careful about using the full data set for any training/validation steps. For example, exploration and dimension reduction, if you choose to pursue this approach, should be done of the training data only.

Make sure to discuss:

Overall performance and how you assess this

Are all or only some classes well separated

*To which degree the dimensionality and/or possible collinearity impacts performance

Class imbalance, modeling assumptions, training metrics

The full training pipeline you set up and motivate all your choices

1b (25p)

Which set of features are optimal classification performance? Does this differ for different classifiers in 1a? Does this differ between the classes?

Use at least 3 different methods to answer the above questions. Discuss the stability of selection and the confidence you can attach to your estimate of the number of features needed or features selected, and how you assessed this.

Make sure to discuss:

Method comparisons

Interpretation of the results - is feature selection an easy or difficult task for this data set? why/why not?

Be clear in how you attach confidence to your estimates

1c (25p)

Use linear and nonlinear dimension reduction techniques (e.g. SVD, sparseSVD, kernelPCA, NMF, AE, tSNE), filtering (testing) and feature selection methods and compare classification performance.

Make sure to discuss:

How the results differ between methods

How the dimension reduction techniques improved (or not) classification performance

1d(25p)

Investigate classification and feature selection performance as a function of label contamination on this data set. Make sure to use a wide range of classification methods (at least 6) and discuss how methods differ in terms of being more or less robust to mislabeling.

Consider if observations can be classified with confidence as belonging to one class or if set prediction is more appropriate for some observations. Is low confidence associated with the mislabeled observations?

Make sure to discuss:

Method comparisons and possible reak-down points - how much mislabeling can the methods handle?

Impact on classification performance at the class level

How you quantify how many observations can be classified with confidence, and which sets of predictions are allocated to which observations

If you could identify mislabeled observations in the original data