Homework 4: Instructions
Before 10/28/2015, when a draft project paper is due, you will have an opportunity to fit a model to your data and see how well it performs.
Given the model you have chosen to study your data, split the data into training and test sets:
- is the precise model, or a close version of the model, available to use?
- if not, what are your plans to fit your model to data?
- does this application scale to the size of your data?
- if not, what alternatives are there (e.g., going from MCMC to variational approaches)?
- does this application scale to the number of features in your data?
- if not, how can you reduce the number of features?
First pass analysis of the fitted model results:
- what is the training set error?
- what is the generalization error?
- how robust is this approach to starting point and local minima? (fit multiple times and convince us that results are fairly close)
- how robust is this approach to parameterization?
It may not work well the first time we do an analysis. Where should you go with this?
- Changes to model or methods?
- Adding additional replication data?
- adding additional hierarchical structure into model?