COMP222 – 2019 – Second CA Assignment
Individual coursework Train Deep Learning Agents
This assignment requires you to implement deep neural networks for the two datasets, i.e.,
- Optical recognition of handwritten digits dataset
- RCV1 dataset
from https://scikit-learn.org/stable/datasets/index.html, and apply the model eval-
uation methods to compare them with the two models in Assignment 1. Please make sure
that you select the same dataset as you did for the Assignment 1, if you completed the
2 DNN-based Classiﬁcation
2.1 Requirement and Description
Language and Platform Python (version 3.5 or above) and Tensorﬂow or Keras (latest
version). You can use some libraries available on Python platform, including numpy, scipy,
scikit-learn, and matplotlib. If you intend to use libraries other than these, please consult
the demonstrator or the lecturer.
Learning Task You can choose either classiﬁcation (preferred) or regression, but needs to be the same choice as your Assignment 1 submission. Classification.
Assignment Tasks You need to implement the following functionalities:
f1 design and build two diﬀerent deep neural networks, one with convolutional layer and
the other without convolutional layer;
f2 apply model evaluation on the learned models. For the materials on model evaluation,
you may take a look at the metrics explained in the lecture “model evaluation”. You
are required to implement by yourself (i.e., do not call built-in libraries)
(a) the cross-validation of 5 subsamples,
(b) the confusion matrix, and
(c) the ROC curve for one class vs. all other classes
(a) the two neural networks you trained in f1, and
(b) the two traditional machine learning algorithms in the ﬁrst assignment.
Please also summarise your observation on the results.
Additional Requirements We have additional requirements that,
- the marker can run your code directly, i.e., see the results of functionality f1 by loading
the saved models, without training.
- You need to provide clear instructions on how to train the two models. The instructions
may be e.g., a diﬀerent command or an easy way of adapting the source code.
Documentation You need to write a proper document
- detailing how to run your program, including the software dependencies,
- explaining how the functionalities and additional requirements are implemented, and
- providing the details of your implementation, including e.g., the meaning of parameters
and variables, the description of your model evaluation, etc.
Submission ﬁles Your submission should include the following ﬁles:
- a ﬁle for source code,
- two ﬁles for saved models, and
- a document.
Please see Section 3 for instructions on how to package your submission ﬁles, and read the
Q&A on whether to upload the two trained models from the ﬁrst assignment.
2.2 Marking Criteria
The assignment is split in a number of steps. Every step gives you some marks.
Note 1 At the beginning of the document, please include a check list indicating whether
the below marking points have been implemented successfully. Unless exceptional cases, the
length of the submitted document needs to be within 4 pages (A4 paper, 11pt font size).
Note 2 The marking of a functionality will also consider the quality of coding and the quality of documentation. A run-able implementation alone will have up to 50% of the marks.
functionality f1: 50%
For each model (with and without convolutional layer), 20% will be for the model construction and 5% will be on the model saving and the model ﬁle in the submission.
functionality f2: 50%
The model evaluation between will include
- cross validation (10%)
- confusion matrix (10%)
- ROC curve (20%)
- discussion on the discovery (10%)
For each of the four parts, 80% of the marks are for deep learning models, while 20% are for
the traditional models in the ﬁrst assignment. For example, for cross validation part, if you
only do deep learning models, your marks are capped at 8% instead of 10%.
The marker will mark according to the quality of both your evaluation and the docu-
Q: The ROC curve we taught in the lecture is for binary classiﬁcation, but
the models we trained are for multiple classes. What can we do?
A: As indicated, you can have one class vs. all other classes, where all other classes are deemed as a single class.
Q: My models in the ﬁrst assignment can output a classiﬁcation but not a
conﬁdence probability. What can we do for ROC curve?
A: If you think some functionality is hard to implement, please explain in the docu- ment. The marker will then evaluate your explanation to give you a reasonable mark.
Q: Since we are requested to evaluate the two models from our ﬁrst assign-
ment, shall we upload again?
A: You can upload them again if needed. Note that, the marker won’t be able to access the ﬁrst assignment when they are marking the second assignment.
Q: My runtime for the functionality f2 are longer than 5 minutes. Will this
aﬀect my marks?
A: Marking is based on the quality of your implementation and your documenta-
tion, and will not take the runtime into consideration. On the other hand, you are
recommended to explain the details of your program (including the runtime) in your
本网站支持 Alipay WeChatPay PayPal等支付方式
E-mail: email@example.com 微信号:vipnxx