Note: This unit version is currently being edited and is subject to change!

COMP5617: Empirical Security Analysis and Engineering (2020 - Semester 2)

Download UoS Outline

Unit: COMP5617: Empirical Security Analysis and Engineering (6 CP)
Mode: Normal-Evening
On Offer: Yes
Level: Postgraduate
Faculty/School: School of Computer Science
Unit Coordinator/s: Dr Holz, Ralph
Session options: Semester 2
Versions for this Unit:
Site(s) for this Unit:
Campus: Camperdown/Darlington
Pre-Requisites: INFO3616 OR ELEC5616.
Brief Handbook Description: This unit will present the lessons from recent research and from case studies of practice to bring students the skills to assess and improve the security of deployed systems. A particular focus is on data-driven approaches to collect operational data about a system's security. We explore deployment issues at local and global scale, e.g. for X.509, DNS, and BGP, and also take human factors explicitly into account. As a result, students will learn to put building blocks of security together in a sound way, to arrive at engineering solutions that are empirically verifiable, functional, and secure against realistic threats. As Dan Geer once famously said: "Any security technology whose effectiveness can't be empirically determined is indistinguishable from blind luck."
Assumed Knowledge: None.
Additional Notes: Students are expected to be competent programmers in a language such as Go, Python, or C (this is particularly required in Assignment 1). They will be able to choose their programming language for the second assignment; however, if they choose a language that the teaching staff is not familiar with, they will have to acknowledge that there is reduced support available. The teaching staff is very familiar with Go and Python; they are OK programmers in Java, C, C++, and C#.
Lecturer/s: Dr Holz, Ralph
Tutor/s: tba.
Timetable: COMP5617 Timetable
Time Commitment:
# Activity Name Hours per Week Sessions per Week Weeks per Semester
1 Lecture 2.00 1 13
2 Tutorial 2.00 1 12
3 Independent Study 5.00 1 12

Learning outcomes are the key abilities and knowledge that will be assessed in this unit. They are listed according to the course goal supported by each. See Assessment Tab for details how each outcome is assessed.

Unassigned Outcomes
1. Knowledge of privacy-preserving technologies
2. Awareness of security vs. usability trade-offs
3. Ability to research and analyse information about current IT security topics.
4. Understanding of data-driven security defences
5. Practical experience with scanning and monitoring of Internet services to determine deployment security
6. Understanding of the building blocks of Internet services such as the Internet naming and routing system, and the WWW
7. Understanding of the main security protocols in the Internet stack
8. Ability to design and conduct an empirical security analysis
Assessment Methods:
# Name Group Weight Due Week Outcomes
1 Intro: empirically studying security No 10.00 Week 3 4, 6, 7, 8,
2 Project work: studying security deployments Yes 40.00 Multiple Weeks 3, 5, 8,
3 Final Examination No 50.00 Exam Period 1, 2, 4, 6, 7,
Assessment Description: We give out an assignment in Week 1, a 10% practical coding exercise, which doubles as a self-check for students to test if they meet the assumed skill requirements before the census date.

Enrolment numbers allowing, the second assignment is an all-semester project work. Students engage in guided research to identify a topic that they would like to investigate more deeply. They write a study proposal, outlining the study`s goals and anticipated difficulties and coding requirements, defining a roadmap. This document is going to be signed off by the lecturer-in-charge. Students write a measurement tool or design a toolchain. They are allocated accounts on one of our (very powerful) scanning machines to run their scans on the real Internet. Students can choose the programming language for the tool they write; data may be analysed in Jupyter Notebook (Python/Pandas) or R Studio. They present their results at the end of the semester. Note: empirical analysis is subject to ethical considerations. Students choosing to do live scans must sign a document that they understand fundamental ethics of Internet and security measurement. This assignments contributes a total of 40% to the overall grade; it is itself broken down into the following graded components: study proposal (20%), tool/toolchain (30%), carrying out the experiment with scientific rigour (20%), analysis (20%), and presentation (10%).

The security analysis and engineering concepts will be assessed in a 2 hour written final exam in the examination period. Students must get 40% in the final exam to pass the unit, regardless of the sum of individual marks.

General statements:

The late penalty for all practical exercises is 20% of the awarded mark per day late; maximum 5 days late (after that: 0 credits).

There may be statistically defensible moderation when combining the marks from each component to ensure consistency of marking between markers, and alignment of final grades with unit outcomes.
Assessment Feedback: Tutorial hours will be used for feedback on the ongoing project and helping with difficulties. Tutorials will also give students an opportunity to practice their skills for each topic we discuss.
Grade Type Description
Standards Based Assessment Final grades in this unit are awarded at levels of HD for High Distinction, DI (previously D) for Distinction, CR for Credit, PS (previously P) for Pass and FA (previously F) for Fail as defined by University of Sydney Assessment Policy. Details of the Assessment Policy are available on the Policies website at . Standards for grades in individual assessment tasks and the summative method for obtaining a final mark in the unit will be set out in a marking guide supplied by the unit coordinator.
Minimum Pass Requirement It is a policy of the School of Computer Science that in order to pass this unit, a student must achieve at least 40% in the written examination. For subjects without a final exam, the 40% minimum requirement applies to the corresponding major assessment component specified by the lecturer. A student must also achieve an overall final mark of 50 or more. Any student not meeting these requirements may be given a maximum final mark of no more than 45 regardless of their average.
Policies & Procedures: IMPORTANT: School policy relating to Academic Dishonesty and Plagiarism.

In assessing a piece of submitted work, the School of Computer Science may reproduce it entirely, may provide a copy to another member of faculty, and/or to an external plagiarism checking service or in-house computer program and may also maintain a copy of the assignment for future checking purposes and/or allow an external service to do so.

Other policies

See the policies page of the faculty website at for information regarding university policies and local provisions and procedures within the Faculty of Engineering and Information Technologies.
Prescribed Text/s: Note: Students are expected to have a personal copy of all books listed.
Recommended Reference/s: Note: References are provided for guidance purposes only. Students are advised to consult these books in the university library. Purchase is not required.
Online Course Content: Canvas site will be available.

Note that the "Weeks" referred to in this Schedule are those of the official university semester calendar

Week Description
Week 1 Lecture/Tutorial: Introduction

• Unit organisation

• Intro: security engineering

• Recap: Internet protocol stack
Week 2 Lecture/Tutorial: Cryptography

• Recap: symmetric and public-key cryptography

• The importance of randomness

• Measurement of cipher deployment and weak keys
Week 3 Lecture/Tutorial: Measurement principles and tools

• Active and passive methods for measurement

• Common security measurement tools

• Fundamental statistics
Assessment Due: Intro: empirically studying security
Week 4 Lecture/Tutorial: Measuring TLS and X.509

• Deployment of TLS and X.509 as the security backbone of the Internet

• Successful subversion, lessons learnt

• Mistakes in deployment, lessons learnt
Week 5 Lecture/Tutorial: Measuring X.509 improvements

• The notary principle

• Append-only auditable logs

• Cross-validation and monitoring to achieve better security
Week 6 Lecture/Tutorial: Measuring the DNS and associated security standards

• Naming systems

• DNSSEC extensions

• Deployment issues
Week 7 Lecture/Tutorial: Measuring Internet routing security

• The insecurity of global routing: threat model and effects

• BGPSec and RPKI

• Measurement-based defences: BGP threat detection
Week 8 Lecture/Tutorial: Mobile and IoT security

• Attack surface in deployment

• Malware

• IoT technology
Week 9 Other: Public holiday
Week 10 Lecture/Tutorial: Machine learning for security

• Principles

• Examples

• Limitations
Week 11 Tutorial: Double tutorial hours: project work
Week 12 Lecture/Tutorial: Network privacy

• Network interference and forms of censorship

• Measurement of censorship and interference
Week 13 Lecture: Presentations and Q&A
Exam Period Assessment Due: Final Examination

Course Relations

The following is a list of courses which have added this Unit to their structure.

Course Year(s) Offered
Bachelor of Advanced Computing (Computational Data Science) 2019, 2020, 2021, 2022
Bachelor of Advanced Computing (Cybersecurity) 2022
Bachelor of Advanced Computing (Software Development) 2019, 2020, 2021, 2022
Graduate Certificate in Information Technology 2017, 2018, 2019, 2020, 2021, 2022
Graduate Certificate in Information Technology Management 2017, 2018, 2019, 2020, 2021, 2022
Graduate Diploma in Computing 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022
Graduate Diploma in Information Technology 2017, 2018, 2019, 2020, 2021, 2022
Graduate Diploma in Information Technology Management 2017, 2018, 2019, 2020, 2021, 2022
Graduate Certificate in Computing 2020, 2021, 2022
Master of Information Technology 2017, 2018, 2019, 2020, 2021, 2022
Master of Information Technology Management 2017, 2018, 2019, 2020, 2021, 2022
Master of IT/Master of IT Management 2017, 2018, 2019, 2020, 2021, 2022

Course Goals

This unit contributes to the achievement of the following course goals:

Attribute Practiced Assessed
(6) Communication and Inquiry/ Research (Level 4) No 0%
(8) Professional Effectiveness and Ethical Conduct (Level 3) No 0%
(5) Interdisciplinary, Inclusiveness, Influence (Level 3) No 0%
(4) Design (Level 3) No 0%
(2) Engineering/ IT Specialisation (Level 4) No 0%
(3) Problem Solving and Inventiveness (Level 3) No 0%
(1) Maths/ Science Methods and Tools (Level 4) No 0%

These goals are selected from Engineering & IT Graduate Outcomes Table 2018 which defines overall goals for courses where this unit is primarily offered. See Engineering & IT Graduate Outcomes Table 2018 for details of the attributes and levels to be developed in the course as a whole. Percentage figures alongside each course goal provide a rough indication of their relative weighting in assessment for this unit. Note that not all goals are necessarily part of assessment. Some may be more about practice activity. See Learning outcomes for details of what is assessed in relation to each goal and Assessment for details of how the outcome is assessed. See Attributes for details of practice provided for each goal.