Note: This unit version has not been officially published yet and is subject to change!

COMP5617: Empirical Security Analysis and Engineering (2019 - Semester 2)

Download UoS Outline

Unit: COMP5617: Empirical Security Analysis and Engineering (6 CP)
Mode: Normal-Day
On Offer: Yes
Level: Postgraduate
Faculty/School: School of Computer Science
Unit Coordinator/s: Dr Holz, Ralph
Session options: Semester 2
Versions for this Unit:
Site(s) for this Unit:
Campus: Camperdown/Darlington
Pre-Requisites: ELEC5616 OR INFO3616 OR INFO2315.
Brief Handbook Description: This unit will present the lessons from recent research and from case studies of practice to bring students the skills to assess and improve the security of deployed systems. A particular focus is on data-driven approaches to collect operational data about a system's security. We explore deployment issues at local and global scale, e.g. for X.509, DNS, and BGP, and also take human factors explicitly into account. As a result, students will learn to put building blocks of security together in a sound way, to arrive at engineering solutions that are empirically verifiable, functional, and secure against realistic threats. As Dan Geer once famously said: "Any security technology whose effectiveness can't be empirically determined is indistinguishable from blind luck."

This unit runs a world-first lab exercise: students will work with colleagues at the Technical University of Munich on a joint security measurement exercise (globally distributed measurement) and produce a report on this activity.
Assumed Knowledge: Students are expected to have: - Good programming skills in Go, Python, or C. - UNIX/Linux command-line and tools - Technical orientation and foundational networking knowledge - Sufficient mathematical skills to understand cryptography - Experience working with version control
Additional Notes: We are going to use the Go programming language, which is why we require a background in either Python or C for students who do not know Go yet. Students are encouraged to familiarise themselves with Go:

This unit runs a globally distributed lab exercise with students at Technical University of Munich. We expect students in this unit to commit their time to cross-continental coordination and team work. Willingness to learn a new programming language without assistance is expected.
Lecturer/s: Dr Holz, Ralph
Tutor/s: tba.
Timetable: COMP5617 Timetable
Time Commitment:
# Activity Name Hours per Week Sessions per Week Weeks per Semester
1 Lecture 2.00 1 13
2 Tutorial 2.00 1 12
3 Independent Study 5.00 12
4 Project Work - own time 4.00 8

Learning outcomes are the key abilities and knowledge that will be assessed in this unit. They are listed according to the course goal supported by each. See Assessment Tab for details how each outcome is assessed.

Unassigned Outcomes
1. Knowledge of privacy-preserving technologies
2. Awareness of security vs. usability trade-offs
3. Ability to research and analyse information about current IT security topics.
4. Understanding of data-driven security defences
5. Practical experience with scanning and monitoring of Internet services to determine deployment security
6. Understanding of the building blocks of Internet services such as the Internet naming and routing system, and the WWW
7. Understanding of the main security protocols in the Internet stack
8. Ability to design and conduct an empirical security analysis
Assessment Methods:
# Name Group Weight Due Week Outcomes
1 Modelling Empirical Security Measurements No 10.00 Week 4 4, 6, 7, 8,
2 Security Measurement / Scan Task No 20.00 Week 9 5, 8,
3 Case Study / Paper No 10.00 Week 12 4, 6, 7, 8,
4 Final Examination No 60.00 Exam Period 1, 2, 4, 6, 7,
Assessment Description: The late penalty for all practical exercises is 20% of the awarded mark per day late; maximum 5 days late (after that: 0 credits).

The security analysis and engineering concepts will be assessed in a 2 hour written final exam in the examination period.

Students must get 40% in the final exam to pass the unit, regardless of the sum of individual marks.

There may be statistically defensible moderation when combining the marks from each component to ensure consistency of marking between markers, and alignment of final grades with unit outcomes.
Assessment Feedback: Feedback on the progress of the projects will be given throughout the semester in the tutorial after the lecture.
Grade Type Description
Standards Based Assessment Final grades in this unit are awarded at levels of HD for High Distinction, DI (previously D) for Distinction, CR for Credit, PS (previously P) for Pass and FA (previously F) for Fail as defined by University of Sydney Assessment Policy. Details of the Assessment Policy are available on the Policies website at . Standards for grades in individual assessment tasks and the summative method for obtaining a final mark in the unit will be set out in a marking guide supplied by the unit coordinator.
Minimum Pass Requirement It is a policy of the School of Computer Science that in order to pass this unit, a student must achieve at least 40% in the written examination. For subjects without a final exam, the 40% minimum requirement applies to the corresponding major assessment component specified by the lecturer. A student must also achieve an overall final mark of 50 or more. Any student not meeting these requirements may be given a maximum final mark of no more than 45 regardless of their average.
Policies & Procedures: IMPORTANT: School policy relating to Academic Dishonesty and Plagiarism.

In assessing a piece of submitted work, the School of Computer Science may reproduce it entirely, may provide a copy to another member of faculty, and/or to an external plagiarism checking service or in-house computer program and may also maintain a copy of the assignment for future checking purposes and/or allow an external service to do so.

Other policies

See the policies page of the faculty website at for information regarding university policies and local provisions and procedures within the Faculty of Engineering and Information Technologies.
Prescribed Text/s: Note: Students are expected to have a personal copy of all books listed.
Recommended Reference/s: Note: References are provided for guidance purposes only. Students are advised to consult these books in the university library. Purchase is not required.
Online Course Content: Canvas site will be available.

Note that the "Weeks" referred to in this Schedule are those of the official university semester calendar

Week Description
Week 1 Lecture: Introduction

• Unit organisation

• Intro: security engineering

• Threat modelling
Week 2 Lecture: Building block: Crypto

• Functional overview: symmetric and public-key cryptography

• Case study: the proliferation of weak keys in the wild

• Measurement of cipher deployment

Short assignment starts: modelling/measurement task
Week 3 Lecture: Building block: protocols

• Functional overview: security protocols in the Internet stack

• Introduction: scanning and monitoring to determine deployment security

• Real-world trust anchors: hierarchical PKIs, flat PKIs
Week 4 What goes wrong in deployment: Internet protocol use

• Deployment of TLS and X.509 as the security backbone of the Internet

• Successful subversion, lessons learnt

• Mistakes in deployment, lessons learnt
Assessment Due: Modelling Empirical Security Measurements
Week 5 Lecture: Data-driven modern defences

• The notary principle

• Append-only auditable logs

• Cross-validation and monitoring to achieve better security

Long assignment starts: security measurement/scan
Week 6 Lecture: Naming and security

• Naming systems

• DNSSEC extensions

• Deployment issues
Week 7 Internet routing security

• The insecurity of global routing: threat model and effects

• BGPSec and RPKI

• Measurement-based defences: BGP threat detection
Week 8 Lecture: The WWW: the Achilles heel of deployment

• Attack surface in deployment

• Engineering a secure Web application

• Case study: Websockets, WebRTC
Week 9 Lecture: Usable security

• Security vs. usability trade-offs

• User psychology

• Case study: passwords (CRAM/SCRAM, best practices)

Short assignment starts: case study/paper
Assessment Due: Security Measurement / Scan Task
Week 10 Guest lecture: Real-world security: the world of finances (TBC)

This is a placeholder as the lecture in wk 10 falls on public holiday - currently looking for a solution for this.
Week 11 Lecture: Special topic: Privacy-preserving technologies

• Engineering for anonymity/pseudonymity

• Anonymity and censorship evasion

• Deniable communication: OTR
Week 12 Lecture: Alternative 1: Guest lecture on Real-World security: the world of finances

Alternative 2:

Special topic: Censorship and network interference

• Network interference and forms of censorship

• Measurement of censorship and interference
Assessment Due: Case Study / Paper
Week 13 Lecture: Review of unit

• Discussion of final exam

• Q&A

• Feedback
Exam Period Assessment Due: Final Examination

Course Relations

The following is a list of courses which have added this Unit to their structure.

Course Year(s) Offered
Bachelor of Advanced Computing (Computational Data Science) 2019
Bachelor of Advanced Computing (Software Development) 2019
Graduate Certificate in Information Technology 2017, 2018, 2019
Graduate Certificate in Information Technology Management 2017, 2018, 2019
Graduate Diploma in Computing 2015, 2016, 2017, 2018, 2019
Graduate Diploma in Information Technology 2017, 2018, 2019
Graduate Diploma in Information Technology Management 2017, 2018, 2019
Master of Information Technology 2017, 2018, 2019
Master of Information Technology Management 2017, 2018, 2019
Master of IT/Master of IT Management 2017, 2018, 2019

Course Goals

This unit contributes to the achievement of the following course goals:

Attribute Practiced Assessed
(6) Communication and Inquiry/ Research (Level 4) No 0%
(8) Professional Effectiveness and Ethical Conduct (Level 3) No 0%
(5) Interdisciplinary, Inclusiveness, Influence (Level 3) No 0%
(4) Design (Level 3) No 0%
(2) Engineering/ IT Specialisation (Level 4) No 0%
(3) Problem Solving and Inventiveness (Level 3) No 0%
(1) Maths/ Science Methods and Tools (Level 4) No 0%

These goals are selected from Engineering & IT Graduate Outcomes Table 2018 which defines overall goals for courses where this unit is primarily offered. See Engineering & IT Graduate Outcomes Table 2018 for details of the attributes and levels to be developed in the course as a whole. Percentage figures alongside each course goal provide a rough indication of their relative weighting in assessment for this unit. Note that not all goals are necessarily part of assessment. Some may be more about practice activity. See Learning outcomes for details of what is assessed in relation to each goal and Assessment for details of how the outcome is assessed. See Attributes for details of practice provided for each goal.