Note: This unit version is currently under review and is subject to change!

COMP5617: Empirical Security Analysis and Engineering (2017 - Semester 2)

Download UoS Outline

Unit: COMP5617: Empirical Security Analysis and Engineering (6 CP)
Mode: Normal-Day
On Offer: Yes
Level: Postgraduate
Faculty/School: School of Information Technologies
Unit Coordinator/s: Dr Holz, Ralph
Session options: Semester 2
Versions for this Unit:
Site(s) for this Unit:
Campus: Camperdown/Darlington
Pre-Requisites: None.
Brief Handbook Description: This unit will present the lessons from recent research and from case studies of practice to bring students the skills to assess and improve the security of deployed systems. A particular focus is on data-driven approaches to collect operational data about a system's security. We explore deployment issues at local and global scale, e.g. for X.509, DNS, and BGP, and also take human factors explicitly into account. As a result, students will learn to put building blocks of security together in a sound way, to arrive at engineering solutions that are empirically verifiable, functional, and secure against realistic threats. As Dan Geer once famously said: "Any security technology whose effectiveness can't be empirically determined is indistinguishable from blind luck."
Assumed Knowledge: ELEC5616 OR INFO2315. Good programming skills in Python or C. A reasonable technical orientation and basic networking knowledge is required. Students should bring the mathematical skills to understand cryptography; the unit will introduce the functional principles as background. Prior exposure to security is helpful, but not a pre-requisite.
Lecturer/s: Dr Holz, Ralph
Tutor/s: tba.
Timetable: COMP5617 Timetable
Time Commitment:
# Activity Name Hours per Week Sessions per Week Weeks per Semester
1 Lecture 2.00 1 13
2 Tutorial 2.00 1 12
3 Independent Study 5.00 12
4 Project Work - own time 4.00 8

Attributes listed here represent the key course goals (see Course Map tab) designated for this unit. The list below describes how these attributes are developed through practice in the unit. See Learning Outcomes and Assessment tabs for details of how these attributes are assessed.

Attribute Development Method Attribute Developed
Demonstrated by modelling the empirical security measurements as part of the practical assignment Design (Level 3)
Lectures and tutorials on Internet service engineering and security analysis concepts; practical assignment on security engineering Engineering/IT Specialisation (Level 4)
Prescribed readings and case study paper; conducing an empirical security analysis Maths/Science Methods and Tools (Level 4)
Case study paper Information Seeking (Level 4)
Through lectures on security engineering, privacy-preserving security measures and usability of security practices Professional Conduct (Level 3)

For explanation of attributes and levels see Engineering & IT Graduate Outcomes Table.

Learning outcomes are the key abilities and knowledge that will be assessed in this unit. They are listed according to the course goal supported by each. See Assessment Tab for details how each outcome is assessed.

Professional Conduct (Level 3)
1. Knowledge of privacy-preserving technologies
2. Awareness of security vs. usability trade-offs
Information Seeking (Level 4)
3. Ability to research and analyse information about current IT security topics.
Maths/Science Methods and Tools (Level 4)
4. Understanding of data-driven security defences
5. Practical experience with scanning and monitoring of Internet services to determine deployment security
Engineering/IT Specialisation (Level 4)
6. Understanding of the building blocks of Internet services such as the Internet naming and routing system, and the WWW
7. Understanding of the main security protocols in the Internet stack
Design (Level 3)
8. Ability to design and conduct an empirical security analysis
Assessment Methods:
# Name Group Weight Due Week Outcomes
1 Modelling Empirical Security Measurements No 10.00 Week 4 4, 6, 7, 8,
2 Security Measurement / Scan Task No 20.00 Week 9 5, 8,
3 Case Study / Paper No 10.00 Week 12 4, 6, 7, 8,
4 Final Examination No 60.00 Exam Period 1, 2, 4, 6, 7,
Assessment Description: The late penalty for all practical exercises is 20% of the awarded mark per day late; maximum 5 days late (after that: 0).

The security analysis and engineering concepts will be assessed in a 2 hour written final exam in the examination period.

You must get 40% in the final exam to pass the unit, regardless of the sum of your individual marks.

There may be statistically defensible moderation when combining the marks from each component to ensure consistency of marking between markers, and alignment of final grades with unit outcomes.
Assessment Feedback: Feedback on the progress of the projects will be given throughout the semester in the tutorial after the lecture.
Grade Type Description
Standards Based Assessment Final grades in this unit are awarded at levels of HD for High Distinction, DI (previously D) for Distinction, CR for Credit, PS (previously P) for Pass and FA (previously F) for Fail as defined by University of Sydney Assessment Policy. Details of the Assessment Policy are available on the Policies website at . Standards for grades in individual assessment tasks and the summative method for obtaining a final mark in the unit will be set out in a marking guide supplied by the unit coordinator.
Minimum Pass Requirement It is a policy of the School of Information Technologies that in order to pass this unit, a student must achieve at least 40% in the written examination. For subjects without a final exam, the 40% minimum requirement applies to the corresponding major assessment component specified by the lecturer. A student must also achieve an overall final mark of 50 or more. Any student not meeting these requirements may be given a maximum final mark of no more than 45 regardless of their average.
Policies & Procedures: IMPORTANT: School policy relating to Academic Dishonesty and Plagiarism.

In assessing a piece of submitted work, the School of IT may reproduce it entirely, may provide a copy to another member of faculty, and/or to an external plagiarism checking service or in-house computer program and may also maintain a copy of the assignment for future checking purposes and/or allow an external service to do so.

Other policies

See the policies page of the faculty website at for information regarding university policies and local provisions and procedures within the Faculty of Engineering and Information Technologies.
Prescribed Text/s: Note: Students are expected to have a personal copy of all books listed.
Recommended Reference/s: Note: References are provided for guidance purposes only. Students are advised to consult these books in the university library. Purchase is not required.
Online Course Content: On the unit`s eLearning site (Blackboard), there will be available the lecture slides, readings, lab handouts and any background information.

Note that the "Weeks" referred to in this Schedule are those of the official university semester calendar

Week Description
Week 1 Lecture: Introduction

• Unit organisation

• Intro: security engineering

• Threat modelling
Week 2 Lecture: Building block: Crypto

• Functional overview: symmetric and public-key cryptography

• Case study: the proliferation of weak keys in the wild

• Measurement of cipher deployment

Short assignment starts: modelling/measurement task
Week 3 Lecture: Building block: protocols

• Functional overview: security protocols in the Internet stack

• Introduction: scanning and monitoring to determine deployment security

• Real-world trust anchors: hierarchical PKIs, flat PKIs
Week 4 What goes wrong in deployment: Internet protocol use

• Deployment of TLS and X.509 as the security backbone of the Internet

• Successful subversion, lessons learnt

• Mistakes in deployment, lessons learnt
Assessment Due: Modelling Empirical Security Measurements
Week 5 Lecture: Data-driven modern defences

• The notary principle

• Append-only auditable logs

• Cross-validation and monitoring to achieve better security

Long assignment starts: security measurement/scan
Week 6 Lecture: Naming and security

• Naming systems

• DNSSEC extensions

• Deployment issues
Week 7 Internet routing security

• The insecurity of global routing: threat model and effects

• BGPSec and RPKI

• Measurement-based defences: BGP threat detection
Week 8 Lecture: The WWW: the Achilles heel of deployment

• Attack surface in deployment

• Engineering a secure Web application

• Case study: Websockets, WebRTC
Week 9 Lecture: Usable security

• Security vs. usability trade-offs

• User psychology

• Case study: passwords (CRAM/SCRAM, best practices)

Short assignment starts: case study/paper
Assessment Due: Security Measurement / Scan Task
Week 10 Guest lecture: Real-world security: the world of finances (TBC)

This is a placeholder as the lecture in wk 10 falls on public holiday - currently looking for a solution for this.
Week 11 Lecture: Special topic: Privacy-preserving technologies

• Engineering for anonymity/pseudonymity

• Anonymity and censorship evasion

• Deniable communication: OTR
Week 12 Lecture: Alternative 1: Guest lecture on Real-World security: the world of finances

Alternative 2:

Special topic: Censorship and network interference

• Network interference and forms of censorship

• Measurement of censorship and interference
Assessment Due: Case Study / Paper
Week 13 Lecture: Review of unit

• Discussion of final exam

• Q&A

• Feedback
Exam Period Assessment Due: Final Examination

Course Relations

The following is a list of courses which have added this Unit to their structure.

Course Year(s) Offered
Graduate Certificate in Information Technology 2017
Graduate Certificate in Information Technology Management 2017
Graduate Diploma in Computing 2015, 2016, 2017
Graduate Diploma in Information Technology 2017
Graduate Diploma in Information Technology Management 2017
Master of Information Technology 2017
Master of Information Technology Management 2017
Master of IT/Master of IT Management 2017

Course Goals

This unit contributes to the achievement of the following course goals:

Attribute Practiced Assessed
Professional Conduct (Level 3) Yes 24%
Information Seeking (Level 4) Yes 0%
Maths/Science Methods and Tools (Level 4) Yes 28.5%
Engineering/IT Specialisation (Level 4) Yes 32%
Design (Level 3) Yes 15.5%

These goals are selected from Engineering & IT Graduate Outcomes Table which defines overall goals for courses where this unit is primarily offered. See Engineering & IT Graduate Outcomes Table for details of the attributes and levels to be developed in the course as a whole. Percentage figures alongside each course goal provide a rough indication of their relative weighting in assessment for this unit. Note that not all goals are necessarily part of assessment. Some may be more about practice activity. See Learning outcomes for details of what is assessed in relation to each goal and Assessment for details of how the outcome is assessed. See Attributes for details of practice provided for each goal.