Note: This unit version is currently under review and is subject to change!

COMP5617: Empirical Security Analysis and Engineering (2018 - Semester 2)

Download UoS Outline

Unit: COMP5617: Empirical Security Analysis and Engineering (6 CP)
Mode: Normal-Day
On Offer: Yes
Level: Postgraduate
Faculty/School: School of Computer Science
Unit Coordinator/s: Dr Holz, Ralph
Session options: Semester 2
Versions for this Unit:
Site(s) for this Unit:
Campus: Camperdown/Darlington
Pre-Requisites: ELEC5616 OR INFO3616 OR INFO2315.
Brief Handbook Description: This unit will present the lessons from recent research and from case studies of practice to bring students the skills to assess and improve the security of deployed systems. A particular focus is on data-driven approaches to collect operational data about a system's security. We explore deployment issues at local and global scale, e.g. for X.509, DNS, and BGP, and we take human factors explicitly into account. We present modern data analysis methods that are based on machine-learning and we take deep dives into privacy and mobile security.

As a result, students will learn to put building blocks of security together in a sound way, to arrive at engineering solutions that are empirically verifiable, functional, and secure against realistic threats. As Dan Geer once famously said: "Any security technology whose effectiveness can't be empirically determined is indistinguishable from blind luck." This unit runs a world-first lab-based assignment: students will work with colleagues at the Technical University of Munich on joint security measurement (globally distributed measurement) and produce a report.

This unit is limited to 24 students to accomodate the joint lab with Technical University of Munich.
Assumed Knowledge: - This unit runs a globally distributed lab assignment with students at Technical University of Munich. We expect students in this unit to be dedicated to cross-continental coordination and team work. - Good programming skills in Go, Python, or C. - Willingness to learn a new programming language without assistance is expected. - Willingness to do a Go tutorial at semester start unless students are already proficient in Go. - Ability to work with UNIX/Linux command-line and tools is expected. - Ability to work with version control is expected. - Technical orientation and foundational networking knowledge is required. - Students should bring the mathematical skills to understand cryptography.
Additional Notes: We are going to use the Go programming language, which is why we require a background in either Python or C for students who do not know Go yet. Students are encouraged to familiarise themselves with Go:
Department Permission Department permission is required for enrollment in this session.
Lecturer/s: Dr Holz, Ralph
Seneviratne, Suranga
Tutor/s: tba.
Timetable: COMP5617 Timetable
Time Commitment:
# Activity Name Hours per Week Sessions per Week Weeks per Semester
1 Lecture 2.00 1 13
2 Tutorial 2.00 1 12
3 Independent Study 5.00 12
4 Project Work - own time 4.00 8

Attributes listed here represent the key course goals (see Course Map tab) designated for this unit. The list below describes how these attributes are developed through practice in the unit. See Learning Outcomes and Assessment tabs for details of how these attributes are assessed.

Attribute Development Method Attribute Developed
Demonstrated by modelling the empirical security measurements as part of the practical assignment Design (Level 3)
Lectures and tutorials on Internet service engineering and security analysis concepts; practical assignment on security engineering Engineering/IT Specialisation (Level 4)
Prescribed readings and case study paper; conducing an empirical security analysis Maths/Science Methods and Tools (Level 4)
Case study paper Information Seeking (Level 4)
Through lectures on security engineering, privacy-preserving security measures and usability of security practices Professional Conduct (Level 3)

For explanation of attributes and levels see Engineering & IT Graduate Outcomes Table 2018.

Learning outcomes are the key abilities and knowledge that will be assessed in this unit. They are listed according to the course goal supported by each. See Assessment Tab for details how each outcome is assessed.

Professional Conduct (Level 3)
1. Knowledge of privacy-preserving technologies
2. Awareness of security vs. usability trade-offs
Information Seeking (Level 4)
3. Ability to research and analyse information about current IT security topics.
Maths/Science Methods and Tools (Level 4)
4. Understanding of data-driven security defences
5. Practical experience with scanning and monitoring of Internet services to determine deployment security
6. Analysis of security data
Engineering/IT Specialisation (Level 4)
7. Understanding of the building blocks of Internet services such as the Internet naming and routing system, and the WWW
8. Understanding of the main security protocols in the Internet stack
Design (Level 3)
9. Ability to design and conduct an empirical security analysis
Assessment Methods:
# Name Group Weight Due Week Outcomes
1 Self-assessment No 10.00 Week 3 4, 5, 8, 9,
2 Analysis of security data with machine learning No 20.00 Week 10 3, 4, 6, 8, 9,
3 Global security measurement (with TU Munich) Yes 20.00 STUVAC (Week 14) 5, 6, 9,
4 Final Examination No 50.00 Exam Period 1, 2, 4, 7, 8, 9,
Assessment Description: The late penalty for all practical exercises is 20% of the awarded mark per day late; maximum 5 days late (after that: 0 credits).

Please note that Assessment 3 (Global Security Measurement) is conducted as a joint exercise with students of TU Munich;

there is little room for extensions beyond extensions for medical reasons. The coordinators may offer assessment adjustments

in justified cases.

The final exam is 90 minutes.

Students must get 40% in the final exam to pass the unit, regardless of the sum of individual marks.

There may be statistically defensible moderation when combining the marks from each component to ensure consistency of marking between markers, and alignment of final grades with unit outcomes.
Assessment Feedback: Feedback on the progress of the projects will be given throughout the semester in the tutorial after the lecture.
Grade Type Description
Standards Based Assessment Final grades in this unit are awarded at levels of HD for High Distinction, DI (previously D) for Distinction, CR for Credit, PS (previously P) for Pass and FA (previously F) for Fail as defined by University of Sydney Assessment Policy. Details of the Assessment Policy are available on the Policies website at . Standards for grades in individual assessment tasks and the summative method for obtaining a final mark in the unit will be set out in a marking guide supplied by the unit coordinator.
Minimum Pass Requirement It is a policy of the School of Computer Science that in order to pass this unit, a student must achieve at least 40% in the written examination. For subjects without a final exam, the 40% minimum requirement applies to the corresponding major assessment component specified by the lecturer. A student must also achieve an overall final mark of 50 or more. Any student not meeting these requirements may be given a maximum final mark of no more than 45 regardless of their average.
Policies & Procedures: IMPORTANT: School policy relating to Academic Dishonesty and Plagiarism.

In assessing a piece of submitted work, the School of IT may reproduce it entirely, may provide a copy to another member of faculty, and/or to an external plagiarism checking service or in-house computer program and may also maintain a copy of the assignment for future checking purposes and/or allow an external service to do so.

Other policies

See the policies page of the faculty website at for information regarding university policies and local provisions and procedures within the Faculty of Engineering and Information Technologies.
Prescribed Text/s: Note: Students are expected to have a personal copy of all books listed.
Recommended Reference/s: Note: References are provided for guidance purposes only. Students are advised to consult these books in the university library. Purchase is not required.
Online Course Content: Canvas site will be available.

Note that the "Weeks" referred to in this Schedule are those of the official university semester calendar

Week Description
Week 1 Lecture: Unit organisation; Introduction to security engineering
Tutorial: Warming up with security engineering
Week 2 Lecture: Security issues and usability; measurement
Tutorial: Usability
Week 3 Lecture: Symmetric cryptography and cryptographically secure generation of random numbers; measurement

Asymmetric cryptography and weak keys in the wild; measurement
Tutorial: Practicing symmetric encryption and finding weaknesses

Practicing public-key cryptography and finding weaknesses
Assessment Due: Self-assessment
Week 4 Design of cryptographic protocols
Tutorial: Breaking cryptographic protocols
Week 5 Lecture: Key Distribution Centres and Public Key Infrastructures
Tutorial: Understanding KDCs and PKIs (Kerberos and X.509)
Week 6 Lecture: Deployment and reinforcement: the example of X.509
Tutorial: Practicing advanced reinforcement concepts for X.509
Week 7 Lecture: Security data analysis with machine learning
Tutorial: Practicing anomaly detection and ML
Week 8 Lecture: Security data analysis with machine learning 2
Tutorial: Security data analysis with machine learning 2
Week 9 Other: Free slot for guest lecture. This may collide with a public holiday; in this case we reschedule the guest lecture to another time in this week.
Week 10 Lecture: Mobile security and IoT security
Tutorial: Mobile security and IoT security
Assessment Due: Analysis of security data with machine learning
Week 11 Lecture: Critical infrastructure: security for the DNS; measurement
Tutorial: DNSSEC
Week 12 Lecture: Critical Infrastructure 2: Internet routing
Tutorial: BGP and RPKI
Week 13 Lecture: Privacy
Tutorial: k-anonymity
STUVAC (Week 14) Assessment Due: Global security measurement (with TU Munich)
Exam Period Assessment Due: Final Examination

Course Relations

The following is a list of courses which have added this Unit to their structure.

Course Year(s) Offered
Bachelor of Advanced Computing (Computational Data Science) 2019
Bachelor of Advanced Computing (Software Development) 2019
Graduate Certificate in Information Technology 2017, 2018, 2019
Graduate Certificate in Information Technology Management 2017, 2018, 2019
Graduate Diploma in Computing 2015, 2016, 2017, 2018, 2019
Graduate Diploma in Information Technology 2017, 2018, 2019
Graduate Diploma in Information Technology Management 2017, 2018, 2019
Master of Information Technology 2017, 2018, 2019
Master of Information Technology Management 2017, 2018, 2019
Master of IT/Master of IT Management 2017, 2018, 2019

Course Goals

This unit contributes to the achievement of the following course goals:

Attribute Practiced Assessed
Professional Conduct (Level 3) Yes 15%
Information Seeking (Level 4) Yes 2%
Maths/Science Methods and Tools (Level 4) Yes 42%
Engineering/IT Specialisation (Level 4) Yes 23%
Design (Level 3) Yes 18%

These goals are selected from Engineering & IT Graduate Outcomes Table 2018 which defines overall goals for courses where this unit is primarily offered. See Engineering & IT Graduate Outcomes Table 2018 for details of the attributes and levels to be developed in the course as a whole. Percentage figures alongside each course goal provide a rough indication of their relative weighting in assessment for this unit. Note that not all goals are necessarily part of assessment. Some may be more about practice activity. See Learning outcomes for details of what is assessed in relation to each goal and Assessment for details of how the outcome is assessed. See Attributes for details of practice provided for each goal.