Note: This unit version has not been officially published yet and is subject to change!

CSYS5030: Information Theory and Self-Organisation (2019 - Semester 2)

Download UoS Outline

Unit: CSYS5030: Information Theory and Self-Organisation (6 CP)
Mode: Normal-Day
On Offer: Yes
Level: Postgraduate
Faculty/School: Faculty of Engineering and Information Technologies
Unit Coordinator/s: Dr Lizier, Joseph
Session options: Semester 2
Versions for this Unit:
Campus: Camperdown/Darlington
Pre-Requisites: None.
Brief Handbook Description: The dynamics of complex systems are often described in terms of how they process information and self-organise; for example regarding how genes store and utilise information, how information is transferred between neurons in undertaking cognitive tasks, and how swarms process information in order to collectively change direction in response to predators. The language of information also underpins many of the central concepts of complex adaptive systems, including order and randomness, self-organisation and emergence. Shannon information theory, which was originally founded to solve problems of data compression and communication, has found contemporary application in how to formalise such notions of information in the world around us and how these notions can be used to understand and guide the dynamics of complex systems.

This unit of study introduces information theory in this context of analysis of complex systems, foregrounding empirical analysis using modern software toolkits, and applications in time-series analysis, nonlinear dynamical systems and data science. Students will be introduced to the fundamental measures of entropy and mutual information, as well as dynamical measures for time series analysis and information flow such as transfer entropy, building to higher-level applications such as feature selection in machine learning and network inference. They will gain experience in empirical analysis of complex systems using comprehensive software toolkits, and learn to construct their own analyses to dissect and design the dynamics of self-organisation in applications such as neural imaging analysis, natural and robotic swarm behaviour, characterisation of risk factors for and diagnosis of diseases, and financial market dynamics.
Assumed Knowledge: Competency in 1st year mathematics, and basic computer programming skills are assumed. Competency in 1st year undergraduate level statistics (for example, covering probabilities, conditional probabilities, Gaussian distribution, correlations, statistical significance/hypothesis testing and p-values). An exposure to linear algebra would be useful but not mandatory.
Lecturer/s: Dr Lizier, Joseph
Timetable: CSYS5030 Timetable
Time Commitment:
# Activity Name Hours per Week Sessions per Week Weeks per Semester
1 Lecture 2.00 1 13
2 Laboratory 1.00 1 13
3 Independent Study 7.00 1 13
T&L Activities: This unit of study comprises of regular lectures, as well as laboratory / tutorial sessions. These sessions will take place in a lab with access to relevant computer facilities. Depending on the syllabus, some weeks will comprise tutorials where students will solve problems with the help of tutors, and other weeks will comprise programming or software based laboratory experiments.

During lab sessions with a programming/software component, tutors will be present to assist students develop relevant programming or other computing skills.

Students will use independent study time to further develop their computing skills and to practise solving analytical problems.

Learning outcomes are the key abilities and knowledge that will be assessed in this unit. They are listed according to the course goal supported by each. See Assessment Tab for details how each outcome is assessed.

Unassigned Outcomes
1. Capacity to critically evaluate investigations of self-organisation and relationships in complex systems using information theory, and the insights provided
2. Develop scientific programming skills which can be applied in complex system analysis and design
3. Ability to apply and make informed decisions in selecting and using information-theoretic measures and software tools to analyse complex systems
4. Ability to create information-theoretic analyses of real-world data sets, in particular in a student’s domain area of expertise
5. Understand basic information-theoretic measures, and advanced measures for time-series, and how to use these to analyse and dissect the nature, structure, function and evolution of complex systems
6. To be able to understand the design of and to extend the design of a piece of software using techniques from class and your own readings
Assessment Methods:
# Name Group Weight Due Week Outcomes
1 Literature review* No 15.00 Week 4 (Friday, 11 pm) 1, 3, 5,
2 Information theory exercises* No 25.00 Week 7 (Friday, 11 pm) 2, 3, 4, 5, 6,
3 Project presentation* No 20.00 Week 12 (During your timetabled class) 1, 2, 3, 4, 5, 6,
4 Project report* No 40.00 STUVAC (Week 14) (Friday, 11 pm) 1, 2, 3, 4, 5, 6,
Assessment Description: The assessments will consist of four assignments. Assignments 1-4 will be completed individually.

Literature review: You will write a short appraisal of an article / report / blog post which provides an information-theoretic analysis of empirical data, and critically evaluate that study. The reviews will be socially shared and discussed in class.

Information theory exercises: You will demonstrate your understanding of the information-theoretic concepts taught in class in calculation exercises (involving both involve mathematical and computational tasks) and short answer questions.

Info theory project presentation and report: You will create an information-theoretic analysis of a data set of your choosing, selecting appropriate tools to answer a question of interest regarding relationships in that data. You will deliver an in-class presentation, and write a report, describing your approach, the results, discussing implications for the system under study, and critically evaluating your findings.

Written assignments and exercises (without formally applying for special consideration) will be assessed a penalty of 7% per day until reaching the day limit where special consideration is required. Extensions for the Project Presentation will only be granted in the case where formal Special Consideration has been applied for and approved.

Then university has authorised and mandated the use of text based similarity detecting software TURNITIN for all text based written assignments.

IMPORTANT: There may be statistically defensible moderation when combining the marks from each component to ensure consistency of marking between markers, and alignment of final grades with unit outcomes and grade descriptors.

* means this assessment must be repeated or will be replaced with a different assessment if missed due to special consideration
Assessment Feedback: Feedback for assignments will be through Canvas/TurnItIn e-learning portal, though which these assignments will be submitted.
Grade Type Description
Standards Based Assessment Final grades in this unit are awarded at levels of HD for High Distinction, DI (previously D) for Distinction, CR for Credit, PS (previously P) for Pass and FA (previously F) for Fail as defined by University of Sydney Assessment Policy. Details of the Assessment Policy are available on the Policies website at . Standards for grades in individual assessment tasks and the summative method for obtaining a final mark in the unit will be set out in a marking guide supplied by the unit coordinator.
Policies & Procedures: See the policies page of the faculty website at for information regarding university policies and local provisions and procedures within the Faculty of Engineering and Information Technologies.
Recommended Reference/s: Note: References are provided for guidance purposes only. Students are advised to consult these books in the university library. Purchase is not required.

Note that the "Weeks" referred to in this Schedule are those of the official university semester calendar

Week Description
Week 1 Lecture/Tutorial: Introduction to information theory and entropy
Week 2 Lecture/Tutorial: Entropy and mutual information I
Week 3 Lecture/Tutorial: Entropy and mutual information II
Week 4 Lecture/Tutorial: Information-theoretic estimators and the JIDT toolkit
Assessment Due: Literature review*
Week 5 Lecture/Tutorial: Self-organisation and case studies
Week 6 Lecture/Tutorial: Statistical significance
Week 7 Lecture/Tutorial: Information dynamics (time-series analysis) I
Assessment Due: Information theory exercises*
Week 8 Lecture/Tutorial: Information dynamics (time-series analysis) II
Week 9 Lecture/Tutorial: Information dynamics (time-series analysis) III
Week 10 Lecture/Tutorial: Network inference
Week 11 Lecture/Tutorial: Project work
Week 12 Lecture/Tutorial: Project presentations
Assessment Due: Project presentation*
Week 13 Lecture/Tutorial: Wrap up / Project work
STUVAC (Week 14) Assessment Due: Project report*

Course Relations

The following is a list of courses which have added this Unit to their structure.

Course Year(s) Offered
Graduate Diploma in Complex Systems 2017, 2018, 2019
Master of Complex Systems 2017, 2018, 2019

Course Goals

This unit contributes to the achievement of the following course goals:

Attribute Practiced Assessed
(5) Interdisciplinary, Inclusiveness, Influence (Level 4) No 0%
(6) Communication and Inquiry/ Research (Level 3) No 0%
(4) Design (Level 4) No 0%
(3) Problem Solving and Inventiveness (Level 4) No 0%
(2) Engineering/ IT Specialisation (Level 5) No 0%
(1) Maths/ Science Methods and Tools (Level 5) No 0%

These goals are selected from Engineering & IT Graduate Outcomes Table 2018 which defines overall goals for courses where this unit is primarily offered. See Engineering & IT Graduate Outcomes Table 2018 for details of the attributes and levels to be developed in the course as a whole. Percentage figures alongside each course goal provide a rough indication of their relative weighting in assessment for this unit. Note that not all goals are necessarily part of assessment. Some may be more about practice activity. See Learning outcomes for details of what is assessed in relation to each goal and Assessment for details of how the outcome is assessed. See Attributes for details of practice provided for each goal.