Schedule of tutorials - 19th July, 2020

11:30 - 13:30
14:00 - 16:00
16:30 - 18:30
19:00 - 21:00
11:30 - 13:30
Tutorial Title Presenter Conference Email
Adversarial Machine Learning: On The Deeper Secrets of Deep Learning Danilo Vargas IJCNN
Brain-Inspired Spiking Neural Network Architectures for Deep, Incremental Learning and Knowledge Evolution       Nikola Kasabov IJCNN
Fundamentals of Fuzzy Networks Alexander Gegov, Farzad Arabikhan FUZZ
Instance Space Analysis for Rigorous and Insightful Algorithm Testing Kate Smith-Miles, Mario Andres, Munoz Acosta WCCI
Advances in Deep Reinforcement Learning Thanh Thi Nguyen, Vijay Janapa Reddi,Ngoc Duy Nguyen, IJCNN 
Selection Exploration and Exploitation Stephen Chen, James Montgomery CEC
Visualising the search process of EC algorithms Su Nguyen, Yi Mei, and Mengjie Zhang CEC
Evolutionary Machine Learning  Masaya Nakata, Shinichi Shirakawa, Will Browne CEC
Evolutionary Many-Objective Optimization Hisao Ishibuchi, Hiroyuki Sato CEC
14:00 - 16:00
Tutorial Title Presenter Conference Email
Deep Learning for Graphs Davide Bacciu IJCNN
Randomization Based Deep and Shallow Learning Methods for Classification and Forecasting P N Suganthan IJCNN
Multi-modality Helps in Solving Biomedical Problems: Theory and Applications Sriparna Saha, Pratik Dutta WCCI
Deep Stochastic Learning and Understanding Jen-Tzung Chien IJCNN
Paving the way from Interpretable Fuzzy Systems to Explainable AI Systems José M. Alonso, Ciro Castiello, Corrado Menca, Luis Magdalena FUZZ
Pareto Optimization for Subset Selection: Theories and Practical Algorithms Chao Qian, Yang Yu CEC
Benchmarking and Analyzing Iterative Optimization Heuristics with IOHprofiler Carola Doerr, Thomas Bäck, Ofer Shir, Hao Wang CEC
Differential Evolution Rammohan Mallipeddi,  Guohua Wu         CEC
Evolutionary computation for games: learning, planning, and designing

Julian Togelius , Jialin Liu 

16:30 - 18:30
Tutorial Title Presenter Conference Email
From brains to deep neural networks Saeid Sanei, Clive Cheong Took IJCNN
Evolution of Neural Networks Risto Miikkulainen IJCNN
Experience Replay for Deep Reinforcement Learning Abdul Rahman Al Tahhan, Vasile Palade IJCNN
Fuzzy Systems for Neuroscience and Neuro-engineering Applications Javier Andreu, CT Lin FUZZ
Dynamic Parameter Choices in Evolutionary Computation Carola Doerr, Gregor Papa  CEC
Evolutionary Computation for Dynamic Multi-objective Optimization Problems Shengxiang Yang CEC
Evolutionary Algorithms and Hyper-Heuristics Nelishia Pillay CEC
Large-Scale Global Optimization Mohammad Nabi Omidvar, Antonio LaTorre CEC 
Bilevel optimization Ankur Sinha, Kalyanmoy Deb CEC
19:00 - 21:00
Tutorial Title Presenter Conference Email
How to combine human and computational intelligence? Peter Erdi WCCI
Machine learning  for data streams in Python with scikit-multi flow Jacob Montiel, Heitor Gomes,Jesse Read, Albert Bifet IJCNN
Deep randomized neural networks Claudio Gallicchio, Simone Scardapane IJCNN
Mechanisms of Universal Turing Machines: Vision, Audition, Natural Language, APFGP and Consciousness Juyang Weng IJCNN
Patch Learning: A New Method of Machine Learning, Implemented by Means of Fuzzy Sets Jerry Mendel FUZZ
Self-Organizing Migrating Algorithm - Recent Advances and Progress in Swarm Intelligence Algorithms

Roman Senkerik CEC
Large-Scale Global Optimization - PART 2 Mohammad Nabi Omidvar, Antonio LaTorre CEC 
Recent Advances in Particle Swarm Optimization Analysis and Understanding Andries Engelbrecht, Christopher Cleghorn CEC
Nature-Inspired Techniques for Combinatorial Problems Malek Mouhoub

Niching Methods for Multimodal Optimization Xiaodong Li, Mike Preuss, Michael G. Epitropakis


Adversarial Machine Learning: On The Deeper Secrets of Deep Learning

Danilo V. Vargas, Associate Professor
Faculty of Information Science and Electrical Engineering, Kyushu University


Recent research has found out that Deep Neural Networks (DNN) behave strangely to slight changes in the input. This tutorial will talk about this curious, and yet, still poorly understood behavior. Moreover, it will dig deep into the meaning of this behavior and its links to the understanding of DNNs.

In this tutorial, I will explain the basic concepts underlying adversarial machine learning and briefly review the state-of-the-art with many illustrations and examples. In the latter part of the tutorial, I will demonstrate how attacks are helping to understand the behavior of DNNs as well as show how many defenses proposed are not improving the robustness. There are still many challenges and puzzles left unsolved. I will present some of them as well as delineate a couple of paths to a solution. Lastly, the tutorial will be closed with an open discussion and promotion of cross-community collaborations.

Tutorial Presenters (names with affiliations):

Associate Professor at Kyushu University

Tutorial Presenters’ Bios:

Danilo Vasconcellos Vargas is currently an Associate Professor at Kyushu University, Japan. His research interests span Artificial Intelligence (AI), evolutionary computation, complex adaptive systems, interdisciplinary studies involving or using an AI’s perspective and AI applications. Many of his works were published in prestigious journals such as Evolutionary Computation (MIT Press), IEEE Transactions on Evolutionary Computation and and IEEE Transactions of Neural Networks and Learning Systems with press coverage in news magazines such as BBC news. He received awards such as the IEEE Excellent Student Award and scholarships to study in Germany and Japan for many years. Regarding his community activities, he was the presenter of two tutorials at the renowned GECCO conference.

Regarding adversarial machine learning, he has more than 5 invited talks about the subject. One given in a workshop in CVPR 2019. He has authored more than 10 articles and three chapters on books about adversarial machine learning, one of its research output was published on BBC news (about the paper “One pixel attack for fooling deep neural networks”).

Currently, he leads the Laboratory of Intelligent Systems aimed at building a new age of robust and adaptive artificial intelligence. More info can be found both in his website and his Lab Page

External website with more information on Tutorial (if applicable):


Brain-Inspired Spiking Neural Network Architectures for Deep, Incremental Learning and Knowledge Representation   

Prof. Nikola Kasabov
Director, Knowledge Engineering and Discovery Research Institute, Auckland University of Technology, Auckland, New Zealand


The 2 hour tutorial demonstrates that the third generation of artificial neural networks, the spiking neural networks (SNN) are not only capable of deep, incremental learning of temporal or spatio-temporal data, but also enabling the extraction of knowledge representation from the learned data and tracing the knowledge evolution over time from the incoming data. Similarly to how the brain learns, these SNN models do not need to be restricted in number of layers, neurons in each layer, etc. as they adopt self-organising learning principles of the brain. The tutorial consists of 2 parts:

  1. Algorithms for deep, incremental learning in SNN.
  2. Algorithms for knowledge representation and for tracing the knowledge evolution in SNN over time from incoming data. Representing fuzzy spatio-temporal rules from SNN.
  3. Selected Applications

The material is illustrated on an exemplar SNN architecture NeuCube (free software and open source along with a cloud-based version available from Case studies are presented of brain and environmental data modelling and knowledge representation using incremental and transfer learning algorithms. These include: predictive modelling of EEG and fMRI data measuring cognitive processes and response to treatment; AD prediction; understanding depression; predicting environmental hazards and extreme events.

It is also demonstrated that brain-inspired SNN architectures, such as the NeuCube, allow for  knowledge transfer between humans and machines through building brain-inspired Brain-Computer Interfaces (BI-BCI). These are used to understand human-to-human knowledge transfer through hyper-scanning and also to create brain-like neuro-rehabilitation robots. This opens the way to build a new type of AI systems – the open and transparent  AI.

Reference: N.Kasabov, Time-Space, Spiking Neural Networks and Brain-Inspired Artificial Intelligence, Springer, 2019,


Prof. Nikola Kasabov, Director, Knowledge Engineering and Discovery Research Institute,

Auckland University of Technology, Auckland, New Zealand,,


Professor Nikola Kasabov is Fellow of IEEE, Fellow of the Royal Society of New Zealand, Fellow of the INNS College of Fellows, DVF of the Royal Academy of Engineering UK. He is the Founding Director of the Knowledge Engineering and Discovery Research Institute (KEDRI), Auckland and Professor at the School of Engineering, Computing and Mathematical Sciences at Auckland University of Technology, New Zealand. Kasabov is the Immediate Past President of the Asia Pacific Neural Network Society (APNNS) and Past President of the International Neural Network Society (INNS). He is member of several technical committees of IEEE Computational Intelligence Society and Distinguished Lecturer of IEEE (2012-2014). He is Editor of Springer Handbook of Bio-Neuroinformatics, Springer Series of Bio-and Neuro-systems and Springer journal Evolving Systems. He is Associate Editor of several journals, including Neural Networks, IEEE TrNN, Tr CDS, Information Sciences, Applied Soft Computing. Kasabov holds MSc and PhD from TU Sofia, Bulgaria. His main research interests are in the areas of neural networks, intelligent information systems, soft computing, bioinformatics, neuroinformatics. He has published more than 620 publications. He has extensive academic experience at various academic and research organisations in Europe and Asia, including: TU Sofia Bulgaria; University of Essex UK; University of Otago, NZ; Advisory Professor at Shanghai Jiao Tong University and CASIA China, Visiting Professor at ETH/University of Zurich and Robert Gordon University UK, Honorary Professor of Teesside University, UK; George Moore Professor of data analytics at the University of Ulster. Prof. Kasabov has received a number of awards, among them: Doctor Honoris Causa from Obuda University, Budapest; INNS Ada Lovelace Meritorious Service Award; NN Best Paper Award for 2016; APNNA ‘Outstanding Achievements Award’; INNS Gabor Award for ‘Outstanding contributions to engineering applications of neural networks’; EU Marie Curie Fellowship; Bayer Science Innovation Award; APNNA Excellent Service Award;  RSNZ Science and Technology Medal; 2015 AUT Medal; Honorable Member of the Bulgarian, the Greek and the Scottish Societies for Computer Science. More information of Prof. Kasabov can be found from:

Advances in Deep Reinforcement Learning

Thanh Thi Nguyen, Deakin University, Victoria, Australia

Vijay JanapaReddi, Harvard University, Massachusetts, USA


This tutorial presents in detail the current state-of-the-art deep RL theory and applications. We highlight the differences between various types of deep RL methods and their corresponding applications. We present our recent algorithms in the multi-objective and multi-agent domains. Real-world examples for each type of deep RL methods are given along with their demonstrations. This tutorial offers a unique opportunity to disseminate in-depth knowledge on deep RL and how to use those algorithms to solve real-world problems such as autonomous vehicles (cars and drones), autonomous surgical robotics, applications in finance, cyber security, and the Internet of Things.

Tutorial Presenters (names with affiliations):

  1. Thanh Thi Nguyen, Deakin University, Victoria, Australia.



  1. Vijay JanapaReddi, Harvard University, Massachusetts, USA.


Tutorial Presenters’ Bios:

Thanh Thi Nguyen is a leading researcher in Australia in the field of Artificial Intelligence, recognized by The Australian Newspaper in a report published in 2018. He has been invited to edit the Special Issue “Deep Reinforcement Learning: Methods and Applications” for the Electronics Journal. Dr Nguyen was a Visiting Scholar with the Computer Science Department at Stanford University in 2015 and the Edge Computing Lab at Harvard University in 2019. He received an Alfred Deakin Postdoctoral Research Fellowship in 2016, a European-Pacific Partnership for ICT Expert Exchange Program Award from European Commission in 2018, and an Australia–India Strategic Research Fund Early- and Mid-Career Fellowship 2020 awarded by the Australian Academy of Science. Dr. Nguyen obtained a PhD in Mathematics and Statistics from Monash University, Australia in 2013 and has expertise in various areas, including artificial intelligence, deep learning, deep reinforcement learning, cyber security, IoT, and data science. He is currently a Senior Lecturer in the School of Information Technology, Deakin University, Victoria, Australia.

Vijay JanapaReddi completed his PhD in computer science at Harvard University in 2010. He is a recipient of multiple awards, including the National Academy of Engineering (NAE) Gilbreth Lecturer Honor (2016), IEEE TCCA Young Computer Architect Award (2016), Intel Early Career Award (2013), Google Faculty Research Awards (2012, 2013, 2015, 2017), Best Paper at the 2005 International Symposium on Microarchitecture, Best Paper at the 2009 International Symposium on High Performance Computer Architecture, and IEEE’s Top Picks in Computer Architecture Awards (2006, 2010, 2011, 2016, 2017). Dr. Reddi is currently an Associate Professor in the John A. Paulson School of Engineering and Applied Sciences at Harvard University where he directs the Edge Computing Lab. His research interests include computer architecture and system-software design, specifically in the context of mobile and edge computing platforms based on machine learning for domains like autonomous systems and mobile robotics.

External website with more information on Tutorial (if applicable):


Deep learning for graphs

Davide Bacciu
Università di Pisa


The tutorial will introduce the lively field of deep learning for graphs and its applications.  Dealing with graph data requires learning models capable of adapting to structured samples of varying size and topology, capturing the relevant structural patterns to perform predictive and explorative tasks while maintaining the efficiency and scalability necessary to process large scale networks. The tutorial will first introduce foundational aspects and seminal models for learning with graph structured data. Then it will discuss the most recent advancements in terms of deep learning for network and graph data, including learning structure embeddings, graph convolutions, attentional models and graph generation.

Tutorial Presenters (names with affiliations):

Davide Bacciu (Università di Pisa)

Tutorial Presenters’ Bios:

Davide Bacciu is Assistant Professor at the Computer Science Department, University of Pisa. The core of his research is on Machine Learning (ML) and deep learning models for structured data processing, including sequences, trees and graphs. He is the PI of an Italian National project on ML for structured data and the Coordinator of the H2020-RIA project TEACHING (2020-2023).  He has been teaching courses of Artificial Intelligence (AI) and ML at undergraduate and graduate levels since 2010. He is an IEEE Senior Member, a member of the IEEE NN Technical Committee and of the IEEE CIS Task Force on Deep Learning. He is an Associate Editor of the IEEE Transactions on Neural Networks and Learning Systems. Since 2017 he is the Secretary of the Italian Association for Artificial Intelligence (AI*IA).

External website with more information on Tutorial (if applicable):

Randomization Based Deep and Shallow Learning Methods for Classification and Forecasting

 Dr. P. N.Suganthan
Nanyang Technological University, Singapore.


This tutorial will first introduce the main randomization-based learning paradigms with closed-form solutions such as the randomization-based feedforward neural networks, randomization based recurrent neural networks and kernel ridge regression. The popular instantiation of the feedforward type called random vector functional link neural network (RVFL) originated in early 1990s. Other feedforward methods are random weight neural networks (RWNN), extreme learning machines (ELM), etc. Reservoir computing methods such as echo state networks (ESN) and liquid state machines (LSM) are randomized recurrent networks. Another paradigm is based on kernel trick such as the kernel ridge regressionwhich includes randomization for scaling to large training data. The tutorial will also consider computational complexity with increasing scale of the classification/forecasting problems. Another randomization-based paradigm is the random forest which exhibits highly competitive performances.The tutorial will also present extensive benchmarking studies using classification and forecasting datasets.

 Key Papers:

 General Bio-sketch:

PonnuthuraiNagaratnamSuganthanreceived the B.A degree, Postgraduate Certificate and M.A degree in Electrical and Information Engineering from the University of Cambridge, UK in 1990, 1992 and 1994, respectively. He received an honorary doctorate (i.e. Doctor Honoris Causa) in 2020 from University of Maribor, Slovenia.After completing his PhD research in 1995, he served as a pre-doctoral Research Assistant in the Dept of Electrical Engineering, University of Sydney in 1995–96 and a lecturer in the Dept of Computer Science and Electrical Engineering, University of Queensland in 1996–99. He moved to Singapore in 1999. He is an Editorial Board Member of the Evolutionary Computation Journal, MIT Press (2013-2018). He is/was an associate editor of the Applied Soft Computing (Elsevier, 2018-), Neurocomputing (Elsevier, 2018-), IEEE Trans on Cybernetics (2012 – 2018), IEEE Trans on Evolutionary Computation (2005 – ), Information Sciences (Elsevier) (2009 – ), Pattern Recognition (Elsevier) (2001 – ) and Int. J. of Swarm Intelligence Research (2009 – ) Journals. He is a founding co-editor-in-chief of Swarm and Evolutionary Computation (2010 – ), an SCI Indexed Elsevier Journal. His co-authored SaDE paper (published in April 2009) won the “IEEE Trans. on Evolutionary Computation outstanding paper award” in 2012. His former PhD student, Dr Jane Jing Liang, won the IEEE CIS Outstanding PhD dissertation award, in 2014. His research interests include swarm and evolutionary algorithms, pattern recognition, big data, deep learning and applications of swarm, evolutionary & machine learning algorithms. His publications have been well cited. His SCI indexed publications attracted over 1000 SCI citations in each calendar year since 2013. He was selected as one of the highly cited researchers by Thomson Reuters yearly from 2015 to 2019 in computer science. He served as the General Chair of the IEEE SSCI 2013. He has been a member of the IEEE (S’90, M’92, SM’00, F’15) since 1991 and an elected AdCom member of the IEEE Computational Intelligence Society (CIS) in 2014-2016. He is an IEEE CIS distinguished lecturer (DLP) in 2018-2020.

Deep Stochastic Learning and Understanding

National Chiao Tung University



This tutorial addresses the advances in deep Bayesian learning for sequence data which are ubiquitous in speech, music, text,

image, video, web, communication and networking applications. Spatial and temporal contents are analyzed and represented to fulfill a variety of tasks ranging from classification, synthesis, generation, segmentation, dialogue, search, recommendation, summarization, answering, captioning, mining, translation, adaptation to name a few. Traditionally, “deep learning” is taken to be a learning process where the inference or optimization is based on the real-valued deterministic model. The “latent semantic structure” in words, sentences, images, actions, documents or videos learned from data may not be well expressed or correctly optimized in mathematical logic or computer programs. The “distribution function” in discrete or continuous latent variable model for spatial and temporal sequences may not be properly decomposed or estimated. This tutorial addresses the fundamentals of statistical models and neural networks, and focus on a series of advanced Bayesian models and deep models including recurrent neural network, sequence-to-sequence model, variational auto-encoder (VAE), attention mechanism, memory-augmented neural network, skip neural network, temporal difference VAE, stochastic neural network, stochastic temporal convolutional network, predictive state neural network, and policy neural network. Enhancing the prior/posterior representation is addressed. We present how these models are connected and why they work for a variety of applications on symbolic and complex patterns in sequence data. The variational inference and sampling method are formulated to tackle the optimization for complicated models. The embeddings, clustering or co-clustering of words, sentences or objects are merged with linguistic and semantic constraints. A series of case studies, tasks and applications are presented to tackle different issues in deep Bayesian learning and understanding. At last, we will point out a number of directions and outlooks for future studies. This tutorial serves the objectives to introduce novices to major topics within deep Bayesian learning, motivate and explain a topic of emerging importance for natural language understanding, and present a novel synthesis combining distinct lines of machine learning work.

Tutorial Presenters (names with affiliations):

Jen-TzungChien, National Chiao Tung University, Taiwan

Tutorial Presenters’ Bios:

Jen-TzungChien is the Chair Professor at the National Chiao Tung University, Taiwan. He held the Visiting Professor position at the IBM T. J. Watson Research Center, Yorktown Heights, NY, in 2010. His research interests include machine learning, deep learning, computer vision and natural language processing. Dr. Chien served as the associate editor of the IEEE Signal Processing Letters in 2008-2011, the general co-chair of the IEEE International Workshop on Machine Learning for Signal Processing in 2017, and the tutorial speaker of the ICASSP in 2012, 2015, 2017, the INTERSPEECH in 2013, 2016, the COLING in 2018, the AAAI, ACL, KDD, IJCAI in 2019. He received the Best Paper Award of IEEE Automatic Speech Recognition and Understanding Workshop in 2011 and the AAPM Farrington Daniels Award in 2018. He has published extensively, including the books “Bayesian Speech and Language Processing”, Cambridge University Press, in 2015, and “Source Separation and Machine Learning”, Academic Press, in 2018. He is currently serving as an elected member of the IEEE Machine Learning for Signal Processing Technical Committee.

External website with more information on Tutorial:

From Brain to Deep Neural Networks

Saeid Sanei
Nottingham Trent University UK

Clive Cheong Took
Royal Holloway University of London UK


The aim of this tutorial is to provide the stepping stone for machine learning enthusiasts into the area of brain pathway modelling using innovative deep learning techniques through processing and learning from electroencephalogram (EEG). An insight into EEG generation and processing will provide the audience with a better understanding of deep network structures used to learn and detect the insightful information about the deep brain function.

Tutorial Presenters

Saeid Sanei, Nottingham Trent University UK

Clive Cheong Took, Royal Holloway University of London UK


SaeidSanei is a full professor with Nottingham Trent University and a visiting professor at Imperial College London. He leads a group where several young researchers working in EEG Processing and its application in brain computer interface (BCI). He authored two research monographs on electroencephalogram (EEG) processing and pattern recognition. Saeid has delivered numerous workshops on EEG Signal Processing & Machine Learning with diverse applications all over the world particularly in Europe, China, and Singapore.

Clive Cheong Took is a senior lecturer (associate professor) at Royal Holloway, University of London. Clive has a background in machine learning and investigates its applications in biomedical problems for more than 10 years. He is an associate editor for IEEE Transactions on Neural Networks and Learning Systems since 2013, and has co-organised special issues on deep learning for healthcare and security. At WCCI 2020, he will also co-organise a special session on Generative Adversarial Learning with Ariel Ruiz-Garcia, Vasile Palade, Jürgen Schmidhuber, and Danilo Mandic.

External Website


Evolution of Neural Networks

The University of Texas at Austin and Cognizant Technology Solutions


Evolution of artificial neural networks has recently emerged as a powerful technique in two areas. First, while the standard value-function based reinforcement learning works well when the environment is fully observable, neuroevolution makes it possible to disambiguate hidden state through memory. Such memory makes new applications possible in areas such as robotic control, game playing, and artificial life. Second, deep learning performance depends crucially on the network architecture and hyperparameters. While many such architectures are too complex to be optimized by hand, neuroevolution can be used to do so automatically. Such evolutionary AutoML can be used to achieve good deep learning performance even with limited resources, or state=of-the art performance with more effort. It is also possible to optimize other aspects of the architecture, like its size, speed, or fit with hardware. In this tutorial, I will review (1) neuroevolution methods that evolve fixed-topology networks, network topologies, and network construction processes, (2) methods for neural architecture search and evolutionary AutoML, and (3) applications of these techniques in control, robotics, artificial life, games, image processing, and language.

Tutorial Presenters (names with affiliations):

The University of Texas at Austin and Cognizant Technology Solutions

Tutorial Presenters’ Bios:

RistoMiikkulainen is a Professor of Computer Science at the University of Texas at Austin and a AVP of Evolutionary AI at Cognizant Technology Solutions. He received an M.S. in Engineering from the Helsinki University of Technology, Finland, in 1986, and a Ph.D. in Computer Science from UCLA in 1990. His research focuses on methods and applications of neuroevolution, as well as neural network models of natural language processing and self-organization of the visual cortex; he is an author of over 430 articles in these research areas. He is an IEEE Fellow, recipient of the 2020 IEEE CIS EC Pioneer Award, recent awards from INNS and ISAL, as well as nine Best-Paper Awards at GECCO.

External website with more information on Tutorial (if applicable):

Experience Replay for Deep Reinforcement Learning
A Comprehensive Review

Leeds Beckett University, UK.

Coventry University, UK.

Primary contact ( )


Reinforcement learning is expected to play an important role in our AI and machine learning era, this is evident by latest major advances, particularly in games. This is due to its flexibility and arguably minimum designer intervention especially when the feature extraction process is left to a robust model such as a deep neural network. Although deep learning alleviated the long-standing burden of manual feature design, another important issue remains to be tackled, that is the experience-hungry nature of RL models which is mainly due to bootstrapping and exploration. One important technique that will play a centre stage role in tackling this issue is experience replay. Naturally, it allows us to capitalise on the already gained experience and to shorten the time needed to train an RL agent. The frequency and depth of the replay can vary significantly and currently a unifying view and a clear understanding of the issues related to off-policy and on-policy replay is generally lacking. For example, on the far end of the spectrum, extensive experience-replay, although should conceivably help reduce the data-intensity of the training period, when done naively, put significant constrains on the practicality of the model and requires both extra time and space that can grow significantly; relegating the method impractical. On the other hand, in its optimal form, whether it is a target re-evaluation or a re-update, when importance sampling ratio uses bootstrapping, the methods computational requirements matches other model based RL methods for planning. In this tutorial we will be tackling the issues and techniques related to the theory and application of deep reinforcement learning and experience replay, and how and when these techniques can be applied effectively to produce a robust model. In addition, we will promote a unified view of experience replay that involves replaying and re-evaluation of the target updates. What is more, we will show that the generalised intensive experience replay method can be used to derive several important algorithms as special cases of other methods including n-steps true online TD and LSTD. This surprising but important view can help immensely the neuro-dynamic/RL community to move this concept further forward and will benefit both the researchers and practitioners in their quest for a better and more practical RL methods and models.


Deep reinforcement learning combined with experience replay allows us to capitalise on the gained experience; capping the model appetite for new experience. Experience replay can be performed in several ways some of which may or may not be suitable for the problem in hand. For example, intensive experience replay, if performed optimally, could shorten the learning cycle of an RL agent and allow it to be taken away from the virtual arena such as games/simulation to the physical/mechanical arena such as robotics. The type of intensive training required for RL models, which can be afforded by a virtual agent, may not be tolerated, or may at least not be desired, for a physical agent. Of course, one way to move to the mechanical world is by utilising model-free policy gradient (search) methods that are based on simulation and then migrate/map the model to the physical world. However, constructing a simulation for the physical world is a tedious process that requires extra time and calibration making it impractical to the type of pervasive RL models that we hope to achieve. In both cases, whether it is a policy gradient or value-function with policy iteration, experience replay plays and important role to make them more practical. For example, for policy gradient methods adopting a softmax policy is preferable over the ε-greedy policy as it can approach asymptotically a deterministic policy after some exploration, while ε-greedy will always have a fixed percentage of exploratory actions regardless of the maturity of the policy being developed/improved.

The tutorial is timely and novel in its treatment and packaging of the topic. It will lay the necessary foundation for a unified overview of the subject. Which will allow other researcher to take it to the next level and will allow the subject area to take off on solid and unified grounds.

It turns out that extensive experience replay can be used as a generalised model from which several n-steps modern reinforcement learning techniques can be deduced, offering an easy way to unify several popular reinforcement learning methods and giving rise to several new more.

In this tutorial I will be giving a step by step detailed overview of the framework that allows us to safely deploy replay methods.
The tutorial will review all major advances of RL replay algorithms and will categorise them into: occasional replay, regular replay, and intensive regular replay.

Bellman equations are the fundamental of individual RL updates, however all the n-steps aggregate methods that have driven the latest breakthrough of RL need different treatment.

The unified view through experience replay offer a new theoretical framework to study the inner traits/properties of those techniques. For example, convergence of several new RL algorithms can be proven by proving the convergence of the unified replay algorithm and then projecting each algorithm as a special case of the general method.

——————————–First part about one hour————————by A. Altahhan————-
• Deep Reinforcement Learning a concise review
• Traditional Replay and Type of Replay: Occasional, Regular, Intensive or Both
• Unified View of Experience Replay
o Replay Past Update vs Target Re-evaluation: how to integrate
o Off-policy vs On-policy! Experience Replay
o The Role of Importance Sampling and Bootstrapping
o Emphatic TD and its Cousins
o Unifying Algorithm for Regular Intensive Sarsa(λ)-Replay
o N-steps Methods as a Special cases of Experience Replay
o Policy Search Methods and Unified Replay
o Exploration, Exploitation and Replay
o Convergence of Replay Methods
——————————–Second part about one hour———————-by V. Palade————–
• Practical Consideration for DRL and Experience Replay:
o To Replay or Not to Replay!
o Time and Space Complexity of Replay
o Combing Deep Learning and Replay in a Meaningful Way
o Softmax or ε-Greedy for Replay
o Replay for Games vs Replay for Robotics
o Live Demonstration on the Game of Packman
o Live Demo on Cheap Affordable Robot
o Audience Running the Code and Connect to the Robot
• Q&A, Discussion and Closing Notes
• To develop a deep understanding of the capabilities and limitations of deep reinforcement learning
• To develop a unified view of the different types of experience replay and how and when to apply them in the context of deep reinforcement learning

• Demonstrate how to apply experience replay on policy search methods
• Demonstrate how to combine of experience replay and deep learning
• Demonstrate first-hand the effect of replay on multiple platforms including Games and Robotics domains

Expected Outcomes
• To gain an in-depth understanding of recent developments in DRL and experience replay
• To gain an in-depth understanding of which update rules to adopt, on-policy or off-policy
• To contrast traditional replay with the more recent re-evaluation that has been termed as replay

Target audience and session organisation:

The target audience are researchers and practitioners in the growing reinforcement learning community who are seeking a better understanding of the issues surrounding combining experience replay, deep learning and off-policy learning in their quest for a more practical methods and models.

The tutorial will take 2 hours to be completed and will provide code that can be easily run on Octave or MATLAB.

The two-hour tutorial will be delivered into two integrated sections, the first will cover the theory and the second will cover the application. The presenters will alternate between each other on both the theoretical part and the application part. Two applications will be covered one is the game of packman and the other is a hacked mini robot that will learn to navigate in small 2x1m easy to assemble arena. The robot is a cheap affordable robot, such as Lego, that is equipped with a Raspberry PI module and camera. It will use vision and deep learning combined with experience replay to learn to perform a homing task. The audience will be provided with the Octave/MATLAB code to experience first-hand with the algorithms and see how they are developed form the inside. The code is general enough to be reused for other RL problems. For remote access audiences the code will be shared on Git, so they will be able to experiments with the model directly and a web camera will be mounted on top of the small robot arena to broadcast how the robot will gradually learn to navigate towards it home using vision to learn an optimal path and using infra-red sensors for obstacle avoidance behaviour. For those attending the tutorial they can SSH to the controlling laptop, to which the intensive processing is off-boarded, in order to try and change the Octave code that is driving the robot and see its effect. If a VPN can be setup, then the same the remote audiences can be provided with the same experience.

Previous tutorials:

We have given a successful tutorial on RL and Deep Learning at the IJCNN 2018 on July 2018 in Rio.


Senior Lecturer in Computing Email: Dr Abdulrahman Altahhan has been teaching AI and related topics since 2000, currently he is a Senior Lecturer in Computing as Leeds Beckett University. He served as the Programme Director of MSc in Data Science at Coventry University, UK. Previously, Dr Altahhan worked in Dubai as an Assistant Professor and Acting Dean. He has a PhD in 2008 in Reinforcement Learning and Neural Networks and an MPhil in Fuzzy Expert Systems. Dr Abdulrahman is actively researching in the area of Deep Reinforcement Learning applied to robotics and autonomous agents with publications in this front. He has extensively prepared designed and developed a novel reinforcement learning family of methods and studied their mathematical underlying properties. Recently he established a new set of algorithms and findings where he combined deep learning with reinforcement learning in a unique way that is hoped to contribute to the development of this new research area. He presented in prestigious conferences and venues in the area of machine learning and neural network. Dr Abdulrahman is a reviewer for important Neural Networks related journals, and venues from Springer and the IEEE; including Neural Computing and Applications journal, International Conference of Robotics and Automation ICRA, and he serves in the programme committees for related conferences such as INNS Big Data 2016. Dr Abdulrahman has organised several special sessions in Deep Reinforcement Learning in IJCNN2016 and IJCNN 2017 as well as ICONIP 2016 and 2017 conferences. Dr Abdulrahman is an EPSRC reviewer and taught Machine Learning, Neural Networks and Big Data Analysis modules in the MSc of Data Science, he is an IEEE Member, a member of the IEEE Computational Intelligence Society and International Neural Network Society.


Professor: Pervasive Computing Email: Prof Vasile Palade has joined Coventry University in 2013 as a Reader in Pervasive Computing, after working for many years as a Lecturer in the Department of Computer Science, of the University of Oxford, UK. His research interests lie in the wide area of machine learning, and encompass mainly neural networks and deep learning, neuro-fuzzy systems, various nature inspired algorithms such as swarm optimization algorithms, hybrid intelligent systems, ensemble of classifiers, class imbalance learning. Application areas include image processing, social network data analysis, Bioinformatics problems, fault diagnosis, web usage mining, health, among others. Dr Palade is author and co-author of more than 160 papers in journals and conference proceedings as well as books on computational intelligence and applications (which attracted 4250 citations and an h-index of 29 according to Scholar Google). He has also co-edited several books including conference proceedings. He is an Associate Editor for several reputed journals, such as Knowledge and Information Systems (Elsevier), Neurocomputing (Elsevier), International Journal on Artificial Intelligence Tools (World Scientific), International Journal of Hybrid Intelligent Systems (IOS Press). He has delivered keynote talks to international conferences on machine learning and applications. Prof. Vasile Palade is an IEEE Senior Member and a member of the IEEE Computational Intelligence Society.

References that will be covered

[1] L.-J. Lin, “Self-improving reactive agents based on reinforcement learning, planning and teaching,” Machine Learning, vol. 8, no. 3, pp. 293-321, 1992.
[2] V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, p. 529, 2015
[3] A. Altahhan, “TD(0)-Replay: An Efficient Model-Free Planning with full Replay,” in 2018 International Joint Conference on Neural Networks (IJCNN), 2018, pp. 1-7.
[4] A. Altahhan, “Deep Feature-Action Processing with Mixture of Updates,” in 2015 International Conference on Neural Information Processing, Istanbul, Turkey, 2015, pp. 1-10.
[5] S. Zhang and R. S. Sutton, “A Deeper Look at Experience Replay,” eprint arXiv:1712.01275, 2017
[6] H. Vanseijen and R. Sutton, “A Deeper Look at Planning as Learning from Replay,” presented at the Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research, 2015.
[7] Y. Pan, M. Zaheer, A. White, A. Patterson, and M. White, “Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domains,” eprint arXiv:1806.04624, 2018.
[8] van Hasselt, H. and Sutton, R. S. (2015). Learning to predict independent of span. arXiv:1508.04582.
[9] H. van Seijen, A. Rupam Mahmood, P. M. Pilarski, M. C. Machado, and R. S. Sutton, “True Online Temporal-Difference Learning,” Journal of Machine Learning Research, vol. 17, no. 145, pp. 1-40, 2016.
[10] Sutton, R. S. and Barto, A. G. (2017). Reinforcement Learning: An Introduction. 2nd Edition, Accessed online, MIT Press, Cambridge.
[11] Watkins, C.J.C.H. & Dayan, P., Q-learning, Mach Learn (1992) 8: 279.
[12] J. Modayil, A. White, and R. S. Sutton, “Multi-timescale nexting in a reinforcement learning robot,” Adaptive Behavior, vol. 22, no. 2, pp. 146-160, 2014/04/01 2014.
[13] D. Precup, R. S. Sutton, and S. Dasgupta, “Off-Policy Temporal Difference Learning with Function Approximation,” presented at the Proceedings of the Eighteenth International Conference on Machine Learning, 2001.
[14] R. S. Sutton, A. Rupam Mahmood, and M. White, “An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning,” Journal of Machine Learning Research, vol. 17, pp. 1-29, 2016.
[15] H. Yu, “On Convergence of Emphatic Temporal-Difference Learning,” eprint arXiv:1506.02582, 2015.
[16] A. Hallak, A. Tamar, R. Munos, and S. Mannor, “Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis,” eprint arXiv:1509.05172, 2015.
[17] X. Gu, S. Ghiassian, and R. S. Sutton, “Should All Temporal Difference Learning Use Emphasis?,” eprint arXiv:1903.00194, 2019.
[18] M. P. Deisenroth, G. Neumann, J. Peters, et al., “A survey on policy search for robotics,” Foundations and Trends R in Robotics, vol. 2, no. 1–2, pp. 1–142, 2013.
[19] R. S. Sutton, C. Szepesvari, A. Geramifard, and M. P. Bowling, “Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping,” eprint arXiv:1206.3285, p. arXiv:1206.3285, 2012.

Machine Learning for Data Dtreams in Python with Scikit-Multiflow

Jacob Montiel
University of Waikato

HeitorMurilo Gomes
University of Waikato

Jesse Read
École Polytechnique

Albert Bifet
University of Waikato


Data stream mining has gained a lot of attention in recent years as an exciting researc

h topic. However, there is still a gap between the pure research proposals and the practical applications to real world machine learning problems. In this tutorial we are going to introduce attendees to data stream mining procedures and examples of big data stream mining applications. Besides the theory we will also present examples using the \skmultiflow framework, a novel open source Python framework.

 Tutorial Presenters (names with affiliations): 

Jacob Montiel (University of Waikato), HeitorMurilo Gomes (University of Waikato), Jesse Read (École Polytechnique), Albert Bifet (University of Waikato)

 Tutorial Presenters’ Bios: 

Jacob Montiel

Jacob is a research fellow at the University of Waikato in New Zealand and the lead maintainer of \skmultiflow. His research interests are in the field of machine learning for evolving data streams. Prior to focusing on research, Jacob led development work for onboard software for aircraft and engine’s prognostics at GE Aviation; working in the development of GE’s Brilliant Machines, part of the IoT and GE’s approach to Industrial Big Data.


HeitorMurilo Gomes

Heitor is a senior research fellow at the University of Waikato in New Zealand. His main research area is Machine Learning, specially Evolving Data Streams, Concept Drift, Ensemble methods and Big Data Streams. He is an active contributor to the open data stream mining project MOA and a co-leader of the StreamDM project, a real-time analytics open-source software library built on top of Spark Streaming.


Jesse Read

Jesse is a Professor at the DaSciM team in LIX at École Polytechnique in France. His research interests are in the areas of Artificial Intelligence, Machine Learning, and Data Science and Mining. Jesse is the maintainer of the open-source software MEKA, a multi-label/multi-target extension to Weka.


Albert Bifet

Albert is a Professor at University of Waikato and Télécom Paris. His research focuses on Data Stream mining, Big Data Machine Learning and Artificial Intelligence. Problems he investigate are motivated by large scale data, the Internet of Things (IoT), and Big Data Science.

He co-leads the open source projects MOA (Massive On-line Analysis), Apache SAMOA (Scalable Advanced Massive Online Analysis) and StreamDM.



External website with more information on Tutorial (if applicable): NA

Fast and Deep Neural Networks

 Claudio Gallicchio
University of Pisa (Italy)

Simone Scardapane
La Sapienza University of Rome (Italy)


Deep Neural Networks (DNNs) are a fundamental tool in the modern

development of Machine Learning. Beyond the merits of properly designed training strategies, a great part of DNNs success is undoubtedly due to the inherent properties of their layered architectures, i.e., to the introduced architectural biases. In this tutorial, we analyze how far we can go by relying almost exclusively on these architectural biases. In particular, we explore recent classes of DNN models wherein the majority of connections are randomized or more generally fixed according to some specific heuristic, leading to the development of Fast and Deep Neural Network (FDNN) models. Examples of such systems consist of multi-layered neural network architectures where the connections to the hidden layer(s) are left untrained after initialization. Limiting the training algorithms to operate on a reduced set of weights implies a number of intriguing features. Among them, the extreme efficiency of the resulting learning processes is undoubtedly a striking advantage with respect to fully trained architectures. Besides, despite the involved simplifications, randomized neural systems possess remarkable properties both in practice, achieving state-of-the-art results in multiple domains, and theoretically, allowing to analyze intrinsic properties of neural architectures (e.g. before training of the hidden layers’ connections). In recent years, the study of randomized neural networks has been extended towards deep architectures, opening new research directions to the design of effective yet extremely efficient deep learning models in vectorial as well as in more complex data domains.

This tutorial will cover all the major aspects regarding the design and analysis of Fast and Deep Neural Networks, and some of the key results with respect to their approximation capabilities. In particular, the tutorial will first introduce the fundamentals of randomized neural models in the context of feedforward networks (i.e., Random Vector Functional Link and equivalent models), convolutional filters, and recurrent systems (i.e., Reservoir Computing networks). Then, it will focus specifically on recent results in the domain of deep randomized systems, and their application to structured domains (trees, graphs).

Tutorial Presenters (names with affiliations):

Claudio Gallicchio, University of Pisa (Italy)

Simone Scardapane, La Sapienza University of Rome (Italy)

*Tutorial Presenters’ Bios:

Claudio Gallicchio is Assistant Professor at the Department of Computer Science, University of Pisa. He is Chair of the IEEE CIS Task Force on Reservoir Computing, and member of IEEE CIS Data Mining and Big Data Analytics Technical Committee, and of the IEEE CIS Task Force on Deep Learning. Claudio Gallicchio has organized several events (special sessions and workshops) in major international conferences (including IJCNN/WCCI, ESANN, ICANN) on themes related to Randomized Neural Networks. He serves as member of several program committees of conferences and workshops in Machine Learning and Artificial Intelligence. He has been invited speaker for several national and international conference. His research interests include Machine Learning, Deep Learning, Randomized Neural Networks, Reservoir Computing, Recurrent and Recursive Neural Networks, Graph Neural Networks.

Simone Scardapane is Assistant Professor at the the “Sapienza” University of Rome. He is active as co-organizer of special sessions and special issues on themes related to Randomized Neural Networks and Randomized Machine Learning approaches. His research interests include Machine Learning, Neural Networks, Reservoir Computing and Randomized Neural Networks, Distributed and Semi-supervised Learning, Kernel Methods, and Audio Classification. Simone Scardapane is an Honorary Research Fellow with the CogBID Laboratory, University of Stirling, Stirling, U.K. Simone Scardapane is the co-organizer of the Rome Machine Learning & Data Science Meetup, that organizes monthly events in Rome, and a member of the advisory board for Codemotion Italy. He is also a co-founder of the Italian Association for Machine Learning, a not-for-profit organization with the aim of promoting machine learning concepts in the public. In 2017 he has been certified as a Google Developer expert for machine learning. Currently, he is the track director for the CNR sponsored “Advanced School of AI”


* External website with more information on Tutorial (if applicable):

Mechanisms of Universal Turing Machines: Vision, Audition, Natural Language, APFGP and Consciousness

Juyang Weng


Finite automata (i.e., finite-state machines) have been taught in almost all electrical engineering programs.  However, Turing machines, especially universal Turing machines (UTM), have not been taught in many electrical engineering programs and were dropped in many computer science and engineering programs as a required course.   This resulted in major knowledge weakness in many people working on neural networks and AI since without knowing UTM, researchers have considered neural networks as merely general-purpose function approximators, but not general-purpose computers.   This tutorial first briefly explains what a Turing machine is, what a UTM is, why a UTM is a general-purpose computer, and why Turing machines and UTMs are all symbolic and handcrafted.  In contrast, a Developmental Network (DN) not only is a new kind of neural network, but also can learn to become a general-purpose computer by learning an emergent Turing machine.  It does so by first taking a sequence of instructions as a user provided program and the data for the program to run on, and then running the program on the data.  Therefore, a universal Turing machine inside a DN emerges autonomously on the fly.  It can perform Autonomous Programming For General Purposes (APFGP).  The DN learns UTM transitions one at a time incrementally, without iterations, and refines UTM transitions from the physical experience through network’s lifetime.  Consciousness, whether natural or artificial, requires APFGP.

Presenter Biographies:

Juyang Weng: Professor at the Department of Computer Science and Engineering, the Cognitive Science Program, and the Neuroscience Program, Michigan State University, East Lansing, Michigan, USA. He is also a visiting professor at Fudan University, Shanghai, China. He received his BS degree from Fudan University in 1982, his MS and PhD degrees from University of Illinois at Urbana-Champaign, 1985 and 1989, respectively, all in Computer Science.  From August 2006 to May 2007, he is also a visiting professor at the Department of Brain and Cognitive Science of MIT.   His research interests include computational biology, computational neuroscience, computational developmental psychology, biologically inspired systems, computer vision, audition, touch, behaviors, and intelligent robots.  He has published over 300 research articles on related subjects, including task muddiness, intelligence metrics, mental architectures, emergent Turing machines, autonomous programing for general purposes (APFGP), vision, audition, touch, attention, detection, recognition, autonomous navigation, and natural language understanding.  He, T. S. Huang and N. Ahuja published the first deep learning system for 3D world, called Cresceptron and a research monograph titled Motion and Structure from Image Sequences.  He authored  a book titled Natural and Artificial Intelligence: Computational Introduction to Computational Brain-Mind.  He is an editor-in-chief of the International Journal of Humanoid Robotics and an associate editor of the IEEE Transactions on Autonomous Mental Development. He has chaired and co-chaired some conferences, including the NSF/DARPA funded Workshop on Development and Learning 2000 (1st ICDL), 2nd ICDL (2002), 7th ICDL (2008), 8th ICDL (2009), and INNS NNN 2008. He was the Chairman of the Governing Board of the International Conferences on Development and Learning (ICDLs) (2005-2007), chairman of the Autonomous Mental Development Technical Committee of the IEEE Computational Intelligence Society (2004-2005), an associate editor of IEEE Trans. On Pattern Recognition and Machine Intelligence, an associate editor of IEEE Trans. on Image Processing.  He was the General Chair of AIML Contest 2016 and taught BMI 831, BMI 861 and BMI 871 that prepared the contestants for the AIML Contest session in IJCNN 2017 in Alaska.  He is a Fellow of IEEE.


Venue: With WCCI 2020 being held as a virtual conference, there will be a virtual experience of Glasgow, Scotland accessible through the virtual platform. We hope that everyone will have a chance to visit one of Europe’s most dynamic cultural capitals and the “World’s Friendliest City” soon in the future!.