Deep Network Development

This course, delivered by the University of Trento in Italy, introduces the principles and practices of deep learning and the development of neural networks. Learners cover the fundamentals of neural network architectures, training methods and practical applications, together with the libraries used to build such models, and gain hands-on experience implementing and testing deep networks in PyTorch.

About this course

Deep learning underpins most of the AI systems now moving into production across Europe, from computer vision and speech to language and multimodal models. This course provides a structured introduction to the principles and practices behind those systems and to how neural networks are actually developed. It covers the fundamentals of neural networks (i.e., key architectures, training methods and practical applications) alongside the libraries used to develop such models, with PyTorch as the working environment.

The module moves from the motivation for depth and the mechanics of the artificial neuron through supervised learning, backpropagation, activation functions and optimisers, and then into the practical craft of getting networks to train: training dynamics, regularisation through both data and architecture, and hyperparameter search. Two dedicated PyTorch sessions and a series of optional coding exercises translate the theory into working code. The second half of the course addresses the major architecture families (convolutional networks, recurrent networks and Transformers), the difficulties that arise as networks get deeper, and the learning paradigms beyond supervision, including unsupervised, generative and self-supervised approaches. The closing lectures follow these ideas to their current conclusion: language modelling, the data challenges that come with training at scale, the shift toward foundation models, and the ethical questions that large deep learning systems raise for the people who build and deploy them. Each lecture is paired with a quiz or an exercise, and the material is self-paced throughout.

The course is delivered at advanced level and assumes working knowledge of Python and a basic grounding in linear algebra, calculus and probability. By the end, participants should be able to implement and test a deep neural network on a problem of their own choosing.

Course structure

• Foundations: why deep networks, artificial neural networks, supervised learning, backpropagation.
• Training mechanics: activation functions, optimisers, training dynamics.
• Generalisation: regularisation through data, regularisation through architecture, hyperparameter search.
• Practice in PyTorch: introduction to the framework in two parts, from first tensors to a working digit classifier, with optional exercises on convolutional models, sequence tagging and gradient monitoring.
• Architectures: convolutional neural networks, recurrent neural networks, issues when going deeper, Transformers.
• Beyond supervision: unsupervised learning, generative modelling, self-supervised learning.
• Scale and responsibility: language modelling, data challenges, the move toward foundation models, and the ethical challenges raised by large-scale deep learning.

The course comprises 24 lessons, 5 practical topics and 22 quizzes, and carries a course certificate on completion.

Learning outcomes

Upon successful completion of this course, students will be able to:

• Explain the principles of deep learning and the reasons depth improves representation and generalisation.
• Describe how neural networks are trained, including backpropagation, loss functions, activation functions and optimisation algorithms.
• Diagnose and address common training problems such as overfitting, vanishing and exploding gradients, and poor convergence.
• Apply regularisation strategies and conduct systematic hyperparameter search.
• Select appropriate architectures (convolutional, recurrent or attention-based) for a given data modality and task.
• Implement, train and evaluate a deep neural network in PyTorch on a problem of choice.
• Distinguish between supervised, unsupervised, generative and self-supervised learning paradigms and identify where each applies.
• Explain how language models are built and trained, and how they relate to the wider development of foundation models.
• Recognise the data challenges involved in training at scale, and assess the ethical implications of developing and deploying deep learning systems.

Further details

This course is developed within the framework of the EMAI4EU project, with the support of the Digital Europe Programme of the European Union under grant agreement no. 101123289. More information on the course is available on the corresponding website.