Course Description

This course introduces you to some fundamental concepts of parallel and distributed computing, from parallel algorithmic models to the development of parallel and distributed programs intended for HPC systems. Starting with the PRAM (Parallel Random Access Machine) model, you will understand the potential and limits of parallelism to solve computationally complex problems, and learn how to design parallel algorithms assuming an idealized parallel computing platform. Then, you will shift to more realistic architectures, and study how to develop parallel programs for both shared-memory architectures using OpenMP annotations and distributed-memory architectures with the MPI library.
By examining both theoretical frameworks and implementation paradigms, you will gain insights into how to significantly increase performance across diverse computational environments, from algorithmic design to actual distributed deployment. Designed for professionals with foundational knowledge in computing, this course builds on intermediate-level expertise to explore advanced techniques in parallel and distributed high-performance computing.

Prerequisites

Foundations of algorithms and complexity
Notions of parallelism and concurrency in computing

Course Learning Outcomes

After successfully finishing the course, you will be able to:

Understand the capabilities and constraints of parallel computing
Design efficient parallel algorithms to solve common computationnally heavy problems
Adapt existing sequential programs into parallel implementations (using Open MP)
Develop programs intended for distributed platforms, specifically through message-passing techniques (using MPI)

Time Commitment & Schedule

This course is self-paced, allowing you to progress through the material at a speed that suits your professional and personal commitments. The estimated time required to complete all content and activities is approximately 25 hours. This includes engaging with lectures, participating in hands-on labs, and completing assessments.

Course Outline

Part I – PRAM: Parallel Programming in an Ideal World (10 hours)
Part II – Open MP: Parallelizing Sequential Code (6 hours)
Part III – Message Passing Interface (MPI): High-Performance Distributed Computing (8 hours)

Textbook and Reference Materials

H. Casanova, A. Legrand and Y. Robert. Parallel Algorithms. Chapman and Hall/CRC Press, 2008.

I. Reif (editor). Synthesis of Parallel Algorithms. Morgan-Kaufmann,1993.

Course Conduct

Initial Assessment: The course starts with a brief knowledge check to confirm prerequisites.
Interactive Learning: Short questions are integrated into each lecture to reinforce understanding.
Chapter Reinforcement: Each chapter concludes with a quiz, assessing both theoretical and practical knowledge.
Hands-On Practice: Practical sessions are included within each chapter to apply learned concepts. Detailed solution walkthroughs are provided.
Automated Feedback: Coding assignments are graded automatically, offering immediate feedback.