Parallel Programming With MPI

Pacheco's book is a strong, gently paced introduction to a very complex API. MPI, the message passing interface, is the most common coordination tool for parallel scientific computing. When a Blue Gene programmer has 1,000 or 100,000 processors all working on different parts of one calculation, there's a big problem in getting partial results from where they're computed to where they're needed. That's what MPI is for.

When the problem is so complicated, the solution is also complicated. Pacheco does a good job of breaking MPI down into digestible pieces, starting with the basic send and receive primitives that you'll use most often. He presents each new part of the API in terms of some problem to be solved, keeping a concrete and practical tone to this book. He gradually adds more pieces in terms of more practical exercises: broadcasts and reductions, scatter and gather, data structuring, communicators, and asynchronous IO.

Along the way, Pacheco introduces algorithms that even experienced uniprocessor programmers may not be familiar with, including bitonic sorting and Fox's algorithm for matrix multiplication. This isn't gratuitous intellectual showmanship. It's a pointed demonstration that, when communication barriers change the computation landscape, old paths to solutions may not be the best routes any more. After finishing with the MPI API itself, Pacheco presents debugging techniques and common kinds of support libraries, as well as basic techniques for analyzing the potential and actual acceleration possible for a given problem.

If you're serious about MPI, you'll need the official standard for understanding the fussy details of these complex APIs. That's a pretty brutal way for a beginner to get going, though. This introduces not only the basic concepts of MPI, but also the basics of how to think about highly parallel programming. And, as multi-threaded multi-core multi-processor systems become common, that's an increasing percentage of all programming.

//wiredweird

I read this book over the past week, covering chapters 1 through 6, skimming 7-10, and reading 11 through the final chapter 16. Its basically applied MPI programming, done up very well and clearly, starting with architectural history & motivations and leading into a simple numerical integrator example program in chapter 4 (chapter 3 was the MPI `hello, world'). The coding used is C, and I wrote my own integrator after finishing chapter 4 to also explore floating point numbers in calculations, loop control, and to integrate arbitrary functions on arbitrary intervals with adjustable resolution. The integrator is developed more fully throughout the book wherein MPI performance issues of the original design are pointed out and polished off as additional MPI functions and techniques are introduced. Some of these techniques included tree-structured initialization & broadcasts, data communication optimizations (such as derived types, packing / unpacking, etc), and guidance as to when certain techniques would be more useful than others offered by MPI. Communications are further advanced later in the book where the important non-blocking forms and more advanced concepts are brought to light & illustrated. Empirical analysis of algorithmic performance occupies two full chapters and is very interesting, including a detailed look into Amdahl's law. Its an excellent example of why we should keeps our eyes open in research. Program design & troubleshooting are also covered, but I only skimmed those chapters. Several parallel algorithms and some parallel libraries are also treated well in the text. At only 362 pages, (minus appendices) this book is a quick read and a superb lab manual. If you are a software developer just now getting into MPI, this book will certainly accelerate you onto MPI with the confidence that you can do anything with it. Just give it a week of your time. Its the perfect self-study manual.
5-stars

I was looking for a good introduction into MPI to parallelize some software I had written. Somehow, no online resources seemed to cover the topic well, so $30 seemed like a worthwhile investment, compared to my time. I got this book and the more recent one by Quinn (PP in C with MPI and OpenMP). This one's the hands-down winner. I basically scanned it in bed for three nights, and two weeks later my code is running like a charm. Just the right mix of reference and tutorial, very little distraction, and a pleasant read throughout.

Highly recommended.

The book is written very well and goes through how the MPI functions work and all their parameters in pretty fine detail. He even talks about the simpler things many books overlook. The only thing I wish was in this book, and this has nothing to do with the quality of the book, is some C++ reference or talk of the C++ MPI calls. This book is written with examples in C only (I think FORTRAN one may be available online), but the theory he teaches and the design of the programs will work for any language, of course. It would be nice to have a good reference chapter that lists the FORTRAN MPI functions and the C++ ones too though.

This book is a great introduction not only to MPI but to parallel programming itself. Initially, one learns essential concepts of parallel programming, including hardware and network topologies. An MPI version of ``Hello World!'' gently introduces the reader to the mechanics of running MPI programs (if you haven't set up your parallel computing environment yet, this may well be the most time consuming chapter of the book!) In what follows, the author lays out the basics of each topic in parallel programming, followed by MPI-specific details (i.e. which MPI function is used in that situation). Particularly worthy of praise is the example-driven approach, whereby in each chapter a problem is presented (like numerical integration or matrix multiplication), and the author shows how to solve them with techniques taught in that chapter. Some problems are rehashed in other sections/chapters, in order to show different PP techniques to solve them. The book doesn't suffer from the black-box syndrome of hiding implementation details: the reader is usually shown the inner workings of, say, each communication technique, while being advised to use MPI's built-in functions instead of rolling out his own versions.

Topics covered include collective communication methods, I/O, debugging, design, performance, parallel algorithms and libraries, and advanced topics in communication. A handy appendix on MPI functions wraps up the book.

As an aside, I will be forever grateful to this book due to its clear exposition of the butterfly broadcast method, which helped me understand the Cooley-Tukey iterative version of the FFT (which uses the same butterfly idea), something I had been struggling with for a couple of years. I would have gladly bought the book for this explanation alone.