 | In the lower left corner, the low number of tasks and small input size makes tightly-coupled Message Passing Interface (MPI) quite manageable. This is the traditional terrain of HPC.
As the data size increases (vertically), we move into the analytics category, such as data mining and analysis.
In the lower right corner, data size remains modest, but the increasing number of tasks moves us into loosely-coupled applications involving many tasks. HTC can be considered a subset of this category.
Finally, the combination of both many tasks and large datasets in the upper right corner moves us into the province of Many-Task Computing. MTC can also be considered as part of the high-task, low data (lower right) area.
Image courtesy of Ioan Raicu, University of Chicago. |
Tightly-coupled applications for which jobs must communicate between each other during execution are typically best served by clustered High Performance Computing (HPC). Applications with many independent job streams, on the other hand, are better suited to distributed High Throughput Computing (HTC). But there are still other kinds of applications.
Over the past half decade we’ve examined many applications from astrophysics, bioinformatics, data mining and other fields, and have found that high-performance computations comprising multiple distinct activities and coupled via file system operations (as opposed to the standard message passing interface commonly found in HPC) don’t fit nicely in either category. To address this, we’ve defined the concept of “Many Task Computing”. We believe that it bridges a gap between these two dominant computing paradigms and opens up opportunities to apply HPC systems in new ways for increasingly complex applications that were simply intractable just a few years ago. Millions to billions of tasks
Many Task Computing (MTC) involves applications with tasks that may be small or large, single or multiprocessor, compute-intensive or data-intensive. The set of tasks may be static or dynamic, homogeneous or heterogeneous, and loosely- or tightly-coupled. Applications may span millions to billions of tasks, entail tens of thousands of processor years, incorporate a degree of parallelism able to occupy the largest supercomputers at hundreds of thousands of processors, and operate on terabyte- to petabyte-size datasets. Resource-, communication- and data-intensive applications MTC differs from HTC in the timescale of task completion, and the often data-intensive nature of applications. It emphasizes the use of many resources over short periods of time to accomplish many computational tasks, both dependent and independent, with primary metrics measured in seconds (e.g., FLOPS, tasks/sec, megabytes/s, I/O rates), rather than operations (e.g., jobs per month). | 
Headlines | Site Map
