What is FD4?
FD4 has been originally developed for the parallelization of spectral bin cloud models and their coupling to atmospheric models. But FD4 is not restricted to this kind of application and can be used in other areas which benefit from FD4's features:
- Dynamic load balancing: FD4 provides dynamic load balancing of a three-dimensional block-decomposed regular grid using either the Hilbert space-filling curve or graph partitioning based on ParMETIS.
- Model coupling: The simulation model based on FD4 can be coupled to external models. An external model could be, for example, any CFD model or weather model.
- Adaptive block mode: If data and computations are only required for a specific spatial subset of the domain, FD4 saves memory by allocating only required blocks. Combined with model coupling, this feature is particularly suitable for many multiphase problems like drops, clouds, combustion processes, and flame fronts.
- 4th dimension: In addition to the three spatial dimensions a variable may have an extra dimension which is not decomposed. It may act, for example, as array of gas phase tracers or as a size spectrum for size-resolving particle models. FD4 is optimized to manage a large number of values per grid cell.
Here's a list of key words that apply to FD4: High performance computing, multiphase modeling, multiphysics, high scalability, Fortran95, MPI-2, model coupling, dynamic data structures, dynamic load balancing, Hilbert space-filling curve, ParMETIS, Vis5D, NetCDF
The development of FD4 has been funded by the DFG with the project Parallel coupling framework and advanced time integration methods for detailed cloud processes in atmospheric models (more info), a cooperation of Leibniz Institute for Tropospheric Research and ZIH, TU Dresden.
FD4 Videos
Dynamic Load Balancing of Detailed Cloud Simulations with Visualization of Hilbert SFC Partitioning and Vampir Performance Analysis.
Comparison of Graph Partitioning and SFC Partitioning for Dynamic Load Balancing of Detailed Cloud Simulations.
Vampir Message Statistics of a Hilbert SFC partitioning benchmark with massively changing communication pattern.
Manual
The FD4 manual is available as html and pdf. The manual contains:
- Introduction to the FD4 data structure and parallelization
- Installation instructions
- FD4 tutorial
FD4 Library Hands-on Training
The best way to learn FD4 is to use the material from the hands-on training at ICTP on November 10 2014.
Download
FD4 is free software. It is available under the GNU GPLv3 license.
Copyright 2010-2016 Matthias Lieber, ZIH, TU Dresden, Germany
Download:
fd4-2016-06-09.tar.gz
Older FD4 versions can be downloaded
here.
Additional input files for 1D partitioning benchmark are
here.
If you are using FD4, it would be nice if you drop me a line.
Reproducing measurements of FD4's scalable high-quality 1D partitioning
1D partitioning is the core problem for SFC-based partitioners. FD4 comes with a hierarchical 1D partitioning algorithm that is scalable like simple heuristics but also achieves nearly the optimal load balance like (serial) exact methods. FD4's algorithm runs ca. 270 times faster compared to an exact method. The algorithm is described in the FGCS paper and in much more detail in my PhD thesis.
You can compare the methods on your system with the 1D partitioning benchmark contained in the FD4 package. See README.partbench on how to run the benchmark. Additionally you need to download the input files.
New in Nov 2016: Comparison to Zoltan and new benchmark data set with ca. 10x number of tasks. This dataset contains instructions to compile and run the 1D partitioning benchmark partbench_zz. This benchmark also runs Zoltan as FD4's partitioner to enable comparing FD4's own methods to other methods.
FD4 Publications
- M. Lieber and W.E. Nagel, Highly scalable SFC-based dynamic load balancing and its application to atmospheric modeling, Future Generation Computer Systems, vol. 82, pp. 575-590, 2018 (doi:10.1016/j.future.2017.04.042, post-print version)
- M. Lieber and W.E. Nagel, Scalable High-Quality 1D Partitioning, in Proceedings of the 2014 International Conference on High Performance Computing and Simulation (HPCS 2014), pp. 112-119, 2014, Outstanding Paper Award (doi:10.1109/HPCSim.2014.6903676, full paper, presentation)
- M. Lieber, W.E. Nagel, and H. Mix, Scalability Tuning of the Load Balancing and Coupling Framework FD4, in "NIC Symposium 2014", NIC Series, Vol. 47, pp. 363-370, 2014 (full paper, full proceedings)
- M. Lieber, Dynamische Lastbalancierung und Modellkopplung zur hochskalierbaren Simulation von Wolkenprozessen, PhD thesis, Technische Universität Dresden, 2012 (abstract and full text)
- M. Lieber, V. Grützun, R. Wolke, M.S. Müller, and W.E. Nagel, Highly Scalable Dynamic Load Balancing in the Atmospheric Modeling System COSMO-SPECS+FD4, in Proc. of PARA 2010, "Applied Parallel and Scientific Computing", LNCS Vol. 7133, pp. 131-141, 2012 (doi:10.1007/978-3-642-28151-8_13, full paper, presentation, The original publication is available at www.springerlink.com)
- M. Lieber, V. Grützun, R. Wolke, M.S. Müller, and W.E. Nagel, FD4: A Framework for Highly Scalable Load Balancing and Coupling of Multiphase Models, in Proc. of ICNAAM 2010, AIP Conference Proceedings 1281(1), 1639-1642, 2010 (doi:10.1063/1.3498143, full paper, presentation)
- M. Lieber, R. Wolke, V. Grützun, M.S. Müller, and W.E. Nagel, A framework for detailed multiphase cloud modeling on HPC systems, in Proc. of ParCo09, "Advances in Parallel Computing", Vol. 19, pp. 281-288, 2010 (doi:10.3233/978-1-60750-530-3-281, full paper, presentation)
FD4 Talks
- Scalable Dynamic Load Balancing of Detailed Cloud Physics with FD4, June 01 2015, Talk at the Minisymposium on Advances in Numerics and Physical Modeling for Geophysical Fluid Dynamics at PASC15, ETH Zürich (slides)
- Dynamic Load Balancing of High Performance Computing Applications, November 25 2014, Talk at the Echtzeit-AG at TU Dresden (slides)
- FD4: A Framework for Highly Scalable Dynamic Load Balancing and Model Coupling, November 13 2014, Talk at the Symposium on HPC and Data-Intensive Applications in Earth Sciences, ICTP, Trieste (slides)
- Introduction to the usage and application of the FD4 library, November 10 2014, Talk at the Developer School for HPC applications in Earth Sciences, ICTP, Trieste (slides)
- High performance computing methods for spectral cloud microphysics in the COSMO model, October 11 2012, Talk at Leibniz Institute for Tropospheric Research (IfT), Leipzig (slides)
- HPC Methods for Coupling Spectral Cloud Microphysics with the COSMO Model, March 17 2011, Talk at Max Planck Institute for Meteorology, Hamburg (slides)
- FD4: A Framework for Highly Scalable Load Balancing and Coupling of Multiphase Models, December 6 2010, ZIH HPC User Forum 2010 (slides)
FD4 Poster
|
Oct 2009
|
Nov 2009
|
Jan 2011
|
Funding / Related Projects
Contact
- Feel free to contact matthias.lieber@tu-dresden.de for any comments, questions, suggestions etc. regarding FD4.
- My homepage at ZIH, TU Dresden