Newer
Older
`hotwheels` is a codebase for hydrodynamic N-body cosmological simulations, developed by **Antonio Ragagnin**.
As part of my commitment to advancing simulation tools, I am working on a flexible and modular implementation of a [Gadget](https://wwwmpa.mpa-garching.mpg.de/gadget/)-like code (temporarily named `hotwheels`). This new implementation incorporates lessons learned from over a decade of experience working with HPC and Gadget codes (e.g., OpenGadget3, see Dolag et al., in prep).
`hotwheels` is designed to leverage CPU and GPU parallelism, with a strong emphasis on modularity. This modular approach facilitates collaboration with HPC engineers and GPU vendors, enabling efficient utilization of HPC systems. Modularity also enhances testability, simplifies the addition of new features, and allows rethinking code structure based on past experiences.
### Key Features
- **Modularity:** Each module is self-contained, adhering to the [Open-Closed Principle](https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle). `hotwheels` consists of C libraries wrapped in Python for high-level processing, following the successful paradigm of machine learning frameworks ([reference](https://www.mdpi.com/2078-2489/11/4/193)). Developers can focus solely on their module’s files without altering other parts of the code.
- **Mini-Apps:** From over 10 years of experience collaborating with HPC engineers and GPU vendors, I found it crucial to deploy the code as minimal, standalone applications. This approach allows engineers to focus on improving code quality and porting it to different architectures without being distracted by unrelated components.
- **Testability:** A robust testing framework is a priority in `hotwheels`. Each module is isolated and can be independently tested, leveraging the modular structure to facilitate efficient debugging and verification.
- **Parallelism:** Recognizing the dominance of **GPUs** in HPC, especially driven by AI advancements, `hotwheels` is designed from scratch to run efficiently on GPUs, multi-core CPUs, and MPI-parallel systems.
- **Documentation:** Navigation through a large codebase can be challenging. `hotwheels` prioritizes comprehensive documentation, including high-level overviews and detailed API references for each component. Documentation is generated automatically, and commits to the `main` branch are rejected if they fail the indentation and documentation pipelines.
## Modules
- [core](https://www.ict.inaf.it/gitlab/hotwheels/core): Provides essential utilities for: (1) Compiling and executing C code from Python (`core.make`); (2) Generating and managing particle data structures as structures of arrays (SoAs) (`core.soa`); (3) General C utilities (e.g., MPI patterns, MyMalloc, panic handling, and timers in `core.c_utils`).
- [IO](https://www.ict.inaf.it/gitlab/hotwheels/io): Handles parallel reading and writing of Gadget1-2 snapshots (thus including [N-GenIC](https://www.h-its.org/2014/11/05/ngenic-code/) and [MUSIC](https://www-n.oca.eu/ohahn/MUSIC/) formats), as well as HDF5 files.
- [tree](https://www.ict.inaf.it/gitlab/hotwheels/tree): A standalone octree implementation designed for GPU compatibility. It supports Hilbert or Morton ordering, multi-particle leafs, and can be used in standalone C projects.
- [domain](https://www.ict.inaf.it/gitlab/hotwheels/domain): A GPU-friendly domain decomposition module based on Hilbert particle ordering.
- [PM](https://www.ict.inaf.it/gitlab/hotwheels/pm): A standalone particle-mesh (PM) module, heavily inspired by the Gadget PM implementation.
- [integrate](https://www.ict.inaf.it/gitlab/hotwheels/integrate): Provides APIs and callbacks for cosmological and non-cosmological integration. This module also manages cosmological parameters.
- [barnes_hut](https://www.ict.inaf.it/gitlab/hotwheels/barnes_hut): **[To-Do]** A Gadget-like short-range gravity and potential computation module, based on the [Barnes & Hut (1986)](https://www.nature.com/articles/324446a0) algorithm.
- [FoF](https://www.ict.inaf.it/gitlab/hotwheels/fof): **[To-Do]** a simple stand-alone C friend-of-friend (FoF) halo identification.
- [PESPH](https://www.ict.inaf.it/gitlab/hotwheels/PESPH): **[To-Do]** A pressure-entropy smoothed particle hydrodynamics (SPH) module.
- [SH03](https://www.ict.inaf.it/gitlab/hotwheels/SH03): **[To-Do]** A star formation recipe based on [Springel & Hernquist (2003)](https://ui.adsabs.harvard.edu/abs/2003MNRAS.339..289S/abstract), with cooling processes handled by [Grackle](https://grackle.readthedocs.io/en/latest/).
## Run Your Own Simulations
Currently it is possible to run particle-mesh gravity only simulations with and without comoving integrations:
- [A refined PM DMO run with fixed timesteps](run_pm_dmo_NFW_fixed_timestep.md)
- **soon:** A cosmological PM DMO run with fixed timesteps
## The roadmap (to do list)
Milestone 1: **cosmological DMO PM sim w fixed timesteps**
- [x] una domain decomposition migliore
- [x] pensare a SoAs
- [x] PM
- [x] fixed timestep
- [x] PM + fixed timestep
- [x] cosmology integration
- [x] PM + fixed timestep + cosmology
- [x] refactor: pass struct per puntatore
- [x] clean PM code
- [ ] scaling test PM da solo
- [ ] dividere in sotto progetti
- [ ] rilasciare la versione 0.0.1 dell'infrastruttura base
Milestone 3: **cosmological DMO TreePM w adaptive timetep**
- [ ] rilasciare modulo PM
- [ ] PM on GPU, (cuFFTW?)
Milestone 4: **cosmological DMO TreePM w adaptive timetep**
- [ ] openmpizzare per bene
- [ ] openmp target per bene
- [ ] scaling tests su GPU
- [ ] confronto performance con Gadget (su CPU e GPU)
- [ ] trovare dove testare openmp target
- [x] caricare da snapshot
- [ ] salvare snapshot gadget
- [ ] scaling test DMO con e senza GPU
Milestone 6: **hotwheels full physics run**
- [ ] read_gadget.py per + tipi
- [ ] SPH base
- [ ] SPH pressure-entropy in mod. separato
- [ ] SPH standard tests
- [ ] scaling test no-rad con e senza GPU
- [ ] SH03
- [ ] scaling test SH03 con e senza GPU
- [ ] FoF
## Licence
The code has no licence (yet). Most of the code is written from scratch by Antonio Ragagnin and re-distribution of said code requires authorisation. It also contains parts copy-pasted from GPL codes (see comments in the code) which can be re-distributed following their licence: Gadget code ([Springel 2001](https://ui.adsabs.harvard.edu/abs/2001NewA....6...79S/abstract)), Gadget4 ([Springel et al. 2021](https://ui.adsabs.harvard.edu/abs/2021MNRAS.506.2871S/abstract)), Blue Tides ([Rupert et al. 2015](https://ui.adsabs.harvard.edu/abs/2015CSE....17b..40C/abstract)).
## Author
The current developer is [Antonio Ragagnin](https://aragagnin.github.io/)