`hotwheels` is a code for hydrodynamic N-body cosmological simulations, `hotwheels` components are made of **C low-level kernels driven by python wrappers**. `hotwheels` is developed by **Antonio Ragagnin**. And, as part of my commitment to advancing simulation tools, I am working on a flexible and modular implementation of a [Gadget](https://wwwmpa.mpa-garching.mpg.de/gadget/)-like code (temporarily named `hotwheels`). This new implementation incorporates lessons learned from over a decade of experience working with HPC and Gadget codes, for instance see my greentree in [Ragagnin et al. (2017)](https://ebooks.iospress.nl/publication/42679), my GPU scheme for N-body codes as in [Ragagnin et al. (2020)](https://ebooks.iospress.nl/publication/53922) (and Ragagnin et al., 2025, in review), my parallel tree build tests (fund by HPC Europa3, ID HPC17YMAKH) in the context of OpenGadget3 (PI. Klaus Dolag). `hotwheels` is designed to leverage CPU and GPU parallelism, with a strong emphasis on modularity. This modular approach facilitates collaboration with HPC engineers and GPU vendors, enabling efficient utilization of HPC systems. Modularity also enhances testability, simplifies the addition of new features, and allows rethinking code structure based on past experiences. Check the here the [TODO list](TODO). # Key Features - **Modularity:** Each module is self-contained, adhering to the [Open-Closed Principle](https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle). `hotwheels` consists of C libraries wrapped in Python for high-level processing, following the successful paradigm of machine learning frameworks ([reference](https://www.mdpi.com/2078-2489/11/4/193)). Developers can focus solely on their module’s files without altering other parts of the code. - **Mini-Apps:** From over 10 years of experience collaborating with HPC engineers and GPU vendors, I found it crucial to deploy the code as minimal, standalone applications. This approach allows engineers to focus on improving code quality and porting it to different architectures without being distracted by unrelated components. See [Cavelan et al.](https://dl.acm.org/doi/10.1145/3394277.3401855) (2020, [SPH-EXA](https://github.com/sphexa-org/sphexa?tab=readme-ov-file)) for external interesting past cases of Mini-App usages. - **Testability:** A robust testing framework is a priority in `hotwheels`. Each module is isolated and can be independently tested, leveraging the modular structure to facilitate efficient debugging and verification. - **Parallelism:** Recognizing the dominance of **GPUs** in HPC, especially driven by AI advancements, `hotwheels` is designed from scratch to run efficiently on GPUs, multi-core CPUs, and MPI-parallel systems. - **Documentation:** Navigation through a large codebase can be challenging. `hotwheels` prioritizes comprehensive documentation, including high-level overviews and detailed API references for each component. Documentation is generated automatically, and commits to the `main` branch are rejected if they fail the indentation and documentation pipelines. # Modules **Access to modules will be granted on request.** **Email: ragagnin.antonio@gmail.com, antonio.ragagnin@inaf.it** - [easyc](https://www.ict.inaf.it/gitlab/hotwheels/easyc): **[Private repo]** A standalone hotwheels-unrelated package for compiling and executing C code from Python, and easily passing of numpy arrays as C arrays, enums, typedefs, arbitrarely nested structures, in particular structures or array and array of structures, and for dynamically generating C headers for structures of arrays (SoAs). - [utils](https://www.ict.inaf.it/gitlab/hotwheels/utils): **[Private repo]** Provides essential C utilities for MPI patterns, Gadget-like MyMalloc, panic, timer - [IO](https://www.ict.inaf.it/gitlab/hotwheels/io): **[Private repo]** Handles parallel reading and writing of Gadget1-2 snapshots (thus including [N-GenIC](https://www.h-its.org/2014/11/05/ngenic-code/) and [MUSIC](https://www-n.oca.eu/ohahn/MUSIC/) formats), as well as HDF5 files. - [octree](https://www.ict.inaf.it/gitlab/hotwheels/octree): **[Private repo]** A standalone octree implementation designed for GPU compatibility. It supports Hilbert or Morton ordering, multi-particle leafs, and can be used in standalone C projects. - [domain](https://www.ict.inaf.it/gitlab/hotwheels/domain): **[Private repo]** A GPU-friendly domain decomposition module based on Hilbert particle ordering. - [PM](https://www.ict.inaf.it/gitlab/hotwheels/PM): **[Private repo]** A standalone particle-mesh (PM) module, heavily inspired by the Gadget PM implementation. - [integrate](https://www.ict.inaf.it/gitlab/hotwheels/integrate): **[Private repo]** Provides APIs and callbacks for cosmological and non-cosmological integration. This module also manages cosmological parameters. - [barnes_hut](https://www.ict.inaf.it/gitlab/hotwheels/barnes_hut): **[To-Do]** A Gadget-like short-range gravity and potential computation module, based on the [Barnes & Hut (1986)](https://www.nature.com/articles/324446a0) algorithm. - [FoF](https://www.ict.inaf.it/gitlab/hotwheels/fof): **[To-Do]** a simple stand-alone C friend-of-friend (FoF) halo identification. - [PESPH](https://www.ict.inaf.it/gitlab/hotwheels/PESPH): **[To-Do]** A pressure-entropy smoothed particle hydrodynamics (SPH) module. - [SH03](https://www.ict.inaf.it/gitlab/hotwheels/SH03): **[To-Do]** A star formation recipe based on [Springel & Hernquist (2003)](https://ui.adsabs.harvard.edu/abs/2003MNRAS.339..289S/abstract), with cooling processes handled by [Grackle](https://grackle.readthedocs.io/en/latest/). # Tutorials - [Install and test](install_and_test.md) - [Google collab: hotwheels C wrapper](https://colab.research.google.com/drive/1CjmosUB0a2A3H1YdMe5vNVrFIUSRy8Ow) - [Google collab: use the octree with multiple particles per leaf](https://colab.research.google.com/drive/1EObmEt7XK56EpwHh7PQ-viR7PHZ5H0pl) - [Google collab: run the octree build on GPU](https://colab.research.google.com/drive/1GmlFxnou2geo_pRzeOMebVVKAIhPZbM9) - [GPU vs CPU octree build on Leonardo BOOSTER](test_gpu_treebuild_on_leonardo.md) - [A PM DMO run with arbitrary refinement levels and fixed timesteps](run_pm_dmo_NFW_fixed_timestep.md) - **soon:** A cosmological PM DMO run with fixed timesteps # Licence This code is currently unlicensed. The majority of the code has been written from scratch by Antonio Ragagnin, and redistribution of this code requires explicit authorization from the author. Certain portions of the code are adapted from GPL-licensed software, and their respective licenses apply to those parts (see code comments). Specifically, this includes: Gadget code ([Springel 2001](https://ui.adsabs.harvard.edu/abs/2001NewA....6...79S/abstract)), Gadget4 ([Springel et al. 2021](https://ui.adsabs.harvard.edu/abs/2021MNRAS.506.2871S/abstract)), and Blue Tides ([Rupert et al. 2015](https://ui.adsabs.harvard.edu/abs/2015CSE....17b..40C/abstract)).