MPI Point-to-Point Refactoring + New Periodic BC Implementation by economon · Pull Request #652 · su2code/SU2

economon · 2019-02-15T20:58:35Z

Proposed Changes

Give a brief overview of your contribution here in a few sentences.

This is one for v7 and needs some testing. This PR contains a complete refactoring of the point-to-point MPI infrastructure and an entirely new implementation of the periodic BC for the FV solvers.

For the MPI refactoring:

All calls are abstracted to parent classes (geometry, solver, matrix), meaning we will remove all Set_MPI_* routines.
Point-to-point comms are all non-blocking calls that have been separated into InitiateComms() & CompleteComms(), so that, in theory, they can be overlapped with computation where possible. It would also be possible now to group the comms into larger messages easily. These optimizations are a next step (not for this PR).
Persistent communication schedules and memory are used throughout the calculation.
Results are the same, this is just a refactoring to improve performance, scalability, etc. while making the code much simpler and separating the periodic part.

For the periodic BC:

No longer need a preprocessing with SU2_MSH - use the original grid and just call SU2_CFD.
Adjacent periodic faces are now possible, e.g., triply-periodic cube.
Concept is now based on completing the residuals from partial control volumes on either side of a periodic face instead of creating extra halo cells.
All periodic comms have been separated from the MPI point-to-point comms and are performed first for consistency.

In a follow-up PR, I will remove many lines of code that are no longer needed (all Set_MPI_* routines, many outdated geometry routines, etc.)

Related Work

Resolve any issues (bug fix or feature request), note any related PRs, or mention interactions with the work of others, if any.

I could use some help with testing this one, in particular the rotational periodicity, which might still need some changes. Unfortunately, I do not have the original meshes for the periodic cases in the repo that are failing (other than Poiseuille)..

@salvovitale @LaSerpe @arubino: do you have the original grids for the periodic cases before adding the halos with SU2_MSH? I would like to make sure that everything is working ok for all previous capability.

All periodic meshes in the regressions have been updated for the new periodic BC, and a python script has been added to remove halo cells from old meshes that were preprocessed with SU2_MSH for the previous periodic BC implementation.

Resolves #550
Resolves #431
Resolves #416
Resolves #252

PR Checklist

Put an X by all that apply. You can fill this out after submitting the PR. If you have any questions, don't hesitate to ask! We want to help. These are a guide for you to know what the reviewers will be looking for in your contribution.

I am submitting my contribution to the develop branch.
My contribution generates no new compiler warnings (try with the '-Wall -Wextra -Wno-unused-parameter -Wno-empty-body' compiler flags).
My contribution is commented and consistent with SU2 style.
I have added a test case that demonstrates my contribution, if necessary.

…be generalized to all solvers and parallelization should be improved.

…sors, and eigenvalues. Both Euler and N-S operational. Fixed multiple bugs.

economon · 2019-04-24T01:26:23Z

Thanks for the input, everyone. Time to finally merge this one.

@pcarruscag : let's finish up #653 as a next step and get that merged asap. I can help fix up any conflicts in the matrix classes.

TobiKattmann

A bit late to the party, I guess...
I thought i better submit that now instead of not submitting at all.
Most of it is quite pedantic.

TobiKattmann · 2019-03-21T14:26:52Z

  Sens_Remove_Sharp,			/*!< \brief Flag for removing or not the sharp edges from the sensitivity computation. */
  Hold_GridFixed,	/*!< \brief Flag hold fixed some part of the mesh during the deformation. */
  Axisymmetric, /*!< \brief Flag for axisymmetric calculations */
+  TaylorGreen,


Description missing

TobiKattmann · 2019-03-21T14:27:37Z

   */
  bool GetAxisymmetric(void);

+    bool GetTaylorGreen(void);


Description missing + indentation

TobiKattmann · 2019-03-21T14:31:22Z

  Boundary,                           /*!< \brief To see if a point belong to the boundary (including MPI). */
  PhysicalBoundary,                   /*!< \brief To see if a point belong to the physical boundary (without includin MPI). */
-  SolidBoundary;                      /*!< \brief To see if a point belong to the physical boundary (without includin MPI). */
+  SolidBoundary,                      /*!< \brief To see if a point belong to the physical boundary (without includin MPI). */


Copied description.
including instead includin.
Difference between solid and physical not clear to me.

TobiKattmann · 2019-03-21T14:43:56Z

 	 */
 	virtual ~CGeometry(void);
+
+  /*!


2x brief, not sure if that has a meaning within doxygen?

TobiKattmann · 2019-03-21T14:49:07Z


+  /*!
+   * \brief A virtual member.
+   * \param[in] config - Definition of the particular problem.


Missing input val_periodic. Can be copied from the non-virtual MatchPeriodic from l. 2119

TobiKattmann · 2019-05-24T08:14:51Z

+   * \param[in] config - Definition of the particular problem.
+   * \param[in] val_periodic - Index of the first periodic face in a pair.
+   */
+  void MatchPeriodic(CConfig *config, unsigned short val_periodic);


Input parameters as a list

TobiKattmann · 2019-05-24T08:15:12Z

   * \param[in] monitoring - turn on priting residuals from solver to screen.
+   * \param[in] config - Definition of the particular problem.
   */
  unsigned long CG_LinSolver(const CSysVector & b, CSysVector & x, CMatrixVectorProduct & mat_vec,


Input parameters as a list

TobiKattmann · 2019-05-24T08:15:19Z

   * \param[in] monitoring - turn on priting residuals from solver to screen.
+   * \param[in] config - Definition of the particular problem.
   */
  unsigned long FGMRES_LinSolver(const CSysVector & b, CSysVector & x, CMatrixVectorProduct & mat_vec,


Input parameters as a list

TobiKattmann · 2019-05-24T08:15:24Z

   * \param[in] monitoring - turn on priting residuals from solver to screen.
+   * \param[in] config - Definition of the particular problem.
   */
  unsigned long BCGSTAB_LinSolver(const CSysVector & b, CSysVector & x, CMatrixVectorProduct & mat_vec,


Input parameters as a list

TobiKattmann · 2019-05-24T08:15:37Z

 #include "../include/toolboxes/printing_toolbox.hpp"

-CConfig::CConfig(char case_filename[MAX_STRING_SIZE], unsigned short val_software, unsigned short val_iZone, unsigned short val_nZone, unsigned short val_nDim, unsigned short verb_level) {
+CConfig::CConfig(char case_filename[MAX_STRING_SIZE], unsigned short val_software, unsigned short val_iZone, unsigned short val_nZone, unsigned short val_nDim, bool verb_high) {


Input parameters as a list

…and limiters (#2961) ## Proposed Changes This is one of **seven related periodic-boundary bug-fix PRs**. | PR | Scope | |---|---| | #2961 (this PR) | Rotational Jacobian, multigrid velocity rotation and limiters | | #2963 | Axis points, slip-wall normals and moving donor volumes | | #2964 | Periodic volumes, neighbor counts, sensors, stencils and coarse flags | | #2965 | Streamwise periodicity, auxiliary gradients and flamelet metadata | | #2966 | Inverse affine transforms in the parser and FEM | | #2968 | Diagnostics for unsupported periodic solver combinations | | #2969 | Translated wall-distance images | The other bug-fix PRs are drafts with their remaining validation and implementation limits documented. Their grouping does not imply a linear merge order; #2963 depends on this PR. The fully coupled implicit operator proposed in #2967 is a separate experimental enhancement. It has been closed and deferred because no overall benefit has been established and convergence/validation issues remain. This PR retains the existing implicit approximation. Three fixes for rotational periodic boundaries, one commit each. The test cases are in the comment below. - **Implicit runs:** the Jacobian block that a periodic point gets from its match was rotated on one side only. It is now rotated on both sides. - **Multigrid:** the coarse levels did not rotate the velocity, so the run did not converge. The velocity is now rotated on all levels. - **Limiters:** the limiters of the velocity components were rotated like a vector, but they are separate numbers. On the periodic boundary one velocity component became first order, and limiters could be negative. Each side now sends the velocity bounds and reconstruction increments in the frame of the other side, so each limiter uses the complete stencil. Nothing changes for a translation. New tests `periodic2d_no_limiter` and `periodic2d_multigrid`; `periodic2d` is now also in the MPI regression. Their x86 reference values pass in CI. Existing reference values that change: `aachen_turbine_restart`, `jones_turbocharger_restart` and `multi_interface` (rotational periodic, the Jacobian block), and the hybrid `periodic2d` (the Jacobian block and the limiters). The x86 periodic tests and the other changed cases pass in CI; the Jones restart residual references were refreshed from the serial, MPI and hybrid CI logs. The aarch64 references still need the CI runs. No conflict with #2945 and #2959. #2956 also changes the reference values of `jones_turbocharger_restart`; whichever is merged second takes them from the CI. ## Related Work - #652 added the present implementation of the periodic boundaries. - #1563 fixed the rotation in 2D and added the `periodic2D` case used here. - #1467, #763 and #1585 discuss the incomplete linear system at periodic points; this PR keeps the existing approximation. ## PR Checklist - [x] I am submitting my contribution to the develop branch. - [x] My contribution generates no new compiler warnings (try with --warnlevel=3 when using meson). - [x] My contribution is commented and consistent with SU2 style (https://su2code.github.io/docs_v7/Style-Guide/). - [x] I used the pre-commit hook to prevent dirty commits and used `pre-commit run --all` to format old commits. - [x] I have added a test case that demonstrates my contribution, if necessary. - [ ] I have updated appropriate documentation (Tutorials, Docs Page, config_template.cpp), if necessary. --------- Co-authored-by: rois1995 <rauxa1995@hotmail.it> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Josh Kelly <81244680+joshkellyjak@users.noreply.github.com> Co-authored-by: Nijso <bigfootedrockmidget@hotmail.com>

economon added 30 commits June 12, 2018 11:58

Basic implementation of entirely new periodic BC for Euler. Needs to …

9d0c59a

…be generalized to all solvers and parallelization should be improved.

Updating with develop.

223a82e

Prototype routine for communicating gradients at periodic boundaries.

44d5233

Merged with develop and fixed conflicts.

fdcc27e

Periodic BCs now handle gradients, limiters, undivided laplacian, sen…

8f57b71

…sors, and eigenvalues. Both Euler and N-S operational. Fixed multiple bugs.

Merge branch 'develop' into feature_periodic

f306b5b

Added vorticity and q-criterion to output for testing periodic BC.

2a24b3f

Merge branch 'develop' into feature_periodic

5c99356

Added periodic BC routines to incompressible solver. Needs testing.

ef123aa

Some fixes for incompressible periodic BCs.

2b2a113

Merge branch 'develop' into feature_periodic

b2be497

First prototype periodic BC version for RANS.

6718b48

Merged develop and fixed conflicts.

02f59e1

Merge branch 'fix_xcode' into feature_periodic

97e5240

Moved over rotation matrix routines.

3673357

Merge branch 'develop' into feature_periodic

8e36413

Added geometry routines for P2P comms.

a501552

Added matrix routines for P2P comms.

dfd9655

Added solver routines for P2P comms.

0d832d6

Added comm level to replace verbosity.

b003c1e

Activated new mpi comms for matrix class only.

c648de4

Added comm level full as default.

daca8f5

Removed unused mpi comm routines.

721282d

Adjusted preprocessing for non-CFD modules.

4b6f3e5

Activated new P2P routines for geometry and solver classes.

b3995dc

Small change.

c521fa4

Comm level checks to save mpi overhead when requested.

2685cb5

Fixed problem with MPI_Waitall.

123a635

Update with develop and fixed conflicts.

700dd7c

Creating new MPI structures for periodicity.

abbefd2

economon merged commit 1571f0c into develop Apr 24, 2019

economon deleted the feature_mpi_periodic branch April 24, 2019 01:27

This was referenced Apr 30, 2019

Order of parallel communication #252

Closed

Current status of periodic BCs #416

Closed

Segfault when running problem with periodic boundary conditions #431

Closed

MPI Failure when Running with 16+ Cores #550

Closed

TobiKattmann reviewed May 24, 2019

View reviewed changes

This was referenced Jun 5, 2019

Remove Dead Code 2 #699

Merged

Remove Dead Code 3 #701

Merged

pcarruscag mentioned this pull request Jul 30, 2019

Small fix affecting MPI+MG+RANS #744

Merged

4 tasks done

clarkpede mentioned this pull request Aug 21, 2019

Periodic boundary conditions and simple timestepping don't combine well #763

Closed

TobiKattmann mentioned this pull request Aug 26, 2019

Streamwise periodicity for incompressible flow #773

Merged

4 tasks done

talbring added changelog:chore changelog:feature and removed changelog:chore labels Nov 7, 2019

talbring changed the title ~~MPI Point-to-Point Refactoring + New Periodic BC~~ MPI Point-to-Point Refactoring + New Periodic BC Implementation Nov 8, 2019

pr-triage Bot added the PR: merged label Nov 8, 2019

pcarruscag mentioned this pull request Feb 29, 2020

SU2 v7 is slower on Ethernet cluster #894

Closed

cvencro mentioned this pull request Aug 16, 2020

Fixes for incompressible solver - rotating frame and convergence rate for unsteady problems #1067

Merged

1 of 5 tasks

TobiKattmann mentioned this pull request Sep 26, 2021

Add periodic boundary conditions to the heat-equation solver #1382

Merged

3 of 5 tasks

Navigation Menu

Repository navigation

MPI Point-to-Point Refactoring + New Periodic BC Implementation - #652

MPI Point-to-Point Refactoring + New Periodic BC Implementation#652
economon merged 86 commits into
developfrom
feature_mpi_periodic

economon commented Feb 15, 2019 •

edited

Loading

Uh oh!

economon commented Apr 24, 2019

Uh oh!

TobiKattmann left a comment

Uh oh!

TobiKattmann Mar 21, 2019

Uh oh!

TobiKattmann Mar 21, 2019

Uh oh!

TobiKattmann Mar 21, 2019

Uh oh!

TobiKattmann Mar 21, 2019

Uh oh!

TobiKattmann Mar 21, 2019

Uh oh!

TobiKattmann May 24, 2019

Uh oh!

TobiKattmann May 24, 2019

Uh oh!

TobiKattmann May 24, 2019

Uh oh!

TobiKattmann May 24, 2019

Uh oh!

TobiKattmann May 24, 2019

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

economon commented Feb 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes

Related Work

PR Checklist

Uh oh!

economon commented Apr 24, 2019

Uh oh!

TobiKattmann left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

economon commented Feb 15, 2019 •

edited

Loading