Error in FSI - runs fine in PC but not on HPC

  • 212 Views
  • Last Post 03 June 2019
  • Topic Is Solved
ansysuser posted this 03 May 2019

Hello,

 

I am trying to get a coupled system to run.  I found that the system runs to completion on my PC, but I get an error when I try to run it on HPC.

 *** ERROR ***                           CP =       1.126   TIME= 18:040
 The FSIN interface names of the mechanical model are different from the
 participant region names of the system coupling. 

 

Progress = 0; Message = "System coupling run completed with errors.  Fluid Flow (Fluent) (Solution) reported: The FSIN interface names of the mechanical model are different from the participant region names of the system coupling."

 

 

 

Order By: Standard | Newest | Votes
ansysuser posted this 08 May 2019

Anyone have any insight on this?

ansysuser posted this 08 May 2019

  +====================================================================+
 |                    Data Transfer Information (2)                   |
 +====================================================================+

 +--------------------------------------------------------------------+
 | Data Transfer: Data Transfer                                       |
 +--------------------------------------------------------------------+

 Source : Fluid Flow (Fluent)                                         
   Region                = fsi                                        
   Variable              = force                                      

 Target : Transient Structural                                        
   Region                = Fluid Solid Interface                      
   Variable              = Force                                      

 General Information :                                                
   Name                  = Data Transfer                              
   Execute Transfer At   = Start Of Iteration                         
   Convergence Option    = RMS Change In Data                         
     Target Value        = 0.01                                       
   Under Relax. Factor   = 1                                          
   Ramping               = None                                       

 +--------------------------------------------------------------------+
 | Data Transfer: Data Transfer 2                                     |
 +--------------------------------------------------------------------+

 Source : Transient Structural                                        
   Region                = Fluid Solid Interface                      
   Variable              = Incremental Displacement                   

 Target : Fluid Flow (Fluent)                                         
   Region                = fsi                                        
   Variable              = displacement                               

 General Information :                                                
   Name                  = Data Transfer 2                            
   Execute Transfer At   = Start Of Iteration                         
   Convergence Option    = RMS Change In Data                         
     Target Value        = 0.01                                       
   Under Relax. Factor   = 1                                          
   Ramping               = None     

ansysuser posted this 08 May 2019

  +====================================================================+
 |                    Data Transfer Information (2)                   |
 +====================================================================+

 +--------------------------------------------------------------------+
 | Data Transfer: Data Transfer                                       |
 +--------------------------------------------------------------------+

 Source : Fluid Flow (Fluent)                                         
   Region                = fsi                                        
   Variable              = force                                      

 Target : Transient Structural                                        
   Region                = Fluid Solid Interface                      
   Variable              = Force                                      

 General Information :                                                
   Name                  = Data Transfer                              
   Execute Transfer At   = Start Of Iteration                         
   Convergence Option    = RMS Change In Data                         
     Target Value        = 0.01                                       
   Under Relax. Factor   = 1                                          
   Ramping               = None                                       

 +--------------------------------------------------------------------+
 | Data Transfer: Data Transfer 2                                     |
 +--------------------------------------------------------------------+

 Source : Transient Structural                                        
   Region                = Fluid Solid Interface                      
   Variable              = Incremental Displacement                   

 Target : Fluid Flow (Fluent)                                         
   Region                = fsi                                        
   Variable              = displacement                               

 General Information :                                                
   Name                  = Data Transfer 2                            
   Execute Transfer At   = Start Of Iteration                         
   Convergence Option    = RMS Change In Data                         
     Target Value        = 0.01                                       
   Under Relax. Factor   = 1                                          
   Ramping               = None     

ansysuser posted this 08 May 2019

Here is the system coupling window.  The solution ran just fine on my PC.

 

 

kkourbat posted this 22 May 2019

1) how are you transferring the project from PC to HPC, and 2) is your HPC linux-based?

ansysuser posted this 22 May 2019

Hello, 

Since posting several weeks ago I have made some progress.  I will updated the situation below:

 

I have Mechanical export a .dat file. 

I have Fluent export .cas.gz and .dat.gz

I have the Coupler export .sci

Then I put them all on the same Linux directory and call them with the slurm commands:

module load ansys
echo "Made to system coupling"
/apps/r/ansys/v193/aisol/.workbench -cmd ansys.services.systemcoupling.exe -inputFile ${COUPLING} &
echo "Made it past system coupling"
fluent 3ddp -g -t${NPF} -driver null -ssh -scport=${HOSTPORT[0]} -schost=${HOSTPORT[1]} -scname="${JOB[4]}" < ${JOURNAL} > output.FLUENT &
ansys193 -b -scport=${HOSTPORT[0]} -schost=${HOSTPORT[1]} -scname="${JOB[2]}" -i ${STRUCTURALDATA} -o output.ANSYS -np ${NPA}

And in the slurm output file I see this:

Made to system coupling
Made it past system coupling
[0] MPI startup(): Single-threaded optimized library
... lots of lines of MPI startup...
Cleanup script file is /pfs/tsfs1/gscratch/....
forrtl: No such file or directory
forrtl: No such file or directory
forrtl: No such file or directory
forrtl: No such file or directory
forrtl: No such file or directory
forrtl: No such file or directory
forrtl: severe (28): CLOSE error, unit 24, file "Unknown"

In output.FLUENT I see it loads the case and data, builds mesh, distributed mesh, parts, faces, etc, then gets to Done. Then the next thing I see in there is:

> Deleting Client...done
Shutting down Server...done
No error handler available
System Coupling Service encountered fatal error.#f

In the output.ANSYS file I see:

 *** ERROR ***
 Another ANSYS job with the same job name (file) is already running in this
 directory or the file.lock file has not been deleted from an abnormally
 terminated ANSYS run.  To disable this check, set the ANSYS_LOCK environment
 variable to OFF.

Now I know the file.lock is deleted between each try of this, so that is not the problem.  Also, this is the only job running when I try to run it.  No other ANSYS jobs at all on the whole cluster.

 

ANSYS wrote out a whole bunch of other files, so let me know if you need to know what is in those as well.

 

Thanks

 

 

 

 

 

Steve posted this 22 May 2019

Hi there,

It looks like you're using the old command line method to start this case. Please review the following tutorial that goes over how to run System Coupling from the command line in 2019R1.

https://ansyshelp.ansys.com/account/secured?returnurl=/Views/Secured/corp/v193/sysc_tut/sysc_tut_oscplate_commandline.html

You can put all the System Coupling commands into a text file, (run.py for example, but the name isn't important). Then in your slurm commands you would have ...systemcoupling -R run.py which would start all participants.

Steve

 

ansysuser posted this 22 May 2019

 Thank you, I will give that a try if I can and let you know how it goes.

ansysuser posted this 24 May 2019

Hello,

I have followed the above as far as setting up the problem and getting the files onto HPC. Then I use this command in the batch file:

/apps/r/ansys/v193/SystemCoupling/bin/systemcoupling -R run.py > outsyscoup.txt

The only thing that is in the outsyscoup.txt is:

Executing from: /apps/r/ansys/v193/SystemCoupling/bin/systemcoupling
... list of Point Releases and Patches....
executing script 'run.py'

+----------------------------------------------------------------------------+
| The most recent execution of the LoadParticipants command reset the        |
| datamodel.  All content has been cleared.                                  |
+----------------------------------------------------------------------------+

The contents of run.py are as follows (I was hoping to get a printout of the names as in the documentation so I could then do the next step, i.e., use GetRegionNamesForParticipant and then modify the timestep).  But these commands did not print out the ChildNames, as you can see from the output above.

LoadParticipants(InputFiles = ['Mech_sys_couple_files.scp', 'Fluent_coupler.scp'])
DatamodelRoot().CouplingParticipant.GetChildNames()

 

What has gone wrong?

 

Thanks

 

 

 

Steve posted this 27 May 2019

- Is the current directory of your linux terminal the same as the location of the .scp files?

- Are the .scp files located in the same directory? You could also have them in separate directories with: LoadParticipants(InputFiles = ['./Structural/Mech_sys_couple_files.scp', './Fluid/Fluent_coupler.scp'])

- If you open both .scp files in a text editor, is the <WorkingDirectory> set correctly?

You could also use the workflow with setup in Workbench and execution in the command line. The workflow is a bit easier. I meant to send this link last time. https://ansyshelp.ansys.com/account/secured?returnurl=/Views/Secured/corp/v193/sysc_tut/sysc_tut_oscplate_sci-commandline.html.

One other note, ANSYS doesn't officially support slurm.

Steve

  • Liked by
  • ansysuser
ansysuser posted this 28 May 2019

Hello Steve,

 

Thanks for your follow up.

For more background: I am working remotely on HPC, so I don't exactly have a "terminal" that I can use real time.  I have to use a scheduler and the cluster uses SLURM through bash files.  All commands to run a job have to be in the bash file. I have been using ANSYS on SLURM for over 5 years on different clusters, but only with Fluent, and I have never had a problem until using the system coupling.  

Now, all files are in the same directory, including the bash file that initiates the job, and the ANSYS files.  I made sure that the <WorkingDirectory> in the scp files is set to the local directory on the cluster, so this is not the problem.  Here is a screenshot of the directory, showing all files inside it.

 

I can try the other method you suggest and let you know if that works.

 

Thanks again.

ansysuser posted this 28 May 2019

Hello,

 

I tried to make it through that tutorial you linked to above, using the oscillating plate even, but there is a part where there seems to be something missing in the tutorial instructions.  On this page (which is the very first link of the page you gave)

 

https://ansyshelp.ansys.com/account/secured?returnurl=/Views/Secured/corp/v193/sysc_tut/sysc_tut_oscplate_sci-commandline_steps.html

 

Look under 1. where it says, "When setting up project systems, prepare input files for each system, (Structural, Fluid, and System Coupling) before closing the associated application"


I went through the instructions under Structural and Fluid through those links, but when I click on the System Coupling link the instructions say, "On the Project Schematic, ensure that the System Coupling system's Setup cell (C2) is in an Up-to-Date (  state."  My System Coupling has a question mark.  The tutorial has not covered how to set up System Coupling.  It skipped that step.

Help!

ansysuser posted this 29 May 2019

Hello,

I was able to get to the system coupler setup by gong to the other track of the tutorial (setting up and solving in Workbench, as opposed to setting up in Workbench and solving at command line).  So when I ran the case this way, I get:

 

Starting Command Console...
Type ^Z<Enter> to exit
>>> Exiting Command Console...
/var/spool/slurmd/job2752553/slurm_script: line 12: syntax error near unexpected token `FilePath'

I am not sure why the error about FilePath is there, because this is the exact command simply copied and pasted from the tutorial linked above.

I appreciate any further input, as I am trying to evaluate the performance of HPC for coupled analysis for our group. We want to know if it is worthwhile to purchase several HPC licenses if the runs are significantly faster on HPC for these types of simulations than on our high end desktops.  I have been trying to get a simple coupled simulation to run on HPC for over a month now. Fluent runs just fine, and so I have been able to compare performance, but this coupled simulation seems to be much more trouble!

 

Thanks

 

 

kkourbat posted this 29 May 2019

could be a result of copy-pasting  Are you operating with text files with the script and systems coupling commands on windows and then using them on linux, or are you creating those files directly on linux? I am asking because there is a difference between end of line character on DOS and Unix which may lead to an error when using Windows-created scripts on linux. The fix is to save text files in Unix format, which many text editors allow, or to use dos2unix command to convert the file on linux.

If this doesn't help, we'll need to see your slurm and systems coupling scripts 

 

  • Liked by
  • ansysuser
ansysuser posted this 29 May 2019

Hello kkoubat,

I have gotten further.  I am facing a new error, which is detailed below.  Recall that I am following Steve's advice for the setup under the tutorial "Setup with Workbench and run from Command line" for the oscillating plate model.

Here is how I call ANSYS in the bash file:

/apps/r/ansys/v193/SystemCoupling/bin/systemcoupling < inputfile.in

And below is the contents of the file inputfile.in.  Note that this is taken directly from the tutorial.  I am aware of the eol difference across OS, so I made sure to paste into linux and then backspace and return over each line.  I don't think that is the issue.

ImportSystemCouplingInputFile(FilePath = 'ScInput.sci')
execCon = DatamodelRoot().CouplingParticipant
execCon['Solution'].ExecutionControl.InitialInput = 'mapdl.dat'
execCon['Solution'].ExecutionControl.WorkingDirectory = 'Structural_Run'
execCon['Solution 1'].ExecutionControl.InitialInput = 'fluidFlow.cas'
execCon['Solution 1'].ExecutionControl.WorkingDirectory = 'Fluid_Run'
execCon['Solution'].ExecutionControl.PrintState()
execCon['Solution 1'].ExecutionControl.PrintState()
Solve()

Now below is the content of the output from SLURM.  Note that it appears to be seeing the input files for both Fluent and Mechanical, and that system coupler seems to be starting.

Starting Command Console...
Type ^Z<Enter> to exit
           
>>> >>> >>> >>> >>> >>> >>> CouplingParticipant:Solution
  ExecutionControl
    WorkingDirectory = Structural_Run
    ParallelArguments =
    InitialInput = mapdl.dat
    Option = ProgramControlled
    AdditionalArguments =

>>> CouplingParticipant:Solution 1
  ExecutionControl
    WorkingDirectory = Fluid_Run
    ParallelArguments =
    InitialInput = fluidFlow.cas
    Option = ProgramControlled
    AdditionalArguments =

>>> Host spawning Node 0 on machine "t416.cluster" (unix).
/apps/r/ansys/v193/SystemCoupling/runTime/linx64/runTimeLibraries/launcher/fluent18.0.0/../fluent18.0.0/bin/fluent -r18.0.0 3ddp -mpi=ibmmpi -p -node -t1 -nodehomedir=False -flux -wait -nosge -mport 10.98.4.168:10.98.4.1687343:0
ls: cannot access /apps/r/ansys/v193/SystemCoupling/runTime/linx64/runTimeLibraries/launcher/fluent18.0.0/../fluent18.0.0/cortex/lnamd64/cortex.*.*.*: No such file or directory
Starting /apps/r/ansys/v193/SystemCoupling/runTime/linx64/runTimeLibraries/launcher/fluent18.0.0/../fluent18.0.0/multiport/mpi/lnamd64/ibmmpi/bin/mpirun -e MPI_IBV_NO_FORK_SAFE=1 -e MPI_USE_MALLOPT_MMAP_MAX=0 -np 1 /pfs/tsfs1/apps/el7-x86_64/r/ansys/v193/commonfiles/CPython/2_7_15/linx64/Release/python/bin/python2 /apps/r/ansys/v193/SystemCoupling/PyLib/kernel/Engine/ComputeNode.py /apps/r/ansys/v193/SystemCoupling/runTime/linx64/runTimeLibraries/launcher/fluent18.0.0/../fluent18.0.0/lnamd64/3ddp_node/fluent_mpi.18.0.0 node -mpiw ibmmpi -pic shmem -mport 10.98.4.168:10.98.4.1687343:0 -nodehomedir=False -wait -nosge

-------------------------------------------------------------------------------
ID          Hostname      Core  O.S.      PID   Vendor                     
-------------------------------------------------------------------------------
e0          t416.cluster  1/32  Linux-64  3320  Intel(R) Xeon(R) E5-2683 v4
controller  t416.cluster        Linux-64  3002  Intel(R) Xeon(R) E5-2683 v4

MPI Option Selected: ibmmpi
-------------------------------------------------------------------------------


==============================================================================
+============================================================================+
|                                                                            |
|                         Summary of Coupling Setup                          |
|                                                                            |
+============================================================================+
==============================================================================
           

+============================================================================+
|                              Coupling Control                              |
+============================================================================+
|                                                                            |
| Analysis Type :                                                  Transient |
| Duration Control                                                           |
|    Option :                                                        EndTime |
|    End Time :                                                     10.0 [s] |
| Step Control                                                               |
|    Maximum Number Of Iterations Per Step :                               5 |
|    Minimum Number Of Iterations Per Step :                               1 |
|    Time Step Size :                                                0.1 [s] |
|                                                                            |
+============================================================================+
|                               Output Control                               |
+============================================================================+
|                                                                            |
| Output Control Option :                                       StepInterval |
| Output Frequency :                                                       2 |
|                                                                            |
+============================================================================+
|                        Participants Information (2)                        |
+============================================================================+
|                                                                            |
| Participant: Structural                                                    |
|    Type :                                                            MAPDL |
|    Update Controls:                                                        |
|       Option :                                           ProgramControlled |
|    Region: Fluid Solid Interface                                           |
|       Topology :                                                   Surface |
|       Input Variables :                                              Force |
|       Output Variables :                          Incremental Displacement |
|                                                                            |
| Participant: Fluid                                                         |
|    Type :                                                           FLUENT |
|    Update Controls:                                                        |
|       Option :                                           ProgramControlled |
|    Region: wall_deforming                                                  |
|       Topology :                                                   Surface |
|       Input Variables :                                       displacement |
|       Output Variables :                                             force |
|                                                                            |
+============================================================================+
|                         Interfaces Information (1)                         |
+============================================================================+
|                                                                            |
| Interface: interface-1                                                     |
|    Sidene                                                                |
|       Coupling Participant :                                    Structural |
|       Region List :                                  Fluid Solid Interface |
|    Side:Two                                                                |
|       Coupling Participant :                                         Fluid |
|       Region List :                                         wall_deforming |
|    Transfer To Side One:Force                                              |
|       Side One Variable :                                            Force |
|       Side Two Variable :                                            force |
|       Convergence Target :                                        1.00E-02 |
|       Ramping Option :                                                None |
|       Relaxation Factor :                                         1.00E+00 |
|       Mapping Type :                                  Surface Conservative |
|    Transfer To Side Two:Incremental Displacement                           |
|       Side One Variable :                         Incremental Displacement |
|       Side Two Variable :                                     displacement |
|       Convergence Target :                                        1.00E-02 |
|       Ramping Option :                                                None |
|       Relaxation Factor :                                         1.00E+00 |
|       Mapping Type :                            Surface Profile Preserving |
|                                                                            |
+============================================================================+
+============================================================================+
|                     Participants Execution Information                     |
+============================================================================+
|                                                                            |
| Structural                                                                 |
|   Execution Command:                                                       |
|     "/apps/r/ansys/v193/ansys/bin/ansys193" -b nolist -s noread -o Solutio |
|     n.out -scport 42633 -schost t416.cluster -scname "Solution" -scid="332 |
|     0_t416.cluster" -sclic=2325@tlic1 -i mapdl.dat                         |
|   Working Directory:                                                       |
|     REDACTED                     |
|                                                                            |
| Fluid                                                                      |
|   Execution Command:                                                       |
|     "/apps/r/ansys/v193/fluent/bin/fluent" 3ddp -g -scport=42633 -schost=t |
|     416.cluster -scname="Solution 1" -scid="3320_t416.cluster" -sclic="232 |
|     5@tlic1" -i Solution1.jou                                              |
|   Working Directory:                                                       |
|     REDACTED                            |
+============================================================================+
Awaiting connections from coupling participants...Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(805)....: fail failed
MPID_Init(1832)..........: channel initialization failed
MPIDI_CH3_Init(126)......: fail failed
MPID_nem_init_ckpt(1087).: fail failed
MPIDI_PG_SetConnInfo(711): PMI_KVS_Put returned 14
In: PMI_Abort(69253391, Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(805)....: fail failed
MPID_Init(1832)..........: channel initialization failed
MPIDI_CH3_Init(126)......: fail failed
MPID_nem_init_ckpt(1087).: fail failed
MPIDI_PG_SetConnInfo(711): PMI_KVS_Put returned 14)
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.

kkourbat posted this 29 May 2019

hmm, looks to be mechanical end throwing MPI errors. This is ibmmpi. Let me bring our installation folks into this discussion

  • Liked by
  • ansysuser
ansysuser posted this 29 May 2019

Thank you so much, kkourbat.

Steve posted this 29 May 2019

Hi,

As a sanity check, can you try: systemcoupling -R inputfile.in, instead of: /apps/r/ansys/v193/SystemCoupling/bin/systemcoupling < inputfile.in. Please post the output for this. I have a few other things to check if this doesn't work.

Steve

ansysuser posted this 29 May 2019

Hello Steve,

It looks the same to me.  I don't know if this helps or not, but I noticed when I look in my job queue that the node is still running this job even after no more output is printed to the output file.  It runs until the time requested is up.

executing script 'inputfile.in'
CouplingParticipant:Solution
  ExecutionControl
    WorkingDirectory = Structural_Run
    ParallelArguments =
    InitialInput = mapdl.dat
    Option = ProgramControlled
    AdditionalArguments =

CouplingParticipant:Solution 1
  ExecutionControl
    WorkingDirectory = Fluid_Run
    ParallelArguments =
    InitialInput = fluidFlow.cas
    Option = ProgramControlled
    AdditionalArguments =

Host spawning Node 0 on machine "t416.cluster" (unix).
/apps/r/ansys/v193/SystemCoupling/runTime/linx64/runTimeLibraries/launcher/fluent18.0.0/../fluent18.0.0/bin/fluent -r18.0.0 3ddp -mpi=ibmmpi -p -node -t1 -nodehomedir=False -flux -wait -nosge -mport 10.98.4.168:10.98.4.168:41642:0
ls: cannot access /apps/r/ansys/v193/SystemCoupling/runTime/linx64/runTimeLibraries/launcher/fluent18.0.0/../fluent18.0.0/cortex/lnamd64/cortex.*.*.*: No such file or directory
Starting /apps/r/ansys/v193/SystemCoupling/runTime/linx64/runTimeLibraries/launcher/fluent18.0.0/../fluent18.0.0/multiport/mpi/lnamd64/ibmmpi/bin/mpirun -e MPI_IBV_NO_FORK_SAFE=1 -e MPI_USE_MALLOPT_MMAP_MAX=0 -np 1 /pfs/tsfs1/apps/el7-x86_64/r/ansys/v193/commonfiles/CPython/2_7_15/linx64/Release/python/bin/python2 /apps/r/ansys/v193/SystemCoupling/PyLib/kernel/Engine/ComputeNode.py /apps/r/ansys/v193/SystemCoupling/runTime/linx64/runTimeLibraries/launcher/fluent18.0.0/../fluent18.0.0/lnamd64/3ddp_node/fluent_mpi.18.0.0 node -mpiw ibmmpi -pic shmem -mport 10.98.4.168:10.98.4.168:41642:0 -nodehomedir=False -wait -nosge

-------------------------------------------------------------------------------
ID          Hostname      Core  O.S.      PID   Vendor                     
-------------------------------------------------------------------------------
e0          t416.cluster  1/32  Linux-64  5532  Intel(R) Xeon(R) E5-2683 v4
controller  t416.cluster        Linux-64  5206  Intel(R) Xeon(R) E5-2683 v4

MPI Option Selected: ibmmpi
-------------------------------------------------------------------------------

Cleanup script file is /pfs/tsfs1/gscratch/mfig/SysC_Run/cleanup-afd-t416.cluster-5206.sh


==============================================================================
+============================================================================+
|                                                                            |
|                         Summary of Coupling Setup                          |
|                                                                            |
+============================================================================+
==============================================================================

+============================================================================+
|                              Coupling Control                              |
+============================================================================+
|                                                                            |
| Analysis Type :                                                  Transient |
| Duration Control                                                           |
|    Option :                                                        EndTime |
|    End Time :                                                     10.0 [s] |
| Step Control                                                               |
|    Maximum Number Of Iterations Per Step :                               5 |
|    Minimum Number Of Iterations Per Step :                               1 |
|    Time Step Size :                                                0.1 [s] |
|                                                                            |
+============================================================================+
|                               Output Control                               |
+============================================================================+
|                                                                            |
| Output Control Option :                                       StepInterval |
| Output Frequency :                                                       2 |
|                                                                            |
+============================================================================+
|                        Participants Information (2)                        |
+============================================================================+
|                                                                            |
| Participant: Structural                                                    |
|    Type :                                                            MAPDL |
|    Update Controls:                                                        |
|       Option :                                           ProgramControlled |
|    Region: Fluid Solid Interface                                           |
|       Topology :                                                   Surface |
|       Input Variables :                                              Force |
|       Output Variables :                          Incremental Displacement |
|                                                                            |
| Participant: Fluid                                                         |
|    Type :                                                           FLUENT |
|    Update Controls:                                                        |
|       Option :                                           ProgramControlled |
|    Region: wall_deforming                                                  |
|       Topology :                                                   Surface |
|       Input Variables :                                       displacement |
|       Output Variables :                                             force |
|                                                                            |
+============================================================================+
|                         Interfaces Information (1)                         |
+============================================================================+
|                                                                            |
| Interface: interface-1                                                     |
|    Sidene                                                                |
|       Coupling Participant :                                    Structural |
|       Region List :                                  Fluid Solid Interface |
|    Side:Two                                                                |
|       Coupling Participant :                                         Fluid |
|       Region List :                                         wall_deforming |
|    Transfer To Side One:Force                                              |
|       Side One Variable :                                            Force |
|       Side Two Variable :                                            force |
|       Convergence Target :                                        1.00E-02 |
|       Ramping Option :                                                None |
|       Relaxation Factor :                                         1.00E+00 |
|       Mapping Type :                                  Surface Conservative |
|    Transfer To Side Two:Incremental Displacement                           |
|       Side One Variable :                         Incremental Displacement |
|       Side Two Variable :                                     displacement |
|       Convergence Target :                                        1.00E-02 |
|       Ramping Option :                                                None |
|       Relaxation Factor :                                         1.00E+00 |
|       Mapping Type :                            Surface Profile Preserving |
|                                                                            |
+============================================================================+
+============================================================================+
|                     Participants Execution Information                     |
+============================================================================+
|                                                                            |
| Structural                                                                 |
|   Execution Command:                                                       |
|     "/apps/r/ansys/v193/ansys/bin/ansys193" -b nolist -s noread -o Solutio |
|     n.out -scport 42400 -schost t416.cluster -scname "Solution" -scid="553 |
|     2_t416.cluster" -sclic=2325@tlic1 -i mapdl.dat                         |
|   Working Directory:                                                       |
|                           |
|                                                                            |
| Fluid                                                                      |
|   Execution Command:                                                       |
|     "/apps/r/ansys/v193/fluent/bin/fluent" 3ddp -g -scport=42400 -schost=t |
|     416.cluster -scname="Solution 1" -scid="5532_t416.cluster" -sclic="232 |
|     5@tlic1" -i Solution1.jou                                              |
|   Working Directory:                                                       |
|                                 |
+============================================================================+
Awaiting connections from coupling participants...Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(805)....: fail failed
MPID_Init(1832)..........: channel initialization failed
MPIDI_CH3_Init(126)......: fail failed
MPID_nem_init_ckpt(1087).: fail failed
MPIDI_PG_SetConnInfo(711): PMI_KVS_Put returned 14
In: PMI_Abort(69253391, Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(805)....: fail failed
MPID_Init(1832)..........: channel initialization failed
MPIDI_CH3_Init(126)......: fail failed
MPID_nem_init_ckpt(1087).: fail failed
MPIDI_PG_SetConnInfo(711): PMI_KVS_Put returned 14)
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** STEP 2753601.0 ON t416 CANCELLED AT 2019-05-29T13:22:20 ***
[mpiexec@t416.cluster] control_cb (../../pm/pmiserv/pmiserv_cb.c:798): connection to proxy 0 at host t416 failed
[mpiexec@t416.cluster] HYDT_dmxu_poll_wait_for_event (../../tools/demux/demux_poll.c:76): callback returned error status
[mpiexec@t416.cluster] HYD_pmci_wait_for_completion (../../pm/pmiserv/pmiserv_pmci.c:501): error waiting for event
[mpiexec@t416.cluster] main (../../ui/mpich/mpiexec.c:1147): process manager error waiting for completion

Steve posted this 30 May 2019

Thanks. We'll take a look into this in more detail and get back to you with what we find.

Steve

  • Liked by
  • ansysuser
Steve posted this 31 May 2019

Hi,

We're thinking this might be an MPI issues with Mechanical. As a test we need to see what happens when Mechanical is forced to run with Shared Memory Parallel. Please add the following line to your inputfile.in, run again, and send the output.

execCon['Solution 2'].ExecutionControl.AdditionalArguments = '-smp'

Thanks,

Steve

 

  • Liked by
  • ansysuser
ansysuser posted this 31 May 2019

Hello,

Thank you for your reply.  Here is the output where it deviates from above:

 

+============================================================================+
|                        Participants Information (3)                        |
+============================================================================+
|                                                                            |
| Participant: Coupling Participant                                          |
|    Type :                                                          DEFAULT |
|    Update Controls:                                                        |
|       Option :                                           ProgramControlled |
|                                                                            |
| Participant: Fluid                                                         |
|    Type :                                                           FLUENT |
|    Update Controls:                                                        |
|       Option :                                           ProgramControlled |
|    Region: wall_deforming                                                  |
|       Topology :                                                   Surface |
|       Input Variables :                                       displacement |
|       Output Variables :                                             force |
|                                                                            |
| Participant: Structural                                                    |
|    Type :                                                            MAPDL |
|    Update Controls:                                                        |
|       Option :                                           ProgramControlled |
|    Region: Fluid Solid Interface                                           |
|       Topology :                                                   Surface |
|       Input Variables :                                              Force |
|       Output Variables :                          Incremental Displacement |
|                                                                            |
+============================================================================+
|                         Interfaces Information (1)                         |
+============================================================================+
|                                                                            |
| Interface: interface-1                                                     |
|    Sidene                                                                |
|       Coupling Participant :                                    Structural |
|       Region List :                                  Fluid Solid Interface |
|    Side:Two                                                                |
|       Coupling Participant :                                         Fluid |
|       Region List :                                         wall_deforming |
|    Transfer To Side One:Force                                              |
|       Side One Variable :                                            Force |
|       Side Two Variable :                                            force |
|       Convergence Target :                                        1.00E-02 |
|       Ramping Option :                                                None |
|       Relaxation Factor :                                         1.00E+00 |
|       Mapping Type :                                  Surface Conservative |
|    Transfer To Side Two:Incremental Displacement                           |
|       Side One Variable :                         Incremental Displacement |
|       Side Two Variable :                                     displacement |
|       Convergence Target :                                        1.00E-02 |
|       Ramping Option :                                                None |
|       Relaxation Factor :                                         1.00E+00 |
|       Mapping Type :                            Surface Profile Preserving |
|                                                                            |
+============================================================================+
+============================================================================+
|                     Participants Execution Information                     |
+============================================================================+
|                                                                            |
| Coupling Participant                                                       |

+----------------------------------------------------------------------------+
| argument of type 'NoneType' is not iterable                                |
+----------------------------------------------------------------------------+
Traceback (most recent call last):
  File "/apps/r/ansys/v193/SystemCoupling/PyLib/Controller.py", line 139, in <module>
    _run(sys.argv)
  File "/apps/r/ansys/v193/SystemCoupling/PyLib/Controller.py", line 135, in _run
    _executeScript(options)
  File "/apps/r/ansys/v193/SystemCoupling/PyLib/Controller.py", line 107, in _executeScript
    kernel.commands.readScriptFile(scriptFile)
  File "PyLib/kernel/commands/__init__.py", line 31, in readScriptFile
  File "PyLib/kernel/commands/CommandManager.py", line 168, in readScriptFile
  File "inputfile.in", line 10, in <module>
    Solve()
  File "PyLib/kernel/commands/CommandDefinition.py", line 72, in func
  File "PyLib/kernel/commands/__init__.py", line 28, in executeCommand
  File "PyLib/kernel/commands/CommandManager.py", line 120, in executeCommand
  File "PyLib/cosimulation/externalinterface/cosim_commands/solver.py", line 94, in execute
  File "PyLib/cosimulation/solver/__init__.py", line 105, in solve
  File "PyLib/cosimulation/solver/__init__.py", line 796, in __startParticipantsControlled
cosimulation.solver.CosimulationError.CosimulationError
Shutting down compute-node processes

 

 

Steve posted this 31 May 2019

My mistake, it should be execCon['Solution'].ExecutionControl.AdditionalArguments = '-smp', not 'Solution 2'

  • Liked by
  • ansysuser
ansysuser posted this 31 May 2019

O.k., now we might be getting somewhere.  The new error looks to me like the coupled run started, but it thinks there is no Fluid-Structure-Interface in Mechanical.  But I know there is because I ran this on my PC first to make sure it would work before exporting.  Here is the error and a picture of my model tree which shows the FSI like created in the example.

==============================================================================
+============================================================================+
|                                                                            |
|                              Coupled Solution                              |
|                                                                            |
+============================================================================+
==============================================================================
+============================================================================+
| COUPLING STEP = 1                                SIMULATION TIME = 0.1 [s] |
+----------------------------------------------------------------------------+
|                                     |      Source            Target        |
+----------------------------------------------------------------------------+
|                           COUPLING ITERATION = 1                           |
| Participant Solution encountered a fatal error. Message: No FSIN           |
| interfaces are defined for the mechanical model.                           |

==============================================================================
Stack backtrace generated for process id 8199 on signal 11 :

+----------------------------------------------------------------------------+
| Solution: No FSIN interfaces are defined for the mechanical model.         |
+----------------------------------------------------------------------------+
1000000: fluent() [0x6ae149]
1000000: /usr/lib64/libc.so.6(+0x35270) [0x2b02bb834270]
1000000: /apps/r/ansys/v193/fluent/../tp/IntelCompiler/2017.6.256/linx64/lib/intel64/libintlc.so.5(__intel_sse2_strcat+0x95) [0x2b02b56498f5]
1000000: fluent(FatalError+0x35) [0x7048d5]
1000000: fluent() [0x6ff3f5]
1000000: fluent(_RpcServeOneRequest+0x1a1) [0x89d9d1]
1000000: fluent(ServeUntilCondition+0x16) [0x703646]
1000000: fluent(CX_PauseAtSyncPoint+0x132) [0x7031e2]
1000000: fluent() [0x6d0d05]
1000000: fluent(eval+0x497) [0x71d677]
1000000: fluent(eval+0xd1e) [0x71defe]
1000000: fluent(eval+0xd1e) [0x71defe]
1000000: fluent(eval+0xd1e) [0x71defe]
1000000: fluent() [0x71e6f6]
1000000: fluent(eval_errprotect+0x4e) [0x71ed7e]
Please include this information with any bug report you file on this issue!
==============================================================================

Traceback (most recent call last):
  File "/apps/r/ansys/v193/SystemCoupling/PyLib/Controller.py", line 139, in <module>
    _run(sys.argv)
  File "/apps/r/ansys/v193/SystemCoupling/PyLib/Controller.py", line 135, in _run
    _executeScript(options)
  File "/apps/r/ansys/v193/SystemCoupling/PyLib/Controller.py", line 107, in _executeScript
    kernel.commands.readScriptFile(scriptFile)
  File "PyLib/kernel/commands/__init__.py", line 31, in readScriptFile
  File "PyLib/kernel/commands/CommandManager.py", line 168, in readScriptFile
  File "inputfile.in", line 10, in <module>
    Solve()
  File "PyLib/kernel/commands/CommandDefinition.py", line 72, in func
  File "PyLib/kernel/commands/__init__.py", line 28, in executeCommand
  File "PyLib/kernel/commands/CommandManager.py", line 120, in executeCommand
  File "PyLib/cosimulation/externalinterface/cosim_commands/solver.py", line 94, in execute
  File "PyLib/cosimulation/solver/__init__.py", line 107, in solve
  File "PyLib/cosimulation/solver/__init__.py", line 254, in __stepControlled

Error:   File "PyLib/cosimulation/solver/__init__.py", line 358, in __iterate
Cortex received a fatal signal (SEGMENTATION VIOLATION).
Error Object: ()
  File "PyLib/cosimulation/solver/__init__.py", line 1112, in __reachSynchronizationPointAllActive
  File "PyLib/cosimulation/solver/__init__.py", line 1098, in __reachSynchronizationPoint
  File "PyLib/cosimulation/solver/__init__.py", line 1119, in __yieldControlTo
  File "PyLib/cosimulation/solver/__init__.py", line 1002, in __serveRequests
  File "PyLib/cosimulation/solver/__init__.py", line 1070, in __handleRequest
  File "PyLib/cosimulation/solver/__init__.py", line 1012, in __transmitExceptionToAllParticipants
cosimulation.solver.CosimulationError.CosimulationError
Shutting down compute-node processes

Steve posted this 31 May 2019

Thanks, we'll continue investigating with this new information.

Steve

  • Liked by
  • ansysuser
Steve posted this 03 June 2019

Hi,

This does look like a setup issue, as you mentioned. Can you zip the run directory and attach it so that I can have a look?

Steve

  • Liked by
  • ansysuser
ansysuser posted this 03 June 2019

Hello Steve,

I found the problem.  This is something your documentation team might want to fix to avoid future user confusion.  The oscillating plate FSI tutorial has several categories for how to run it as listed below.  Note that many users will select one of these tracts depending on their needs and my not even look at the other tracks.

When I follow the link you gave for, "Co-Simulation with Workbench Setup and Command-Line Execution" it says to write the the structural input using tools>Write Input Files, which results in only one file (mapdl.dat) for use in the simulation (on this page):

https://ansyshelp.ansys.com/account/secured?returnurl=/Views/Secured/corp/v193/sysc_tut/sysc_tut_oscplate_sci-commandline_setup_prepare_mech.html

That is what I did last week.  However, if I had chosen the other route, "Co-Simulation with Setup and Execution from the Command Line" instead, I would have discovered that I was supposed to also write another file, not mentioned in the Workbench Setup tutorial.  Here I am directed to right click on Transient in the model tree and "Write System Coupling Files" which is a needed file (mapdl.scp).  Once I put the mapdl.scp in the same HPC directory as the mapdl.dat file I had from last week, it seems to have run successfully.

Thank you for your time, Steve.  I appreciate it!

Close