Page 1 of 3 1 2 3 >
Topic Options
#37402 - 03/20/19 11:55 AM Free version of CHARMM in parallel
pmj Offline
Forum Member

Registered: 01/15/19
Posts: 34
Dear all,

I am using the free version of charmm (v43) which I complied with:

./install.com gnu xlarge x86_64
./install.com gnu M

I am using GBSW solvation model and I was doing some testing to see how fast was the procedure. Also, my system is constrained with EMAP.

I got really strange results, so I assume CHARMM has not complied properly.

When I run it on 4, 8, 12 and 16 cores the CPU time shows it takes more to run the script on 16 cores than on 4:

NP 4 8 12 16
CPU time/min 25.53 27.27 29.09 29.87

I also tried with ./install.com gnu M MPICH ADM64 and some other like ./install.com gnu xlarge ifort x86_64, and than recompile it with mpi option, but it does not work.

I tried to recompile it with em47t, but I could not do it successfully as it was possible to compile it only in the serial version.

Can anyone help me with this issue? I suppose I am just not doing it properly, so I would appreciate the help.

Thank you!

Top
#37403 - 03/20/19 12:03 PM Re: Free version of CHARMM in parallel [Re: pmj]
rmv Offline

Forum Member

Registered: 09/17/03
Posts: 8336
Loc: 39 03 48 N, 77 06 54 W
Have you run the test case suite?

Are the cores all on a single machine?

It's entirely possible that GBSW and EMAP may not be as well optimized as other parts of the code.

The 'em64t' option requires a licensed Intel compiler; it does work with MPI libraries, but the setup may be a bit different, and an Intel-built version of the mpi.mod flie is required.
_________________________
Rick Venable
computational chemist


Top
#37404 - 03/20/19 12:23 PM Re: Free version of CHARMM in parallel [Re: pmj]
pmj Offline
Forum Member

Registered: 01/15/19
Posts: 34
Dear Prof. Venable,

I run the test cases and there were no failures, unfinished or abnormally finished jobs. (Number of test cases is 168). I am not sure if this good or bad.

I tested the following:

./test.com M 8 gnu

When I tried to do the same with ./parallel-tests it did not really work.

I also tried with ./parallel-test ../pathtocharmm/gnu_M and it showed no apparent error.

All the cores are on the single machine.

Top
#37405 - 03/20/19 12:45 PM Re: Free version of CHARMM in parallel [Re: pmj]
rmv Offline

Forum Member

Registered: 09/17/03
Posts: 8336
Loc: 39 03 48 N, 77 06 54 W
There are a lot more than 168 test cases, close to 600 in all.

You should probably look at the output from test cases with the strings 'emap' and 'gbsw' in the file name.

The command "./test.com M 8 gnu" should be okay for testing the parallel code in CHARMM, although some tests will be skipped.

The ./parallel-tests script allows running multiple test cases at once, and is not specifically related to the parallel energy evaluation code in CHARMM.
_________________________
Rick Venable
computational chemist


Top
#37406 - 03/20/19 12:47 PM Re: Free version of CHARMM in parallel [Re: pmj]
pmj Offline
Forum Member

Registered: 01/15/19
Posts: 34
Dear Prof. Venable,

I wrote one mistake, there were 168 test cases that have not been run, everything else was okay.

I will check the files as you wrote.

Thank you

Top
#37407 - 03/20/19 01:02 PM Re: Free version of CHARMM in parallel [Re: pmj]
pmj Offline
Forum Member

Registered: 01/15/19
Posts: 34
Dear Prof. Venable,

I checked all the outputs regarding the gbsw and emap and except for few warnings, there were no major errors (normal termination).

However, I run my script without gbsw and emap and I obtained following results:

NP 4 8 12 16
CPU time/min

1.12 min. 37s 340ms 34s 210ms 37s 100ms
57s 830ms 36s 600ms 46s 540ms 35s 480ms

It seems bit odd that they are almost the same for all cases or longer on 16 cores.

Thanks

Top
#37408 - 03/20/19 01:51 PM Re: Free version of CHARMM in parallel [Re: pmj]
rmv Offline

Forum Member

Registered: 09/17/03
Posts: 8336
Loc: 39 03 48 N, 77 06 54 W
It should be noted that more cores is not always faster; that kind of behavior can be observed for systems with too few atoms. When the energy calculations done by esch core are very fast, the communication between cores starts to dominate, and using more cores becomes slower.
_________________________
Rick Venable
computational chemist


Top
#37409 - 03/20/19 01:57 PM Re: Free version of CHARMM in parallel [Re: pmj]
pmj Offline
Forum Member

Registered: 01/15/19
Posts: 34
Dear Prof. Venable,

the system is rather big as it has approximately 120 000 atoms.

Is there any additional test I could perform to see whether parallel charmm is working properly?

Thank you so much

Top
#37412 - 03/21/19 04:54 AM Re: Free version of CHARMM in parallel [Re: pmj]
SNOW001 Offline
Forum Member

Registered: 11/12/18
Posts: 16
Have you ever check the charmm/build/gnu_M/pref.dat, if it contains term: MPI,PARALLEL,PARAFULL, I think charmm can parallel.
Then I recommend you check the output file with different cores, If the out energy is exactly same after some steps (such as 1ns), I guess each core do same things repeatly instead of each core do part of the run.
Also, you can check the charmm/build/gnu_M/new.log file, If they are some 'Error or Warning', Maybe you compiled it wrong.


Edited by SNOW001 (03/21/19 05:23 AM)

Top
#37414 - 03/21/19 05:51 AM Re: Free version of CHARMM in parallel [Re: SNOW001]
pmj Offline
Forum Member

Registered: 01/15/19
Posts: 34
Hi!

I checked the pref.data file and it containes all three keywords. There are no errors in new.log file as well.

I will try and see if the energies are the same as you said when I compile it with ./install gnu M. Perhaps it has not complied properly.

I was trying to compile CHARMM with ./install.com gnu M MPICH AMD64, but when I was trying to give the input where mpif.h file is located it is not recognised (like /path/mpi/intel64/lib or /path/mpi/intel64/).

Also, where do I change number of cores in the test cases?

Thank you for your help.

Current operating system I am using: Linux-3.16.0-4-amd64(x86_64)


Edited by pmj (03/21/19 06:15 AM)

Top
#37417 - 03/21/19 08:14 AM Re: Free version of CHARMM in parallel [Re: pmj]
SNOW001 Offline
Forum Member

Registered: 11/12/18
Posts: 16
./install.com gnu M is ok, I have not try ./install.com gnu M MPICH AMD64. What compiler do you use ?
When do ./test.com gnu_M , you may see something like this: ~/charmm/exec/gnu_M/charmm -input ~/char mm/test/c31test/gbsw.inp -prevclcg >& output/gbsw.out, it use one core, if you want 8 cores can try: mpirun -np 8 ~/charmm/exec/gnu_M/charmm -input ~/charmm/test/c31test/gbsw.inp -prevclcg >& output/c 31test/gbsw.out

Top
#37420 - 03/21/19 08:49 AM Re: Free version of CHARMM in parallel [Re: pmj]
lennart Offline

Forum Member

Registered: 09/25/03
Posts: 4724
Loc: ~ 59N, 15E
In CHARMM give the command pref. It lists the keywords used when CHARMM was installed. If you see MPI PARALLEL in the list you have a parallel version.

Try test.com without arguments. It will tell you how to run the testcases in parallel.
_________________________
Lennart Nilsson
Karolinska Institutet
Stockholm, Sweden

Top
#37424 - 03/21/19 09:48 AM Re: Free version of CHARMM in parallel [Re: pmj]
tru Offline
Forum Member

Registered: 01/30/04
Posts: 8
Loc: France
$ /bin/echo -e '*\nshow\nstop'|/c7/shared/c43b2/gcc-openmpi-2.1.5-cuda-9.2.148_396.37/bin/charmm|grep -e 'PARA' -e 'MPI'
MISCOM: label: MPI value: 1
MISCOM: label: PARAFULL value: 1
MISCOM: label: PARALLEL value: 1

Top
#37430 - 03/22/19 04:24 AM Re: Free version of CHARMM in parallel [Re: pmj]
pmj Offline
Forum Member

Registered: 01/15/19
Posts: 34
Dear all,

I made all the tests as you have suggested and it seems CHARMM has complied properly.

From the data I obtained, it seems that GBSW and EMAP, however, are not really parallelised. Is it possible to actually parallelised GBSW and EMAP by switching on some options in the code?

I was wondering if EEF1 is parallelised perhaps and if it is possible to use emap constraints with it?

Thank you all for help.



Edited by pmj (03/22/19 07:00 AM)

Top
#37431 - 03/25/19 04:48 AM Re: Free version of CHARMM in parallel [Re: pmj]
SNOW001 Offline
Forum Member

Registered: 11/12/18
Posts: 16
Both GBSW and EEF1 can parallel, I am not sure about EMAP. When view file charmm/source/emap/emapdock.src emapfield.src emapop.src emapsubs.src, only emapsubs.src has key words:
#if KEY_PARALLEL==1
use parallel
I am not sure which command to call when you run (may be it will display in run.out if you add 'PRNLEV 15' in your run.inp file, or may not).
If emapdock.src, emapfield.src or emapop.src were used, I guess it cannot parallel. Actually, I'm not familiar with EMAP.

Top
#37432 - 03/25/19 05:55 AM Re: Free version of CHARMM in parallel [Re: SNOW001]
pmj Offline
Forum Member

Registered: 01/15/19
Posts: 34
Hi!

First of all thanks for your help.

I checked the files for emap and as you said and in the emapsubs.src:

#if KEY_PARALLEL==1
NSTA=(NEMAT*MYNOD)/NUMNOD+1
NEND+(NEMAT*MYNODP)/NUMNOD
#else /**/
NSTA=1
NEND=NEMAT

The thing is when I test the same scripts with gbsw or eef1 and emap (in all combinations w or w/o gbsw or eef1 and emap) and I measure the time I get random values (the time does not decrease when I use more cores.

My system is quite large (120 000 atoms) so it is really important for me to have gbsw or eef1 and emap parallelised properly. Initially, I thought I complied charmm wrongly, but I checked some scripts with explicit solvent and they are rather fast with 16 cores (with 4 cores it is approx 10 min and with 16 is around 4-5 min). I do not get the same with gbsw, eef1 or when I use emap.

Also, I was wondering is it possible to comple free version charmm with

./install gnu xxlarge ifort x86_64 i8 D M MPICH?

Thank you!


Edited by pmj (03/25/19 07:00 AM)

Top
#37433 - 03/25/19 07:58 AM Re: Free version of CHARMM in parallel [Re: pmj]
SNOW001 Offline
Forum Member

Registered: 11/12/18
Posts: 16
Have you try only using gbsw, or only using eef1, I think they can parallel.


Edited by SNOW001 (03/25/19 08:00 AM)

Top
#37434 - 03/25/19 08:09 AM Re: Free version of CHARMM in parallel [Re: SNOW001]
pmj Offline
Forum Member

Registered: 01/15/19
Posts: 34
I tested it and for gbsw I get normal reduction in cpu time when I remove mandatory nonbonded keywords.

For eef1 it seems it not parallelised if I use emap.

Top
#37435 - 03/25/19 08:23 AM Re: Free version of CHARMM in parallel [Re: pmj]
SNOW001 Offline
Forum Member

Registered: 11/12/18
Posts: 16
Could you paste your input file, may be I can test it with my computer.


Edited by SNOW001 (03/25/19 08:24 AM)

Top
#37436 - 03/26/19 05:42 AM Re: Free version of CHARMM in parallel [Re: SNOW001]
pmj Offline
Forum Member

Registered: 01/15/19
Posts: 34
Hi!

Thank you for testing it.

Since I could not send the files through forum here I uploaded it to:

https://ufile.io/pmsao

Top
#37437 - 03/26/19 10:17 AM Re: Free version of CHARMM in parallel [Re: pmj]
SNOW001 Offline
Forum Member

Registered: 11/12/18
Posts: 16
Hello,

I run your file with different cores, when include:
{
stream gbsw_2.inp
stream dyna_2.inp (100 steps)
}
the time consumed:
1 core:
$$$$$ JOB ACCOUNTING INFORMATION $$$$$
ELAPSED TIME: 18.00 MINUTES
CPU TIME: 18.01 MINUTES
4 cores:
$$$$$ JOB ACCOUNTING INFORMATION $$$$$
ELAPSED TIME: 9.08 MINUTES
CPU TIME: 8.80 MINUTES


and when include:
{
stream gbsw_2.inp
stream emap.str
stream dyna_2.inp (100 steps)
}
the time consumed:
1 core:
$$$$$ JOB ACCOUNTING INFORMATION $$$$$
ELAPSED TIME: 18.62 MINUTES
CPU TIME: 18.63 MINUTES
4 cores:
$$$$$ JOB ACCOUNTING INFORMATION $$$$$
ELAPSED TIME: 10.50 MINUTES
CPU TIME: 9.91 MINUTES



Edited by SNOW001 (03/26/19 10:20 AM)

Top
#37438 - 03/27/19 04:20 AM Re: Free version of CHARMM in parallel [Re: SNOW001]
pmj Offline
Forum Member

Registered: 01/15/19
Posts: 34
Thank you so much for testing it. I see now that there is nothing wrong with the compilation I suppose. This the CPU time I get while running the script on 1, 4, 8, 12 and 16 cores. But in the end, I find it really slow on 16 cores for approximately 120 000 atoms (I managed to do around 10 ns/per day). I also tested the EEF1 with the EMAP and it seems not parallelised at all. Perhaps I will look into some other options.

GBSW EMAP NProc 1 4 8 12 16

+ + CPU time/min 23.38 11.16 10.17 12.28 10.2
+ - CPU time/min 21.12 9.40 8.13 11.16 8.17

Once more thank you for your help!
Have a great day

Top
#37439 - 03/27/19 04:28 AM Re: Free version of CHARMM in parallel [Re: pmj]
lennart Offline

Forum Member

Registered: 09/25/03
Posts: 4724
Loc: ~ 59N, 15E
Note that for benchmarking parallel jobs it is generally the elapsed time that is important (ie, the time YOU have to wait), not the CPU-time.
_________________________
Lennart Nilsson
Karolinska Institutet
Stockholm, Sweden

Top
#37440 - 03/27/19 04:37 AM Re: Free version of CHARMM in parallel [Re: lennart]
pmj Offline
Forum Member

Registered: 01/15/19
Posts: 34
Dear Prof. Nilsson,

I used the time ./run.sh command for each test. The time in the table is real time.

From what I saw in the CHARMM output files, the elapsed time and the cpu time are more or less the same.

Top
Page 1 of 3 1 2 3 >

Moderator:  John Legato, lennart