Previous Thread
Next Thread
Print Thread
Page 1 of 2 1 2
CHARMM on GPU machine
#37230 12/11/18 01:54 AM
Joined: Jun 2017
Posts: 12
M
mala Offline OP
Forum Member
OP Offline
Forum Member
M
Joined: Jun 2017
Posts: 12
Hi,

We just compiled the non-free version of CHARMM enhanced for GPUs on a machine with 4 NVidia GPUs and 20 Intel CPU cores. When I execute the following command:

/usr/local/bin/mpirun -n 4 /usr/local/software/charmm-parallel/charmm/bin/charmm -i test_parallel_md.test > output.out

on a test MD simulation, I notice the following odd things:

1) The output file shows the output duplicated 4 times (this didn't happen on previously compiled parallel non-GPU versions I have)
2) The GPUs do not seem to be engaged at all
(running
watch -n 0.5 nvidia-smi
shows that the GPUs are not being used at all.
3) When I read the scarce online documentation for running CHARMM on GPUs (am I missing some obvious documentation?), it mentioned adding PARA commands or GRAPe commands, both of which just give me "error" messages -- that these commands aren't recognized.

I definitely obtained the GPU-enhanced (non-free) CHARMM 43b1 officially from Harvard, but it seems like it doesn't even engage my GPUs or understand the commands it should. Can you please provide any insight as to what I might be missing?

Thanks so much!

Mala

Re: CHARMM on GPU machine
mala #37231 12/11/18 07:32 AM
Joined: Sep 2003
Posts: 4,784
Likes: 2
Forum Member
Online Content
Forum Member
Joined: Sep 2003
Posts: 4,784
Likes: 2
Even the free CHARMM version has GPU support through OpenMM.

You have to install CHARMM with the appropriate options for GPU, either using DOMDEC (domedc.doc; good for very large systems) or OpenMM (openmm.doc; probably faster for medium sized systems).

With OpenMM you do not need to run on more than one CPU since all the work is done on the GPU(s).

Neither install.doc or gpu.doc are really up-to-date. The following helps you find more information on CHARMM and GPUs:
cd ~charmm/doc
grep -i gpu *.doc


Lennart Nilsson
Karolinska Institutet
Stockholm, Sweden
Re: CHARMM on GPU machine
mala #37233 12/11/18 05:39 PM
Joined: Sep 2003
Posts: 8,486
rmv Online Content
Forum Member
Online Content
Forum Member
Joined: Sep 2003
Posts: 8,486
Additional features must be specified at compile time to enable either DOMDEC_GPU or OpenMM code. They work very differently from each other, and have different strengths and weaknesses.

The domdec.doc and openmm.doc files have not been kept up to date very well, either. I've had to play mix and match with CUDA and OpenMM versions over the past few years, depending on the CHARMM release version.


Rick Venable
computational chemist

Re: CHARMM on GPU machine
mala #37237 12/19/18 11:55 AM
Joined: Jun 2017
Posts: 12
M
mala Offline OP
Forum Member
OP Offline
Forum Member
M
Joined: Jun 2017
Posts: 12
Hi -- ok, so we did the following:

1. Installed CUDA 10.0
2. Compiled and installed OpenMM (Pre-Requisite for Charmm).
3. Compiled and installed Swig, PCRE, OpenCL, Flex, Doxygen and GCC 5.4)
4. Compiled and installed Charmm.

When I run
/usr/local/bin/mpirun -np 4 /usr/local/software/charmm-parallel/charmm/bin/charmm -i test_parallel_md.test > output.out

It now does run in parallel but only with the CPUs. It still does not engage the GPUs. What am I doing wrong? How do I get it to use the GPUs?

thank you.

Re: CHARMM on GPU machine
mala #37238 12/19/18 01:29 PM
Joined: Sep 2003
Posts: 4,784
Likes: 2
Forum Member
Online Content
Forum Member
Joined: Sep 2003
Posts: 4,784
Likes: 2
1. You don't need to use -np 4 (you only need one CPU);
2. Use an input file that activates OpenMM/GPU, see openmm.doc or testcases ~charmm/test/c*test/omm*.inp.


Lennart Nilsson
Karolinska Institutet
Stockholm, Sweden
Re: CHARMM on GPU machine
mala #37245 12/20/18 09:16 PM
Joined: Jun 2017
Posts: 12
M
mala Offline OP
Forum Member
OP Offline
Forum Member
M
Joined: Jun 2017
Posts: 12
Hi,

Thanks. However, when I try to run on the test examples -- in this case c37test/omm_dynamics.inp, I get a LOT of errors, but the first of which is:

terminate called after throwing an instance of 'OpenMM::OpenMMException'
what(): There is no registered Platform called "CUDA"

CUDA has in fact been installed. The installation instructions in doc/openmm.doc were followed precisely. Any ideas?

THanks,

M

Re: CHARMM on GPU machine
mala #37247 12/21/18 08:16 AM
Joined: Sep 2003
Posts: 4,784
Likes: 2
Forum Member
Online Content
Forum Member
Joined: Sep 2003
Posts: 4,784
Likes: 2
Only certain combinations of CHARMM, OpenMM and CUDA versions work together, so this may be due to a version mismatch.

I just checked my own installation of CHARMM 43b1, and it works with this:

ifort 17.0.2
cuda 8.0
OpenMM 7.1.1

install.com em64t M openmm


Last edited by lennart; 12/23/18 03:29 PM. Reason: added charmm versiion

Lennart Nilsson
Karolinska Institutet
Stockholm, Sweden
Re: CHARMM on GPU machine
mala #37593 08/07/19 02:37 PM
Joined: Jul 2019
Posts: 25
C
Forum Member
Offline
Forum Member
C
Joined: Jul 2019
Posts: 25
Hey guys, just wanted to add my own setup for those of us just starting with charmm and the omm interface.

cuda 9.0
OMM 7.3.1 from Anaconda (current latest)
charmm 43b2

Ran successfully on a small metal/peptide system on a cluster with Slurm on a single Tesla k80, ~80ns/day at this stage.

Last edited by Corey Taylor; 08/07/19 02:37 PM.
Re: CHARMM on GPU machine
mala #37594 08/07/19 03:27 PM
Joined: Sep 2003
Posts: 8,486
rmv Online Content
Forum Member
Online Content
Forum Member
Joined: Sep 2003
Posts: 8,486
CUDA 9.2 is strongly recommended over 9.0, for performance reasons.


Rick Venable
computational chemist

Re: CHARMM on GPU machine
mala #37595 08/08/19 07:22 AM
Joined: Jul 2019
Posts: 25
C
Forum Member
Offline
Forum Member
C
Joined: Jul 2019
Posts: 25
Cheers Rick, have recompiled CHARMM to use 9.2 and now it's in the queue for a test calculation. What magnitude of speedup could one expect?

Page 1 of 2 1 2

Moderated by  lennart, rmv 

Link Copied to Clipboard
Powered by UBB.threads™ PHP Forum Software 7.7.4
(Release build 20200307)
Responsive Width:

PHP: 5.6.33-0+deb8u1 Page Time: 0.015s Queries: 35 (0.008s) Memory: 0.9860 MB (Peak: 1.1173 MB) Data Comp: Off Server Time: 2020-08-15 08:08:27 UTC
Valid HTML 5 and Valid CSS