CHARMM Development Project
Posted By: mala CHARMM on GPU machine - 12/11/18 01:54 AM

We just compiled the non-free version of CHARMM enhanced for GPUs on a machine with 4 NVidia GPUs and 20 Intel CPU cores. When I execute the following command:

/usr/local/bin/mpirun -n 4 /usr/local/software/charmm-parallel/charmm/bin/charmm -i test_parallel_md.test > output.out

on a test MD simulation, I notice the following odd things:

1) The output file shows the output duplicated 4 times (this didn't happen on previously compiled parallel non-GPU versions I have)
2) The GPUs do not seem to be engaged at all
watch -n 0.5 nvidia-smi
shows that the GPUs are not being used at all.
3) When I read the scarce online documentation for running CHARMM on GPUs (am I missing some obvious documentation?), it mentioned adding PARA commands or GRAPe commands, both of which just give me "error" messages -- that these commands aren't recognized.

I definitely obtained the GPU-enhanced (non-free) CHARMM 43b1 officially from Harvard, but it seems like it doesn't even engage my GPUs or understand the commands it should. Can you please provide any insight as to what I might be missing?

Thanks so much!

Posted By: lennart Re: CHARMM on GPU machine - 12/11/18 07:32 AM
Even the free CHARMM version has GPU support through OpenMM.

You have to install CHARMM with the appropriate options for GPU, either using DOMDEC (domedc.doc; good for very large systems) or OpenMM (openmm.doc; probably faster for medium sized systems).

With OpenMM you do not need to run on more than one CPU since all the work is done on the GPU(s).

Neither install.doc or gpu.doc are really up-to-date. The following helps you find more information on CHARMM and GPUs:
cd ~charmm/doc
grep -i gpu *.doc
Posted By: rmv Re: CHARMM on GPU machine - 12/11/18 05:39 PM
Additional features must be specified at compile time to enable either DOMDEC_GPU or OpenMM code. They work very differently from each other, and have different strengths and weaknesses.

The domdec.doc and openmm.doc files have not been kept up to date very well, either. I've had to play mix and match with CUDA and OpenMM versions over the past few years, depending on the CHARMM release version.
Posted By: mala Re: CHARMM on GPU machine - 12/19/18 11:55 AM
Hi -- ok, so we did the following:

1. Installed CUDA 10.0
2. Compiled and installed OpenMM (Pre-Requisite for Charmm).
3. Compiled and installed Swig, PCRE, OpenCL, Flex, Doxygen and GCC 5.4)
4. Compiled and installed Charmm.

When I run
/usr/local/bin/mpirun -np 4 /usr/local/software/charmm-parallel/charmm/bin/charmm -i test_parallel_md.test > output.out

It now does run in parallel but only with the CPUs. It still does not engage the GPUs. What am I doing wrong? How do I get it to use the GPUs?

thank you.
Posted By: lennart Re: CHARMM on GPU machine - 12/19/18 01:29 PM
1. You don't need to use -np 4 (you only need one CPU);
2. Use an input file that activates OpenMM/GPU, see openmm.doc or testcases ~charmm/test/c*test/omm*.inp.
Posted By: mala Re: CHARMM on GPU machine - 12/20/18 09:16 PM

Thanks. However, when I try to run on the test examples -- in this case c37test/omm_dynamics.inp, I get a LOT of errors, but the first of which is:

terminate called after throwing an instance of 'OpenMM::OpenMMException'
what(): There is no registered Platform called "CUDA"

CUDA has in fact been installed. The installation instructions in doc/openmm.doc were followed precisely. Any ideas?


Posted By: lennart Re: CHARMM on GPU machine - 12/21/18 08:16 AM
Only certain combinations of CHARMM, OpenMM and CUDA versions work together, so this may be due to a version mismatch.

I just checked my own installation of CHARMM 43b1, and it works with this:

ifort 17.0.2
cuda 8.0
OpenMM 7.1.1 em64t M openmm

Posted By: Corey Taylor Re: CHARMM on GPU machine - 08/07/19 02:37 PM
Hey guys, just wanted to add my own setup for those of us just starting with charmm and the omm interface.

cuda 9.0
OMM 7.3.1 from Anaconda (current latest)
charmm 43b2

Ran successfully on a small metal/peptide system on a cluster with Slurm on a single Tesla k80, ~80ns/day at this stage.
Posted By: rmv Re: CHARMM on GPU machine - 08/07/19 03:27 PM
CUDA 9.2 is strongly recommended over 9.0, for performance reasons.
Posted By: Corey Taylor Re: CHARMM on GPU machine - 08/08/19 07:22 AM
Cheers Rick, have recompiled CHARMM to use 9.2 and now it's in the queue for a test calculation. What magnitude of speedup could one expect?
Posted By: rmv Re: CHARMM on GPU machine - 08/08/19 02:59 PM
Not sure, but I read some reports and heard first hand accounts of a performance regression with 9.0 that was fixed with 9.2
Posted By: Corey Taylor Re: CHARMM on GPU machine - 08/09/19 10:39 AM
Yep and looks to be a significant hit using 9.0 vs 9.2. Again, thanks very much for the tip!
© CHARMM forums