Monday 17 February 2014

Compiling and setting up GAMESS

Small guide on how to setup the QM software GAMESS on a normal Ubuntu computer and work in parallel with multiple nodes (via sockets). Loosely this is based on this guide on how to compile GAMESS.

I'm going to pretend that you are working on Ubuntu 12.04 LTS, but I'm sure you can relate it to whatever distribution you are working on.

Download and compile


2. Run config


./config

and answer the questions. Answer them truthfully.

3. Compile DDI

DDI is used to run the GAMESS executable, compile it by


make ddi

4. Compile GAMESS

After compiling DDI and setting up the config, we can just write


make

and everything will happen automatically. You can add the flag "-j4" if you have 4 CPU's and want the compiling to go a little faster. It takes a few minutes.

Note: Compiling without Math library

Note on compiling without a math library, like MKL. In some versions of GAMESS the linking will fail with error message like "including missing subroutines for vector operations failed". This is what the math libraries are used for. For Ubuntu you can install  BLAS and LAPACK easily with


sudo apt-get install libblas3gf libblas-doc libblas-dev
sudo apt-get install liblapack3gf liblapack-doc liblapack-dev

Now we just need to change the ./liked script and add some compiler flags. If you are compiling with gfortran then you need to find and update the following two lines in ./lked under the gfortran section.


- set BLAS='blas.o'
- set MATHLIBS=' '
+ set BLAS=''
+ set MATHLIBS='-lblas -llapack '

and similar if you are using ifort to compile edit the section in lked and set the correct compiler flags.
and then it should liked alright.

Update rungms

To run GAMESS there is a included a run script in the root folder, which needs to be updated located in the beginning, for the scratch folder and GAMESS path. so, edit (using VIM, if you are awesome, EMACS, if you are not).


vi rungms

and set following paths to the correct (and obviously not my username);


set SCR=/home/charnley/scr
set USERSCR=/home/charnley/scr
set GMSPATH=/home/charnley/opt/gamess

Note: GAMESS doesn't work?!

Pro tip by +Anders Steen Christensen
If this is the first time you are trying to get GAMESS working on your Linux machine, you will need to set kernel.shmmax to 1610612736. This is done by either;


/sbin/sysctl -w kernel.shmmax=1610612736

or if you don't want to do the command (or don't have root access) every time you boot up, open the file called:


/etc/sysctl.conf

and add the following line:


kernel.shmmax=1610612736

Running the GAMESS testcase (exam)

Included in GAMESS is a list of input files to be tested to see if the software is working as it should. This is also useful to run if you do changes to the source code. Go to the root of the GAMESS folder and write.


./runall
./tests/standard/checktst

This will then output all the exams. If all is passed, then GAMESS should be working alright.


Setting up cluster calculation

If you want to have GAMESS working with a cluster system then there are a few things to change. This is to get the parallelization‎ with sockets to work.

1. Update rungms

You'll need to edit the part of rungms that checks the hostname of the current node. As present the rungms check if you are running calculations on any of the Iowa nodes (where GAMESS is developed), and you are probably not.

Find the following if-statement and switch;


if ($NCPUS > 1) then
    switch (`hostname`)

and locate the default switch case, and outcomment the "exit", just because the rungms doesn't regonize your hostname. If you are using a cluster system, you are probably using some kind of queuing system, like PBS og LoadLeveler. Here is what you need to change on the specific system;

1.1 PBS specific

PBS should be working out-of-the box. As long as you have fixed the above hostname exit. If you don't need the user-scratch file, you can set \$USERSCR to "/scratch/\$PBS_JOBID" as \$SCR, under


if ($SCHED == PBS) then

1.2 LoadLeveler specific

As default, rungms does not contain loadleveler default settings, so you will need to setup the following test.



if ($?LOADL_STEP_ID) set SCHED=LOADL

under 'batch schedular' section, and then add the case



if ($SCHED == LOADL) then
    set SCR=/scratch/$LOADL_STEP_ID
endif

If you don't use the $USERSCR output, you can set that too the local scratch folder as well.

2. Setting up input file

If you are running the calculation across multiple nodes, you'll need to tell the GAMESS executable how many nodes you are using, this is done simply by stating, fx. for 10 nodes;


 $gddi
  ngroup=10
 $end

 $system
  mwords=256
 $end

in the input file, with mwords being how much memory you want to allocate.

Remember: Remember that you need 1 space before the sections on GAMESS input files.

The end


And that's it. So simple and easy out-of-the-box. You should be running loads of QM calculations now.

Happy GAMESS'ing


4 comments:

  1. Always select "sockets" and never "MPI"!

    ReplyDelete
  2. Actually, PBS does _not_ work out of the box because the defaults are some obscure and based on a computer cluster at ISU in Ames.

    You should change

    if (`uname` == Linux) set NETEXT=".myri"

    into

    if (`uname` == Linux) set NETEXT=""

    because you do most likely not have a myrinet network and similarly you should set the line

    set spacer2=":netext="

    to

    set spacer2=""

    That will make it run on PBS.

    ReplyDelete
  3. Actually, there is more:

    1) For an FMO calculation, the ridiculous amount of memory you have specified (5000 mega words = 40 GB) is a serious waste. Use something more like 256 (which is 2 GB) or even half of that. This memory will be allocated per core depending on your setup and the type of calculation you are running.

    2) if you are not going to make something other than having 10 groups (10 nodes) then it is a waste to actually have the NGRFMO(1)=10 specified. NGRFMO is used to control the number of groups specified for each part of a calculation and if you do not specify anything, then NGRFMO(1)=NGROUP.

    3) in general it is a bad idea to set $USERSCR to $SCR when you are running on a cluster. There might be files you are interested in which will be lost when the queue-system cleans up after you.

    ReplyDelete
  4. Hello, I am trying to compile it with MKL library (the one that if free at Intel's website, Parallel Studio XE 2015). Please, see if you can help me with this error.
    My install.info file is:
    #!/bin/csh
    # compilation configuration for GAMESS
    # generated on lab221-cayo2
    # generated at Sex Ago 21 19:57:50 BRT 2015
    setenv GMS_PATH /home/lab221-cayo2/gamess
    setenv GMS_BUILD_DIR /home/lab221-cayo2/gamess
    # machine type
    setenv GMS_TARGET linux64
    # FORTRAN compiler setup
    setenv GMS_FORTRAN gfortran
    setenv GMS_GFORTRAN_VERNO 4.8
    # mathematical library setup
    setenv GMS_MATHLIB mkl
    setenv GMS_MATHLIB_PATH /opt/intel/mkl/lib/intel64
    setenv GMS_MKL_VERNO 12
    # parallel message passing model setup
    setenv GMS_DDI_COMM sockets
    # LIBCCHEM CPU/GPU code interface
    setenv GMS_LIBCCHEM false

    I the make the compddi step and the compall without problems, but when I try to execute the lked program, it get this message:
    ...

    Choices for some optional plug-in codes are
    Tinker/SIMOMM code skipped, using dummy file qmmm.o
    Valence Bond program VB2000 object files are vb2000.o vb2gms.o
    Nuclear Electron Orbital code skipped, using dummy file neostb.o
    Natural Bond Orbital (NBO) code skipped, using dummy file nbostb.o
    MPQC code skipped, using dummy file mpqcst.o

    The message passing libraries searched are
    /home/lab221-cayo2/gamess/ddi/libddi.a -lpthread

    Other libraries, including math libraries, to be searched are
    -Wl,--start-group /opt/intel/mkl/lib/intel64/libmkl_gf_ilp64.a /opt/intel/mkl/lib/intel64/libmkl_sequential.a /opt/intel/mkl/lib/intel64/libmkl_core.a -Wl,--end-group

    Linker messages (if any) follow...
    /opt/intel/mkl/lib/intel64/libmkl_core.a(mkl_aa_fw_load_orsl_lite_lib.o): na função `mkl_aa_fw_load_orsl_lite_lib':
    ../../../../serv/offload/framework/core/mkl_aa_fw_load_orsl_lite_lib.c:(.text+0xbe): referência indefinida para `dlopen'
    ../../../../serv/offload/framework/core/mkl_aa_fw_load_orsl_lite_lib.c:(.text+0xd9): referência indefinida para `dlsym'
    ../../../../serv/offload/framework/core/mkl_aa_fw_load_orsl_lite_lib.c:(.text+0xf7): referência indefinida para `dlsym'
    ../../../../serv/offload/framework/core/mkl_aa_fw_load_orsl_lite_lib.c:(.text+0x115): referência indefinida para `dlsym'
    ../../../../serv/offload/framework/core/mkl_aa_fw_load_orsl_lite_lib.c:(.text+0x133): referência indefinida para `dlsym'
    /opt/intel/mkl/lib/intel64/libmkl_core.a(load_library.o): na função `mkl_ueaa_prv_load_backend_lib':
    load_library.c:(.text+0x1d1): referência indefinida para `dlopen'
    load_library.c:(.text+0x1f3): referência indefinida para `dlvsym'
    load_library.c:(.text+0x21c): referência indefinida para `dlvsym'
    load_library.c:(.text+0x245): referência indefinida para `dlvsym'
    load_library.c:(.text+0x26e): referência indefinida para `dlvsym'
    load_library.c:(.text+0x297): referência indefinida para `dlvsym'
    /opt/intel/mkl/lib/intel64/libmkl_core.a(load_library.o):load_library.c:(.text+0x2c0): mais referências indefinidas para seguir `dlvsym'
    /opt/intel/mkl/lib/intel64/libmkl_core.a(mkl_libc_is_static.o): na função `mkl_serv_libc_is_static':
    ../../../../serv/kernel/mkl_libc_is_static.c:(.text+0x10): referência indefinida para `dladdr'
    collect2: error: ld returned 1 exit status

    Unfortunately, there was an error while linking GAMESS.
    0.2u 0.0s 0:00.34 97.0% 0+0k 0+76480io 0pf+0w

    Do you know what it is?

    ReplyDelete