profile fortran + mpi

어떻게 fortran+mpi 프로그램을 프로파일링 하는지 공부.

인터넷을 찾아보니 주로 사용하는 방법은 

(1) gprof 과 gprof2dot 조합 : 컴파일 시 -pg option필요

(2) valgrind, callgrind, kcashegrind 조합 : 컴파일 시 -g option 필요. 

인 것 같다. 

How to use gprof 

(1) compile code with option -pg 

(2) run the code  : (mpirun generate multiple gmon.out )

    export GMON_OUT_PREFIX=gmon.out  #optional? 

    mpirun -np 3 ./run 

(3) once gmon.out file is created, run gprof.  

    gprof run  gmon.out.xxx > prof_result.txt 

(4) reading prof_result.txt seems to show which subroutine is most time-consumming. 

Binary file problem with intel fortran (ifort) and gfortran (GNU fortran)

 I found that reading a binary file created with the same source file but compiled by ifort and gfortan give different results. By searching internet, I found that the way binary file is written in ifort is not the same sa gfortran. Thus, to make the same code gives the same results, one have to use open binary files with ACCESS='stream', FORM='unformatted'. 

• Non-standard: open(unit=20,file=fname,form='binary')
• Standard: open(unit=20,file=fname,form='unformatted',access='stream',status='replace')

But, this seems to be not the full story. 

I need to check the correct way to convert following gfortran code to ifort code.

 open(pin_file_id, file=trim(pin_file), status='unknown',form='unformatted', &

                 access='direct',action='write',recl=nsize,iostat=ios)

* After testing, following seems to be equivalent to the above one. 

open(pin_file_id, file=trim(pin_file), status='unknown',form='BINARY', &

                 access='direct',action='write',recl=nsize,iostat=ios)

* According to ChatGPT, ifort and gfortran interpret differently the recl= argument in open statement. In gfortran, 'recl' is interpreted as length of bytes. In ifort, 'recl' is interpreted as length of 4 bytes. Thus, to get the same results, if "recl= nsize" in gfortran , "recl= nsize/4" in ifort have to be used. Or/And(?) one have to use compile option in "ifort -assume byterecl .

This also imply that the "recl" in gfortran must be a multiple of 4. 

* But, modifying the source code depending on the compiler or machine is not desirable. access='stream' option without 'recl=' parameter will behave similarily in both ifort and gfortran. keep form='unformatted'.  status='replace' create a new file if it does not exist, or replace it if it exist. Thus, it may be better than status='unknown' to ensure freshfile in each run. (Or default status='unknown' may be okay??)

But, to access specific data, one have to use 'pos=' parameter which should be counted in whole file. For example, if data is stored as  'real(8), integer(4), real(4)', to read these 

read(file_id, pos=1) real_value

read(file_id, pos=9) int_value

read(file_id, pos=13) real4_value

In other words, 'pos' specify the starting byte position, and type of variable specify the length of the data to read. 

compile problems of various manybody codes (in Ubuntu)

List of compile problems for various manybody nuclear physics codes. 

 (1) BigstickPublick : https://github.com/cwjsdsu/BigstickPublick

            gfortran had problems with MPI commands. 

        --> ifort could compile without problem

        or add compile option "-fallow-argument-mismatch"

 (2) imsrg : https://github.com/ragnarstroberg/imsrg

               missing libraries have to be installed. But, python binding was problem. 

        --> include python3 path in Makefile. (replace 'python-config' into 'python3-config') 

       (2-1) to build : goto "build/" and run "cmake .." 

                then "make install" 

       (2-2) After compile, one can run  "imsrg++  [arguments]" 

               Python script to specify command line arguments can be found in "work/" folder. 

       (2-3)  To get documentation, run "doxygen  [.dox file]" in "doc/" folder.  

 (3) NuHamil : https://github.com/Takayuki-Miyagi/NuHamil-public

          *       no submodule

        --> "git submodule init" and "git submodule update"               

            *       missing hdf5.mod 

        --> add include path " -I/usr/include/hdf5/serial" 

               (first install "sudo apt install libhdf5-dev hdf5-tools")

             *      cannot find -lhdf5_fortran 

        --> add path "-L/usr/lib/x86_64-linux-gnu/hdf5/serial" 

             * may need additional libraries installed 

                      "sudo apt install libgsl-dev" 

(4) HartreeFock:  https://github.com/Takayuki-Miyagi/HartreeFock 

                    no submodule 

        --> same solution as above 

                     argument mismatch 

        --> edit Makefile, Debug=off and add compile option "-fallow-argument-mismatch"

 Tips: In general, if some library is already installed but could not be found, solutions are usually add correct path/location. How to find the path os installed library in Ubuntu? 

 try " dpkg --get-selections | grep -v deinstall | grep [name]" 

then " dpkg -L packagename "to find the path.

Sorting of an array in FORTAN (using MINLOC)

There is no intrinsic sorting function in FORTAN.
Though it is easy to write a sorting program by using various sorting algorithm
or import the exising codes from internet,
one of the simple method is to use MINLOC intrinsic function in FORTRAN.
(Though it may be not an efficient one, it is short.)

Suppose there is an array size N, narray(1:N). Then, one can sort the array by

Do i=1,N
  l = MINLOC(narray(i:N),1)
  temp = narray(i)
  narray(i)=narray(i+l-1)
  narray(i+l-1)= temp
End do

Now, narray(1:N) is sorted by its values.

If one want to keep track of original index, prepare another array, index(1:N).

Do i=1,N
 index(i)=i
End Do
Do i=1,N
  l = MINLOC(narray(i:N),1)
  temp = narray(i)
  narray(i)=narray(i+l-1)
  narray(i+l-1)= temp

  temp=index(i) 

  index(i)=index(i+l-1)

  index(i+l-1)=temp

End do

Now, index(1:N) is the index in original array.

Intel Math Kernel Library (MKL) and FFTW3 링크하기

먼저 MKL library의 위치를 찾을 것. 
MKL library에 FFTW가 포함되어 있다고 하지만 FFTW의 위치도 알아둘 것.

MKL libary를 사용하는 방법은 두가지가 있다.

(1) -mkl 옵션을 사용하는 방법
(2) 모든 옵션을 일일이 지정해 주는 방법
-mkl 옵션의 경우 default 설정이 무엇인가에  따라 달라지게 된다.일반적인 경우라면,

ifort test.f90 -mkl -o test_f
icc test.c -mkl -o test_c

만으로 충분하고 FFTW를 사용하는 경우에는 

icc test.c -w -DMKL_DOUBLE -I/opt/intel/Compiler/11.1/069/mkl/include/fftw -mkl -o test_c

ifort -I/opt/intel/Compiler/11.1/069/mkl/include/fftw test.f90 -mkl -o test_f

icc test.c -I/opt/intel/Compiler/11.1/069/mkl/include/fftw -mkl -o test_c

ifort test.f90 -I/opt/intel/Compiler/11.1/069/mkl/include/fftw -mkl -o test_f

와 같이 include path를 지정해 주는 것이 좋다고 한다. 

-mkl 옵션을 쓰지 않으면 아래와 같이 일일이 써 주어야한다. 

icc test.c -I/opt/intel/Compiler/11.1/069/mkl/include -I/opt/intel/Compiler/11.1/069/mkl/include/fftw
-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm 
-L/opt/intel/Compiler/11.1/069/mkl/lib/em64t -o c_test 

ifort test.f90 -I/opt/intel/Compiler/11.1/069/mkl/include -I/opt/intel/Compiler/11.1/069/mkl/include/fftw 
-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm 
-L/opt/intel/Compiler/11.1/069/mkl/lib/em64t -o f_test


(주의1) -lmkl_intel_lp64 는 64bit cpu 용 libary를 사용한다는 뜻이고,  
-lmkl_intel_ilp64 는 64bit cpu 용 libary를 사용하되 
정수도 64bit인 libary를 사용한다는 뜻으로 compile 할 때 -i8 option을 추가해 주어야 한다.   
ilp64를 사용할 경우 FFTW와 integer type이 다를 수 있으므로 주의해야 한다.

(주의2) FFTW2 를 사용할 경우는 option을 다르게 주어야한다.

icc test2.c -o test2_c -I/opt/intel/Compiler/11.1/069/mkl/include/fftw 
-L/opt/intel/Compiler/11.1/069/mkl/lib/em64t -lfftw2xc_intel -mkl 

ifort test2.f -o test2_f -I/opt/intel/Compiler/11.1/069/mkl/include/fftw 
-L/opt/intel/Compiler/11.1/069/mkl/lib/em64t -lfftw2xf_intel -mkl

만약 외부 FFTW library를 사용하는 경우에는 아래와 같이 사용한다.

ifort -O3 -o fftw_example.exe fftw_example.f -I/u/username/fftw/include -L/u/username/fftw/lib -lfftw3

참고:

http://geco.mines.edu/software/mkl/fftw/

http://www.nas.nasa.gov/hecc/support/kb/MKL-FFTW-Interface_204.html



(Details on the -mkl option )

-mkl[=]
          link to the Intel(R) Math Kernel Library (Intel(R) MKL) and
          bring in the associated headers
            parallel   - link using the threaded Intel(R) MKL libraries.
                         This is the default when -mkl is specified
            sequential - link using the non-threaded Intel(R) MKL libraries
            cluster    - link using the Intel(R) MKL Cluster libraries plus
                         the sequential Intel(R) MKL libraries

The libraries that are linked in for:

    * -mkl=parallel

          --start-group \
          -lmkl_solver_lp64 \
          -lmkl_intel_lp64 \
          -lmkl_intel_thread \
          -lmkl_core \
          -liomp5 \
          --end-group \

    * -mkl=sequential

          --start-group \
          -lmkl_solver_lp64_sequential \
          -lmkl_intel_lp64 \
          -lmkl_sequential \
          -lmkl_core \
          --end-group \

    * -mkl=cluster

          --start-group \
          -lmkl_solver_lp64 \
          -lmkl_intel_lp64 \
          -lmkl_cdft_core \
          -lmkl_scalapack_lp64 \
          -lmkl_blacs_lp64 \
          -lmkl_sequential \
          -lmkl_core \
          -liomp5 \
          --end-group \

Fortran : string 관련 tips

1. Change between numbers and characters(string)

  (1) Use CHAR(48+num) trick: However this works only for 1 digit number

  (2) Use internal file method :

* To convert number into strings, write

          WRITE(string_name,format) number

          (be careful that we need to use I3.3 instead of I3 to

          put zeroes on the left of number )

        * To convert strings into number, read

         READ(string_name,format) number

   (3) If we need only part of  string, use slicing

         string(begin:end)

       *In case of one character, string(num) does not work.
        Have to use

         string(num:num)  

   예를 들어서 길이가 달라지는 변수에 대해 format string을 쓰고 싶은 경우. 

   길이가 n 인 실수 array v(1:n) 를 출력한다고 하자. 

 

    WRITE(fmt_str,'(a,i2.2,a)') '(1x,' , n , '(f15.7,1x))'     

    ( n=3인 경우) fmt_str 은 '(1x,03(f15.7,1x))' 가 되고 

    WRITE(* , trim(fmt_str)) v    

    로 v 를 출력할 수 있다. 

  

2. Deal with system commands in Fortran

(1) get_command_argument(i,arg): get command arguments  

                             return i-th command argument as string

                             0-th argument is command itself

(2) SYSTEM( strings  )  : use system command

(3) TRIM : Removes trailing blank characters of a string

(4) call getcwd(path): get current working directory

      call chdir(path)   :change current working directory

      포트란의 기본 인풋/아웃풋 들, 파일 핸들링은 current working directory를

      기준으로 한다. call SYSTEM(‘cd path’)

      does not actually change the environment for fortran

3. quotes in strings : ‘’ represent ‘ in string. thus ‘’’ x’’’ -> ‘x’

4. Fw.d : 전체 캐릭터 수가 w 이고,( 부호, . 포함), 소수점뒤로 d 캐릭터

   Ew.d:  전체 캐릭터 수가 w 이고,( 부호, ., E+00 포함) 소수점 뒤로 d 캐릭터