atomicAdd in GPU program(CUDA fortran)

When a subroutine is called in GPU such as 

call sum_array<<<numBlocks, blockSize>>>(A_d, n, total_d)

Basically, all individual threads works independently. 

To make a sum(or reduction) across blocks or across threads withn a block,

one have to use global device memory or block shared memory variable. 

However, if read-modify-write operation is done independently,

there will be a race-condition and result can be wrong. 

Atomic operation is a serialized operation across blocks or threads in a block. 

Scope of operation is determined by the memory location of variable. 

Note if the sums are partially done in each block, one have to be sure to call atomicAdd within only one thread in a block to get correct global sums. 

     ! Atomic reduction to shared memory

    !    dummy = atomicAdd(variable, value)

    !    atomic(indivisible) operation : read-modify-write as a single hardware instruction

    !                                    avoids race conditions. (ie. serialized by the hardware)

    !                                    variable have to be global in device or shared in block    

    !                                    variable have to be integer or real (not complex).  

    !    scope of operation(within block or across all blocks) is determined by the memory space.

    !       REAL(8),DEVICE or argument = Global memory, shared by all blocks

    !       REAL(8),SHARED =  Shared memory within the same block

    !       REAL(8)        = local memory within a thread.

PIP error: externally-managed-environment ( How to use virtual environment python)

 I encountered an error "error: externally-managed-environment" when trying to install library in Ubuntu. The message said to use "virtual environment" or risk breaking system. 

To create a virtual environment and install 

python -m venv my-venv

my-venv/bin/pip install some-python-library

This creates folder my-venv in the current location. 

To use virtual environment, activate 

source myenv/bin/activate

This enters virtual environment. 

deactivate

 

subprocess.Popen

 Suppose 

(1) current foldder = '/a/' 

(2) exe file 'run.exe' and '_.input' files are located at folder ='/a/b/' 

(3) necessary input file absolute path is "/a/b/c/xxx.dat"

* To run the code using Popen, from (1) 

 p= Popen("./run.exe < _.input ", shell=True, cwd = 'b/') ; p.communicate() 

where '_.input' have relative path './c/xxx.dat' 

In other words, relative path ( ".","..") are interpreted from "cwd" path . 

*

p = Popen("../pikoe1 < 12Cp2pTDXinv.cnt",shell=True,cwd ="pikoe1/sample4/" ); p.communicate()

Nurion 누리온 설정 bashrc 등등..

 bashrc 설정

# User specific aliases and functions

module load craype-mic-knl intel impi fftw_mpi/3.3.7 python/3.9.5

export PATH=$PATH:/home01/x3030a01/bin

#--check is any files will be deleted

find /scratch/x3030a01/ -type f -name ToBeDelete_* > ToBeDeletefiles.txt

echo `du -sh ToBeDeletefiles.txt`

ToBeDelete 가 있는 경우,

rename ToBeDelete_ '' ToBeDelete_*;touch *

miniforge/miniconda/nvidia(cuda) download/install (certification problems)

* 기본적으로 miniforge3 를 사용하는 것이 나을 것 같다. (miniconda의 경우 repository를 바꾸는 등의 작업이 추가로 필요하기 때문) 

(1) miniforge install : https://github.com/conda-forge/miniforge/ 에서 다운로드

(2) conda를 사용할때, SSL certificate errorr 가 생기는 경우. 다음과 같이 설정을 바꾼다. 

     conda config --set ssl_verify false

* apt 를 사용할때 certification poroblem ( "Certificate verification failed: ") 이 생기는 경우:

   https://askubuntu.com/questions/1095266/apt-get-update-failed-because-certificate-verification-failed-because-handshake

  의 예시를 따라서, /etc/apt/apt.conf.d/99verify-peer.conf 파일을 만들고, Acquire { https::Verify-Peer false } 를 추가해 준다. 위 예시에서는 update후 99verify... file을 지우라고 하는데, 나의 경우에는 유지해야했다....

*CUDA-FORTAN 을 사용하기 위해서는 https://developer.nvidia.com/cuda-fortran 에서 프로그램의 설명대로 진행한다. (위의 방법으로 apt의 certification 문제 해결) 이때, 주의할 것은 설치된 sdk 에는 openmpi 를 비롯한 compiler들이 포함되어 있고, 기존의 mpif90 등과 구분을 하여야 한다는 것이다. 설명대로 bashrc 에 nvidia/openmpi 용 mpif90 등의 path를 추가할 것. 

* 미니콘다(mini conda)를 다운로드하기 위해서는 아래 명령어를 widows command prompt에서 사용한다. ( -k 가 없으면 인증서 문제가 있을 수 있다.) 

 curl -k https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe -o .\miniconda.exe

2025 Daily work log

올해 부터는 간단하게라도 매일 무엇을 했는지 적어보기로 했다. 

2025-05-22: 한동안 잊고 있었네.. 계속 작성할 필요가 있을까? 어제 PC 교체. 

2025-04-08: 

  8He pinhole density plotting. Because of memory limitation, mpi gather have to be done with slices.   

2025-03-24: 

   Fix the problem in pinhole calculation of 3He with SU(4) interaction. 

   (initialization of nsh_A was not compatible with initial waves.)

   TOPTIER indico 페이지 수정.  

2025-03-18:

     8He(0+) and 8He(0+,2nd) Energy seems to be okay. 

     But, 8He(1-) is not okay. Testing 8He(1-) and 8He(2+).... 

2025-03-12:  Updated the NLEFT code to separate pp,pn,nn in GIB,GIR contributions. 

                 currently runnig 8He calculations and 12C for separation benchmark. 

2025-03-11: Study Shihang's pinhole storage code... 

          ( @numba.jit(nopython=True) could be new thing I need to try.) 

2025-03-08: Still working on the p-40Ar elastic scattering. 

                 Dean suggest to study pinhole calculations of 8He with WFM interaction for the 4 neutron correlations....

2025-02-28 ~ 03-04: Tavel to Beijing for the Frontiers meeting. 

2025-02-25:

(1) Still tweaking the p-40Ar scattering. 

(2) SU(4) pinhole seems to be not the best to study the charge radius. 

2025-02-11:

(1) NLEFT SU(4) pinhole calculation

(2) Fit elastic scattering experiments. Try to get dispersion relation.

2025-02-02:

(1) update GUI code  

(2) Fit elastic scattering experiments with optical potential. 

2025-01-22:

(1) study FRESCO output (meaning of each fort files)

(2) modify the code to get SU(4) essential Hamiltonian. 

(3) introduce skipping in perturb_PIN calculation. Need to check with 6He.

    However, somehow I used Rpro_phys=0 in test run. 

-------------------------------

2025-01-16:

(1) running test code of savepin : v0(no savepin) v2 and v3

(2) how to profile fortran and mpi 

------------------------------

2025-01-14:

(1) update mod_savepin_v3.f90. and python code. 

(2) study lightClsuterDistillation paper.

(3) DWIA formalism study  

--------------------------------

2025-01-12:

(1) DWIA study: understand the T-amplitude expression.

(2) pinhole calculation Lt=200, 12C,14C,16O,18O,20Ne,22Ne done. However, the calculation seems to be too slow.

-------------------------------    

2025-01-10:

(1) DWIA study 

(2) KPS meeting bilateral session proposal 

(3) 20Ne, 22Ne gs. energy calculation 거의 완료. 분석은 아직 

     20Ne, 22Ne,12C,14C,18O Lt=200 pinhole 계산중

(4) linear determinant method in YM's rank one note... not sure I understand this... 

-------------------------------

2025-01-08:

(1) Quantum computing 공부중. VQE 코드 실행

(2) 20Ne, 22Ne gs. energy 계산중. 추가로 pinhole 계산 시작. (아직은 savepin 코드 수정안함.)

    12C,14C, 16O, 18O 의 경우 계산 전에 메모리 테스트중.  

------------------------------------

2025-01-03: 

(1) https://qc.ascsn.net/landing.html 공부시작 

(2) Nurion Ne isotope 계산 메모리 테스트 

qstat 작업 상태 확인

 - 종료된 작업확인

$ qstat -xu [ id ] 

- 작업 상세정보 중 memory 확인
$ qstat -xf [job id] |grep mem

resources_used.mem = 2392727700kb

 는 총 노드 수 * 노드당 사용된 메모리 에 해당. 

여기서, 의문

(1) N 노드, M threads/node 일때, 하나의 thread 가 사용한 메모리는? 

    어차피 중요한 것은 하나의 노드의 총메모리를 모두 사용했는가일테니, 별로 중요하지 않은가?

(2) kb 는 kilo bits?