From arccwiki
Jump to: navigation, search

DENISE-Black-Edition: 2D time-domain isotropic (visco)elastic FD modeling and full waveform inversion (FWI) code for P/SV-waves
2D time-domain isotropic (visco)elastic finite-difference modeling and full waveform inversion (FWI) code for P/SV-waves, which I developed together with André Kurzmann, Denise De Nil and Thomas Bohlen. Since then the code has been extended by Lisa Groos, Sven Heider, Martin Schäfer, Linbin Zhang and Daniel Wehner.

Module: Example

[]$ module spider denise
  denise: denise/1.3
    This module can be loaded directly: module load denise/1.3
module load denise/1.3


Testing: Based on Chapter 7, Example 1 in the Manual This assumes you have:

  • Cloned the DENISE-Black-Edition and have a copy of the par subfolder.
  • Cloned the DENISE-Benchmark and copied (as instructed) the Marmousi test files.

The documentation talks about using mpirun to run a simulation. You need to use the following method:

  • Create a batch file that you'll submit using sbatch or use salloc to create an interactive session.
  • Use srun to start you simulation.


  • The value of nodes * ntasks-per-node must match the srun -n value.
  • This value in turn must match NPROCX * NPROCY as defined in the DENISE_marm_OBC.inp file:

Example 1:

#SBATCH --account=type-your-project-name
#SBATCH --time=00:05:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=15
#SBATCH --cpus-per-task=1
#SBATCH --output=denise_%A.out

module load denise/1.3

srun -n 15 denise DENISE_marm_OBC.inp
#-------------- Domain Decomposition -----------------------------
number_of_processors_in_x-direction_(NPROCX) = 5
number_of_processors_in_y-direction_(NPROCY) = 3

Example 2:

#SBATCH --account=type-your-project-name
#SBATCH --time=00:05:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=30
#SBATCH --cpus-per-task=1
#SBATCH --output=denise_%A.out

module load denise/1.3

srun -n 60 denise DENISE_marm_OBC.inp
#-------------- Domain Decomposition -----------------------------
number_of_processors_in_x-direction_(NPROCX) = 10
number_of_processors_in_y-direction_(NPROCY) = 6

The number of cores you want to use must satisfy the conditions for the domain decomposition:

  • where NX, NY denote the number of FD grid points in x- and y-direction,
  • NPROCX, NPROCY are the number of MPI-processes in each spatial direction
  • and the % (modulo) operator yields the remainder from the division of the first argument by the second.

For the given Marmousi-2 model discretization: NX = 500 NY = 174
The following domain decomposition in the DENISE parameter file NPROCX = 10 NPROCY = 6 should run DENISE on NP = NPROCX * NPROCY = 60 cores
And in the batch file we define 2 nodes, each using 30 cores: 2 * 30 = 60
Or 4 nodes, each using 15 cores: 4 * 15 = 60

  • This software is dependent on the following modules:
    • swset/2018.05
    • intel/18.0.1
    • intel-mpi/2018.2.199
    • fftw/3.3.8-impi - compiled using intel/18.0.1
    • The module load denise/1.3 line will automatically load these modules for you.

Back to HPC Installed Software