Compilation

Manual (alternative)

Setup

  • Download WRFotron, WRFChem, make a copy for WRFMeteo without the chemistry folder, download WPS, download WPS Geography files:
cd /nobackup/${USER}
git clone https://github.com/wrfchem-leeds/WRFotron.git
git clone https://github.com/wrf-model/WRF.git WRFChem
git clone https://github.com/wrf-model/WPS.git

cp -r WRFChem WRFMeteo
cd WRFMeteo
rm -rf chem
  • Or copy these folders over from /nobackup/WRFChem:
cd /nobackup/${USER}
cp -r /nobackup/WRFChem/{WRFotron,WRFChem4.2,WRFMeteo4.2,WPS4.2,WPSGeog4} .
  • You will need to remove, or at minimum, change the module load line at the top of config.bash. The modules intel, openmpi, and WRFchem are for the CEMAC installation, and keeping these (and potentially others) can interfere with executables. These need to be removed. NCL, NCO, and conda can be used from CEMAC for manual runs, or you could have your own personal conda environments with NCL and NCO (see below). You can see the manual blueprint in the repository: config.bash.blueprint_manual.
# cemac compilation uses
module load intel/19.0.4 openmpi/3.1.4 WRFchem/4.2 ncl/6.5.0 nco/4.6.0 wrfchemconda/3.7 sge

# for manual compilation remove (at least) intel, openmpi, and WRFchem
module load ncl/6.5.0 nco/4.6.0 wrfchemconda/3.7 sge
  • The executables within pre.bash need to be copied over directly, rather than just linked which is adequate for the CEMAC method. To do this make both of the following replacements. You can see the manual blueprint in the repository: pre.bash.blueprint_manual.
# on line 21, replace:
for aFile in util geogrid ungrib link_grib.csh metgrid
# with:
for aFile in util geogrid geogrid.exe ungrib ungrib.exe link_grib.csh metgrid metgrid.exe

# and then on line 80, replace:
cp -r ${WRFdir}/run/* .
# with:
cp -r ${WRFdir}/run/* .
rm *.exe
cp -r ${WRFdir}/main/*.exe .
cp -r ${WRFmeteodir}/main/wrf.exe wrfmeteo.exe
  • All executables and preprocessors will need to have ./ before them to execute. This includes ungrib.exe, geogrid.exe, metgrid.exe, real.exe, megan_bio_emiss, wesely, exo_coldens, anthro_emiss, fire_emis, and mozbc in pre.bash. Also, wrfmeteo.exe and wrf.exe in main.bash. Also, wrf.exe in main_restart.bash. You can see the manual blueprints in the repository: pre.bash.blueprint_manual, main.bash.blueprint_manual, and main_restart.bash.blueprint_manual.
  • Add links to the preprocessor executables anthro_emis, fire_emis, and mozbc by adding the following code. You can see the manual blueprints in the repository: pre.bash.blueprint_manual.
ln -s ${WRFanthrodir}/anthro_emis . # section 4.a, line 148
ln -s ${WRFfiredir}/fire_emis .     # section 4.b, line 164
ln -s ${WRFMOZARTdir}/mozbc .         # section 6,   line 186
  • Download flex (tool for generating scanners: programs which recognize lexical patterns in text).

  • Download and compile (in serial) preprocessors:
    • anthro_emis (anthropogenic emissions preprocessor).

    • fire_emiss (fire emissions preprocessor).

    • megan (biogenic emissions preprocessor).

    • mozbc (preprocessor for lateral boundary and initial conditions).

    • wes-coldens (exocoldens and season_wesely, O2 and O3 column densities and dry deposition).

    • Check preprocessors have the correct modules and libraries linked via: ldd preprocessor.

conda deactivate # maybe multiple times
module purge
module load intel netcdf openmpi
export NETCDF=$(nc-config --prefix)
export NETCDF_DIR=$NETCDF
export FC=ifort

./make_anthro

./make_fire_emis

./make_util megan_bio_emiss

./make_mozbc

./make_util wesely

./make_util exo_coldens

Conda

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
  • Run bash script, read terms, and set path:
bash Miniconda3-latest-Linux-x86_64.sh
  • Create conda environment with Python 3 (with some libraries for analysis), NCL, and NCO:
conda create -n python3_ncl_nco -c conda-forge xarray salem xesmf numpy scipy pandas matplotlib rasterio affine ncl nco wrf-python dask geopandas descartes
  • To activate/deactivate conda environment:
conda activate python3_ncl_nco
conda deactivate
  • For more information on conda, visit.
  • Create separate environments for downloading ECMWF data (requires Python 2) and ncview, which you can then load temporarily to execute these functions:
conda create -n python2_ecmwf -c conda-forge ecmwf-api-client
conda create -n ncview -c eumetsat -c conda-forge ncview libpng

Compile WPS, WRFMeteo, and WRFChem

  • Setup:
# compiler setup
COMPILER_VER='intel:19.0.4'
MPI_VER='openmpi:3.1.4' # could use intelmpi instead of openmpi, and then also need: export I_MPI_HYDRA_TOPOLIB=ipl
CMP=${COMPILER_VER%:*}
CMP_VER=${COMPILER_VER#*:}
MP=${MPI_VER%:*}
MP_VER=${MPI_VER#*:}
FLAVOUR="${CMP}-${CMP_VER}-${MP}-${MP_VER}"

# modules
conda deactivate # maybe multiple times if many environments activated
module purge
module load licenses sge ${CMP}/${CMP_VER} ${MP}/${MP_VER} netcdf hdf5 patchelf

# environment variables - shell
NETCDF=$(nc-config --prefix)
NETCDF_DIR=$NETCDF
YACC='/usr/bin/yacc -d'
FLEX_LIB_DIR='/nobackup/WRFChem/flex/lib'
LD_LIBRARY_PATH=$FLEX_LIB_DIR:$LD_LIBRARY_PATH
JASPERLIB='/usr/lib64'
JASPERINC='/usr/include'

# environment variables - WRFChem
WRF_EM_CORE=1 # selects the ARW core
WRF_NMM_CORE=0 # ensures that the NMM core is deselected
WRF_CHEM=1 # selects the WRF-Chem module
WRF_KPP=1 # turns on Kinetic Pre-Processing (KPP)
WRFIO_NCD_LARGE_FILE_SUPPORT=1 # supports large wrfout files

# export variables
export FC CC NETCDF NETCDF_DIR YACC FLEX_LIB_DIR LD_LIBRARY_PATH JASPERLIB JASPERINC
export WRFIO_NCD_LARGE_FILE_SUPPORT WRF_KPP WRF_CHEM WRF_NMM_CORE WRF_EM_CORE
  • WRFChem compilation:
    • HPC option will be specific to your HPC architecture.
    • ARC4 = 15 = INTEL (ifort/icc) (dmpar) e.g. Distributed-Memory Parallelism MPI.
    • Compile for basic nesting: option 1.
    • Compile real (as oppose to ideal simulations).
    • Thousands of messages will appear. Compilation takes about 20-30 minutes.
    • How do you know your compilation was successful? If you have all four main/*.exe.
    • Check the executables have all relevant linked libraries: ldd main/wrf.exe.
cd /nobackup/${USER}/WRFChem
./clean -a
./configure # 15 for intel (ifort/icc) (dmpar) hpc architecture, 1 for basic nesting

./compile em_real >& log.compile
  • WPS compilation (requires a successfully compiled WRF):
    • HPC option will be specific to your HPC architecture.
    • ARC4 = 17 = INTEL (ifort/icc) (serial).
    • Sometimes configure.wps can assign the incorrect path to WRFChem, check and edit if required.
    • How do you know your compilation was successful? If you have geogrid.exe, metgrid.exe, and ungrib.exe.
    • Check the executables have all relevant linked libraries: ldd geogrid.exe.
cd /nobackup/${USER}/WPS
./clean -a
./configure # 17 for intel (ifort/icc) (serial) hpc architecture

gedit configure.wps
WRF_DIR="/nobackup/${USER}/WRFChem"

./compile >& log.compile
  • WRFMeteo compilation:

    • Deselect the WRFChem module.
    • HPC option will be specific to your HPC architecture.
    • ARC4 = 15 = INTEL (ifort/icc) (dmpar).
    • Compile for basic nesting: option 1.
    • Compile real (as oppose to ideal simulations).
    • Thousands of messages will appear. Compilation takes about 20-30 minutes.
    • How do you know your compilation was successful? If you have all four main/*.exe.
    • Check the executables have all relevant linked libraries: ldd main/wrf.exe.
export WRF_CHEM=0

cd /nobackup/${USER}/WRFMeteo
./clean -a
./configure # 15 for intel (ifort/icc) (dmpar) hpc architecture, 1 for basic nesting

./compile em_real >& log.compile
  • Preprocessors
    • If make any changes to preprocessors then they require recompilation.
    • Raw preprocessors downloaded from here.
    • Ensure the setup is the same as above for manual compilation of WPS/WRFChem/WRFMeteo.
      • May need to check if preprocessor requires a different module version that currently compiled with.
      • If Makefile cannot locate the correct NetCDF path, may need to add -lnetcdff.
      • Note for wes_coldens: FC hardcoded in make_util.
  • If need JASPER:
wget http://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/jasper-1.900.1.tar.gz
tar xvfz jasper-1.900.1.tar.gz
./configure
make
make install
export JASPERLIB=/usr/lib64 # not installed need own jasper
export JASPERINC=/usr/include
  • If need FLEX:
cd /nobackup/${USER}/flex/lib
./configure --prefix=$(pwd)/../flex
export FLEX_LIB_DIR='/nobackup/${USER}/flex/lib'