Common errors

We hope you don’t run into these errors. But if you do, we hope they will be a useful reference.

Tip: Use Ctrl+F (or Command+F) in your browser to search this page, it’ll be much faster!

Note

WRF-GC has several output files. Most GEOS-Chem output is in the root CPU output file, rsl.out.0000. Look in this file for any GEOS-Chem related errors. HEMCO-related errors are in HEMCO.log. WRF-related errors can be in any of the files numbered rsl.error.* and rsl.out.*. You may want to use tail -n 5 rsl.* | less to quickly look at all log files to identify errors.

Warning

THe HEMCO error log, HEMCO.log, only writes errors/warnings from the root process. In order to see HEMCO errors for all processes, edit HEMCO_Config.rc and set LogFile to *. Then you will be able to see HEMCO logs for all processes in the rsl.out.* files.

Compiling errors

error #7013: This module file was not generated by any release of this compiler.

Did you copy your WRF-GC directory from somewhere else? Some module files appear to be lingering over from a previous compile. Try running ./clean -a, reconfiguring WRF-GC (./configure -hyb for WRFV3, ./configure for WRFV4), and rebuliding (./compile em_real).

ld: cannot find -lGIGC -lHistory -lGeosCore -lHistory -lKpp -lGeosCore -lIsorropia -lHCOI -lHCOX -lHCO -lGeosUtil -lKpp -lHeaders -lNcUtils

A few things to check:

1. Go to WRF/chem/gc/lib. Do you see all these files?

WRF/chem/gc/lib$ ls
libGeosCore.a  libGIGC.a  libHCOI.a  libHeaders.a  libIsorropia.a  libNcUtils.a
libGeosUtil.a  libHCO.a   libHCOX.a  libHistory.a  libKpp.a        libObsPack.a

If yes, proceed to 2. If no, GEOS-Chem was not compiled correctly. Go to WRF/chem, run make compile_chem, then check if there are any errors.

2. Go to WRF/main/Makefile. Search for something like chem/gc/lib: ... -L/n/home04/hplin/wrfgc_dev_202110/WRF/chem/gc/lib -lHistory ....

Does this path, which points to chem/gc/lib, exist and is correct? Is is the same path as 1?

If no, go to step 3. The error is because you (or someone) moved your WRF-GC directly after compiling it once. We can fix this.

3. Move WRF/main/Makefile.bak to WRF/main/Makefile. Make sure this replaced version does not have -lGIGC anywhere within this file. If yes, download a fresh copy of WRF and move the main/Makefile from there into WRF/main/Makefile.

4. Now, go to WRF/chem, run make compile_chem. Verify that WRF/chem/main/Makefile has chem/gc/lib pointing to the right path. Go back to the root WRF directory and compile again: ./compile em_real. No clean is necessary.

mo_mozart_lib.f90:118: undefined reference to nf_open_

This happens in mozbc. Check if you have netCDF-C and netCDF-Fortran in the same directory. Your $NETCDF_DIR should include include/libnetcdff.a - if not, you’re missing the netCDF-Fortran.

lib64: file not recognized: Is a directory; make[1]: [geogrid.exe] Error 1 (ignored)`.

This may happen if you are on a cluster and are compiling WPS. Run export MPI_LIB="" then recompile WPS.

Cannot find input.geos, HEMCO_Config.rc or other configuration files in the run directory after compile

Make sure that you have compiled with export WRF_CHEM=1, which enables chemistry compilation. If this was not done, you must start over going to WRF/, ./clean -a, then ./compile em_real again.

Error: Dummy argument ‘datasetname’ with INTENT(IN) in variable definition context (actual argument to INTENT = OUT/INOUT) at (1)

This happens if you are using GNU Fortran (gfortran) compilers version 10 and above to try to build WRF v3. You may have to apply these patches manually to your WRF installation to get gfortran to compile.

WPS-related errors

ERROR: The mandatory field TT was not found in any input data.

This is because GFS was updated in June 2019. Download and compile WPS version 4 (only WPS) and copy the ungrib.exe from WPS v4 and use that. Haipeng has previously written on this

WRF-related errors

ERROR: Mismatch between namelist and input file dimensions NOTE: 1 namelist vs input data inconsistencies found.

If you see an error similar to:

d01 2016-05-11_12:00:00  input_wrf.F: SIZE MISMATCH:  namelist   ide,jde,num_metgrid_levels=         245         181          27
d01 2016-05-11_12:00:00  input_wrf.F: SIZE MISMATCH:  input file ide,jde,kde               =         245         181          32
d01 2016-05-11_12:00:00 ---- ERROR: Mismatch between namelist and input file dimensions
NOTE:       1 namelist vs input data inconsistencies found.

or

d01 2016-04-23_00:00:00  input_wrf.F: SIZE MISMATCH:  namelist num_metgrid_levels           =           34
d01 2016-04-23_00:00:00  input_wrf.F: SIZE MISMATCH:  input file BOTTOM-TOP_GRID_DIMENSION  =           27
d01 2016-04-23_00:00:00 ---- ERROR: Mismatch between namelist and input file dimensions
NOTE:       1 namelist vs input data inconsistencies found.

You’ve hit a problem where your meteorology data source (e.g., GFS) has updated during your model run times! This doesn’t happen often but you are bound to run into it sometime - see the list of updates and their dates on this changelog page from NOAA.

If your model run time did not go across: Then maybe you’re using old data and num_metgrid_levels is different. For example, try changing the namelist.input so it has the right number (in this case 32 instead of 27). If this doesn’t fix the issue, see above.

If your model run time did go across two GFS/FNL updates: Usually this can be fixed by making a custom data request, e.g., in FNL at the NCAR/UCAR Research Data Archive (RDA) with a consistent number of vertical levels, and this can be solved.

GEOS-CHEM USES A VERTICAL HYBRID-SIGMA GRID. WRF MUST BE CONFIGURED TO USE THIS GRID

If you see this warning in WRF-GC’s run logs:

=====================================================
|               W A R N I N G (WRF-GC)              |
=====================================================
  GEOS-CHEM USES A VERTICAL HYBRID-SIGMA GRID.
  WRF MUST BE CONFIGURED TO USE THIS GRID using the
  namelist option &dynamics: hybrid_opt = 2, and
  RECOMPILING with ./configure -hyb.

  WE WERE *NOT* ABLE TO DETECT THIS IN YOUR WRF-GC
  CONFIGURATION, WHICH MEANS THE VERTICAL LEVELS MAY
  BE INACCURATE AND OUTRIGHT WRONG. PLEASE CHECK.
=====================================================

Make sure that: * If you’re running WRFV3, make sure WRF-GC was configured with ./configure -hyb. If not, you have to recompile. Backup your namelist and HEMCO configuration, and ./clean -a, ./configure -hyb, then ./compile em_real in the WRF directory. * You have hybrid_opt = 2 in the &dynamics section of your namelist.input.

Crash right after W-DAMPING BEGINS AT W-COURANT NUMBER = 1.000000

The species numbers in the diagnostics (Additional diagnostics) are incorrect. Try resetting the diagnostic species numbers in the &chem section:

gc_diagn_spc_n0                     = 0,
gc_diagn_spc_n1                     = 0,
gc_diagn_spc_n2                     = 0,
gc_diagn_spc_n3                     = 0,

the domain size is too small for this many processors, or the decomposition aspect ratio is poor.

Short answer: Try to reduce the number of processors used in running WRF-GC.

Long answer: This warning message was added by WRF in order to avoid domains to be decomposed into too small chunks, where the overhead in cross-processor communication may exceed improvements by scaling. However, it appears that this error does not consider that chemistry needs more CPUs to scale and improvements may be seen even with a decomposed domain smaller than 10x10 in each core.

You may be able to work around this by patching WRF/share/module_check_a_mundo.F

IF ( ( model_config_rec % e_we(i) /  model_config_rec % nproc_x .LT. 10 ) .OR. &
   ( model_config_rec % e_sn(i) /  model_config_rec % nproc_y .LT. 10 ) ) THEN

And changing the parameters 10 to something like 5. But take care that you may see diminishing returns as the number of processors increase.

GEOS-Chem related errors

## INTEGRATE FAILED TWICE !!! or error in flexchem_mod_mp_d 1057 flexchem_mod.F90

If you see --> Step size too small: T + 10*H = T or H < Roundoff, this means that the conditions in that grid box are not optimal and resulted in integration errors. Try a better set of initial / boundary conditions. Also, see My NOx / HNO3 or nitrogen-related species are extremely high!.

WARNING: Negative concentration for species O3 at (I,J,L) =

This happens during boundary layer mixing. It is a crash of the ParaNOx extension. If you are sure you have realistic boundary / initial conditions (from GEOS-Chem output) and you are still getting this error, please see My NOx / HNO3 or nitrogen-related species are extremely high!.

My NOx / HNO3 or nitrogen-related species are extremely high!

If this is happening near the coast, this is a bug with the ParaNOx extension. This may even cause crashes in the model with negative species concentrations or KPP errors.

A temporary fix is to disable the ship plume extension (ParaNOx). Go to HEMCO_Config.rc and turn off the ParaNOx extension:

102     ParaNOx                : on    NO/NO2/O3/HNO3

Changing on to off. We are looking for a more permanent fix and are aware that this change may introduce differences in your simulation.

FAST-JX (RD_XXX): REQUIRED FILE NOT FOUND

In GEOS-Chem versions 12 or 13:

Make sure you have updated input.geos’s configuration with the correct path to the Fast-JX CHEM_INPUTS:

%%% PHOTOLYSIS MENU %%% :
FAST-JX directory       : /n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/data/ExtData/CHEM_INPUTS/FAST_JX/v2019-10/

Make sure to update the path on top of input.geos as well:

Root data directory     : /n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/data/ExtData/

In GEOS-Chem version 14 and above:

Make sure you have updated geoschem_config.yml’s configuration with the correct path to CHEM_INPUTS: there are three locations to update.

root_data_dir: /n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/data/ExtData
chem_inputs_dir: /n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/data/ExtData/CHEM_INPUTS/

...

photolysis:
  input_dir: /n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/data/ExtData/CHEM_INPUTS/FAST_JX/v2021-10/

Data can be obtained from GEOS-Chem input data at WUSTL.

HEMCO related errors

HEMCO ERROR: Invalid time index in (file)

Check the file that it is pointing to. There are a few likely reasons:

• The file was corrupted / not fully downloaded. Try opening it with ncview and checking!

• This inventory does not have the appropriate file for this date/time.

If (and only if) you have daily updating emissions data stored in monthly files and you are getting this error at the last day of the month, you are being affected by this bug. There is a temporary fix for WRF-GC available, contact Haipeng Lin for details.

GEOS-Chem ERROR: Error encountered in “HCO_Run”! -> at HCOI_GC_Run (in module GeosCore/hcoi_gc_main_mod.F90)

This is a HEMCO error. Check HEMCO.log. If nothing is there, change LogFile to * in HEMCO_Config.rc, and check all rsl.out.* files after re-running the model.

You are generally looking for something that looks like this:

HEMCO ERROR: Cannot find file for current simulation time:
/ExtData/HEMCO/OFFLINE_BIOVOC/v2019-10/0.5x0.625/2015/08/biovoc_05.20150820.nc
  - Cannot get field BIOGENIC_ACET. Please check file name and time (incl. time
 range flag) in the config. file

HEMCO ERROR: Error encountered in routine HCOIO_Read!

HEMCO ERROR: Error in HCOIO_DATAREAD called from HEMCO ReadList_Fill: BIOGENIC
 _ACET
 --> LOCATION: ReadList_Fill (HCO_ReadList_Mod.F90)

HEMCO ERROR: Error in ReadList_Fill (4) called from HEMCO ReadList_Read
 --> LOCATION: ReadList_Read (HCO_ReadList_Mod.F90)
 Error in ReadList_Read called from hco_run
===============================================================================
GEOS-Chem ERROR: Error encountered in "HCO_Run"!
 -> at HCOI_GC_Run (in module GeosCore/hco_interface_gc_mod.F90)

THIS ERROR ORIGINATED IN HEMCO!  Please check the HEMCO log file for
additional error messages!
===============================================================================

This may be much, much further up than the error about the subscripts and may even be in a different file. Always look through all the log files.

mozbc related errors

mo_wrfchem_lib.o: … undefined reference to nf_open_ / nf_inq_varid_ / nf_get_vara_real_ / nf_close_ / nf_def_var_

This error when compiling mozbc is usually because the path to netCDF library, NETCDF_DIR, is incorrect.

Make sure that your NETCDF_DIR is set to the root directory of your netCDF installation, where under $NETCDF_DIR/lib you can find the appropriate netCDF library (libnetcdff.so).

chk_moz_vars: could not find …_VMR_inst

This error is because the input netCDF file (outputs from GEOS-Chem) to create initial/boundary conditions does not have the aforementioned species. This may be because you are using outputs from an older GEOS-Chem version (e.g., 12.8.3) to feed initial/boundary conditions for a newer WRF-GC version (e.g., 14.0.0).

To solve this, simply go to the .inp file used for mozbc and remove the relevant line to ignore this species. e.g., for the error “chk_moz_vars: could not find NAP_VMR_inst”, the NAP species can be removed:

...

    'n2o5 -> N2O5',
    'nap -> NAP',
    'nh3 -> NH3',

    ...

Remove the line corresponding to the problematic species.

Red herrings

If you see anything on this list, this is not the root cause - this means that there’s an error somewhere else! Always check other log files (rsl.*) and also further upstream in the error files first.

HEMCO ERROR: MaxNest too low, cannot enter GET_TIMEIDX (hco_read_std_mod.F90)

Any kind of error that says HEMCO ERROR: MaxNest too low means that there is an error somewhere above, in HEMCO.log. Check further! See GEOS-Chem ERROR: Error encountered in “HCO_Run”! -> at HCOI_GC_Run (in module GeosCore/hcoi_gc_main_mod.F90).

forrtl: severe (408): fort: (2): Subscript #1 of the array LOC has value 11 which is greater than the upper bound of 10

This means that there is an error in HEMCO.log - check the HEMCO log instead! Maybe inventories are missing, etc. Also, see GEOS-Chem ERROR: Error encountered in “HCO_Run”! -> at HCOI_GC_Run (in module GeosCore/hcoi_gc_main_mod.F90).

forrtl: severe (408): fort: (2): Subscript #1 of the array ZPJ has value 1 which is greater than the upper bound of -1

Same! Check rsl.error. files for another error - it may be hundreds of lines above (searching for GEOS-Chem ERROR: or REQUIRED FILE NOT FOUND helps).

It may be helpful to first check this: FAST-JX (RD_XXX): REQUIRED FILE NOT FOUND to make sure your input.geos paths are correct.

This error indicates that another error occurred somewhere upstream.

forrtl: error (78): process killed (SIGTERM)

Check if your system has enough memory. Also, check rsl.out.0000 – this is usually an error within GEOS-Chem.

forrtl: severe (174): SIGSEGV, segmentation fault occurred

Check if your system has enough memory and try to run with ulimit -c unlimited, ulimit -s unlimited.

This may also mean there is an error somewhere else upstream. Look further below for a “stack trace”, e.g.,

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
wrf.exe            xxxxxxxxxxxxxxxx  Unknown               Unknown  Unknown
libpthread-2.17.s  xxxxxxxxxxxxxxxx  Unknown               Unknown  Unknown
wrf.exe            xxxxxxxxxxxxxxxx  error_mod_mp_erro         437  error_mod.F90
wrf.exe            xxxxxxxxxxxxxxxx  flexchem_mod_mp_d        1057  flexchem_mod.F90
wrf.exe            xxxxxxxxxxxxxxxx  chemistry_mod_mp_         299  chemistry_mod.F90
wrf.exe            xxxxxxxxxxxxxxxx  gigc_chunk_mod_mp        1277  gigc_chunk_mod.F90

Then look in this page for the error that corresponds to where the model has crashed.

Exit 152 mpirun -np ./wrf.exe

Check if your system / cluster job is allowing for enough memory. Also, check errors in other rsl.* files.

forrtl: error (78): process killed (SIGTERM)

Check other files first - this CPU is not the root cause of the problem.

Not actually errors

Cannot find -lGCHPint

This is not an error, ignore. If you cannot successfully compile WRF-GC, there is an error above in the compile log.

cannot stat ‘geos’: No such file or directory

Also shows sometimes as make[5]: [exe] Error 1 (ignored). This is not an error, ignore. If you cannot successfully compile WRF-GC, there is an error above in the compile log.

Trapping levels are weird and they should not be equal

This doesn’t seem to be an issue. If your run was interrupted, there might be an error elsewhere.

47 more processes have sent help message help-mpi-btl-openib.txt / no device params found

This is usually not an issue. If your run stopped, check all the other run files.

tropopause_climate: Warning: Done finding tropopause

This is not an issue, but may be related to your model vertical level configuration. If your outputs look reasonable, it is safe to ignore.