Common errors

We hope you don’t run into these errors. But if you do, we hope they will be a useful reference.

Tip: Use Ctrl+F (or Command+F) in your browser to search this page, it’ll be much faster!

Note

WRF-GC has several output files. Most GEOS-Chem output is in the root CPU output file, rsl.out.0000. Look in this file for any GEOS-Chem related errors. HEMCO-related errors are in HEMCO.log. WRF-related errors can be in any of the files numbered rsl.error.* and rsl.out.*. You may want to use tail -n 5 rsl.* | less to quickly look at all log files to identify errors.

Warning

THe HEMCO error log, HEMCO.log, only writes errors/warnings from the root process. In order to see HEMCO errors for all processes, edit HEMCO_Config.rc and set LogFile to *. Then you will be able to see HEMCO logs for all processes in the rsl.out.* files.

Compiling errors

error #7013: This module file was not generated by any release of this compiler.

Did you copy your WRF-GC directory from somewhere else? Some module files appear to be lingering over from a previous compile. Try running ./clean -a, reconfiguring WRF-GC (./configure -hyb for WRFV3, ./configure for WRFV4), and rebuliding (./compile em_real).

ld: cannot find -lGIGC -lHistory -lGeosCore -lHistory -lKpp -lGeosCore -lIsorropia -lHCOI -lHCOX -lHCO -lGeosUtil -lKpp -lHeaders -lNcUtils

A few things to check:

  1. Go to WRF/chem/gc/lib. Do you see all these files?

WRF/chem/gc/lib$ ls
libGeosCore.a  libGIGC.a  libHCOI.a  libHeaders.a  libIsorropia.a  libNcUtils.a
libGeosUtil.a  libHCO.a   libHCOX.a  libHistory.a  libKpp.a        libObsPack.a

If yes, proceed to 2. If no, GEOS-Chem was not compiled correctly. Go to WRF/chem, run make compile_chem, then check if there are any errors.

  1. Go to WRF/main/Makefile. Search for something like chem/gc/lib: ... -L/n/home04/hplin/wrfgc_dev_202110/WRF/chem/gc/lib -lHistory ....

Does this path, which points to chem/gc/lib, exist and is correct? Is is the same path as 1?

If no, go to step 3. The error is because you (or someone) moved your WRF-GC directly after compiling it once. We can fix this.

  1. Move WRF/main/Makefile.bak to WRF/main/Makefile. Make sure this replaced version does not have -lGIGC anywhere within this file. If yes, download a fresh copy of WRF and move the main/Makefile from there into WRF/main/Makefile.

  2. Now, go to WRF/chem, run make compile_chem. Verify that WRF/chem/main/Makefile has chem/gc/lib pointing to the right path. Go back to the root WRF directory and compile again: ./compile em_real. No clean is necessary.

mo_mozart_lib.f90:118: undefined reference to nf_open_

This happens in mozbc. Check if you have netCDF-C and netCDF-Fortran in the same directory. Your $NETCDF_DIR should include include/libnetcdff.a - if not, you’re missing the netCDF-Fortran.

lib64: file not recognized: Is a directory; make[1]: [geogrid.exe] Error 1 (ignored)`.

This may happen if you are on a cluster and are compiling WPS. Run export MPI_LIB="" then recompile WPS.

Cannot find input.geos, HEMCO_Config.rc or other configuration files in the run directory after compile

Make sure that you have compiled with export WRF_CHEM=1, which enables chemistry compilation. If this was not done, you must start over going to WRF/, ./clean -a, then ./compile em_real again.

Red herrings

If you see anything on this list, this is not the root cause - this means that there’s an error somewhere else! Always check other log files (rsl.*) and also further upstream in the error files first.

HEMCO ERROR: MaxNest too low, cannot enter GET_TIMEIDX (hco_read_std_mod.F90)

Any kind of error that says HEMCO ERROR: MaxNest too low means that there is an error somewhere above, in HEMCO.log. Check further! See GEOS-Chem ERROR: Error encountered in “HCO_Run”! -> at HCOI_GC_Run (in module GeosCore/hcoi_gc_main_mod.F90).

forrtl: severe (408): fort: (2): Subscript #1 of the array LOC has value 11 which is greater than the upper bound of 10

This means that there is an error in HEMCO.log - check the HEMCO log instead! Maybe inventories are missing, etc. Also, see GEOS-Chem ERROR: Error encountered in “HCO_Run”! -> at HCOI_GC_Run (in module GeosCore/hcoi_gc_main_mod.F90).

forrtl: severe (408): fort: (2): Subscript #1 of the array ZPJ has value 1 which is greater than the upper bound of -1

Same! Check rsl.error. files for another error - it may be hundreds of lines above (searching for GEOS-Chem ERROR: or REQUIRED FILE NOT FOUND helps).

It may be helpful to first check this: FAST-JX (RD_XXX): REQUIRED FILE NOT FOUND to make sure your input.geos paths are correct.

This error indicates that another error occurred somewhere upstream.

forrtl: error (78): process killed (SIGTERM)

Check if your system has enough memory. Also, check rsl.out.0000 – this is usually an error within GEOS-Chem.

forrtl: severe (174): SIGSEGV, segmentation fault occurred

Check if your system has enough memory and try to run with ulimit -c unlimited, ulimit -s unlimited.

This may also mean there is an error somewhere else upstream. Look further below for a “stack trace”, e.g.,

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
wrf.exe            xxxxxxxxxxxxxxxx  Unknown               Unknown  Unknown
libpthread-2.17.s  xxxxxxxxxxxxxxxx  Unknown               Unknown  Unknown
wrf.exe            xxxxxxxxxxxxxxxx  error_mod_mp_erro         437  error_mod.F90
wrf.exe            xxxxxxxxxxxxxxxx  flexchem_mod_mp_d        1057  flexchem_mod.F90
wrf.exe            xxxxxxxxxxxxxxxx  chemistry_mod_mp_         299  chemistry_mod.F90
wrf.exe            xxxxxxxxxxxxxxxx  gigc_chunk_mod_mp        1277  gigc_chunk_mod.F90

Then look in this page for the error that corresponds to where the model has crashed.

Exit 152 mpirun -np ./wrf.exe

Check if your system / cluster job is allowing for enough memory. Also, check errors in other rsl.* files.

forrtl: error (78): process killed (SIGTERM)

Check other files first - this CPU is not the root cause of the problem.

Not actually errors

Cannot find -lGCHPint

This is not an error, ignore. If you cannot successfully compile WRF-GC, there is an error above in the compile log.

cannot stat ‘geos’: No such file or directory

Also shows sometimes as make[5]: [exe] Error 1 (ignored). This is not an error, ignore. If you cannot successfully compile WRF-GC, there is an error above in the compile log.

Trapping levels are weird and they should not be equal

This doesn’t seem to be an issue. If your run was interrupted, there might be an error elsewhere.

47 more processes have sent help message help-mpi-btl-openib.txt / no device params found

This is usually not an issue. If your run stopped, check all the other run files.

tropopause_climate: Warning: Done finding tropopause

This is not an issue, but may be related to your model vertical level configuration. If your outputs look reasonable, it is safe to ignore.