Common errors ============== We hope you don't run into these errors. But if you do, we hope they will be a useful reference. **Tip:** Use ``Ctrl+F`` (or ``Command+F``) in your browser to search this page, it'll be much faster! .. note:: WRF-GC has several output files. Most GEOS-Chem output is in the root CPU output file, ``rsl.out.0000``. **Look in this file for any GEOS-Chem related errors.** HEMCO-related errors are in ``HEMCO.log``. WRF-related errors can be in any of the files numbered ``rsl.error.*`` and ``rsl.out.*``. You may want to use ``tail -n 5 rsl.* | less`` to quickly look at all log files to identify errors. .. warning:: THe HEMCO error log, ``HEMCO.log``, only writes errors/warnings from the root process. In order to see HEMCO errors for all processes, edit ``HEMCO_Config.rc`` and set ``LogFile`` to ``*``. Then you will be able to see HEMCO logs for all processes in the ``rsl.out.*`` files. Compiling errors ----------------- error #7013: This module file was not generated by any release of this compiler. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Did you copy your WRF-GC directory from somewhere else? Some module files appear to be lingering over from a previous compile. Try running ``./clean -a``, reconfiguring WRF-GC (``./configure -hyb`` for WRFV3, ``./configure`` for WRFV4), and rebuliding (``./compile em_real``). ld: cannot find -lGIGC -lHistory -lGeosCore -lHistory -lKpp -lGeosCore -lIsorropia -lHCOI -lHCOX -lHCO -lGeosUtil -lKpp -lHeaders -lNcUtils ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A few things to check: 1. Go to ``WRF/chem/gc/lib``. Do you see **all** these files? .. code-block:: WRF/chem/gc/lib$ ls libGeosCore.a libGIGC.a libHCOI.a libHeaders.a libIsorropia.a libNcUtils.a libGeosUtil.a libHCO.a libHCOX.a libHistory.a libKpp.a libObsPack.a If yes, proceed to 2. If no, GEOS-Chem was not compiled correctly. Go to ``WRF/chem``, run ``make compile_chem``, then check if there are any errors. 2. Go to ``WRF/main/Makefile``. Search for something like ``chem/gc/lib``: ``... -L/n/home04/hplin/wrfgc_dev_202110/WRF/chem/gc/lib -lHistory ...``. Does this path, which points to ``chem/gc/lib``, **exist** and **is correct**? Is is the same path as 1? If no, go to step 3. **The error is because you (or someone) moved your WRF-GC directly after compiling it once. We can fix this.** 3. Move ``WRF/main/Makefile.bak`` to ``WRF/main/Makefile``. Make sure this replaced version does not have ``-lGIGC`` **anywhere** within this file. If yes, download a fresh copy of WRF and move the ``main/Makefile`` from there into ``WRF/main/Makefile``. 4. Now, go to ``WRF/chem``, run ``make compile_chem``. Verify that ``WRF/chem/main/Makefile`` has ``chem/gc/lib`` pointing to the right path. **Go back to the root WRF directory and compile again**: ``./compile em_real``. No clean is necessary. mo_mozart_lib.f90:118: undefined reference to nf_open_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This happens in ``mozbc``. Check if you have **netCDF-C** and **netCDF-Fortran** in the same directory. Your ``$NETCDF_DIR`` should include ``include/libnetcdff.a`` - if not, you're missing the netCDF-Fortran. lib64: file not recognized: Is a directory; make[1]: [geogrid.exe] Error 1 (ignored)`. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This may happen if you are on a cluster and are compiling :term:`WPS`. Run ``export MPI_LIB=""`` then recompile WPS. Cannot find input.geos, HEMCO_Config.rc or other configuration files in the run directory after compile ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Make sure that you have compiled with ``export WRF_CHEM=1``, which enables chemistry compilation. If this was not done, you **must** start over going to ``WRF/``, ``./clean -a``, then ``./compile em_real`` again. Error: Dummy argument ‘datasetname’ with INTENT(IN) in variable definition context (actual argument to INTENT = OUT/INOUT) at (1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This happens if you are using GNU Fortran (gfortran) compilers version 10 and above to try to build WRF v3. You may have to `apply these patches manually to your WRF installation `_ to get gfortran to compile. WPS-related errors ------------------- ERROR: The mandatory field TT was not found in any input data. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This is because GFS was updated in June 2019. Download and compile WPS version 4 (only WPS) and copy the ``ungrib.exe`` from WPS v4 and use that. `Haipeng has previously written on this `_ WRF-related errors ------------------ ERROR: Mismatch between namelist and input file dimensions NOTE: 1 namelist vs input data inconsistencies found. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you see an error similar to: .. code-block:: d01 2016-05-11_12:00:00 input_wrf.F: SIZE MISMATCH: namelist ide,jde,num_metgrid_levels= 245 181 27 d01 2016-05-11_12:00:00 input_wrf.F: SIZE MISMATCH: input file ide,jde,kde = 245 181 32 d01 2016-05-11_12:00:00 ---- ERROR: Mismatch between namelist and input file dimensions NOTE: 1 namelist vs input data inconsistencies found. or .. code-block:: d01 2016-04-23_00:00:00 input_wrf.F: SIZE MISMATCH: namelist num_metgrid_levels = 34 d01 2016-04-23_00:00:00 input_wrf.F: SIZE MISMATCH: input file BOTTOM-TOP_GRID_DIMENSION = 27 d01 2016-04-23_00:00:00 ---- ERROR: Mismatch between namelist and input file dimensions NOTE: 1 namelist vs input data inconsistencies found. You've hit a problem where your meteorology data source (e.g., GFS) has updated **during your model run times!** This doesn't happen often but you are bound to run into it sometime - see the list of updates and their dates `on this changelog page `_ from NOAA. **If your model run time did not go across:** Then maybe you're using old data and ``num_metgrid_levels`` is different. For example, try changing the ``namelist.input`` so it has the right number (in this case ``32`` instead of ``27``). If this doesn't fix the issue, see above. **If your model run time did go across two GFS/FNL updates:** Usually this can be fixed by making a custom data request, e.g., in FNL `at the NCAR/UCAR Research Data Archive (RDA) `__ with a consistent number of vertical levels, and this can be solved. GEOS-CHEM USES A VERTICAL HYBRID-SIGMA GRID. WRF MUST BE CONFIGURED TO USE THIS GRID ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you see this warning in WRF-GC's run logs: .. code-block:: ===================================================== | W A R N I N G (WRF-GC) | ===================================================== GEOS-CHEM USES A VERTICAL HYBRID-SIGMA GRID. WRF MUST BE CONFIGURED TO USE THIS GRID using the namelist option &dynamics: hybrid_opt = 2, and RECOMPILING with ./configure -hyb. WE WERE *NOT* ABLE TO DETECT THIS IN YOUR WRF-GC CONFIGURATION, WHICH MEANS THE VERTICAL LEVELS MAY BE INACCURATE AND OUTRIGHT WRONG. PLEASE CHECK. ===================================================== Make sure that: * If you're running WRFV3, make sure WRF-GC was configured with ``./configure -hyb``. If not, you have to recompile. **Backup your namelist and HEMCO configuration**, and ``./clean -a``, ``./configure -hyb``, then ``./compile em_real`` in the WRF directory. * You have ``hybrid_opt = 2`` in the ``&dynamics`` section of your ``namelist.input``. Crash right after W-DAMPING BEGINS AT W-COURANT NUMBER = 1.000000 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The species numbers in the diagnostics (:doc:`/extra-diagnostics`) are incorrect. Try resetting the diagnostic species numbers in the ``&chem`` section: .. code-block:: gc_diagn_spc_n0 = 0, gc_diagn_spc_n1 = 0, gc_diagn_spc_n2 = 0, gc_diagn_spc_n3 = 0, the domain size is too small for this many processors, or the decomposition aspect ratio is poor. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Short answer: Try to reduce the number of processors used in running WRF-GC. Long answer: This warning message was added by WRF in order to avoid domains to be decomposed into too small chunks, where the overhead in cross-processor communication may exceed improvements by scaling. However, it appears that this error does not consider that chemistry needs more CPUs to scale and improvements may be seen even with a decomposed domain smaller than 10x10 in each core. You may be able to work around this by patching ``WRF/share/module_check_a_mundo.F`` .. code-block:: IF ( ( model_config_rec % e_we(i) / model_config_rec % nproc_x .LT. 10 ) .OR. & ( model_config_rec % e_sn(i) / model_config_rec % nproc_y .LT. 10 ) ) THEN And changing the parameters ``10`` to something like ``5``. But take care that you may see diminishing returns as the number of processors increase. GEOS-Chem related errors ------------------------ ## INTEGRATE FAILED TWICE !!! or error in flexchem_mod_mp_d 1057 flexchem_mod.F90 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you see ``--> Step size too small: T + 10*H = T or H < Roundoff``, this means that the conditions in that grid box are not optimal and resulted in integration errors. Try a better set of initial / boundary conditions. Also, see :ref:`paranox-bug`. WARNING: Negative concentration for species O3 at (I,J,L) = ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This happens during boundary layer mixing. It is a crash of the ParaNOx extension. If you are sure you have realistic boundary / initial conditions (from GEOS-Chem output) and you are still getting this error, please see :ref:`paranox-bug`. .. _paranox-bug: My NOx / HNO3 or nitrogen-related species are extremely high! ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If this is happening near the coast, this is a bug with the ParaNOx extension. This may even cause crashes in the model with negative species concentrations or KPP errors. A temporary fix is to disable the ship plume extension (ParaNOx). Go to ``HEMCO_Config.rc`` and turn off the `ParaNOx` extension: .. code-block:: 102 ParaNOx : on NO/NO2/O3/HNO3 Changing ``on`` to ``off``. We are looking for a more permanent fix and are aware that this change may introduce differences in your simulation. FAST-JX (RD_XXX): REQUIRED FILE NOT FOUND ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ **In GEOS-Chem versions 12 or 13:** Make sure you have updated ``input.geos``'s configuration with the correct path to the Fast-JX CHEM_INPUTS: .. code-block:: %%% PHOTOLYSIS MENU %%% : FAST-JX directory : /n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/data/ExtData/CHEM_INPUTS/FAST_JX/v2019-10/ Make sure to update the path on top of ``input.geos`` as well: .. code-block:: Root data directory : /n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/data/ExtData/ **In GEOS-Chem version 14 and above:** Make sure you have updated ``geoschem_config.yml``'s configuration with the correct path to ``CHEM_INPUTS``: there are three locations to update. .. code-block:: root_data_dir: /n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/data/ExtData chem_inputs_dir: /n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/data/ExtData/CHEM_INPUTS/ ... photolysis: input_dir: /n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/data/ExtData/CHEM_INPUTS/FAST_JX/v2021-10/ Data can be obtained from `GEOS-Chem input data `__ at WUSTL. HEMCO related errors -------------------- HEMCO ERROR: Invalid time index in (file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Check the file that it is pointing to. There are a few likely reasons: * The file was corrupted / not fully downloaded. Try opening it with ``ncview`` and checking! * This inventory does not have the appropriate file for this date/time. If (and only if) you have daily updating emissions data stored in monthly files and you are getting this error at the last day of the month, you are being affected `by this bug `_. There is a temporary fix for WRF-GC available, contact Haipeng Lin for details. .. _hco-run-error: GEOS-Chem ERROR: Error encountered in "HCO_Run"! -> at HCOI_GC_Run (in module GeosCore/hcoi_gc_main_mod.F90) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This is a HEMCO error. Check ``HEMCO.log``. If nothing is there, change ``LogFile`` to ``*`` in ``HEMCO_Config.rc``, and check **all** ``rsl.out.*`` files after re-running the model. You are generally looking for something that looks like this: .. code-block:: HEMCO ERROR: Cannot find file for current simulation time: /ExtData/HEMCO/OFFLINE_BIOVOC/v2019-10/0.5x0.625/2015/08/biovoc_05.20150820.nc - Cannot get field BIOGENIC_ACET. Please check file name and time (incl. time range flag) in the config. file HEMCO ERROR: Error encountered in routine HCOIO_Read! HEMCO ERROR: Error in HCOIO_DATAREAD called from HEMCO ReadList_Fill: BIOGENIC _ACET --> LOCATION: ReadList_Fill (HCO_ReadList_Mod.F90) HEMCO ERROR: Error in ReadList_Fill (4) called from HEMCO ReadList_Read --> LOCATION: ReadList_Read (HCO_ReadList_Mod.F90) Error in ReadList_Read called from hco_run =============================================================================== GEOS-Chem ERROR: Error encountered in "HCO_Run"! -> at HCOI_GC_Run (in module GeosCore/hco_interface_gc_mod.F90) THIS ERROR ORIGINATED IN HEMCO! Please check the HEMCO log file for additional error messages! =============================================================================== **This may be much, much further up than the error about the subscripts and may even be in a different file.** Always look through all the log files. mozbc related errors --------------------- mo_wrfchem_lib.o: ... undefined reference to nf_open_ / nf_inq_varid_ / nf_get_vara_real_ / nf_close_ / nf_def_var_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This error when compiling ``mozbc`` is usually because the path to netCDF library, ``NETCDF_DIR``, is incorrect. Make sure that your ``NETCDF_DIR`` is set to the root directory of your netCDF installation, where under ``$NETCDF_DIR/lib`` you can find the appropriate netCDF library (``libnetcdff.so``). chk_moz_vars: could not find ..._VMR_inst ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This error is because the input netCDF file (outputs from GEOS-Chem) to create initial/boundary conditions does not have the aforementioned species. This may be because you are using outputs from an older GEOS-Chem version (e.g., 12.8.3) to feed initial/boundary conditions for a newer WRF-GC version (e.g., 14.0.0). To solve this, simply go to the ``.inp`` file used for ``mozbc`` and remove the relevant line to ignore this species. e.g., for the error "chk_moz_vars: could not find NAP_VMR_inst", the ``NAP`` species can be removed: .. code-block:: ... 'n2o5 -> N2O5', 'nap -> NAP', 'nh3 -> NH3', ... Remove the line corresponding to the problematic species. Red herrings ------------- If you see anything on this list, this is not the root cause - **this means that there's an error somewhere else!** Always check other log files (``rsl.*``) and also further upstream in the error files first. HEMCO ERROR: MaxNest too low, cannot enter GET_TIMEIDX (hco_read_std_mod.F90) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Any kind of error that says ``HEMCO ERROR: MaxNest too low`` means that there is an error somewhere above, in ``HEMCO.log``. Check further! See :ref:`hco-run-error`. forrtl: severe (408): fort: (2): Subscript #1 of the array LOC has value 11 which is greater than the upper bound of 10 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This means that there is an error in ``HEMCO.log`` - check the HEMCO log instead! Maybe inventories are missing, etc. Also, see :ref:`hco-run-error`. forrtl: severe (408): fort: (2): Subscript #1 of the array ZPJ has value 1 which is greater than the upper bound of -1 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Same! Check `rsl.error.` files for **another error** - it may be hundreds of lines above (searching for ``GEOS-Chem ERROR:`` or ``REQUIRED FILE NOT FOUND`` helps). It may be helpful to first check this: :ref:`FAST-JX (RD_XXX): REQUIRED FILE NOT FOUND` to make sure your ``input.geos`` paths are correct. This error indicates that another error occurred somewhere upstream. forrtl: error (78): process killed (SIGTERM) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Check if your system has enough memory. Also, check ``rsl.out.0000`` -- this is usually an error within GEOS-Chem. forrtl: severe (174): SIGSEGV, segmentation fault occurred ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Check if your system has enough memory and try to run with ``ulimit -c unlimited``, ``ulimit -s unlimited``. This may also mean there is an error somewhere else upstream. Look further below for a "stack trace", e.g., .. code-block:: forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source wrf.exe xxxxxxxxxxxxxxxx Unknown Unknown Unknown libpthread-2.17.s xxxxxxxxxxxxxxxx Unknown Unknown Unknown wrf.exe xxxxxxxxxxxxxxxx error_mod_mp_erro 437 error_mod.F90 wrf.exe xxxxxxxxxxxxxxxx flexchem_mod_mp_d 1057 flexchem_mod.F90 wrf.exe xxxxxxxxxxxxxxxx chemistry_mod_mp_ 299 chemistry_mod.F90 wrf.exe xxxxxxxxxxxxxxxx gigc_chunk_mod_mp 1277 gigc_chunk_mod.F90 Then look in this page for the error that corresponds to where the model has crashed. Exit 152 mpirun -np ./wrf.exe ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Check if your system / cluster job is allowing for enough memory. Also, check errors in other ``rsl.*`` files. forrtl: error (78): process killed (SIGTERM) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Check other files first - this CPU is not the root cause of the problem. Not actually errors -------------------- Cannot find -lGCHPint ^^^^^^^^^^^^^^^^^^^^^^ This is not an error, ignore. If you cannot successfully compile WRF-GC, there is an error above in the compile log. cannot stat ‘geos’: No such file or directory ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Also shows sometimes as ``make[5]: [exe] Error 1 (ignored)``. This is not an error, ignore. If you cannot successfully compile WRF-GC, there is an error above in the compile log. Trapping levels are weird and they should not be equal ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This doesn't seem to be an issue. If your run was interrupted, there might be an error elsewhere. 47 more processes have sent help message help-mpi-btl-openib.txt / no device params found ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This is usually not an issue. If your run stopped, check all the other run files. tropopause_climate: Warning: Done finding tropopause ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This is not an issue, but may be related to your model vertical level configuration. If your outputs look reasonable, it is safe to ignore.