NASA - National Aeronautics and Space Administration

In 2011, the ESPO data archive changed file formats, from the Ames-format (Hispkind and Gaines format) to ICARTT-format data files. While the ICARTT format is conceptually just an update to the Ames format and shares many similarities, there are also significant changes between the two formats. This page summarizes the most important changes.

File Names

The order of the components in the filenames has been changed, as have the allowed filecodes for individual files. Some examples of how the overall names have changed:

ICARTT-format files always end in the extension ".ict" (the platform information is now provided earlier in the name)

File Codes

The filecodes (i.e., the first part of the filename that identifies which instrument/parameters are provided in the file) used by the Ames format were typically only two characters in length, due to filename length restrictions typical when the Ames format was first introduced.  With ICARTT, the obscure 2-letter identifiers can be replaced by somewhat longer and more self-explanatory codes. ICARTT filecodes are also case-sensitive. Filecodes can only contain the characters: a-zA-Z0-9- (upper-case and lower-case letters, numbers, or hyphen). Spaces and underscores (_) are not allowed. An instrument's acronym makes a good filecode; if an instrument has multiple filenames the acronym plus a word or two describing the specific file can be used.

Revision Number

To be completed

File Header

The overall contents of the file header are the same as in Ames files, with the same file format indexes (FFI, e.g., 1001) and same row contents. The single biggest overall difference is that whenever multiple values appear on the same line they must be separated by commas (,) not just spaces. So the first line of the file might be "36, 1001" instead of "36 1001". The second critical difference is that missing numbers are now negative and must contain only the digit 9 (previously 9s were widely used but not mandatory). So a missing number might be "-9999" instead of "9999".

Normal Comments

The ICARTT format requires that supporting information about the file be added to the normal comments section of the file. A dozen different fields must be added to the normal comments. The specific fields and their contents are documented in the ICARTT reference. Remember when adding this information that the extra lines count towards the length of the normal contents section and the overall length of the header -- so the lines providing those lengths must be updated accordingly.

File Content

ICARTT introduces two noticeable changes to the file contents, specifically related to how record times are reported:

• If your instrument reports data at less than 1Hz (i.e., if the interval between any two measurement time points is more than 1 second) you must report both Stop and Start times for every record. In addition, if the mid-point of your sampling interval is not the average of the start and stop times it must also be reported. To report Stop and Start times:
  • Make your primary index (the first number on each line of data) be Start_UTC
  • Make the first measurement variable (the second number on each line of data) be Stop_UTC
  • See the ICARTT documentation for complete examples.
• If your instrument reports data at 1Hz or greater, but does not report data at regular intervals (e.g., if you state on line 8 of your file that DX-1=0), your file will be rejected by ICARTT's Fscan software, which will insist that you must provide start and stop times. This is not necessary if you are submitting the file to the ESPO archive. However, to satisfy the intent behind the ICARTT restrictions, you must report data every second. If your instrument did not make measurements during any interval (e.g., calibration), you must explicitly state that your data is missing for every time point (fill the gap with lines containing the time followed by -9999 values for every measurement).

The other changes to the file contents (i.e., the actual data, starting after the file header) are generally just extensions of changes already mentioned in the File Header section. Specifically:

• Commas separate values, not spaces.
• Missing values are negative -- so checks for missing values must use 'value<=VMISS' instead of 'value>=VMISS'
• Upper and/or lower LOD flags (e.g., '-7777', '-8888') may be present in the data, as specified in the normal contents of the header using the ULOD_FLAG or LLOD_FLAG keywords.

Transition Assistance

Various tools are being planned to ease the transition between Ames and ICARTT formats. Details on these tools will be made available as they are finalized. However, the accelerated roll-out of the new ESPO archive for MACPEX means that some of the tools might not be ready in time for the first MACPEX flights.

Conversion Limitations

If you are uploading Ames-format files and need them converted to ICARTT-format files, an extra web form will need to be filled out during the upload process, where you will need to provide any ICARTT-required supporting information that is missing from your files. One restriction of Ames-format files is that you cannot use the new ICARTT limit-of-detection flags in your files If you are downloading ICARTT-format files as Ames files, any limit-of-detection flags used in the ICARTT-format file will be mangled. You will be warned when downloading the files if this has been necessary for the files you've downloaded. If you do not make any changes to the software being used to read the Ames-format files, any values that are outside the limits of detection will be read as missing values. However, if the software is modified to recognize the ULOD_FLAG and LLOD_FLAG values provided in the data header, any such values can in fact be read correctly.