cdf-rs

This is a standalone Rust library to parse and write to files written in NASA's Common Data Format (CDF), which is different from UCAR's netCDF format. The CDF format is commonly used in space physics to store data from various instruments onboard NASA spacecraft.

There are various existing parsers to read and write CDF files. cdf-rs is written from scratch in Rust following the CDF internal format specification. It does not use or interact in any way with the official CDF C library provided by NASA. In addition to the official C library, other parsers for the CDF format include,

• cdflib (Python, standalone),
• pysatCDF (Python wrapper to the official C-library),
• cdfj (Java), and
• CDFpp (C++).

cdf-rs is not zero-copy. At the primitive level, there is a call to from_le_bytes or from_be_bytes on a byte-slice buffer.

Usage

Install by adding cdf to Cargo.toml or using cargo.

$ cargo add cdf

Or, if you want to derive serde::Serialize and serde::Deserialize,

$ cargo add cdf --features serde

cdf-rs decodes the CDF file in a heirarchical manner by recursively calling decode_* on each constituent. Calling the top-level Cdf::read_cdf_file function is the easiest. This reads in the contents of the CDF file into one struct representative of the CDF data model.

use cdf::cdf::Cdf;

fn main() {
    let cdf_contents = Cdf::read_cdf_file("examples/data/test_alltypes.cdf").unwrap();
}

Dependencies

By default cdf-rs has no dependencies (as of yet). serde support is optional and for that you need to enable the serde feature.

The CDF data model

A CDF file is a collection of 'records'. There are different kinds of records, and some records point to other records of a different type, or different records of the same type (creating a linked-list). But, at the lowest level, data is stored in the form of integers, floats, etc. Different kinds of CDF records, and different kinds of CDF primitive types are defined in the CDF Internal Format specification.

Heirarchy of a CDF file
The CDF format is heirarchical and cdf-rs makes use of this to deserialize (and eventually serialize) .cdf files.

• Arrows indicate the presence of file-offset pointer. Think of | and --> as "points to".
• Some records point to another record of the same type, creating a linked-list.
• The VXR is the only record that can point to a lower-level VXR.
• The Variable Values Record points to a group of contiguous variable records. Any variable can use several VXRs that can each contain several VVRs (or several VXRs).

CDR                                                            Variable Records
|                                                              |
| --> GDR                                                      |
      |             (for each rVariable)                       |  
      | --> rVDR --> rVDR --> ... rVDR                         |
      |     |                                            |---> | #11
      |     |--> VXR --> VXR ... VXR                     |     |
      |          |                                       |     |
      |          |--> VVR -------------------------------|     |
      |          |                                       |     |
      |          |--> CVVR                               |     |
      |          |                                       |---> | #17
      |          |--> VXR --> VXR ...                    +     |
      |               | ...                              ...   |
      |                                                        |
      |             (for each zVariable)                       |
      | --> zVDR --> zVDR --> ... zVDR                         |
      |     |                                            |---> | #4123
      |     |--> VXR --> VXR ... VXR                     |     |
      |          |                                       |     |
      |          |--> VVR -------------------------------|     |
      |          |                                       |---> | #4127
      |          |--> CVVR                               +     |
      |          |                                       ...   |
      |          |--> VXR --> VXR ...                          |
      |               | ...                                    |
      |                                                        |
      |             (for each attribute)                       |
      | --> ADR  --> ADR  --> ... ADR                          |
      |     |                                                  |
      |     |--> AGREDR --> AGREDR --> ... AGREDR              |
      |     |                                                  |
      |     |--> AZEDR  --> AZEDR  --> ... AZEDR               |
      |                                                        |
      | --> UIR  --> UIR  --> ... UIR                          |

Using cdf-rs with serde

In a way, cdf-rs mimics serde's strategy by creating its own data model via types that wrap around native Rust types. In addition, nearly all "CdfTypes" implement serde::Serialize and serde::Deserialize and can be used, for example, to store the contents of the CDF file into a JSON file, or any other format that has serde support.

                     _____________
                     | .cdf file |
                     |___________|
                           |
_____________      ________|_________      ____________________      _________________
| User data | ---> | CDF data model | ---> | serde data model | ---> | Other formats |
|___________|      | (this lib)     |      |__________________|      |_______________|
                   |________________|

For example, after enabling the serde feature, you can use an external crate like serde_json to convert previously read CDF data into a JSON string that is stored into a .json file.

// This example will only compile with the `serde` feature enabled.
#![cfg(feature = "serde")]
use cdf::{cdf::Cdf, error::CdfError};
use std::{fs::File, io::Write};

fn main() -> Result<(), CdfError> {
    
    let cdf_contents = Cdf::read_cdf_file("examples/data/test_alltypes.cdf")?;

    let json_str = serde_json::to_string_pretty(&cdf_contents)
        .map_err(|err| CdfError::Other(err.to_string()))?;

    let mut buffer = File::create("examples/data/test_alltypes.json")?;
    write!(buffer, "{}", json_str)?;
    Ok(())
}

At the moment, any user that wishes to use this model needs to convert their data into the CDF data model. But that is something we could work on later to simplify.

Work in progress

This is a new project and so will likely go through some revisions. Some parts of the CDF specification are not currently implemented.

If you are interested in helping, please raise an issue on Github with whatever you'd like to work on.

To-do:

Short Term
[ ] A proper test for the VXR and VVR. [ ] Handle TimeTt2000, Epoch, and Epoch16 data types appropriately.
[ ] Profile and improve performance.
[ ] Consolidate tests into one (?)
[ ] CDF versions after v3.8.1 support UTF-8 strings.
[ ] Simplifying the record structs by removing unused values.

Long Term
[ ] Encode / serialize into the CDF format.
[ ] Provide a way to easily convert to-from simple data and the CDF data model.
[ ] Implement serializing / deserializing of multi-file CDFs.