ODB - Ordinary Data Base - a simplified interface to NetCDF and HDF libraries.

Beginner's Guide to ODB: (This is a "Point and Click" Index but you can also scroll it.)

(HOME - means goto the BEGINNING of this document, END - goto the END:-)
Send your comments or questions to:

Senya Basin

LDGO of Columbia University

Palisades NY 10964 senya@rosie.ldgo.columbia.edu

Introduction

The purpose of this library is to help users to get access to a data stored in a popular data formats like NetCDF or HDF. Our objective was to give a user the simplest interface we could think of as well as help in maintaining some obvious data structures widely used in the Oceanographic community. In the near future an interface to a new powerful "CUF-Format" will be available via ODB-library.

Two Simple Examples.

We will begin with some examples which should help to start using ODB.

Reading a NetCDF/HDF File.

Suppose you want to read the temperature data from a NetCDF(or HDF) file GCMoutput.cdf which has a variable temp among others (like salt, vel, dens). We will assume that you know the dimensions of temp (NX = 360 and NY = 180) obtained by using ncdump for example.

The fragment of a Fortran code which does this will be the following:


     dimension temp(360,180) 
c.....................open a file 
     call odb_open  (idfile, 'GCMoutout.cdf', 0)  
c.....................read the data for temperature
     call odb_rdvar (idfile, 'temp', temp)  
c.....................close the file 
     call odb_close (idfile) 

Writing a NetCDF File.

Now we want to show how to write your data in a NetCDF format so it will be self documented, self descriptive and recognized by other without additional comments from you on almost any computer on the Internet!

In order to make it easier for Fortran-programmers to understand the usage of the ODB library we have included two Fortran fragments which write similar information, but one - in a form which is undocumented as well as a machine-dependent binary and another in Network transparent NetCDF.

Fortran Unformatted File:                       NetCDF File:
parameter(MX=45, MY=32, MT=12)    parameter (MX=45,MY=32,MT=12)               
real aa(MX,MY),                   real aa(MX,MY),                             
*    xx(MX), yy(MY), tt(MT)       *    xx(MX),yy(MY),tt(MT)                   
                                                                              
open (unit = 1, file = 'abc',     call odb_open (idf, 'abc', 1)               
*     form = 'unformatted')                                                   
                                  call odb_wrdesc(idf, 'The Super-Duper GCM'//
                                  * 'Temperature; 0.0001 sec of run')         

write (1) MX, MY, MT              call odb_setxyt(idf, 'longitude',MX,xx,     
                                  * 'latitude',MY,yy, 'Time',MT,tt, 'TEMP')   
write (1) xx                                                                  
write (1) yy                                                                  
write (1) tt                                                                  
                                                                              
do IT = 1, MT                     do IT = 1, MT                               
   call compute (MX, MY, IT, aa)     call compute (MX, MY, IT, aa)            
   write (1) aa                      call odb_wrxy (idf, IT, 'TEMP', aa)      
enddo                             enddo                                       

close (1)                         call odb_close (idf)                        

Top 10 ODB Functions

[?] odb_open(idf, file, key) - opens a file
idf - /output/ integer variable used to store the file identifier
file - character string: file name
key - 0-read only, 1-update or create; 2-always re-create
[?] odb_dfgr(idf, gname, nn) - defines a grid with a given name & dimension
idf - from previos call to odb_open
gname - character grid name (one word)
nn - grid's dimension (integer)
[?] odb_dfvar?(idf, gn1, gn2, ... gnN, vname) - defines a variable based on a specific set of grids
gn1...gnN - grid names from the previous call to odb_dfgr
vname - variable's name (one word)
[?] odb_setxyt(idf,xunits,NX,xx,yunits,NY,yy,tunits,NT,tt,vname) - defines a variable based on X,Y and T grids.
[x|y|t]units - units names for X, Y, and T grids
NX, NY, NT - number of points for X, Y, T grids
xx, yy, tt - arrays with values for X,Y and T grid data(real)
vname - variable's name (one word)
[?] odb_wrgr(idf, gname, xx) - writes the gridpoints values to a NetCDF file.
gname - grid's name
xx - array with grid's data (real)
[?] odb_wrvar(idf, vname, var) - writes the values of a variable to NetCDF file.
vname - variable's name
var - array of data (real by default.)
[?] odb_rdgr(idf, gname, nx, xx) - reads the gridpoints values from a NetCDF file
gname - /input/ grid's name
nn - /output/ grid's dimension
xx - /output/ array with grid's data (real)
[?] odb_rdvar(idf, vname, var) - reads the values of a variable
vname - variable's name
var - array to br filled by the data (real)
[?] odb_set?attr(idf, vname, aname, val) - write an attribute for a variable
vname - variable's name
aname - attribute name
val - attribute value (if ? =c - character; =r real; =i integer; =d double)
[?] odb_wrdesc(idf, desc) - write a description for the file
desc - description character string.

Major Hints & Tricks

* How to Compile your Program?

Just add the following libraries to your f77 options list: '-lodb -lnetcdf -lsun' - if you are going to read or write/create a NetCDF files. (Use '-lodb -lnetcdfdf -ldf -lsun' if you want to read HDF files as well.

Example: f77 -o abc abc.f -lodb -lnetcdf -lsun

* How to Convert your old Fortran Unformatted Data Files into NetCDF format?

The following example may be helpful:
   subroutine tocdf(MX, MY, MT, xx, yy, tt, func)
c------------------------------------------------
   real xx(MX), yy(MY), tt(MT), func(MX,MY)
   common /grids/ xstart, xstep, ystart, ystep, tstart, tstep

   do i = 1, MX
10 xx(i) = xstart + xstep*real(i-1)
   do i = 1, MY
20 yy(i) = ystart + ystep*real(i-1)
   do i = 1, MT
30 tt(i) = tstart + tstep*real(i-1)

   open (unit = 1, file = 'psi.data', form = 'unformatted')
   call odb_open (idf, 'psi.cdf', 1)
   
   call odb_wrdesc (idf, 'Vorticity from the 1972 Barotropic Model.')
   call odb_setxyt (idf,'longitude',MX,xx,'latitude',MY,yy,'time','psi')  

   do i = 1, MT
      read(1) func
      call odb_wrxy (idf, i, 'psi', func)
   enddo

   call odb_close(idf)
   return	
   end

* How to Create an Expandable NetCDF Data Set?

We will call a dataset an expandable one if it has an unlimited dimension like a time grid. Such data sets are essential as an output of time-dependent models. There is a limit (of NetCDF protocol) for only one such dimension (or grid) per a datafile. (See "CUF-format" for an unlimited number of such dimensions). In order to use it:
  1. Define an expandable grid using odb_dftm(idf,gname) instead of odb_dfgr.
  2. Make sure that this grid is the last for all variables which have it, so you will write data in chunks along this grid
  3. Do not write the gridvalues for this grid before other data, but add them gradually as you are adding new chunks of data along that grid with: odb_wrtm (idf, gridname, indx, gridvalue), or just call once: odb_wrgr (idf, gridname, gridvalues) after all other data has been already written in the file.
You can reopen such a file and start adding new points to the expandable grid .

* How to Get a Grid's Dimension from a NetCDF file?

The routine odb_rddm (idf, gridname, ndim) will return a grid's dimension in NDIM.

* How to Write a Vector (multicomponent) Data?

We assume that for 2D and 3D vectors each component should be treated as a separate variable with a different name (like "TAUX" & "TAUY"), but with the same grids. The greater number of vector components may be represented by adding an additional grid (or dimension). For a "standard" Cartesian X-Y-Z or X-Y-T type of data, there are few functions which will define 2D & 3D vectors:
for X-Y-T grids:

call odb_set2xyt(idf,xunits,nx,xx,yunits,ny,yy,tunits,nt,tt,vname1,vname2)
call odb_set3xyt(idf,xunits,nx,xx,yunits,ny,yy,tunits,nt,tt,vname1,vname2,vname3)

or for expandable dimension 'T':

call odb_set2xyte(idf,xunits,nx,xx,yunits,ny,yy,tunits,nt,tt,vname1,vname2)
call odb_set3xyte(idf,xunits,nx,xx,yunits,ny,yy,tunits,nt,tt,vname1,vname2,vname3)

for X-Y-Z grids:

call odb_set2xyz(idf,xunits,nx,xx,yunits,ny,yy,zunits,nz,zz,vname1,vname2)
call odb_set3xyz(idf,xunits,nx,xx,yunits,ny,yy,zunits,nz,zz,vname1,vname2,vname3)

* How to Set a MISSING VALUE Flag?

If a variable has a missing data it is a good practice to add an attribute for such variable which will store the flag:

call odb_setrattr(idf, vname, 'missing_value', flag)

* How to Use, Update or Create the HDF Files?

If you link with a standard NetCDF library (-lnetcdf -lsun) you will be able to use/generate ONLY NetCDF files. BUT!!

There is an NetCDF interface for HDF library which is now a part of HDF distribution. After you relink with (-lnetcdfdf -ldf -lsun) you will be capable of reading and updating HDF files without any change for your ODB-based code!!!

If you want to create HDF files using ODB, you should:

Please note that a call to: odb_open, will still be creating NetCDF files as by default and will be capable of reading/updating existed HDF files, but you may want to use an explicit : odb_opcdf(idf, filename, key) instead.

* How to change a Format Of Data?

You can READ data using odb-library from an existed file not worrying about the internal format: just use a correct specification for a receptor variable.

When you CREATING a new variable it will be an 32-bit floating point by default. If you want to change the format, you may use "odb_setfmt(fmt)" before you define a variable (or a grid). (Don't forget to set it back if you plan to define a variable of another type later in your code). The following parameters are valid:

an example below shows how to define and write INTEGER*2 and DOUBLE PRECISION variables:
INTEGER*2        mask
DOUBLE PRECISION temp
...
call odb_setfmt('i2')
call odb_dfvar2(idf, 'X','Y', 'Mask')
call odb_setfmt('r8')
call odb_dfvar2(idf, 'X','Y', 'Temperature')
call odb_setfmt('r4')
...
call odb_wrvar(idf, 'Mask', mask)
call odb_wrvar(idf, 'Temperature', temp)
...

* How to get a copy of ODB library?

The source for ODB library is available here: One may want to tune the Makefile a little bit as it was designed for SGI make clone: "pmake".

Full Index


subroutine odb_open (idf, file, key) subroutine odb_close (idf) subroutine odb_dfgr (idf, gname, nn) subroutine odb_dftm (idf, gn1) subroutine odb_dfvar1 (idf, gn1, vname) subroutine odb_dfvar2 (idf, gn1, gn2, vname) subroutine odb_dfvar3 (idf, gn1, gn2, gn3, vname) subroutine odb_dfvar4 (idf, gn1, gn2, gn3, gn4, vname) subroutine odb_dfvar (idf, ngr, vgrn, vname) subroutine odb_wrgr (idf, gn1, xx) subroutine odb_wrtm (idf, tname, indx, tval) subroutine odb_wrvar (idf, vname, var) subroutine odb_wrxv (idf, igrn, vname, var) subroutine odb_wr1v2 (idf, igr2, vname, var) subroutine odb_wr1v3 (idf, igr2, igr3, vname, var) subroutine odb_wr1v4 (idf, igr2, igr3, igr4, vname, var) subroutine odb_wr2v3 (idf, igr3, vname, var) subroutine odb_wr2v4 (idf, igr3, igr4, vname, var) subroutine odb_wr3v4 (idf, igr4, vname, var) subroutine odb_rddm (idf, gn1, nn) subroutine odb_rdgr (idf, gn1, nn, xx) subroutine odb_rdvar (idf, vname, var) subroutine odb_rdxv (idf, igrn, vname, var) subroutine odb_rd1v2 (idf, igr2, vname, var) subroutine odb_rd1v3 (idf, igr2, igr3, vname, var) subroutine odb_rd1v4 (idf, igr2, igr3, igr4, vname, var) subroutine odb_rd2v3 (idf, igr3, vname, var) subroutine odb_rd2v4 (idf, igr3, igr4, vname, var) subroutine odb_rd3v4 (idf, igr4, vname, var) subroutine odb_getvdim (idf, vname, ndim, mdim) subroutine odb_setcattr (idf, vname, aname, val) subroutine odb_setiattr (idf, vname, aname, val) subroutine odb_setrattr (idf, vname, aname, val) subroutine odb_setdattr (idf, vname, aname, val) subroutine odb_getdattr (idf, vname, aname, val) subroutine odb_getrattr (idf, vname, aname, val) subroutine odb_getiattr (idf, vname, aname, ival) subroutine odb_wrdesc (idf, desc) subroutine odb_setfmt (fmt) subroutine odb_setxyt (idf,xunits,nx,xx,yunits,ny,yy,tunits,nt,tt,vname) subroutine odb_set2xyt (idf,xunits,nx,xx,yunits,ny,yy,tunits,nt,tt,v1,v2) subroutine odb_setxyte (idf,xunits,nx,xx,yunits,ny,yy,tunits,vname) subroutine odb_set2xyte (idf,xunits,nx,xx,yunits,ny,yy,tunits,v1,v2) subroutine odb_setxyz (idf,xunits,nx,xx,yunits,ny,yy,zunits,nz,zz,vname) subroutine odb_set2xyz (idf,xunits,nx,xx,yunits,ny,yy,zunits,nz,zz,v1,v2) subroutine odb_wrxy (idf, indxz, vname, var) subroutine odb_wrxz (idf, indxy, vname, var) subroutine odb_wryz (idf, indxx, vname, var) subroutine odb_wrxye (idf, time, vname, var) subroutine odb_rdxy (idf, indxz, vname, var) subroutine odb_rdxz (idf, indxy, vname, var) subroutine odb_rdyz (idf, indxx, vname, var)

A Simple MATLAB extention

As a part of the project an extremely simplified Matlab interface to read NetCDF files is implemented (just use odb() command in Matlab)

SYNOPSIS:

    [object] = odb( [idf], 'COMMAND', [args...]) 

COMMANDS:

    'open', 'dim', 'var','attr','help'...,
    may be arbitrary abbreviated. See odb('help') for details.

DESCRIPTION:

    idf = odb('open','name')
          - opens file  for reading, returns a file ID
            which may be used in further references

    nx = odb(idf,'dim','X')
          - returns a value of a dimension 'NX' from
            a previously opened file with ID=idf
    v1 = odb(idf,'var','v1')
          - returns values of 1-D variable V1 as a vector v1
    a12 = odb(idf,'var','A12')
          - returns values of 2D variable  as a matrix a12
            (if A12 was A12(NY,NX) variable in Netcdf file then
            matrix a12 has NX rows and NY columns)
    v1 = odb(idf,'var','A12','X', ix)
          - extracts a column from a 2D variable , which
            corresponds to the index  of the dimension 
    v = odb(idf,'var','V', [ i1 i2 ... iN] )
          - extracts values of N-dimensional variable  
            as a 1-2D matrix in accordance with an index array.
            If value of index in that array equal 0 the whole
            dimension will be retrieved, otherwise only values
            for specified index will be read.
    a = odb(idf,'attr','V','A')
          - reads the value(s) of attribute  of a variable 

    odb('help') - prints this help

EXAMPLE:

For plotting the temperature contours from the following NetCDF file:

netcdf sst {
dimensions:
        X = 90 ;
        Y = 38 ;
variables:
        float X(X) ;
                X:units = "longitude" ;
        float Y(Y) ;
                Y:units = "latitude" ;
        float sst(Y, X) ;
                sst:description = "Sea Surface Temperature";
                sst:missing_value = NaNf ;
}
...one may want to use the following commands:

>> id  = odb('open', 'sst');
>> x   = odb(id, 'var', 'X');
>> y   = odb(id, 'var', 'Y');
>> sst = odb(id, 'var', 'sst');
>> contour(x,y,sst');
>> title(odb(id, 'attr', 'sst', 'description'));

In order to use this matlab interface you should compile the "modb.c"
file with the matlab's cmex compiler and properly install the library.

A Long Example


      program 
c....An example: How to convert the ASCII files to netCDF files.

      parameter(NX =72, NY =36)
      parameter(ALON1 = 122.5, DLON = 5., ALAT1 = 87.5, DLAT =-5.)
      parameter(cmul = 0.1, cref = 5000.)
      parameter(VALICE = -1000, VALAND = -32768, ZOTL = -999.9, ZOTI = -1.7)
c
      integer NORMAL(NX,NY), MONTH
      real sst(NX,NY), alon(NX), alat(NY), amon(12)
      character*80 cdfnam
c
      call getarg(1,cdfnam)
c 
c.....Open new netCDF file
      call odb_open(idf,cdfnam,2)

c.....Calculate values of grid and output
      do 10 n=1,nx
10    alon(n)=alon1+(n-1)*dlon
      do 20 n=1,ny
20    alat(n)=alat1+(n-1)*dlat
      do 30 m=1,12
30    amon(m)=m-1

c.....Write a file description
      call odb_wrdesc(idf,'UKMO SST Normals')

c.....Define grids and set a variable
      call odb_setxyt(idf,'longitude',nx,alon,
     *     'latitude',ny,alat,'month',12,amon,'SST')
c
c.....Define missing value
      call odb_setrattr(idf,'SST','missing_value',zotl)

c.....Read data, convert to real numbers and output
      DO k = 1,12
         read (5,'i6') MONTH
         read (5,'72i6')) NORMAL
         do j = 1, ny
            do i = 1, nx
               if    (NORMAL(i,j) .ge. cref) then
                  sst(i,j) = (NORMAL(i,j)-cref)*cmul
               elseif(NORMAL(i,j).eq.valice) then
                  sst(i,j) = ZOTI
               elseif(NORMAL(i,j).eq.valand) then
                  sst(i,j) = ZOTL
               else
                  sst(i,j) = NORMAL(i,j)*cmul
               endif
            enddo
         enddo
c.....Write one time-slice of data
         call odb_wrxy(idf, K, 'SST',sst)
      ENDDO

c.....Close NetCDF file
      call odb_close(idf)
      stop
      end


senya@rosie.ldgo.columbia.edu