Accessing Data Files

Accessing Data Files#

GBTIDL reads SDFITS files generated by the sdfits filler program run in Green Bank. During an observation the data are automatically filled into the SDFITS format in real time and these data can be accessed in online mode. After an observing session, data are accessed in offline mode.

Working Online#

In Green Bank, recent raw (unfilled) data can be found in /home/gbtdata. As scans are completed, SDFITS files and their corresponding index files and an empty flag file are automatically generated in the directory /home/sdfits. Files are grouped into subdirectories by project name. Note that SDFITS files are only produced for the VEGAS, Spectrometer, Spectral Processor, and Zpectrometer. No SDFITS files are produced automatically for the DCR.

To view data being filled into the online directory, simply issue the command online. The project name and location of the online data file do not need to be specified. This command also sets GBTIDL into online mode so subsequent GBTIDL commands automatically detect new data as they are filled. There is no need to create SDFITS and index files manually, nor any need to issue special commands to pick up the latest scans. For example:

GBTIDL -> online

connects to most recent project’s online SDFITS file and provides the following output:

Connecting to file: /home/sdfits/project/project.raw.acs.fits
File has not been updated in 2.35 minutes.

Now, get some recent data, while scan 11 is integrating

GBTIDL -> getfs, 10

When scan 11 is finished, its data are available with no additional filein command necessary, just type

GBTIDL -> getfs, 11 ;4

If problems are encountered with the online system, contact the on-duty telescope operator. Note that all data in /home/sdfits is read only except for the flag files, which need to be writable so that GBTIDL users can flag and unflag the online data. GBTIDL will remain in online mode until an offline data set is specified for input using the offline, filein or dirin command.

Accessing SDFITS Data After an Observing Run#

Data created by the online SDFITS filler will remain in the /home/sdfits directory for approximately 3 months before it is automatically deleted. You may wish to preserve your SDFITS files by copying them from /home/sdfits/project to your own account (where ”project” here is the name of the project that you wish to preserve). To connect to data in the /home/sdfits directory after the project has finished observing, you can either use the filein command (described in the next section) or the offline command:

GBTIDL -> offline, ’AGBT06C_028_05’

Creating SDFITS Files#

After about 3 months time, the SDFITS data in /home/sdfits are deleted and the “unfilled” data in /home/gbtdata are moved to /home/archive. You will need to create new SDFITS files in your account if you haven’t copied the originals or if the default filling method is not what you want. If you are storing large data files in Green Bank it is preferred that you use scratch space, such as the /home/scratch area. This area is meant for temporary storage of large data files. To create SDFITS files from data in /home/gbtdata or /home/archive, use the sdfits program, which runs from the unix prompt, only in Green Bank. Help is available for the sdfits program as follows:

sdfits -help

In addition to the online help, the sdfits program has full documentation at http://safe.nrao.edu/wiki/bin/view/GB/Data/Sdfits including development status, known issues, request list, and usage examples.

Todo

Transfer this wiki page to gbtdocs.

The sdfits program offers three levels of calibration, identified by the -mode switch as “raw” (default), “cal”, or “avg”. Users of GBTIDL will generally want to use the default of no calibration (-mode=raw) and use GBTIDL routines to do the calibration, instead. With -mode=raw, the SDFITS file will contain one row of data for each data phase. That is, sig calon, sig caloff, ref calon, and ref caloff phases are all stored individually in the SDFITS file. The other modes (cal and avg) are not recommended since they use un-maintained, older calibration code. In the future these will use the same calibration code used by GBTIDL.

Todo

Check is the information above still applies.

The sdfits program writes data into the current directory by default, so it is best to change into a directory in which you can write large files before running sdfits. In Green Bank, you have a quota on your home directory so it is best not to fill large datasets into your home area. Typical use looks like this:

cd /home/scratch/[username]
sdfits -scans=1:100 /home/gbtdata/AGBT01A_001_01

The resulting SDFITS file, assuming it contains spectrometer (ACS) scans, is called AGBT01A 001 01.raw.acs.fits. If the specified scan range also includes data from the DCR or Spectral Processor, then a separate output file will be created for each backend.

Accessing SDFITS Files#

The filein command is used to identify an SDFITS file as the source of input. The name of the SDFITS file can be supplied as a parameter. If the file is in the directory from which GBTIDL was started, the full path is not required. For example:

GBTIDL -> filein,’mydata.fits’

However, if GBTIDL was not started in the directory in which the file is stored, you must include the path. For example,

GBTIDL -> filein,’/users/aeinstein/mydata.fits’

If you omit the filename and simply type filein, you can select the input file using the file selection GUI.

GBTIDL associates an index file with each SDFITS file. If the file mydata.fits does not already have an up-to-date index file, it will be created when the filein procedure is run. The index file, in this case, would be called mydata.index and would reside in the same directory as the SDFITS file. The index file is simply an ASCII listing containing information about each row in the SDFITS file. It is used in the GBTIDL I/O system to speed data retrieval and enable searching the data.

Accessing Multiple SDFITS Files Simultaneously#

It is easy to work with data from multiple SDFITS files at the same time in GBTIDL, provided the data are stored in a single directory. Rather than using filein, simply use dirin to identify the directory and all the SDFITS data in that directory will be available to the Data I/O commands. A single index file (called “dir.index”) is created and there will be a unique index for each row of data throughout the directory. The dirin command takes one parameter, the name of the directory:

dirin,’/users/aeinstein’

You can also simply type dirin and select the desired directory from the file selection GUI. Since scan numbers may repeat from one observing session to another (even within the same project) it often is simpler to work with one SDFITS file at a time. However, GBTIDL does provide ways to work with multiple occurances of the same scan number. For example, data access and calibration commands such as getfs have a parameter called instance that allows you to select which instance of a given scan number to acquire.

Summary of the Location of Data#

Recent raw data can be found in /home/gbtdata.
- These files need to be converted to SDFITS before they can be used in GBTIDL.
- After approximately 3 months, these files are moved to /home/archive.
Data are archived in /home/archive for long term storage.
- These files need to be converted to SDFITS before they can be used in GBTIDL.
While observing, SDFITS files are produced automatically in /home/sdfits/project.
- The online and offline commands in GBTIDL know about and can use these files.
- Files in /home/sdfits are removed as needed to make room for newer data. SDFITS files will remain available for at least 3 months and often longer. They are not archived since they can be recreated from the data in /home/gbtdata or /home/archive.
- The data in /home/sdfits/project was filled using defaults that may not be right for you. You may want to create SDFITS files using your own choice of sdfits settings.
- ”project” in /home/sdfits/project is the name of the project being observed (e.g. AGBT01A 001 01).
SDFITS files in user accounts are for longer term storage.
- The sdfits program creates SDFITS files in your personal acoount.
- The filein and dirin commands in GBTIDL work on files in your account.

Listing the Contents of Data Files#

After a file is loaded into GBTIDL using one of the methods described above, there are several ways to summarize the contents of that file.

summary: This procedure gives a summary of the scans in an input data file. You can either print the summary to the screen or to a file. For example, to print to the screen, the command is:

GBTIDL -> summary

and its result would be something like this:

Scan Source Vel Proc Seq RestF nIF nInt nFd Az El
-------------------------------------------------------------------------------
W3OH -44.0 Track 0 1.667 2 6 1 379.2 16.1
W3OH -44.0 Track 0 1.667 2 6 1 379.4 16.2
W3OH -44.0 Track 0 1.667 2 6 1 379.5 16.3
W3OH -44.0 Track 0 1.667 2 6 1 379.6 16.4
W3OH -44.0 Track 0 1.667 2 6 1 379.8 16.4

If you wish to save the summary to a file, use:

GBTIDL -> summary, ’myfile.summary’

list: The records in the input data set can be summarized with the list command. You can choose to list all of the records or only a subset of them using optional parameters. To display a range of records, two optional parameters are needed, a beginning and ending record number. For example, the following will list the first 11 records in the input data set:

GBTIDL -> list,0,10

#INDEX SOURCE SCAN PROCEDURE POL IFNUM FDNUM INT SIG CAL
W3(OH) 5 Track XX 0 0 0 F T
W3(OH) 5 Track XX 0 0 0 T T
W3(OH) 5 Track XX 0 0 0 F F
W3(OH) 5 Track XX 0 0 0 T F
W3(OH) 5 Track XX 0 0 1 F T
W3(OH) 5 Track XX 0 0 1 T T
W3(OH) 5 Track XX 0 0 1 F F
W3(OH) 5 Track XX 0 0 1 T F
W3(OH) 5 Track XX 0 0 2 F T
W3(OH) 5 Track XX 0 0 2 T T
W3(OH) 5 Track XX 0 0 2 F F

For more details, type: usage, ’list’, /verbose.

files: The files command prints to the terminal screen the file names being used for I/O. For example, if you have loaded both continuum and spectral line data:
```
filein, ’/home/line.fits’
cont
filein, ’/home/continuum.fits’
files, /full ; print full path names
```
the files command will return
```
spectral line in : /home/line.fits
spectral line out : /home/GBTIDL_KEEP.fits
continuum in : /home/continuum.fits
```
Note that GBTIDL KEEP.fits is the default output file and it is opened automatically on startup