#################### Accessing Data Files #################### GBTIDL reads SDFITS files generated by the sdfits filler program run in Green Bank. During an observation the data are automatically filled into the SDFITS format in real time and these data can be accessed in *online* mode. After an observing session, data are accessed in *offline* mode. Working Online -------------- In Green Bank, recent raw (unfilled) data can be found in ``/home/gbtdata``. As scans are completed, SDFITS files and their corresponding index files and an empty flag file are automatically generated in the directory ``/home/sdfits``. Files are grouped into subdirectories by project name. Note that SDFITS files are only produced for the VEGAS, Spectrometer, Spectral Processor, and Zpectrometer. No SDFITS files are produced automatically for the DCR. To view data being filled into the online directory, simply issue the command :code:`online`. The project name and location of the online data file do not need to be specified. This command also sets GBTIDL into online mode so subsequent GBTIDL commands automatically detect new data as they are filled. There is no need to create SDFITS and index files manually, nor any need to issue special commands to pick up the latest scans. For example: .. code-block:: IDL GBTIDL -> online connects to most recent project’s online SDFITS file and provides the following output: .. code-block:: text Connecting to file: /home/sdfits/project/project.raw.acs.fits File has not been updated in 2.35 minutes. Now, get some recent data, while scan 11 is integrating .. code-block:: IDL GBTIDL -> getfs, 10 When scan 11 is finished, its data are available with no additional filein command necessary, just type .. code-block:: IDL GBTIDL -> getfs, 11 ;4 If problems are encountered with the online system, contact the on-duty telescope operator. Note that all data in ``/home/sdfits`` is read only except for the flag files, which need to be writable so that GBTIDL users can flag and unflag the online data. GBTIDL will remain in online mode until an offline data set is specified for input using the offline, filein or dirin command. Accessing SDFITS Data After an Observing Run -------------------------------------------- Data created by the online SDFITS filler will remain in the ``/home/sdfits`` directory for approximately 3 months before it is automatically deleted. You may wish to preserve your SDFITS files by copying them from ``/home/sdfits/project`` to your own account (where ”project” here is the name of the project that you wish to preserve). To connect to data in the ``/home/sdfits`` directory after the project has finished observing, you can either use the :idl:pro:`filein` command (described :ref:`in the next section `) or the `offline` command: .. code-block:: IDL GBTIDL -> offline, ’AGBT06C_028_05’ Creating SDFITS Files --------------------- After about 3 months time, the SDFITS data in ``/home/sdfits`` are deleted and the “unfilled” data in ``/home/gbtdata`` are moved to ``/home/archive``. You will need to create new SDFITS files in your account if you haven’t copied the originals or if the default filling method is not what you want. If you are storing large data files in Green Bank it is preferred that you use scratch space, such as the ``/home/scratch`` area. This area is meant for temporary storage of large data files. To create SDFITS files from data in ``/home/gbtdata`` or ``/home/archive``, use the ``sdfits`` program, which runs from the unix prompt, only in Green Bank. Help is available for the sdfits program as follows: .. code-block:: bash sdfits -help In addition to the online help, the sdfits program has full documentation at http://safe.nrao.edu/wiki/bin/view/GB/Data/Sdfits including development status, known issues, request list, and usage examples. .. todo:: Transfer this wiki page to gbtdocs. The ``sdfits`` program offers three levels of calibration, identified by the -mode switch as “raw” (default), “cal”, or “avg”. Users of GBTIDL will generally want to use the default of no calibration (-mode=raw) and use GBTIDL routines to do the calibration, instead. With -mode=raw, the SDFITS file will contain one row of data for each data phase. That is, sig calon, sig caloff, ref calon, and ref caloff phases are all stored individually in the SDFITS file. The other modes (cal and avg) are not recommended since they use un-maintained, older calibration code. In the future these will use the same calibration code used by GBTIDL. .. todo:: Check is the information above still applies. The sdfits program writes data into the current directory by default, so it is best to change into a directory in which you can write large files before running sdfits. In Green Bank, you have a quota on your home directory so it is best not to fill large datasets into your home area. Typical use looks like this: .. code-block:: bash cd /home/scratch/[username] sdfits -scans=1:100 /home/gbtdata/AGBT01A_001_01 The resulting SDFITS file, assuming it contains spectrometer (ACS) scans, is called AGBT01A 001 01.raw.acs.fits. If the specified scan range also includes data from the DCR or Spectral Processor, then a separate output file will be created for each backend. Accessing SDFITS Files ---------------------- The filein command is used to identify an SDFITS file as the source of input. The name of the SDFITS file can be supplied as a parameter. If the file is in the directory from which GBTIDL was started, the full path is not required. For example: .. code-block:: IDL GBTIDL -> filein,’mydata.fits’ However, if GBTIDL was not started in the directory in which the file is stored, you must include the path. For example, .. code-block:: IDL GBTIDL -> filein,’/users/aeinstein/mydata.fits’ If you omit the filename and simply type :code:`filein`, you can select the input file using the file selection GUI. GBTIDL associates an index file with each SDFITS file. If the file mydata.fits does not already have an up-to-date index file, it will be created when the filein procedure is run. The index file, in this case, would be called mydata.index and would reside in the same directory as the SDFITS file. The index file is simply an ASCII listing containing information about each row in the SDFITS file. It is used in the GBTIDL I/O system to speed data retrieval and enable searching the data. Accessing Multiple SDFITS Files Simultaneously ---------------------------------------------- It is easy to work with data from multiple SDFITS files at the same time in GBTIDL, provided the data are stored in a single directory. Rather than using filein, simply use dirin to identify the directory and all the SDFITS data in that directory will be available to the Data I/O commands. A single index file (called “dir.index”) is created and there will be a unique index for each row of data throughout the directory. The dirin command takes one parameter, the name of the directory: .. code-block:: IDL dirin,’/users/aeinstein’ You can also simply type dirin and select the desired directory from the file selection GUI. Since scan numbers may repeat from one observing session to another (even within the same project) it often is simpler to work with one SDFITS file at a time. However, GBTIDL does provide ways to work with multiple occurances of the same scan number. For example, data access and calibration commands such as getfs have a parameter called instance that allows you to select which instance of a given scan number to acquire. Summary of the Location of Data ------------------------------- 1. Recent raw data can be found in /home/gbtdata. * These files need to be converted to SDFITS before they can be used in GBTIDL. * After approximately 3 months, these files are moved to /home/archive. 2. Data are archived in /home/archive for long term storage. * These files need to be converted to SDFITS before they can be used in GBTIDL. 3. While observing, SDFITS files are produced automatically in /home/sdfits/project. * The online and offline commands in GBTIDL know about and can use these files. * Files in /home/sdfits are removed as needed to make room for newer data. SDFITS files will remain available for at least 3 months and often longer. They are not archived since they can be recreated from the data in ``/home/gbtdata`` or ``/home/archive``. * The data in /home/sdfits/project was filled using defaults that may not be right for you. You may want to create SDFITS files using your own choice of sdfits settings. * ”project” in ``/home/sdfits/project`` is the name of the project being observed (e.g. AGBT01A 001 01). 4. SDFITS files in user accounts are for longer term storage. * The sdfits program creates SDFITS files in your personal acoount. * The :idl:pro:`filein` and :idl:pro:`dirin` commands in GBTIDL work on files in your account. Listing the Contents of Data Files ---------------------------------- After a file is loaded into GBTIDL using one of the methods described above, there are several ways to summarize the contents of that file. * **summary**: This procedure gives a summary of the scans in an input data file. You can either print the summary to the screen or to a file. For example, to print to the screen, the command is: .. code-block:: IDL GBTIDL -> summary and its result would be something like this: .. code-block:: text Scan Source Vel Proc Seq RestF nIF nInt nFd Az El ------------------------------------------------------------------------------- 79 W3OH -44.0 Track 0 1.667 2 6 1 379.2 16.1 80 W3OH -44.0 Track 0 1.667 2 6 1 379.4 16.2 81 W3OH -44.0 Track 0 1.667 2 6 1 379.5 16.3 82 W3OH -44.0 Track 0 1.667 2 6 1 379.6 16.4 83 W3OH -44.0 Track 0 1.667 2 6 1 379.8 16.4 If you wish to save the summary to a file, use: .. code-block:: IDL GBTIDL -> summary, ’myfile.summary’ * **list**: The records in the input data set can be summarized with the :idl:pro:`list` command. You can choose to list all of the records or only a subset of them using optional parameters. To display a range of records, two optional parameters are needed, a beginning and ending record number. For example, the following will list the first 11 records in the input data set: .. code-block:: IDL GBTIDL -> list,0,10 .. code-block:: text #INDEX SOURCE SCAN PROCEDURE POL IFNUM FDNUM INT SIG CAL 0 W3(OH) 5 Track XX 0 0 0 F T 1 W3(OH) 5 Track XX 0 0 0 T T 2 W3(OH) 5 Track XX 0 0 0 F F 3 W3(OH) 5 Track XX 0 0 0 T F 4 W3(OH) 5 Track XX 0 0 1 F T 5 W3(OH) 5 Track XX 0 0 1 T T 6 W3(OH) 5 Track XX 0 0 1 F F 7 W3(OH) 5 Track XX 0 0 1 T F 8 W3(OH) 5 Track XX 0 0 2 F T 9 W3(OH) 5 Track XX 0 0 2 T T 10 W3(OH) 5 Track XX 0 0 2 F F For more details, type: :code:`usage, ’list’, /verbose`. * **files**: The files command prints to the terminal screen the file names being used for I/O. For example, if you have loaded both continuum and spectral line data: .. code-block:: IDL filein, ’/home/line.fits’ cont filein, ’/home/continuum.fits’ files, /full ; print full path names the :idl:pro:`files` command will return .. code-block:: text spectral line in : /home/line.fits spectral line out : /home/GBTIDL_KEEP.fits continuum in : /home/continuum.fits Note that GBTIDL KEEP.fits is the default output file and it is opened automatically on startup