How to observe with MUSTANG-2

1. Project Preparation

1.1 Prepare observing scripts

Before you observe you need to have prepared your observing scripts and chosen your primary/flux calibrators, your OOF sources, and secondary calibrators. For a guide on how to do all of these things see this guide for instructions on preparing your scripts.

2. Observing Preparation

2.1 Observing Log

During observing, you are expected to edit the MUSTANG-2 observing run notes wiki and take notes of what’s occurred throughout the night.

1. Well in advance of your observations, create a new page and entry at the bottom of the observing logs wiki by clicking “Edit Wiki text”

2. Follow the naming convention of entry above <AGBTsemester_project-code_session-number>, e.g. AGBT18A_014_01

3. On the new log page you have created you can put in the text from a log template. On the log template page, go to “Edit”, copy that text, and paste it into your new log (also in the “Edit” mode). You will have to get rid of some extra spaces.

Attention

Save your log often (not only at the end of the observation) to ensure that your notes are being saved!

Tip

When you are actually recording information during observing you can be in either the “Edit Wiki Text” or “Edit” modes. But for some reason copying the formatting from the log template to your log has to be done in the “Edit” mode.

Note

If you don’t have permissions yet to edit the wiki and are observing, you can take notes in a text document and email them to the MUSTANG-2 team afterwards to upload to the wiki for you.

2.2 Connect

Open and connect to VNC session or start an XFCE FastX session on titania or ariel via FastX.

Connection Issues?

The internet at GBO can be intermitent at times. Specifically there are days that the internet goes down for 30-60 seconds at a time quite often. Are you having issues with FastX or your VNC being really laggy? Check this status page to see the status of the ssh gateways. See these instructions for FastX and VNC workarounds using Charlottesville to potentially better your connection.

2.3 AstrID

Open an AstrID session and navigate to your corresponding MUSTANG-2 project. The MUSTANG-2 instrument team should have already populated your AstrID area with appropriate scripts.

2.4 CLEO

The following are suggested CLEO windows to have open during observing:

• Launch → Receivers → Mustang2

  • To monitor specific thermometers, click the gray box next to the titles:

    • PT Fridge 1

    • PT Fridge 2

    • Array

    • HE4 Fridge 1 Charcoal

    • He4 Fridge 2 Charcoal

    • He3 Charcoal

    • He4 Fridge 1 Evap

    • He4 Fridge 2 Evap

    • He3 Evap.

    These thermometers are of interest because they can indicate that things are wrong before they affect the array, or help diagnose what is wrong and how to fix it if the array temperature starts to go up. Sometimes a cryocycle gets started by accident - in which case if you are looking at the charcoal you can hit abort quickly and no damage is done. Other times a helium4 might run out and that can pull up the array temperature - not much you can do but often you can still collect some good data for a while.

• Launch → Status

• Launch → Antenna

• Launch → Observer Tools → Scheduler & Skyview

  • Click on Real-time mode

  • Load in catalogs:

    • your science target catalogs

      • Click Catalog... in the upper right-hand corner

      • Click Add/Select/DeSelect Catalogs ...

      • Select Add User Catalog

      • Select the path to your science catalog (consult your observing scripts). They may be in /users/penarray/Public/Catalogs/Science_catalogs/

      • Click Apply

    • ALMA grid calibrators (flux calibration and often OOF sources)

      • Click Catalog... in the upper right-hand corner

      • Click Add/Select/DeSelect Catalogs ...

      • Select Add User Catalog

      • Select the path /users/penarray/Public/Catalogs/alma_gridcal.cat

      • Click Apply

    • mustang_pointing

      • Click Catalog... in the upper right-hand corner

      • Click Add/Select/DeSelect Catalogs ...

      • Select mustang_pointing

      • Click Apply

      • It is suggested that you enter in a minimum of 0.5 Jy

• Launch → Observer Tools → Talk and Draw

2.5 Checklist if setup was done awhile ago

If the setup (tuning, get data flowing, etc.) was done a while ago (several hours) and there is no other M2 project before you, there are a few things you should check before starting to take data.

2.5.1 Reconnect to Roaches

Over many hours temperature drifts will make the tuning less optimal. Therefore if the tuning was done a while ago (several hours), you need to reconnect to the roaches and check the tuning.

To check the tuning:

1. To reconnect to the roaches execute /users/penarray/Public/startMUSTANG.bash <project1> reconnect where <project> is the project code (e.g., AGBT26A_XXX_sessionNum). For more information about the ./startMUSTANG.bash script see Advanced Useage of the startMUSTANGbash.

2. Run um1.plotIQ() in each roach’s terminal.

3. If needed, run um1.fixIQ() per roach.

2.5.2 Check that data is flowing

Go to the M2 CLEO manager and check that data is flowing (see how to do this and some fixes if data is not flowing here).

2.5.3 Check biases

If biases were set manually by hand before the observations, check that they are not 0 in the Miscelaneous tab of the CLEO M2 manager. If they are 0 it means that the manager likely crashed and you’ll need to Restart the manager.

3. Observing Procedure

3.1. Communicate with operator

At least 15 minutes before your observing start time, get on Talk & Draw, tell the operator who you are and what project you are observing for. At this point you can also ask who the operator is.

Hint

You can check who the current operator is by checking the current shift of the operators log.

3.2. Fill in AstrID info

In AstrID under ObservationManagement, go to the Run tab and fill in the Observer and Operator information.

3.3. Fill in weather and correction info in observing log

Go to your observing log and fill in the following:

• Weather conditions:

  • Temperature

  • Humidity

  • Wind Velocity

  Note

  You can get all of this weather information from the GbtStatus tab in AstrID, the “Weather” CLEO application, or from the “Status” CLEO application.

• The focus and pointing corrections before you submit any corrections:

  • LFCy - to find this value, look at YFC in the CLEO Status screen.

  • Az LPC - to find this value, look at Az LPC in the CLEO Status screen.

  • El LPC - to find this value, look at El LPC in the CLEO Status screen.

Attention

If the pointing corrections are >30” ask the operator to zero them out. If you leave large corrections in you likely will not see anything in your OOF scans.

3.4. Take control

Once the member of the M2 instrument team has finished biasing and the operator tells you are in the gateway/gives you the go ahead, in AstrID → File → Real time mode … → Select work online with control of the telescope.

3.5. Configure

Run the 1_m2setup script in AstrID.

3.6. OOF

Attention

Before and after OOF is a good time to check that the active surface settings are as they should be. See Monitoring the Surface Models or M2 active surface documentation for what to check and how to check them.

Hint

If operator asks you if you want to zero out any corrections (they might ask about pointing, focus, and thermals), the M2 instrument team suggests that you keep all the solutions. But make sure that you note in the observing log what the corrections were before the OOF (see 3.3. Fill in weather and correction info in observing log). If you get a bad OOF solution, that is when you can consider zeroing out various corrections.

OOF procedure:

1. Make sure that you have changed mySrc in 2_m2oof and run the 2_m2oof script in AstrID.

2. For the first OOF of the night, you need to have calSeq=True so that a skydip is done as a part of the OOFing process. An OOF will take ~20 minutes to run.

3. Check the OOF results in AstrID → DataDisplay → OOF and re-rerun if necessary.

   For M2, we typically apply the z5 corrections. When the corrections are available, press the button that reads After selecting the Zernike solution above, click this green button to send the solutions to the telescope.

   Note

   For advice on what you are looking for in OOF solutions see the Inspecting AutoOOF Solutions section of the AutoOOF Guide. For good and bad M2 OOF examples see the MUSTANG-2 section of the AutoOOF guide.

Hint

While your OOF is running, it is a good time to:

• Start the m2gui to check M2 data while observing. It can be useful to start two m2guis: one for analyzing calibrator data and one to analyze science data.

• In the m2gui

  • Once the skydip has finished, Check the Tipping Scan (Skydip) and write down the number of live detector in the log.

  • check what each of the OOF images looks like (see Make Calibrator Map)

Note

During this initial data acquisition (and to some extent, throughout the night) check your MUSTANG-2 manager CLEO screen, and make sure that the numbers in sections such as Frame Cntr and Roach Data are continuing to change with time (if so, the boxes will mostly be blue). However, if they stop (indicated when the boxes turn lavender) then the M2 manager has crashed, and you’ll need to Restart the manager. Note that M2 manager is known to crash on the 1st scan of a session (the skydip).

3.7 Quick daisy on OOF source

1. Run the 2_m2quickDaisyPrimary script on your OOF/calibrator source

   It’s best if you can make your OOF source and your calibrator source the same.

2. Use the m2gui to Fit Calibrator Map and determine:

   • beam shape (WidthA & WidthB)

   • peak of the source (Peak_Height)

3. Record these values in your observing log

4. It’s a good idea to check the time streams (see the check time streams section for instructions and examples)

3.8 Quick daisy on secondary calibrator

1. Run the 3_m2quickDaisySecondary script on your secondary calibrator

2. Use the m2gui again to Fit Calibrator Map and determine:

   • beam shape (WidthA & WidthB)

   • peak of the source (Peak_Height)

3. Record these values in your observing log

Note

Occasionally, one or more detectors may show glitches in their time streams. These faulty detectors appear as artifacts in the maps and can distort the map’s scaling. As a result, the source may appear fainter than it truly is, and this can lead to a poor fit to a point source or even prevent the GUI from fitting the point source altogether. See Use crmask to Mask Bad Detectors on how to mask the bad detector(s).

3.9 Take science data

Take ~30 minutes of science data followed by a quick daisy on your seconday calibrator. Often this is accomplished by submitting several science scripts (e.g., 5_science_rX) in AstrID. For example, often for cluster science each individual science scan is ~8-9 minutes in length. So if you are submitting individual science scans (which 5_science_rX are), you can submit 4 of the science scripts in a row followed by your secondary calibrator scan.

You should check the time streams in the m2gui (see the check time streams section for instructions and examples) of each science scan.

You can make maps and SNR maps of your science target using the m2gui (see make science maps section for instructions).

3.10 Continue to take science data

1. Continue to do ~30 minutes of science data followed by a quick daisy on the secondary calibrator for the rest of the night.

2. Monitor the beam size (WidthA and WidthB) and the Peak_Height using the m2gui to determine if you need to OOF again.

3.11 Observing primary calibrators

It is safest to observe multiple primary/flux calibrators in an observing session. You should have your OOF source be a primary calibrator but when you observe an additional primary calibrator later in the observing session, the sequence of observing should be:

• secondary

• science scans

• secondary

• primary calibrator

• science scans

• secondary

Primary and secondary calibrators need to be observed as close in time as is feasible.

Note

In February, ALMA is shutdown and thus ALMA grid cals are not observed during this period. See 3.1.1 Primary Calibrators in February for more information on which flux calibrators to observe during the ALMA shutdown.

3.12 When to OOF again?

First, know that there is no right answer and you have to gain experience to get a feel for when to OOF again. But the following is some advice for when to consider OOFing again.

The two criteria for determining if you need to OOF again are the Peak_Height and the width of the beam.

The Peak_Height is important as it is related to the efficiency of the telescope. If the PeakHeight is down by more than ~15-25%, you will want to do another OOF. It can be a good idea to take another.

Optional

If you don’t have much observing time left, once the PeakHeight is down by more than 15%, instead of redoing the OOF scan, you can do another m2QuickDaisy on the secondary source to be sure that it is that low, and then do two more science scans until the PeakHeight has gone down by another 15% (so a cumulative 30%).

Generally, you want to monitor the size and shape of the beam. If the largest diameter of the beam (whichever of WidthA or WidthB) is 12-13” you’ll want to consider doing another OOF. For more extended sources (like clusters), a slightly more diffuse beam isn’t so harmful; you can stretch a beam up to 13” for diffuse sources. But for point sources the beam size matters more so a largest beam width of ~12” is worrisome. Additionally, you should monitor the shape of the beam. If the beam has become quite elliptical (i.e., WidthA and WidthB become very different from one another), you’ll want to consider doing another OOF. In the case of M2, a nice beam is 9”x9”. An elliptical beam is when there is a \(≳\) 3” difference between WidthA and WidthB.

Often you will see that a decrease in PeakHeight and a degradation of the beam will come hand in hand.

If you have a 1/2 hour of a project left, it isn’t worth it to OOF again because OOF takes ~a half hour so you likely wouldn’t get much if any observing time done after the OOF.

Attention

Make sure that you are not being fooled by a bad detector or two. A bad detector will affect your measurements of the calibrator. See Use crmask to Mask Bad Detectors for how to mask bad detectors. You will see a bad detector in your map as a bright daisy pattern or in the time streams.

3.13 Be aware - Issue with quadrant detector

In early 2023 it was discovered that over the past year or two the quadrant detector sometimes isn’t working and doesn’t write files to /home/gbtdata/project_code_sesion/QuadrantDetector as we expect. The GUI now will pop up a warning box (WARNING QD Values are missing for scans: ...) if it detects that the quadrant detector files are not being written.

If this happens during observing, press ok and ask the operator to restart the quadrant detector manager.

Warning

However if you get a warning about just ONE file, this is not a problem. Most likely the scan is not finished yet. There may be an issue with the quadrant detector only if you get a pop-up notification about SEVERAL scans.

3.14 Observing Tips and Tricks

1. You can open two m2gui windows, one for working with your calibrators and one for making science maps.

2. Save your wiki observing log often (not only at the end of the observation) to ensure that your notes are being saved!

3. We note that when observing with MUSTANG-2 on the GBT, the preferred maximum elevation limit of a target is 75 degrees. It is possible to observe targets up to 80 degrees elevation but this is not preferable. The hard limit is around 84 degrees. At these higher elevations, the MUSTANG-2 beam becomes large because the GBT cannot keep up with the slewing speeds required to map and track the source. Conversely, the preferred minimum elevation is 30 degrees. However, it is possible to but can go lower, but lower than 30 is hard on the hardware.

4. General Advice for Determining “Bad Weather“

Once you have some indication of bad weather (bad skydip, bad time streams, or physical weather indication), you will want to make an educated guess as to what the trajectory of the weather/data is in order to determine whether or not to keep observing or give up the time. There are many tools that you can use to an assessment of this trajectory. Consider, do the following suggest that the remainder of your scans would be scientifically useful? (this can be used as a checklist of sorts)

• Time streams

  • Check the time streams of the science scans as laid out above in Checking Time Streams. Are they wiggly? How wiggly? See examples below in 4.1 Examples of effect of bad weather.

  • How many “bad” science scans have there been in a row?

• Skydip(s)

  • How does the first skydip of night look? How wiggly is it? See examples below 4.1 Examples of effect of bad weather.

  • If you are seeing indications of bad weather and you decide to OOF again one could add a skydip in to test the weather (calSeq=True).

  • One could even do a one off skydip.

• Beam

  • Has the beam been deteriorating?

• Weather forecast

  • Ask the operator what the weather is like

  • Use the m2gui to get Weather Info about the wind and radar

  • Weather Underground for Green Bank, WV

  • Windy for Green Bank, WV

  • NOAA for Green Bank, WV

  • NOAA Atlantic Coast Radar

  • Check the All High Frequency Overview Plots via the DSS

  • Check other reputable weather forecasters (Weather underground, weather.forcast.gov, Windy, etc.)

• Direct communication with the operator

  • Ask the operator what the weather is like. Since you asked at the beginning of the observation you have one data point.

  • This also serves as a way to keep the operator in the loop and aware of a potentially imminent decision to relinquish telescope control.

Note

The observer should reach out to the operator once the concern of bad weather is identified to let them know that the weather is a concern. This could be as early as the first bad scan (time streams, whether a science scan or those from a skydip). A good practice is that if there are two consecutive scans with bad time streams, the operator should be notified and consulted at this point. That doesn’t mean a decision needs to be made this early on, but it lays the groundwork so that both parties are aware of a potentially imminent decision to relinquish telescope control. If the observer has doubts, reach out to an M2 team member after a second bad scan.

A few data/weather trajectories are as follows:

• Improve

  • Is it a one off? As in its just a cloud passing by?

  • Is the or will the weather improve?

• Stay the same. Is the weather staying bad and not improving?

• Get worse. Is the trajectory getting worse and worse?

You will need to monitoring the situation over time and over multiple scans in order to make a guess about the trajectory of the data. One note is the it is usually never sufficient to come across one bad scan and call it quits. There is usually always some nebulous time span (~half hour to an hour) to determine that things are bad and staying bad. If you think the weather will improve and the improvement should happen soon and give ample time for valuable science scans, then the suggestion is to try to endure the bad weather. However, for weather staying the same and getting worse, the advice is to rely on the other metrics to make a determination, except for the case that the operator identifies clear precipitation with no expectation for improvement. At that point, one can give up the time promptly if it’s heavily raining or snowing.

When making a judgment call as to whether to give up the time due to bad weather, consider the following cases:

• How much time is left? If there is not much time left it is less likely that the weather will change.

• Are you observing a faint target? If you give up amount of time you have left, will that amount of time you have left make a difference for your science?

• How much time has been observed for the project and how much time is left in the project? We ask for a factor of 2 of overheads so maybe there is time to tolerate bad weather.

Note

~30 minutes is a rough minimum amount of time to relinquish control, but the operator will need some time to prepare a backup project so this is why it is good to keep in touch with the operator throughout this process. So the general advice is that if you give up the time near the end of an observation, the minimum time left in an observing session would be ~45 minutes.

Note

The flip side of overheads (i.e. maybe the project can tolerate bad weather) is that if you are observing the last session (using up all awarded time), any rescheduled observing would all go to overheads. If it’s not the last session, then the advice is to give up the remainder of time for bad weather (if all bad-weather items are checked).

Again, when in doubt you can always call an M2 team member to help you make the call of whether or not to give up the time.

4.1 Examples of effect of bad weather

Here are some examples of science time streams and skydips in good and bad weather.

Faint Science

Good Time Stream

This is what a good, unaffected faint science time streams (a cluster) looks like in good weather - flat.

Bad Time Stream

This is what faint science time streams look like when they are heavily affected by weather - very wiggly.

Skydip

Good Skydip

This is what a good skydip looks like in good weather - not wiggly.

Bad Skydip

This is what a skydip looks like when it is heavily affected by weather - very wiggly.

Note

It is difficult to see the affect of weather in calibrator time streams as the signal from the point source is quite bright.

5. Checklist for Changing M2 Projects

Multiple M2 projects can be scheduled in a single night. If you are observing for an M2 project that is not the first M2 project of the night (a subsequent M2 project), you need to do the following:

5.1 Check for Tuning Files

Tuning files need to be linked to an observing session. This is done one of two ways either: - the tuner includes the second project in the tuning process (put the second observing project/session as a second argument separated by a comma in the tuning process) - or if they did not, you will have to create a symlink.

If you are observing for a second project in the night, it is best to communicate with the tuner to make sure they include the second project. But if you didn’t, before you start observing check to see if the tuning directory for this second project exists /home/gbtlogs/Rcvr_MBA1_5tuning/. If it does not follow the instructions below to create a symlink for the tuning.

Before you begin observing, login to egret as lmonctrl (ssh lmonctrl@egret.gbt.nrao.edu) and type:

cd /home/gbtlogs/Rcvr_MBA1_5tuning/
ln -s <old_project_session> <new_project_session>

where old_project_session is the full name of the previous M2 project and new_project_session is the second M2 project of the night that you are observing for.

Warning

Be very careful to put in the right project and session ID or this step will not work and you won’t get any data. You can ask the previous observer for the old project session ID, or look for it by typing:

ls -ltr /home/gbtdata/

The last modified file will tell you what the most recent project ID was.

5.2 Check data is flowing

Go to the M2 CLEO manager and check that data is flowing (see how to do this and some fixes if data is not flowing here).

5.3 Determine if you need to OOF

At least 15-30 minutes before your observations, you’ll need to get a sense of how the beam is doing and if you need to OOF at the beginning of the project. To do this, (a) ask the observer of the previous M2 project via Talk and Draw how the beam is and if they recommend OOFing, and (b) check the observing log of the previous session to see how the beam is doing.

5.4 Configure

When the observing time for the second project starts, you need run 1_m2setup in AstrID as usual.

Warning

Some people think they can skip this step when changing from another MUSTANG-2 run. This is not the case. It’s very important to still run 1_m2setup at the beginning of your session.

5.5 Get a Skydip

You need a skydip at the beginning of this project.

You can get this skydip one of two ways: - If you need to re-OOF, make sure that calSeq=True to get a skydip. - If you do not need to re-OOF, do a stand-alone skydip (typically called skydip) and change myAz to the Azimuth of whatever your first source will be (calibrator, etc.). The telescope will slew to that Azimuth and do the skydip.

Note

Note that if you run the stand-alone skydip in the m2gui you will see that scan 1 is a “Track” scan and scan 2 is a “Tip” scan. The skydip is scan 2. You only get one scan when you run the skydip as part of an OOF.

5.4 Flux calibrator

You’ll also want to still observe your flux calibrator using the m2quickdaisy script.

Warning

This is another thing people think they can skip, but it makes reduction later more difficult. Check the beam with this flux calibrator.

6. Observing Troubleshooting

6.1 MUSTANG-2 Manager

See M2 manager documentation and typical issues on the MUSTANG-2 Manager page.

6.2 You have less detectors than you are expecting

If have less live detectors than expected (usually found through skydip or seeing many detectors missing from map), reconnect to the roaches and run um1.fixIQ() in each roach’s terminal.

Attention

If you change anything about the instrument during an observing session (like bringing detectors back via um1.fixIQ()), when you OOF after those changes, make sure that you take a skydip as a part of AutoOOF (i.e., set calseq=True). More information about the discovery of this procedure here.

7. Closing up for the night

7.1 Go offline

In AstrID, go from working online to working offline:

• File → Real time mode … → work offline.

7.2 Shutdown M2

For the shutdown process you can either do this (a) automatically or (b) manually. For BOTH you need to be in the gateway for MUSTANG-2 (not just the observing gateway).

Automatic Shutdown

Execute the following in a terminal:

/users/penarray/Public/stopMUSTANG.bash

Manual Shutdown

1. Set detector biases to zero

   • Go to the Mustang Manager in CLEO

   • Click on the miscellaneous tab

   • In the top middle, you will see 4 rows of Det Bias 1-4, corresponding to the 4 roaches.

   • Unlock the manager

   • roach-by-roach:

     • type 0 in the left DetBias box

     • press enter

     • wait until the blue box (right DetBias box) shows a DetBias of 0

     • repeat this step for all 4 roaches.

2. Turn off data transmission

   • Mustang2 CLEO scan turn off DataXinit for all four roaches.

   Note

   You will need to be in gateway AND unlock both the unlock and advanced features unlock buttons to do this.

3. Turn off components

   In VNC session, go to http://mustangboot.gbt.nrao.edu and turn off the roaches, HEMTs, and Function Generator by checking those three boxes then go to left of the screen and click ‘Off’ (gray button).

4. Turn on daily cycle

   Mustang2 CLEO window

   • go to Housekeeping

   • unlock

   • recheck daily cycle to be on and put autocycle trigger to HE4

     This means that if either of the He4 fridges run out it starts a cycle.

   • set the daily cycle time = 0.65 of a day in UT

     This is the time of day that the daily cycle starts measured in fraction of a day (UT). 0.65 is a nice balance between ensuring the cycle is over by the time any observations are likely to come up, yet not so early that there is no time to work with the receiver in the morning.

7.3 Finish updates to log

Finish updates to your observing log. It is best to do this while it is all fresh in your memory.

7.4 Send observing summary to M2 team

Send a short summary of how observing went and a link to the observing log to the M2 instrument team via the google group email address. You can simply just respond to the email thread where observing was coordinated with this information.

7.5 Kill VNC session

Either kill your FastX session or your VNC session via the terminal.

Congratulations!

You’re all done! Now, let’s do some science with that data!