Analysis of Broadband RFI Flagging during uGMRT observations

Analysis of Broadband RFI Flagging during uGMRT observations

This utility is meant to read and process the data from the CSV file containing the RFI count and Total Count from the real-time BB RFI filtering in GMRT Wideband Backend (GWB). This file is generated during a uGMRT observation ONLY if the broadband filtering is enabled and the counter commands are provided during the observing run.

The CSV file contains the total samples flagged as RFI and the total number of samples that were processed in a given duration for each antenna and polarization.

The duration of recording and the number of records will depend on how many times and at what interval the counter was written to the file. A single file is generated from for one day. For analysis, the user needs to select the timestamps and the rows corresponding to the duration of the observation.

Detailed document about the method of flagging, the data format of the CSV file and the output of the Python utility can be found here

A detailed Standard Operating Procedure (SOP) for analyzing the RFI counter files is available here

Sample CSV file can be downloaded here

The utility can perform the following functions. It has been tested on 64-bit Linux platform.

It reads the input CSV file and provides a file and bar charts containing

1. Percentage values of all antennas and polarization (generates an output file and bar charts)

2. Percentage values of single antenna individually over time (generates an output file and a single bar chart)

3. Average percentage flagging (generates an output file and single bar chart with average percentage flagging)

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

To analyze counter data for three different test modes, respective python scripts are provided.

1. Regular GTAC observatios : rfi_counter_flagging_v6.py

2. 1:2 Digital copy mode : rfi_counter_flagging_v6_1to2.py

3. 1:4 Digital copy mode : rfi_counter_flagging_v6_1to4.py

Python code for regular GTAC observations (Version 6.0; latest release)

Python code for 1:2 Digital copy mode (Version 7.0; latest release)

Python code for 1:4 Digital copy mode (Version 6.0; latest release)

Note:

When observation crosses the boundary of midnight counter data is stored datewise in two different files. In order to analyze these files can be appended using following command.

Use command from the command prompt: awk 'FNR==1 && NR!=1{next;}{print}' file1.csv file2.csv > output.csv

Files will get appended in the order of input files provided by user.

Software dependencies : Python version 3

Python Packages required :

csv

math

sys

numpy

from operator import truediv

matplotlib

itertools

statistics

command to run this code in Linux from command prompt

python3 ./rfi_counter_flagging_v6.py < csv_input_filename > < csv_out_file > < start row number > < stop row number > < y-limit >

Example : python3 ./rfi_counter_flagging_v6.py rfi_test_19_09_2019.csv out2.csv 17 97 1

For average flagging across all the antennas (polarization-wise), use the Python code for averaging across all the antennas (Version 1.0; latest release)

For any queries/feedback on this utility or for getting a desired CSV files, please contact Kaushal D. Buch (kdbuch@gmrt.ncra.tifr.res.in).