Monitoring Magnetic Fields: Data Transmission to INTERMAGNET

The earth’s magnetic field is constantly changing. Most of us, of course, go about our lives unconcerned about the phenomenon. But, for scientists, the mystery of the magnetic field and its variations are engrossing, and, for more than 180 years, there have been attempts to record changes in the magnetic field using magnetometers.

A digital fluxgate magnetometer to measure variations in three perpendicular directions
and an Overhauser magnetometer for total field measurement used by CSIR-NGRI

More recently, magnetic data from different parts of the world is collected by INTERMAGNET, the International Real-time Magnetic Observatory Network. It is not easy to bring together data from more than 140 magnetic observatories. So there are Geomagnetic Information Nodes in five countries to collect the data for collating and sharing.

Initially, INTERMAGNET used to collect one-minute magnetic data in hourly chunks. But the need for higher time resolution has pushed most magnetic observatories to record and share one-second data.

An example of one-minute data of the field in X, Y and Z directions and the total field

The international effort is fuelled by more than sheer curiosity about this basic force of nature. The earth’s magnetic field sometimes undergoes abrupt changes due to changes in the density of particles and the speed of solar wind. This leads to geomagnetic storms, disturbing navigation systems and creating electrical spikes in transmission lines. So there are some practical problems that need to be tackled based on our understanding.

India too, contributes to this effort. The CSIR-National Geophysical Research Institute has had an observatory in Hyderabad since 1964. And, in 2012, another magnetic observatory was established in the Choutuppal campus, 65 kilometres from Hyderabad.  The recording and transmission of data went on fine till 2017.

After the cooperation between the German Research Centre for Geosciences, GFZ, and NGRI ended, real-time data transmission from the two NGRI observatories to the information node at Edinburgh became challenging. The system for uninterrupted data transmission was too costly for NGRI to procure, manage and maintain. And not transmitting the data would prove to be scientifically costly.

So Sai Vijay Kumar Potharaju and N. Phani Chandrasekhar decided to create software for the purpose de novo, using Python. Python is open source, available out of the box in Linux. It is easy to write and test code in Python. That should take care of the software requirement. But what about the hardware?

The data acquisition systems connected to magnetometers at Hyderabad and Choutuppal do not have internal memory. They provide information with real-time plots of acquired and filtered data, logs of different events and errors of the recording systems, and even information about any unexpected shutdown. Any problem with the limited disc space and goodbye valuable data.

So the researchers configured the data acquisition system to transfer the data to workstations deployed at Choutuppal with the Linux OS. And, with an adequately logical script, data from Choutuppal could be transferred to the server at Hyderabad.

But sending one-minute data in hourly chunks by email or as email attachment to the Geomagnetic Information Node at Edinburgh demanded constant effort, leaving little time for the scientists to attend to other important tasks. So there was a need to automate the process. Moreover, there was a bottleneck in the transfer of data from Choutuppal. A fibre optic cable from the rural area was too costly an option. So the connection was made through a local internet provider. The solution to the problem needed to take into account low bandwidth and occasional interruption of services. Password protected, secure file transfer protocol was not adequate. The initial cross platform data transmission had a batch file to transfer data with a delay of one minute. It was fortified with an abort option in case of interruptions in connection. And there had to be methods to compare and synchronise the data to avoid loss or gaps during transfer. The researchers set up the system so that every 120 seconds data is double-checked automatically for completeness. Now they were ready for the final lap: transmitting the data to the Geomagnetic Information Network.

Establishing a perfect ‘handshake’ with the Centos system at the Geomagnetic Information Network did not pose any problem. The synchronisation and repetition of the data every 300 seconds took place as envisaged. The sorted data from the Linux systems was transmitted as per the information standards of the Geomagnetic Information Network with a latency of only 300 seconds. Perfect.

But…

The same data was being sent repeatedly with the same latency period to the receiving web service. Huge volumes of data from both observatories were clogging up the web service and the disc space.

Sai Vijay Kumar and Phani Chandrasekhar were quick to respond. They dropped the synchronisation protocol and created background daemons instead to recheck the data and execute the Python code every 300 seconds. That made transmission smooth and there was no duplication of data.

Then came another minor problem: the complete data was not being plotted at the INTERMAGNET. Why? The data was indeed being transmitted and successfully.

Experts at the INTERMAGNET suggested uploading a full day’s data to double check. And that too went off smoothly. So the problem was at the plotting system, and not of the automatic transmission of data.

Thus, Sai Vijay Kumar and Phani Chandrasekhar at CSIR-NGRI successfully created a stable, sturdy system for transferring geomagnetic data at a fraction of the cost. Many magnetic observatories in India and in other developing countries will, one day, benefit from these consistent efforts to create an indigenous system for the purpose.

Reference: Sai Vijay Kumar Potharaju and Phani Chandrasekhar Nelapatla, Development of a robust real‑time synchronized data transmission technique from a Magnetic Observatory to an INTERMAGNET GIN Scientific Reports 12:10277 (2022); DOI: 10.1038/s41598-022-13820-y