On April 24, 2024, Misha Mesarcik, one of the PhD students I had the pleasure to supervise, successfully defended his PhD thesis on Machine learning-based anomaly detection for radio telescopes at the University of Amsterdam and ASTRON. Misha worked in the NWO-funded Efficient Deep Learning (EDL) programme. Congratulations Misha!
Many thanks to the committee members who read and judged the thesis, and in some cases traveled far to attend the defense. Many thanks also to the other Supervisors: Albert-Jan Boonstra (ASTRON), Elena Ranguelova (Netherlands eSciecne Center), and Cees de Laat (University of Amsterdam).
In modern radio telescopes such as LOFAR, system health management is crucial for early detection of errors and for remedying them. As radio telescope data volumes are ever increasing, manually searching for system errors is becoming untenable. AI approaches to detect and classify error patterns are potentially much more accurate and complete than the manual inspection route. Misha developed AI tools to detect and cluster different error patterns and regular events in LOFAR spectrogram data. The picture above shows an example of nine clusters of event types that were detected with high accuracy.
Following this success, a pilot proposal was presented to the LOFAR stakeholders and is awaiting implementation and testing.
Misha’s thesis is titled “Machine learning-based anomaly detection for radio telescopes“. Here the abstract:
Radio telescopes are getting bigger and are generating increasing amounts of data to improve their sensitivity and resolution. The growing system size and resulting complexity increases the likelihood of unexpected events occurring thereby producing datasets containing anomalies. These events include failures in instrument electronics, miscalibrated observations, environmental and astronomical effects such as lightning and solar storms as well as problems in data processing systems among many more. Currently, efforts to diagnose and mitigate these events are performed by human operators, who manually inspect intermediate data products to determine the success or failure of a given observation. The accelerating data-rates coupled with the lack of automation results in operator-based data quality inspection becoming increasingly infeasible.
This thesis focuses on applying machine learning-based anomaly detection to spectrograms obtained from the LOFAR telescope for the purpose of System Health Management (SHM). It does this across several chapters, with each chapter focusing on a different aspect of SHM in radio telescopes. We provide an overview of the data processing systems in LOFAR so to create a workflow for SHM that could effectively be integrated into the scientific data processing pipeline.