Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

LoFAR

From EGIWiki
Revision as of 12:57, 30 July 2015 by Ychen (talk | contribs)
Jump to navigation Jump to search
Engagement overview Community requirements Community events Training EGI Webinars Documentations


Community Information

Community Name

The Low Frequency Array

Community Short Name

LOFAR

Community Website

http://www.lofar.org/

Community Description

LoFAR will be the first large radio telescope system wherein a huge amount of small sensors are used to achieve its sensitivity instead of a small number of big dishes. For the astronomy application, LOFAR is an aperture synthesis array composed of phased array stations. The antennas in each station form a phased array, producing one or many station beams on the sky. Multi-beaming is a major advantage of the phased array concept. It is not only used to increase observational efficiency, but may be vital for calibration purposes. The phased array stations are combined into an aperture synthesis array. The Remote Stations are distributed over a large area with a maximum baseline of 100 km within the Netherlands and 1500 km within Europe.

Community Objectives

LOFAR started as a new and innovative effort to force a breakthrough in sensitivity for astronomical observations at radio-frequencies below 250 MHz

Main Contact Institutions

ASTRON, CSIC, BSC, IAA

Main Contact

  • Michael Wise (ASTRON, wise@astron.nl),
  • R.F.Pizzo (ASTRON, pizzo@astron.nl ),
  • Susana Sanchez-Exposito (CSIC, sse@iaa.es)
  • Daniele Lezzi (BSC, daniele.lezzi@bsc.es)
  • Jose Sabater Montes (IAA, jsm@iaa.es)

Prior EGI Collaboration

EGI and LoFAR have been collaborating since Oct 2014 to integrate calibration, analysis and modelling pipelines of radio-astronomy data into a cloud infrastructure. It is developed jointly by users of the [www.lofar.org LOFAR] radio-telescope and members of the AMIGA4GAS project. https://wiki.egi.eu/wiki/FedCloudLOFAR

Science Viewpoint

Scientific challenges

  • In most cases, ingest jobs by the Radio Observatory need to be monitored closely to verify that all files are ingested and to manually recover the situation after a failure. This causes quite some inconvenience for some users, who have to wait for several days to get their data. 

  • Instability of the ingest system can cause long ingest queues and, inevitably, can make CEP2 very full. In extreme cases, the observing schedule needs to be rearranged because there is not enough disk space available on CEP2 to store more data till important ingest jobs are completed and the corresponding data can be removed from the cluster. This obviously limits the observing efficiency.
  • Larger file number/size for staging required 

  • Fully exploit processing resources offered by the LOFAR Long-Term Archive

Objectives

  • Efficient user data retrieval.
    • Optimise the data staging, e.g., using pre-staging technology to move data from tapes to computing facilities to reduce the waiting for staging when a user requests to retrieve data
    • Allowing user to process large amount of data and retrieve results only to avoid downloading the data to their local computer.
  • Elastic disk storage space to allow data ingest jobs to be smoothly executed when handling bust computation

User Stories

LOFAR requests a future system to efficiently support large volumes of data access and burst data access, in particularly, to support the following 2 scenarios:

  • User wants to retrieve large volumes of data from LTA. He finds the desired datasets by using the searching facilities provided by LoFAR data portal. An optimisation mechanism is installed which accelerates data staging process. User also starts data processing/analysing service/application running at LTA HPC/Cloud which are near the datasets. The dataset is injected into the processing/analysing service/application, and produced the results. User examines the results using a visualisation service, and downloads the results on his local PC.
  • LTA encounters a burst access from users, and existing disk space is too small to handle the requests. Since LTA is federated with EGI Cloud, additional resources are immediately assigned to LOFAR LTA to handle the burst access. After that, the additional resources from EGI FedCloud are released.

Information Viewpoint

Data

Data Object Types

  • Imaging data
  • Pulsar data

Data Size

Observational data at rates up to 60 Gbps (650 TB per day), once processed, the amount of data to be kept for a longer time

Data Collection Size

  • Exceeded 19 PB of data in the Long-Term Archive (LTA)
  • Current growth: 3PB per year

Data Format

Datacubes (3D data): two Fourier spatial coordinate axes plus a spectral axis. A datacube can reach several TB. LOFAR telescope allows up to 488 subbands, which can reach several GBs. Each subband is processed independently.

Data Locations

Currently involves sites in the Netherlands and Germany

Privacy policy

  • LoFAR data made public as of March 2nd 2015.
  • Data that has passed the proprietary period becomes public and can be retrieved by anyone.
  • Currently, data are still mainly 
retrieved by project PIs and collaborators 


Data Lifecycle

[[Image:|500px|LOFAR SYSTEM Dataflow]]

The LOFAR Archive stores data on magnetic tapes. Data cannot be downloaded right away, but has to be copied from tape to disk first. This process is called 'staging’ 
 Current limitations:

  • Stage no more than 10 TB at a time and no more than 20,000 files 

  • Staging data from tape to disk might take drives are shared with all users (also non- LOFAR) and requests are queued 

  • Staging space is limited and shared between all LOFAR users – system might temporarily run low on disk space 

  • Data copy remains on disk for 2 weeks 

  • Maintenance and small outages experienced regularly 


Require efficient solutions for data retrieval