Difference between revisions of "LoFAR"
(Created page with "==Community Information== ===Community Name=== The Low Frequency Array ===Community Short Name === LOFAR ===Community Website=== http://www.lofar.org/ ===Community Description ...") |
m |
||
Line 1: | Line 1: | ||
{{Template:Community_Engagement}} | |||
{{TOC_right}} | |||
[[Category:Community Engagement]] | |||
==Community Information== | ==Community Information== | ||
===Community Name=== | ===Community Name=== | ||
Line 37: | Line 41: | ||
* LTA encounters a burst access from users, and existing disk space is too small to handle the requests. Since LTA is federated with EGI Cloud, additional resources are immediately assigned to LOFAR LTA to handle the burst access. After that, the additional resources from EGI FedCloud are released. | * LTA encounters a burst access from users, and existing disk space is too small to handle the requests. Since LTA is federated with EGI Cloud, additional resources are immediately assigned to LOFAR LTA to handle the burst access. After that, the additional resources from EGI FedCloud are released. | ||
==Information Viewpoint== | |||
===Data=== | |||
====Data Object Types==== | |||
* Imaging data | |||
* Pulsar data | |||
====Data Size==== | |||
Observational data at rates up to 60 Gbps (650 TB per day), once processed, the amount of data to be kept for a longer time | |||
====Data Collection Size==== | |||
* Exceeded 19 PB of data in the Long-Term Archive (LTA) | |||
* Current growth: 3PB per year | |||
====Data Format==== | |||
Datacubes (3D data): two Fourier spatial coordinate axes plus a spectral axis. A datacube can reach several TB. LOFAR telescope allows up to 488 subbands, which can reach several GBs. Each subband is processed independently. | |||
====Data Locations==== | |||
Currently involves sites in the Netherlands and Germany | |||
====Privacy policy ==== | |||
* LoFAR data made public as of March 2nd 2015. | |||
* Data that has passed the proprietary period becomes public and can be retrieved by anyone. | |||
* Currently, data are still mainly
retrieved by project PIs and collaborators
| |||
===Data Lifecycle=== | |||
[[Image:|500px|LOFAR SYSTEM Dataflow]] | |||
The LOFAR Archive stores data on magnetic tapes. Data cannot be downloaded right away, but has to be copied from tape to disk first. This process is called 'staging’
| |||
Current limitations: | |||
* Stage no more than 10 TB at a time and no more than 20,000 files
| |||
* Staging data from tape to disk might take drives are shared with all users (also non- LOFAR) and requests are queued
| |||
* Staging space is limited and shared between all LOFAR users – system might temporarily run low on disk space
| |||
* Data copy remains on disk for 2 weeks
| |||
* Maintenance and small outages experienced regularly
| |||
Require efficient solutions for data retrieval |
Revision as of 13:57, 30 July 2015
Engagement overview | Community requirements | Community events | Training | EGI Webinars | Documentations |
Community Information
Community Name
The Low Frequency Array
Community Short Name
LOFAR
Community Website
Community Description
LoFAR will be the first large radio telescope system wherein a huge amount of small sensors are used to achieve its sensitivity instead of a small number of big dishes. For the astronomy application, LOFAR is an aperture synthesis array composed of phased array stations. The antennas in each station form a phased array, producing one or many station beams on the sky. Multi-beaming is a major advantage of the phased array concept. It is not only used to increase observational efficiency, but may be vital for calibration purposes. The phased array stations are combined into an aperture synthesis array. The Remote Stations are distributed over a large area with a maximum baseline of 100 km within the Netherlands and 1500 km within Europe.
Community Objectives
LOFAR started as a new and innovative effort to force a breakthrough in sensitivity for astronomical observations at radio-frequencies below 250 MHz
Main Contact Institutions
ASTRON, CSIC, BSC, IAA
Main Contact
- Michael Wise (ASTRON, wise@astron.nl),
- R.F.Pizzo (ASTRON, pizzo@astron.nl ),
- Susana Sanchez-Exposito (CSIC, sse@iaa.es)
- Daniele Lezzi (BSC, daniele.lezzi@bsc.es)
- Jose Sabater Montes (IAA, jsm@iaa.es)
Prior EGI Collaboration
EGI and LoFAR have been collaborating since Oct 2014 to integrate calibration, analysis and modelling pipelines of radio-astronomy data into a cloud infrastructure. It is developed jointly by users of the [www.lofar.org LOFAR] radio-telescope and members of the AMIGA4GAS project. https://wiki.egi.eu/wiki/FedCloudLOFAR
Science Viewpoint
Scientific challenges
- In most cases, ingest jobs by the Radio Observatory need to be monitored closely to verify that all files are ingested and to manually recover the situation after a failure. This causes quite some inconvenience for some users, who have to wait for several days to get their data.
- Instability of the ingest system can cause long ingest queues and, inevitably, can make CEP2 very full. In extreme cases, the observing schedule needs to be rearranged because there is not enough disk space available on CEP2 to store more data till important ingest jobs are completed and the corresponding data can be removed from the cluster. This obviously limits the observing efficiency.
- Larger file number/size for staging required
- Fully exploit processing resources offered by the LOFAR Long-Term Archive
Objectives
- Efficient user data retrieval.
- Optimise the data staging, e.g., using pre-staging technology to move data from tapes to computing facilities to reduce the waiting for staging when a user requests to retrieve data
- Allowing user to process large amount of data and retrieve results only to avoid downloading the data to their local computer.
- Elastic disk storage space to allow data ingest jobs to be smoothly executed when handling bust computation
User Stories
LOFAR requests a future system to efficiently support large volumes of data access and burst data access, in particularly, to support the following 2 scenarios:
- User wants to retrieve large volumes of data from LTA. He finds the desired datasets by using the searching facilities provided by LoFAR data portal. An optimisation mechanism is installed which accelerates data staging process. User also starts data processing/analysing service/application running at LTA HPC/Cloud which are near the datasets. The dataset is injected into the processing/analysing service/application, and produced the results. User examines the results using a visualisation service, and downloads the results on his local PC.
- LTA encounters a burst access from users, and existing disk space is too small to handle the requests. Since LTA is federated with EGI Cloud, additional resources are immediately assigned to LOFAR LTA to handle the burst access. After that, the additional resources from EGI FedCloud are released.
Information Viewpoint
Data
Data Object Types
- Imaging data
- Pulsar data
Data Size
Observational data at rates up to 60 Gbps (650 TB per day), once processed, the amount of data to be kept for a longer time
Data Collection Size
- Exceeded 19 PB of data in the Long-Term Archive (LTA)
- Current growth: 3PB per year
Data Format
Datacubes (3D data): two Fourier spatial coordinate axes plus a spectral axis. A datacube can reach several TB. LOFAR telescope allows up to 488 subbands, which can reach several GBs. Each subband is processed independently.
Data Locations
Currently involves sites in the Netherlands and Germany
Privacy policy
- LoFAR data made public as of March 2nd 2015.
- Data that has passed the proprietary period becomes public and can be retrieved by anyone.
- Currently, data are still mainly retrieved by project PIs and collaborators
Data Lifecycle
[[Image:|500px|LOFAR SYSTEM Dataflow]]
The LOFAR Archive stores data on magnetic tapes. Data cannot be downloaded right away, but has to be copied from tape to disk first. This process is called 'staging’ Current limitations:
- Stage no more than 10 TB at a time and no more than 20,000 files
- Staging data from tape to disk might take drives are shared with all users (also non- LOFAR) and requests are queued
- Staging space is limited and shared between all LOFAR users – system might temporarily run low on disk space
- Data copy remains on disk for 2 weeks
- Maintenance and small outages experienced regularly
Require efficient solutions for data retrieval