Difference between revisions of "EGI-Engage:TASK JRA1.3 Accounting"
Jump to navigation
Jump to search
(14 intermediate revisions by 5 users not shown) | |||
Line 1: | Line 1: | ||
{{Template:EGI-Engage menubar}} {{TOC_right}} | {{Template:EGI-Engage menubar}} {{TOC_right}} | ||
[[Category:EGI-Engage]] | [[Category:EGI-Engage]] | ||
'''Task leader: ''' | '''Task leader: '''Adrian Coveney/STFC <BR> | ||
'''Task leader deputy: ''' | '''Task leader deputy: '''Greg Corbett/STFC | ||
'''Mailing list:'''TBD | '''Mailing list:'''TBD | ||
Line 48: | Line 48: | ||
|- | |- | ||
| 3.1.1 | | 3.1.1 | ||
| Cloud Accounting: Usage Record V0.4 (Benchmarks etc) | | '''Cloud Accounting: Usage Record V0.4 (Benchmarks etc)''' | ||
| 03/15 | | 03/15 | ||
| | | 07/16 | ||
| | | Closed | ||
| | | | ||
| | | | ||
|- | |- | ||
| 3.1.2 | | 3.1.2 | ||
| Storage Accounting: Contact sites and test of the current solution | | '''Storage Accounting: Contact sites and test of the current solution''' | ||
| 03/15 | | 03/15 | ||
| | | 10/15 | ||
| | | Closed | ||
| | | | ||
| Fix rolled out and new format being loaded successfully. | |||
|- | |- | ||
| 3.1.3 | | 3.1.3 | ||
| Cloud Accounting: Long running VMs | | '''Cloud Accounting: Long running VMs''' - Adding support for VMs that run for a period that crosses a month boundary so that their usage is apportioned to each month. | ||
| 03/15 | | 03/15 | ||
| | | 07/16 | ||
| Open | |||
| | |||
| | | | ||
| Several attempts at a solution proposed. All rejected by developers. New solution being tested at central repository. | |||
|- | |- | ||
| 3.1.4 | | 3.1.4 | ||
| | | '''ARC Parser''' | ||
| 04/15 | | 04/15 | ||
| | | 07/16 | ||
| Open | |||
| | |||
| | | | ||
| Parser code written and tested against test data. Contacting sites to test. | |||
|- | |- | ||
| 3.1.5 | | 3.1.5 | ||
| Data Accounting: proof of concept | | '''Data Accounting: proof of concept''' | ||
| 06/15 | | 06/15 | ||
| 01/16 | | 01/16 | ||
| | | Closed | ||
| | | | ||
| | | | ||
|- | |- | ||
| 3.1.6 | | 3.1.6 | ||
| Storage Accounting: Usage Record V2.0 | | '''Storage Accounting: Usage Record V2.0''' | ||
| 07/ | | 07/16 | ||
| | | 10/16 | ||
| | | Open | ||
| 3.1.2 | | 3.1.2 | ||
| | | | ||
|- | |- | ||
| 3.1.7 | | 3.1.7 | ||
| XML Parser | | '''XML Parser''' | ||
| 07/16 | | 07/16 | ||
| 11/16 | |||
| Planned | | Planned | ||
| | | | ||
| | | Requirements need developing. | ||
|- | |- | ||
| 3.1.8 | | 3.1.8 | ||
| D3.3: Accounting Repository Release | | '''D3.3: Accounting Repository Release''' | ||
| 02/16 | | 02/16 | ||
| 02/16 | | 02/16 | ||
| | | Closed | ||
| | | 3.1.2, 3.1.5 | ||
| | | | ||
|- | |- | ||
| 3.1.9 | | 3.1.9 | ||
| Data Accounting: minimal requirements implemented | | '''Data Accounting: minimal requirements implemented''' | ||
| 02/16 | | 02/16 | ||
| 09/16 | | 09/16 | ||
| | | Closed | ||
| 3.1.5 | | 3.1.5 | ||
| | | | ||
|- | |- | ||
| 3.1.10 | | 3.1.10 | ||
| Storage Accounting: support new storage systems | | '''Storage Accounting: support new storage systems''' | ||
| | | 09/16 | ||
| | | 02/17 | ||
| Planned | | Planned | ||
| 3.1.6 | | 3.1.6 | ||
Line 128: | Line 128: | ||
|- | |- | ||
| 3.1.11 | | 3.1.11 | ||
| D3.6: Analysis on techniques to manage big data on the EGI accounting system | | '''D3.6: Analysis on techniques to manage big data on the EGI accounting system''' | ||
| | | 02/16 | ||
| | | 02/16 | ||
| | | Closed | ||
| | | | ||
| Date moved much earlier than originally expected. | |||
|- | |- | ||
| 3.1.12 | | 3.1.12 | ||
| Investigating Big Data tools to improve accounting repository | | '''Investigating Big Data tools to improve accounting repository''' | ||
| 07/16 | | 07/16 | ||
| 02/17 | | 02/17 | ||
| | | Closed | ||
| 3.1.11 | | 3.1.11 | ||
| | | | ||
|- | |- | ||
| 3.1.13 | | 3.1.13 | ||
| | | '''SQL parsers''' | ||
| 08/16 | | 08/16 | ||
| 03/17 | | 03/17 | ||
| | | Open | ||
| | | | ||
| Requirements need developing. | |||
|- | |- | ||
| 3.1.14 | | 3.1.14 | ||
| D3.8: First data accounting prototype | | '''D3.8: First data accounting prototype''' | ||
| 10/16 | | 10/16 | ||
| 10/16 | | 10/16 | ||
| | | Closed | ||
| 3.1.9 | | 3.1.9 | ||
| | | | ||
|- | |- | ||
| 3.1.15 | | 3.1.15 | ||
| Data Accounting: production requirements | | '''Data Accounting: production requirements''' | ||
| 10/16 | | 10/16 | ||
| 05/17 | | 05/17 | ||
Line 168: | Line 168: | ||
|- | |- | ||
| 3.1.16 | | 3.1.16 | ||
| GPGPU Accounting | | '''GPGPU Accounting''' | ||
| 10/16 | | 10/16 | ||
| 05/17 | | 05/17 | ||
| | | Open | ||
| | | | ||
| Batch systems still do not support accounting for GPGPU usage. Looking into accounting for VM usage of GPGPUs. | |||
|- | |- | ||
| 3.1.17 | | 3.1.17 | ||
| Support AAI evolution | | '''Support AAI evolution''' | ||
| 01/17 | | 01/17 | ||
| 08/17 | | 08/17 | ||
| | | Open | ||
| | | | ||
| | | | ||
|- | |- | ||
| 3.1.18 | | 3.1.18 | ||
| D3.12: Second release of the Accounting Repository | | '''D3.12: Second release of the Accounting Repository''' | ||
| 02/17 | | 02/17 | ||
| 02/17 | | 02/17 | ||
| | | Closed | ||
| 3.8, 3.9, 3.10, 3.12, 3.14 | | 3.1.1, 3.1.3, 3.1.4, 3.1.6, 3.1.8, 3.1.9, 3.1.10, 3.1.12, 3.1.14 | ||
| | | | ||
|- | |- | ||
| 3.1.19 | | 3.1.19 | ||
| D3.13: Report on Data Accounting | | '''D3.13: Report on Data Accounting''' | ||
| 02/17 | | 02/17 | ||
| 02/17 | | 02/17 | ||
| | | Open | ||
| 3.1.15 | | 3.1.15 | ||
| | | | ||
|- | |- | ||
| 3.1.20 | | 3.1.20 | ||
| D3.15: Second data accounting prototype | | '''D3.15: Second data accounting prototype''' | ||
| 06/17 | | 06/17 | ||
| 06/17 | | 06/17 | ||
| Planned | | Planned | ||
| 3.15 | | 3.1.15 | ||
| | | | ||
|- | |- | ||
| 3.1.21 | | 3.1.21 | ||
| D3.17: Final release of the Accounting Repository | | '''D3.17: Final release of the Accounting Repository''' | ||
| 08/17 | | 08/17 | ||
| 08/17 | | 08/17 | ||
Line 252: | Line 252: | ||
| 03/15 | | 03/15 | ||
| 08/15 | | 08/15 | ||
| | | Done | ||
| | | | ||
| | | | ||
Line 260: | Line 260: | ||
| 03/15 | | 03/15 | ||
| 08/15 | | 08/15 | ||
| | | Done | ||
| 3.2.1 | | 3.2.1 | ||
| | | | ||
Line 289: | Line 289: | ||
|- | |- | ||
| 3.2.6 | | 3.2.6 | ||
| EGI Federated AAI Integration | |||
| 04/16 | |||
| 04/16 | |||
| Planned | |||
| Depends on outcome of AAI TF and EGI policy | |||
| | |||
|- | |||
| 3.2.7 | |||
| D3.5: First release of the new Accounting Portal deployed in production | | D3.5: First release of the new Accounting Portal deployed in production | ||
| 04/ | | 04/16 | ||
| 04/16 | | 04/16 | ||
| Planned | | Planned | ||
Line 296: | Line 304: | ||
| | | | ||
|- | |- | ||
| 3.2. | | 3.2.8 | ||
| Define a complete API to get accounting data directly from the accounting portal | | Define a complete API to get accounting data directly from the accounting portal | ||
| 05/16 | | 05/16 | ||
Line 304: | Line 312: | ||
| | | | ||
|- | |- | ||
| 3.2. | | 3.2.9 | ||
| Integrate data in maps for a graphical distribution of the information | | Integrate data in maps for a graphical distribution of the information | ||
| 07/16 | | 07/16 | ||
Line 312: | Line 320: | ||
| | | | ||
|- | |- | ||
| 3.2. | | 3.2.10 | ||
| Integrate analytics to extract intelligence operation from the data | | Integrate analytics to extract intelligence operation from the data | ||
| 08/16 | | 08/16 | ||
Line 320: | Line 328: | ||
| | | | ||
|- | |- | ||
| 3.2. | | 3.2.11 | ||
| D3.10: Second release of the new Accounting Portal deployed in production | | D3.10: Second release of the new Accounting Portal deployed in production | ||
| 02/17 | | 02/17 | ||
| 02/17 | | 02/17 | ||
| Planned | | Planned | ||
| 3.2. | | 3.2.7, 3.2.8, 3.2.9, 3.2.10 | ||
| | | | ||
|- | |- | ||
| 3.2. | | 3.2.12 | ||
| Support Data Accounting | | Support Data Accounting | ||
| 01/17 | | 01/17 | ||
| 08/17 | | 08/17 | ||
| Planned | | Planned | ||
| | | 3.1.5, 3.1.8, 3.1.9, 3.1.15, 3.1.18 | ||
| | | | ||
|- | |- | ||
| 3.2. | | 3.2.13 | ||
| Support GPGPU Accounting | | Support GPGPU Accounting | ||
| 01/17 | | 01/17 | ||
| 08/17 | | 08/17 | ||
| Planned | | Planned | ||
| | | 3.1.16, 3.1.18 | ||
| | | | ||
|- | |- | ||
| 3.2. | | 3.2.14 | ||
| Support Big Data tools | | Support Big Data tools | ||
| 01/17 | | 01/17 | ||
| 08/17 | | 08/17 | ||
| Planned | | Planned | ||
| | | 3.1.12, 3.1.18 | ||
| | | | ||
|- | |- | ||
| 3.2. | | 3.2.15 | ||
| D3.19: Final release of the new Accounting Portal deployed in production | | D3.19: Final release of the new Accounting Portal deployed in production | ||
| 08/17 | | 08/17 | ||
| 08/17 | | 08/17 | ||
| Planned | | Planned | ||
| 3.2. | | 3.2.11, 3.2.12, 3.2.13, 3.2.14 | ||
| | | | ||
|} | |} |
Latest revision as of 14:00, 23 March 2017
Task leader: Adrian Coveney/STFC
Task leader deputy: Greg Corbett/STFC
Mailing list:TBD
Involved Partner
- STFC
- CESGA
Releases
Roadmap
Accounting Repository
Task number (3.1.X) |
Task name |
Start date (MM/YY) |
Release Date (MM/YY) |
Status (Open/Closed) |
Dependencies from other tasks (insert only task number) |
Issues |
---|---|---|---|---|---|---|
3.1.1 | Cloud Accounting: Usage Record V0.4 (Benchmarks etc) | 03/15 | 07/16 | Closed | ||
3.1.2 | Storage Accounting: Contact sites and test of the current solution | 03/15 | 10/15 | Closed | Fix rolled out and new format being loaded successfully. | |
3.1.3 | Cloud Accounting: Long running VMs - Adding support for VMs that run for a period that crosses a month boundary so that their usage is apportioned to each month. | 03/15 | 07/16 | Open | Several attempts at a solution proposed. All rejected by developers. New solution being tested at central repository. | |
3.1.4 | ARC Parser | 04/15 | 07/16 | Open | Parser code written and tested against test data. Contacting sites to test. | |
3.1.5 | Data Accounting: proof of concept | 06/15 | 01/16 | Closed | ||
3.1.6 | Storage Accounting: Usage Record V2.0 | 07/16 | 10/16 | Open | 3.1.2 | |
3.1.7 | XML Parser | 07/16 | 11/16 | Planned | Requirements need developing. | |
3.1.8 | D3.3: Accounting Repository Release | 02/16 | 02/16 | Closed | 3.1.2, 3.1.5 | |
3.1.9 | Data Accounting: minimal requirements implemented | 02/16 | 09/16 | Closed | 3.1.5 | |
3.1.10 | Storage Accounting: support new storage systems | 09/16 | 02/17 | Planned | 3.1.6 | |
3.1.11 | D3.6: Analysis on techniques to manage big data on the EGI accounting system | 02/16 | 02/16 | Closed | Date moved much earlier than originally expected. | |
3.1.12 | Investigating Big Data tools to improve accounting repository | 07/16 | 02/17 | Closed | 3.1.11 | |
3.1.13 | SQL parsers | 08/16 | 03/17 | Open | Requirements need developing. | |
3.1.14 | D3.8: First data accounting prototype | 10/16 | 10/16 | Closed | 3.1.9 | |
3.1.15 | Data Accounting: production requirements | 10/16 | 05/17 | Planned | 3.1.14 | |
3.1.16 | GPGPU Accounting | 10/16 | 05/17 | Open | Batch systems still do not support accounting for GPGPU usage. Looking into accounting for VM usage of GPGPUs. | |
3.1.17 | Support AAI evolution | 01/17 | 08/17 | Open | ||
3.1.18 | D3.12: Second release of the Accounting Repository | 02/17 | 02/17 | Closed | 3.1.1, 3.1.3, 3.1.4, 3.1.6, 3.1.8, 3.1.9, 3.1.10, 3.1.12, 3.1.14 | |
3.1.19 | D3.13: Report on Data Accounting | 02/17 | 02/17 | Open | 3.1.15 | |
3.1.20 | D3.15: Second data accounting prototype | 06/17 | 06/17 | Planned | 3.1.15 | |
3.1.21 | D3.17: Final release of the Accounting Repository | 08/17 | 08/17 | Planned | 3.1.13, 3.1.15, 3.1.16, 3.1.17, 3.1.18 |
Accounting Portal
Task number (3.2.X) |
Task name |
Start date (MM/YY) |
Release Date (MM/YY) |
Status (Planned/Done) |
Dependencies from other tasks (insert only task number) |
Issues |
---|---|---|---|---|---|---|
3.2.1 | Requirements collection | 03/15 | 08/15 | Done | ||
3.2.2 | D3.1: Technical design of the new Accounting Portal and implementation plan | 03/15 | 08/15 | Done | 3.2.1 | |
3.2.3 | Modernize the accounting Portal with the adoption of technologies easier to maintain | 06/15 | 01/16 | Planned | 3.2.1 | |
3.2.4 | Simplify access to some basic functionality. Avoid the use of complex forms for common statistics and get accounting information for some common queries | 07/15 | 04/16 | Planned | ||
3.2.5 | Support Cloud Usage Record V0.4 and accounting of long running VMs | 07/15 | 04/16 | Planned | 3.1.1, 3.1.3 | |
3.2.6 | EGI Federated AAI Integration | 04/16 | 04/16 | Planned | Depends on outcome of AAI TF and EGI policy | |
3.2.7 | D3.5: First release of the new Accounting Portal deployed in production | 04/16 | 04/16 | Planned | 3.2.2, 3.2.3, 3.2.4, 3.2.5 | |
3.2.8 | Define a complete API to get accounting data directly from the accounting portal | 05/16 | 12/16 | Planned | ||
3.2.9 | Integrate data in maps for a graphical distribution of the information | 07/16 | 01/16 | Planned | ||
3.2.10 | Integrate analytics to extract intelligence operation from the data | 08/16 | 05/17 | Planned | ||
3.2.11 | D3.10: Second release of the new Accounting Portal deployed in production | 02/17 | 02/17 | Planned | 3.2.7, 3.2.8, 3.2.9, 3.2.10 | |
3.2.12 | Support Data Accounting | 01/17 | 08/17 | Planned | 3.1.5, 3.1.8, 3.1.9, 3.1.15, 3.1.18 | |
3.2.13 | Support GPGPU Accounting | 01/17 | 08/17 | Planned | 3.1.16, 3.1.18 | |
3.2.14 | Support Big Data tools | 01/17 | 08/17 | Planned | 3.1.12, 3.1.18 | |
3.2.15 | D3.19: Final release of the new Accounting Portal deployed in production | 08/17 | 08/17 | Planned | 3.2.11, 3.2.12, 3.2.13, 3.2.14 |