Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "EGI-Engage:TASK JRA1.4 Operations Tools"

From EGIWiki
Jump to navigation Jump to search
 
(45 intermediate revisions by 4 users not shown)
Line 58: Line 58:
| 05/2015<br>  
| 05/2015<br>  
| 02/2016<br>  
| 02/2016<br>  
| Planned<br>
| Released
| <br>  
| <br>  
|  
|  
Line 74: Line 74:
| 05/2015<br>  
| 05/2015<br>  
| 02/2016<br>  
| 02/2016<br>  
| Planned<br>
| Released
| 4.1.1<br>  
| 4.1.1<br>  
| <br>
| <br>
Line 86: Line 86:
| 06/2015  
| 06/2015  
| 02/2016  
| 02/2016  
| Planned
| Released
|  
| <br>
|  
| <br>
|-
|-
! 4.1.4<br>  
! 4.1.4<br>  
Line 98: Line 98:
| 06/2015<br>  
| 06/2015<br>  
| 02/2016  
| 02/2016  
| Planned<br>
| Released
| 4.1.3<br>  
| 4.1.3<br>  
| <br>
| <br>
Line 110: Line 110:
| 06/2015  
| 06/2015  
| 02/2016  
| 02/2016  
| Planned<br>
| Released
| 4.1.3<br>  
| 4.1.3<br>  
| <br>
| <br>
Line 121: Line 121:


| 10/2015<br>  
| 10/2015<br>  
| 02/2016
| 02/2016  
| Planned<br>
| Released
| 4.1.3<br>  
| 4.1.3<br>  
| <br>
| <br>
Line 130: Line 130:
ReplaceGstat  
ReplaceGstat  


Replace Gstat Main Features<br>  
Add Gstat Main Features into Glue2Browser<br>  


| 11/2015<br>  
| 11/2015<br>  
| 02/2017<br>  
| 02/2017<br>  
| Planned<br>  
| On-going<br>  
| <br>  
| <br>  
| <br>
| <br>
Line 145: Line 145:


| 04/2016  
| 04/2016  
| 02/2017
| 02/2017  
| Planned  
| Planned  
|  
| <br>
|  
| <br>
|-
|-
! 4.1.9  
! 4.1.9  
Line 158: Line 158:
| 04/2016  
| 04/2016  
| 02/2017  
| 02/2017  
| Planned
| Released
|  
| <br>
|  
| <br>
|-
|-
! 4.1.10  
! 4.1.10  
Line 166: Line 166:
ExtendAPI3  
ExtendAPI3  


Extend Gstat API  
Extend Glue2Browser API  


| 04/2016  
| 04/2016  
| 02/2017  
| 02/2017  
| Planned
| Released
| 4.1.7  
| 4.1.7  
|  
| <br>
|-
|-
! 4.1.11<br>  
! 4.1.11<br>  
Line 181: Line 181:


| 06/2016<br>  
| 06/2016<br>  
| 02/2017
| 02/2017  
| Planned<br>  
| Planned<br>  
| <br>  
| <br>  
Line 207: Line 207:
| 08/2017  
| 08/2017  
| Planned  
| Planned  
| <br>
| <br>
|-
! 4.1.14
| OpsPortalEgiMetrics<br>Extends the current metrics and add new ones
| 05/2016
| 09/2016
| Released
| <br>
| <br>
|-
! 4.1.15
|  
|  
OpsPortalVoAcknowledgement
Add Vo Acknowledgement section in VO ID card
| 10/2016
| 11/2016
| Released
| <br>
| <br>
|-
! 4.1.16
|  
|  
OpsPortalVoRobotCertificate
Add certificate robot section
| 11/2016
| 12/2016
| Released
| <br>
| <br>
|}
|}


== GOCDB ==
== GOCDB ==
* Note, targets and dates are indicative and subject to change and augmentation as requirements/priorities evolve.  
 
*Note, targets and dates are indicative and subject to change and augmentation as requirements/priorities evolve.
 
{| class="wikitable"
{| class="wikitable"
|-
|-
Line 218: Line 252:
Task number  
Task number  


(4.2.X)
(4.2.X)  


! Task name  
! Task name  
Line 224: Line 258:
Start date  
Start date  


(MM/YY)
(MM/YY)  


!  
!  
Release Date  
Release Date  


(MM/YY)
(MM/YY)  


!  
!  
Line 243: Line 277:
! Issues
! Issues
|-
|-
| 4.2.0
| 4.2.0  
Support changing infrastructure requirements:
| Extend data model  
* Extend Data Model where necessary
| 03/15
| 03/15
| 08/17
| 08/17
| Done/ongoing
| Planned
|
|-
| 4.2.1
| v5.4
*'''RoleActionLogging'''
*'''Fine grained content rendering '''(PermitAll and Protected pages)
*'''Downtime declaration in local timezone'''
 
| 04/15
| 07/15
| Done
|
|
|-
| 4.2.2
| v5.5
*'''Role Abstractions for Multi-tenant''' (host multiple projects in same goc instance and apply different roles per-project - users can have roles in different projects https://rt.egi.eu/rt/Ticket/Display.html?id=9429)
*'''SAML/FIM authentication''', Elixir (https://rt.egi.eu/rt/Ticket/Display.html?id=7493)
 
| 06/15
| 12/15
| Done
|
|
*Issues related to data protection and the Elixir/GEANT CoCo delay the production release of FIM. This is an EGI wide issue - all ops tools need to use the same eppn (most likely hashed given data protection concerns) in order to correlate accounts across services.
 
|-
| 4.2.3
| D3.4: First release of the Operational tools - GOCDB
| 02/16
| 02/16
| Done
|
|-
| 4.2.4
| v5.6
*'''Reserved Scope Tags''' - Restrict user tagging of resources using reserved/protected scope tags. Will allow WLCG/Elixir sites to apply tags only to their resources and prevent other sites from using same tags.
*'''Downtime Calendar '''with fine grained filtering by (reserved) scopes and other params.
*'''Bulk addition/upload of multiple custom properties''' including support for .properties file uploads (https://rt.egi.eu/rt/Ticket/Display.html?id=9427)
* '''Integration with EGI ProxyIdp''' - Allow access for users without client certificate.
 
| 11/15
| 07/03/16
| Done
|
|
|-
| 4.2.4.1
| v5.7
*'''Optional Paging on PI''' - Selected PI queries should page by default. If page URL parameters are not provided, then query will return the first page of results by default. As a minimum, apply to get_downtime*, get_service* methods and review other methods for paging. Use cursor-based paging and not offset-based paging.   
 
| 05/16
| <strike>07/16</strike> 12/16
| Done
|
| Announced for early/EGI testing 09/08/2016. Was delayed in order to implement new cursor-based paging logic which is required to solve data inconsistency issues associated with offset-based paging.
Released as part of v5.7 07/12/2016
|-
| 4.2.5
| <strike>v5.8</strike> features merged into v5.7
*'''Writable REST API '''to post updates to sites/services (requested by WLCG and EUDAT). Requires Site's to manage their own DN based ACLs per site required to post updates for a site/service.
*'''Add new monitoring attributes for ARGO''' Enables EGI ARGO service to removed dependency on BDII
 
| 03/16
| <strike>09/16</strike> 12/16
| Done
|
| <strike>Planned alpha release W/C September 19th.</strike> Further details on [[GOCDB/Write_API_Development|dedicated wiki page]]. Planned alpha release w/c 24th Oct.
Released as part of v5.7 release 07/12/16
|-
| 4.2.6
|
*'''Schedule OTAG to re-prioritise GOCDB developments for next release''' Make request to EGI for GOCDB OTAG.
| 01/17
| ~02/17  
| Completed (27/02/2017)
|
|
|  
| Details: https://indico.egi.eu/indico/event/3296/
|-
| 4.2.7
| '''D3.11: Second release of the Operational tools - GOCDB'''
| 02/17
| 02/17
| Completed
| 4.2.4, 4.2.4.1, 4.2.5, 4.2.6
|
|-
|-
| 4.2.1
| 4.2.8
| RoleActionLogging (v5.4)
| '''Write API Extensions for Services/ServiceEndpoints'''
Also:  
* https://rt.egi.eu/rt/Ticket/Display.html?id=11020
* Finer grained content rendering (PermitAll and Protected pages)
| 03/17
* Downtime declaration in local timezone
| 07/17
|  04/15
| Planned
| 05/15 
| Done (06/07/15)
|  
|   
|   
|
|-
|-
| 4.2.2
| 4.2.9
| RoleAbstractions (v5.5)
| '''Verify Data Freshness Check'''
* Multi-tennant (apply different rules/roles on a per-project basis)
* https://rt.egi.eu/rt/Ticket/Display.html?id=8240
| 06/15
| 03/17
| 08/15 
| 07/17
| Planned
| Planned  
|  
|
| Running ~1month behind, v.5.5 release scheduled for ~end Sept for testing
|  
|-
|-
| 4.2.3
| 4.2.10
| Object Diff Auditing (v5.6)
| '''GUI Improvements'''
| 09/15
* https://rt.egi.eu/rt/Ticket/Display.html?id=11546
| 12/15 
* https://rt.egi.eu/rt/Ticket/Display.html?id=11217
| Planned
| 03/17
|  
| 07/17
| Delayed, EUDAT funded developer to start end Sept to start this work.  
| Planned  
|  
| No API changes - GUI only meaning the terminology in the API and GUI would diverge.  
|-
|-
| 4.2.4
| 4.2.11
| Support Marketplace developments
| '''List of smaller issues reported in Github'''
| 05/15
* https://github.com/GOCDB/gocdb/issues
| 19/15 
| 03/17
| Planned
| 07/17
|  
| Planned  
|
|
|
|-
|-
| 4.2.5
| 4.2.12
| D3.4: First release of the Operational tools - GOCDB
| '''~Downtime classification changes'''
| 02/16
* Tentative - depends on complexity of requirements
| 02/16 
* https://rt.egi.eu/rt/Ticket/Display.html?id=10845
Planned
| 03/17
| 4.2.1, 4.2.2
| 07/17
|
| Tentative
|   
| If DT > 1 month in duration, declare 1 month in advance to be SCHEDULED. If DT < than 5 days, can declare 24hrs in advance to be SCHEDULED. If there are no requirements for complex DT rules based on different Site/SE groupings allowing the same rules to apply to all DTs, then this could be implemented. Pending - To agree with WLCG/EGI
|-
|-
| 4.2.6
| 4.2.13
| EGI_Fed_AAI_Integration, LoA integration.  
| '''~Unique Constraint on hostName/serviceType pair'''
| 01/16
* Tentative - may not be a requirement any longer
| 06/16 
* https://rt.egi.eu/rt/Ticket/Display.html?id=10368
| Planned
| 03/17
Depends on outcome of AAI TF and EGI policy.
| 07/17
| Many. We need to consider Level of Assurance for different auth schemes. Requires policy agreement in EGI.  
| Tentative
|   
| Not an EGI requirement, but happy to re-prioritise if still needed by WLCG.  
|-
|-
| 4.2.7
| 4.2.14
| MVC + GUI refactoring:
| '''~NGI Certification Status Rules'''
* Replace proprietary MVC with Symfony2
* Tentative - Nice to have, depends on remaining effort
| 04/16
* https://rt.egi.eu/rt/Ticket/Display.html?id=9084
| 08/16 
| 03/17
| Planned
| 07/17
|  
| Tentative
|
|
| Useful to have a quick way of changing CertStatus of all Sites belonging to an NGI. Determine if following account portal requirement is related/needed: https://rt.egi.eu/rt/Ticket/Display.html?id=11481
|-
|-
| 4.2.8
| 4.2.15
| D3.11: Second release of the Operational tools - GOCDB
| '''D3.18: Final release of the Operational tools - GOCDB'''
| 02/17
| 08/17  
| 02/17
| 08/17  
| Planned
| Planned  
| 4.2.3, 4.2.4, 4.2.6
| 4.2.8 to 4.2.14
|
|  
|-
|-
4.2.9
|   
| D3.18: Final release of the Operational tools - GOCDB
| '''E-Infra12 proposals'''
08/17
* Replace UI with modern Web framework
| 08/17 
* More Change logging
| Planned
* New requirements will probably emerge relating to dynamic attributes and extending GOCDB in info-service space
4.2.7
* DB migration scripts for higher resilience 
|
|   
|  
|  
|   
|  
|}
|}


Line 353: Line 480:
Status  
Status  


(Open/Closed)  
(Planned/Done)  


!  
!  
Line 588: Line 715:
Status  
Status  


(Open/Closed)  
(Planned/Done)  


!  
!  
Line 654: Line 781:
Status  
Status  


(Open/Closed)  
(Planned/Done)  


!  
!  

Latest revision as of 14:33, 24 March 2017

EGI-Engage project: Main page WP1(NA1) WP3(JRA1) WP5(SA1) PMB Deliverables and Milestones Quality Plan Risk Plan Data Plan
Roles and
responsibilities
WP2(NA2) WP4(JRA2) WP6(SA2) AMB Software and services Metrics Project Office Procedures


Task leader: Cyril Lorphelin/CNRS
Task leader deputy: TBD

Mailing list:TBD


Involved Partner

  • CNRS
  • CESNET
  • GRNET
  • STFC
  • SRCE

Releases

Roadmap

Operations Portal

Task number

(4.1.X)

Task name

Start date

(MM/YY)

Release Date

(MM/YY)

Status

(Planned/Done)

Dependencies from other tasks

(insert only task number)

Issues
4.1.1

LavUpgrade

Upgrade Vapor configuration for lavoisier

05/2015
02/2016
Released



4.1.2

LavGlue

Replace Glue1.3 queries by Glue 2

05/2015
02/2016
Released 4.1.1

4.1.3

LavCloud

Catpure Cloud Resources in Lavoisier

06/2015 02/2016 Released

4.1.4

RBExtension

Extend the resource browser to cloud resources

06/2015
02/2016 Released 4.1.3

4.1.5

VaporExtension

Extend Vapor views to cloud resources

06/2015 02/2016 Released 4.1.3

4.1.6

DashboardExtension

Extend dashboard views to cloud resources

10/2015
02/2016 Released 4.1.3

4.1.7

ReplaceGstat

Add Gstat Main Features into Glue2Browser

11/2015
02/2017
On-going


4.1.8

ExtendAPI1

Extend dashboard API

04/2016 02/2017 Planned

4.1.9

ExtendAPI2

Extend Vapor API

04/2016 02/2017 Released

4.1.10

ExtendAPI3

Extend Glue2Browser API

04/2016 02/2017 Released 4.1.7
4.1.11

MonitorVM1

monitor running/creation requests VM

06/2016
02/2017 Planned


4.1.12

MonitorVM2

monitor success/error/time-out rates for cloud sites

06/2016 02/2017 Planned
4.1.11

4.1.13

ExtendVoDataManagement

Support cloud storage solutions / File catalog

01/2017 08/2017 Planned

4.1.14 OpsPortalEgiMetrics
Extends the current metrics and add new ones
05/2016 09/2016 Released

4.1.15

OpsPortalVoAcknowledgement

Add Vo Acknowledgement section in VO ID card

10/2016 11/2016 Released

4.1.16

OpsPortalVoRobotCertificate

Add certificate robot section

11/2016 12/2016 Released

GOCDB

  • Note, targets and dates are indicative and subject to change and augmentation as requirements/priorities evolve.

Task number

(4.2.X)

Task name

Start date

(MM/YY)

Release Date

(MM/YY)

Status

(Planned/Done)

Dependencies from other tasks

(insert only task number)

Issues
4.2.0 Extend data model 03/15 08/17 Done/ongoing
4.2.1 v5.4
  • RoleActionLogging
  • Fine grained content rendering (PermitAll and Protected pages)
  • Downtime declaration in local timezone
04/15 07/15 Done
4.2.2 v5.5 06/15 12/15 Done
  • Issues related to data protection and the Elixir/GEANT CoCo delay the production release of FIM. This is an EGI wide issue - all ops tools need to use the same eppn (most likely hashed given data protection concerns) in order to correlate accounts across services.
4.2.3 D3.4: First release of the Operational tools - GOCDB 02/16 02/16 Done
4.2.4 v5.6
  • Reserved Scope Tags - Restrict user tagging of resources using reserved/protected scope tags. Will allow WLCG/Elixir sites to apply tags only to their resources and prevent other sites from using same tags.
  • Downtime Calendar with fine grained filtering by (reserved) scopes and other params.
  • Bulk addition/upload of multiple custom properties including support for .properties file uploads (https://rt.egi.eu/rt/Ticket/Display.html?id=9427)
  • Integration with EGI ProxyIdp - Allow access for users without client certificate.
11/15 07/03/16 Done
4.2.4.1 v5.7
  • Optional Paging on PI - Selected PI queries should page by default. If page URL parameters are not provided, then query will return the first page of results by default. As a minimum, apply to get_downtime*, get_service* methods and review other methods for paging. Use cursor-based paging and not offset-based paging.
05/16 07/16 12/16 Done Announced for early/EGI testing 09/08/2016. Was delayed in order to implement new cursor-based paging logic which is required to solve data inconsistency issues associated with offset-based paging.

Released as part of v5.7 07/12/2016

4.2.5 v5.8 features merged into v5.7
  • Writable REST API to post updates to sites/services (requested by WLCG and EUDAT). Requires Site's to manage their own DN based ACLs per site required to post updates for a site/service.
  • Add new monitoring attributes for ARGO Enables EGI ARGO service to removed dependency on BDII
03/16 09/16 12/16 Done Planned alpha release W/C September 19th. Further details on dedicated wiki page. Planned alpha release w/c 24th Oct.

Released as part of v5.7 release 07/12/16

4.2.6
  • Schedule OTAG to re-prioritise GOCDB developments for next release Make request to EGI for GOCDB OTAG.
01/17 ~02/17 Completed (27/02/2017) Details: https://indico.egi.eu/indico/event/3296/
4.2.7 D3.11: Second release of the Operational tools - GOCDB 02/17 02/17 Completed 4.2.4, 4.2.4.1, 4.2.5, 4.2.6
4.2.8 Write API Extensions for Services/ServiceEndpoints 03/17 07/17 Planned
4.2.9 Verify Data Freshness Check 03/17 07/17 Planned
4.2.10 GUI Improvements 03/17 07/17 Planned No API changes - GUI only meaning the terminology in the API and GUI would diverge.
4.2.11 List of smaller issues reported in Github 03/17 07/17 Planned
4.2.12 ~Downtime classification changes 03/17 07/17 Tentative If DT > 1 month in duration, declare 1 month in advance to be SCHEDULED. If DT < than 5 days, can declare 24hrs in advance to be SCHEDULED. If there are no requirements for complex DT rules based on different Site/SE groupings allowing the same rules to apply to all DTs, then this could be implemented. Pending - To agree with WLCG/EGI
4.2.13 ~Unique Constraint on hostName/serviceType pair 03/17 07/17 Tentative Not an EGI requirement, but happy to re-prioritise if still needed by WLCG.
4.2.14 ~NGI Certification Status Rules 03/17 07/17 Tentative Useful to have a quick way of changing CertStatus of all Sites belonging to an NGI. Determine if following account portal requirement is related/needed: https://rt.egi.eu/rt/Ticket/Display.html?id=11481
4.2.15 D3.18: Final release of the Operational tools - GOCDB 08/17 08/17 Planned 4.2.8 to 4.2.14
E-Infra12 proposals
  • Replace UI with modern Web framework
  • More Change logging
  • New requirements will probably emerge relating to dynamic attributes and extending GOCDB in info-service space
  • DB migration scripts for higher resilience

Monitoring

Task number

(4.3.X)

Task name

Start date

(MM/YY)

Release Date

(MM/YY)

Status

(Planned/Done)

Dependencies from other tasks

(insert only task number)

Issues
4.3.1 ARGO Compute Engine & Web API:
  • automatic recomputation triggers
  • multi-tenant support
  • stability and performance improvements
04/15 06/15 Planned
4.3.2 ARGO Monitoring Engine:
  • probe framework
  • support documentation (Guides)
  • stability and performance improvements
04/15 06/15 Planned
4.3.3 ARGO EGI Web UI
  • ACL mechanism (support groups/roles)
  • UI Enhancements
04/15 06/15 Planned
4.3.4 ARGO EGI Connectors & Consumer:
  • improved support for VOs
  • stability and performance improvements
04/15 06/15 Planned
4.3.5 ARGO POEM:
  • ACL mechanism (support groups/roles)
  • stability and performance improvements
04/15 06/15 Planned
4.3.6 ARGO Compute Engine & Web API:
  • API for data ingestion specification
  • separation of A/R and Metric stores
  • APIv2 Specification
  • stability and performance improvements
07/15 09/15 Planned 4.3.1
4.3.7 ARGO Monitoring Engine:
  • fedcloud probes
  • stability and performance improvements
07/15 09/15 Planned 4.3.2
4.3.8 ARGO EGI Web UI:
  • UI Enhancements
  • initial support for federated logins using SAML
07/15 09/15 Planned 4.3.3
4.3.9 ARGO EGI Connectors & Consumer:
  • use of CE ingestion api
  • stability and performance improvements
07/15 09/15 Planned 4.3.4
4.3.10 ARGO POEM
  • initial support for federated logins using SAML
  • support for probe management
  • stability and performance improvements
07/15 09/15 Planned 4.3.5
4.3.11 ARGO Compute Engine & Web API:
  • API for data ingestion specification
  • APIv2
  • stability and performance improvements
09/15 12/15 Planned 4.3.6
4.3.12 ARGO Monitoring Engine:
  • fedcloud probes update
  • stability and performance improvements
09/15 12/15 Planned 4.3.7
4.3.13 ARGO EGI Web UI:
  • UI Enhancements
  • support for federated logins with IdP Discovery
09/15 12/15 Planned 4.3.8
4.3.14 ARGO EGI Connectors & Consumer:
  • stability and performance improvements
09/15 12/15 Planned 4.3.9
4.3.15 ARGO POEM:
  • support for federated logins with IdP Discovery
  • stability and performance improvements
09/15 12/15 Planned 4.3.10
4.3.16 ARGO Compute Engine & Web API:
  • stability and performance improvements
01/16 03/16 Planned 4.3.11
4.3.17 ARGO Monitoring Engine:
  • fedcloud probes update
  • stability and performance improvements
01/16 03/16 Planned 4.3.12
4.3.18 ARGO EGI Web UI:
  • join as a SP to eduGAIN
01/16 03/16 Planned 4.3.13
4.3.19 ARGO EGI Connectors & Consumer
  • stability and performance improvements
01/16 03/16 Planned 4.3.14
4.3.20 ARGO POEM
  • join as a SP to eduGAIN
  • stability and performance improvements
01/16 03/16 Planned 4.3.15

Messaging

Task number

(4.4.X)

Task name

Start date

(MM/YY)

Release Date

(MM/YY)

Status

(Planned/Done)

Dependencies from other tasks

(insert only task number)

Issues
4.4.1 Preparatory phase 04/05 06/05 Planned
4.4.2 APIv1 alpha specification 07/05 09/05 Planned 4.4.1
4.4.3 APIv1 test implementation

APIv1 final draft specification (ready for external party review)

09/05 12/05 Planned 4.4.2
4.4.4 APIv1 final specification 01/06 03/06 Planned 4.4.2

Security Monitoring

Task number

(4.5.X)

Task name

Start date

(MM/YY)

Release Date

(MM/YY)

Status

(Planned/Done)

Dependencies from other tasks

(insert only task number)

Issues