Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Intranet"

From EGIWiki
Jump to navigation Jump to search
Line 1: Line 1:
{{Template:Op menubar}} {{Template:Tools menubar}} {{TOC_right}}  
{{Template:Op menubar}} {{Template:Tools menubar}} {{TOC_right}}[[EGI Collaboration tools|<< go to EGI Collaboration tools main page]]


[[Category:Tools]]
This page contains technical details on the services of [http://www.egi.eu/about/intranet EGI intranet] provided by [http://www.cesnet.cz CESNET].
 
Core of the services was prepared before the actual start of the EGI-InSPIRE project and they were available from day 1.


This page describes the status of the services as of mid June 2010.


[[EGI_Collaboration_tools| << go to EGI Collaboration tools main page]]
== Technical background  ==


This page contains technical details on the services of [http://www.egi.eu/about/intranet EGI intranet] provided by [http://www.cesnet.cz CESNET].
=== Hardware  ===


Core of the services was prepared before the actual start of the EGI-InSPIRE
There are two identical servers:
project and they were available from day 1.


This page describes the status of the services as of mid June 2010.
*[http://www.supermicro.com/products/system/1U/6016/SYS-6016T-NTRF.cfm SuperMicro SuperServer 6016T-NTRF]
*2x Intel Xeon X5560 (QuadCore Nehalem 2.8 GHz)
*48 GB
*2x Gbit ethernet
*redundant power supply


== Technical background ==
Both the machines are connected to the same disk array:


=== Hardware ===
*[http://www.flexystor.cz/54-r162ss.html FlexySTOR 162SS]
*16x 450 GB SAS, 15 krpm disks
*RAID controller, 2 GB cache
*the disks are arranged into 2 RAID-10 partitions, yielding 2x 1.8 TB effective capacity


There are two identical servers:
In normal operation each of the machine works in one of the disk array partition, The actual services are implemented in virtual machines, and they are distributed between the physical machines, in order to optimize load.  
* [http://www.supermicro.com/products/system/1U/6016/SYS-6016T-NTRF.cfm SuperMicro SuperServer 6016T-NTRF]
* 2x Intel Xeon X5560 (QuadCore Nehalem 2.8 GHz)
* 48 GB
* 2x Gbit ethernet
* redundant power supply


Both the machines are connected to the same disk array:
In case of failure of any of the physical machines the other one takes over hosting the affected virtual machines. Due to the dual connection of the disk array this can be done without the need of any cable switching. Eventually, an automatic fail-over mechanism can be deployed.  
* [http://www.flexystor.cz/54-r162ss.html FlexySTOR 162SS]
* 16x 450 GB SAS, 15 krpm disks
* RAID controller, 2 GB cache
* the disks are arranged into 2 RAID-10 partitions, yielding 2x 1.8 TB effective capacity


In normal operation
Failure of a single disk in the array is handled transparently by the RAID controller. The disks are hot-swappable, allowing seamless replacement of the failed disk.  
each of the machine works in one of the disk array partition,
The actual services are implemented in virtual machines, and
they are distributed between the physical machines,
in order to optimize load.


In case of failure of any of the physical
The whole system is covered with Next-Business-Day On-Site warranty agreement.  
machines the other one takes over hosting
the affected virtual machines. Due to the dual connection of the disk array
this can be done without the need of any cable switching.
Eventually, an automatic fail-over mechanism can be deployed.


Failure of a single disk in the array is handled transparently by the RAID
The machines are situated in the computer room of [http://www.ics.muni.cz Institute of Computer Science] of Masaryk University, Brno, CZ.  
controller. The disks are hot-swappable, allowing seamless replacement
of the failed disk.


The whole system is covered with Next-Business-Day On-Site warranty agreement.
=== Virtual hosts  ===


The machines are situated in the computer room of [http://www.ics.muni.cz Institute of Computer Science] of Masaryk University, Brno, CZ.
The physical hardware hosts several virtual hosts which provide the actual services in turn. We trade off flexibility and the cost of management, yielding three virtual hosts currently:  


=== Virtual hosts ===
*mail server and request tracker: optimized for high email traffic
*web server, including wiki and the document server: optimized for the web traffic
*backend server providing database backends to the services


The physical hardware hosts several virtual hosts which provide the actual
Currently the former two are hosted by one of the physical host, the database backend by the other in order to balance the overall load.  
services in turn.
We trade off flexibility and the cost of management, yielding three virtual
hosts currently:


* mail server and request tracker: optimized for high email traffic
=== Network connectivity  ===
* web server, including wiki and the document server: optimized for the web traffic
* backend server providing database backends to the services


Currently the former two are hosted by one of the physical host,
The computer room where the machines are located is in the same building as the Point of Presence of the CESNET network backbone. The LAN segment of the servers is directly attached to the backbone router port.  
the database backend by the other in order to balance the overall load.


=== Network connectivity ===
=== Backup  ===


The computer room where the machines are located
Besides the redundancy provided by the hot-swappable RAID-10 disk array all the systems are backed up with the [http://meta.cesnet.cz/cms/opencms/en/resources/backups.html CESNET tape systems].  
is in the same building as the Point of Presence
of the CESNET network backbone.
The LAN segment of the servers is directly attached to the backbone router
port.


=== Backup ===
In general, full file systems are backed up (with the exception of large database files where the usual approach of snapshot + transaction logs is used), therefore disaster recovery is limited by the time to restore full backup, no manual configuration recovery should be required.


Besides the redundancy provided by the hot-swappable RAID-10 disk array
=== Monitoring  ===
all the systems are backed up with the [http://meta.cesnet.cz/cms/opencms/en/resources/backups.html CESNET tape systems].


In general, full file systems are backed up (with the exception of
The services are covered by the monitoring system of NGI CZ based on nagios. The following probes are deployed:
large database files where the usual approach of snapshot + transaction logs
is used), therefore disaster recovery is limited by the time to restore
full backup, no manual configuration recovery should be required.


=== Monitoring ===
*CPU load and utilization
*memory usage, including kernel memory
*critical system messages
*network interface status
*file system usage
*HTTP/HTTPS request sanity on selected URLs
*pakiti -- up to date status of installed software (missing security fixes in particular)


The services are covered by the monitoring system of NGI CZ based on nagios.
<br>
The following probes are deployed:


* CPU load and utilization
=== Operating system and software environment  ===
* memory usage, including kernel memory
* critical system messages
* network interface status
* file system usage
* HTTP/HTTPS request sanity on selected URLs
* pakiti -- up to date status of installed software (missing security fixes in particular)


The hardware servers run Debian 5.0, Xen Dom0. Otherwise there are virtually no services installed.


=== Operating system and software environment ===
The virtual servers are run as Xen DomU, running Debian 5.0 as the guest OS as well. Debian was chosen because of stability; among free Linux distributions it has the longest lifetime of stable major releases. We do not expect the need for bleeding edge functionalities in these services therefore stability is prefered.


The hardware servers run Debian 5.0, Xen Dom0. Otherwise there are virtually no services installed.
As a rule of thumb, the EGI services do not depend on any external services outside of this system. Exceptions are DNS and email, relying on the services provided by Masaryk University and CESNET.  


The virtual servers are run as Xen DomU, running Debian 5.0 as the guest OS as well.
=== Server certificates  ===
Debian was chosen because of stability; among free Linux distributions
it has the longest lifetime of stable major releases.
We do not expect the need for bleeding edge functionalities in these services
therefore stability is prefered.


As a rule of thumb, the EGI services do not depend on any external services outside of this system.
Certificates issued by TERENA SSL CA (generally recognised by web browsers) are used for all the services. Administratively, they are issued to Stichting FOM/Nikhef -- the owner of the egi.eu DNS domain.  
Exceptions are DNS and email, relying on the services provided by Masaryk University and CESNET.


=== Server certificates ===
=== Software customization  ===


Certificates issued by TERENA SSL CA (generally recognised by web browsers)
When setting up the services we could have not avoided modifications of the used sofware (adding or customizing functionality, integration with the common AuthN/Z etc.). We keep records of trivial modifications, non-trivial modifications are kept in CVS repository, allowing fairly easy merging on upgrade to new versions of the software.  
are used for all the services.
Administratively, they are issued to Stichting FOM/Nikhef -- the owner
of the egi.eu DNS domain.


=== Software customization ===
=== Backend server  ===


When setting up the services we could have not avoided modifications of the
Hostname: aldor.ics.muni.cz
used sofware (adding or customizing functionality, integration with the common
AuthN/Z etc.).  
We keep records of trivial modifications,
non-trivial modifications are kept in CVS repository, allowing fairly easy
merging on upgrade to new versions of the software.


=== Backend server ===
Service machine (invisible from outside) hosting database backends of the other services. It is a separate Xen host, so that we are able to move it to other hardware for performance tuning.


Hostname: aldor.ics.muni.cz
== Common authentication and authorization  ==


Service machine (invisible from outside) hosting database backends
Due to the nature of the services, the primary authentication method will be username/password. Over the time we will investigate possibilities to integrate Shibboleth and X509 certificate based AuthN, however, the username/password will remain as the fallback method.  
of the other services.  
It is a separate Xen host, so that we are able to move it to other hardware
for performance tuning.


== Common authentication and authorization ==
The goal is having a single username/password for all the services. A technical solution is LDAP backend; most services are prepared for LDAP-based authentication out of box, and adaptation of others is relatively easy. Currently we use direct LDAP-based authentication in all the services apart of Mailman, where the user passwords are synchronized with the LDAP every hour.


Due to the nature of the services, the primary authentication method
All users of the services and all people working on the EGI projects are required to register an account at the [https://www.egi.eu/sso EGI SSO] system. The users can edit properties of their account, and request password reset eventually.  
will be username/password.  
Over the time we will investigate possibilities to integrate Shibboleth
and X509 certificate based AuthN, however, the username/password
will remain as the fallback method.


The goal is having a single username/password for all the services.
Besides user accounts the LDAP server stores user groups (as <code>groupOfNames</code>) objects. The attribute <code>businessCategory</code> is used to distinguish purpose of the group (multiple values can be specified yielding multi-purpose group):
A technical solution is LDAP backend; most services are prepared
for LDAP-based authentication out of box, and adaptation of others
is relatively easy.
Currently we use direct LDAP-based authentication in all the services
apart of Mailman, where the user passwords are synchronized
with the LDAP every hour.


All users of the services and all people working on the EGI-InSPIRE
*<code>mailman</code> -- members of the group are subscribed to the mailing list of the same name
project are required to register an account at the [https://www.egi.eu/sso EGI SSO] system.
*<code>RT</code> -- group of the same name and members is created in the Request Tracker and can be used for authorization there
The users can edit properties of their account, and request password reset eventually.
*<code>DocDB</code> -- dtto in the Document Database
*<code>wiki</code> -- dtto at wiki


Besides user accounts the LDAP server stores user groups (as <code>groupOfNames</code>) objects.
Group membership is managed at [https://www.egi.eu/sso EGI SSO] as well. Besides adding and removing users from groups, the group owner can invite extrnal people to create their account and to be subscribed to the group.
The attribute <code>businessCategory</code>
is used to distinguish purpose of the group (multiple values can be specified
yielding multi-purpose group):


* <code>mailman</code> -- members of the group are subscribed to the mailing list of the same name
Besides the intranet services, the EGI SSO is used to authenticate users of the project PPT (timesheet submission system), and integration with the software repository and the application database is planned.<br>
* <code>RT</code> -- group of the same name and members is created in the Request Tracker and can be used for authorization there
* <code>DocDB</code> -- dtto in the Document Database
* <code>wiki</code> -- dtto at wiki


Group membership is managed at  [https://www.egi.eu/sso EGI SSO] as well.
=== Removing SSO group procedure <br> ===
Besides adding and removing users from groups, the group owner can invite
extrnal people to create their account and to be subscribed to the group.


Besides the intranet services, the EGE SSO is used to authenticate
<br>
users of the project PPT (timesheet submission system), and integration
with the software repository and the application database is planned.


== Mailing lists ==
<br>


https://mailman.egi.eu/mailman/listinfo
== Mailing lists  ==


[http://www.gnu.org/software/mailman/index.html GNU Mailman] software
https://mailman.egi.eu/mailman/listinfo
is used, in the version (2.1 currently) provided by Debian OS, with modifications integrating it with the EGI SSO.


List subscribers and list administrators can use their EGI SSO passwords for authentication to the Mailman web interface.
[http://www.gnu.org/software/mailman/index.html GNU Mailman] software is used, in the version (2.1 currently) provided by Debian OS, with modifications integrating it with the EGI&nbsp;SSO.  


With a few exceptions the mailing lists membership is controlled by EGI SSO. The exceptions are:
List subscribers and list administrators can use their EGI&nbsp;SSO passwords for authentication to the Mailman web interface.
* the ngi-security-contacts and site-security-contacts lists that have members synchronized with the [http://www.ngs.ac.uk/egee/gocdb GODCB]
* the announce list that anybody can subscribe and all users from the EGI SSO are added to it
* the eef-members list which is hosted for the European E-infrastructure Forum


Mailing lists are exposed by the canonical names
With a few exceptions the mailing lists membership is controlled by EGI SSO. The exceptions are:
<code>list-name@mailman.egi.eu</code>.


==== HTTP server ====
*the ngi-security-contacts and site-security-contacts lists that have members synchronized with the [http://www.ngs.ac.uk/egee/gocdb GODCB]
Apache2, out of the Debian distribution.
*the announce list that anybody can subscribe and all users from the EGI&nbsp;SSO&nbsp;are added to it
Its purpose is administrative Mailman interface and access to the mail list
*the eef-members list which is hosted for the European E-infrastructure Forum
archives only.
Because most of traffic is expected to be authenticated, port 80 (HTTP default)
is redirected to 443 (HTTPS).


==== Incoming email ====
Mailing lists are exposed by the canonical names <code>list-name@mailman.egi.eu</code>.


The only MX DNS record for mailman.egi.eu points to the Masaryk University
==== HTTP server  ====
mail relay (located in the same building, serving in the same way for
several other domains).
The relay forwards all mail to mailman.egi.eu via special rule in its config.
In this way we gain additional reliability and advanced features of the
relay (spam and virus protection).


Apache2, out of the Debian distribution. Its purpose is administrative Mailman interface and access to the mail list archives only. Because most of traffic is expected to be authenticated, port 80 (HTTP default) is redirected to 443 (HTTPS).


==== Outgoing email ====
==== Incoming email ====


Using "smart host" relay.muni.cz for all outgoing email.
The only MX DNS record for mailman.egi.eu points to the Masaryk University mail relay (located in the same building, serving in the same way for several other domains). The relay forwards all mail to mailman.egi.eu via special rule in its config. In this way we gain additional reliability and advanced features of the relay (spam and virus protection).  
This is agreed with the relay administrator, and the symmetric setup may have benefits in case of paranoid recipients.


<br>


==== Spam and virus protection ====
==== Outgoing email  ====


relay.muni.cz (our MX) implements
Using "smart host" relay.muni.cz for all outgoing email. This is agreed with the relay administrator, and the symmetric setup may have benefits in case of paranoid recipients.  
[http://en.wikipedia.org/wiki/Greylisting Grey listing]
technique to ban naive spam attacks.


In addition, we plan to add spam detection set up locally on mailman.egi.eu
<br>
with [http://spamassassin.apache.org/ Spamassassin], using combination
of reliable black lists, static rules for well-known spam patterns
(Viagra, Nigerian spam, ...), and dynamic Bayes filters tuned with real
trafic gradually.


Exact strategy what to do with spam positives has still to be defined, and it
==== Spam and virus protection  ====
may vary among different lists.
In general, as long as it's possible with the amount of the traffic,
we are in favour of moderating to let false positives pass
rather than discarding automatically.


Viruses are detected at relay.muni.cz with
relay.muni.cz (our MX) implements [http://en.wikipedia.org/wiki/Greylisting Grey listing] technique to ban naive spam attacks.  
[http://www.kaspersky.com/ Kaspersky Antivirus], and positives are bounced
back to the sender.


== Web server ==
In addition, we plan to add spam detection set up locally on mailman.egi.eu with [http://spamassassin.apache.org/ Spamassassin], using combination of reliable black lists, static rules for well-known spam patterns (Viagra, Nigerian spam, ...), and dynamic Bayes filters tuned with real trafic gradually.


http://www.egi.eu
Exact strategy what to do with spam positives has still to be defined, and it may vary among different lists. In general, as long as it's possible with the amount of the traffic, we are in favour of moderating to let false positives pass rather than discarding automatically.  


This is the project web site and a web front-end for all the services.
Viruses are detected at relay.muni.cz with [http://www.kaspersky.com/ Kaspersky Antivirus], and positives are bounced back to the sender.  
We use Apache2 from Debian distribution.


Content of the web site is managed by [http://www.opencms.org/ OpenCMS].
== Web server  ==
Google Analytics gathering statistics on the access is deployed.


== Meeting planner ==
http://www.egi.eu


http://www.egi.eu/indico
This is the project web site and a web front-end for all the services. We use Apache2 from Debian distribution.  


General meeting planner using the [http://cern.ch/indico CERN Indico]
Content of the web site is managed by [http://www.opencms.org/ OpenCMS]. Google Analytics gathering statistics on the access is deployed.  
software. It allows scheduling meetings in the full range of size
from informal meetings of few participants to large conferences.
Meeting agenda can be scheduled, and various material attached
to the sections and talks.


The latest stable release of Indico 0.96.2 is used. It was modified to use the EGI SSO LDAP as the source of external users.
== Meeting planner  ==


== Document server ==
http://www.egi.eu/indico


http://documents.egi.eu (public access)
General meeting planner using the [http://cern.ch/indico CERN Indico] software. It allows scheduling meetings in the full range of size from informal meetings of few participants to large conferences. Meeting agenda can be scheduled, and various material attached to the sections and talks.  


https://documents.egi.eu/secure (authenticated access)
The latest stable release of Indico 0.96.2 is used. It was modified to use the EGI&nbsp;SSO&nbsp;LDAP as the source of external users.  


Storing large document files directly on the web site or wiki is
== Document server ==
not optimal. Instead we provide a dedicated document server  
for this purpose. Besides optimizing the storage and access,
the document server offers the following capabilities:


* metadata associated with each document
http://documents.egi.eu (public access)  
* versioning of the documents
* provision of the documents in multiple formats (Word, PDF, ...)
* fine grained access control based on hierarchy of groups


We use [http://docdb-v.sourceforge.net/ DocDB] software with a few local
https://documents.egi.eu/secure (authenticated access)
customizations.


Storing large document files directly on the web site or wiki is not optimal. Instead we provide a dedicated document server for this purpose. Besides optimizing the storage and access, the document server offers the following capabilities:


*metadata associated with each document
*versioning of the documents
*provision of the documents in multiple formats (Word, PDF, ...)
*fine grained access control based on hierarchy of groups


== Wiki ==
We use [http://docdb-v.sourceforge.net/ DocDB] software with a few local customizations.


=== General purpose wiki ===
<br>
https://wiki.egi.eu/wiki/


General purpose wiki for the use in the project, based on
== Wiki  ==
[http://www.mediawiki.org/ MediaWiki] software.


Write access is limited to users registered with EGI SSO.
=== General purpose wiki  ===
Writing to specific areas (namespaces) is further restricted to SSO groups, as is described at [[EGIWiki:Community_Portal]].


=== Special wikis ===
https://wiki.egi.eu/wiki/


Due to the open nature of MediaWiki, it is not possible to reliably restrict '''read access'''
General purpose wiki for the use in the project, based on [http://www.mediawiki.org/ MediaWiki] software.  
of selected pages and keep other pages open for reading. Thus there is a dedicated wiki at


https://wiki.egi.eu/csirt/
Write access is limited to users registered with EGI SSO. Writing to specific areas (namespaces) is further restricted to SSO groups, as is described at [[EGIWiki:Community Portal]].  


which is '''read and write restricted''' for the members of the EGI-CSIRT-Team group from EGI SSO.
=== Special wikis  ===
Because the whole URL space with <nowiki>https://wiki.egi.eu/csirt/</nowiki> prefix is closed,
it is guaranteed that nobody outside of the group can get any access.


== Request tracker ==
Due to the open nature of MediaWiki, it is not possible to reliably restrict '''read access''' of selected pages and keep other pages open for reading. Thus there is a dedicated wiki at


https://rt.egi.eu/
https://wiki.egi.eu/csirt/  


[[RT_Help |RT help]]
which is '''read and write restricted''' for the members of the EGI-CSIRT-Team group from EGI&nbsp;SSO. Because the whole URL space with <nowiki>https://wiki.egi.eu/csirt/</nowiki> prefix is closed, it is guaranteed that nobody outside of the group can get any access.


Work on the project involves tracking wide range of issues,
== Request tracker  ==
starting from resolution of problems with the intranet services,
through managing UMD software releases, upto tracking progress
of project tasks and formal deliverables.


We use the [http://www.bestpractical.com/rt RT] system], version 3.8.
https://rt.egi.eu/  
All issues are tracked in terms of ''tickets'' arranged in various ''queues''.
A ticket follows a defined sequence of major states, custom minor states
can be added per queue.
Similarly, to the default set of fields in a ticket (owner, priority, etc.)
a queue can define additional custom fields with defined data types,
value constraints etc.
Finally, custom actions can be performed on virtually any change of a ticket.
This customizability is starting to be used quite extensively in EGI,
allowing to adapt the system to specific needs of the various groups.


The system provides both web and email interfaces, as well as web-service
[[RT Help|RT help]]
programmatic interface (planned to be used for integration with the
EGI software repository).


Whereas most of the queues setup in this RT instance are dedicated for
Work on the project involves tracking wide range of issues, starting from resolution of problems with the intranet services, through managing UMD software releases, upto tracking progress of project tasks and formal deliverables.  
internal use in the project and EGI.eu, a special [https://wiki.egi.eu/wiki/Requirements_Tracking requirements queue] was put
in place to be the main gateway to store, monitor and resolve requirements and
recommendations communicated by user communities and NGIs to EGI. On the other hand, bug report
helpdesk is served by the GGUS system.


== Jabber ==
We use the [http://www.bestpractical.com/rt RT] system], version 3.8. All issues are tracked in terms of ''tickets'' arranged in various ''queues''. A ticket follows a defined sequence of major states, custom minor states can be added per queue. Similarly, to the default set of fields in a ticket (owner, priority, etc.) a queue can define additional custom fields with defined data types, value constraints etc. Finally, custom actions can be performed on virtually any change of a ticket. This customizability is starting to be used quite extensively in EGI, allowing to adapt the system to specific needs of the various groups.


jabber.egi.eu
The system provides both web and email interfaces, as well as web-service programmatic interface (planned to be used for integration with the EGI software repository).  


We provide a jabber (XMPP instant messaging) server attached to
Whereas most of the queues setup in this RT instance are dedicated for internal use in the project and EGI.eu, a special [https://wiki.egi.eu/wiki/Requirements_Tracking requirements queue] was put in place to be the main gateway to store, monitor and resolve requirements and recommendations communicated by user communities and NGIs to EGI. On the other hand, bug report helpdesk is served by the GGUS system.  
the EGI SSO (each user gets an account automatically) in order not to
rely on publicly available services (jabber.org etc.).


The server runs jabberd14 software available in Debian,
== Jabber  ==
including the ''multi-user conference'' extension which provides
chat rooms.
Any SSO user can create a chat room, however, the rooms are
available to users at jabber.egi.eu only.


Integration with the SSO is done by the <code>xdb_auth_cpile</code> jabberd component.
jabber.egi.eu


== Hosted servers ==
We provide a jabber (XMPP instant messaging) server attached to the EGI SSO (each user gets an account automatically) in order not to rely on publicly available services (jabber.org etc.).


We host two additional web sites:
The server runs jabberd14 software available in Debian, including the ''multi-user conference'' extension which provides chat rooms. Any SSO user can create a chat room, however, the rooms are available to users at jabber.egi.eu only.


* [http://www.eu-emi.eu www.eu-emi-eu] Website of the EU EMI project.
Integration with the SSO is done by the <code>xdb_auth_cpile</code> jabberd component.  
* [http://www.einfrastructure-forum.eu www.einfrastructure-forum.eu] Informal organization, forum for the discussion of principles and practices to create synergies for distributed Infrastructures.


Both are provided as  
== Hosted servers  ==
virtual hosts (in terms of Apache, not Xen) on www.egi.eu and managed in the OpenCMS content management system. The EMI site
 
is inside the OpenCMS enclosed in a separate so-called ''organizational unit'', which allows to show the web site admins only
We host two additional web sites:
the files and folders that are relevant to them. The OpenCMS extensive permission and user role system is used to restrict
 
the users to their own sites only.
*[http://www.eu-emi.eu www.eu-emi-eu] Website of the EU EMI project.
*[http://www.einfrastructure-forum.eu www.einfrastructure-forum.eu] Informal organization, forum for the discussion of principles and practices to create synergies for distributed Infrastructures.
 
Both are provided as virtual hosts (in terms of Apache, not Xen) on www.egi.eu and managed in the OpenCMS content management system. The EMI site is inside the OpenCMS&nbsp;enclosed in a separate so-called ''organizational unit'', which allows to show the web site admins only the files and folders that are relevant to them. The OpenCMS extensive permission and user role system is used to restrict the users to their own sites only.  


== EGI.eu domain websites  ==
== EGI.eu domain websites  ==
Line 537: Line 419:
<br>  
<br>  


<br>
<br>  
 
[[Category:Tools]]

Revision as of 15:16, 21 April 2015

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Tools menu: Main page Instructions for developers AAI Proxy Accounting Portal Accounting Repository AppDB ARGO GGUS GOCDB
Message brokers Licenses OTAGs Operations Portal Perun EGI Collaboration tools LToS EGI Workload Manager


<< go to EGI Collaboration tools main page

This page contains technical details on the services of EGI intranet provided by CESNET.

Core of the services was prepared before the actual start of the EGI-InSPIRE project and they were available from day 1.

This page describes the status of the services as of mid June 2010.

Technical background

Hardware

There are two identical servers:

Both the machines are connected to the same disk array:

  • FlexySTOR 162SS
  • 16x 450 GB SAS, 15 krpm disks
  • RAID controller, 2 GB cache
  • the disks are arranged into 2 RAID-10 partitions, yielding 2x 1.8 TB effective capacity

In normal operation each of the machine works in one of the disk array partition, The actual services are implemented in virtual machines, and they are distributed between the physical machines, in order to optimize load.

In case of failure of any of the physical machines the other one takes over hosting the affected virtual machines. Due to the dual connection of the disk array this can be done without the need of any cable switching. Eventually, an automatic fail-over mechanism can be deployed.

Failure of a single disk in the array is handled transparently by the RAID controller. The disks are hot-swappable, allowing seamless replacement of the failed disk.

The whole system is covered with Next-Business-Day On-Site warranty agreement.

The machines are situated in the computer room of Institute of Computer Science of Masaryk University, Brno, CZ.

Virtual hosts

The physical hardware hosts several virtual hosts which provide the actual services in turn. We trade off flexibility and the cost of management, yielding three virtual hosts currently:

  • mail server and request tracker: optimized for high email traffic
  • web server, including wiki and the document server: optimized for the web traffic
  • backend server providing database backends to the services

Currently the former two are hosted by one of the physical host, the database backend by the other in order to balance the overall load.

Network connectivity

The computer room where the machines are located is in the same building as the Point of Presence of the CESNET network backbone. The LAN segment of the servers is directly attached to the backbone router port.

Backup

Besides the redundancy provided by the hot-swappable RAID-10 disk array all the systems are backed up with the CESNET tape systems.

In general, full file systems are backed up (with the exception of large database files where the usual approach of snapshot + transaction logs is used), therefore disaster recovery is limited by the time to restore full backup, no manual configuration recovery should be required.

Monitoring

The services are covered by the monitoring system of NGI CZ based on nagios. The following probes are deployed:

  • CPU load and utilization
  • memory usage, including kernel memory
  • critical system messages
  • network interface status
  • file system usage
  • HTTP/HTTPS request sanity on selected URLs
  • pakiti -- up to date status of installed software (missing security fixes in particular)


Operating system and software environment

The hardware servers run Debian 5.0, Xen Dom0. Otherwise there are virtually no services installed.

The virtual servers are run as Xen DomU, running Debian 5.0 as the guest OS as well. Debian was chosen because of stability; among free Linux distributions it has the longest lifetime of stable major releases. We do not expect the need for bleeding edge functionalities in these services therefore stability is prefered.

As a rule of thumb, the EGI services do not depend on any external services outside of this system. Exceptions are DNS and email, relying on the services provided by Masaryk University and CESNET.

Server certificates

Certificates issued by TERENA SSL CA (generally recognised by web browsers) are used for all the services. Administratively, they are issued to Stichting FOM/Nikhef -- the owner of the egi.eu DNS domain.

Software customization

When setting up the services we could have not avoided modifications of the used sofware (adding or customizing functionality, integration with the common AuthN/Z etc.). We keep records of trivial modifications, non-trivial modifications are kept in CVS repository, allowing fairly easy merging on upgrade to new versions of the software.

Backend server

Hostname: aldor.ics.muni.cz

Service machine (invisible from outside) hosting database backends of the other services. It is a separate Xen host, so that we are able to move it to other hardware for performance tuning.

Common authentication and authorization

Due to the nature of the services, the primary authentication method will be username/password. Over the time we will investigate possibilities to integrate Shibboleth and X509 certificate based AuthN, however, the username/password will remain as the fallback method.

The goal is having a single username/password for all the services. A technical solution is LDAP backend; most services are prepared for LDAP-based authentication out of box, and adaptation of others is relatively easy. Currently we use direct LDAP-based authentication in all the services apart of Mailman, where the user passwords are synchronized with the LDAP every hour.

All users of the services and all people working on the EGI projects are required to register an account at the EGI SSO system. The users can edit properties of their account, and request password reset eventually.

Besides user accounts the LDAP server stores user groups (as groupOfNames) objects. The attribute businessCategory is used to distinguish purpose of the group (multiple values can be specified yielding multi-purpose group):

  • mailman -- members of the group are subscribed to the mailing list of the same name
  • RT -- group of the same name and members is created in the Request Tracker and can be used for authorization there
  • DocDB -- dtto in the Document Database
  • wiki -- dtto at wiki

Group membership is managed at EGI SSO as well. Besides adding and removing users from groups, the group owner can invite extrnal people to create their account and to be subscribed to the group.

Besides the intranet services, the EGI SSO is used to authenticate users of the project PPT (timesheet submission system), and integration with the software repository and the application database is planned.

Removing SSO group procedure



Mailing lists

https://mailman.egi.eu/mailman/listinfo

GNU Mailman software is used, in the version (2.1 currently) provided by Debian OS, with modifications integrating it with the EGI SSO.

List subscribers and list administrators can use their EGI SSO passwords for authentication to the Mailman web interface.

With a few exceptions the mailing lists membership is controlled by EGI SSO. The exceptions are:

  • the ngi-security-contacts and site-security-contacts lists that have members synchronized with the GODCB
  • the announce list that anybody can subscribe and all users from the EGI SSO are added to it
  • the eef-members list which is hosted for the European E-infrastructure Forum

Mailing lists are exposed by the canonical names list-name@mailman.egi.eu.

HTTP server

Apache2, out of the Debian distribution. Its purpose is administrative Mailman interface and access to the mail list archives only. Because most of traffic is expected to be authenticated, port 80 (HTTP default) is redirected to 443 (HTTPS).

Incoming email

The only MX DNS record for mailman.egi.eu points to the Masaryk University mail relay (located in the same building, serving in the same way for several other domains). The relay forwards all mail to mailman.egi.eu via special rule in its config. In this way we gain additional reliability and advanced features of the relay (spam and virus protection).


Outgoing email

Using "smart host" relay.muni.cz for all outgoing email. This is agreed with the relay administrator, and the symmetric setup may have benefits in case of paranoid recipients.


Spam and virus protection

relay.muni.cz (our MX) implements Grey listing technique to ban naive spam attacks.

In addition, we plan to add spam detection set up locally on mailman.egi.eu with Spamassassin, using combination of reliable black lists, static rules for well-known spam patterns (Viagra, Nigerian spam, ...), and dynamic Bayes filters tuned with real trafic gradually.

Exact strategy what to do with spam positives has still to be defined, and it may vary among different lists. In general, as long as it's possible with the amount of the traffic, we are in favour of moderating to let false positives pass rather than discarding automatically.

Viruses are detected at relay.muni.cz with Kaspersky Antivirus, and positives are bounced back to the sender.

Web server

http://www.egi.eu

This is the project web site and a web front-end for all the services. We use Apache2 from Debian distribution.

Content of the web site is managed by OpenCMS. Google Analytics gathering statistics on the access is deployed.

Meeting planner

http://www.egi.eu/indico

General meeting planner using the CERN Indico software. It allows scheduling meetings in the full range of size from informal meetings of few participants to large conferences. Meeting agenda can be scheduled, and various material attached to the sections and talks.

The latest stable release of Indico 0.96.2 is used. It was modified to use the EGI SSO LDAP as the source of external users.

Document server

http://documents.egi.eu (public access)

https://documents.egi.eu/secure (authenticated access)

Storing large document files directly on the web site or wiki is not optimal. Instead we provide a dedicated document server for this purpose. Besides optimizing the storage and access, the document server offers the following capabilities:

  • metadata associated with each document
  • versioning of the documents
  • provision of the documents in multiple formats (Word, PDF, ...)
  • fine grained access control based on hierarchy of groups

We use DocDB software with a few local customizations.


Wiki

General purpose wiki

https://wiki.egi.eu/wiki/

General purpose wiki for the use in the project, based on MediaWiki software.

Write access is limited to users registered with EGI SSO. Writing to specific areas (namespaces) is further restricted to SSO groups, as is described at EGIWiki:Community Portal.

Special wikis

Due to the open nature of MediaWiki, it is not possible to reliably restrict read access of selected pages and keep other pages open for reading. Thus there is a dedicated wiki at

https://wiki.egi.eu/csirt/

which is read and write restricted for the members of the EGI-CSIRT-Team group from EGI SSO. Because the whole URL space with https://wiki.egi.eu/csirt/ prefix is closed, it is guaranteed that nobody outside of the group can get any access.

Request tracker

https://rt.egi.eu/

RT help

Work on the project involves tracking wide range of issues, starting from resolution of problems with the intranet services, through managing UMD software releases, upto tracking progress of project tasks and formal deliverables.

We use the RT system], version 3.8. All issues are tracked in terms of tickets arranged in various queues. A ticket follows a defined sequence of major states, custom minor states can be added per queue. Similarly, to the default set of fields in a ticket (owner, priority, etc.) a queue can define additional custom fields with defined data types, value constraints etc. Finally, custom actions can be performed on virtually any change of a ticket. This customizability is starting to be used quite extensively in EGI, allowing to adapt the system to specific needs of the various groups.

The system provides both web and email interfaces, as well as web-service programmatic interface (planned to be used for integration with the EGI software repository).

Whereas most of the queues setup in this RT instance are dedicated for internal use in the project and EGI.eu, a special requirements queue was put in place to be the main gateway to store, monitor and resolve requirements and recommendations communicated by user communities and NGIs to EGI. On the other hand, bug report helpdesk is served by the GGUS system.

Jabber

jabber.egi.eu

We provide a jabber (XMPP instant messaging) server attached to the EGI SSO (each user gets an account automatically) in order not to rely on publicly available services (jabber.org etc.).

The server runs jabberd14 software available in Debian, including the multi-user conference extension which provides chat rooms. Any SSO user can create a chat room, however, the rooms are available to users at jabber.egi.eu only.

Integration with the SSO is done by the xdb_auth_cpile jabberd component.

Hosted servers

We host two additional web sites:

Both are provided as virtual hosts (in terms of Apache, not Xen) on www.egi.eu and managed in the OpenCMS content management system. The EMI site is inside the OpenCMS enclosed in a separate so-called organizational unit, which allows to show the web site admins only the files and folders that are relevant to them. The OpenCMS extensive permission and user role system is used to restrict the users to their own sites only.

EGI.eu domain websites

Domain Contact person Contact Email Purpose
operations-portal.egi.eu Cyril Lorphelin
cic-information@in2p3.fr Portal
gstat.egi.eu
project-grid-info-support@cern.ch Portal
mon.egi.eu Emir Imamagic
argo-ggus-support@grnet.gr Portal
midmon.egi.eu Emir Imamagic argo-ggus-support@grnet.gr Portal
opsmon.egi.eu Emir Imamagic argo-ggus-support@grnet.gr Portal
secmon.egi.eu
Emir Imamagic argo-ggus-support@grnet.gr Portal
accounting.egi.eu Carlos Fernandez
grid-admin@cesga.es Portal
metrics.egi.eu Carlos Fernandez grid-admin@cesga.es Portal
helpdesk.egi.eu Guenter Grein
support@ggus.org Portal
goc.egi.eu
David Meredith gocdb-admins@mailman.egi.eu Portal
appdb.egi.eu Marios Chatziangelou

appdb-support@iasa.gr

Portal
site-certification.egi.eu Kostas Koumantaros

egi-catch-all-services@lists.grnet.gr

Portal
www.egi.eu

it-support@egi.eu

Portal
wiki.egi.eu

it-support@egi.eu

Portal
documents.egi.eu

it-support@egi.eu

Portal
forum.egi.eu

it-support@egi.eu

Portal
mailman.egi.eu

it-support@egi.eu

Portal and mail
rt.egi.eu

it-support@egi.eu

Portal
indico.egi.eu 

it-support@egi.eu

Portal
survey.egi.eu

it-support@egi.eu

Portal
pakiti.egi.eu Daniel Kouril daniel.kouril@cesnet.cz Portal
dirac.egi.eu
dirac-support@mailman.egi.eu Portal
e-grant.egi.eu Tomasz Szepieniec
egrant-support@mailman.egi.eu Portal
portal.egi.eu Diego Scardaci
Diego.Scardaci@egi.eu Portal
unity.egi.eu Tomasz Szepieniec t.szepieniec@cyfronet.pl

perun.egi.eu Michal Prochazka
michalp@ics.muni.cz Portal
repository.egi.eu Kostas Koumantaros

egi-repo@hellasgrid.gr

Portal
admin-repo.egi.eu
Kostas Koumantaros

egi-repo@hellasgrid.gr

Portal