Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Intranet"

From EGIWiki
Jump to navigation Jump to search
(Created page with 'This page contains technical details on the services of [http://www.egi.eu/about/intranet EGI intranet] provided by [http://www.cesnet.cz CESNET]. == Technical background == ==…')
 
 
(65 intermediate revisions by 8 users not shown)
Line 1: Line 1:
This page contains technical details on the services of [http://www.egi.eu/about/intranet EGI intranet] provided by [http://www.cesnet.cz CESNET].
{{Template:Op menubar}} {{Template:Tools menubar}} {{TOC_right}}[[EGI Collaboration tools|<< go to EGI Collaboration tools main page]]  


== Technical background ==
This page contains technical details on the services of [http://www.egi.eu/about/intranet EGI intranet] provided by [http://www.cesnet.cz CESNET].


=== Hardware ===
Core of the services was prepared before the actual start of the EGI-InSPIRE project and they were available from day 1.


There are two identical servers:
This page was updated to describe the status of the services as of June 2015.  
* [http://www.supermicro.com/products/system/1U/6016/SYS-6016T-NTRF.cfm SuperMicro SuperServer 6016T-NTRF]
* 2x Intel Xeon X5560 (QuadCore Nehalem 2.8 GHz)
* 48 GB
* 2x Gbit ethernet
* redundant power supply


Both the machines are connected to the same disk array:
== Technical background  ==
* [http://www.flexystor.cz/54-r162ss.html FlexySTOR 162SS]
* 16x 450 GB SAS, 15 krpm disks
* RAID controller, 2 GB cache
* the disks are arranged into 2 RAID-10 partitions, yielding 2x 1.8 TB effective capacity


The system is covered with Next-Business-Day On-Site warranty agreement.
=== Hardware  ===


In normal operation
There are two identical servers:
each of the machine works in one of the disk array partition,
The actual services are implemented in virtual machines, and
they are distributed between the physical machines,
in order to optimize load.


In case of failure of any of the physical
*[http://www.supermicro.com/products/system/1U/6016/SYS-6016T-NTRF.cfm SuperMicro SuperServer 6016T-NTRF]
machines the other one takes over hosting
*2x Intel Xeon X5560 (QuadCore Nehalem 2.8 GHz)
the affected virtual machines. Due to the dual connection of the disk array
*48 GB
this can be done without the need of any cable switching.
*2x Gbit ethernet
Eventually, an automatic failover mechanism can be deployed.
*redundant power supply


The machines are situated in the computer room of [http://www.ics.muni.cz Institute of Computer Science] of Masaryk University, Brno, CZ.
Both the machines are connected to the same disk array:  


'''Status:''' done
*[http://www.flexystor.cz/54-r162ss.html FlexySTOR 162SS]
*16x 450 GB SAS, 15 krpm disks
*RAID controller, 2 GB cache
*the disks are arranged into 2 RAID-10 partitions, yielding 2x 1.8 TB effective capacity


=== Network connectivity ===
In normal operation each of the machine works in one of the disk array partition, The actual services are implemented in virtual machines, and they are distributed between the physical machines, in order to optimize load.


The computer room where the machines are located
In case of failure of any of the physical machines the other one takes over hosting the affected virtual machines. Due to the dual connection of the disk array this can be done without the need of any cable switching. Eventually, an automatic fail-over mechanism can be deployed.  
is in the same building as the Point of Presence
of the CESNET network backbone.
The LAN segment of the servers is directly attached to the backbone router
port currently, planned final connection is via Masaryk University 10 GE
backbone (still short connection within adjacent computer rooms
in the same building).


'''Status:''' done
Failure of a single disk in the array is handled transparently by the RAID controller. The disks are hot-swappable, allowing seamless replacement of the failed disk.


=== Backup ===
The whole system is covered with Next-Business-Day On-Site warranty agreement.


Besides the redundancy provided of hot-swappable RAID-10 disk arrays
The machines are situated in the computer room of [http://www.ics.muni.cz Institute of Computer Science] of Masaryk University, Brno, CZ.  
all the systems are backed up with the [http://meta.cesnet.cz/cms/opencms/en/resources/backups.html CESNET tape systems].


'''Status:''' to be configured by each virtual machine once its main
=== Virtual hosts  ===
configuration is finished.


=== Operating system and software environment ===
The physical hardware hosts several virtual hosts which provide the actual services in turn. We trade off flexibility and the cost of management, yielding the following virtual hosts currently:


The servers above run Debian 5.0, Xen Dom0. Otherwise there are virtually no services installed.
* mail server and request tracker: optimized for high email traffic
* web server, including wiki, the document server, blog, SAML identity provider: optimized for the web traffic
* server with Indico, phpBB forum, jabber
* server with LimeSurvey and Radius
* backend server providing database backends to the services


The virtual servers described bellow are run as Xen DomU, running Debian 5.0 as the guest OS as well.
=== Network connectivity  ===
Debian was chosen because of stability; among free Linux distributions
it has the longest lifetime of stable major releases.
We do not expect the need for bleeding edge functionalities in these services
therefore stability is prefered.


As a rule of thumb, the EGI services do not depend on any external services outside of this
The computer room where the machines are located is in the same building as the Point of Presence of the CESNET network backbone. The LAN segment of the servers is directly attached to the backbone router port.  
system, apart of DNS and email.


=== Server certificates ===
=== Backup  ===


The policy of TERENA SSL CA (used by CESNET servers) does not allow
Besides the redundancy provided by the hot-swappable RAID-10 disk array all the systems are backed up with the [http://meta.cesnet.cz/cms/opencms/en/resources/backups.html CESNET tape systems].  
issuing a certificate for servers in the egi.eu domain to CESNET,
we have to receive it via EGI.EU organization and/or NIKHEF.


'''Status:''' done
In general, full file systems are backed up (with the exception of large database files where the usual approach of snapshot + transaction logs is used), therefore disaster recovery is limited by the time to restore full backup, no manual configuration recovery should be required.


=== Software customization ===
=== Monitoring  ===


The services are covered by the monitoring system of NGI CZ based on nagios. The following probes are deployed:


== Common authentication and authorization ==
*CPU load and utilization
*memory usage, including kernel memory
*critical system messages
*network interface status
*file system usage
*HTTP/HTTPS request sanity on selected URLs
*pakiti -- up to date status of installed software (missing security fixes in particular)


Due to the nature of the services, the primary authentication method
<br>
will be username/password.
Over the time we will investigate possibilities to integrate Shibboleth
and X509 certificate based AuthN, however, the username/password
will remain as the fallback method.


The goal is having a single username/password for all the services.
=== Operating system and software environment  ===
Candidate technical solution is LDAP.


'''Status:'''
The hardware servers run Debian Linux, Xen Dom0. Otherwise there are virtually no services installed.  
* LDAP server installed and running
* user accounts created from mailing lists' members, username were manually invented
* groups for lists created, marked with attribute "businessCategory: mailman", users assigned to groups
* Mailman list members and their passwords are synchronized (daily) with LDAP
* Mediawiki has LDAP extension installed, all users from LDAP can log in
* DocDB authentication is based on LDAP
* web application for editing passwords and registration is at [https://egi.eu/sso/ https://egi.eu/sso/]
* group owners can manage group members thru the same [https://egi.eu/sso/ https://egi.eu/sso/]
* group owners can ask new people to become users and specify groups for them


'''TODO'''
The virtual servers are run as Xen DomU, running Debian as the guest OS as well. Debian was chosen because of stability; among free Linux distributions it has the longest lifetime of stable major releases. We do not expect the need for bleeding edge functionalities in these services therefore stability is prefered.
* Mediawiki needs exntension for page protrection based on LDAP groups
* OpenCMS needs LDAP plugin
* Indico LDAP support is not known, but there are some remarks about CERN lightweight account in the documentation


== Mailing lists ==
As a rule of thumb, the EGI services do not depend on any external services outside of this system. Exceptions are DNS and email, relying on the services provided by Masaryk University and CESNET.


=== Server certificates  ===


'''Status:''' Done.
Certificates are issued by a commercial certification authority, because the owner of the egi.eu DNS domain is the EGI.eu foundation based in the Netherlands, and certificates issued from the academic TERENA TCS CA through the local NREN (SURFNet) would be far more expensive.


=== Disk partitions ===
=== Software customization  ===
For the purpose of mutual isolation, separate partitions are used:


<table>
When setting up the services we could have not avoided modifications of the used sofware (adding or customizing functionality, integration with the common AuthN/Z etc.). We keep records of trivial modifications, non-trivial modifications are kept in CVS repository, allowing fairly easy merging on upgrade to new versions of the software.
<tr>
<td align="left"> / </td>
<td align="left"> 5 GB </td>
<td align="left"> root filesystem</td>
</tr>


<tr>
=== Backend server  ===
<td align="left"> /var/logs </td>
<td align="left"> 2 GB </td>
<td align="left"> all logs </td>
</tr>


<tr>
Hostname: aldor.ics.muni.cz
<td align="left"> /var/lib/mailman/archives </td>
<td align="left"> 2 GB </td>
<td align="left"> mailman archives </td>
</tr>


The sizes are minimalistic, and the filesystems (using XFS) can be extended
Service machine (invisible from outside) hosting database backends of the other services. It is a separate Xen host, so that we are able to move it to other hardware for performance tuning.  
as needed.


</table>
== Provided services ==


See also the page [[EGI_Collaboration_tools]]


=== HTTP server ===
=== DNS for egi.eu domain ===
Apache2, out of the Debian distribution.
Its purpose is administrative Mailman interface and access to the mailist
archives only.


Because most of traffic is expected to be authenticated, port 80 (HTTP default)
The egi.eu DNS domain is registered at http://www.eurid.eu/ by the EGI.eu foundation, but its DNS servers are provided by CESNET.
is redirected to 443 (HTTPS).


'''Status:''' done
=== EGI SSO ===


=== Mailman software ===
The goal is having a single username/password for all the services. A technical solution is LDAP backend; most services are prepared for LDAP-based authentication out of box, and adaptation of others is relatively easy. Currently we use direct LDAP-based authentication in all the services apart of Mailman, where the user passwords are synchronized with the LDAP every hour.


Out of Debian distribution.
All users of the services and all people working on the EGI projects are required to register an account at the [https://www.egi.eu/sso EGI SSO] system. The users can edit properties of their account, arequest password reset and delete their own account.  


Individual mailing lists will be created and removed by the server admins
Some of the services were modified to accept also an X509 digital certificate - Mailman, wiki, DocDB and RT. A certificate can be registered to a user account in the SSO by logging into https://www.egi.eu/sso/user with a certificate and user/password in the same time.
(the set is expected to be semi-static).


Management of the individual lists can be delageted to any of the system
'''Security considerations:''' Anybody with an email address can get an EGI SSO account ! Only the e-mail address is verified, all other information is supplied by the user and thus can be fake. The security of the EGI services is based on group membership, as group membership can be granted only by group owners.
users. Technically done by access control in Apache configuration
(each Mailman list has a unique URL prefix).


'''Status:''' done.
==== Groups ====
Besides user accounts the LDAP server stores user groups (as <code>groupOfNames</code>) objects. The attribute <code>businessCategory</code> is used to distinguish purpose of the group (multiple values can be specified yielding multi-purpose group):  


=== Incoming email ===
*<code>mailman</code> -- members of the group are subscribed to the mailing list of the same name, owners of the group are administrators of the list, the group description is used as list description
*<code>RT</code> -- group of the same name and members is created in the Request Tracker and can be used for authorization there
*<code>DocDB</code> -- dtto in the Document Database
*<code>wiki</code> -- dtto at wiki
*<code>forum</code> -- dtto at phpBB forum
* the Indico imports all groups automatically


The only MX DNS record for mailman.egi.eu points to the Masaryk University
Group membership is managed at [https://www.egi.eu/sso EGI SSO] as well. Besides adding and removing users from groups, the group owner can invite external people to create their account and to be subscribed to the group.  
mail relay (located in the same building, serving in the same way for
several other domains).
The relay forwards all mail to mailman.egi.eu via special rule in its config.


In this way we gain additional reliability and advanced features of the
Groups can form a hierarchy. A group can be '''owned''' by a group, then all members of the owning group can manage the group. A group can also be a '''subgroup''' of another group (linked by <code>seeAlso</code> atribute), then all members of the subgroup are also members of the supergroup.
relay (spam and virus protection).


'''Status''': done
Group membership on the services is synchronized with the LDAP server on every full hour.


=== Outgoing email ===
==== Removing SSO group procedure <br>  ====


Using "smart host" relay.muni.cz for all outgoing email.
Following steps are taken once request for SSO group removal is received<br>
The relay admin accepts it, and the symmetric setup may have benefits in case of paranoid recipients.


'''Status:''' done
#Creating a snapshot of members list&nbsp; (members name and email,&nbsp; name of the group and short description of the grou)
#Informing members of the group that it will be removed (if mailing list was created)
#Removing members from list
#DocDB, Wiki (namespace) data ownership is transferred to ''EGI catch-all'' SSO group  (only if no other group would be assigned for the document/namespace)
#Closing and archiving mailing list with response message that group has been closed


=== Spam and virus protection ===


relay.muni.cz (our MX) implements
=== IdP for external services ===
[http://en.wikipedia.org/wiki/Greylisting Grey listing]
technique to ban naive spam attacks.


In addition, spam detection will be set up locally on mailman.egi.eu
The SSO user database can be used for external services using [http://en.wikipedia.org/wiki/Security_Assertion_Markup_Language SAML] '''Identity Provider''' (IdP). The external services must deploy a SAML '''Service Provider''' (SP) like [https://shibboleth.net/products/service-provider.html Shibboleth] or [https://simplesamlphp.org/samlsp SimpleSAMLphp], and establish mutual trust with the EGI IdP.
with [http://spamassassin.apache.org/ Spamassassin], using combination
of reliable black lists, static rules for well-known spam patterns
(Viagra, Nigerian spam, ...), and dynamic Bayes filters tuned with real
trafic gradually.


Exact strategy what to do with spam positives has still to be defined, and it
A prospective SP should do the following steps:
may vary among different lists.
# install the SP software
In general, as long as it's possible with the amount of the traffic,  
# establish mutual trust with EGI IdP
I'm in favour of moderating to let false positives pass
#* add EGI IdP's metadata from https://www.egi.eu/idp/shibboleth to the SP configuration
rather than discarding automatically.
#* generate the metadata of their SP (in Shibboleth, available at <nowiki>https://www.example.org/Shibboleth.sso/Metadata</nowiki>) and send them to it-support@egi.eu
# configure EGI IdP in their discovery service or as a single IdP for their SP


Viruses are detected at relay.muni.cz with
The IdP provides the following attributes:
[http://www.kaspersky.com/ Kaspersky Antivirus], and positives are bounced
{| class="wikitable"
back to the sender.
! attribute
! friendlyName
! content
|-
|urn:oid:0.9.2342.19200300.100.1.1 || uid || username from EGI SSO
|-
| urn:oid:0.9.2342.19200300.100.1.3 || mail || email address registered in EGI SSO
|-
| urn:oid:1.3.6.1.4.1.5923.1.1.1.6 || eduPersonPrincipalName || username with appended @egi.eu
|-
| urn:oid:2.5.4.3 || cn || full name in English, ASCII characters only
|-
| urn:oid:2.16.840.1.113730.3.1.241 || displayName || full name in original alphabet, can be Greek, Chinese, etc.
|-
| urn:oid:2.5.4.42 || givenName || first name
|-
| urn:oid:2.5.4.4 || sn || surname, suitable for alphabetical ordering
|-
| urn:oid:1.3.6.1.4.1.5923.1.1.1.7 || eduPersonEntitlement || contains list of groups in the form of URIs like ''urn:egi.eu:group:somegroup''
|-
| urn:oid:1.3.6.1.4.1.11433.2.2.1.9 || userCertificateSubject || distinguished names from X509 certificates in OpenSSL notation
|-
| <nowiki>https://www.egi.eu/attribute-def/egiPartnerOrg</nowiki> || egiPartnerOrg || value from a fixed list of partner organizations, or "other". The fixed list is specified in the drop-down list of organization in [https://www.egi.eu/sso/user?edit= SSO edit form]
|-
| <nowiki>https://www.egi.eu/attribute-def/egiFreeTypedOrg</nowiki> || egiFreeTypedOrg || anything the user typed, can be misspelled
|-
| urn:oid:2.5.4.10 || o || organization - if egiPartnerOrg!=other then same as egiPartnerOrg,  else same as  egiFreeTypedOrg
|}


'''Status:'''
* Spamassassin to be deployed and configured
* Spam handling strategies to be defined


== Web server ==
=== Mailing lists  ===


www.egi.eu
https://mailman.egi.eu/mailman/listinfo


WWW front-end for all the services.
[http://www.gnu.org/software/mailman/index.html GNU Mailman] software is used, in the version (2.1 currently) provided by Debian OS, with modifications integrating it with the EGI&nbsp;SSO.  


Apache2 from Debian distribution.
List subscribers and list administrators can use their EGI&nbsp;SSO passwords for authentication to the Mailman web interface.  


http://www.egi.eu/ https://www.egi.eu/
With a few exceptions the mailing lists membership is controlled by EGI SSO. The exceptions are:  


=== OpenCMS ===
*the ngi-security-contacts and site-security-contacts lists that have members synchronized with the [http://www.ngs.ac.uk/egee/gocdb GODCB]
*the announce list that anybody can subscribe and all users from the EGI&nbsp;SSO&nbsp;are added to it
*the eef-members list which is hosted for the European E-infrastructure Forum


* Installation of [http://www.opencms.org/ OpenCMS] done.
Mailing lists are exposed by the canonical names <code>list-name@mailman.egi.eu</code>.  
* Accounts for content maintainers created.
* page template created from eu-egi.eu pages
* menu and breadcrumbs implemented
* Google Analytics deployed
* news and their RSS feed created
* administration interface is at [https://www.egi.eu/cms/system/login/ https://www.egi.eu/cms/system/login/]
* real SSL certificate installed


'''TODO''':
==== HTTP server  ====


* install LDAP plugin for CMS
Apache2, out of the Debian distribution. Its purpose is administrative Mailman interface and access to the mail list archives only. Because most of traffic is expected to be authenticated, port 80 (HTTP default) is redirected to 443 (HTTPS).


== Document server ==
==== Incoming email  ====


Basic setup of [http://docdb-v.sourceforge.net/ DocDB] done [https://www.egi.eu/DocDB/cgi/DocumentDatabase here]
The only MX DNS record for mailman.egi.eu points to the Masaryk University mail relay (located in the same building, serving in the same way for several other domains). The relay forwards all mail to mailman.egi.eu via special rule in its config. In this way we gain additional reliability and advanced features of the relay (spam and virus protection).  
(requires authentication) but not fully working yet.


TODO:
<br>
* broken help
* accents in names
* manually introduced institutions


'''Status:''' done except LDAP authentication in OpenCMS
==== Outgoing email  ====


== Meeting planner ==
Using "smart host" relay.muni.cz for all outgoing email. This is agreed with the relay administrator, and the symmetric setup may have benefits in case of paranoid recipients.


== Wiki ==
<br>


Virtual host (in terms of Apache, not Xen), on www.egi.eu.
==== Spam and virus protection  ====


http://wiki.egi.eu/
relay.muni.cz (our MX) implements [http://en.wikipedia.org/wiki/Greylisting Grey listing] technique to ban naive spam attacks.


https://wiki.egi.eu/
In addition, we plan to add spam detection set up locally on mailman.egi.eu with [http://spamassassin.apache.org/ Spamassassin], using combination of reliable black lists, static rules for well-known spam patterns (Viagra, Nigerian spam, ...), and dynamic Bayes filters tuned with real trafic gradually.


'''TODO''':
Exact strategy what to do with spam positives has still to be defined, and it may vary among different lists. In general, as long as it's possible with the amount of the traffic, we are in favour of moderating to let false positives pass rather than discarding automatically.
* select and install some extension for restricting access to selected pages


'''Status:'''
Viruses are detected at relay.muni.cz with [http://www.kaspersky.com/ Kaspersky Antivirus], and positives are bounced back to the sender.
* webserver running
* MediaWiki installed
* LDAP plugin installed
* google analytics activated
* real SSL certificate installed


=== Web sites ===


== Request tracker ==
==== Main web site with OpenCMS and Peble blog ====


== Jabber ==
http://www.egi.eu


== Other and auxiliary services ==
This is the project web site and a web front-end for all the services. We use Apache2 from Debian distribution.


=== www.einfrastructure-forum.eu ===
Content of the web site is managed by [http://www.opencms.org/ OpenCMS]. Google Analytics gathering statistics on the access is deployed.  


Virtual host (in terms of Apache, not Xen), on www.egi.eu.
Part of the suite is a blog managed by Pebble blog software.


http://www.einfrastructure-forum.eu/
==== Hosted servers  ====


https://www.einfrastructure-forum.eu/
We host additional web sites:  


'''Status:''' webserver running with a self-signed SSL certificate, CMS configured, page template finished
* [http://www.einfrastructure-forum.eu www.einfrastructure-forum.eu] - hosted on OpenCMS
* [https://www.opensciencecommons.org/ www.opensciencecommons.org] - hosted on Wordpress CMS


'''TODO''':
* get a real SSL certificate


=== aldor.ics.muni.cz ===
=== Indico meeting planner  ===


Service machine (invisible from outside) hosting database backends
http://www.egi.eu/indico
of the services.  


A separate Xen host, so that we are able to move it to other hardware
General meeting planner using the [http://cern.ch/indico CERN Indico] software. It allows scheduling meetings in the full range of size from informal meetings of few participants to large conferences. Meeting agenda can be scheduled, and various material attached to the sections and talks.
for performance tuning.
 
 
=== Document server  ===
 
http://documents.egi.eu (public access)
 
https://documents.egi.eu/secure (authenticated access)
 
Storing large document files directly on the web site or wiki is not optimal. Instead we provide a dedicated document server for this purpose. Besides optimizing the storage and access, the document server offers the following capabilities:
 
*metadata associated with each document
*versioning of the documents
*provision of the documents in multiple formats (Word, PDF, ...)
*fine grained access control based on hierarchy of groups
 
We use [http://docdb-v.sourceforge.net/ DocDB] software with a few local customizations.
 
=== Wiki  ===
 
==== General purpose wiki  ====
 
https://wiki.egi.eu/wiki/
 
General purpose wiki for the use in the project, based on [http://www.mediawiki.org/ MediaWiki] software.
 
Write access is limited to users registered with EGI SSO. Writing to specific areas (namespaces) is further restricted to SSO groups, as is described at [[EGIWiki:Community Portal]].
 
==== Special wikis  ====
 
Due to the open nature of MediaWiki, it is not possible to reliably restrict '''read access''' of selected pages and keep other pages open for reading. Thus there is a dedicated wiki at
 
https://wiki.egi.eu/csirt/
 
which is '''read and write restricted''' for the members of the EGI-CSIRT-Team group from EGI&nbsp;SSO. Because the whole URL space with <nowiki>https://wiki.egi.eu/csirt/</nowiki> prefix is closed, it is guaranteed that nobody outside of the group can get any access.
 
=== Request tracker  ===
 
https://rt.egi.eu/
 
[[RT Help|RT help]]
 
Work on the project involves tracking wide range of issues, starting from resolution of problems with the intranet services, through managing UMD software releases, upto tracking progress of project tasks and formal deliverables.  
 
We use the [http://www.bestpractical.com/rt RT] system], version 3.8. All issues are tracked in terms of ''tickets'' arranged in various ''queues''. A ticket follows a defined sequence of major states, custom minor states can be added per queue. Similarly, to the default set of fields in a ticket (owner, priority, etc.) a queue can define additional custom fields with defined data types, value constraints etc. Finally, custom actions can be performed on virtually any change of a ticket. This customizability is starting to be used quite extensively in EGI, allowing to adapt the system to specific needs of the various groups.
 
The system provides both web and email interfaces, as well as web-service programmatic interface (planned to be used for integration with the EGI software repository).
 
Whereas most of the queues setup in this RT instance are dedicated for internal use in the project and EGI.eu, a special [https://wiki.egi.eu/wiki/Requirements_Tracking requirements queue] was put in place to be the main gateway to store, monitor and resolve requirements and recommendations communicated by user communities and NGIs to EGI. On the other hand, bug report helpdesk is served by the GGUS system.
 
=== Jabber  ===
 
jabber.egi.eu
 
We provide a jabber (XMPP instant messaging) server attached to the EGI SSO (each user gets an account automatically) in order not to rely on publicly available services (jabber.org etc.).
 
The server runs jabberd14 software available in Debian, including the ''multi-user conference'' extension which provides chat rooms. Any SSO user can create a chat room, however, the rooms are available to users at jabber.egi.eu only.
 
Integration with the SSO is done by the <code>xdb_auth_cpile</code> jabberd component.
 
=== Eduroam ===
 
For the EGI.eu staff included in the SSO group '''eduroam''', we provide [[Eduroam]] authentication server.
 
== EGI.eu domain websites  ==
Last update: 2020-02-06
 
{| class="wikitable sortable"
|-
! scope="col" | Domain
! scope="col" | Contact person
! scope="col" | Contact Email
! scope="col" | Purpose
|-
| operations-portal.egi.eu || Cyril Lorphelin || cic-information@in2p3.fr || Portal
|-
| gstat.egi.eu || || project-grid-info-support@cern.ch || Portal
|-
| mon.egi.eu || Emir Imamagic || argo-ggus-support@grnet.gr || Portal
|-
| midmon.egi.eu || Emir Imamagic || argo-ggus-support@grnet.gr || Portal
|-
| opsmon.egi.eu || Emir Imamagic || argo-ggus-support@grnet.gr || Portal
|-
| secmon.egi.eu || Emir Imamagic || argo-ggus-support@grnet.gr || Portal
|-
| accounting.egi.eu || Carlos Fernandez || grid-admin@cesga.es || Portal
|-
| metrics.egi.eu || Carlos Fernandez || grid-admin@cesga.es || Portal
|-
| helpdesk.egi.eu || Guenter Grein || support@ggus.org || Portal
|-
| goc.egi.eu || David Meredith || gocdb-admins@mailman.egi.eu || Portal
|-
| appdb.egi.eu || Marios Chatziangelou || appdb-support@iasa.gr || Portal
|-
| site-certification.egi.eu || Kostas Koumantaros || egi-catch-all-services@lists.grnet.gr || Portal
|-
| www.egi.eu || || it-support@egi.eu || Main website, blog, identity provider
|-
| wiki.egi.eu || || it-support@egi.eu || Wiki
|-
| csirt-wiki.egi.eu || || it-support@egi.eu || Wiki
|-
| wiki.eosc-hub.eu  || || it-support@egi.eu || Wiki
|-
| documents.egi.eu  || || it-support@egi.eu || Document server
|-
| mailman.egi.eu || || it-support@egi.eu || Mailing lists
|-
| mailman.eosc-hub.eu || || it-support@egi.eu || Mailing lists
|-
| rt.egi.eu || || it-support@egi.eu || Request Tracking
|-
| indico.egi.eu || || it-support@egi.eu || Meeting planner
|-
| confluence.egi.eu || || it-support@egi.eu || Collaboration wiki
|-
| jira.egi.eu || || it-support@egi.eu || Collaboration Tool
|-
| pakiti.egi.eu || Daniel Kouril || daniel.kouril@cesnet.cz || software security updates
|-
| dirac.egi.eu || || dirac-support@mailman.egi.eu || Portal
|-
| e-grant.egi.eu || Tomasz Szepieniec|| egrant-support@mailman.egi.eu || Portal
|-
| portal.egi.eu || Diego Scardaci || Diego.Scardaci@egi.eu || Portal
|-
| unity.egi.eu || Tomasz Szepieniec || t.szepieniec@cyfronet.pl|| Authorization service for LTOS based on UNITY
|-
| perun.egi.eu || Michal Prochazka || michalp@ics.muni.cz || Portal
|-
| repository.egi.eu || Kostas Koumantaros || egi-repo@hellasgrid.gr || Portal
|-
| admin-repo.egi.eu || Kostas Koumantaros || egi-repo@hellasgrid.gr || Portal
|-
| appliance-repo.egi.eu|| Kostas Koumantaros || egi-repo@hellasgrid.gr || Portal
|-
| access.egi.eu || Tomasz Szepieniec || t.szepieniec@cyfronet.pl || Portal
|-
| argo.egi.eu || Christos Kanellopoulos || skanct@admin.grnet.gr || Portal
|-
| aai.egi.eu      || Peter Solagna|| peter.solagna@egi.eu|| AAI pilot
|-
| eduroam.egi.eu  || || it-support@egi.eu || Portal
|-
| council.egi.eu  || || it-support@egi.eu || Portal
|-
| community.egi.eu || || it-support@egi.eu || Portal
|-
| sso.egi.eu      || || it-support@egi.eu || Authentication Portal
|}
 
[[Category:Tools]]

Latest revision as of 16:04, 6 February 2020

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Tools menu: • Main page • Instructions for developers • AAI Proxy • Accounting Portal • Accounting Repository • AppDB • ARGO • GGUS • GOCDB
• Message brokers • Licenses • OTAGs • Operations Portal • Perun • EGI Collaboration tools • LToS • EGI Workload Manager


<< go to EGI Collaboration tools main page

This page contains technical details on the services of EGI intranet provided by CESNET.

Core of the services was prepared before the actual start of the EGI-InSPIRE project and they were available from day 1.

This page was updated to describe the status of the services as of June 2015.

Technical background

Hardware

There are two identical servers:

Both the machines are connected to the same disk array:

  • FlexySTOR 162SS
  • 16x 450 GB SAS, 15 krpm disks
  • RAID controller, 2 GB cache
  • the disks are arranged into 2 RAID-10 partitions, yielding 2x 1.8 TB effective capacity

In normal operation each of the machine works in one of the disk array partition, The actual services are implemented in virtual machines, and they are distributed between the physical machines, in order to optimize load.

In case of failure of any of the physical machines the other one takes over hosting the affected virtual machines. Due to the dual connection of the disk array this can be done without the need of any cable switching. Eventually, an automatic fail-over mechanism can be deployed.

Failure of a single disk in the array is handled transparently by the RAID controller. The disks are hot-swappable, allowing seamless replacement of the failed disk.

The whole system is covered with Next-Business-Day On-Site warranty agreement.

The machines are situated in the computer room of Institute of Computer Science of Masaryk University, Brno, CZ.

Virtual hosts

The physical hardware hosts several virtual hosts which provide the actual services in turn. We trade off flexibility and the cost of management, yielding the following virtual hosts currently:

  • mail server and request tracker: optimized for high email traffic
  • web server, including wiki, the document server, blog, SAML identity provider: optimized for the web traffic
  • server with Indico, phpBB forum, jabber
  • server with LimeSurvey and Radius
  • backend server providing database backends to the services

Network connectivity

The computer room where the machines are located is in the same building as the Point of Presence of the CESNET network backbone. The LAN segment of the servers is directly attached to the backbone router port.

Backup

Besides the redundancy provided by the hot-swappable RAID-10 disk array all the systems are backed up with the CESNET tape systems.

In general, full file systems are backed up (with the exception of large database files where the usual approach of snapshot + transaction logs is used), therefore disaster recovery is limited by the time to restore full backup, no manual configuration recovery should be required.

Monitoring

The services are covered by the monitoring system of NGI CZ based on nagios. The following probes are deployed:

  • CPU load and utilization
  • memory usage, including kernel memory
  • critical system messages
  • network interface status
  • file system usage
  • HTTP/HTTPS request sanity on selected URLs
  • pakiti -- up to date status of installed software (missing security fixes in particular)


Operating system and software environment

The hardware servers run Debian Linux, Xen Dom0. Otherwise there are virtually no services installed.

The virtual servers are run as Xen DomU, running Debian as the guest OS as well. Debian was chosen because of stability; among free Linux distributions it has the longest lifetime of stable major releases. We do not expect the need for bleeding edge functionalities in these services therefore stability is prefered.

As a rule of thumb, the EGI services do not depend on any external services outside of this system. Exceptions are DNS and email, relying on the services provided by Masaryk University and CESNET.

Server certificates

Certificates are issued by a commercial certification authority, because the owner of the egi.eu DNS domain is the EGI.eu foundation based in the Netherlands, and certificates issued from the academic TERENA TCS CA through the local NREN (SURFNet) would be far more expensive.

Software customization

When setting up the services we could have not avoided modifications of the used sofware (adding or customizing functionality, integration with the common AuthN/Z etc.). We keep records of trivial modifications, non-trivial modifications are kept in CVS repository, allowing fairly easy merging on upgrade to new versions of the software.

Backend server

Hostname: aldor.ics.muni.cz

Service machine (invisible from outside) hosting database backends of the other services. It is a separate Xen host, so that we are able to move it to other hardware for performance tuning.

Provided services

See also the page EGI_Collaboration_tools

DNS for egi.eu domain

The egi.eu DNS domain is registered at http://www.eurid.eu/ by the EGI.eu foundation, but its DNS servers are provided by CESNET.

EGI SSO

The goal is having a single username/password for all the services. A technical solution is LDAP backend; most services are prepared for LDAP-based authentication out of box, and adaptation of others is relatively easy. Currently we use direct LDAP-based authentication in all the services apart of Mailman, where the user passwords are synchronized with the LDAP every hour.

All users of the services and all people working on the EGI projects are required to register an account at the EGI SSO system. The users can edit properties of their account, arequest password reset and delete their own account.

Some of the services were modified to accept also an X509 digital certificate - Mailman, wiki, DocDB and RT. A certificate can be registered to a user account in the SSO by logging into https://www.egi.eu/sso/user with a certificate and user/password in the same time.

Security considerations: Anybody with an email address can get an EGI SSO account ! Only the e-mail address is verified, all other information is supplied by the user and thus can be fake. The security of the EGI services is based on group membership, as group membership can be granted only by group owners.

Groups

Besides user accounts the LDAP server stores user groups (as groupOfNames) objects. The attribute businessCategory is used to distinguish purpose of the group (multiple values can be specified yielding multi-purpose group):

  • mailman -- members of the group are subscribed to the mailing list of the same name, owners of the group are administrators of the list, the group description is used as list description
  • RT -- group of the same name and members is created in the Request Tracker and can be used for authorization there
  • DocDB -- dtto in the Document Database
  • wiki -- dtto at wiki
  • forum -- dtto at phpBB forum
  • the Indico imports all groups automatically

Group membership is managed at EGI SSO as well. Besides adding and removing users from groups, the group owner can invite external people to create their account and to be subscribed to the group.

Groups can form a hierarchy. A group can be owned by a group, then all members of the owning group can manage the group. A group can also be a subgroup of another group (linked by seeAlso atribute), then all members of the subgroup are also members of the supergroup.

Group membership on the services is synchronized with the LDAP server on every full hour.

Removing SSO group procedure

Following steps are taken once request for SSO group removal is received

  1. Creating a snapshot of members list  (members name and email,  name of the group and short description of the grou)
  2. Informing members of the group that it will be removed (if mailing list was created)
  3. Removing members from list
  4. DocDB, Wiki (namespace) data ownership is transferred to EGI catch-all SSO group (only if no other group would be assigned for the document/namespace)
  5. Closing and archiving mailing list with response message that group has been closed


IdP for external services

The SSO user database can be used for external services using SAML Identity Provider (IdP). The external services must deploy a SAML Service Provider (SP) like Shibboleth or SimpleSAMLphp, and establish mutual trust with the EGI IdP.

A prospective SP should do the following steps:

  1. install the SP software
  2. establish mutual trust with EGI IdP
    • add EGI IdP's metadata from https://www.egi.eu/idp/shibboleth to the SP configuration
    • generate the metadata of their SP (in Shibboleth, available at https://www.example.org/Shibboleth.sso/Metadata) and send them to it-support@egi.eu
  3. configure EGI IdP in their discovery service or as a single IdP for their SP

The IdP provides the following attributes:

attribute friendlyName content
urn:oid:0.9.2342.19200300.100.1.1 uid username from EGI SSO
urn:oid:0.9.2342.19200300.100.1.3 mail email address registered in EGI SSO
urn:oid:1.3.6.1.4.1.5923.1.1.1.6 eduPersonPrincipalName username with appended @egi.eu
urn:oid:2.5.4.3 cn full name in English, ASCII characters only
urn:oid:2.16.840.1.113730.3.1.241 displayName full name in original alphabet, can be Greek, Chinese, etc.
urn:oid:2.5.4.42 givenName first name
urn:oid:2.5.4.4 sn surname, suitable for alphabetical ordering
urn:oid:1.3.6.1.4.1.5923.1.1.1.7 eduPersonEntitlement contains list of groups in the form of URIs like urn:egi.eu:group:somegroup
urn:oid:1.3.6.1.4.1.11433.2.2.1.9 userCertificateSubject distinguished names from X509 certificates in OpenSSL notation
https://www.egi.eu/attribute-def/egiPartnerOrg egiPartnerOrg value from a fixed list of partner organizations, or "other". The fixed list is specified in the drop-down list of organization in SSO edit form
https://www.egi.eu/attribute-def/egiFreeTypedOrg egiFreeTypedOrg anything the user typed, can be misspelled
urn:oid:2.5.4.10 o organization - if egiPartnerOrg!=other then same as egiPartnerOrg, else same as egiFreeTypedOrg


Mailing lists

https://mailman.egi.eu/mailman/listinfo

GNU Mailman software is used, in the version (2.1 currently) provided by Debian OS, with modifications integrating it with the EGI SSO.

List subscribers and list administrators can use their EGI SSO passwords for authentication to the Mailman web interface.

With a few exceptions the mailing lists membership is controlled by EGI SSO. The exceptions are:

  • the ngi-security-contacts and site-security-contacts lists that have members synchronized with the GODCB
  • the announce list that anybody can subscribe and all users from the EGI SSO are added to it
  • the eef-members list which is hosted for the European E-infrastructure Forum

Mailing lists are exposed by the canonical names list-name@mailman.egi.eu.

HTTP server

Apache2, out of the Debian distribution. Its purpose is administrative Mailman interface and access to the mail list archives only. Because most of traffic is expected to be authenticated, port 80 (HTTP default) is redirected to 443 (HTTPS).

Incoming email

The only MX DNS record for mailman.egi.eu points to the Masaryk University mail relay (located in the same building, serving in the same way for several other domains). The relay forwards all mail to mailman.egi.eu via special rule in its config. In this way we gain additional reliability and advanced features of the relay (spam and virus protection).


Outgoing email

Using "smart host" relay.muni.cz for all outgoing email. This is agreed with the relay administrator, and the symmetric setup may have benefits in case of paranoid recipients.


Spam and virus protection

relay.muni.cz (our MX) implements Grey listing technique to ban naive spam attacks.

In addition, we plan to add spam detection set up locally on mailman.egi.eu with Spamassassin, using combination of reliable black lists, static rules for well-known spam patterns (Viagra, Nigerian spam, ...), and dynamic Bayes filters tuned with real trafic gradually.

Exact strategy what to do with spam positives has still to be defined, and it may vary among different lists. In general, as long as it's possible with the amount of the traffic, we are in favour of moderating to let false positives pass rather than discarding automatically.

Viruses are detected at relay.muni.cz with Kaspersky Antivirus, and positives are bounced back to the sender.

Web sites

Main web site with OpenCMS and Peble blog

http://www.egi.eu

This is the project web site and a web front-end for all the services. We use Apache2 from Debian distribution.

Content of the web site is managed by OpenCMS. Google Analytics gathering statistics on the access is deployed.

Part of the suite is a blog managed by Pebble blog software.

Hosted servers

We host additional web sites:


Indico meeting planner

http://www.egi.eu/indico

General meeting planner using the CERN Indico software. It allows scheduling meetings in the full range of size from informal meetings of few participants to large conferences. Meeting agenda can be scheduled, and various material attached to the sections and talks.


Document server

http://documents.egi.eu (public access)

https://documents.egi.eu/secure (authenticated access)

Storing large document files directly on the web site or wiki is not optimal. Instead we provide a dedicated document server for this purpose. Besides optimizing the storage and access, the document server offers the following capabilities:

  • metadata associated with each document
  • versioning of the documents
  • provision of the documents in multiple formats (Word, PDF, ...)
  • fine grained access control based on hierarchy of groups

We use DocDB software with a few local customizations.

Wiki

General purpose wiki

https://wiki.egi.eu/wiki/

General purpose wiki for the use in the project, based on MediaWiki software.

Write access is limited to users registered with EGI SSO. Writing to specific areas (namespaces) is further restricted to SSO groups, as is described at EGIWiki:Community Portal.

Special wikis

Due to the open nature of MediaWiki, it is not possible to reliably restrict read access of selected pages and keep other pages open for reading. Thus there is a dedicated wiki at

https://wiki.egi.eu/csirt/

which is read and write restricted for the members of the EGI-CSIRT-Team group from EGI SSO. Because the whole URL space with https://wiki.egi.eu/csirt/ prefix is closed, it is guaranteed that nobody outside of the group can get any access.

Request tracker

https://rt.egi.eu/

RT help

Work on the project involves tracking wide range of issues, starting from resolution of problems with the intranet services, through managing UMD software releases, upto tracking progress of project tasks and formal deliverables.

We use the RT system], version 3.8. All issues are tracked in terms of tickets arranged in various queues. A ticket follows a defined sequence of major states, custom minor states can be added per queue. Similarly, to the default set of fields in a ticket (owner, priority, etc.) a queue can define additional custom fields with defined data types, value constraints etc. Finally, custom actions can be performed on virtually any change of a ticket. This customizability is starting to be used quite extensively in EGI, allowing to adapt the system to specific needs of the various groups.

The system provides both web and email interfaces, as well as web-service programmatic interface (planned to be used for integration with the EGI software repository).

Whereas most of the queues setup in this RT instance are dedicated for internal use in the project and EGI.eu, a special requirements queue was put in place to be the main gateway to store, monitor and resolve requirements and recommendations communicated by user communities and NGIs to EGI. On the other hand, bug report helpdesk is served by the GGUS system.

Jabber

jabber.egi.eu

We provide a jabber (XMPP instant messaging) server attached to the EGI SSO (each user gets an account automatically) in order not to rely on publicly available services (jabber.org etc.).

The server runs jabberd14 software available in Debian, including the multi-user conference extension which provides chat rooms. Any SSO user can create a chat room, however, the rooms are available to users at jabber.egi.eu only.

Integration with the SSO is done by the xdb_auth_cpile jabberd component.

Eduroam

For the EGI.eu staff included in the SSO group eduroam, we provide Eduroam authentication server.

EGI.eu domain websites

Last update: 2020-02-06

Domain Contact person Contact Email Purpose
operations-portal.egi.eu Cyril Lorphelin cic-information@in2p3.fr Portal
gstat.egi.eu project-grid-info-support@cern.ch Portal
mon.egi.eu Emir Imamagic argo-ggus-support@grnet.gr Portal
midmon.egi.eu Emir Imamagic argo-ggus-support@grnet.gr Portal
opsmon.egi.eu Emir Imamagic argo-ggus-support@grnet.gr Portal
secmon.egi.eu Emir Imamagic argo-ggus-support@grnet.gr Portal
accounting.egi.eu Carlos Fernandez grid-admin@cesga.es Portal
metrics.egi.eu Carlos Fernandez grid-admin@cesga.es Portal
helpdesk.egi.eu Guenter Grein support@ggus.org Portal
goc.egi.eu David Meredith gocdb-admins@mailman.egi.eu Portal
appdb.egi.eu Marios Chatziangelou appdb-support@iasa.gr Portal
site-certification.egi.eu Kostas Koumantaros egi-catch-all-services@lists.grnet.gr Portal
www.egi.eu it-support@egi.eu Main website, blog, identity provider
wiki.egi.eu it-support@egi.eu Wiki
csirt-wiki.egi.eu it-support@egi.eu Wiki
wiki.eosc-hub.eu it-support@egi.eu Wiki
documents.egi.eu it-support@egi.eu Document server
mailman.egi.eu it-support@egi.eu Mailing lists
mailman.eosc-hub.eu it-support@egi.eu Mailing lists
rt.egi.eu it-support@egi.eu Request Tracking
indico.egi.eu it-support@egi.eu Meeting planner
confluence.egi.eu it-support@egi.eu Collaboration wiki
jira.egi.eu it-support@egi.eu Collaboration Tool
pakiti.egi.eu Daniel Kouril daniel.kouril@cesnet.cz software security updates
dirac.egi.eu dirac-support@mailman.egi.eu Portal
e-grant.egi.eu Tomasz Szepieniec egrant-support@mailman.egi.eu Portal
portal.egi.eu Diego Scardaci Diego.Scardaci@egi.eu Portal
unity.egi.eu Tomasz Szepieniec t.szepieniec@cyfronet.pl Authorization service for LTOS based on UNITY
perun.egi.eu Michal Prochazka michalp@ics.muni.cz Portal
repository.egi.eu Kostas Koumantaros egi-repo@hellasgrid.gr Portal
admin-repo.egi.eu Kostas Koumantaros egi-repo@hellasgrid.gr Portal
appliance-repo.egi.eu Kostas Koumantaros egi-repo@hellasgrid.gr Portal
access.egi.eu Tomasz Szepieniec t.szepieniec@cyfronet.pl Portal
argo.egi.eu Christos Kanellopoulos skanct@admin.grnet.gr Portal
aai.egi.eu Peter Solagna peter.solagna@egi.eu AAI pilot
eduroam.egi.eu it-support@egi.eu Portal
council.egi.eu it-support@egi.eu Portal
community.egi.eu it-support@egi.eu Portal
sso.egi.eu it-support@egi.eu Authentication Portal
Retrieved from "https://wiki.egi.eu/w/index.php?title=Intranet&oldid=111042"