Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

APEL/SSM2AddingFiles

From EGIWiki
< APEL
Revision as of 17:42, 1 November 2018 by Acatstfc (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Adding files to SSM2

In SSM2 we changed to using the EMI Messaging team's 'directory queue' libraries: https://twiki.cern.ch/twiki/bin/view/EMI/MessagingLibraries. This changes how messages are added to SSM2.

There are two ways of adding messages to SSM2:

  • Programmatically, using perl or python
  • Manually, writing files to the filesystem

Programmatically

You can add messages to SSM2 using the python-dirq or the perl-Directory-Queue libraries.

Create a QueueSimple object with path /var/spool/apel/outgoing and add messages. Follow the above link or email apel-admins@stfc.ac.uk for more help.

Manually

If you are not using perl or python, you can simply write files to the filesystem and SSM2 will send them. However, there are constraints on the filenames:

  • Files must be in the location /var/spool/apel/outgoing/<directory>/<file>
  • <directory> must be 8 hex characters - that is 0-9 or a-f
  • <file> must be 14 hex characters

Example naming scheme

These notes are taken from the dirq documentation and describe a possible (not compulsory) naming scheme.

Directory Structure


The toplevel directory contains intermediate directories that contain the stored elements, each of them in a file.

The names of the intermediate directories are time based: the element insertion time is used to create a 8-digits long hexadecimal number. The granularity (see the constructor) is used to limit the number of new directories. For instance, with a granularity of 60 (the default), new directories will be created at most once per minute.

Since there is usually a filesystem limit in the number of directories a directory can hold, there is a trade-off to be made. If you want to support many added elements per second, you should use a low granularity to keep small directories. However, in this case, you will create many directories and this will limit the total number of elements you can store.

The elements themselves are stored in files (one per element) with a 14-digits long hexadecimal name SSSSSSSSMMMMMR where:

  • SSSSSSSS represents the number of seconds since the Epoch
  • MMMMM represents the microsecond part of the time since the Epoch
  • R is a random digit used to reduce name collisions