MirroringUMD

From EGIWiki
Jump to: navigation, search
Technology Software Component Delivery Software Provisioning UMD Middleware Cloud Middleware Distribution Containers Distribution Technology Glossary


Software Provisioning menu: Software Provisioning Process UMD Release Process Quality Assurance UMD Staged Rollout





Mirroring of UMD Repositories

We provide some basic information needed in order to mirror UMD repositories. Available repos:

Communication

  • Mailing-lists
    • for announcements (changes, new repos, etc): mirror-egi at egi.eu - TO BE SET-UP
    • for discussion: FORUM - mirror-egi - TO BE SET-UP

Repos size estimates

  • please see the content of the following file for insfo on available directories and their estimated sizes:

Export License

Mirroring

Using lftp

Depending on what repositories you want to mirror and the space you have available (see #Repos size estimates you can use something similar:

  • to create the mirror for the first time:
 # lftp -c mirror --verbose --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
  • to removes any local packages that no longer exist on the source, and only downloads newer package versions from the host mirror:
lftp -c mirror –-only-newer --ignore-time -–delete --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}

Using wget

  • at the moment "wget -r" will not work, please use the lftp solution described above.

Using rsync

  • A rsync server is availabe at rsync.athena.hellasgrid.gr
  • it can be use with:
# rsync -avzH --delete --exclude cas/ --exclude sam/ --exclude umd/1/ --exclude umd/2/ --exclude umd/candidate/ rsync://rsync.athena.hellasgrid.gr/production/ <path_to_your_directory/
  • or you can create a script to be run as a cron job:
# cat mirror_UMD.sh 
#!/bin/sh
DATE=`/bin/date +%Y-%m-%d`
OUTDIR='/tmp'
MIRROR=/rep/repo/UMD/production/umd/
[ -d $OUTDIR ] || mkdir -p $OUTDIR
OUTFILE=$OUTDIR/mirror-UMD-$DATE.txt

/bin/nice /usr/bin/rsync --verbose --progress \
   --stats --archive --partial -H --delete \
   --exclude cas/ \
   --exclude sam/ \
   --exclude umd/1 \
   --exclude umd/2 \
   --exclude umd/candidate \
   rsync.athena.hellasgrid.gr::production/ $MIRROR/ \
   >> $OUTFILE