Difference between revisions of "MirroringUMD"

From EGIWiki
Jump to: navigation, search
(Mirroring of UMD Repositories)
(Mirroring)
 
Line 32: Line 32:
 
=== Using lftp ===
 
=== Using lftp ===
  
* Depending on what repositories you want to mirror and the space you have available (see [[#Repos size estimates]] you can use something similar:
+
Depending on what repositories you want to mirror and the space you have available (see [[#Repos size estimates]] you can use something similar:
** to create the mirror for the first time:
+
* to create the mirror for the first time:
<blockquote>
+
<pre> # lftp -c mirror --verbose --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}</pre>
lftp -c mirror --verbose --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
+
* to removes any local packages that no longer exist on the source, and only downloads newer package versions from the host mirror:
</blockquote>
+
<pre>
** to removes any local packages that no longer exist on the source, and only downloads newer package versions from the host mirror:
 
<blockquote>
 
 
lftp -c mirror –-only-newer --ignore-time -–delete --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
 
lftp -c mirror –-only-newer --ignore-time -–delete --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
</blockquote>
+
</pre>
  
 
=== Using wget ===
 
=== Using wget ===
 
* at the moment "wget -r" will not work, please use the lftp solution described above.
 
* at the moment "wget -r" will not work, please use the lftp solution described above.
<!--
 
* Taking as example [http://fedoranews.org/alex/tutorial/yum/ Fedora's tutorial on using wget to create a local mirror] you can use the following [https://raw.githubusercontent.com/caifti/repos_mirror/master/UMD_yum_repository.sh script] modify it to reflect the repositories you want to mirror, if you want to use the crearerepo or not, etc.
 
  
* You can find other examples on how to use wget for mirroring repositories:
+
=== Using rsync ===
** [https://peterpap.net/index.php/Creating_a_CentOS_mirror_site Creating a CentOS mirror site]
 
-->
 
  
=== Using rsync ===
+
* A rsync server is availabe at '''rsync.athena.hellasgrid.gr'''
 +
 
 +
* it can be use with:
 +
<pre>
 +
# rsync -avzH --delete --exclude cas/ --exclude sam/ --exclude umd/1/ --exclude umd/2/ --exclude umd/candidate/ rsync://rsync.athena.hellasgrid.gr/production/ <path_to_your_directory/
 +
</pre>
 +
* or you can create a script to be run as a cron job:
 +
<pre>
 +
# cat mirror_UMD.sh
 +
#!/bin/sh
 +
DATE=`/bin/date +%Y-%m-%d`
 +
OUTDIR='/tmp'
 +
MIRROR=/rep/repo/UMD/production/umd/
 +
[ -d $OUTDIR ] || mkdir -p $OUTDIR
 +
OUTFILE=$OUTDIR/mirror-UMD-$DATE.txt
  
* at the moment we do not offer a rsync server for UMD repositories
+
/bin/nice /usr/bin/rsync --verbose --progress \
 +
  --stats --archive --partial -H --delete \
 +
  --exclude cas/ \
 +
  --exclude sam/ \
 +
  --exclude umd/1 \
 +
  --exclude umd/2 \
 +
  --exclude umd/candidate \
 +
  rsync.athena.hellasgrid.gr::production/ $MIRROR/ \
 +
  >> $OUTFILE
 +
</pre>

Latest revision as of 11:47, 19 February 2015

Technology Software Component Delivery Software Provisioning UMD Middleware Cloud Middleware Distribution Containers Distribution Technology Glossary


Software Provisioning menu: Software Provisioning Process UMD Release Process Quality Assurance UMD Staged Rollout





Mirroring of UMD Repositories

We provide some basic information needed in order to mirror UMD repositories. Available repos:

Communication

  • Mailing-lists
    • for announcements (changes, new repos, etc): mirror-egi at egi.eu - TO BE SET-UP
    • for discussion: FORUM - mirror-egi - TO BE SET-UP

Repos size estimates

  • please see the content of the following file for insfo on available directories and their estimated sizes:

Export License

Mirroring

Using lftp

Depending on what repositories you want to mirror and the space you have available (see #Repos size estimates you can use something similar:

  • to create the mirror for the first time:
 # lftp -c mirror --verbose --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
  • to removes any local packages that no longer exist on the source, and only downloads newer package versions from the host mirror:
lftp -c mirror –-only-newer --ignore-time -–delete --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}

Using wget

  • at the moment "wget -r" will not work, please use the lftp solution described above.

Using rsync

  • A rsync server is availabe at rsync.athena.hellasgrid.gr
  • it can be use with:
# rsync -avzH --delete --exclude cas/ --exclude sam/ --exclude umd/1/ --exclude umd/2/ --exclude umd/candidate/ rsync://rsync.athena.hellasgrid.gr/production/ <path_to_your_directory/
  • or you can create a script to be run as a cron job:
# cat mirror_UMD.sh 
#!/bin/sh
DATE=`/bin/date +%Y-%m-%d`
OUTDIR='/tmp'
MIRROR=/rep/repo/UMD/production/umd/
[ -d $OUTDIR ] || mkdir -p $OUTDIR
OUTFILE=$OUTDIR/mirror-UMD-$DATE.txt

/bin/nice /usr/bin/rsync --verbose --progress \
   --stats --archive --partial -H --delete \
   --exclude cas/ \
   --exclude sam/ \
   --exclude umd/1 \
   --exclude umd/2 \
   --exclude umd/candidate \
   rsync.athena.hellasgrid.gr::production/ $MIRROR/ \
   >> $OUTFILE