Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "MirroringUMD"

From EGIWiki
Jump to navigation Jump to search
 
(One intermediate revision by the same user not shown)
Line 9: Line 9:
* [http://repository.egi.eu/sw/production/cas/1/ CAS]
* [http://repository.egi.eu/sw/production/cas/1/ CAS]
* [http://repository.egi.eu/sw/production/sam/1/ SAM]
* [http://repository.egi.eu/sw/production/sam/1/ SAM]
* [http://repository.egi.eu/sw/production/umd/ UMD production (including rReleaseCandidate]
* [http://repository.egi.eu/sw/production/umd/ UMD production (including ReleaseCandidate]
* [http://repository.egi.eu/sw/stagerollout/ UMD staged-rollout]
* [http://repository.egi.eu/sw/stagerollout/ UMD staged-rollout]
* [http://repository.egi.eu/sw/testing/umd/ UMD testing]-  includes everything that is currently in the Stage Rollout or UMD-Store State
* [http://repository.egi.eu/sw/testing/umd/ UMD testing]-  includes everything that is currently in the Stage Rollout or UMD-Store State
Line 32: Line 32:
=== Using lftp ===
=== Using lftp ===


* Depending on what repositories you want to mirror and the space you have available (see [[#Repos size estimates]] you can use something similar:
Depending on what repositories you want to mirror and the space you have available (see [[#Repos size estimates]] you can use something similar:
** to create the mirror for the first time:
* to create the mirror for the first time:
<blockquote>
<pre> # lftp -c mirror --verbose --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}</pre>
lftp -c mirror --verbose --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
* to removes any local packages that no longer exist on the source, and only downloads newer package versions from the host mirror:
</blockquote>
<pre>
** to removes any local packages that no longer exist on the source, and only downloads newer package versions from the host mirror:
<blockquote>
lftp -c mirror –-only-newer --ignore-time -–delete --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
lftp -c mirror –-only-newer --ignore-time -–delete --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
</blockquote>
</pre>


=== Using wget ===
=== Using wget ===
* at the moment "wget -r" will not work, please use the lftp solution described above.
* at the moment "wget -r" will not work, please use the lftp solution described above.
<!--
* Taking as example [http://fedoranews.org/alex/tutorial/yum/ Fedora's tutorial on using wget to create a local mirror] you can use the following [https://raw.githubusercontent.com/caifti/repos_mirror/master/UMD_yum_repository.sh script] modify it to reflect the repositories you want to mirror, if you want to use the crearerepo or not, etc.


* You can find other examples on how to use wget for mirroring repositories:
=== Using rsync ===
** [https://peterpap.net/index.php/Creating_a_CentOS_mirror_site Creating a CentOS mirror site]
-->


=== Using rsync ===
* A rsync server is availabe at '''rsync.athena.hellasgrid.gr'''
 
* it can be use with:
<pre>
# rsync -avzH --delete --exclude cas/ --exclude sam/ --exclude umd/1/ --exclude umd/2/ --exclude umd/candidate/ rsync://rsync.athena.hellasgrid.gr/production/ <path_to_your_directory/
</pre>
* or you can create a script to be run as a cron job:
<pre>
# cat mirror_UMD.sh
#!/bin/sh
DATE=`/bin/date +%Y-%m-%d`
OUTDIR='/tmp'
MIRROR=/rep/repo/UMD/production/umd/
[ -d $OUTDIR ] || mkdir -p $OUTDIR
OUTFILE=$OUTDIR/mirror-UMD-$DATE.txt


* at the moment we do not offer a rsync server for UMD repositories
/bin/nice /usr/bin/rsync --verbose --progress \
  --stats --archive --partial -H --delete \
  --exclude cas/ \
  --exclude sam/ \
  --exclude umd/1 \
  --exclude umd/2 \
  --exclude umd/candidate \
  rsync.athena.hellasgrid.gr::production/ $MIRROR/ \
  >> $OUTFILE
</pre>

Latest revision as of 11:47, 19 February 2015

Technology Software Component Delivery Software Provisioning UMD Middleware Cloud Middleware Distribution Containers Distribution Technology Glossary


Software Provisioning menu: Software Provisioning Process UMD Release Process Quality Assurance UMD Staged Rollout





Mirroring of UMD Repositories

We provide some basic information needed in order to mirror UMD repositories. Available repos:

Communication

  • Mailing-lists
    • for announcements (changes, new repos, etc): mirror-egi at egi.eu - TO BE SET-UP
    • for discussion: FORUM - mirror-egi - TO BE SET-UP

Repos size estimates

  • please see the content of the following file for insfo on available directories and their estimated sizes:

Export License

Mirroring

Using lftp

Depending on what repositories you want to mirror and the space you have available (see #Repos size estimates you can use something similar:

  • to create the mirror for the first time:
 # lftp -c mirror --verbose --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
  • to removes any local packages that no longer exist on the source, and only downloads newer package versions from the host mirror:
lftp -c mirror –-only-newer --ignore-time -–delete --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}

Using wget

  • at the moment "wget -r" will not work, please use the lftp solution described above.

Using rsync

  • A rsync server is availabe at rsync.athena.hellasgrid.gr
  • it can be use with:
# rsync -avzH --delete --exclude cas/ --exclude sam/ --exclude umd/1/ --exclude umd/2/ --exclude umd/candidate/ rsync://rsync.athena.hellasgrid.gr/production/ <path_to_your_directory/
  • or you can create a script to be run as a cron job:
# cat mirror_UMD.sh 
#!/bin/sh
DATE=`/bin/date +%Y-%m-%d`
OUTDIR='/tmp'
MIRROR=/rep/repo/UMD/production/umd/
[ -d $OUTDIR ] || mkdir -p $OUTDIR
OUTFILE=$OUTDIR/mirror-UMD-$DATE.txt

/bin/nice /usr/bin/rsync --verbose --progress \
   --stats --archive --partial -H --delete \
   --exclude cas/ \
   --exclude sam/ \
   --exclude umd/1 \
   --exclude umd/2 \
   --exclude umd/candidate \
   rsync.athena.hellasgrid.gr::production/ $MIRROR/ \
   >> $OUTFILE