Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "MirroringUMD"

From EGIWiki
Jump to navigation Jump to search
 
(18 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{|
{{Tech menubar}} {{SWProv menubar}}  
|-
{{TOC_right}}  
|-
 
|}
{{TOC right}}


=Mirroring of UMD Repositories =
=Mirroring of UMD Repositories =


We provide some basic information needed in order to mirror UMD repositories
We provide some basic information needed in order to mirror [http://repository.egi.eu/sw/ UMD repositories].
Available repos:
* [http://repository.egi.eu/sw/production/cas/1/ CAS]
* [http://repository.egi.eu/sw/production/sam/1/ SAM]
* [http://repository.egi.eu/sw/production/umd/ UMD production (including ReleaseCandidate]
* [http://repository.egi.eu/sw/stagerollout/ UMD staged-rollout]
* [http://repository.egi.eu/sw/testing/umd/ UMD testing]-  includes everything that is currently in the Stage Rollout or UMD-Store State
* [http://repository.egi.eu/sw/untested/ UMD untested] - includes everything that is currently in the Unverified or In Verification State


== Communication ==  
== Communication ==  
* Mailing-lists
* Mailing-lists
** for announcements (changes, new repos, etc): mirror-egi at egi.eu
** for announcements (changes, new repos, etc): mirror-egi at egi.eu - TO BE SET-UP
** for discussion: mirror-egi-d at egi.eu or ''can we suggest FORUM - mirror-egi''
** for discussion: FORUM - mirror-egi'' - TO BE SET-UP


== Repos size estimates ==  
== Repos size estimates ==  
* file.txt with "du -hs /<dir_repos>
* please see the content of the following file for insfo on available directories and their estimated sizes:
<!--** [[File:EGI_repos_sizes.pdf|EGI repos sizes]]-->
**[https://wiki.egi.eu/w/images/7/7e/EGI_repos_sizes.pdf EGI repositories sizes]


== Export License ==
== Export License ==
Do we need something similar to http://fedoraproject.org/wiki/Infrastructure/Mirroring#Export_Compliance ?
<!-- Do we need something similar to http://fedoraproject.org/wiki/Infrastructure/Mirroring#Export_Compliance ? -->


== Mirroring ==  
== Mirroring ==  
Line 26: Line 32:
=== Using lftp ===
=== Using lftp ===


* Depending on what repositories you want to mirror and the space you have available (see [[#Repos size estimates]] you can use something similar:
Depending on what repositories you want to mirror and the space you have available (see [[#Repos size estimates]] you can use something similar:
** to create the mirror for the first time:
* to create the mirror for the first time:
<blockquote>
<pre> # lftp -c mirror --verbose --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}</pre>
lftp -c mirror --verbose --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
* to removes any local packages that no longer exist on the source, and only downloads newer package versions from the host mirror:
</blockquote>
<pre>
** to removes any local packages that no longer exist on the source, and only downloads newer package versions from the host mirror:
<blockquote>
lftp -c mirror –-only-newer --ignore-time -–delete --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
lftp -c mirror –-only-newer --ignore-time -–delete --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
</blockquote>
</pre>
 
=== Using wget ===
* at the moment "wget -r" will not work, please use the lftp solution described above.


=== Using rsync ===
=== Using rsync ===


* at the moment we do not offer a rsync server for UDM reposotories
* A rsync server is availabe at '''rsync.athena.hellasgrid.gr'''
 
* it can be use with:
<pre>
# rsync -avzH --delete --exclude cas/ --exclude sam/ --exclude umd/1/ --exclude umd/2/ --exclude umd/candidate/ rsync://rsync.athena.hellasgrid.gr/production/ <path_to_your_directory/
</pre>
* or you can create a script to be run as a cron job:
<pre>
# cat mirror_UMD.sh
#!/bin/sh
DATE=`/bin/date +%Y-%m-%d`
OUTDIR='/tmp'
MIRROR=/rep/repo/UMD/production/umd/
[ -d $OUTDIR ] || mkdir -p $OUTDIR
OUTFILE=$OUTDIR/mirror-UMD-$DATE.txt
 
/bin/nice /usr/bin/rsync --verbose --progress \
  --stats --archive --partial -H --delete \
  --exclude cas/ \
  --exclude sam/ \
  --exclude umd/1 \
  --exclude umd/2 \
  --exclude umd/candidate \
  rsync.athena.hellasgrid.gr::production/ $MIRROR/ \
  >> $OUTFILE
</pre>

Latest revision as of 11:47, 19 February 2015

Technology Software Component Delivery Software Provisioning UMD Middleware Cloud Middleware Distribution Containers Distribution Technology Glossary


Software Provisioning menu: Software Provisioning Process UMD Release Process Quality Assurance UMD Staged Rollout





Mirroring of UMD Repositories

We provide some basic information needed in order to mirror UMD repositories. Available repos:

Communication

  • Mailing-lists
    • for announcements (changes, new repos, etc): mirror-egi at egi.eu - TO BE SET-UP
    • for discussion: FORUM - mirror-egi - TO BE SET-UP

Repos size estimates

  • please see the content of the following file for insfo on available directories and their estimated sizes:

Export License

Mirroring

Using lftp

Depending on what repositories you want to mirror and the space you have available (see #Repos size estimates you can use something similar:

  • to create the mirror for the first time:
 # lftp -c mirror --verbose --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
  • to removes any local packages that no longer exist on the source, and only downloads newer package versions from the host mirror:
lftp -c mirror –-only-newer --ignore-time -–delete --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}

Using wget

  • at the moment "wget -r" will not work, please use the lftp solution described above.

Using rsync

  • A rsync server is availabe at rsync.athena.hellasgrid.gr
  • it can be use with:
# rsync -avzH --delete --exclude cas/ --exclude sam/ --exclude umd/1/ --exclude umd/2/ --exclude umd/candidate/ rsync://rsync.athena.hellasgrid.gr/production/ <path_to_your_directory/
  • or you can create a script to be run as a cron job:
# cat mirror_UMD.sh 
#!/bin/sh
DATE=`/bin/date +%Y-%m-%d`
OUTDIR='/tmp'
MIRROR=/rep/repo/UMD/production/umd/
[ -d $OUTDIR ] || mkdir -p $OUTDIR
OUTFILE=$OUTDIR/mirror-UMD-$DATE.txt

/bin/nice /usr/bin/rsync --verbose --progress \
   --stats --archive --partial -H --delete \
   --exclude cas/ \
   --exclude sam/ \
   --exclude umd/1 \
   --exclude umd/2 \
   --exclude umd/candidate \
   rsync.athena.hellasgrid.gr::production/ $MIRROR/ \
   >> $OUTFILE