Difference between revisions of "MirroringUMD"
Jump to navigation
Jump to search
(9 intermediate revisions by the same user not shown) | |||
Line 5: | Line 5: | ||
=Mirroring of UMD Repositories = | =Mirroring of UMD Repositories = | ||
We provide some basic information needed in order to mirror UMD repositories | We provide some basic information needed in order to mirror [http://repository.egi.eu/sw/ UMD repositories]. | ||
Available repos: | |||
* [http://repository.egi.eu/sw/production/cas/1/ CAS] | |||
* [http://repository.egi.eu/sw/production/sam/1/ SAM] | |||
* [http://repository.egi.eu/sw/production/umd/ UMD production (including ReleaseCandidate] | |||
* [http://repository.egi.eu/sw/stagerollout/ UMD staged-rollout] | |||
* [http://repository.egi.eu/sw/testing/umd/ UMD testing]- includes everything that is currently in the Stage Rollout or UMD-Store State | |||
* [http://repository.egi.eu/sw/untested/ UMD untested] - includes everything that is currently in the Unverified or In Verification State | |||
== Communication == | == Communication == | ||
* Mailing-lists | * Mailing-lists | ||
** for announcements (changes, new repos, etc): mirror-egi at egi.eu | ** for announcements (changes, new repos, etc): mirror-egi at egi.eu - TO BE SET-UP | ||
** for discussion: mirror-egi | ** for discussion: FORUM - mirror-egi'' - TO BE SET-UP | ||
== Repos size estimates == | == Repos size estimates == | ||
Line 18: | Line 25: | ||
== Export License == | == Export License == | ||
Do we need something similar to http://fedoraproject.org/wiki/Infrastructure/Mirroring#Export_Compliance ? | <!-- Do we need something similar to http://fedoraproject.org/wiki/Infrastructure/Mirroring#Export_Compliance ? --> | ||
== Mirroring == | == Mirroring == | ||
Line 25: | Line 32: | ||
=== Using lftp === | === Using lftp === | ||
Depending on what repositories you want to mirror and the space you have available (see [[#Repos size estimates]] you can use something similar: | |||
* to create the mirror for the first time: | |||
< | <pre> # lftp -c mirror --verbose --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}</pre> | ||
lftp -c mirror --verbose --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR} | * to removes any local packages that no longer exist on the source, and only downloads newer package versions from the host mirror: | ||
</ | <pre> | ||
< | |||
lftp -c mirror –-only-newer --ignore-time -–delete --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR} | lftp -c mirror –-only-newer --ignore-time -–delete --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR} | ||
</ | </pre> | ||
=== Using wget === | === Using wget === | ||
* at the moment "wget -r" will not work, please use the lftp solution described above. | |||
=== Using rsync === | |||
* A rsync server is availabe at '''rsync.athena.hellasgrid.gr''' | |||
* it can be use with: | |||
<pre> | |||
# rsync -avzH --delete --exclude cas/ --exclude sam/ --exclude umd/1/ --exclude umd/2/ --exclude umd/candidate/ rsync://rsync.athena.hellasgrid.gr/production/ <path_to_your_directory/ | |||
=== | </pre> | ||
* or you can create a script to be run as a cron job: | |||
<pre> | |||
# cat mirror_UMD.sh | |||
#!/bin/sh | |||
DATE=`/bin/date +%Y-%m-%d` | |||
OUTDIR='/tmp' | |||
MIRROR=/rep/repo/UMD/production/umd/ | |||
[ -d $OUTDIR ] || mkdir -p $OUTDIR | |||
OUTFILE=$OUTDIR/mirror-UMD-$DATE.txt | |||
/bin/nice /usr/bin/rsync --verbose --progress \ | |||
--stats --archive --partial -H --delete \ | |||
--exclude cas/ \ | |||
--exclude sam/ \ | |||
--exclude umd/1 \ | |||
--exclude umd/2 \ | |||
--exclude umd/candidate \ | |||
rsync.athena.hellasgrid.gr::production/ $MIRROR/ \ | |||
>> $OUTFILE | |||
</pre> |
Latest revision as of 12:47, 19 February 2015
Technology | Software Component Delivery | Software Provisioning | UMD Middleware | Cloud Middleware Distribution | Containers Distribution | Technology Glossary |
Software Provisioning menu: | Software Provisioning Process | UMD Release Process | Quality Assurance | UMD | Staged Rollout |
Mirroring of UMD Repositories
We provide some basic information needed in order to mirror UMD repositories. Available repos:
- CAS
- SAM
- UMD production (including ReleaseCandidate
- UMD staged-rollout
- UMD testing- includes everything that is currently in the Stage Rollout or UMD-Store State
- UMD untested - includes everything that is currently in the Unverified or In Verification State
Communication
- Mailing-lists
- for announcements (changes, new repos, etc): mirror-egi at egi.eu - TO BE SET-UP
- for discussion: FORUM - mirror-egi - TO BE SET-UP
Repos size estimates
- please see the content of the following file for insfo on available directories and their estimated sizes:
Export License
Mirroring
Using lftp
Depending on what repositories you want to mirror and the space you have available (see #Repos size estimates you can use something similar:
- to create the mirror for the first time:
# lftp -c mirror --verbose --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
- to removes any local packages that no longer exist on the source, and only downloads newer package versions from the host mirror:
lftp -c mirror –-only-newer --ignore-time -–delete --exclude 'candidate/' --exclude '1/' --exclude '2/' http://repository.egi.eu/sw/production/umd/ ${LOCAL_DIR}
Using wget
- at the moment "wget -r" will not work, please use the lftp solution described above.
Using rsync
- A rsync server is availabe at rsync.athena.hellasgrid.gr
- it can be use with:
# rsync -avzH --delete --exclude cas/ --exclude sam/ --exclude umd/1/ --exclude umd/2/ --exclude umd/candidate/ rsync://rsync.athena.hellasgrid.gr/production/ <path_to_your_directory/
- or you can create a script to be run as a cron job:
# cat mirror_UMD.sh #!/bin/sh DATE=`/bin/date +%Y-%m-%d` OUTDIR='/tmp' MIRROR=/rep/repo/UMD/production/umd/ [ -d $OUTDIR ] || mkdir -p $OUTDIR OUTFILE=$OUTDIR/mirror-UMD-$DATE.txt /bin/nice /usr/bin/rsync --verbose --progress \ --stats --archive --partial -H --delete \ --exclude cas/ \ --exclude sam/ \ --exclude umd/1 \ --exclude umd/2 \ --exclude umd/candidate \ rsync.athena.hellasgrid.gr::production/ $MIRROR/ \ >> $OUTFILE