We have private partial mirrors of various repositories which we use for testing packages and then to install the packages on each box. These private partial mirrors are synced nightly.
Because of a bug in lftp we can't use Tim's neat redirector and instead we mirror Trinity from mirrorservice.org.
Last night there was no change in our partial Trinity mirror.
The night before some old files were deleted but no new files were added.
The night before that mirroring seemed to be working normally. It brought in a few dozen files from tdeartwork and tdebase.
Also I see nightly builds updating on mirrorservice.org. I wonder if RC2 updates are backed up behind nightly build updates. If so I wonder if it would be better to serve the volatile but lightly used nightly builds directly so that the less volatile and more heavily used releases could more readily migrate to the public mirrors.
--Mike
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA224
We have private partial mirrors of various repositories which we use for testing packages and then to install the packages on each box. These private partial mirrors are synced nightly.
Because of a bug in lftp we can't use Tim's neat redirector and instead we mirror Trinity from mirrorservice.org.
Last night there was no change in our partial Trinity mirror.
The night before some old files were deleted but no new files were added.
The night before that mirroring seemed to be working normally. It brought in a few dozen files from tdeartwork and tdebase.
Also I see nightly builds updating on mirrorservice.org. I wonder if RC2 updates are backed up behind nightly build updates. If so I wonder if it would be better to serve the volatile but lightly used nightly builds directly so that the less volatile and more heavily used releases could more readily migrate to the public mirrors.
Yes, the R14 update is probably backed up behind the nightlies. Problem is, if I disable the nightly mirror mirrorservice.org will purge all of the nightly files, causing the TDE servers to re-upload over 50 GB of data (this is something I cannot afford to do).
I have had a couple of offers in the past for hosting a primary mirror with push rsync; while I have been hesitant to make mirrorservice.org a secondary mirror due to its high availability and bandwidth I may have to do this anyway. Any changes would not occur until after R14 release and full sync, so this decision will be delayed for now
Tim
On Fri December 5 2014 09:08:56 Timothy Pearson wrote:
Yes, the R14 update is probably backed up behind the nightlies. Problem is, if I disable the nightly mirror mirrorservice.org will purge all of the nightly files, causing the TDE servers to re-upload over 50 GB of data (this is something I cannot afford to do).
Do you have enough disk space to LVM snapshot the repository upon RC or release and temporarily serve rsync from the snapshot instead of the live repository until the RC or release is mirrored?
If RCs and releases can be mirrored sooner that will reduce pressure on your bandwidth from RC and release users.
--Mike
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA224
On Fri December 5 2014 09:08:56 Timothy Pearson wrote:
Yes, the R14 update is probably backed up behind the nightlies. Problem is, if I disable the nightly mirror mirrorservice.org will purge all of the nightly files, causing the TDE servers to re-upload over 50 GB of data (this is something I cannot afford to do).
Do you have enough disk space to LVM snapshot the repository upon RC or release and temporarily serve rsync from the snapshot instead of the live repository until the RC or release is mirrored?
If RCs and releases can be mirrored sooner that will reduce pressure on your bandwidth from RC and release users.
--Mike
I don't have that amount of control over the remote mirror; mirrorservice.org simply rsyncs via a cron job from the server here.
Tim
On Fri December 5 2014 23:42:27 Timothy Pearson wrote:
I don't have that amount of control over the remote mirror; mirrorservice.org simply rsyncs via a cron job from the server here.
Can the source repository on your server be frozen by LVM snapshot or just by stopping updates until the mirror syncs?
//
FWIW (1) the latest 24 hours saw the following changes on mirrorservice.org as seen through our filter which only mirrors for Wheezy and Jessie:
* R14 and builddeps - none. * NB and builddeps - 28 new files.
//
FWIW (2) the most recent R14 file I see on mirrorservice.org is dists/wheezy/main/binary-amd64/Packages Which is 2½ weeks old.
//
It looks like the build farm generates something like 32 versions of nightly builds. I wonder what fraction of the nightly builds are ever downloaded from the mirror before a new version comes along. It might save bandwidth overall if the nightly builds were served direct from your server without mirroring.
--Mike
In the last couple of days there has been a dramatic improvement in mirroring speed - roughly a factor of 20 - and mirror state has progressed from roughly 88% of RC1 to roughly 62% of RC2.
If this continues RC2 should be mirrored in roughly one more day.
--Mike
-
Mike Bird -
Timothy Pearson