24 Jun
2018
24 Jun
'18
6:59 p.m.
On Sun June 24 2018 10:14:39 Slávek Banko wrote:
at mirror.xcer.cz, colleague has not yet made the
removal of distributions
that are no longer build for Preliminary Stable Builds. Therefore, packages
for these distributions are still available on mirror.xcer.cz. While on
mirror.ppa.trinitydesktop.org is cleaning old packages more actively. This
is why the content may vary. Both of these mirrors are synchronized from my
main server using apt-mirror. Therefore, removing old packages can be done
differently on each mirror.
If you want to synchronize Preliminary Stable Builds repository to your
mirror, I can setup rsync on mirror.ppa.trinitydesktop.org. Would it be
useful for you?
// Retitled and moved from trinity-users to trinity-devel.
Hi Slavek,
The mirrors are here to serve TDE so the question is what would be useful to
TDE devs and users?
Here's a little background FYI.
(1) The primary mirror has roughly 240GB of which 192GB is currently in use.
This can be increased but of course none of us want to spend more money
on renting data center disk space than necessary. I don't know how much
disk space is available for TDE on the secondary mirrors.
27G ./cdimages
1.7G ./git-images
14G ./libreoffice-trinity
38M ./openldap
24G ./releases
62G ./trinity
384M ./trinity-builddeps
1.5G ./trinity-builddeps-r14.0.0
929M ./trinity-builddeps-v3.5.13
1.5G ./trinity-nightly-build-dependencies
412K ./trinity-nightly-builds-01
41G ./trinity-r14.0.0
20G ./trinity-v3.5.13
553M ./ulab
(2) I have not yet exceeded monthly transfer limits and hope not to do so.
I can serve normal traffic and new releases but please contact mirror
admins before any overall reorganization which could be done more
efficiently with mv than rsync.
(3) I am not personally in direct contact with any of the other mirrors, even
though they all pull from us. Contact between the mirror admins has AFAIK
always been through Tim. We created the primary mirror to alleviate
bandwidth problems at Tim's build farm in the past when all mirrors
pulled direct from him.
(4) The primary mirror uses rsync rather than apt-mirror, and I suspect the
same is true for the other mirrors.
(5) I appreciate heads up on any planned major changes to mirror content so
I can increase monitoring and if necessary change mirroring parameters.
It might be best to provide such heads up off-list because ...
(6) To avoid overloading build farm bandwidth, it is best not to announce
releases until they are fully mirrored. If users start pulling unmirrored
files they are served from the build farm, overloading its bandwidth and
slowing mirroring, so things get very slow for everyone.
As a TDE user myself I would find it convenient but not critical if there
were fewer differences between PSB and Stable. Ideally they would be in
the same repo pool - like Debian testing and stable.
--Mike
21 Jul
21 Jul
5:20 p.m.
New subject: [trinity-devel] Mirrors and PSB
Hi Mike,
Sorry for a later answer.
Answers are inscribed on individual points.
On Sunday 24 of June 2018 20:59:46 Mike Bird wrote:
On Sun June 24 2018 10:14:39 Slávek Banko wrote:
The advantage of existing solutions where Preliminary Stable Builds and
Preliminary Testing Builds are completely out of official mirrors is that
there is no need to address additional space on mirrors. The official
mirrors contain the final official packages. Redirector provides access
to pre-packages. Everyone can be happy.
at mirror.xcer.cz, colleague has not yet made the
removal of
distributions that are no longer build for Preliminary Stable Builds.
Therefore, packages for these distributions are still available on
mirror.xcer.cz. While on mirror.ppa.trinitydesktop.org is cleaning
old packages more actively. This is why the content may vary. Both of
these mirrors are synchronized from my main server using apt-mirror.
Therefore, removing old packages can be done differently on each
mirror.
If you want to synchronize Preliminary Stable Builds repository to
your mirror, I can setup rsync on mirror.ppa.trinitydesktop.org.
Would it be useful for you?
// Retitled and moved from trinity-users to trinity-devel.
Hi Slavek,
The mirrors are here to serve TDE so the question is what would be
useful to TDE devs and users?
Here's a little background FYI.
(1) The primary mirror has roughly 240GB of which 192GB is currently in
use. This can be increased but of course none of us want to spend more
money on renting data center disk space than necessary. I don't know
how much disk space is available for TDE on the secondary mirrors.
27G ./cdimages
1.7G ./git-images
14G ./libreoffice-trinity
38M ./openldap
24G ./releases
62G ./trinity
384M ./trinity-builddeps
1.5G ./trinity-builddeps-r14.0.0
929M ./trinity-builddeps-v3.5.13
1.5G ./trinity-nightly-build-dependencies
412K ./trinity-nightly-builds-01
41G ./trinity-r14.0.0
20G ./trinity-v3.5.13
553M ./ulab
(2) I have not yet exceeded monthly transfer limits
and hope not to do
so. I can serve normal traffic and new releases but please contact
mirror admins before any overall reorganization which could be done
more efficiently with mv than rsync.
Because Preliminary Stable Builds and Preliminary Testing Builds are
completely independent of the official mirrors, this solution provides
the advantage that synchronization from my server to the repository is
done more often - packages are checked every two hours.
Therefore, packages can be delivered to users faster. At the same time,
there is no risk that the transfer limits on mirrors will be exceeded.
(3) I am not personally in direct contact with any of
the other
mirrors, even though they all pull from us. Contact between the mirror
admins has AFAIK always been through Tim. We created the primary
mirror to alleviate bandwidth problems at Tim's build farm in the past
when all mirrors pulled direct from him.
Yes, this is a well-designed solution. I thank you for your effort on this
solution. By creating a separate master redirector, we have further
improved our situation.
There is a cache on the redirector, which continuously synchronizes the
packages for the new versions before official release. Here are both
Debian / Ubuntu packages (synchronized from QuickBuild) and RPM packages
(synchronized from François Andriot primary source). At the time of the
release of the new TDE version, the redirector can serve packages to
users directly from this cache before they are available on any mirror.
(4) The primary mirror uses rsync rather than
apt-mirror, and I suspect
the same is true for the other mirrors.
Yes, that's perfectly fine. The official primary server contains not only
deb packages, so using apt-mirror for mirroring the primary server would
make no sense.
If anyone is interested in synchronizing Preliminary Stable Builds or
Preliminary Testing Builds, just let me know and I can also make them
accessible via rsync. However, as I wrote above - these are repositories
that may be updated faster than once a day.
(5) I appreciate heads up on any planned major changes
to mirror
content so I can increase monitoring and if necessary change mirroring
parameters. It might be best to provide such heads up off-list because
...
Changes to the mirror system or content are not currently planned. The
current concept seems to work well enough. Let me say that it was
well-proven at the time of release R14.0.4.
(6) To avoid overloading build farm bandwidth, it is
best not to
announce releases until they are fully mirrored. If users start
pulling unmirrored files they are served from the build farm,
overloading its bandwidth and slowing mirroring, so things get very
slow for everyone.
This is no longer necessary. As I mentioned above, the redirector has a
cache that is filled with new packages before the official release.
Therefore, at the time of release, if packages are not yet available on
official mirrors, they may be served directly from this cache. This makes
the packages available to users quickly and without overloading the line
to the primary server. Already R14.0.4 has successfully used this cache
at the time of release.
As a TDE user myself I would find it convenient but
not critical if
there were fewer differences between PSB and Stable. Ideally they
would be in the same repo pool - like Debian testing and stable.
Yes, you are quite right - a lot of time has elapsed since R14.0.4. It
would definitely be better if we released the maintenance versions more
often. We hope that after releasing R14.0.5 we will again be able to
manage more frequent releases.
--Mike
Cheers
--
Slávek
10:12 p.m.
New subject: [trinity-devel] Mirrors and PSB
Hi Slavek,
Thank you for the comprehensive response. A few points come to mind.
(1) TDE mirrors are not synced "daily".
Due to master server bandwidth limitations, primary sync runs can
take anywhere from a minute to a week.
The primary mirror normally syncs from Tim's master server
continuously subject to a bandwidth limit selected by Tim and
with a short delay between the completion of one sync run
and the start of the next. During major release rollouts
syncing is changed e.g. by reducing delay between sync runs
and by deferring dbg/debuginfo/dev/devel/tar/src files when more
urgent files need to be mirrored.
Secondary mirrors sync from the primary on their own schedules.
Currently some secondaries sync three times per day and some four.
On Sat July 21 2018 10:20:33 Slávek Banko wrote:
On Sunday 24 of June 2018 20:59:46 Mike Bird wrote:
(2) This seems counter-productive. The reason for implementing a
distinct primary mirror was so that each file would only be
transferred once from Tim's master server due to bandwidth
limitations.
(6) To avoid overloading build farm bandwidth, it
is best not to
announce releases until they are fully mirrored. If users start
pulling unmirrored files they are served from the build farm,
overloading its bandwidth and slowing mirroring, so things get very
slow for everyone.
This is no longer necessary. As I mentioned above, the redirector has a
cache that is filled with new packages before the official release.
Therefore, at the time of release, if packages are not yet available on
official mirrors, they may be served directly from this cache. This makes
the packages available to users quickly and without overloading the line
to the primary server. Already R14.0.4 has successfully used this cache at
the time of release.
(3) There were severe slowdowns for several days during R14.0.4 rollout
which appeared to be due to master server bandwidth limitations.
--Mike
11:55 p.m.
New subject: [trinity-devel] Mirrors and PSB
On Sunday 22 of July 2018 00:12:21 Mike Bird wrote:
Yes, releasing the version will always cause a load of bandwidth to the
primary server for several days. However, thanks to the cache on the
redirector, this load can be lower and the packages are accessible to
users earlier.
Note: If I remember correctly, the RPM packages cache on the redirector
have not yet been set at time of release R14.0.4. Therefore, when we
release R14.0.5, I expect the additional benefit of this cache.
Hi Slavek,
Thank you for the comprehensive response. A few points come to mind.
(1) TDE mirrors are not synced "daily".
Due to master server bandwidth limitations, primary sync runs can
take anywhere from a minute to a week.
The primary mirror normally syncs from Tim's master server
continuously subject to a bandwidth limit selected by Tim and
with a short delay between the completion of one sync run
and the start of the next. During major release rollouts
syncing is changed e.g. by reducing delay between sync runs
and by deferring dbg/debuginfo/dev/devel/tar/src files when more
urgent files need to be mirrored.
Secondary mirrors sync from the primary on their own schedules.
Currently some secondaries sync three times per day and some four.
Interesting. I supposed that the synchronization is initialized once a
day, and then it only depends on whether it has finished or that it will
be delayed when previous synchronization is still running.
On Sat July 21 2018 10:20:33 Slávek Banko wrote:
Do not worry, this caching is not counterproductive, just the opposite.
The synchronization of packages into the redirector cache is done before
the packages are published to the official repository == before the
synchronization to the primary mirror starts. This means that
synchronization into redirector cache uses the time when the bandwidth on
the primary server is not used. This is the main trick of this caching -
that's why this caching is very beneficial.
Separate substantial benefit is caching of RPM packages. RPM packages are
being prepared by François on their own repository. From this repository,
the packages are first synchronized into the primary server => load of
bandwidth on the incoming. Then are synchronized to the primary mirror =>
load of bandwidth on the outgoing. This both represents a long time. On
the redirector, packages are cached directly from the François repository
== this does not cause any bandwidth load on the primary server, and the
packages may be available before they are downloaded to the primary
server.
Therefore, I would like to say that caching on the redirector is very
beneficial both for saving bandwidth to the primary server and also for
faster availability of packages to users. Both sides wins :)
On Sunday 24 of June 2018 20:59:46 Mike Bird
wrote:
(2) This seems counter-productive. The reason for implementing a
distinct primary mirror was so that each file would only be
transferred once from Tim's master server due to bandwidth
limitations.
(6) To avoid overloading build farm bandwidth, it
is best not to
announce releases until they are fully mirrored. If users start
pulling unmirrored files they are served from the build farm,
overloading its bandwidth and slowing mirroring, so things get very
slow for everyone.
This is no longer necessary. As I mentioned above, the redirector has
a cache that is filled with new packages before the official release.
Therefore, at the time of release, if packages are not yet available
on official mirrors, they may be served directly from this cache.
This makes the packages available to users quickly and without
overloading the line to the primary server. Already
R14.0.4 has successfully used this cache at the time of
release.
(3) There were severe slowdowns for several days during R14.0.4 rollout
which appeared to be due to master server bandwidth limitations.
--Mike
Cheers
--
Slávek
2104
days inactive
2131
days old
3 comments
2 participants
participants (2)
-
Mike Bird -
Slávek Banko