27 Jul
2018
27 Jul
'18
1:38 p.m.
(sorry if I broke the threading, not having a mail to reply to here)
The primary mirror has roughly 240GB of which 192GB is
currently in use.
This can be increased but of course none of us want to spend more money
on renting data center disk space than necessary. I don't know how much
disk space is available for TDE on the secondary mirrors.
The question is, aren't most of these 192 GB useless by now?
Looking at what we currently have in our mirror dir, it seems there is a
lot of stuff there that could be gotten rid of. To me it really doesn't
make much sense to keep e.g. the 3.5.12-release from 2010 available on
all mirrors. Or a Maverick-iso from 2010. Other projects, e.g. CentOS,
solve this by only keeping the currently maintained versions on the main
mirror network, and at some point moving the things that are no longer
maintained to an archive site. This has a lot of advantages: That
content usually gets _extremely_ few requests, so it uses up far more
traffic to keep the mirrors updated than the mirrors will ever receive
requests for it. It also reduces the space usage on the mirrors. It
saves everyone space and traffic. And the few requests for archive
content can be handled by a server with very low bandwidth.
I am not personally in direct contact with any of the
other mirrors, even
though they all pull from us. Contact between the mirror admins has AFAIK
always been through Tim.
The primary mirror uses rsync rather than apt-mirror, and I suspect the
same is true for the other mirrors.
We're running one of these mirrors at ftp.fau.de. Both your assumptions
are correct for us.
(From Sláveks mail)
If anyone is interested in synchronizing Preliminary Stable Builds or
Preliminary Testing Builds, just let me know and I can also make them
accessible via rsync
We would gladly mirror this _if_ there is a demand for it, i.e. we don't
spend more traffic syncing the mirror than the mirror will ever see
requests.
As a TDE user myself I would find it convenient but
not critical if
there were fewer differences between PSB and Stable. Ideally they
would be in the same repo pool - like Debian testing and stable.
As a TDE user myself, I would very much support the idea of putting the
PSBs, signed with a proper trinitydesktop.org and not a personal key,
onto the official mirror network. Rename them into "testing" builds so
it's clear what they are (that name is a lot more self-explaining than
"PSB").
Regards,
--
Michael Meier, FTP-Admin
Friedrich-Alexander-Universitaet Erlangen-Nuernberg
Regionales Rechenzentrum Erlangen
Martensstrasse 1, 91058 Erlangen, Germany
Tel.: +49 9131 85-28973, Fax: +49 9131 302941
rrze-ftp-admins(a)fau.de
blogs.fau.de/ftp/
29 Jul
29 Jul
9:09 a.m.
New subject: [trinity-devel] Re: Mirrors and PSB
On Friday 27 of July 2018 15:38:59 Michael Meier wrote:
Yes, some data are seldom used, some suitable for removal (for example,
git-images, old development ISO files in trinity/cdimages/ubuntu).
To postpone old packages to the archive, I see two problems here:
1) Repositories, although for old versions, are connected to QuickBuild
repositories. Here is a good reason to keep this state. Indeed,
repository for v3.5.12 have a very reduced number of distributions that
have been preserved.
2) An appropriate location for a possible archive would probably be the
server on the primary site. But it would mean that all the downloads from
the archive would lead to overloading the bandwidth to the primary
server. But that's something we're trying to prevent - just by publishing
content on mirrors. Only if the access speed to the archive was
artificially reduced, but that does not seem like a good idea. Such an
archive would be counterproductive.
As I mentioned above, all official repositories (signed by the official
Trinity key) are managed on QuickBuild. As I explained in previous
emails, Preliminary Stable Builds are created independently of
QuickBuild - independent builders (used pbuilder on my builders),
independent repository maintenance (used reprepro on my server).
That's why PSB repositories simply can not be signed with an official
Trinity key that is integrated with QuickBuild. I could create some
general GPG key for signing repository instead of my personal, but it
would still be a key other than the official Trinity key. So it seems
like a futile change.
Regarding the propose to rename from 'stable' to 'testing' builds, this
change is not possible. Existing Preliminary Stable Builds are built on
a 'stable' branch (now r14.0.x) == this is preliminary packages for the
next maintenance relase - therefore 'stable' in the repository name.
Additionally, the second repository named Preliminary Testing Builds has
already been prepared. This second repository is built on a 'master'
branch == this is preliminary packages for the next major release, which
rightfully deserves the naming of 'testing'. The official announcement of
this 'testing' repository can be expected soon.
(sorry if I broke the threading, not having a mail to
reply to here)
> The primary mirror has roughly 240GB of which 192GB is currently in
use.
> This can be increased but of course none of
us want to spend more
> money on renting data center disk space than necessary. I don't know
how
much disk
space is available for TDE on the secondary mirrors.
The question is, aren't most of these 192 GB useless by now?
Looking at what we currently have in our mirror dir, it seems there is a
lot of stuff there that could be gotten rid of. To me it really doesn't
make much sense to keep e.g. the 3.5.12-release from 2010 available on
all mirrors. Or a Maverick-iso from 2010. Other projects, e.g. CentOS,
solve this by only keeping the currently maintained versions on the main
mirror network, and at some point moving the things that are no longer
maintained to an archive site. This has a lot of advantages: That
content usually gets _extremely_ few requests, so it uses up far more
traffic to keep the mirrors updated than the mirrors will ever receive
requests for it. It also reduces the space usage on the mirrors. It
saves everyone space and traffic. And the few requests for archive
content can be handled by a server with very low bandwidth.
> I am not personally in direct contact with any of
the other mirrors,
even
> though they all pull from us. Contact
between the mirror admins
has
> AFAIK always been through Tim.
> The primary mirror uses rsync rather than apt-mirror, and I suspect
the
same is
true for the other mirrors.
We're running one of these mirrors at ftp.fau.de. Both your assumptions
are correct for us.
(From Sláveks mail)
If anyone is interested in synchronizing Preliminary Stable Builds or
Preliminary Testing Builds, just let me know and I can also make them
accessible via rsync
We would gladly mirror this _if_ there is a demand for it, i.e. we don't
spend more traffic syncing the mirror than the mirror will ever see
requests.
As a TDE user myself I would find it convenient
but not critical if
there were fewer differences between PSB and Stable. Ideally they
would be in the same repo pool - like Debian testing and stable.
As a TDE user myself, I would very much support the idea of putting the
PSBs, signed with a proper trinitydesktop.org and not a personal key,
onto the official mirror network. Rename them into "testing" builds so
it's clear what they are (that name is a lot more self-explaining than
"PSB").
Regards,
Thank you for good ideas. I hope your proposed clean up of old unnecessary
things will be successful. Tim is the only one who can do this. I hope
you understand why some of the proposals can not be done.
Thank you for your efforts in supporting the project.
Cheers
--
Slávek
10:15 a.m.
New subject: [trinity-devel] Re: Mirrors and PSB
On Sun July 29 2018 02:09:22 Slávek Banko wrote:
As I mentioned above, all official repositories
(signed by the official
Trinity key) are managed on QuickBuild. As I explained in previous
emails, Preliminary Stable Builds are created independently of
QuickBuild - independent builders (used pbuilder on my builders),
independent repository maintenance (used reprepro on my server).
That's why PSB repositories simply can not be signed with an official
Trinity key that is integrated with QuickBuild. I could create some
general GPG key for signing repository instead of my personal, but it
would still be a key other than the official Trinity key. So it seems
like a futile change.
Regarding the propose to rename from 'stable' to 'testing' builds, this
change is not possible. Existing Preliminary Stable Builds are built on
a 'stable' branch (now r14.0.x) == this is preliminary packages for the
next maintenance relase - therefore 'stable' in the repository name.
Additionally, the second repository named Preliminary Testing Builds has
already been prepared. This second repository is built on a 'master'
branch == this is preliminary packages for the next major release, which
rightfully deserves the naming of 'testing'. The official announcement of
this 'testing' repository can be expected soon.
Hi Slávek,
I am not yet understanding why we have so much duplication. Why is there
not a "testing" release in the main repository instead of a having a main
repo and a DEB PSB repo and a RPM PSB repo in different locations and
using different tooling? (And soon in addition a new DEB testing repo?)
I understand that PSB updates every two hours whereas the primary mirror
currently checks for updates three hours after the completion of the
previous run, but I could change the three hour delay to a ten minute delay
without adversely impacting Tim's bandwidth. What is the average volume
(GB) of DEB PSB updates published per day?
The standard single Debian repository structure is well understood by all
concerned. It reduces the number of downloads and upgrades needed for
new minor releases and generally provides a smoother and more efficient
transition from testing to release. Indeed Trinity successfully used a
single repo incorporating trinity-nightly-builds for many years.
--Mike
5:33 p.m.
New subject: [trinity-devel] Re: Mirrors and PSB
On Sunday 29 of July 2018 12:15:07 Mike Bird wrote:
Hi Slávek,
I am not yet understanding why we have so much duplication. Why is
there not a "testing" release in the main repository instead of a
having a main repo and a DEB PSB repo and a RPM PSB repo in different
locations and using different tooling? (And soon in addition a new DEB
testing repo?)
I understand that PSB updates every two hours whereas the primary
mirror currently checks for updates three hours after the completion of
the previous run, but I could change the three hour delay to a ten
minute delay without adversely impacting Tim's bandwidth. What is the
average volume (GB) of DEB PSB updates published per day?
The standard single Debian repository structure is well understood by
all concerned. It reduces the number of downloads and upgrades needed
for new minor releases and generally provides a smoother and more
efficient transition from testing to release. Indeed Trinity
successfully used a single repo incorporating trinity-nightly-builds
for many years.
Hi Mike,
I'll start with the simplest part - RPM packages:
For RPM packages, there is nothing that QuickBuild could offer because it
is only for DEB packages. All RPM packages are created by François, where
the creation of source packages, building and publishing to the
repository takes place completely outside of the primary server - on
François server. From the François server, RPM repositories are first
synchronized to the primary TDE server => it takes some time because the
bandwidth to the primary server. Subsequently, RPM repositories are
synchronized from primary TDE server to the primary mirror => it takes
some time, because the bandwidth from the primary server. When this is
done, RPM packages are synchronized from primary mirror to other mirrors.
With a major update, such as a new TDE release, it usually takes several
days or even weeks. Note: The current RPM repository size is 43 GiB.
On a VPS that has the role of the main redirector, the RPM repository is
synchronized directly from the François server to the local cache on the
redirector. Due to the fact, that the redirector is on wide bandwidth
(300 Mbps) and the François server also has a good bandwidth,
synchronization takes place quickly - it usually takes only a few hours.
Once the RPM packages are available in the cache on the redirector, they
can be served to users directly from this cache, regardless of whether
synchronization from the François server to the primary TDE server has
already been completed, regardless of whether all subsequent
synchronizations are completed.
Advantages?
+ packages are accessible to users very soon after publishing by François
+ the time during which the RPM repository is inconsistent is very short
+ users' access to packages does not cause load of bandwidth to the primary
server
+ once some packages begin to be available at least on the primary mirror, the
redirector will then direct users' access to the mirror
Disadvantages?
+ on the redirector is required space for whole RPM repository
+ once mirrors synchronization is complete, cache of RPM packages on
redirector remain unused (until the next update by François)
Any comments, questions or suggestions for this part?
--
Slávek
30 Jul
30 Jul
1:07 a.m.
New subject: [trinity-devel] Re: Mirrors and PSB
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
On 2018/07/30 02:33 AM, Slávek Banko wrote:
From the François server, RPM repositories are first
synchronized to the primary TDE server => it takes some time
because the bandwidth to the primary server. Subsequently, RPM repositories are
synchronized from primary TDE
server to the primary mirror
Why don't we make the primary mirror the main provide of user content? Yes, it would
not be an exact mirror of the
primary server, but more a "collector" of packages from various server (see
attached primitive sketch).
In this way, the primary server would become a provider to the primary mirror,
Francois's repo could become a provider
in the same way (removing the need of copy from Francois -> main server -> primary
mirror), Slavek's PSB/PTB can
become a provider in the same way.
I don't see any problem with PSB/PTB not being signed with official key, since many
users are already using them and
Francois' RPM are also not signed.
This would speed up availability of packages and reduce load on primary server.
It also offers a fallback solution should the primary server disappear in the long term,
since packages would still be
available from Francois and Slavek repos.
Cheers
Michele
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEjhl1z5vbYB3YbFTiKnW3yore1c8FAlteZLkACgkQKnW3yore
1c9C9xAAjF4cZsAFBH4b99NDfaMWZw2hwBGjsrUr/lm/5Ebm/exjWuQgVKbcapHl
lskvlaph/sT43TR5jG4U5N7wi/iqedvQc7dsYzxNdejIgceL5m/uMbhaOJ/LZ17E
pMEfPPyTJTZyTmDDzEtTDKxHaQhpsrbQUB1t+RfURS+Fp9pNNYgYVj+c3HnvmGXj
lbe6Hgrp+MUkPHegvnfiTQBdk850KjCf4KsFJ+CyicvlLlekXIlQFY0fKQ1c/BUJ
sDuctjLXtTUfz7dzq6KzzjMjlzI7sYz3MnKUE5UumXTwKVAiUBtlaDlJX8ykrGbl
L4ZzsJr8Wu/iiH1Q+uN5tJ/5Jw8kuUlzzOLjAnxiHmgXUzY7W+Mut0xfLUasIZ3Y
w7wNhlLFmddVQ3CduntQWNTq5CnkwDzVgqkk0YkER/cSg3K7Izgif8XCRz36N74l
p1SfYJJZVTa9AUSJXejb+/mntmHiA2LA8IGVWm6GRaVTpPOtRplGlkn7oyNrYH5p
faSqMP0EUqXTNxu0pr6aQjft43b9TP8vD8tCNndYzrsVcrEdUA+EgDzEllKmURW1
hk+grkY3S7/4J7lfDP6pv9mb7SFonr8Yo7dKo9fvGTogATsWLbyVIOZNkGwDwqiM
QCKA+w6BPphyUdhmWTar0FFkSIVkoTxQHOt+OPAwqUD910FljI8=
=8Ucl
-----END PGP SIGNATURE-----
29 Jul
29 Jul
6:58 p.m.
New subject: [trinity-devel] Re: Mirrors and PSB
On Sunday 29 of July 2018 12:15:07 Mike Bird wrote:
On Sun July 29 2018 02:09:22 Slávek Banko wrote:
Hi Mike,
do you remember when I started preparing the packages, which then resulted in
the maintenance v3.5.13.x branch? Small reminder:
http://trinity-users.pearsoncomputing.net/?0::2776
Do you remember my 'axis' repository on QuickBuild? This repository met your
proposal - it was on the primary server - on QuickBuild. However, placement
of this repository brought a combination of all the disadvantages that it
could have:
+ although the repository is on the primary server, but under my account on
QuickBuild == repository is not signed by the official TDE key, but the GPG
key corresponding to my QuickBuild account
+ only official repositories are published on mirrrrors == all access to
the 'axis' repository causes total bandwidth load on the primary server - all
users constantly complained about very slow downloading of packages - if the
master server was unavailable, the repository were not accessible at all
+ on QuickBuild at that time often were situations where builders were stuck
and repositories were inconsistent - users could not download and install
packages or broke their machines
+ adding support of new Debian / Ubuntu distributions on QuickBuild is always
a challenging operation - requires local mirror of distribution == a lot of
space, therefore adding support for new distributions was slower
Creating a completely independent Preliminary Stable Builds repository (both
building and repository management) was therefore a significant benefit for
the users but also for the release of new TDE releases.
Some time ago, I and Tim agreed to place Preliminary Stable Builds repository
directly to the redirector. As a result, this independent repository gained
more official status ... and at the same time raised the questions that we
are currently discussing :)
Because I do not want the mail to be too long, now I will finish this part and
I will continue with the next mail.
--
Slávek
As I mentioned above, all official repositories
(signed by the official
Trinity key) are managed on QuickBuild. As I explained in previous
emails, Preliminary Stable Builds are created independently of
QuickBuild - independent builders (used pbuilder on my builders),
independent repository maintenance (used reprepro on my server).
That's why PSB repositories simply can not be signed with an official
Trinity key that is integrated with QuickBuild. I could create some
general GPG key for signing repository instead of my personal, but it
would still be a key other than the official Trinity key. So it seems
like a futile change.
Regarding the propose to rename from 'stable' to 'testing' builds, this
change is not possible. Existing Preliminary Stable Builds are built on
a 'stable' branch (now r14.0.x) == this is preliminary packages for the
next maintenance relase - therefore 'stable' in the repository name.
Additionally, the second repository named Preliminary Testing Builds has
already been prepared. This second repository is built on a 'master'
branch == this is preliminary packages for the next major release, which
rightfully deserves the naming of 'testing'. The official announcement of
this 'testing' repository can be expected soon.
Hi Slávek,
I am not yet understanding why we have so much duplication. Why is there
not a "testing" release in the main repository instead of a having a main
repo and a DEB PSB repo and a RPM PSB repo in different locations and
using different tooling? (And soon in addition a new DEB testing repo?)
I understand that PSB updates every two hours whereas the primary mirror
currently checks for updates three hours after the completion of the
previous run, but I could change the three hour delay to a ten minute delay
without adversely impacting Tim's bandwidth. What is the average volume
(GB) of DEB PSB updates published per day?
The standard single Debian repository structure is well understood by all
concerned. It reduces the number of downloads and upgrades needed for
new minor releases and generally provides a smoother and more efficient
transition from testing to release. Indeed Trinity successfully used a
single repo incorporating trinity-nightly-builds for many years.
--Mike
30 Jul
30 Jul
2:18 a.m.
New subject: [trinity-devel] Re: Mirrors and PSB
On Sunday 29 of July 2018 12:15:07 Mike Bird wrote:
On Sun July 29 2018 02:09:22 Slávek Banko wrote:
Hi Mike,
you mention "Indeed Trinity successfully used a single repo incorporating
trinity-nightly-builds for many years." But here is one basic mistake.
The repository trinity-nightly-builds is one of the separate repositories
on QuickBuild. At the time of publishing the new release, the contents of
this repository is on QuickBuild copied into the official repository. And
this is the moment when synchronization of complete content to the mirror
begins. There is no "It reduces the number of downloads and upgrades
needed for new minor releases and generally provides a smoother and more
efficient transition from testing to release." It just does not work in
that way.
To make the situation even more complicated, you know that I also have the
repositories for preliminary stable packages that are on QuickBuild.
However, these repositories is not normally used by any user just for
reasons that were with the 'axis' repository.
These repositories on QuickBuild I mainly use to prepare packages that
will then be published to the official repositories as a new release.
Into these repositories, I'm uploading identical source packages as I
uploaded into Preliminary Stable Builds repository. But on QuickBuild
packages must be built again because into QuickBuild is not possible to
upload binary packages, only source packages.
It is just these repositories that contain the final packages for new TDE
release before they are published into the official repository. And just
these packages are downloaded into the cache on the redirector already
during the build == while the bandwidth to the primary server is not
loaded! And that is the main trick why this cache is so important when
releasing a new version. While to the primary mirror synchronization
starts at the moment of publishing packages, these packages are already
available on the redirector's cache before that time. Simple and very
effective.
Since this synchronization of packages from the primary server to the
redirector cache is for deb packages, the advantate is to use apt-mirror
for this purpose. Apt-mirror works in a way that repositories remain
consistent during the update. This is why I use apt-mirror for DEB
packages, while for rpm packages I use rsync.
Here is a small example of the trick. The current stable version is
R14.0.4. Therefore, listing the tdelibs folder from apt repository
displays packages for R14.0.4:
http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian/pool/ma…
For example DSC file for Debian 8 (Jessie) is as follows:
http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian/pool/ma…
Since the builds of R14.0.5 final packages are currently underway and
these are continuously synchronized to the cache on the redirector, it is
now already possible to get files for the future R14.0.5 version. For
example DSC file for Debian 8 (Jessie) is as follows:
http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian/pool/ma…
Once the packages will be published to the official repository, the folder
listing starts showing new files, the meta-information repositories will
refer to new files, and these files will be ready in the cache on the
redirector. Simply because they are there in advance.
Is it now clear what I mean? Is it now clear why the cache on the
redirector is very useful? And it is clear that keeping Preliminary
Stable Builds independent of QuickBuild is no duplication because the
same duplication is also going on at QuickBuild?
Cheers
--
Slávek
As I mentioned above, all official repositories
(signed by the
official Trinity key) are managed on QuickBuild. As I explained in
previous emails, Preliminary Stable Builds are created independently
of QuickBuild - independent builders (used pbuilder on my builders),
independent repository maintenance (used reprepro on my server).
That's why PSB repositories simply can not be signed with an official
Trinity key that is integrated with QuickBuild. I could create some
general GPG key for signing repository instead of my personal, but it
would still be a key other than the official Trinity key. So it seems
like a futile change.
Regarding the propose to rename from 'stable' to 'testing' builds,
this change is not possible. Existing Preliminary Stable Builds are
built on a 'stable' branch (now r14.0.x) == this is preliminary
packages for the next maintenance relase - therefore 'stable' in the
repository name. Additionally, the second repository named
Preliminary Testing Builds has already been prepared. This second
repository is built on a 'master' branch == this is preliminary
packages for the next major release, which rightfully deserves the
naming of 'testing'. The official announcement of this 'testing'
repository can be expected soon.
Hi Slávek,
I am not yet understanding why we have so much duplication. Why is
there not a "testing" release in the main repository instead of a
having a main repo and a DEB PSB repo and a RPM PSB repo in different
locations and using different tooling? (And soon in addition a new DEB
testing repo?)
I understand that PSB updates every two hours whereas the primary
mirror currently checks for updates three hours after the completion of
the previous run, but I could change the three hour delay to a ten
minute delay without adversely impacting Tim's bandwidth. What is the
average volume (GB) of DEB PSB updates published per day?
The standard single Debian repository structure is well understood by
all concerned. It reduces the number of downloads and upgrades needed
for new minor releases and generally provides a smoother and more
efficient transition from testing to release. Indeed Trinity
successfully used a single repo incorporating trinity-nightly-builds
for many years.
--Mike
17 Aug
17 Aug
7:29 a.m.
New subject: [trinity-devel] Re: Mirrors and PSB
On Monday 30 of July 2018 04:18:03 Slávek Banko wrote:
On Sunday 29 of July 2018 12:15:07 Mike Bird wrote:
Hi Mike,
by the way, right now you can check the cache's advantage on the
redirector. The packages were posted to the official repository a few
hours ago. Mirror synchronization has not yet begun, but users can
already install R14.0.5 from the official repository without causing load
of bandwidth to the primary server.
You will notice - the newly created folder, on the primary mirror folder
does not yet exist:
http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian/pool/ma…
http://tde-mirror.yosemite.net/trinity/trinity-r14.0.0/debian/pool/main/a/a…
...but the redirector gives to the users HTTP code 200 == download file
directly from the cache on the redirector:
$ curl -I
http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian/pool/ma…
HTTP/1.1 200 OK
Date: Fri, 17 Aug 2018 07:18:42 GMT
Server: Apache
Content-Location:
serve/trinity-r14.0.0/debian/pool/main/a/akode/akode_14.0.5.orig.tar.xz
Last-Modified: Sat, 28 Jul 2018 07:46:50 GMT
ETag: "5720a6e572e80"
Content-Length: 166928
That is why I said that new versions are immediately available to users -
thanks to cache == there is no reason to postpone official announcements
for a long time.
How do you like it?
Cheers :)
--
Slávek
On Sun July 29 2018 02:09:22 Slávek Banko wrote:
Hi Mike,
you mention "Indeed Trinity successfully used a single repo
incorporating trinity-nightly-builds for many years." But here is one
basic mistake. The repository trinity-nightly-builds is one of the
separate repositories on QuickBuild. At the time of publishing the new
release, the contents of this repository is on QuickBuild copied into
the official repository. And this is the moment when synchronization of
complete content to the mirror begins. There is no "It reduces the
number of downloads and upgrades needed for new minor releases and
generally provides a smoother and more efficient transition from
testing to release." It just does not work in that way.
To make the situation even more complicated, you know that I also have
the repositories for preliminary stable packages that are on
QuickBuild. However, these repositories is not normally used by any
user just for reasons that were with the 'axis' repository.
These repositories on QuickBuild I mainly use to prepare packages that
will then be published to the official repositories as a new release.
Into these repositories, I'm uploading identical source packages as I
uploaded into Preliminary Stable Builds repository. But on QuickBuild
packages must be built again because into QuickBuild is not possible to
upload binary packages, only source packages.
It is just these repositories that contain the final packages for new
TDE release before they are published into the official repository. And
just these packages are downloaded into the cache on the redirector
already during the build == while the bandwidth to the primary server
is not loaded! And that is the main trick why this cache is so
important when releasing a new version. While to the primary mirror
synchronization starts at the moment of publishing packages, these
packages are already available on the redirector's cache before that
time. Simple and very effective.
Since this synchronization of packages from the primary server to the
redirector cache is for deb packages, the advantate is to use
apt-mirror for this purpose. Apt-mirror works in a way that
repositories remain consistent during the update. This is why I use
apt-mirror for DEB packages, while for rpm packages I use rsync.
Here is a small example of the trick. The current stable version is
R14.0.4. Therefore, listing the tdelibs folder from apt repository
displays packages for R14.0.4:
http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian/poo
l/main/t/tdelibs-trinity/
For example DSC file for Debian 8 (Jessie) is as follows:
http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian/poo
l/main/t/tdelibs-trinity/tdelibs-trinity_14.0.4-0debian8.0.0+0.dsc
Since the builds of R14.0.5 final packages are currently underway and
these are continuously synchronized to the cache on the redirector, it
is now already possible to get files for the future R14.0.5 version.
For example DSC file for Debian 8 (Jessie) is as follows:
http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian/poo
l/main/t/tdelibs-trinity/tdelibs-trinity_14.0.5-0debian8.0.0+0.dsc
Once the packages will be published to the official repository, the
folder listing starts showing new files, the meta-information
repositories will refer to new files, and these files will be ready in
the cache on the redirector. Simply because they are there in advance.
Is it now clear what I mean? Is it now clear why the cache on the
redirector is very useful? And it is clear that keeping Preliminary
Stable Builds independent of QuickBuild is no duplication because the
same duplication is also going on at QuickBuild?
Cheers As I mentioned above, all official repositories
(signed by the
official Trinity key) are managed on QuickBuild. As I explained in
previous emails, Preliminary Stable Builds are created
independently of QuickBuild - independent builders (used pbuilder
on my builders), independent repository maintenance (used reprepro
on my server).
That's why PSB repositories simply can not be signed with an
official Trinity key that is integrated with QuickBuild. I could
create some general GPG key for signing repository instead of my
personal, but it would still be a key other than the official
Trinity key. So it seems like a futile change.
Regarding the propose to rename from 'stable' to 'testing' builds,
this change is not possible. Existing Preliminary Stable Builds are
built on a 'stable' branch (now r14.0.x) == this is preliminary
packages for the next maintenance relase - therefore 'stable' in
the repository name. Additionally, the second repository named
Preliminary Testing Builds has already been prepared. This second
repository is built on a 'master' branch == this is preliminary
packages for the next major release, which rightfully deserves the
naming of 'testing'. The official announcement of this 'testing'
repository can be expected soon.
Hi Slávek,
I am not yet understanding why we have so much duplication. Why is
there not a "testing" release in the main repository instead of a
having a main repo and a DEB PSB repo and a RPM PSB repo in different
locations and using different tooling? (And soon in addition a new
DEB testing repo?)
I understand that PSB updates every two hours whereas the primary
mirror currently checks for updates three hours after the completion
of the previous run, but I could change the three hour delay to a ten
minute delay without adversely impacting Tim's bandwidth. What is
the average volume (GB) of DEB PSB updates published per day?
The standard single Debian repository structure is well understood by
all concerned. It reduces the number of downloads and upgrades
needed for new minor releases and generally provides a smoother and
more efficient transition from testing to release. Indeed Trinity
successfully used a single repo incorporating trinity-nightly-builds
for many years.
--Mike
9:57 a.m.
New subject: [trinity-devel] Re: Mirrors and PSB
On Fri August 17 2018 00:29:21 Slávek Banko wrote:
The packages were posted to the official repository a
few
hours ago. Mirror synchronization has not yet begun ...
How do you like it?
No communication with mirror operators is bad. The current
run in progess is configured for a trickle of daily updates
instead of optimally handling a release.
Still hoping for a sensible Debian-like unified repository, at
least for the Debian/Ubuntu parts, instead of all this copying.
--Mike
10:16 a.m.
New subject: [trinity-devel] Re: Mirrors and PSB
On Fri August 17 2018 00:29:21 Slávek Banko wrote:
You will notice - the newly created folder, on the
primary mirror folder
does not yet exist:
You are apparently unaware of the use of rsync --delay-updates, together
with hiding of partials, to provide a consistent view for our downstreams
and users.
Sadly there was no heads up to mirror operators so the roll out of this
release is proceeding slowly in daily update mode, further limited not
by our gigabit bandwidth but the 2Mbps available from the TDE server.
--Mike
11:18 a.m.
New subject: [trinity-devel] Re: Mirrors and PSB
Dne pá 17. srpna 2018 Mike Bird napsal(a):
On Fri August 17 2018 00:29:21 Slávek Banko wrote:
Oh, of course, I did not realize that you are using --delay-updates for
synchronization.
Yes, I know the main problem is narrow bandwith to the primary TDE server.
That's exactly why a cache on VPS was created - to make the release of
new releases significantly faster.
Cheers
--
Slávek
You will notice - the newly created folder, on
the primary mirror
folder does not yet exist:
You are apparently unaware of the use of rsync --delay-updates,
together with hiding of partials, to provide a consistent view for our
downstreams and users.
Sadly there was no heads up to mirror operators so the roll out of this
release is proceeding slowly in daily update mode, further limited not
by our gigabit bandwidth but the 2Mbps available from the TDE server.
--Mike
2076
days inactive
2097
days old
10 comments
4 participants
participants (4)
-
Michael Meier -
Michele Calgaro -
Mike Bird -
Slávek Banko