-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA224
All,
I am delaying the R14.0.0 release by one day in light of the recently discovered problems with our primary mirror, and also so that I have time to clean up the press release.
Sorry for the inconvenience!
Tim
I am delaying the R14.0.0 release by one day in light of the recently discovered problems with our primary mirror, and also so that I have time to clean up the press release.
Sorry for the inconvenience!
Perhaps the inconvenience is the mirror web pages are not updated. Today I looked for the Fedora 21 repo rpm. When I connect to http://ppa.quickbuild.pearsoncomputing.net/ I get redirected to a mirror. That is probably by design but which mirror I am redirected seems random. The web pages do not seem updated. For example, test the links at this mirror:
http://kuiper.mirrorservice.org/sites/trinitydesktop.org/trinity/trinity/rpm...
404
Notice there are no links for R14.
The web site packages page is not updated:
https://www.trinitydesktop.org/releases.php
Nor is the wiki:
https://wiki.trinitydesktop.org/FedoraInstall
I recommend not pushing announcements until all web page and wiki links are validated as functional. :)
Darrell
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA224
I am delaying the R14.0.0 release by one day in light of the recently discovered problems with our primary mirror, and also so that I have time to clean up the press release.
Sorry for the inconvenience!
Perhaps the inconvenience is the mirror web pages are not updated. Today I looked for the Fedora 21 repo rpm. When I connect to http://ppa.quickbuild.pearsoncomputing.net/ I get redirected to a mirror. That is probably by design but which mirror I am redirected seems random. The web pages do not seem updated. For example, test the links at this mirror:
http://kuiper.mirrorservice.org/sites/trinitydesktop.org/trinity/trinity/rpm...
404
Notice there are no links for R14.
R14 RPM packages have not been built/published at this time. As far as I know this is underway but unfortunately I cannot hold up release for one distribution type.
The web site packages page is not updated:
https://www.trinitydesktop.org/releases.php
Nor is the wiki:
All of those pages will be updated when the release is officially announced. :-)
Tim
R14 RPM packages have not been built/published at this time. As far as I know this is underway but unfortunately I cannot hold up release for one distribution type.
Francois shared yesterday that he uploaded Fedora packages. Should there now be a link in the web site and wiki?
Darrell
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA224
R14 RPM packages have not been built/published at this time. As far as I know this is underway but unfortunately I cannot hold up release for one distribution type.
Francois shared yesterday that he uploaded Fedora packages. Should there now be a link in the web site and wiki?
Darrell
Give it a couple weeks to mirror. Right now we are having some issues with our primary mirror (see the threads by Mike Bird on this list), and the financial state of the project cannot support a high bandwidth to upload them faster.
Sorry about the delay; I wish I could improve the speed but external factors impose constraints as always. :-)
Tim
Give it a couple weeks to mirror. Right now we are having some issues with our primary mirror (see the threads by Mike Bird on this list), and the financial state of the project cannot support a high bandwidth to upload them faster.
Weeks?!?
Sorry about the delay; I wish I could improve the speed but external factors impose constraints as always. :-)
I would have thought minimal QA was to ensure all mirrors were ready before pushing the official announcement. Carts and horses and all that.
Darrell
On Tue December 16 2014 17:32:29 Darrell wrote:
I would have thought minimal QA was to ensure all mirrors were ready before pushing the official announcement. Carts and horses and all that.
Given the mirror problems Tim devised a transparent mechanism which provides packages from mirrors where they are available and delivers them direct from the master server when they are not. It works reliably for Debian and Ubuntu packages, albeit at slower speed when packages have to be downloaded from the master server.
AFAIK this mechanism has not yet been tested with RPM-based distros. If they attempt to scan a directory instead of just downloading specific files the redirector may need adjustment.
--Mike
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA224
Give it a couple weeks to mirror. Right now we are having some issues with our primary mirror (see the threads by Mike Bird on this list), and the financial state of the project cannot support a high bandwidth to upload them faster.
Weeks?!?
Donations have dropped off sharply over the past two years (probably due to the lack of new releases); believe it or not I pull in less than 15% of my total *operating* costs (this does not include the cost of the equipment itself). This is a large donation on my part to the project, in addition to the heavy development I have done on TDE over the past 6 years, and I cannot continue to bleed cash on the required scale just to have a faster upload of packages I don't even use. From what I gather this is the sad state of affairs in many open source projects these days; this is all probably related to global economic issues but I cannot be sure.
I am hoping R14 will revitalize the project financially and I can then revisit these decisions, but for now I am constrained by factors outside of my control. TDE will continue to exist (a GIT server is quite cheap), but the future form of this project will be determined by our users in the coming months.
Sorry about the delay; I wish I could improve the speed but external factors impose constraints as always. :-)
I would have thought minimal QA was to ensure all mirrors were ready before pushing the official announcement. Carts and horses and all that.
They were, but the RPM builds were not ready. I made an administrative decision to release R14 at this time as the source was ready and the Debian/Ubuntu builds were ready. I considered that it would do more damage to both halt development for that length of time and spread out the "release" (thus causing loss of visibility) than it would to simply delay the RPM build availability.
While TDE has grown significantly from its early days we are still very small compared to large corporate-sponsored projects like KDE, Gnome, systemd, etc. This means our development process will, by nature, be slower and bumpier than that of the larger projects.
Tim
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA224
Give it a couple weeks to mirror. Right now we are having some issues with our primary mirror (see the threads by Mike Bird on this list), and the financial state of the project cannot support a high bandwidth to upload them faster.
Weeks?!?
Donations have dropped off sharply over the past two years (probably due to the lack of new releases); believe it or not I pull in less than 15% of my total *operating* costs (this does not include the cost of the equipment itself). This is a large donation on my part to the project, in addition to the heavy development I have done on TDE over the past 6 years, and I cannot continue to bleed cash on the required scale just to have a faster upload of packages I don't even use. From what I gather this is the sad state of affairs in many open source projects these days; this is all probably related to global economic issues but I cannot be sure.
I am hoping R14 will revitalize the project financially and I can then revisit these decisions, but for now I am constrained by factors outside of my control. TDE will continue to exist (a GIT server is quite cheap), but the future form of this project will be determined by our users in the coming months.
Sorry about the delay; I wish I could improve the speed but external factors impose constraints as always. :-)
I would have thought minimal QA was to ensure all mirrors were ready before pushing the official announcement. Carts and horses and all that.
They were, but the RPM builds were not ready. I made an administrative decision to release R14 at this time as the source was ready and the Debian/Ubuntu builds were ready. I considered that it would do more damage to both halt development for that length of time and spread out the "release" (thus causing loss of visibility) than it would to simply delay the RPM build availability.
While TDE has grown significantly from its early days we are still very small compared to large corporate-sponsored projects like KDE, Gnome, systemd, etc. This means our development process will, by nature, be slower and bumpier than that of the larger projects.
Tim
On re-reading this message my tone may have not reflected what I was actually thinking.
The RPM (and Slackware, and <insert distro here> packages are a very important part of TDE. I want to thank Francois for providing the RPM builds; much of my ire above is directed at the technical design of the various packaging systems which require a centralized repository to function. For small projects this is not much of an issue; for something the size of TDE we are suddenly looking at a 200GB archive and very high bandwidth costs even with a fairly sophisticated mirroring system in place.
TDE will continue to support various distributions for as long as their respective builders/maintainers are willing to provide packages; the initial transfer speeds to the mirror system will be degraded depending on the project's financial status but this is a relative nit imposed by the unfortunate constraints of the real world.
Tim
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2014/12/17 11:09 PM, Timothy Pearson wrote:
Give it a couple weeks to mirror. Right now we are having some issues with our primary mirror (see the threads by Mike Bird on this list), and the financial state of the project cannot support a high bandwidth to upload them faster.
This is not a "polemic" mail. Just want to raise awareness. I have been trying to test updating from 3.5.13.2 to R14.0.0 over the last 3 days, but the package download speed has been *ridiculously* slow for several big packages (like tdelibs for example), something like 821 B/s or 1109 B/s just to give some numbers. Not to mention the amount of timeouts I received. I understand the financial situation of the project (so the absence of any type of polemic complaint), but we are definitely not doing any good to us with this situation. Potential new users may just turn away when they try to download Trinity. IMO, in the future we must make sure that a package-wise fully updated mirror system is working properly before announcing the next release.
Cheers Michele
On Thursday 18 of December 2014 10:00:35 Michele Calgaro wrote:
On 2014/12/17 11:09 PM, Timothy Pearson wrote:
Give it a couple weeks to mirror. Right now we are having some issues with our primary mirror (see the threads by Mike Bird on this list), and the financial state of the project cannot support a high bandwidth to upload them faster.
This is not a "polemic" mail. Just want to raise awareness. I have been trying to test updating from 3.5.13.2 to R14.0.0 over the last 3 days, but the package download speed has been *ridiculously* slow for several big packages (like tdelibs for example), something like 821 B/s or 1109 B/s just to give some numbers. Not to mention the amount of timeouts I received. I understand the financial situation of the project (so the absence of any type of polemic complaint), but we are definitely not doing any good to us with this situation. Potential new users may just turn away when they try to download Trinity. IMO, in the future we must make sure that a package-wise fully updated mirror system is working properly before announcing the next release.
Cheers Michele
From my perspective: Slow line on the primary server is a fact about which there is no need to discuss. I dare say that this slow line may not be a fatal problem. But we would have had to draw, how to effectively use it.
I see several problems:
1) The more mirror sites that sync from the master server => the line is more overloaded => synchronization is complete in sight.
2) The longer synchronization time to mirrors causes that more users download files from the master server => the line is more and more overloaded => everything is moving towards infinity.
3) Nightly-builds and preliminary-stable-builds PPAs are not on the mirrors => their users cause additional overload of the line.
As we can see now, every additional user deepens this problem, each additional user is frustrated. I believe that we need to prevent any user to access directly to the master server.
I propose to establish an official primary mirror, which will be on enough bandwidth and will be the only one who will access to the master server. All other mirrors would then be synchronized from this primary mirror. This would solve the problem 1), including the consequent problem 2). On this primary mirror should also be mirrored nightly-builds and preliminary-stable-builds, to avoid also the problem 3).
What is your opinion?
I see several problems:
- The more mirror sites that sync from the master server => the line is more
overloaded => synchronization is complete in sight.
- The longer synchronization time to mirrors causes that more users download
files from the master server => the line is more and more overloaded => everything is moving towards infinity.
- Nightly-builds and preliminary-stable-builds PPAs are not on the mirrors =>
their users cause additional overload of the line.
As we can see now, every additional user deepens this problem, each additional user is frustrated. I believe that we need to prevent any user to access directly to the master server.
I propose to establish an official primary mirror, which will be on enough bandwidth and will be the only one who will access to the master server. All other mirrors would then be synchronized from this primary mirror. This would solve the problem 1), including the consequent problem 2). On this primary mirror should also be mirrored nightly-builds and preliminary-stable-builds, to avoid also the problem 3).
What is your opinion?
Can't the mirrors use bittorrent to sync?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA224
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2014/12/17 11:09 PM, Timothy Pearson wrote:
Give it a couple weeks to mirror. Right now we are having some issues with our primary mirror (see the threads by Mike Bird on this list), and the financial state of the project cannot support a high bandwidth to upload them faster.
This is not a "polemic" mail. Just want to raise awareness.
Understood.
I have been trying to test updating from 3.5.13.2 to R14.0.0 over the last 3 days, but the package download speed has been *ridiculously* slow for several big packages (like tdelibs for example), something like 821 B/s or 1109 B/s just to give some numbers. Not to mention the amount of timeouts I received. I understand the financial situation of the project (so the absence of any type of polemic complaint), but we are definitely not doing any good to us with this situation. Potential new users may just turn away when they try to download Trinity. IMO, in the future we must make sure that a package-wise fully updated mirror system is working properly before announcing the next release.
Our primary mirrorservice.org mirror *was* working a couple days before release as far as I could tell. Then it just stopped downloading R14 packages and started re-downloading the CD images--it still hasn't gotten around to downloading the remaining R14 packages. In addition it downloads each file at least twice, sometimes more. As a result of these issues I am looking to replace the primary mirror.
The mirroring architecture is such that our primary mirror downloads directly from the TDE servers, then all secondary mirrors download from the primary. Adding additional mirrors therefore does not increase the load on the limited TDE server uplink.
Tim
On Thursday 18 of December 2014 20:05:09 Timothy Pearson wrote:
The mirroring architecture is such that our primary mirror downloads directly from the TDE servers, then all secondary mirrors download from the primary. Adding additional mirrors therefore does not increase the load on the limited TDE server uplink.
Thank you for the information. This makes my previous mail obsolete :)
On Wednesday 17 of December 2014 02:32:29 Darrell wrote:
Give it a couple weeks to mirror. Right now we are having some issues with our primary mirror (see the threads by Mike Bird on this list), and the financial state of the project cannot support a high bandwidth to upload them faster.
Weeks?!?
Sorry about the delay; I wish I could improve the speed but external factors impose constraints as always. :-)
I would have thought minimal QA was to ensure all mirrors were ready before pushing the official announcement. Carts and horses and all that.
Darrell
I recall that when we release TDE v3.5.13.2, we waited for synchronizing of mirrors for about a month.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA224
On Wednesday 17 of December 2014 02:32:29 Darrell wrote:
Give it a couple weeks to mirror. Right now we are having some issues with our primary mirror (see the threads by Mike Bird on this list),
and
the financial state of the project cannot support a high bandwidth to upload them faster.
Weeks?!?
Sorry about the delay; I wish I could improve the speed but external factors impose constraints as always. :-)
I would have thought minimal QA was to ensure all mirrors were ready before pushing the official announcement. Carts and horses and all that.
Darrell
I recall that when we release TDE v3.5.13.2, we waited for synchronizing of mirrors for about a month.
-- Slávek
Correct; to avoid that for R14 I had changed two things: 1.) I started synchronizing large packages that would not change from the RC to Final (such as the l18n files) weeks in advance of the final R14 builds. 2.) I created a new mirroring redirector that falls back to the master server if a file is not present on the mirror system. This allows us (theoretically) to release even if the mirror synchronization is at, say, 98% complete instead of having to wait for 100% completion.
Unfortunately this still wasn't enough judging from the mailing lists. The issue in this cycle was our primary mirror re-downloading each file at least twice, thereby wasting at least 50% of the TDE project's upload bandwidth. Needless to say I am working on replacing that mirror.
Tim
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 12/19/2014 04:19 AM, Timothy Pearson wrote:
I recall that when we release TDE v3.5.13.2, we waited for synchronizing of mirrors for about a month.
This allows us (theoretically) to release even if the mirror synchronization is at, say, 98% complete instead of having to wait for 100% completion.
TDE is big, so even a missing 10% from the mirror can cause big delays since every user will have to download that 10% from the same server. In hindsight, the policy followed for 3.5.13.2 (waiting for mirror synchronization) is more sensible. We spend more than 2 years in developing R14, so a couple more weeks of delay would not have been a big issue IMO. I guess this is a lesson learnt for the future :-)
Until we resolve the issues with the mirror, would it be sensible to add a note on the main site to advice users against possible temporarily slow download speeds?
Cheers Michele
-
Darrell -
Dr. Nikolaus Klepp -
Michele Calgaro -
Mike Bird -
Slávek Banko -
Timothy Pearson