All,
The QuickBuild build farm was taken offline this morning and will remain offline indefinitely. While I realize this pushes R14 release further out, the cold hard reality is that donations have fallen so much, along with the bad economy impacting my own finances, that I cannot afford to keep the build farm running at this time. Finances were already strained with the Web server upgrade, as the new software required newer hardware to keep performance at a reasonable level.
My intention was to keep the build farm running until R14 was released, then see if any donations came in to cover some of the costs. However, this morning the main chiller was destroyed due to sudden power brownouts/surges, forcing the build farm offline. As I do not have the funds to replace this chiller, and with the onset of spring/warmer temperatures here, I cannot restart the build farm at this time.
I will do my best to bring one builder online for each architecture in the next few days, but this severe drop in build power will obviously cause full archive rebuilds to take several months at minimum. As a result we need to start looking at dropping support for older Ubuntu versions, though I did not want to take this step as I know some larger deployments depend on this support.
All other TDE services are not affected at this time.
Sorry for the bad news!
Timothy Pearson Trinity Desktop Project
All,
The QuickBuild build farm was taken offline this morning and will remain offline indefinitely. While I realize this pushes R14 release further out, the cold hard reality is that donations have fallen so much, along with the bad economy impacting my own finances, that I cannot afford to keep the build farm running at this time. Finances were already strained with the Web server upgrade, as the new software required newer hardware to keep performance at a reasonable level.
My intention was to keep the build farm running until R14 was released, then see if any donations came in to cover some of the costs. However, this morning the main chiller was destroyed due to sudden power brownouts/surges, forcing the build farm offline. As I do not have the funds to replace this chiller, and with the onset of spring/warmer temperatures here, I cannot restart the build farm at this time.
I will do my best to bring one builder online for each architecture in the next few days, but this severe drop in build power will obviously cause full archive rebuilds to take several months at minimum. As a result we need to start looking at dropping support for older Ubuntu versions, though I did not want to take this step as I know some larger deployments depend on this support.
All other TDE services are not affected at this time.
Sorry for the bad news!
Timothy Pearson Trinity Desktop Project
This little donation will help! Who's next?
-Alexandre
My intention was to keep the build farm running until R14 was
released, then see if any donations came in to cover some of the costs. However, this morning the main chiller was destroyed due to sudden power brownouts/surges, forcing the build farm offline. As I do not have the funds to replace this chiller, and with the onset of spring/warmer temperatures here, I cannot restart the build farm at this time.<
How much money is needed? I am admittedly spending almost all of my money on political expenses right now and may be too broke to do anything until May or June.
Could we make a formal appeal for cash for expenses for getting 14.0.0 out the door, with a specific target, if it's larger than what this list alone could be hoped to raise? It would be very good to at least get the older deployments support for 14 before dropping some of them off.
Do we know which releases are more commonly deployed organizationally? I would assume the LTS releases, but such obvious assumptions prove incorrect all the time -- one sufficiently large deployment using, say, Meerkat, could make that a higher priority for us than Lucid. Do we have statistics on which versions get hit up for updates most frequently? Is there a better measure than that that we might use, given that many deployments might not update frequently at all?
James Gholston
Tim , Robert,
Can we leverage OpenSUSEs OBS project for building trinity? On Mar 10, 2014 7:35 PM, "James Gholston" jamesg@dimensionality.com wrote:
My intention was to keep the build farm running until R14 was released,
then see if any donations came in to cover some of the costs. However, this morning the main chiller was destroyed due to sudden power brownouts/surges, forcing the build farm offline. As I do not have the funds to replace this chiller, and with the onset of spring/warmer temperatures here, I cannot restart the build farm at this time.<
How much money is needed? I am admittedly spending almost all of my money on political expenses right now and may be too broke to do anything until May or June.
Could we make a formal appeal for cash for expenses for getting 14.0.0 out the door, with a specific target, if it's larger than what this list alone could be hoped to raise? It would be very good to at least get the older deployments support for 14 before dropping some of them off.
Do we know which releases are more commonly deployed organizationally? I would assume the LTS releases, but such obvious assumptions prove incorrect all the time -- one sufficiently large deployment using, say, Meerkat, could make that a higher priority for us than Lucid. Do we have statistics on which versions get hit up for updates most frequently? Is there a better measure than that that we might use, given that many deployments might not update frequently at all?
James Gholston
To unsubscribe, e-mail: trinity-devel-unsubscribe@ lists.pearsoncomputing.net For additional commands, e-mail: trinity-devel-help@lists. pearsoncomputing.net Read list messages on the web archive: http://trinity-devel. pearsoncomputing.net/ Please remember not to top-post: http://trinity. pearsoncomputing.net/mailing_lists/#top-posting
Tim , Robert,
Can we leverage OpenSUSEs OBS project for building trinity?
TDE would overwhelm any public build system I am aware of. During a full archive rebuild we typically keep 80 Opteron cores busy 24/7 for several weeks (!), while consuming over 150GB in disk space. Launchpad couldn't even handle TDE; it was taking way too long to build on the public resources.
Tim
Hey, been a long time since I've emailed this list. Life...
On 10 March 2014 20:06, Calvin Morrison mutantturkey@gmail.com wrote:
Tim , Robert,
Can we leverage OpenSUSEs OBS project for building trinity?
We can, but their build system is already overwhelmed with tons of projects, and I'm pretty sure I don't want to add the strain of trinity for now. Also, they don't integrate with git, which may make things difficult.
I would definitely recommend figuring out which releases should no longer be supported - I was looking at the trinitydesktop.org website and there are a lot of old releases that we still seem to have...
Tim - would cloud building be an option? (like those $0.005/hr servers that can be brought up/down on demand?). Just a random thought on that. Could be useful to leverage if QuickBuild could support it.
Hey, been a long time since I've emailed this list. Life...
On 10 March 2014 20:06, Calvin Morrison mutantturkey@gmail.com wrote:
Tim , Robert,
Can we leverage OpenSUSEs OBS project for building trinity?
We can, but their build system is already overwhelmed with tons of projects, and I'm pretty sure I don't want to add the strain of trinity for now. Also, they don't integrate with git, which may make things difficult.
I would definitely recommend figuring out which releases should no longer be supported - I was looking at the trinitydesktop.org website and there are a lot of old releases that we still seem to have...
Tim - would cloud building be an option? (like those $0.005/hr servers that can be brought up/down on demand?). Just a random thought on that. Could be useful to leverage if QuickBuild could support it.
The main problem with this is bandwidth to the build control server (a.k.a. the one with several TB of disk storage dedicated to package archives). Bandwidth in the needed amounts is even more expensive than electricity and cooling, which is why I have generally kept the builders local. I currently use 10Gbps links and from experience I wouldn't want to use anything less than 100Mbps--QuickBuild is already unstable enough without throwing network variability into the mix!
Good ideas all! Keep them coming. :-)
And for those who have already donated, thanks a bunch! Like David said, let's look at this as an opportunity to make the TDE infrastructure even better.
Tim
My intention was to keep the build farm running until R14 was
released, then see if any donations came in to cover some of the costs. However, this morning the main chiller was destroyed due to sudden power brownouts/surges, forcing the build farm offline. As I do not have the funds to replace this chiller, and with the onset of spring/warmer temperatures here, I cannot restart the build farm at this time.<
How much money is needed? I am admittedly spending almost all of my money on political expenses right now and may be too broke to do anything until May or June.
Right now I am looking at needing between $1500 and $2000. This doesn't actually cover the cost of a proper unit, but I should be able to provide the remainder if all goes well. If someone knows of anyone getting rid of an older Liebert system or similar who would be willing to donate the unit to the project, that would work just fine instead. ;-)
Once we hit the summer months I should be able to partially restart the build farm even if the chiller has not been replaced as the existing comfort air system is powerful enough to cool the servers. Obviously I won't be able to run that system until several months from now (45 degrees outside right now), but hey, it's a backup plan at least. :-)
Could we make a formal appeal for cash for expenses for getting 14.0.0 out the door, with a specific target, if it's larger than what this list alone could be hoped to raise? It would be very good to at least get the older deployments support for 14 before dropping some of them off.
Probably. I'm not sure how to approach this yet; give me some time to think it over.
Do we know which releases are more commonly deployed organizationally? I would assume the LTS releases, but such obvious assumptions prove incorrect all the time -- one sufficiently large deployment using, say, Meerkat, could make that a higher priority for us than Lucid. Do we have statistics on which versions get hit up for updates most frequently? Is there a better measure than that that we might use, given that many deployments might not update frequently at all?
A public poll is all I can think of. As you said, many institutions will not upgrade often, which skews the statistics on this end. Add in local mirrors (at least one TDE deployment I am aware of does this) and I have no valid data on which to base a decision.
Tim
On Tuesday 11 of March 2014 03:08:58 Timothy Pearson wrote:
Do we know which releases are more commonly deployed organizationally? I would assume the LTS releases, but such obvious assumptions prove incorrect all the time -- one sufficiently large deployment using, say, Meerkat, could make that a higher priority for us than Lucid. Do we have statistics on which versions get hit up for updates most frequently? Is there a better measure than that that we might use, given that many deployments might not update frequently at all?
A public poll is all I can think of. As you said, many institutions will not upgrade often, which skews the statistics on this end. Add in local mirrors (at least one TDE deployment I am aware of does this) and I have no valid data on which to base a decision.
What if we reduce the supported Ubuntu versions in nightly-builds, but keep support in the stable release? In this way, the load will occur only at the time of building packages for the stable branch.
What do you think about this?
But then you don't find out about breakage on old systems until release. On Mar 10, 2014 10:56 PM, "Slávek Banko" slavek.banko@axis.cz wrote:
On Tuesday 11 of March 2014 03:08:58 Timothy Pearson wrote:
Do we know which releases are more commonly deployed organizationally? I would assume the LTS releases, but such obvious assumptions prove incorrect all the time -- one sufficiently large deployment using, say, Meerkat, could make that a higher priority for us than Lucid. Do we have statistics on which versions get hit up for updates most frequently? Is there a better measure than that that we might use, given that many deployments might not update frequently at all?
A public poll is all I can think of. As you said, many institutions will not upgrade often, which skews the statistics on this end. Add in local mirrors (at least one TDE deployment I am aware of does this) and I have no valid data on which to base a decision.
What if we reduce the supported Ubuntu versions in nightly-builds, but keep support in the stable release? In this way, the load will occur only at the time of building packages for the stable branch.
What do you think about this?
-- Slavek
To unsubscribe, e-mail: trinity-devel-unsubscribe@lists.pearsoncomputing.net For additional commands, e-mail: trinity-devel-help@lists.pearsoncomputing.net Read list messages on the web archive: http://trinity-devel.pearsoncomputing.net/ Please remember not to top-post: http://trinity.pearsoncomputing.net/mailing_lists/#top-posting
On Tuesday 11 of March 2014 04:02:56 Calvin Morrison wrote:
But then you don't find out about breakage on old systems until release.
Not so literally. For the stable branch I mean my ppa, where the building was not so often as in nightly-builds, but will be tested before the final release.
On Tuesday 11 of March 2014 04:02:56 Calvin Morrison wrote:
But then you don't find out about breakage on old systems until release.
Not so literally. For the stable branch I mean my ppa, where the building was not so often as in nightly-builds, but will be tested before the final release.
-- Slavek
While the build farm remains offline this might be a reasonable solution. I may end up restricting the nightly builds to Ubuntu Trusty and Debian Wheezy for now.
Top priority for me right now is getting the two backup i386/amd64 builders online (the armel/armhf builders produce very little heat and therefore were not affected). I should have this complete by Thursday.
Tim
While the build farm remains offline this might be a reasonable solution. I may end up restricting the nightly builds to Ubuntu Trusty and Debian Wheezy for now.
Tim, if it could be useful, I could build Debian packages of the GIT repo on my own computer (using pbuilder) and then upload them somewhere on the TDE servers, so they would be available as a temporary replacement until we come up with a solution for the original problem. It takes about 16-17 hours to do a full build on my system, but I have to split it over two nights. So a possible workflow would be:
- build debian/jessie amd64 (2 days) - upload debian/jessie packages (about 1.2GB .deb only, 2.1 GB including source and build logs). One day should suffice - build debian/jessie i386 (2 days) - upload debian/jessie packages (about 1.2GB). One day
- build debian/wheezy amd64 (2 days) - upload debian/wheezy packages (about 1.2GB). One day - build debian/wheezy i386 (2 days) - upload debian/wheezy packages (about 1.2GB). One day
- build debian/squeeze amd64 (2 days) - upload debian/squeeze packages (about 1.2GB). One day - build debian/squeeze i386 (2 days) - upload debian/squeeze packages (about 1.2GB). One day
So approximately every three weeks I could provide updated Debian packages for the main two architectures (perhaps even more frequently, but I made space for days where I won't be able to do work in TDE). You could then avoid building nightly for wheezy and use the free resources for building another Ubuntu distro.
Perhaps some other developer could do similar things for other Ubuntu distro.
Cheers Michele
While the build farm remains offline this might be a reasonable solution. I may end up restricting the nightly builds to Ubuntu Trusty and Debian Wheezy for now.
Tim, if it could be useful, I could build Debian packages of the GIT repo on my own computer (using pbuilder) and then upload them somewhere on the TDE servers, so they would be available as a temporary replacement until we come up with a solution for the original problem.
Thank you for the offer! While the backup build systems *should* be able to handle the load of only building for Trusty and Wheezy (once I get them online), if they fail or end up building too slowly I may have to take you up on this.
Tim
Thank you for the offer! While the backup build systems *should* be able to handle the load of only building for Trusty and Wheezy (once I get them online), if they fail or end up building too slowly I may have to take you up on this.
No worries, just let me know :)
Michele
---- Timothy Pearson kb9vqf@pearsoncomputing.net wrote:
All,
The QuickBuild build farm was taken offline this morning and will remain offline indefinitely. While I realize this pushes R14 release further out, the cold hard reality is that donations have fallen so much, along with the bad economy impacting my own finances, that I cannot afford to keep the build farm running at this time. Finances were already strained with the Web server upgrade, as the new software required newer hardware to keep performance at a reasonable level.
My intention was to keep the build farm running until R14 was released, then see if any donations came in to cover some of the costs. However, this morning the main chiller was destroyed due to sudden power brownouts/surges, forcing the build farm offline. As I do not have the funds to replace this chiller, and with the onset of spring/warmer temperatures here, I cannot restart the build farm at this time.
I will do my best to bring one builder online for each architecture in the next few days, but this severe drop in build power will obviously cause full archive rebuilds to take several months at minimum. As a result we need to start looking at dropping support for older Ubuntu versions, though I did not want to take this step as I know some larger deployments depend on this support.
All other TDE services are not affected at this time.
Sorry for the bad news!
Timothy Pearson Trinity Desktop Project
Not bad news -- an opportunity for improvement!
Let's pass the hat again between this list and the trinity-user list and get this covered. I'll chip in.
-- David C. Rankin, J.D.,P.E.
-
Alexandre -
Calvin Morrison -
David C. Rankin -
James Gholston -
Michele Calgaro -
Robert Xu -
Slávek Banko -
Timothy Pearson