10 Mar
2014
10 Mar
'14
9:57 p.m.
All,
The QuickBuild build farm was taken offline this morning and will remain
offline indefinitely. While I realize this pushes R14 release further
out, the cold hard reality is that donations have fallen so much, along
with the bad economy impacting my own finances, that I cannot afford to
keep the build farm running at this time. Finances were already strained
with the Web server upgrade, as the new software required newer hardware
to keep performance at a reasonable level.
My intention was to keep the build farm running until R14 was released,
then see if any donations came in to cover some of the costs. However,
this morning the main chiller was destroyed due to sudden power
brownouts/surges, forcing the build farm offline. As I do not have the
funds to replace this chiller, and with the onset of spring/warmer
temperatures here, I cannot restart the build farm at this time.
I will do my best to bring one builder online for each architecture in the
next few days, but this severe drop in build power will obviously cause
full archive rebuilds to take several months at minimum. As a result we
need to start looking at dropping support for older Ubuntu versions,
though I did not want to take this step as I know some larger deployments
depend on this support.
All other TDE services are not affected at this time.
Sorry for the bad news!
Timothy Pearson
Trinity Desktop Project
10 Mar
10 Mar
11:13 p.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
All,
The QuickBuild build farm was taken offline this morning and will remain
offline indefinitely. While I realize this pushes R14 release further
out, the cold hard reality is that donations have fallen so much, along
with the bad economy impacting my own finances, that I cannot afford to
keep the build farm running at this time. Finances were already strained
with the Web server upgrade, as the new software required newer hardware
to keep performance at a reasonable level.
My intention was to keep the build farm running until R14 was released,
then see if any donations came in to cover some of the costs. However,
this morning the main chiller was destroyed due to sudden power
brownouts/surges, forcing the build farm offline. As I do not have the
funds to replace this chiller, and with the onset of spring/warmer
temperatures here, I cannot restart the build farm at this time.
I will do my best to bring one builder online for each architecture in the
next few days, but this severe drop in build power will obviously cause
full archive rebuilds to take several months at minimum. As a result we
need to start looking at dropping support for older Ubuntu versions,
though I did not want to take this step as I know some larger deployments
depend on this support.
All other TDE services are not affected at this time.
Sorry for the bad news!
Timothy Pearson
Trinity Desktop Project
This little donation will help! Who's next?
-Alexandre
11:35 p.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
My intention was to keep the build farm running until
R14 was
released, then see if any donations came in to cover some of the costs.
However, this morning the main chiller was destroyed due to sudden power
brownouts/surges, forcing the build farm offline. As I do not have the
funds to replace this chiller, and with the onset of spring/warmer
temperatures here, I cannot restart the build farm at this time.<
How much money is needed? I am admittedly spending almost all of my
money on political expenses right now and may be too broke to do
anything until May or June.
Could we make a formal appeal for cash for expenses for getting 14.0.0
out the door, with a specific target, if it's larger than what this list
alone could be hoped to raise? It would be very good to at least get
the older deployments support for 14 before dropping some of them off.
Do we know which releases are more commonly deployed organizationally?
I would assume the LTS releases, but such obvious assumptions prove
incorrect all the time -- one sufficiently large deployment using, say,
Meerkat, could make that a higher priority for us than Lucid. Do we
have statistics on which versions get hit up for updates most
frequently? Is there a better measure than that that we might use,
given that many deployments might not update frequently at all?
James Gholston
11 Mar
11 Mar
12:06 a.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
Tim , Robert,
Can we leverage OpenSUSEs OBS project for building trinity?
On Mar 10, 2014 7:35 PM, "James Gholston" <jamesg(a)dimensionality.com>
wrote:
My intention
was to keep the build farm running until R14 was released,
then see if any
donations came in to cover some of the costs. However, this
morning the main chiller was destroyed due to sudden power
brownouts/surges, forcing the build farm offline. As I do not have the
funds to replace this chiller, and with the onset of spring/warmer
temperatures here, I cannot restart the build farm at this time.<
How much money is needed? I am admittedly spending almost all of my money
on political expenses right now and may be too broke to do anything until
May or June.
Could we make a formal appeal for cash for expenses for getting 14.0.0 out
the door, with a specific target, if it's larger than what this list alone
could be hoped to raise? It would be very good to at least get the older
deployments support for 14 before dropping some of them off.
Do we know which releases are more commonly deployed organizationally? I
would assume the LTS releases, but such obvious assumptions prove incorrect
all the time -- one sufficiently large deployment using, say, Meerkat,
could make that a higher priority for us than Lucid. Do we have statistics
on which versions get hit up for updates most frequently? Is there a
better measure than that that we might use, given that many deployments
might not update frequently at all?
James Gholston
---------------------------------------------------------------------
To unsubscribe, e-mail: trinity-devel-unsubscribe@
lists.pearsoncomputing.net
For additional commands, e-mail: trinity-devel-help@lists.
pearsoncomputing.net
Read list messages on the web archive: http://trinity-devel.
pearsoncomputing.net/
Please remember not to top-post: http://trinity.
pearsoncomputing.net/mailing_lists/#top-posting
2:12 a.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
Tim , Robert,
Can we leverage OpenSUSEs OBS project for building trinity?
TDE would overwhelm any public build system I am aware of. During a full
archive rebuild we typically keep 80 Opteron cores busy 24/7 for several
weeks (!), while consuming over 150GB in disk space. Launchpad couldn't
even handle TDE; it was taking way too long to build on the public
resources.
Tim
4:59 a.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
Hey, been a long time since I've emailed this list. Life...
On 10 March 2014 20:06, Calvin Morrison <mutantturkey(a)gmail.com> wrote:
Tim , Robert,
Can we leverage OpenSUSEs OBS project for building trinity?
We can, but their build system is already overwhelmed with tons of
projects, and I'm pretty sure I don't want to add the strain of
trinity for now. Also, they don't integrate with git, which may make
things difficult.
I would definitely recommend figuring out which releases should no
longer be supported - I was looking at the trinitydesktop.org website
and there are a lot of old releases that we still seem to have...
Tim - would cloud building be an option? (like those $0.005/hr servers
that can be brought up/down on demand?). Just a random thought on
that. Could be useful to leverage if QuickBuild could support it.
--
cheers, Robert :: protocol.by/rxu
5:26 a.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
Hey, been a long time since I've emailed this
list. Life...
On 10 March 2014 20:06, Calvin Morrison <mutantturkey(a)gmail.com> wrote:
The main problem with this is bandwidth to the build control server
(a.k.a. the one with several TB of disk storage dedicated to package
archives). Bandwidth in the needed amounts is even more expensive than
electricity and cooling, which is why I have generally kept the builders
local. I currently use 10Gbps links and from experience I wouldn't want
to use anything less than 100Mbps--QuickBuild is already unstable enough
without throwing network variability into the mix!
Good ideas all! Keep them coming. :-)
And for those who have already donated, thanks a bunch! Like David said,
let's look at this as an opportunity to make the TDE infrastructure even
better.
Tim
Tim , Robert,
Can we leverage OpenSUSEs OBS project for building trinity?
We can, but their build system is already overwhelmed with tons of
projects, and I'm pretty sure I don't want to add the strain of
trinity for now. Also, they don't integrate with git, which may make
things difficult.
I would definitely recommend figuring out which releases should no
longer be supported - I was looking at the trinitydesktop.org website
and there are a lot of old releases that we still seem to have...
Tim - would cloud building be an option? (like those $0.005/hr servers
that can be brought up/down on demand?). Just a random thought on
that. Could be useful to leverage if QuickBuild could support it.
2:08 a.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
My intention
was to keep the build farm running until R14 was
released, then see if any
donations came in to cover some of the costs.
However, this morning the main chiller was destroyed due to sudden power
brownouts/surges, forcing the build farm offline. As I do not have the
funds to replace this chiller, and with the onset of spring/warmer
temperatures here, I cannot restart the build farm at this time.<
How much money is needed? I am admittedly spending almost all of my
money on political expenses right now and may be too broke to do
anything until May or June. Could we make a formal appeal for cash for expenses
for getting 14.0.0
out the door, with a specific target, if it's larger than what this list
alone could be hoped to raise? It would be very good to at least get
the older deployments support for 14 before dropping some of them off.
Probably. I'm not sure how to approach this yet; give me some time to
think it over.
Do we know which releases are more commonly deployed
organizationally?
I would assume the LTS releases, but such obvious assumptions prove
incorrect all the time -- one sufficiently large deployment using, say,
Meerkat, could make that a higher priority for us than Lucid. Do we
have statistics on which versions get hit up for updates most
frequently? Is there a better measure than that that we might use,
given that many deployments might not update frequently at all?
A public poll is all I can think of. As you said, many institutions will
not upgrade often, which skews the statistics on this end. Add in local
mirrors (at least one TDE deployment I am aware of does this) and I have
no valid data on which to base a decision.
Tim
2:56 a.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
On Tuesday 11 of March 2014 03:08:58 Timothy Pearson wrote:
What if we reduce the supported Ubuntu versions in nightly-builds, but keep
support in the stable release? In this way, the load will occur only at the
time of building packages for the stable branch.
What do you think about this?
--
Slavek
Do we know
which releases are more commonly deployed organizationally?
I would assume the LTS releases, but such obvious assumptions prove
incorrect all the time -- one sufficiently large deployment using, say,
Meerkat, could make that a higher priority for us than Lucid. Do we
have statistics on which versions get hit up for updates most
frequently? Is there a better measure than that that we might use,
given that many deployments might not update frequently at all?
A public poll is all I can think of. As you said, many institutions will
not upgrade often, which skews the statistics on this end. Add in local
mirrors (at least one TDE deployment I am aware of does this) and I have
no valid data on which to base a decision.
3:02 a.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
But then you don't find out about breakage on old systems until release.
On Mar 10, 2014 10:56 PM, "Slávek Banko" <slavek.banko(a)axis.cz> wrote:
On Tuesday 11 of March 2014 03:08:58 Timothy Pearson
wrote:
What if we reduce the supported Ubuntu versions in nightly-builds, but keep
support in the stable release? In this way, the load will occur only at the
time of building packages for the stable branch.
What do you think about this?
--
Slavek
---------------------------------------------------------------------
To unsubscribe, e-mail:
trinity-devel-unsubscribe(a)lists.pearsoncomputing.net
For additional commands, e-mail:
trinity-devel-help(a)lists.pearsoncomputing.net
Read list messages on the web archive:
http://trinity-devel.pearsoncomputing.net/
Please remember not to top-post:
http://trinity.pearsoncomputing.net/mailing_lists/#top-posting
Do we
know which releases are more commonly deployed organizationally?
I would assume the LTS releases, but such obvious assumptions prove
incorrect all the time -- one sufficiently large deployment using, say,
Meerkat, could make that a higher priority for us than Lucid. Do we
have statistics on which versions get hit up for updates most
frequently? Is there a better measure than that that we might use,
given that many deployments might not update frequently at all?
A public poll is all I can think of. As you said, many institutions will
not upgrade often, which skews the statistics on this end. Add in local
mirrors (at least one TDE deployment I am aware of does this) and I have
no valid data on which to base a decision.
3:18 a.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
On Tuesday 11 of March 2014 04:02:56 Calvin Morrison wrote:
But then you don't find out about breakage on old
systems until release.
Not so literally. For the stable branch I mean my ppa, where the building was
not so often as in nightly-builds, but will be tested before the final
release.
--
Slavek
3:29 a.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
On Tuesday 11 of March 2014 04:02:56 Calvin Morrison
wrote:
While the build farm remains offline this might be a reasonable solution.
I may end up restricting the nightly builds to Ubuntu Trusty and Debian
Wheezy for now.
Top priority for me right now is getting the two backup i386/amd64
builders online (the armel/armhf builders produce very little heat and
therefore were not affected). I should have this complete by Thursday.
Tim
But then you don't find out about breakage on
old systems until release.
Not so literally. For the stable branch I mean my ppa, where the building
was
not so often as in nightly-builds, but will be tested before the final
release.
--
Slavek 
4:46 a.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
While the build farm remains offline this might be a
reasonable solution.
I may end up restricting the nightly builds to Ubuntu Trusty and Debian Wheezy for now.
Tim,
if it could be useful, I could build Debian packages of the GIT repo on my own computer
(using pbuilder) and then upload them somewhere on the TDE servers, so they would be
available as a temporary replacement until we come up with a solution for the original
problem.
It takes about 16-17 hours to do a full build on my system, but I have to split it over
two nights.
So a possible workflow would be:
- build debian/jessie amd64 (2 days)
- upload debian/jessie packages (about 1.2GB .deb only, 2.1 GB including source and build
logs). One day should suffice
- build debian/jessie i386 (2 days)
- upload debian/jessie packages (about 1.2GB). One day
- build debian/wheezy amd64 (2 days)
- upload debian/wheezy packages (about 1.2GB). One day
- build debian/wheezy i386 (2 days)
- upload debian/wheezy packages (about 1.2GB). One day
- build debian/squeeze amd64 (2 days)
- upload debian/squeeze packages (about 1.2GB). One day
- build debian/squeeze i386 (2 days)
- upload debian/squeeze packages (about 1.2GB). One day
So approximately every three weeks I could provide updated Debian packages for the main
two architectures (perhaps even more frequently, but I made space for days where I
won't be able to do work in TDE).
You could then avoid building nightly for wheezy and use the free resources for building
another Ubuntu distro.
Perhaps some other developer could do similar things for other Ubuntu distro.
Cheers
Michele
5:22 a.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
While the
build farm remains offline this might be a reasonable
solution.
I may end up restricting the nightly builds to Ubuntu Trusty and Debian
Wheezy for now.
Tim,
if it could be useful, I could build Debian packages of the GIT repo on my
own computer (using pbuilder) and then upload them somewhere on the TDE
servers, so they would be available as a temporary replacement until we
come up with a solution for the original problem.
5:26 a.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
Thank you for the offer! While the backup build
systems *should* be able
to handle the load of only building for Trusty and Wheezy (once I get them
online), if they fail or end up building too slowly I may have to take you
up on this.
No worries, just let me know :)
Michele
4:14 a.m.
New subject: [trinity-devel] Indefinite QuickBuild outage
---- Timothy Pearson <kb9vqf(a)pearsoncomputing.net> wrote:
All,
The QuickBuild build farm was taken offline this morning and will remain
offline indefinitely. While I realize this pushes R14 release further
out, the cold hard reality is that donations have fallen so much, along
with the bad economy impacting my own finances, that I cannot afford to
keep the build farm running at this time. Finances were already strained
with the Web server upgrade, as the new software required newer hardware
to keep performance at a reasonable level.
My intention was to keep the build farm running until R14 was released,
then see if any donations came in to cover some of the costs. However,
this morning the main chiller was destroyed due to sudden power
brownouts/surges, forcing the build farm offline. As I do not have the
funds to replace this chiller, and with the onset of spring/warmer
temperatures here, I cannot restart the build farm at this time.
I will do my best to bring one builder online for each architecture in the
next few days, but this severe drop in build power will obviously cause
full archive rebuilds to take several months at minimum. As a result we
need to start looking at dropping support for older Ubuntu versions,
though I did not want to take this step as I know some larger deployments
depend on this support.
All other TDE services are not affected at this time.
Sorry for the bad news!
Timothy Pearson
Trinity Desktop Project
Not bad news -- an opportunity for improvement!
Let's pass the hat again between this list and the trinity-user list and get this
covered. I'll chip in.
--
David C. Rankin, J.D.,P.E.
3700
days inactive
3701
days old
15 comments
8 participants
participants (8)
-
Alexandre -
Calvin Morrison -
David C. Rankin -
James Gholston -
Michele Calgaro -
Robert Xu -
Slávek Banko -
Timothy Pearson