21 May
2012
21 May
'12
12:23 a.m.
--- On Sun, 5/20/12, Timothy Pearson <kb9vqf(a)pearsoncomputing.net> wrote:
From: Timothy Pearson
<kb9vqf(a)pearsoncomputing.net>
Subject: Re: [trinity-devel] Que Pasa Bro -- What be the Story Tim?
To: trinity-devel(a)lists.pearsoncomputing.net
Date: Sunday, May 20, 2012, 1:31 PM
Universities or OpenSource
I'm not concerned about the type of outage we experienced. Those things happen.
If the recent outage was a hosting plan squabble and you still had normal customer ISP
access, then a personal email to somebody would have helped to pass the word as much as
possible. If you lost personal services too, then perhaps a trip to the library or local
coffee shop would have been in order to send an email or leave a message in IRC.
The recent outage leaves me concerned about what we can do should something serious happen
long-term. Nasty weather and natural disasters could disrupt an ISP connection for days,
weeks, or months. Serious illness could keep you out of touch too.
We need a basic contingency plan. Something so the project stays alive and the remaining
participants are not left hanging.
When the outage occurred I had the latest (known) GIT sources. Worst case then the project
could have move toward some kind of restoration with that, minus a few patches. Yet when I
think about the bug tracker, etherpad, wiki, and mail list archives disappearing then I do
not feel so comfortable.
Granted, part of what we do is dynamic, but not so much that a basic contingency plan
can't restore.
Currently we have two domains: www.pearsoncomputing.net and www.trinitydesktop.org. If
both are housed from the same location/ISP, then we become dependent upon those people not
to get a bug up their backside. Perhaps we can move one domain elsewhere. Basically,
diversify --- don't put all our eggs in one basket.
We do not need to mirror everything in the traditional sense. Some work-arounds would save
the project.
For example, some of us can rsync our local GIT tree at least daily or more often. We
don't need to "mirror" those sources if several of us do that. GIT is a
distributed system so all we need are people keeping synced.
Losing the bug tracker would be a huge challenge. Is there a way some of us can rsync or
spider the bug tracker to our personal systems? Likewise for the wiki, etherpad, mail list
archives. Such efforts would be the equivalent of a backup system.
None of this need cost additional funds. We just need a way for several project members to
rsync/spider files. Should disaster strike we then have a way to get us back on our feet.
I don't want to make mountains out of molehills. I'm just asking that as a group
we consider basic stewardship. :-)
Darrell
On 05/20/2012 01:05 AM, Timothy
Pearson
wrote:
> Nothing serious on this end fortunately. :-)
>
> The ISP that services the TDE server cluster had a
localized equipment
> failure, but instead of trying to fix the
outage
they decided to blame
> the
> customer (me). This prolonged the outage
significantly, even though in
> the end it did turn out to be upstream ISP
equipment failure.
>
> I do apologize for the outage; as you are probably
aware the TDE
> services
> can only utilize a single carrier due to the
long-term contractual
> expense
> of bringing out a redundant service provider to the
server location.
Tim
Have we approached any of the high-bandwidth outfits
to see if they would be willing to provide mirror
services? I see so many
run-of-the-mill projects that have mirror
services all
over the world. It
sure
seems like a favorable project like TDE would be a
prime candidate to be
accepted and helped by some of these
institutions.
<snip>
We do mirror most of our static binary content on a
worldwide mirror system. However, most of the TDE services are dynamic
(GIT, Wiki, mailing lists, searchable documentation, IRC, bugtracker, patch
lists, etc.) and therefore cannot (easily/safely) be mirrored.
Good idea though!