19 Dec
2020
19 Dec
'20
11 a.m.
Hi all,
the tenth service on the migration list is the primary archive.
-- Migration status --
The change of the primary archive was made before the release of R14.0.7
(in October 2019). This removed the dependency of the primary archive on
QuickBuild, which made it possible to make a change in the archive layout
and cleanup. The reprepro tool is now used to manage repositories of deb
packages, which allows us to publish new releases very smoothly. Thanks to
the use of reprepro, Contents files for apt-file are also available.
Because the primary archive is too large to be placed on a tde-box or
tde-vps, it is located on my home server. In order to reduce the size for
the mirror system, packages of older versions are excluded.
I would like to thank those who provides mirrors for our repositories, and
especially Mike Bird for his efforts and attention in managing the primary
mirror and overseeing the status of other mirrors.
-- What needs to be done --
We may consider placing the primary archive in a hosting center - for
example, by purchasing larger disks for tde-box or by increasing the space
provided for tde-vps. However, it will probably be a good idea to consider
such changes only after we are able to manage the project funds and accept
donations.
-- More ideas and suggestions?
Do you have any other ideas and suggestions?
Cheers
--
Slávek
19 Dec
19 Dec
10:02 p.m.
On 2020/12/20 04:00 AM, Slávek Banko via tde-users wrote:
We may consider placing the primary archive in a
hosting center - for
example, by purchasing larger disks for tde-box or by increasing the space
provided for tde-vps. However, it will probably be a good idea to consider
such changes only after we are able to manage the project funds and accept
donations.
This is defintely something we will need to do in future, to move away from possible
one-person-dependant services.
I agree with you we need to figure out how to handle donations and project funds, then we
can relook at this.
Cheers
Michele
20 Dec
20 Dec
9:12 a.m.
On Sunday 20 December 2020 12:02:26 am Michele Calgaro via tde-users wrote:
On 2020/12/20 04:00 AM, Slávek Banko via tde-users
wrote:
In that vein (away from possible one-person-dependant services):
- How big is the archive? (<250GB? ~4TB?)
- How simple is it to back it up? (a few rsync’s? tarball?)
I have no issues storing a copy, or two, of the archive on one of my backup
servers (which are in hosting centers).
Best,
Michael
* If it’s in the 10-15GB range I can dump multiple copies on my DNS servers as
well.
We may consider placing the primary archive in a
hosting center - for
example, by purchasing larger disks for tde-box or by increasing the
space provided for tde-vps. However, it will probably be a good idea to
consider such changes only after we are able to manage the project funds
and accept donations.
This is defintely something we will need to do in future, to move away from
possible one-person-dependant services. I agree with you we need to figure
out how to handle donations and project funds, then we can relook at this.
11:34 a.m.
Dne Sunday 20 of December 2020 18:12:25 Michael via tde-users napsal(a):
I'm glad you're interested in the details.
Current size and occupied space of the partition:
/dev/mapper/lvm1-tde_data 400G 336G 65G 84% /srv/tde
A subset of this volume is synchronized to the mirror system. First to the
primary mirror, from which the other mirrors are then synchronized. This
synchronization usually takes place several times a day, with pauses of 3
hours between the individual synchronizations.
The size always decreases significantly at the moment of release, when the
contents of the official repository and PSB are identical. The size of the
repository is now constantly growing again as the packages in PSB are
built for R14.0.10~pre. Such an increase can be almost 100 GiB. At the
same time, there is a gradual increase in volume when adding support for
new distributions.
During the outage, which I announced in early December, I replaced my home
server's hardware with slightly newer components, including the purchase
of new, larger disks. So now I have a sufficient reserve of disk capacity
again.
The specified partition is on two disks (RAID1). In addition to these
disks, I make a regular daily local backup on an external usb disk. The
backup disk requires significantly more capacity, as the daily differences
can be large - from hundreds of MiBs to tens of GiBs. Next to this, I'm
making a copy of the archive on my next machine in a remote location (two
disks in RAID1). Copying to this remote location is not regular yet.
On Sunday 20 December 2020 12:02:26 am Michele Calgaro
via tde-users
wrote:
On 2020/12/20
04:00 AM, Slávek Banko via tde-users wrote:
In that vein (away from possible one-person-dependant services):
- How big is the archive? (<250GB? ~4TB?)
- How simple is it to back it up? (a few rsync’s? tarball?)
I have no issues storing a copy, or two, of the archive on one of my
backup servers (which are in hosting centers).
We may consider placing the primary archive in a
hosting center -
for example, by purchasing larger disks for tde-box or by increasing
the space provided for tde-vps. However, it will probably be a good
idea to consider such changes only after we are able to manage the
project funds and accept donations.
This is defintely something we will need to do in future, to move away
from possible one-person-dependant services. I agree with you we need
to figure out how to handle donations and project funds, then we can
relook at this. Best,
Michael
* If it’s in the 10-15GB range I can dump multiple copies on my DNS
servers as well.
____________________________________________________
Cheers
--
Slávek
12:28 p.m.
On Sunday 20 December 2020 01:34:29 pm Slávek Banko via tde-users wrote:
Dne Sunday 20 of December 2020 18:12:25 Michael via
tde-users napsal(a):
I'm glad you're interested in the details.
Current size and occupied space of the partition:
/dev/mapper/lvm1-tde_data 400G 336G 65G 84% /srv/tde
A subset of this volume is synchronized to the mirror system. First to the
primary mirror, from which the other mirrors are then synchronized. This
synchronization usually takes place several times a day, with pauses of 3
hours between the individual synchronizations.
The size always decreases significantly at the moment of release, when the
contents of the official repository and PSB are identical. The size of the
repository is now constantly growing again as the packages in PSB are
built for R14.0.10~pre. Such an increase can be almost 100 GiB. At the
same time, there is a gradual increase in volume when adding support for
new distributions.
During the outage, which I announced in early December, I replaced my home
server's hardware with slightly newer components, including the purchase
of new, larger disks. So now I have a sufficient reserve of disk capacity
again.
The specified partition is on two disks (RAID1). In addition to these
disks, I make a regular daily local backup on an external usb disk. The
backup disk requires significantly more capacity, as the daily differences
can be large - from hundreds of MiBs to tens of GiBs. Next to this, I'm
making a copy of the archive on my next machine in a remote location (two
disks in RAID1). Copying to this remote location is not regular yet.
Hi Slávek,
Lol, interesting, but TMI ;)
- What’s the max size you’ve ever needed?
- How many unique backups do you want? *
- Rsync fine? Push? Pull?(what are the paths and creds?) **
- cron every 4 hours okay?
It’ll take half an hour to setup, and I guess we can stop spamming the rest of
the list, so just send to me personally for the rest...
Best,
Michael
michael@
* I’ll do one here at home off of one of my servers as well.
** I prefer pull, but just from a security issue of not giving people access
to my servers.
PS: I’ll send you a copy of my final script so you can automate the remote
location you were talking about too.
On Sunday 20 December 2020 12:02:26 am Michele
Calgaro via tde-users
wrote:
On
2020/12/20 04:00 AM, Slávek Banko via tde-users wrote:
In that vein (away from possible one-person-dependant services):
- How big is the archive? (<250GB? ~4TB?)
- How simple is it to back it up? (a few rsync’s? tarball?)
I have no issues storing a copy, or two, of the archive on one of my
backup servers (which are in hosting centers). We may consider placing the primary archive in a
hosting center -
for example, by purchasing larger disks for tde-box or by increasing
the space provided for tde-vps. However, it will probably be a good
idea to consider such changes only after we are able to manage the
project funds and accept donations.
This is defintely something we will need to do in future, to move away
from possible one-person-dependant services. I agree with you we need
to figure out how to handle donations and project funds, then we can
relook at this. 
1:01 p.m.
On Sun December 20 2020 11:34:29 Slávek Banko via tde-users wrote:
Current size and occupied space of the partition:
/dev/mapper/lvm1-tde_data 400G 336G 65G 84% /srv/tde
The primary mirror is currently:
/dev/mapper/DISK-TDE 298G 250G 49G 84% /ARCHIVE/TDE
Since the upgrade a year ago we can handle 400G no problem so the
easiest solution might be just to mirror everything. I'm not sure
how much the secondaries can take but it's not like it's a massive
change.
I wonder if Michael <mb_trinity_desktop(a)inet-design.com> would
be interested in becoming a mirror, which would use less primary
archive bandwidth than pulling backups directly.
--Mike
6:45 p.m.
On Sunday 20 of December 2020 21:28:11 Michael via tde-users wrote:
I'm glad you're interested in the details.
Current size and occupied space of the partition:
/dev/mapper/lvm1-tde_data 400G 336G 65G 84% /srv/tde
A subset of this volume is synchronized to the mirror system. First to
the primary mirror, from which the other mirrors are then
synchronized. This synchronization usually takes place several times a
day, with pauses of 3 hours between the individual synchronizations.
The size always decreases significantly at the moment of release, when
the contents of the official repository and PSB are identical. The
size of the repository is now constantly growing again as the packages
in PSB are built for R14.0.10~pre. Such an increase can be almost 100
GiB. At the same time, there is a gradual increase in volume when
adding support for new distributions.
During the outage, which I announced in early December, I replaced my
home server's hardware with slightly newer components, including the
purchase of new, larger disks. So now I have a sufficient reserve of
disk capacity again.
The specified partition is on two disks (RAID1). In addition to these
disks, I make a regular daily local backup on an external usb disk.
The backup disk requires significantly more capacity, as the daily
differences can be large - from hundreds of MiBs to tens of GiBs. Next
to this, I'm making a copy of the archive on my next machine in a
remote location (two disks in RAID1). Copying to this remote location
is not regular yet.
Hi Slávek,
Lol, interesting, but TMI ;)
- What’s the max size you’ve ever needed?
The current partition size is to be understood as "beginning size". Each
newly added distribution adds 1.5 to 12 GiB (depending on the number of
architectures). Once we release R14.1.x, it will mean new repositories, an
estimated new 80 GiB. So here it is necessary to take into account that
the required space will continue to increase.
On Sunday 20 December 2020 01:34:29 pm Slávek Banko
via tde-users wrote:
> Dne Sunday 20 of December 2020 18:12:25 Michael via tde-users
napsal(a):
On Sunday 20 December 2020 12:02:26 am Michele Calgaro
via tde-users
wrote:
On
2020/12/20 04:00 AM, Slávek Banko via tde-users wrote:
> We may consider placing the primary archive in a hosting center
> - for example, by purchasing larger disks for tde-box or by
> increasing the space provided for tde-vps. However, it will
> probably be a good idea to consider such changes only after we
> are able to manage the project funds and accept donations.
This is defintely something we will need to do in future, to move
away from possible one-person-dependant services. I agree with you
we need to figure out how to handle donations and project funds,
then we can relook at this.
In that vein (away from possible one-person-dependant services):
- How big is the archive? (<250GB? ~4TB?)
- How simple is it to back it up? (a few rsync’s? tarball?)
I have no issues storing a copy, or two, of the archive on one of my
backup servers (which are in hosting centers). - How many unique backups do you want? *
I use dirvish for backup. It uses the principle of hardlinks so that each
backup serves as an independent full backup, although it is performed as
an incremental one. Due to this independence, backups can have different
expiration times. A normal daily backup has an expiration date of 15 days,
weekly backups have an expiration date of 3 months, and the first weekly
backup of the month has an expiration date of one year.
- Rsync fine? Push? Pull?(what are the paths and
creds?) **
- cron every 4 hours okay?
An rsync server is provided for synchronization to the primary mirror. So
the primary mirror does the pull. As I mentioned, there are some parts
excluded from syncing to mirrors.
If we want to solve the backup of the entire primary archive and at the
same time keep the exclusion so that the mirror system does not grow
beyond the accepted limit and contains currently useful data, it is
possible to set the second path for rsync.
It’ll take half an hour to setup, and I guess we can
stop spamming the
rest of the list, so just send to me personally for the rest...
It will certainly be good for Mike, as the administrator of the primary
mirror, to take part in the discussion here as well.
Best,
Michael
michael@
* I’ll do one here at home off of one of my servers as well.
** I prefer pull, but just from a security issue of not giving people
access to my servers.
PS: I’ll send you a copy of my final script so you can automate the
remote location you were talking about too.
____________________________________________________
Cheers
--
Slávek
6:53 p.m.
On Sunday 20 of December 2020 22:01:33 Mike Bird via tde-users wrote:
On Sun December 20 2020 11:34:29 Slávek Banko via
tde-users wrote:
As I mentioned in the other mail in this thread - here it can be expected
that the size will continue to increase. So it is possible that 400 GiB
will not be enough again next year. Therefore, it seems like a good idea
that only a subset is published to the mirror system.
Current size and occupied space of the
partition:
/dev/mapper/lvm1-tde_data 400G 336G 65G 84% /srv/tde
The primary mirror is currently:
/dev/mapper/DISK-TDE 298G 250G 49G 84% /ARCHIVE/TDE
Since the upgrade a year ago we can handle 400G no problem so the
easiest solution might be just to mirror everything. I'm not sure
how much the secondaries can take but it's not like it's a massive
change.
I wonder if Michael
<mb_trinity_desktop(a)inet-design.com> would
be interested in becoming a mirror, which would use less primary
archive bandwidth than pulling backups directly.
Yes, this would definitely be good. If Michael has enough free capacity, he
could join as another official mirror.
--Mike
____________________________________________________
Cheers
--
Slávek
6:55 p.m.
On Sunday 20 December 2020 03:01:33 pm Mike Bird via tde-users wrote:
On Sun December 20 2020 11:34:29 Slávek Banko via
tde-users wrote:
Hi Mike,
I asked Tim about becoming a mirror years ago, never got a response...
So, yes there is interest, but first I'd need to know what's the maximum
monthly bandwidth I'd expect to see?
All of my pipes are things like 10 TB / month on a 1Gbps port, so if it's
under 1TB / month easy peasy.
My least used server has 275GB of never used, free space, so if at all
possible I’d like to stay within that limit. If not, I’ll look at the loads
on my other ones and pick one of those (most have TB drives).
Best,
Michael
Current size and occupied space of the
partition:
/dev/mapper/lvm1-tde_data 400G 336G 65G 84% /srv/tde
The primary mirror is currently:
/dev/mapper/DISK-TDE 298G 250G 49G 84% /ARCHIVE/TDE
Since the upgrade a year ago we can handle 400G no problem so the
easiest solution might be just to mirror everything. I'm not sure
how much the secondaries can take but it's not like it's a massive
change.
I wonder if Michael would
be interested in becoming a mirror, which would use less primary
archive bandwidth than pulling backups directly.
7:18 p.m.
On Sunday 20 December 2020 08:53:16 pm Slávek Banko via tde-users wrote:
K, cap for next 12-18 months is 600GB? Doable.
Taking this to private mail, no need to keep spaming the users list ;)
Best,
Michael
I wonder if
Michael <mb_trinity_desktop(a)inet-design.com> would
be interested in becoming a mirror, which would use less primary
archive bandwidth than pulling backups directly.
Yes, this would definitely be good. If Michael has enough free capacity, he
could join as another official mirror.
8:23 p.m.
On Sun December 20 2020 18:55:46 Michael via tde-users wrote:
So, yes there is interest, but first I'd need to
know what's the maximum
monthly bandwidth I'd expect to see?
Primary mirror has 4TB/month bandwidth and has used 876GB about two thirds
of the way thru this month. So about 1.3TB in a quiet month for the primary
mirror. Secondary mirrors will probably see a third of that. There's more
when a release is finalizing - maybe double or triple. So a 2TB/month cap
should be OK for a secondary mirror and you might even get by with 1TB/month.
CAVEAT: I'm wondering if the proposed broadening of mirror scope to full
backup will expose more volatile files which would increase bandwidth.
--Mike
8:24 p.m.
On Sun December 20 2020 19:18:03 Michael via tde-users wrote:
K, cap for next 12-18 months is 600GB? Doable.
Taking this to private mail, no need to keep spaming the users list ;)
Keeping this on list for the archive for the benefit of future historians.
The primary mirror currently has 400GB assigned to TDE but it's an 800GB
VM with Linux and LVM and Apache and not much else so if we can keep TDE
to 600GB or so there shouldn't be a problem.
If Slávek keeps a backup of his rsync filters we could maybe use them if
any secondary mirrors complain about the new size.
---Mike
21 Dec
21 Dec
12:28 a.m.
On Monday 21 of December 2020 05:24:51 Mike Bird via tde-users wrote:
On Sun December 20 2020 19:18:03 Michael via tde-users
wrote:
Ok, that sounds good, so I will gradually enable the directories that are
now excluded. We will see how the traffic will increase as soon as the PTB
repository is available on the mirror system.
Cheers
--
Slávek
K, cap for next 12-18 months is 600GB? Doable.
Taking this to private mail, no need to keep spaming the users list ;)
Keeping this on list for the archive for the benefit of future
historians.
The primary mirror currently has 400GB assigned to TDE but it's an 800GB
VM with Linux and LVM and Apache and not much else so if we can keep TDE
to 600GB or so there shouldn't be a problem.
If Slávek keeps a backup of his rsync filters we could maybe use them if
any secondary mirrors complain about the new size.
---Mike
____________________________________________________
9:40 a.m.
On Sunday 20 December 2020 10:24:51 pm Mike Bird via tde-users wrote:
On Sun December 20 2020 19:18:03 Michael via tde-users
wrote:
Okay, makes sense.
K, cap for next 12-18 months is 600GB? Doable.
Taking this to private mail, no need to keep spaming the users list ;)
Keeping this on list for the archive for the benefit of future historians. The primary mirror currently has 400GB assigned to TDE
but it's an 800GB
VM with Linux and LVM and Apache and not much else so if we can keep TDE
to 600GB or so there shouldn't be a problem.
If Slávek keeps a backup of his rsync filters we could maybe use them if
any secondary mirrors complain about the new size.
Ignore the NVMe host from earlier, they’re nice people, but they don't
customize their plans.
If this meets the needs for a mirror:
http://turnkeyinternet.net/linux-cpanel-web-hosting/
C10 cPanel Hosting
$2.49per month (it’s $90 for 3 years.)
I’ll go talk with them and see if we fit their ‘Unlimited’ using these specs.
Max HD: 800GB
Max Data: 3TB/month
Pipe Needs: 50Mbps *
* They say Gbps, but does a mirror needs that? If not, let them cap data
rates for the storage size?
As we’ve had a lot of different numbers thrown around, Mike/Slávek please
verify the above.
Ref:
50 Mbit/s = ~16 TB per month
# # #
Alternatively:
In March we’re switching to T-Mobile 5G internet at our house and doing away
with cable TV entirely. To support that, I’m going to build a ~30TB file
server for the house.
* 1U Xeon server ($150) w/ 30TB of drives ($160), plus other junk (~$100).
The file server certainly won’t be ‘fast,’ but the up/down bandwidth is 100+
Mbps.
I will have to look into T-Mobile’s TOS, but with the plethora of Plex servers
out their I don’t really see any issues.
The other thing would be the use of a dynamic DNS, but would the mirror
software in use by TDE care or have issues with that?
All for now, best,
Michael
11:49 a.m.
On Mon December 21 2020 09:40:43 Michael via tde-users wrote:
Ignore the NVMe host from earlier, they’re nice
people, but they don't
customize their plans.
If this meets the needs for a mirror:
http://turnkeyinternet.net/linux-cpanel-web-hosting/
C10 cPanel Hosting
$2.49per month (it’s $90 for 3 years.)
I’ll go talk with them and see if we fit their ‘Unlimited’ using these
specs.
Generally "web hosting" plans only allow you to run pre-approved software.
There's a lot listed so I may have missed it but I didn't notice ssh server,
rsync server, or bash scripting.
Max HD: 800GB
Max Data: 3TB/month
Pipe Needs: 50Mbps *
* They say Gbps, but does a mirror needs that? If not, let them cap data
rates for the storage size?
As we’ve had a lot of different numbers thrown around, Mike/Slávek please
verify the above.
Ref:
50 Mbit/s = ~16 TB per month
# # #
Alternatively:
In March we’re switching to T-Mobile 5G internet at our house and doing
away with cable TV entirely. To support that, I’m going to build a ~30TB
file server for the house.
* 1U Xeon server ($150) w/ 30TB of drives ($160), plus other junk (~$100).
The file server certainly won’t be ‘fast,’ but the up/down bandwidth is
100+ Mbps.
I will have to look into T-Mobile’s TOS, but with the plethora of Plex
servers out their I don’t really see any issues.
The other thing would be the use of a dynamic DNS, but would the mirror
software in use by TDE care or have issues with that?
Dynamic DNS is not good for servers as it leads to a trade off between
excessive DNS traffic and temporary failures.
We're doing OK for mirrors right now so you might want to focus on your
new file server and later if you feel like creating a mirror you could
do so and once you have it working reliably contact Slávek to add you
to the list. FWIW I customize our installs to always pull from our
own mirror so this way I test it and get a heads-up if it's misbehaving.
I would suggest that if you get into mirroring TDE you join the TDE
devel mailing list so we can discuss mirroring there.
Best,
--Mike
22 Dec
22 Dec
7:02 p.m.
I would suggest that if you get into mirroring TDE you join the TDE
devel mailing list so we can discuss mirroring there.
Best,
--Mike
Thanks Mike and Michael for the excellent contribution and discussion. It will be good to
have a full mirror (and maybe
more than one) of the TDE archive on Slavek machine.
Nevertheless (and perhaps my fault for not explaining it well), the point of "moving
away from a single person
structure" is that we need the archive (and all TDE tools) on public machines
accessible from more than one person.
Slavek archive/computer is only accessible by him, so it is still a week point, although
having a full mirror will
improve the current situation.
Longer term we need to move all remaining services to public machines, definitely
something to discuss again after
sorting out the project license and donation status.
It would also be good to have at least 2 people who knows what are the required tasks for
maintenance of the various
services.
I do get involved in some of them, but mostly it is Slavek doing the maintenance of the
machines, while I focus more on
the development side. It would be good if more people steps up and help out Slavek's
on the maintenance burden :-)
Cheers
Michele
P.S: perhaps we should discuss on the devels ML rather than users?? so adding devs ML here
too
23 Dec
23 Dec
8:35 a.m.
On Tuesday 22 December 2020 09:02:31 pm Michele Calgaro via tde-users wrote:
Thanks Mike and Michael for the excellent contribution
and discussion. It
will be good to have a full mirror (and maybe more than one) of the TDE
archive on Slavek machine.
Hello All TDE Developers,
This started out as a discussion about a TDE archive, then it morphed into a
discussion about a TDE mirror. I believe I know what the difference is, and
I’m willing to do both, but possibly one of the TDE Dev’s should give us
users an ‘official’ definition of each? Just so everyone knows what is what.
Archive Definition:
Mirror Definition:
Why? A full up VPS, or webhosting service, is less than $100/year[1], which
I, and I’m sure several others on this user’s list, have no issue either
outright ‘donating’, or adding funds for, to TDE[2]. It also completely
solves:
the point
of "moving away from a single person structure" is that we need the archive
(and all TDE tools) on public machines accessible from more than one
person.
But at this point I am completely and utterly frustrated [3], as I have asked
repeatedly what the requirements for each are and have not been given a
concrete answer. If we (the users) don’t know what the requirements are, we
can’t help you (the dev’s).
Just to put everyone on the same page and not having anyone guessing, a ‘my
best guess at Archive needs’ for an example (see prior post for my guess at
Mirror):
Requirements for Archive:
Time frame good for: 24 Months
HD: 2TB (max)
Data: 3TB / month (max)
Pipe: 15 Mbps (min)
CPU: 1 (min)
Software: SSH, rsync (plus standard Linux stack, cron, bash, tar, etc.)
Number of unique Backups w/ time frames:
==Raw data==
1 raw rsync, synced once per day (2 AM)
==Tarballs==
Daily: 1 for ea of last 4 days (2 AM)
Weekly: 1 for ea of last 3 weeks (2 AM Sunday)
Monthly: 1 for ea of last 3 months (2 AM, 1st of Month)
Yearly: 1 for ea of last 2 years (2 AM, Jan 1)
(don’t like tarballs? Fill in the Software requirement above with whatever
floats your boat...)
# # #
Now, PLEASE!, fill in the blanks... (Do add a criteria if I’ve missed it)
Requirements for Archive:
Time frame good for:
HD:
Data:
Pipe:
CPU:
Software:
Number of unique Backups w/ time frames:
Requirements for Mirror:
Time frame good for:
HD:
Data:
Pipe:
CPU:
Software:
# # #
Okay, that’s it from me. If I can get something concrete to work from, I can
help you, otherwise, not so much.
May everyone have a Happy & Merry holiday season!
Best Regards All,
Michael
[1] Yes you have to know how to dig these up, as you won’t find them ‘retail.’
[2] I could give a rats ass about a non-profit donation receipt.
[3] This discussion time alone has ‘cost’ me 5-10x in time of just straight up
buying and donating a 3 year VPS to the ‘cause.’
PS: My source for bulk ~$15 / 3TB SAS2 drives had a warehouse fire, so that
alternative is off the table atm. I really didn’t want a 1U airplane in an
upstairs closet anyway (sour grapes)...
10:17 a.m.
Hi Michael,
As discussed we're now mirroring the whole thing so there's no
separate "backup" option any more. A key factor in mirroring
is to minimize traffic to and from the primary server.
On Wed December 23 2020 08:35:05 Michael via tde-users wrote:
Requirements for Mirror:
Time frame good for:
HD:
Data:
Pipe:
CPU:
Software:
Slávek posted the current size:
Current size and occupied space of the partition:
/dev/mapper/lvm1-tde_data 400G 336G 65G 84% /srv/tde
- and there was
some discussion of trying to stay under 600GB over
the next few years.
I posted the bandwidth info:
Primary mirror has 4TB/month bandwidth and has used
876GB about two thirds
of the way thru this month. So about 1.3TB in a quiet month for the
primary mirror. Secondary mirrors will probably see a third of that.
There's more when a release is finalizing - maybe double or triple. So a
2TB/month cap should be OK for a secondary mirror and you might even get by
with 1TB/month.
All other decisions are up to the mirror operator. I did post some
details of the script I use. Other mirrors use different scripts.
I use Debian Buster and Apache2 currently but that may change and
other mirrors may use entirely different software. The CPU and
RAM will of course depend upon the bandwidth which we have discussed
and the software which you choose but in reality it would probably
run fine on a twenty year old Pentium with a gig of RAM.
Please join the TDE devel list if you wish to discuss mirroring further
as that is a more appropriate forum.
Thanks,
--Mike
1242
days inactive
1246
days old
17 comments
4 participants
participants (4)
-
Michael -
Michele Calgaro -
Mike Bird -
Slávek Banko