23 Dec
2020
23 Dec
'20
6:17 p.m.
Hi Michael,
As discussed we're now mirroring the whole thing so there's no
separate "backup" option any more. A key factor in mirroring
is to minimize traffic to and from the primary server.
On Wed December 23 2020 08:35:05 Michael via tde-users wrote:
Requirements for Mirror:
Time frame good for:
HD:
Data:
Pipe:
CPU:
Software:
Slávek posted the current size:
Current size and occupied space of the partition:
/dev/mapper/lvm1-tde_data 400G 336G 65G 84% /srv/tde
- and there was
some discussion of trying to stay under 600GB over
the next few years.
I posted the bandwidth info:
Primary mirror has 4TB/month bandwidth and has used
876GB about two thirds
of the way thru this month. So about 1.3TB in a quiet month for the
primary mirror. Secondary mirrors will probably see a third of that.
There's more when a release is finalizing - maybe double or triple. So a
2TB/month cap should be OK for a secondary mirror and you might even get by
with 1TB/month.
All other decisions are up to the mirror operator. I did post some
details of the script I use. Other mirrors use different scripts.
I use Debian Buster and Apache2 currently but that may change and
other mirrors may use entirely different software. The CPU and
RAM will of course depend upon the bandwidth which we have discussed
and the software which you choose but in reality it would probably
run fine on a twenty year old Pentium with a gig of RAM.
Please join the TDE devel list if you wish to discuss mirroring further
as that is a more appropriate forum.
Thanks,
--Mike
26 Dec
26 Dec
7:26 p.m.
New subject: [tde-users] Re: Migration of services - 10. primary archive
Dne Wednesday 23 of December 2020 19:17:40 Mike Bird via tde-devels
napsal(a):
Hi Michael,
As discussed we're now mirroring the whole thing so there's no
separate "backup" option any more. A key factor in mirroring
is to minimize traffic to and from the primary server.
I gradually enabled the parts that were excluded, and a moment ago the
synchronization was completed after enabling the last part. Currently,
only directories with reprepro metadata that are subject to build
management remain blocked.
On Wed December 23 2020 08:35:05 Michael via tde-users
wrote:
Now you should also observe the current sizes on the mirror:
$ du -chs /srv/tde/archive/*/
25G /srv/tde/archive/cdimages/
235G /srv/tde/archive/deb/
34G /srv/tde/archive/releases/
37G /srv/tde/archive/rpm/
0 /srv/tde/archive/trinity/
0 /srv/tde/archive/trinity-builddeps-r14.0.0/
0 /srv/tde/archive/trinity-r14.0.0/
1.3G /srv/tde/archive/ulab/
331G total
Requirements for Mirror:
Time frame good for:
HD:
Data:
Pipe:
CPU:
Software:
Slávek posted the current size:
Current size and occupied space of the
partition:
/dev/mapper/lvm1-tde_data 400G 336G 65G 84% /srv/tde
- and there was some discussion of trying to stay under 600GB over
the next few years.
I posted the bandwidth info:
Mike, thank you for the information regarding the primary mirror.
For now, Michael, if you have enough capacity and want to donate it to
Trinity, the best way is to join as a new mirror that syncs data from
Mike's primary mirror. If you have the capacity to keep backups of the
contents of the mirror, well, that may come in handy.
For backups, personally, I am very happy to abandon the principles used for
tapes - that is, tarballs, because it seems to me to be very impractical
in all respects. For disk-to-disk backups, it seems to me to be the best
way use of the hardlink principle. This adds great flexibility to backups
in a very natural way and, above all, eliminates the need to distinguish
between full × incremental backups, as all backups can be used as full
while all are performed as incremental. I use dirvish for this purpose.
For further considerations involving any paid services, we should wait
until the acceptance of donations and project funds is resolved. Without
it, it is premature to devote time to such discussions.
Cheers
--
Slávek
Primary mirror has 4TB/month bandwidth and has
used 876GB about two
thirds of the way thru this month. So about 1.3TB in a quiet month
for the primary mirror. Secondary mirrors will probably see a third
of that. There's more when a release is finalizing - maybe double or
triple. So a 2TB/month cap should be OK for a secondary mirror and
you might even get by with 1TB/month.
All other decisions are up to the mirror operator. I did post some
details of the script I use. Other mirrors use different scripts.
I use Debian Buster and Apache2 currently but that may change and
other mirrors may use entirely different software. The CPU and
RAM will of course depend upon the bandwidth which we have discussed
and the software which you choose but in reality it would probably
run fine on a twenty year old Pentium with a gig of RAM.
Please join the TDE devel list if you wish to discuss mirroring further
as that is a more appropriate forum.
Thanks,
--Mike
____________________________________________________
8:08 p.m.
New subject: [tde-users] Re: Migration of services - 10. primary archive
On Sat December 26 2020 11:26:18 Slávek Banko via tde-devels wrote:
Now you should also observe the current sizes on the
mirror:
$ du -chs /srv/tde/archive/*/
25G /srv/tde/archive/cdimages/
235G /srv/tde/archive/deb/
34G /srv/tde/archive/releases/
37G /srv/tde/archive/rpm/
0 /srv/tde/archive/trinity/
0 /srv/tde/archive/trinity-builddeps-r14.0.0/
0 /srv/tde/archive/trinity-r14.0.0/
1.3G /srv/tde/archive/ulab/
331G total
Close enough:
# du -chs -- *
25G cdimages
233G deb
34G releases
38G rpm
4.0K -synctime
4.0K trinity
4.0K trinity-builddeps-r14.0.0
4.0K trinity-r14.0.0
1.3G ulab
328G total
Differences most likely due to differing directory sizes due to
different numbers of deleted entries. (Could also be different
block sizes or file systems or file system options.)
FYI copernicus and kuiper are syncing fine at the new size. fau.de
appears to have indigestion but let's give them a while to resolve it.
--Mike
27 Dec
27 Dec
9:12 a.m.
New subject: [tde-users] Re: Migration of services - 10. primary archive
On Sat December 26 2020 12:08:22 Mike Bird via tde-devels wrote:
FYI copernicus and kuiper are syncing fine at the new
size. fau.de
appears to have indigestion but let's give them a while to resolve it.
fau.de is now fully synced.
--Mike
1215
days inactive
1219
days old
3 comments
2 participants
participants (2)
-
Mike Bird -
Slávek Banko