On Thu June 15 2023 19:25:53 Michele Calgaro wrote:
if I remember correctly, we are still providing R14.0.13 on the mirror to ease user migration, am I wrong? So R14.0.13 and R14.1.0 would take up lot of space. If that is the case, would make more sense to drop off R14.0.13 from the mirror and keep trinity-testing? What do you think?
If my one-liner to analyze web logs on the primary mirror while filtering out the redirector's status checks is accurate than I think you're right Michele. Here's the top 20 by second-level directory, counting accesses not bytes, over the last 15 days or so.
145327 /deb/trinity-sb 129511 /deb/trinity-testing 48093 /deb/trinity-r14.0.x 25878 /deb/trinity-r14.1.x 22105 /rpm/osstw 21067 /ulab/ubuntu 17410 /deb/trinity-builddeps-r14.0.x 11493 /rpm/f38 10715 /trinity-r14.0.0/debian 10461 /rpm/el6 10172 /rpm/el5 9957 /trinity-r14.0.0/ubuntu 9336 /rpm/el8 9008 /rpm/el7 8930 /rpm/pclinuxos 7487 /rpm/mga8 7233 /rpm/opensuse15.5 6659 /trinity/rpm 6291 /rpm/mga9 5885 /rpm/el9
Unless someone comes up with a better idea in the next 24 hours I'll mask r14.0.x and builddeps on the mirror (on the assumption that most of that traffic will soon move from 14.0.x to 14.1.x) and restore trinity-testing.
FWIW here are the directory sizes:
2.0G ./releases/R14.0.8 2.4G ./releases/R14.0.0 1.9G ./releases/R14.0.11 1.9G ./releases/R14.0.7 2.3G ./releases/R14.0.5 2.3G ./releases/R14.0.4 1.9G ./releases/R14.1.0 2.0G ./releases/R14.0.9 1.9G ./releases/R14.0.6 5.3G ./releases/R14.0.1 2.4G ./releases/R14.0.3 1.9G ./releases/R14.0.12 2.0G ./releases/R14.0.10 1.9G ./releases/R14.0.13 2.4G ./releases/R14.0.2 34G ./releases 650M ./archlinux/x86_64 299M ./archlinux/aarch64 296M ./archlinux/armv7h 432M ./archlinux/powerpc64le 1.7G ./archlinux 78G ./deb/trinity-r14.1.x 52G ./deb/trinity-sb 115G ./deb/trinity-r14.0.x 1.9G ./deb/trinity-builddeps-r14.0.x 246G ./deb 1.3G ./ulab/ubuntu 1.3G ./ulab 3.6G ./rpm/osstw 2.4G ./rpm/el7 2.2G ./rpm/opensuse15.5 2.5G ./rpm/el9 3.5G ./rpm/mga8 2.5G ./rpm/el8 2.5G ./rpm/f37 3.3G ./rpm/el6 4.2G ./rpm/el5 1.7G ./rpm/pclinuxos 2.5G ./rpm/f38 2.2G ./rpm/opensuse15.4 2.5G ./rpm/mga9 2.5G ./rpm/f36 38G ./rpm 321G .
--Mike
On Friday 16 of June 2023 05:51:45 Mike Bird via tde-devels wrote:
On Thu June 15 2023 19:25:53 Michele Calgaro wrote:
if I remember correctly, we are still providing R14.0.13 on the mirror to ease user migration, am I wrong? So R14.0.13 and R14.1.0 would take up lot of space. If that is the case, would make more sense to drop off R14.0.13 from the mirror and keep trinity-testing? What do you think?
If my one-liner to analyze web logs on the primary mirror while filtering out the redirector's status checks is accurate than I think you're right Michele. Here's the top 20 by second-level directory, counting accesses not bytes, over the last 15 days or so.
145327 /deb/trinity-sb 129511 /deb/trinity-testing 48093 /deb/trinity-r14.0.x 25878 /deb/trinity-r14.1.x 22105 /rpm/osstw 21067 /ulab/ubuntu 17410 /deb/trinity-builddeps-r14.0.x 11493 /rpm/f38 10715 /trinity-r14.0.0/debian 10461 /rpm/el6 10172 /rpm/el5 9957 /trinity-r14.0.0/ubuntu 9336 /rpm/el8 9008 /rpm/el7 8930 /rpm/pclinuxos 7487 /rpm/mga8 7233 /rpm/opensuse15.5 6659 /trinity/rpm 6291 /rpm/mga9 5885 /rpm/el9
Unless someone comes up with a better idea in the next 24 hours I'll mask r14.0.x and builddeps on the mirror (on the assumption that most of that traffic will soon move from 14.0.x to 14.1.x) and restore trinity-testing.
FWIW here are the directory sizes:
2.0G ./releases/R14.0.8 2.4G ./releases/R14.0.0 1.9G ./releases/R14.0.11 1.9G ./releases/R14.0.7 2.3G ./releases/R14.0.5 2.3G ./releases/R14.0.4 1.9G ./releases/R14.1.0 2.0G ./releases/R14.0.9 1.9G ./releases/R14.0.6 5.3G ./releases/R14.0.1 2.4G ./releases/R14.0.3 1.9G ./releases/R14.0.12 2.0G ./releases/R14.0.10 1.9G ./releases/R14.0.13 2.4G ./releases/R14.0.2 34G ./releases 650M ./archlinux/x86_64 299M ./archlinux/aarch64 296M ./archlinux/armv7h 432M ./archlinux/powerpc64le 1.7G ./archlinux 78G ./deb/trinity-r14.1.x 52G ./deb/trinity-sb 115G ./deb/trinity-r14.0.x 1.9G ./deb/trinity-builddeps-r14.0.x 246G ./deb 1.3G ./ulab/ubuntu 1.3G ./ulab 3.6G ./rpm/osstw 2.4G ./rpm/el7 2.2G ./rpm/opensuse15.5 2.5G ./rpm/el9 3.5G ./rpm/mga8 2.5G ./rpm/el8 2.5G ./rpm/f37 3.3G ./rpm/el6 4.2G ./rpm/el5 1.7G ./rpm/pclinuxos 2.5G ./rpm/f38 2.2G ./rpm/opensuse15.4 2.5G ./rpm/mga9 2.5G ./rpm/f36 38G ./rpm 321G .
--Mike ____________________________________________________
Hi Mike,
thank you for your supervision and maintenance of the mirror. During that time there were several things:
Before the release of Bookworm, I noticed that a staging repository was accidentally published on the mirror, where the final packages for Bookworm and Raspbian Bookworm were being prepared. This caused these packages to be there twice for a while. In any case, publishing the final packages for Bookworm and Raspbian Bookworm means that a few more GiB have been consumed there.
During the preparation of the special update of TQt3 and tdelibs for R14.1.0, there was another temporary increase in the occupied space. However, it has already been cleaned up again.
We dropped Stretch, Xenial and Impish in PTB and Impish in PSB. This led to some reduction in the space occupied. However, there will be separate packages for Trixie and Mantic in a while, so it will take up some space again.
You may recall that when there was a plan to create a new repository for R14.1.x, I mentioned that it would later represent an increase in volume of about 80 GiB. In short, it could be said that increasing the size of the mirror is exactly as expected.
At the time R14.1.0 was being prepared, there was a very unusual situation where the volume was reduced because the PTB and PSB became identical. As well as the subsequently released R14.1.0 in the new repository. But at the moment of release, new packages for PSB and new packages for PTB started to appear there. That's why these repositories started to take on their usual size again. Even at this point, there are still some packages identical to R14.1.x, PSB and PTB, so you can still expect to consume more space there. So it's not an incorrect or unexpected state.
To reduce the volume on the mirror, we must therefore consider excluding some older data from the mirror. Originally, I assumed that the R14.0.x repository would be removed from the mirror after the release of R14.1.1. If it seems appropriate, we can exclude it earlier. What is your opinion? Probably no one will mind if we exclude the source tarballs for the R14.0.x series. This will help free up some additional space. For cdimmages, I can set excludes on the primary archive.
It will certainly be useful to organize the exclusion of the old data so that the PTB repository can be back on the mirrors.
Cheers
On Mon June 19 2023 16:40:06 Slávek Banko via tde-devels wrote:
To reduce the volume on the mirror, we must therefore consider excluding some older data from the mirror. Originally, I assumed that the R14.0.x repository would be removed from the mirror after the release of R14.1.1. If it seems appropriate, we can exclude it earlier. What is your opinion? Probably no one will mind if we exclude the source tarballs for the R14.0.x series. This will help free up some additional space. For cdimmages, I can set excludes on the primary archive.
It will certainly be useful to organize the exclusion of the old data so that the PTB repository can be back on the mirrors.
I'm in an odd state where I prepay for the mirror VPS a year at a time and there are still several more months left but the provider is discontinuing this particular service so I will have to switch to another service by November.
For now the mirrors have PSB and PTB and R14.1.x but R14.0.x has been removed.
The source tarballs for R14.1.x amount to only 2GB of the 72GB of R14.1.x. I doubt that excluding sources from R14.0.x would reduce the 115GB of R14.0.x enough for it to fit.
Meantime I'm keeping my eyes open for good deals on large VPSs. They don't come up often. This will determine whether I can resume mirroring R14.0.x.
--Mike
On Tuesday 20 of June 2023 02:22:06 Mike Bird via tde-devels wrote:
On Mon June 19 2023 16:40:06 Slávek Banko via tde-devels wrote:
To reduce the volume on the mirror, we must therefore consider excluding some older data from the mirror. Originally, I assumed that the R14.0.x repository would be removed from the mirror after the release of R14.1.1. If it seems appropriate, we can exclude it earlier. What is your opinion? Probably no one will mind if we exclude the source tarballs for the R14.0.x series. This will help free up some additional space. For cdimmages, I can set excludes on the primary archive.
It will certainly be useful to organize the exclusion of the old data so that the PTB repository can be back on the mirrors.
I'm in an odd state where I prepay for the mirror VPS a year at a time and there are still several more months left but the provider is discontinuing this particular service so I will have to switch to another service by November.
For now the mirrors have PSB and PTB and R14.1.x but R14.0.x has been removed.
Oh, I see, no need to think about it, it's already been excluded... I'll add an exclude on the primary archive, so that it matches :)
The source tarballs for R14.1.x amount to only 2GB of the 72GB of R14.1.x. I doubt that excluding sources from R14.0.x would reduce the 115GB of R14.0.x enough for it to fit.
I meant the source tarballs located in the /releases/ folder. There it represents a larger volume than just 2 GiB.
Meantime I'm keeping my eyes open for good deals on large VPSs. They don't come up often. This will determine whether I can resume mirroring R14.0.x.
The intention here was to exclude R14.0.x from mirroring. It just happened sooner than I expected. But it does not matter. It is useful to keep the mirror at some reasonably acceptable size. I hope that you will soon manage to find an alternative for your VPS. Thank you for your contribution to the project!
--Mike ____________________________________________________
Cheers
On Tue June 20 2023 17:38:52 Slávek Banko via tde-devels wrote:
The intention here was to exclude R14.0.x from mirroring. It just happened sooner than I expected. But it does not matter. It is useful to keep the mirror at some reasonably acceptable size. I hope that you will soon manage to find an alternative for your VPS. Thank you for your contribution to the project!
I have a larger replacement mirror now but it will take a few days to configure and sync it and replace the old mirror.
--Mike
The exclude-to-mirror setting is failing to mirror a important thing: trinity-keyring.deb is not mirrored. Only three items inside "deb" are now mirrored: trinity-r14.1.x trinity-sb trinity-testing See: http://tde-mirror.yosemite.net/trinity/deb/ http://kuiper.mirrorservice.org/sites/trinitydesktop.org/trinity/deb/ http://copernicus.mirrorservice.org/sites/trinitydesktop.org/trinity/deb/ http://ftp.fau.de/trinity/deb/ https://ftp.fau.de/trinity/deb/ http://tde-mirror-01.inet-design.com/trinity/deb/ I note a exception: http://mirror.nasutek.com/trinity/deb/ does mirror trinity-keyring.deb Please do "deb/trinity-keyring.deb" mirrorable. Thank you. En miércoles, 21 de junio de 2023, 02:40:56 CEST, Slávek Banko via tde-devels devels@trinitydesktop.org escribió:
On Tuesday 20 of June 2023 02:22:06 Mike Bird via tde-devels wrote:
On Mon June 19 2023 16:40:06 Slávek Banko via tde-devels wrote:
To reduce the volume on the mirror, we must therefore consider excluding some older data from the mirror. Originally, I assumed that the R14.0.x repository would be removed from the mirror after the release of R14.1.1. If it seems appropriate, we can exclude it earlier. What is your opinion? Probably no one will mind if we exclude the source tarballs for the R14.0.x series. This will help free up some additional space. For cdimmages, I can set excludes on the primary archive.
It will certainly be useful to organize the exclusion of the old data so that the PTB repository can be back on the mirrors.
I'm in an odd state where I prepay for the mirror VPS a year at a time and there are still several more months left but the provider is discontinuing this particular service so I will have to switch to another service by November.
For now the mirrors have PSB and PTB and R14.1.x but R14.0.x has been removed.
Oh, I see, no need to think about it, it's already been excluded... I'll add an exclude on the primary archive, so that it matches :)
The source tarballs for R14.1.x amount to only 2GB of the 72GB of R14.1.x. I doubt that excluding sources from R14.0.x would reduce the 115GB of R14.0.x enough for it to fit.
I meant the source tarballs located in the /releases/ folder. There it represents a larger volume than just 2 GiB.
Meantime I'm keeping my eyes open for good deals on large VPSs. They don't come up often. This will determine whether I can resume mirroring R14.0.x.
The intention here was to exclude R14.0.x from mirroring. It just happened sooner than I expected. But it does not matter. It is useful to keep the mirror at some reasonably acceptable size. I hope that you will soon manage to find an alternative for your VPS. Thank you for your contribution to the project!
--Mike ____________________________________________________
Cheers
On Sat June 24 2023 02:44:52 Alexis PM via tde-devels wrote:
The exclude-to-mirror setting is failing to mirror a important thing: trinity-keyring.deb is not mirrored. Only three items inside "deb" are now mirrored: trinity-r14.1.x trinity-sb trinity-testing See: http://tde-mirror.yosemite.net/trinity/deb/ http://kuiper.mirrorservice.org/sites/trinitydesktop.org/trinity/deb/ http://copernicus.mirrorservice.org/sites/trinitydesktop.org/trinity/deb/ http://ftp.fau.de/trinity/deb/ https://ftp.fau.de/trinity/deb/ http://tde-mirror-01.inet-design.com/trinity/deb/ I note a exception: http://mirror.nasutek.com/trinity/deb/ does mirror trinity-keyring.deb Please do "deb/trinity-keyring.deb" mirrorable. Thank you.
Hi Alexis,
Thank you for bringing this to our attention.
It turns out that trinity-keyring.deb is actually mirrored but it's just a symlink into the r14.0.x repository. As r14.0.x is not currently mirrored the symlink is dead and not shown.
I've just finished changing the primary mirror to a larger VPS. After a day or two to check it is running smoothly I will resume mirroring r14.0.x.
Meanwhile Slávek may want to consider changing the master copy of the link to point into the current r14.1.x repository.
FWIW the "official" path to TDE is http://mirror.ppa.trinitydesktop.org/ which is a redirector which knows which mirrors currently have which files. It can always find the keyring for you.
--Mike
On Saturday 24 of June 2023 12:10:31 Mike Bird via tde-devels wrote:
On Sat June 24 2023 02:44:52 Alexis PM via tde-devels wrote:
The exclude-to-mirror setting is failing to mirror a important thing: trinity-keyring.deb is not mirrored. Only three items inside "deb" are now mirrored: trinity-r14.1.x trinity-sb trinity-testing See: http://tde-mirror.yosemite.net/trinity/deb/ http://kuiper.mirrorservice.org/sites/trinitydesktop.org/trinity/deb/ http://copernicus.mirrorservice.org/sites/trinitydesktop.org/trinity/d eb/ http://ftp.fau.de/trinity/deb/ https://ftp.fau.de/trinity/deb/ http://tde-mirror-01.inet-design.com/trinity/deb/ I note a exception: http://mirror.nasutek.com/trinity/deb/ does mirror trinity-keyring.deb Please do "deb/trinity-keyring.deb" mirrorable. Thank you.
Hi Alexis,
Thank you for bringing this to our attention.
It turns out that trinity-keyring.deb is actually mirrored but it's just a symlink into the r14.0.x repository. As r14.0.x is not currently mirrored the symlink is dead and not shown.
I've just finished changing the primary mirror to a larger VPS. After a day or two to check it is running smoothly I will resume mirroring r14.0.x.
Meanwhile Slávek may want to consider changing the master copy of the link to point into the current r14.1.x repository.
FWIW the "official" path to TDE is http://mirror.ppa.trinitydesktop.org/ which is a redirector which knows which mirrors currently have which files. It can always find the keyring for you.
--Mike ____________________________________________________
Hi,
yes, Mike accurately identified the problem. I did not realize that symlink referred to R14.0.x repository, and that after exclude R14.0.x will be invalid on mirrors. I changed symlink to refer to R14.1.x repository.
Cheers
yes, Mike accurately identified the problem. I did not realize that symlink referred to R14.0.x repository, and that after exclude R14.0.x will be invalid on mirrors. I changed symlink to refer to R14.1.x repository.
would be even better if the keyring package was not bound to a specific branch but just general, IMO. Cheers Michele
-
Alexis PM -
Michele Calgaro -
Mike Bird -
Slávek Banko