Hi,
I note on this page
https://www.trinitydesktop.org/mirrorstatus.php
that you list our backend servers (copernicus and kuiper) separately. Could you change that to list a single URL please?
http://www.mirrorservice.org/sites/trinitydesktop.org/trinity/
(you may use https too if you prefer)
We're also testing out shorter URLs if this looks better:
http://trinitydesktop.mirrorservice.org/trinity/
(https available within the next 10 days)
The reason for this request is that these URLs go through our load balancer. This allows us to transparently handle node failure, do patching and maintenance, distribute traffic, etc, without impacting the user visible service. The backends are not intended to be accessed directly.
Thanks,
Tim.
Hi Tim,
Thank you for mirroring TDE!
I don't maintain that web page so this is not a direct response to your request but rather a request for further information.
FYI https://www.trinitydesktop.org/mirrorstatus.php is not the normal route whereby users retrieve files. Users work instead via a smart redirector http://mirror.ppa.trinitydesktop.org/
The status page is more of a debugging tool.
We monitor mirrors to determine when they have pulled updates. There have been several extended periods when copernicus and kuiper have held different files. We use this information in the smart redirector to only send user traffic to mirrors which are known to have the file the user needs.
I was wondering to what extent the UK Mirror Service load balancer addresses this as the last time I saw copernicus rsync from us was June 19th? And earlier in June there was a period of a few days when neither kuiper nor copernicus were updating?
Thanks again!
--Mike
--------------------------------------------------------------------- To unsubscribe, e-mail: trinity-devel-unsubscribe@lists.pearsoncomputing.net For additional commands, e-mail: trinity-devel-help@lists.pearsoncomputing.net Read list messages on the web archive: http://trinity-devel.pearsoncomputing.net/ Please remember not to top-post: http://trinity.pearsoncomputing.net/mailing_lists/#top-posting
Hi Mike,
The two servers sync independently. Whilst this is not an ideal situation, and can lead to them having slightly out of step content, it's the simplest approach and requires the least development and maintenance work. In theory they should sync within an hour of each other, so the likely reason for them to have different files for an extended period is that one of them is failing to sync. Obviously this is something that's quite easy to investigate at the time, but impossible to work out later. So if you see this please let us know at the time so we can see why it's happening.
The load balancer doesn't take much action regarding this. It doesn't know which mirrors are up to date or not, just whether they're functioning. However, it does have "stickiness" enabled, which means a client from the same IP should repeatedly hit the same backend unless there's a failure. So they should at least get consistent content.
My main gripe is about publishing the details of the backend systems directly. This leads to people using them not via the load balancer and then complaining when one is down. In your case you have a smart redirector in front, so presumably that'd deal with that problem without inconveniencing the users. I don't have any problem with you doing that but I'd be happier if the backend URLs weren't so obviously advertised.
The period of downtime in early June was because we had a complete data centre outage for the first time in over 10 years. The UPS systems were being upgraded and a failure occurred which powered everything off - the systems supposed to save us were the ones that failed us :-( The Mirror Service sadly isn't a priority system for the University, compared to core teaching and research systems, so we had to wait until full power was restored before we could turn it back on.
That issue also killed copernicus, or more specifically its RAID controller. I have now replaced that and spent the past couple of weeks rebuilding it and resyncing the data. This has proved quite time consuming but it is starting to come back online this weekend.
Tim.
On Sat, Jul 18, 2020 at 11:35:18AM -0700, Mike Bird wrote:
Hi Tim,
Thank you for mirroring TDE!
I don't maintain that web page so this is not a direct response to your request but rather a request for further information.
FYI https://www.trinitydesktop.org/mirrorstatus.php is not the normal route whereby users retrieve files. Users work instead via a smart redirector http://mirror.ppa.trinitydesktop.org/
The status page is more of a debugging tool.
We monitor mirrors to determine when they have pulled updates. There have been several extended periods when copernicus and kuiper have held different files. We use this information in the smart redirector to only send user traffic to mirrors which are known to have the file the user needs.
I was wondering to what extent the UK Mirror Service load balancer addresses this as the last time I saw copernicus rsync from us was June 19th? And earlier in June there was a period of a few days when neither kuiper nor copernicus were updating?
Thanks again!
--Mike
Hi Slávek, cc: Tim Bishop
Very helpful information here from Tim.
In short, could we change the mirror list on https://www.trinitydesktop.org/mirrorstatus.php to replace his copernicus and kuiper with just http://www.mirrorservice.org/sites/trinitydesktop.org/trinity/ while our redirector could continue to use copernicus and kuiper?
Primary mirror would not be affected by this change.
Thanks,
--Mike
On Sun July 19 2020 03:25:35 Tim Bishop wrote:
Hi Mike,
The two servers sync independently. Whilst this is not an ideal situation, and can lead to them having slightly out of step content, it's the simplest approach and requires the least development and maintenance work. In theory they should sync within an hour of each other, so the likely reason for them to have different files for an extended period is that one of them is failing to sync. Obviously this is something that's quite easy to investigate at the time, but impossible to work out later. So if you see this please let us know at the time so we can see why it's happening.
The load balancer doesn't take much action regarding this. It doesn't know which mirrors are up to date or not, just whether they're functioning. However, it does have "stickiness" enabled, which means a client from the same IP should repeatedly hit the same backend unless there's a failure. So they should at least get consistent content.
My main gripe is about publishing the details of the backend systems directly. This leads to people using them not via the load balancer and then complaining when one is down. In your case you have a smart redirector in front, so presumably that'd deal with that problem without inconveniencing the users. I don't have any problem with you doing that but I'd be happier if the backend URLs weren't so obviously advertised.
The period of downtime in early June was because we had a complete data centre outage for the first time in over 10 years. The UPS systems were being upgraded and a failure occurred which powered everything off - the systems supposed to save us were the ones that failed us :-( The Mirror Service sadly isn't a priority system for the University, compared to core teaching and research systems, so we had to wait until full power was restored before we could turn it back on.
That issue also killed copernicus, or more specifically its RAID controller. I have now replaced that and spent the past couple of weeks rebuilding it and resyncing the data. This has proved quite time consuming but it is starting to come back online this weekend.
Tim.
On Sat, Jul 18, 2020 at 11:35:18AM -0700, Mike Bird wrote:
Hi Tim,
Thank you for mirroring TDE!
I don't maintain that web page so this is not a direct response to your request but rather a request for further information.
FYI https://www.trinitydesktop.org/mirrorstatus.php is not the normal route whereby users retrieve files. Users work instead via a smart redirector http://mirror.ppa.trinitydesktop.org/
The status page is more of a debugging tool.
We monitor mirrors to determine when they have pulled updates. There have been several extended periods when copernicus and kuiper have held different files. We use this information in the smart redirector to only send user traffic to mirrors which are known to have the file the user needs.
I was wondering to what extent the UK Mirror Service load balancer addresses this as the last time I saw copernicus rsync from us was June 19th? And earlier in June there was a period of a few days when neither kuiper nor copernicus were updating?
Thanks again!
--Mike
--------------------------------------------------------------------- To unsubscribe, e-mail: trinity-devel-unsubscribe@lists.pearsoncomputing.net For additional commands, e-mail: trinity-devel-help@lists.pearsoncomputing.net Read list messages on the web archive: http://trinity-devel.pearsoncomputing.net/ Please remember not to top-post: http://trinity.pearsoncomputing.net/mailing_lists/#top-posting
Hi,
I read this thread a few minutes ago. I really like the short address http://trinitydesktop.mirrorservice.org/trinity/ - if such an address can be considered stable (it was mentioned that it is testing), I would prefer to use this URL.
We currently have the mirror list configuration as a simple configuration file, so it's easy to change - see:
https://mirror.git.trinitydesktop.org/gitea/TDE/tdemainweb-core/src/branch/m...
A small disadvantage compared to the current state will be that when using one record for your mirror system, the redirector may obtain inaccurate information about the state of the mirror, because it will not be certain which of your backends has been contacted. Specifically, the redirector may get a different state than the real user will subsequently receive.
Our redirector not only checks the presence of files on the selected mirror, but also checks the up-to-dateness of files before redirecting the user. When there are different states for the redirector and for the real user, there may be a redirector performing the redirection and the user receiving a code 404 ... which is exactly the state our redirector is trying to prevent. Checking the up-to-dateness of files is useful, for example, for APT packaging files. However, this should not be a major problem under normal circumstances.
Cheers
On Sun July 19 2020 11:53:17 Slávek Banko wrote:
A small disadvantage compared to the current state will be that when using one record for your mirror system, the redirector may obtain inaccurate information about the state of the mirror, because it will not be certain which of your backends has been contacted. Specifically, the redirector may get a different state than the real user will subsequently receive.
Hi Slávek,
I'm wondering whether we can decouple the web page status from the redirector. We could continue to use kuiper and copernicus for the redirector as they rsync from us independently.
However Tim would prefer that kuiper and copernicus not be publicized and is asking that the mirror status page show their load balancer rather than kuiper and copernicus.
Copernicus is currently rebuilding but even before that there were often distinct differences, e.g. 2461 files difference between kuiper and copernicus during primary mirror's daily report on June 9th:
0 files missing from copernicus.mirrorservice.org::trinitydesktop.org/ 2461 files missing from kuiper.mirrorservice.org::trinitydesktop.org/
--Mike
--------------------------------------------------------------------- To unsubscribe, e-mail: trinity-devel-unsubscribe@lists.pearsoncomputing.net For additional commands, e-mail: trinity-devel-help@lists.pearsoncomputing.net Read list messages on the web archive: http://trinity-devel.pearsoncomputing.net/ Please remember not to top-post: http://trinity.pearsoncomputing.net/mailing_lists/#top-posting
On Sunday 19 of July 2020 21:40:38 Mike Bird wrote:
On Sun July 19 2020 11:53:17 Slávek Banko wrote:
A small disadvantage compared to the current state will be that when using one record for your mirror system, the redirector may obtain inaccurate information about the state of the mirror, because it will not be certain which of your backends has been contacted. Specifically, the redirector may get a different state than the real user will subsequently receive.
Hi Slávek,
I'm wondering whether we can decouple the web page status from the redirector. We could continue to use kuiper and copernicus for the redirector as they rsync from us independently.
However Tim would prefer that kuiper and copernicus not be publicized and is asking that the mirror status page show their load balancer rather than kuiper and copernicus.
Copernicus is currently rebuilding but even before that there were often distinct differences, e.g. 2461 files difference between
kuiper and copernicus during primary mirror's daily report on June 9th:
0 files missing from copernicus.mirrorservice.org::trinitydesktop.org/ 2461 files missing from kuiper.mirrorservice.org::trinitydesktop.org/
--Mike
Hi,
I'm afraid that will require a slightly more complex solution.
Redirector selects the mirror primarily based on GeoIP information. Therefore, it is very likely that in the current configuration, the same backend will be selected repeatedly in the case of MirrorService.org (in case both urls are accessible and updated). And that would make it impossible to use a balancer on the MirrorService.org side.
The ideal solution would probably be to add the "base URL" and "backend URLs" option to the mirror configuration. A "base URL" would be used for address listing and redirection. To test the usability of the mirror, all available "backend URLs" would then be tested and the overall status of the mirror would be determined by whether at least one "backend URL" is accessible and at the same time whether all of the accessible "backend URLs" are updated.
What is your opinion on this idea? This should ensure reliable operation for use with the redirector, while allowing proper use of the balancer on the MirrorService.org side.
Cheers
On Sun July 19 2020 17:12:12 Slávek Banko wrote:
I'm afraid that will require a slightly more complex solution.
Redirector selects the mirror primarily based on GeoIP information. Therefore, it is very likely that in the current configuration, the same backend will be selected repeatedly in the case of MirrorService.org (in case both urls are accessible and updated). And that would make it impossible to use a balancer on the MirrorService.org side.
I would imagine that is OK as we are but a tiny portion of Tim's traffic.
The important thing is not to redirect user requests for 2,000 newish files to a server that does not have them yet.
The ideal solution would probably be to add the "base URL" and "backend URLs" option to the mirror configuration. A "base URL" would be used for address listing and redirection. To test the usability of the mirror, all available "backend URLs" would then be tested and the overall status of the mirror would be determined by whether at least one "backend URL" is accessible and at the same time whether all of the accessible "backend URLs" are updated.
Maybe some kind of "partial" status on the web page if some but not all backends are up?
What is your opinion on this idea? This should ensure reliable operation for use with the redirector, while allowing proper use of the balancer on the MirrorService.org side.
Sounds great to me. Let's see what Tim B thinks.
--Mike
--------------------------------------------------------------------- To unsubscribe, e-mail: trinity-devel-unsubscribe@lists.pearsoncomputing.net For additional commands, e-mail: trinity-devel-help@lists.pearsoncomputing.net Read list messages on the web archive: http://trinity-devel.pearsoncomputing.net/ Please remember not to top-post: http://trinity.pearsoncomputing.net/mailing_lists/#top-posting
On Monday 20 of July 2020 02:40:48 Mike Bird wrote:
On Sun July 19 2020 17:12:12 Slávek Banko wrote:
I'm afraid that will require a slightly more complex solution.
Redirector selects the mirror primarily based on GeoIP information. Therefore, it is very likely that in the current configuration, the same backend will be selected repeatedly in the case of MirrorService.org (in case both urls are accessible and updated). And that would make it impossible to use a balancer on the MirrorService.org side.
I would imagine that is OK as we are but a tiny portion of Tim's traffic.
The important thing is not to redirect user requests for 2,000 newish files to a server that does not have them yet.
The ideal solution would probably be to add the "base URL" and "backend URLs" option to the mirror configuration. A "base URL" would be used for address listing and redirection. To test the usability of the mirror, all available "backend URLs" would then be tested and the overall status of the mirror would be determined by whether at least one "backend URL" is accessible and at the same time whether all of the accessible "backend URLs" are updated.
Maybe some kind of "partial" status on the web page if some but not all backends are up?
For use in the redirector, it will be important if all accessible backends are updated - ie "green" status. If any of the backends is inaccessible, this is not an obstacle for use with the redirector.
For detailed information suitable for diagnostics, in addition to the summary status, the status of individual backends could also be listed there. Instead of the summary "partial", a real state could be seen.
For information, the redirector uses a special page mode to find usable mirrors, where it gets a plain list of usable urls - mirrors that are not ready are not listed:
https://www.trinitydesktop.org/mirrorstatus.php?mr
What is your opinion on this idea? This should ensure reliable operation for use with the redirector, while allowing proper use of the balancer on the MirrorService.org side.
Sounds great to me. Let's see what Tim B thinks.
--Mike
Cheers
Hi,
On Sun, Jul 19, 2020 at 08:53:17PM +0200, Slávek Banko wrote:
I read this thread a few minutes ago. I really like the short address http://trinitydesktop.mirrorservice.org/trinity/ - if such an address can be considered stable (it was mentioned that it is testing), I would prefer. to use this URL.
This address is stable and uses the load balancer. In just under one week from now it'll also support https - we're just waiting until Let's Encrypt will let us have more certificates :-)
On Sun, Jul 19, 2020 at 12:20:47PM -0700, Mike Bird wrote:
Copernicus is currently rebuilding but even before that there were often distinct differences, e.g. 2461 files difference between kuiper and copernicus during primary mirror's daily report on June 9th:
0 files missing from copernicus.mirrorservice.org::trinitydesktop.org/ 2461 files missing from kuiper.mirrorservice.org::trinitydesktop.org/
Copernicus is now back online. If you see issues like this in the future please get in touch and I'll investigate. Unless the files are very fresh (we sync every 6 hours - happy to change that if needed) this shouldn't happen.
On Mon, Jul 20, 2020 at 09:38:41AM +0200, Slávek Banko wrote:
On Monday 20 of July 2020 02:40:48 Mike Bird wrote:
On Sun July 19 2020 17:12:12 Slávek Banko wrote:
The ideal solution would probably be to add the "base URL" and "backend URLs" option to the mirror configuration. A "base URL" would be used for address listing and redirection. To test the usability of the mirror, all available "backend URLs" would then be tested and the overall status of the mirror would be determined by whether at least one "backend URL" is accessible and at the same time whether all of the accessible "backend URLs" are updated.
Maybe some kind of "partial" status on the web page if some but not all backends are up?
For use in the redirector, it will be important if all accessible backends are updated - ie "green" status. If any of the backends is inaccessible, this is not an obstacle for use with the redirector.
For detailed information suitable for diagnostics, in addition to the summary status, the status of individual backends could also be listed there. Instead of the summary "partial", a real state could be seen.
For information, the redirector uses a special page mode to find usable mirrors, where it gets a plain list of usable urls - mirrors that are not ready are not listed:
A solution that probed the backends and sent users to the load balanced address when the files were present on both mirrors would be fine. Although this does mean effectively not using our service if one backend has issues, but if you have other mirrors that's not really a problem.
I'd also be happy with a solution that meant you weren't publicising or directly giving users the backend addresses. If you have your own redirector that can effectively perform the same, or better, job as our load balancer then I'm fine with that, even if it primarily always selects the same backend. Since you're a relatively low traffic mirror I'm more concerned about availability rather than spreading load - I feel bad whenever users are affected by avoidable downtime, such as patching or hardware maintenance.
I suppose an ideal solution would combine the two; using the load balanced address when both backends have the files, but falling back to a single one in a case where only one has the files. That's likely heading in to the real of unnecessary effort though :-)
Tim.
On Sun July 19 2020 03:25:35 Tim Bishop wrote:
The two servers sync independently. Whilst this is not an ideal situation, and can lead to them having slightly out of step content, it's the simplest approach and requires the least development and maintenance work. In theory they should sync within an hour of each other, so the likely reason for them to have different files for an extended period is that one of them is failing to sync. Obviously this is something that's quite easy to investigate at the time, but impossible to work out later. So if you see this please let us know at the time so we can see why it's happening.
Hi Tim,
The primary mirror daily report issued an hour ago included this:
0 files missing from copernicus.mirrorservice.org::trinitydesktop.org/ 33495 files missing from kuiper.mirrorservice.org::trinitydesktop.org/
--Mike
On Sun July 19 2020 03:25:35 Tim Bishop wrote:
The two servers sync independently. Whilst this is not an ideal situation, and can lead to them having slightly out of step content, it's the simplest approach and requires the least development and maintenance work. In theory they should sync within an hour of each other, so the likely reason for them to have different files for an extended period is that one of them is failing to sync. Obviously this is something that's quite easy to investigate at the time, but impossible to work out later. So if you see this please let us know at the time so we can see why it's happening.
Reposting, as I had mistakenly sent this to the old TDE dev mailing list address.
Hi Tim,
Possible stall on Kuiper:
0 files missing from copernicus.mirrorservice.org::trinitydesktop.org/ 5259 files missing from kuiper.mirrorservice.org::trinitydesktop.org/
--Mike
-
Mike Bird -
Slávek Banko -
Tim Bishop