Hi Tim, Thank you for mirroring TDE! I don't maintain that web page so this is not a direct response to your request but rather a request for further information. FYI https://www.trinitydesktop.org/mirrorstatus.php is not the normal route whereby users retrieve files. Users work instead via a smart redirector http://mirror.ppa.trinitydesktop.org/ The status page is more of a debugging tool. We monitor mirrors to determine when they have pulled updates. There have been several extended periods when copernicus and kuiper have held different files. We use this information in the smart redirector to only send user traffic to mirrors which are known to have the file the user needs. I was wondering to what extent the UK Mirror Service load balancer addresses this as the last time I saw copernicus rsync from us was June 19th? And earlier in June there was a period of a few days when neither kuiper nor copernicus were updating? Thanks again! --Mike

Hi Slávek, cc: Tim Bishop Very helpful information here from Tim. In short, could we change the mirror list on https://www.trinitydesktop.org/mirrorstatus.php to replace his copernicus and kuiper with just http://www.mirrorservice.org/sites/trinitydesktop.org/trinity/ while our redirector could continue to use copernicus and kuiper? Primary mirror would not be affected by this change. Thanks, --Mike On Sun July 19 2020 03:25:35 Tim Bishop wrote: > Hi Mike, > > The two servers sync independently. Whilst this is not an ideal > situation, and can lead to them having slightly out of step content, > it's the simplest approach and requires the least development and > maintenance work. In theory they should sync within an hour of each > other, so the likely reason for them to have different files for an > extended period is that one of them is failing to sync. Obviously this > is something that's quite easy to investigate at the time, but > impossible to work out later. So if you see this please let us know at > the time so we can see why it's happening. > > The load balancer doesn't take much action regarding this. It doesn't > know which mirrors are up to date or not, just whether they're > functioning. However, it does have "stickiness" enabled, which means a > client from the same IP should repeatedly hit the same backend unless > there's a failure. So they should at least get consistent content. > > My main gripe is about publishing the details of the backend systems > directly. This leads to people using them not via the load balancer > and then complaining when one is down. In your case you have a smart > redirector in front, so presumably that'd deal with that problem > without inconveniencing the users. I don't have any problem with you > doing that but I'd be happier if the backend URLs weren't so obviously > advertised. > > The period of downtime in early June was because we had a complete > data centre outage for the first time in over 10 years. The UPS > systems were being upgraded and a failure occurred which powered > everything off - the systems supposed to save us were the ones that > failed us :-( The Mirror Service sadly isn't a priority system for the > University, compared to core teaching and research systems, so we had > to wait until full power was restored before we could turn it back on. > > That issue also killed copernicus, or more specifically its RAID > controller. I have now replaced that and spent the past couple of > weeks rebuilding it and resyncing the data. This has proved quite time > consuming but it is starting to come back online this weekend. > > Tim. > > On Sat, Jul 18, 2020 at 11:35:18AM -0700, Mike Bird wrote: > > Hi Tim, > > > > Thank you for mirroring TDE! > > > > I don't maintain that web page so this is not a direct response > > to your request but rather a request for further information. > > > > FYI https://www.trinitydesktop.org/mirrorstatus.php is not the > > normal route whereby users retrieve files. Users work instead > > via a smart redirector http://mirror.ppa.trinitydesktop.org/ > > > > The status page is more of a debugging tool. > > > > We monitor mirrors to determine when they have pulled updates. > > There have been several extended periods when copernicus and > > kuiper have held different files. We use this information in > > the smart redirector to only send user traffic to mirrors which > > are known to have the file the user needs. > > > > I was wondering to what extent the UK Mirror Service load balancer > > addresses this as the last time I saw copernicus rsync from us > > was June 19th? And earlier in June there was a period of a few days > > when neither kuiper nor copernicus were updating? > > > > Thanks again! > > > > --Mike

On Sun July 19 2020 11:53:17 Slávek Banko wrote: Hi Slávek, I'm wondering whether we can decouple the web page status from the redirector.  We could continue to use kuiper and copernicus for the redirector as they rsync from us independently. However Tim would prefer that kuiper and copernicus not be publicized and is asking that the mirror status page show their load balancer rather than kuiper and copernicus. Copernicus is currently rebuilding but even before that there were often distinct differences, e.g. 2461 files difference between kuiper and copernicus during primary mirror's daily report on June 9th: --Mike

On Sun July 19 2020 17:12:12 Slávek Banko wrote: I would imagine that is OK as we are but a tiny portion of Tim's traffic. The important thing is not to redirect user requests for 2,000 newish files to a server that does not have them yet. Maybe some kind of "partial" status on the web page if some but not all backends are up?

Sounds great to me. Let's see what Tim B thinks. --Mike

The two servers sync independently. Whilst this is not an ideal situation, and can lead to them having slightly out of step content, it's the simplest approach and requires the least development and maintenance work. In theory they should sync within an hour of each other, so the likely reason for them to have different files for an extended period is that one of them is failing to sync. Obviously this is something that's quite easy to investigate at the time, but impossible to work out later. So if you see this please let us know at the time so we can see why it's happening.