Re: [trinity-devel] UK Mirror Service mirrors

Hi Slávek, cc: Tim Bishop Very helpful information here from Tim. In short, could we change the mirror list on https://www.trinitydesktop.org/mirrorstatus.php to replace his copernicus and kuiper with just http://www.mirrorservice.org/sites/trinitydesktop.org/trinity/ while our redirector could continue to use copernicus and kuiper? Primary mirror would not be affected by this change. Thanks, --Mike On Sun July 19 2020 03:25:35 Tim Bishop wrote: > Hi Mike, > > The two servers sync independently. Whilst this is not an ideal > situation, and can lead to them having slightly out of step content, > it's the simplest approach and requires the least development and > maintenance work. In theory they should sync within an hour of each > other, so the likely reason for them to have different files for an > extended period is that one of them is failing to sync. Obviously this > is something that's quite easy to investigate at the time, but > impossible to work out later. So if you see this please let us know at > the time so we can see why it's happening. > > The load balancer doesn't take much action regarding this. It doesn't > know which mirrors are up to date or not, just whether they're > functioning. However, it does have "stickiness" enabled, which means a > client from the same IP should repeatedly hit the same backend unless > there's a failure. So they should at least get consistent content. > > My main gripe is about publishing the details of the backend systems > directly. This leads to people using them not via the load balancer > and then complaining when one is down. In your case you have a smart > redirector in front, so presumably that'd deal with that problem > without inconveniencing the users. I don't have any problem with you > doing that but I'd be happier if the backend URLs weren't so obviously > advertised. > > The period of downtime in early June was because we had a complete > data centre outage for the first time in over 10 years. The UPS > systems were being upgraded and a failure occurred which powered > everything off - the systems supposed to save us were the ones that > failed us :-( The Mirror Service sadly isn't a priority system for the > University, compared to core teaching and research systems, so we had > to wait until full power was restored before we could turn it back on. > > That issue also killed copernicus, or more specifically its RAID > controller. I have now replaced that and spent the past couple of > weeks rebuilding it and resyncing the data. This has proved quite time > consuming but it is starting to come back online this weekend. > > Tim. > > On Sat, Jul 18, 2020 at 11:35:18AM -0700, Mike Bird wrote: > > Hi Tim, > > > > Thank you for mirroring TDE! > > > > I don't maintain that web page so this is not a direct response > > to your request but rather a request for further information. > > > > FYI https://www.trinitydesktop.org/mirrorstatus.php is not the > > normal route whereby users retrieve files. Users work instead > > via a smart redirector http://mirror.ppa.trinitydesktop.org/ > > > > The status page is more of a debugging tool. > > > > We monitor mirrors to determine when they have pulled updates. > > There have been several extended periods when copernicus and > > kuiper have held different files. We use this information in > > the smart redirector to only send user traffic to mirrors which > > are known to have the file the user needs. > > > > I was wondering to what extent the UK Mirror Service load balancer > > addresses this as the last time I saw copernicus rsync from us > > was June 19th? And earlier in June there was a period of a few days > > when neither kuiper nor copernicus were updating? > > > > Thanks again! > > > > --Mike