[07:24:15] <dcaro_away>	 !log toolsbeta started postgresql on puppetdb-02, might have crashed during the ceph issues, now puppet runs on toolsbeta work again
[07:24:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[19:35:45] <Platonides>	 is db replication broken?
[19:36:06] <wm-bb>	 <Samuel> Which section?
[19:36:15] <Platonides>	 commonswiki
[19:36:21] <Platonides>	 I don't remember now, s5?
[19:36:35] <wm-bb>	 <Samuel> And @bd808 it's duplicating
[19:36:40] <Platonides>	 recent changes last entry seems to be from 20220812010311
[19:36:47] <wm-bb>	 <Samuel> Commonswiki is yes
[19:36:50] <Platonides>	 expected something like 20220812193553
[19:36:53] <taavi>	 there are some larger schema changes in progress, which can create occasional replication lag
[19:37:08] <wm-bb>	 <Samuel> Commonswiki is ongoing schema change ye
[19:37:14] <taavi>	 commons is s4, and looks like it's indeed currently lagging behind
[19:37:44] <wm-bb>	 <Samuel> S2 keeps breaking too due to templatelinks
[19:38:14] <AntiComposite>	 https://replag.toolforge.org/
[19:38:57] <RhinosF1>	 That tells you there's lag, doesn't say why though.
[19:41:13] <Platonides>	 it doesn't seem to be slow, just stopped :/
[19:43:02] <RhinosF1>	 That's what happens with schema changes
[19:43:28] <taavi>	 Platonides: this is "expected". mariadb can't replica while a table is being ALTERed, so for schema changes to larger tables (such as templatelinks, likely what's happening here) the replicas unfortunately stop replicating for a while and then will catch up once the schema change is done
[19:44:48] <Platonides>	 ok, as far as it's expected... :)
[19:46:26] <taavi>	 well, 'expected' in a sense that it's known and the best thing we have without spending lots of money on extra hardware and tooling to make schema changes less visible
[19:48:36] <RhinosF1>	 Anyone able to bounce wm-bb as it's duplicating on telegram
[19:49:52] <Platonides>	 we can live with it, taavi 
[19:54:07] <wm-bot>	 !log tools.bridgebot <bd808> Double IRC messages to other bridges
[19:54:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[19:54:38] <bd808>	 message dups should be fixed now
[19:55:07] <wm-bb>	 <MaartenDammers> You and your duplicate? (re @wmtelegram_bot: <Platonides> we can live with it, taavi)
[19:55:14] <bd808>	 someday I'll find the time to look into fixing that in the bot again...
[19:56:29] <RhinosF1>	 bd808: they are fixed ye
[19:57:27] <Platonides>	 maybe they guys at telegram are not exactly sober, Multichill ;)
[20:04:36] <wm-bot>	 !log tools.bridgebot <bd808> Remove protocol label from #wikimedia-hackathon channel messages
[20:04:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL