[01:09:28] PROBLEM - MariaDB sustained replica lag on m1 on db1217 is CRITICAL: 8.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321 [01:10:56] RECOVERY - MariaDB sustained replica lag on m1 on db1217 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321 [06:59:18] There's a old task that got pinged on clinic duty, seems related to Swift: https://phabricator.wikimedia.org/T291137 Is this still an issue or can it be closed? [07:31:48] it might well still be an issue; the first file is no longer available, but it might be worth suggesting they retry the PDF - there has been some work on improving thumbor's PDF handling, so it's possible it'll now upload OK. I'm afraid these sort of issues are very difficult to get to the bottom of (particularly given the lack of request tracing through the MW => swift & thumbor stacks) [07:32:50] thumbor> cf T337649 [07:32:51] T337649: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 [07:37:23] (the other possibility is the PDF that is failing is malformed in some way; thumbor is less forgiving than some other tooling, and I'm not convinced that's a failing in thumbor) [07:45:43] Okay, should I do something with the task or just leave it in the backlog :-) [07:58:48] I think it can probably just stay in the swift backlog-of-doom [07:59:33] [I'm not sure clinic duty person is meant to be poked for "here's an issue that I think the relevant team isn't looking at without enough dilligence"] [08:06:04] I've responded on the ticket [16:37:20] very first draft tutorial on git-based packaging of new software - https://wikitech.wikimedia.org/wiki/User:MVernon/Packaging_Tutorial [17:55:35] thanks! [18:05:41] I was most of the way through it working with GNU Hello, which I naïvely thought would be a good example program 😿