[03:47:06] <jinxer-wm> FIRING: [2x] SystemdUnitFailed: pt-heartbeat-wikimedia.service on pc1017:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:39:11] <jinxer-wm> FIRING: [3x] SystemdUnitFailed: swift_rclone_sync.service on ms-be2069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:47:06] <jinxer-wm> FIRING: [5x] SystemdUnitFailed: swift_rclone_sync.service on ms-be2069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:53:04] <arnaudb> https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1092787 weird spam [08:00:57] <RhinosF1> arnaudb: given the email (https://ldap.toolforge.org/user/PRESELYA1), should probably be disabled [08:01:34] <RhinosF1> I have no idea how to do that though [08:03:22] <RhinosF1> https://wikitech.wikimedia.org/wiki/Disabling_an_account is still outdated. Wikitech is SUL now. [09:39:11] <jinxer-wm> FIRING: PuppetFailure: Puppet has failed on ms-be1056:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:44:58] <Emperor> That's T371192 again; extending silence [09:44:59] <stashbot> T371192: Disk (sdh) failed on ms-be1056 - https://phabricator.wikimedia.org/T371192 [10:34:11] <jinxer-wm> RESOLVED: SystemdUnitFailed: swift_rclone_sync.service on ms-be2069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:47:06] <jinxer-wm> FIRING: [3x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s3.service on db2239:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:33:20] <Emperor> anyone here a commons admin, or know one? T380738 has a image that needs uploading to fix an inconsistent one, and I think we usually ask a commons admin to do the upload. [12:33:21] <stashbot> T380738: Schuur - Nieuwerbrug - 20164513 - RCE.jpg inconsistent, needs new upload - https://phabricator.wikimedia.org/T380738 [12:35:11] <Amir1> I can find one [12:35:24] <Amir1> (meeting now, will do it later) [12:36:36] <Emperor> Thanks :) [13:00:17] <Amir1> https://phabricator.wikimedia.org/T380449#10352765 That's going to be a fun backup change alert [13:12:03] <Emperor> nice [13:15:41] <arnaudb> Emperor: ah I missed the alert on db2239, they seem to come from an expired silence, I think it is still in WIP https://sal.toolforge.org/production?p=0&q=db2239 I'll extend the downtime for the week [13:25:12] <Emperor> thanks :) [14:30:23] <Emperor> my webcam is having a moment, trying to fix it [14:54:20] <Emperor> webcam working with other devices [14:57:00] <jynus> :-( [14:59:03] <Emperor> Seems to be only google meet where it isn't [15:00:45] <Emperor> better, only google meet in my existing browser profile(!) if I start up a private window in a different profile it works OK [15:05:25] <Emperor> Hm, throwing away my google/youtube cookies didn't fix it, but if I disable auto-center it sort-of-works again. [15:12:59] <jynus> that's weird [15:25:57] <Emperor> google and my default browser profile keep falling out for no obvious reason, which is a total pain because all my work state / bookmarks / etc etc are in that profile