[01:18:52] 10SRE, 10ops-codfw, 10DC-Ops, 10observability, 10User-fgiunchedi: codfw: Testing Out Sample PDUs - https://phabricator.wikimedia.org/T265435 (10wiki_willy) Hi @fgiunchedi - sorry for the delay. Just to do a quick check before you put a lot of time and effort in....from your perspective on the monitoring... [01:23:13] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [03:24:49] RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [04:19:19] 10SRE, 10DBA, 10Datacenter-Switchover: Check "Days in advance preparation" for databases before DC switchover - https://phabricator.wikimedia.org/T285069 (10Marostegui) [04:31:31] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1113:3316', diff saved to https://phabricator.wikimedia.org/P16546 and previous config saved to /var/cache/conftool/dbconfig/20210617-043130-marostegui.json [04:31:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:37:04] (03PS1) 10Marostegui: db2080: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/700118 [04:37:45] (03CR) 10Marostegui: [C: 03+2] db2080: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/700118 (owner: 10Marostegui) [04:41:33] !log marostegui@cumin1001 dbctl commit (dc=all): 'Repool db1113:3316', diff saved to https://phabricator.wikimedia.org/P16547 and previous config saved to /var/cache/conftool/dbconfig/20210617-044132-marostegui.json [04:41:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:41:47] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1180', diff saved to https://phabricator.wikimedia.org/P16548 and previous config saved to /var/cache/conftool/dbconfig/20210617-044146-marostegui.json [04:41:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:45:54] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1180 (re)pooling @ 25%: Repool db1180 after schema change', diff saved to https://phabricator.wikimedia.org/P16549 and previous config saved to /var/cache/conftool/dbconfig/20210617-044554-root.json [04:45:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:56:29] PROBLEM - Unmerged changes on repository puppet on puppetmaster1001 is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet, ref HEAD..origin/production). https://wikitech.wikimedia.org/wiki/Monitoring/unmerged_changes [04:58:21] RECOVERY - Unmerged changes on repository puppet on puppetmaster1001 is OK: No changes to merge. https://wikitech.wikimedia.org/wiki/Monitoring/unmerged_changes [05:00:58] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1180 (re)pooling @ 50%: Repool db1180 after schema change', diff saved to https://phabricator.wikimedia.org/P16550 and previous config saved to /var/cache/conftool/dbconfig/20210617-050057-root.json [05:01:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:16:02] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1180 (re)pooling @ 75%: Repool db1180 after schema change', diff saved to https://phabricator.wikimedia.org/P16551 and previous config saved to /var/cache/conftool/dbconfig/20210617-051601-root.json [05:16:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:31:05] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1180 (re)pooling @ 100%: Repool db1180 after schema change', diff saved to https://phabricator.wikimedia.org/P16552 and previous config saved to /var/cache/conftool/dbconfig/20210617-053105-root.json [05:31:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:34:55] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1165', diff saved to https://phabricator.wikimedia.org/P16553 and previous config saved to /var/cache/conftool/dbconfig/20210617-053455-marostegui.json [05:34:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:40:04] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1165 (re)pooling @ 25%: Repool db1165 after schema change', diff saved to https://phabricator.wikimedia.org/P16554 and previous config saved to /var/cache/conftool/dbconfig/20210617-054003-root.json [05:40:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:55:07] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1165 (re)pooling @ 50%: Repool db1165 after schema change', diff saved to https://phabricator.wikimedia.org/P16555 and previous config saved to /var/cache/conftool/dbconfig/20210617-055507-root.json [05:55:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:10:11] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1165 (re)pooling @ 75%: Repool db1165 after schema change', diff saved to https://phabricator.wikimedia.org/P16556 and previous config saved to /var/cache/conftool/dbconfig/20210617-061010-root.json [06:10:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:15:36] (03PS1) 10Ladsgroup: dumps: Migrate miscdumps clean up cron to systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/700123 (https://phabricator.wikimedia.org/T273673) [06:22:15] (03PS1) 10Majavah: Add WMCS public addresses to $wgSoftBlockRanges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700160 [06:25:15] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1165 (re)pooling @ 100%: Repool db1165 after schema change', diff saved to https://phabricator.wikimedia.org/P16557 and previous config saved to /var/cache/conftool/dbconfig/20210617-062514-root.json [06:25:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:25:29] (03CR) 10Elukey: [V: 03+2 C: 03+2] Add initial debianization for istioctl 1.6.14 [debs/istioctl] - 10https://gerrit.wikimedia.org/r/700012 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey) [06:27:30] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [06:31:36] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1098:3316', diff saved to https://phabricator.wikimedia.org/P16558 and previous config saved to /var/cache/conftool/dbconfig/20210617-063135-marostegui.json [06:31:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:32:07] (03PS1) 10Ladsgroup: dumps: Migrate xml dumps clean up cron to systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/700161 (https://phabricator.wikimedia.org/T273673) [06:42:15] (03PS1) 10Elukey: Skip the dwz step [debs/istioctl] - 10https://gerrit.wikimedia.org/r/700162 (https://phabricator.wikimedia.org/T278192) [06:47:19] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1098:3316 (re)pooling @ 25%: Repool db1098:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P16559 and previous config saved to /var/cache/conftool/dbconfig/20210617-064717-root.json [06:47:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:49:06] (03CR) 10Elukey: Add support for knative serving (034 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/699380 (https://phabricator.wikimedia.org/T278194) (owner: 10Elukey) [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210617T0700) [07:02:23] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1098:3316 (re)pooling @ 50%: Repool db1098:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P16560 and previous config saved to /var/cache/conftool/dbconfig/20210617-070222-root.json [07:02:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:10:51] (03CR) 10Jcrespo: "> Is that sufficient and we can stop backing it up regularly? Or should we continue to keep backing it up, knowing that most of the time i" [puppet] - 10https://gerrit.wikimedia.org/r/697637 (https://phabricator.wikimedia.org/T282303) (owner: 10Ladsgroup) [07:17:26] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1098:3316 (re)pooling @ 75%: Repool db1098:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P16561 and previous config saved to /var/cache/conftool/dbconfig/20210617-071726-root.json [07:17:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:32:30] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1098:3316 (re)pooling @ 100%: Repool db1098:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P16562 and previous config saved to /var/cache/conftool/dbconfig/20210617-073229-root.json [07:32:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:33:06] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1168', diff saved to https://phabricator.wikimedia.org/P16563 and previous config saved to /var/cache/conftool/dbconfig/20210617-073305-marostegui.json [07:33:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:39:11] (03CR) 10Jcrespo: [V: 03+2 C: 03+2] api_db: Add working skeleton code for api_db, add dockerfile [software/bernard] - 10https://gerrit.wikimedia.org/r/699915 (https://phabricator.wikimedia.org/T284399) (owner: 10H.krishna123) [07:39:26] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1168 (re)pooling @ 25%: Repool db1168 after schema change', diff saved to https://phabricator.wikimedia.org/P16564 and previous config saved to /var/cache/conftool/dbconfig/20210617-073926-root.json [07:39:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:40:10] (03CR) 10Jcrespo: [V: 03+2 C: 03+2] "We may have to change the base image, as I mentioned, later, and it needs some work for CI integration, but the foundations seems solid (t" [software/bernard] - 10https://gerrit.wikimedia.org/r/699915 (https://phabricator.wikimedia.org/T284399) (owner: 10H.krishna123) [07:40:38] (03CR) 10Jcrespo: "We forgot to add this in our actionables :-)" [software/bernard] - 10https://gerrit.wikimedia.org/r/699915 (https://phabricator.wikimedia.org/T284399) (owner: 10H.krishna123) [07:50:40] (03CR) 10Elukey: [C: 03+1] "LGTM" [software/spicerack] - 10https://gerrit.wikimedia.org/r/700076 (owner: 10Volans) [07:54:30] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1168 (re)pooling @ 50%: Repool db1168 after schema change', diff saved to https://phabricator.wikimedia.org/P16565 and previous config saved to /var/cache/conftool/dbconfig/20210617-075429-root.json [07:54:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:54:43] (03CR) 10Ema: [V: 03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29898/console" [puppet] - 10https://gerrit.wikimedia.org/r/693959 (https://phabricator.wikimedia.org/T281423) (owner: 10Legoktm) [07:58:26] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1096:3315', diff saved to https://phabricator.wikimedia.org/P16566 and previous config saved to /var/cache/conftool/dbconfig/20210617-075825-marostegui.json [07:58:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:04:31] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 25%: Repool db1096:3315 after schema change', diff saved to https://phabricator.wikimedia.org/P16567 and previous config saved to /var/cache/conftool/dbconfig/20210617-080430-root.json [08:04:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:04:37] (03CR) 10David Caro: [C: 03+1] "👌" (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/700076 (owner: 10Volans) [08:09:34] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1168 (re)pooling @ 75%: Repool db1168 after schema change', diff saved to https://phabricator.wikimedia.org/P16568 and previous config saved to /var/cache/conftool/dbconfig/20210617-080933-root.json [08:09:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:10:56] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good, Go binaries don't have dwz symbols, I think that got fixed for bullseye in dh_golang, but we need it for buster." [debs/istioctl] - 10https://gerrit.wikimedia.org/r/700162 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey) [08:11:58] (03CR) 10Elukey: [V: 03+2 C: 03+2] Skip the dwz step [debs/istioctl] - 10https://gerrit.wikimedia.org/r/700162 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey) [08:12:17] moritzm: thanks! [08:12:27] yw :-) [08:17:10] (03PS1) 10Elukey: Fix debian distribution name in changelog [debs/istioctl] - 10https://gerrit.wikimedia.org/r/700165 [08:17:27] (03CR) 10Elukey: [V: 03+2 C: 03+2] Fix debian distribution name in changelog [debs/istioctl] - 10https://gerrit.wikimedia.org/r/700165 (owner: 10Elukey) [08:17:58] (03CR) 10Volans: "replied to comment, will addrss" (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/700076 (owner: 10Volans) [08:19:35] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 50%: Repool db1096:3315 after schema change', diff saved to https://phabricator.wikimedia.org/P16569 and previous config saved to /var/cache/conftool/dbconfig/20210617-081934-root.json [08:19:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:24:10] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1113:3315', diff saved to https://phabricator.wikimedia.org/P16570 and previous config saved to /var/cache/conftool/dbconfig/20210617-082409-marostegui.json [08:24:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:24:22] 10SRE, 10netops: Cleanup confed BGP peerings and policies - https://phabricator.wikimedia.org/T167841 (10cmooney) > The first major issue is that we haven't really thought through the tradeoffs between doing multihop BGP peerings or not (with an almost arbitrary/hard to calculate max hop), between loopbacks or... [08:24:37] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1168 (re)pooling @ 100%: Repool db1168 after schema change', diff saved to https://phabricator.wikimedia.org/P16571 and previous config saved to /var/cache/conftool/dbconfig/20210617-082437-root.json [08:24:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:28:02] !log upload istioctl 1.6.14-1 to buster-wikimedia [08:28:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:29:40] !log marostegui@cumin1001 dbctl commit (dc=all): 'Repool db1113:3315', diff saved to https://phabricator.wikimedia.org/P16572 and previous config saved to /var/cache/conftool/dbconfig/20210617-082939-marostegui.json [08:29:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:30:05] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1110', diff saved to https://phabricator.wikimedia.org/P16573 and previous config saved to /var/cache/conftool/dbconfig/20210617-083005-marostegui.json [08:30:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:34:38] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 75%: Repool db1096:3315 after schema change', diff saved to https://phabricator.wikimedia.org/P16574 and previous config saved to /var/cache/conftool/dbconfig/20210617-083438-root.json [08:34:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:35:10] (03CR) 10Jbond: [C: 03+1] "lgtm" [software/spicerack] - 10https://gerrit.wikimedia.org/r/700076 (owner: 10Volans) [08:35:46] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1110 (re)pooling @ 25%: Repool db1110 after schema change', diff saved to https://phabricator.wikimedia.org/P16575 and previous config saved to /var/cache/conftool/dbconfig/20210617-083545-root.json [08:35:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:43:59] (03CR) 10JMeybohm: [C: 04-1] Add support for knative serving (033 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/699380 (https://phabricator.wikimedia.org/T278194) (owner: 10Elukey) [08:49:42] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1096:3315 (re)pooling @ 100%: Repool db1096:3315 after schema change', diff saved to https://phabricator.wikimedia.org/P16576 and previous config saved to /var/cache/conftool/dbconfig/20210617-084941-root.json [08:49:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:49:49] (03CR) 10H.krishna123: "> Patch Set 3:" [software/bernard] - 10https://gerrit.wikimedia.org/r/699915 (https://phabricator.wikimedia.org/T284399) (owner: 10H.krishna123) [08:50:49] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1110 (re)pooling @ 50%: Repool db1110 after schema change', diff saved to https://phabricator.wikimedia.org/P16577 and previous config saved to /var/cache/conftool/dbconfig/20210617-085048-root.json [08:50:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:05:53] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1110 (re)pooling @ 75%: Repool db1110 after schema change', diff saved to https://phabricator.wikimedia.org/P16578 and previous config saved to /var/cache/conftool/dbconfig/20210617-090552-root.json [09:05:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:09:48] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1161', diff saved to https://phabricator.wikimedia.org/P16579 and previous config saved to /var/cache/conftool/dbconfig/20210617-090947-marostegui.json [09:09:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:16:21] (03PS6) 10Ema: varnish: add timing data to varnishmtail [puppet] - 10https://gerrit.wikimedia.org/r/699223 (https://phabricator.wikimedia.org/T284576) [09:16:23] (03PS3) 10Ema: varnish: add prometheus histogram varnish_processing_seconds [puppet] - 10https://gerrit.wikimedia.org/r/699941 (https://phabricator.wikimedia.org/T284576) [09:19:35] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1161 (re)pooling @ 25%: Repool db1161 after schema change', diff saved to https://phabricator.wikimedia.org/P16580 and previous config saved to /var/cache/conftool/dbconfig/20210617-091934-root.json [09:19:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:20:56] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1110 (re)pooling @ 100%: Repool db1110 after schema change', diff saved to https://phabricator.wikimedia.org/P16581 and previous config saved to /var/cache/conftool/dbconfig/20210617-092056-root.json [09:20:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:23:37] (03PS1) 10Jbond: C:puppetmaster: drop unneeded python dependencies. [puppet] - 10https://gerrit.wikimedia.org/r/700174 [09:25:08] (03PS4) 10Ema: varnish: add prometheus histogram varnish_processing_seconds [puppet] - 10https://gerrit.wikimedia.org/r/699941 (https://phabricator.wikimedia.org/T284576) [09:26:42] (03CR) 10jerkins-bot: [V: 04-1] varnish: add prometheus histogram varnish_processing_seconds [puppet] - 10https://gerrit.wikimedia.org/r/699941 (https://phabricator.wikimedia.org/T284576) (owner: 10Ema) [09:28:17] (03CR) 10Ema: varnish: add prometheus histogram varnish_processing_seconds (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/699941 (https://phabricator.wikimedia.org/T284576) (owner: 10Ema) [09:28:32] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good to me." [puppet] - 10https://gerrit.wikimedia.org/r/700174 (owner: 10Jbond) [09:29:23] (03PS5) 10Ema: varnish: add prometheus histogram varnish_processing_seconds [puppet] - 10https://gerrit.wikimedia.org/r/699941 (https://phabricator.wikimedia.org/T284576) [09:31:52] (03CR) 10Jbond: [C: 03+2] C:puppetmaster: drop unneeded python dependencies. [puppet] - 10https://gerrit.wikimedia.org/r/700174 (owner: 10Jbond) [09:34:39] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1161 (re)pooling @ 50%: Repool db1161 after schema change', diff saved to https://phabricator.wikimedia.org/P16582 and previous config saved to /var/cache/conftool/dbconfig/20210617-093438-root.json [09:34:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:43:49] (03CR) 10MSantos: [C: 03+1] maps: make maps2007 a buster replica of maps2009 [puppet] - 10https://gerrit.wikimedia.org/r/700087 (https://phabricator.wikimedia.org/T269582) (owner: 10Hnowlan) [09:49:42] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1161 (re)pooling @ 75%: Repool db1161 after schema change', diff saved to https://phabricator.wikimedia.org/P16583 and previous config saved to /var/cache/conftool/dbconfig/20210617-094942-root.json [09:49:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:58:51] (03PS2) 10Volans: icinga: rename some IcingaHosts methods [software/spicerack] - 10https://gerrit.wikimedia.org/r/700076 [10:04:41] (03PS1) 10Giuseppe Lavagetto: mediawiki: early and late rewrites are optional [deployment-charts] - 10https://gerrit.wikimedia.org/r/700178 [10:04:46] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1161 (re)pooling @ 100%: Repool db1161 after schema change', diff saved to https://phabricator.wikimedia.org/P16584 and previous config saved to /var/cache/conftool/dbconfig/20210617-100445-root.json [10:04:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:05:34] (03PS1) 10Elukey: Move knative serving's queue image to a different layout [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/700179 (https://phabricator.wikimedia.org/T272919) [10:07:24] (03PS2) 10Elukey: Move knative serving's queue image to a different layout [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/700179 (https://phabricator.wikimedia.org/T272919) [10:13:15] (03CR) 10Dzahn: "Would you like me to just abandon it? Or should it be simply amended to use the ./latest/ path? That would of course be a trivial fix and," [puppet] - 10https://gerrit.wikimedia.org/r/697850 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn) [10:17:43] (03PS1) 10Volans: Add pip's dependencies to the generated wheels [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/700181 [10:18:28] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1130', diff saved to https://phabricator.wikimedia.org/P16585 and previous config saved to /var/cache/conftool/dbconfig/20210617-101827-marostegui.json [10:18:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:19:48] (03CR) 10Jbond: [C: 03+1] "LGTM" [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/700181 (owner: 10Volans) [10:21:46] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1130 (re)pooling @ 25%: Repool db1130 after schema change', diff saved to https://phabricator.wikimedia.org/P16586 and previous config saved to /var/cache/conftool/dbconfig/20210617-102145-root.json [10:21:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:22:30] (03CR) 10Jcrespo: "> The part about the default backup policy makes me wonder which other strategy would be suggested or what makes this different from other" [puppet] - 10https://gerrit.wikimedia.org/r/697850 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn) [10:23:48] (03CR) 10Jcrespo: "> Patch Set 2: -Code-Review" [puppet] - 10https://gerrit.wikimedia.org/r/697850 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn) [10:24:17] (03CR) 10David Caro: [C: 03+1] "👍" [software/spicerack] - 10https://gerrit.wikimedia.org/r/700076 (owner: 10Volans) [10:30:42] (03CR) 10JMeybohm: [C: 03+1] "Oh, wow. That is ... impressive" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/700179 (https://phabricator.wikimedia.org/T272919) (owner: 10Elukey) [10:36:50] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1130 (re)pooling @ 50%: Repool db1130 after schema change', diff saved to https://phabricator.wikimedia.org/P16587 and previous config saved to /var/cache/conftool/dbconfig/20210617-103649-root.json [10:36:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:37:59] (03PS1) 10David Caro: ceph: add latency monitoring stats [puppet] - 10https://gerrit.wikimedia.org/r/700182 (https://phabricator.wikimedia.org/T281254) [10:39:03] (03CR) 10David Caro: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/700182 (https://phabricator.wikimedia.org/T281254) (owner: 10David Caro) [10:42:53] (03CR) 10JMeybohm: [C: 03+1] "Smells like we need a linter for the generated apache config :-)" [deployment-charts] - 10https://gerrit.wikimedia.org/r/700178 (owner: 10Giuseppe Lavagetto) [10:46:21] (03PS1) 10Jcrespo: bacula: Add new jobdefaults/schedule for Github, full backups every day [puppet] - 10https://gerrit.wikimedia.org/r/700183 (https://phabricator.wikimedia.org/T274463) [10:46:57] (03CR) 10Giuseppe Lavagetto: [C: 03+2] mediawiki: early and late rewrites are optional [deployment-charts] - 10https://gerrit.wikimedia.org/r/700178 (owner: 10Giuseppe Lavagetto) [10:47:29] (03PS3) 10Jcrespo: backups: Fix typo on fileset name, resulting on no backups scheduled [puppet] - 10https://gerrit.wikimedia.org/r/684300 (https://phabricator.wikimedia.org/T281369) [10:47:48] (03Abandoned) 10Jcrespo: backups: Fix typo on fileset name, resulting on no backups scheduled [puppet] - 10https://gerrit.wikimedia.org/r/684300 (https://phabricator.wikimedia.org/T281369) (owner: 10Jcrespo) [10:47:52] (03CR) 10jerkins-bot: [V: 04-1] bacula: Add new jobdefaults/schedule for Github, full backups every day [puppet] - 10https://gerrit.wikimedia.org/r/700183 (https://phabricator.wikimedia.org/T274463) (owner: 10Jcrespo) [10:49:15] (03Merged) 10jenkins-bot: mediawiki: early and late rewrites are optional [deployment-charts] - 10https://gerrit.wikimedia.org/r/700178 (owner: 10Giuseppe Lavagetto) [10:49:27] (03CR) 10Jcrespo: "This is the general idea: https://gerrit.wikimedia.org/r/c/operations/puppet/+/700183 (may need some puppet compiler and testing, etc)" [puppet] - 10https://gerrit.wikimedia.org/r/697850 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn) [10:49:59] (03CR) 10Giuseppe Lavagetto: [C: 03+1] Move knative serving's queue image to a different layout (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/700179 (https://phabricator.wikimedia.org/T272919) (owner: 10Elukey) [10:50:22] elukey: code that relies on a hardcoded path reeks of high quality [10:50:52] joe: I feel a little sad inside [10:51:18] (03PS2) 10Jcrespo: bacula: Add new jobdefaults/schedule for Github, full backups every day [puppet] - 10https://gerrit.wikimedia.org/r/700183 (https://phabricator.wikimedia.org/T274463) [10:51:53] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1130 (re)pooling @ 75%: Repool db1130 after schema change', diff saved to https://phabricator.wikimedia.org/P16588 and previous config saved to /var/cache/conftool/dbconfig/20210617-105153-root.json [10:51:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:53:55] (03PS3) 10Jcrespo: bacula: Add new jobdefaults/schedule for Github, full backups every day [puppet] - 10https://gerrit.wikimedia.org/r/700183 (https://phabricator.wikimedia.org/T274463) [10:59:27] (03CR) 10Jcrespo: "Looking good: https://puppet-compiler.wmflabs.org/compiler1002/29899/backup1001.eqiad.wmnet/index.html" [puppet] - 10https://gerrit.wikimedia.org/r/700183 (https://phabricator.wikimedia.org/T274463) (owner: 10Jcrespo) [11:00:37] (03CR) 10Elukey: [V: 03+2 C: 03+2] Move knative serving's queue image to a different layout [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/700179 (https://phabricator.wikimedia.org/T272919) (owner: 10Elukey) [11:02:01] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1144:3315', diff saved to https://phabricator.wikimedia.org/P16589 and previous config saved to /var/cache/conftool/dbconfig/20210617-110200-marostegui.json [11:02:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:04:50] (03CR) 10Hnowlan: [C: 03+2] postgres: fix sync bugs in resync_replica script [puppet] - 10https://gerrit.wikimedia.org/r/699430 (owner: 10Hnowlan) [11:06:57] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1130 (re)pooling @ 100%: Repool db1130 after schema change', diff saved to https://phabricator.wikimedia.org/P16590 and previous config saved to /var/cache/conftool/dbconfig/20210617-110656-root.json [11:06:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:07:04] (03CR) 10Urbanecm: "commit message issue" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700183 (https://phabricator.wikimedia.org/T274463) (owner: 10Jcrespo) [11:08:09] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 25%: Repool db1144:3315 after schema change', diff saved to https://phabricator.wikimedia.org/P16591 and previous config saved to /var/cache/conftool/dbconfig/20210617-110808-root.json [11:08:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:08:55] (03PS1) 10David Caro: grid: php config don't rely on php being installed by puppet [puppet] - 10https://gerrit.wikimedia.org/r/700186 [11:10:00] (03CR) 10Majavah: [C: 04-1] grid: php config don't rely on php being installed by puppet (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700186 (owner: 10David Caro) [11:10:24] (03PS1) 10Jbond: P:puppetdb::database: Add support for bullseye [puppet] - 10https://gerrit.wikimedia.org/r/700187 [11:10:27] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1096:3316', diff saved to https://phabricator.wikimedia.org/P16592 and previous config saved to /var/cache/conftool/dbconfig/20210617-111026-marostegui.json [11:10:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:11:28] (03CR) 10Jbond: [V: 03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29900/console" [puppet] - 10https://gerrit.wikimedia.org/r/700187 (owner: 10Jbond) [11:13:41] (03CR) 10Jbond: [V: 03+1 C: 03+2] P:puppetdb::database: Add support for bullseye [puppet] - 10https://gerrit.wikimedia.org/r/700187 (owner: 10Jbond) [11:15:58] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 25%: Repool db1096:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P16593 and previous config saved to /var/cache/conftool/dbconfig/20210617-111558-root.json [11:16:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:19:23] (03PS2) 10David Caro: grid: php config don't rely on php being installed by puppet [puppet] - 10https://gerrit.wikimedia.org/r/700186 [11:19:31] 10Puppet, 10User-jbond: Prepare puppet master infrastructure for bullseye - https://phabricator.wikimedia.org/T285086 (10jbond) [11:19:33] (03CR) 10David Caro: grid: php config don't rely on php being installed by puppet (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700186 (owner: 10David Caro) [11:20:52] (03PS1) 10Jbond: C:uwsgi: Add support for bullseye [puppet] - 10https://gerrit.wikimedia.org/r/700189 (https://phabricator.wikimedia.org/T285086) [11:21:31] 10Puppet, 10Patch-For-Review, 10User-jbond: Prepare puppet master infrastructure for bullseye - https://phabricator.wikimedia.org/T285086 (10jbond) p:05Triage→03Medium [11:23:13] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 50%: Repool db1144:3315 after schema change', diff saved to https://phabricator.wikimedia.org/P16594 and previous config saved to /var/cache/conftool/dbconfig/20210617-112312-root.json [11:23:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:23:53] (03CR) 10Jbond: [V: 03+1] "PCC SUCCESS (NOOP 11): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29901/console" [puppet] - 10https://gerrit.wikimedia.org/r/700189 (https://phabricator.wikimedia.org/T285086) (owner: 10Jbond) [11:24:31] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1180', diff saved to https://phabricator.wikimedia.org/P16595 and previous config saved to /var/cache/conftool/dbconfig/20210617-112431-marostegui.json [11:24:33] (03CR) 10Jbond: [V: 03+1 C: 03+2] C:uwsgi: Add support for bullseye [puppet] - 10https://gerrit.wikimedia.org/r/700189 (https://phabricator.wikimedia.org/T285086) (owner: 10Jbond) [11:24:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:26:36] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1180 (re)pooling @ 25%: Repool db1180 after schema change', diff saved to https://phabricator.wikimedia.org/P16596 and previous config saved to /var/cache/conftool/dbconfig/20210617-112635-root.json [11:26:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:31:02] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 50%: Repool db1096:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P16597 and previous config saved to /var/cache/conftool/dbconfig/20210617-113101-root.json [11:31:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:31:14] RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [11:35:49] (03PS3) 10Jelto: bacula/gitlab: add a backup::set for gitlab and use it [puppet] - 10https://gerrit.wikimedia.org/r/697850 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn) [11:38:16] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 75%: Repool db1144:3315 after schema change', diff saved to https://phabricator.wikimedia.org/P16598 and previous config saved to /var/cache/conftool/dbconfig/20210617-113816-root.json [11:38:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:39:09] (03CR) 10Jelto: "> Patch Set 2:" [puppet] - 10https://gerrit.wikimedia.org/r/697850 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn) [11:41:40] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1180 (re)pooling @ 50%: Repool db1180 after schema change', diff saved to https://phabricator.wikimedia.org/P16599 and previous config saved to /var/cache/conftool/dbconfig/20210617-114139-root.json [11:41:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:43:24] (03PS1) 10Jbond: C:postgresql::server: Add support for bullseye [puppet] - 10https://gerrit.wikimedia.org/r/700192 (https://phabricator.wikimedia.org/T285086) [11:46:06] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 75%: Repool db1096:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P16600 and previous config saved to /var/cache/conftool/dbconfig/20210617-114605-root.json [11:46:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:46:48] (03CR) 10Jbond: [C: 03+2] C:postgresql::server: Add support for bullseye [puppet] - 10https://gerrit.wikimedia.org/r/700192 (https://phabricator.wikimedia.org/T285086) (owner: 10Jbond) [11:48:47] (03PS1) 10Arturo Borrero Gonzalez: openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 [11:49:36] (03CR) 10jerkins-bot: [V: 04-1] openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 (owner: 10Arturo Borrero Gonzalez) [11:51:47] (03PS2) 10Arturo Borrero Gonzalez: openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 [11:53:20] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1144:3315 (re)pooling @ 100%: Repool db1144:3315 after schema change', diff saved to https://phabricator.wikimedia.org/P16601 and previous config saved to /var/cache/conftool/dbconfig/20210617-115319-root.json [11:53:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:55:13] (03PS1) 10Jbond: C:postgress: Add support for bullseye [puppet] - 10https://gerrit.wikimedia.org/r/700194 (https://phabricator.wikimedia.org/T285086) [11:56:09] (03CR) 10David Caro: [C: 03+1] "Did not test it, but looks ok" [puppet] - 10https://gerrit.wikimedia.org/r/700193 (owner: 10Arturo Borrero Gonzalez) [11:56:15] (03PS3) 10Arturo Borrero Gonzalez: openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 [11:56:26] (03CR) 10Jbond: [V: 03+1] "PCC SUCCESS (NOOP 4): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29904/console" [puppet] - 10https://gerrit.wikimedia.org/r/700194 (https://phabricator.wikimedia.org/T285086) (owner: 10Jbond) [11:56:43] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1180 (re)pooling @ 75%: Repool db1180 after schema change', diff saved to https://phabricator.wikimedia.org/P16602 and previous config saved to /var/cache/conftool/dbconfig/20210617-115643-root.json [11:56:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:58:51] (03CR) 10Klausman: [C: 03+1] Move knative serving's queue image to a different layout (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/700179 (https://phabricator.wikimedia.org/T272919) (owner: 10Elukey) [12:01:04] (03PS4) 10Arturo Borrero Gonzalez: openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 [12:01:09] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 100%: Repool db1096:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P16603 and previous config saved to /var/cache/conftool/dbconfig/20210617-120109-root.json [12:01:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:07:05] (03CR) 10Jbond: [V: 03+1 C: 03+2] C:postgress: Add support for bullseye [puppet] - 10https://gerrit.wikimedia.org/r/700194 (https://phabricator.wikimedia.org/T285086) (owner: 10Jbond) [12:10:04] PROBLEM - rpki grafana alert on alert1001 is CRITICAL: CRITICAL: RPKI ( https://grafana.wikimedia.org/d/UwUa77GZk/rpki ) is alerting: eqiad rsync status alert, rsync status alert. https://wikitech.wikimedia.org/wiki/RPKI%23Grafana_alerts https://grafana.wikimedia.org/d/UwUa77GZk/ [12:11:47] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1180 (re)pooling @ 100%: Repool db1180 after schema change', diff saved to https://phabricator.wikimedia.org/P16604 and previous config saved to /var/cache/conftool/dbconfig/20210617-121146-root.json [12:11:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:12:18] (03PS5) 10Arturo Borrero Gonzalez: openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 [12:13:08] (03CR) 10jerkins-bot: [V: 04-1] openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 (owner: 10Arturo Borrero Gonzalez) [12:14:34] (03PS6) 10Arturo Borrero Gonzalez: openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 [12:17:20] (03PS7) 10Arturo Borrero Gonzalez: openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 [12:21:53] (03PS1) 10Ladsgroup: client: Bring back using the client setting for langlink group [extensions/Wikibase] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700036 (https://phabricator.wikimedia.org/T284854) [12:23:11] (03PS8) 10Arturo Borrero Gonzalez: openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 [12:25:45] (03PS9) 10Arturo Borrero Gonzalez: openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 [12:26:40] (03CR) 10jerkins-bot: [V: 04-1] openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 (owner: 10Arturo Borrero Gonzalez) [12:27:41] (03PS10) 10Arturo Borrero Gonzalez: openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 [12:28:32] (03CR) 10jerkins-bot: [V: 04-1] openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 (owner: 10Arturo Borrero Gonzalez) [12:29:49] (03CR) 10Filippo Giunchedi: [C: 03+1] varnish: add prometheus histogram varnish_processing_seconds [puppet] - 10https://gerrit.wikimedia.org/r/699941 (https://phabricator.wikimedia.org/T284576) (owner: 10Ema) [12:31:44] (03PS11) 10Arturo Borrero Gonzalez: openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 [12:32:35] 10SRE, 10ops-codfw, 10DC-Ops, 10observability, 10User-fgiunchedi: codfw: Testing Out Sample PDUs - https://phabricator.wikimedia.org/T265435 (10fgiunchedi) >>! In T265435#7160137, @wiki_willy wrote: > Hi @fgiunchedi - sorry for the delay. Just to do a quick check before you put a lot of time and effort... [12:33:35] (03PS12) 10Arturo Borrero Gonzalez: openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 [12:41:16] (03CR) 10Ema: [C: 03+2] varnish: add timing data to varnishmtail [puppet] - 10https://gerrit.wikimedia.org/r/699223 (https://phabricator.wikimedia.org/T284576) (owner: 10Ema) [12:41:28] (03CR) 10Ema: [C: 03+2] varnish: add prometheus histogram varnish_processing_seconds [puppet] - 10https://gerrit.wikimedia.org/r/699941 (https://phabricator.wikimedia.org/T284576) (owner: 10Ema) [12:44:47] (03PS1) 10Ema: varnish: install mtail program varnishprocessing.mtail [puppet] - 10https://gerrit.wikimedia.org/r/700198 (https://phabricator.wikimedia.org/T284576) [12:45:47] (03CR) 10Ema: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/700198 (https://phabricator.wikimedia.org/T284576) (owner: 10Ema) [12:47:50] (03CR) 10Ema: [C: 03+2] varnish: install mtail program varnishprocessing.mtail [puppet] - 10https://gerrit.wikimedia.org/r/700198 (https://phabricator.wikimedia.org/T284576) (owner: 10Ema) [12:48:20] (03CR) 10Jcrespo: "ups" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700183 (https://phabricator.wikimedia.org/T274463) (owner: 10Jcrespo) [12:48:48] (03PS4) 10Jcrespo: bacula: Add new jobdefaults/schedule for Gitlab, full backups every day [puppet] - 10https://gerrit.wikimedia.org/r/700183 (https://phabricator.wikimedia.org/T274463) [12:48:50] (03CR) 10Ladsgroup: [C: 03+2] client: Bring back using the client setting for langlink group [extensions/Wikibase] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700036 (https://phabricator.wikimedia.org/T284854) (owner: 10Ladsgroup) [12:49:13] (03CR) 10Jcrespo: bacula: Add new jobdefaults/schedule for Gitlab, full backups every day (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700183 (https://phabricator.wikimedia.org/T274463) (owner: 10Jcrespo) [12:50:58] ty urbanecm for spotting and reporting! [12:51:56] (03PS5) 10Jcrespo: bacula: Add new jobdefaults/schedule for Gitlab, full backups every day [puppet] - 10https://gerrit.wikimedia.org/r/700183 (https://phabricator.wikimedia.org/T274463) [12:57:07] 10SRE, 10SRE-Access-Requests: Access to Search Console for ptwikinews - https://phabricator.wikimedia.org/T285091 (10Edu) [12:58:03] 10SRE, 10SRE-Access-Requests: Access to Search Console for ptwikinews - https://phabricator.wikimedia.org/T285091 (10Edu) @Dzahn could check this task? [13:02:11] any time jynus ! [13:06:58] RECOVERY - rpki grafana alert on alert1001 is OK: OK: RPKI ( https://grafana.wikimedia.org/d/UwUa77GZk/rpki ) is not alerting. https://wikitech.wikimedia.org/wiki/RPKI%23Grafana_alerts https://grafana.wikimedia.org/d/UwUa77GZk/ [13:07:24] (03CR) 10Jcrespo: "> Patch Set 2:" [puppet] - 10https://gerrit.wikimedia.org/r/697850 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn) [13:08:19] 10SRE, 10SRE-Access-Requests: Access to ptwikinews Search Console for Edu - https://phabricator.wikimedia.org/T285091 (10Peachey88) [13:10:10] (03CR) 10Jcrespo: "@Jelto What time do exports run at? 0hours? Is a 4 hours gap between the export and the full backups adequate (e.g. to complete the export" [puppet] - 10https://gerrit.wikimedia.org/r/700183 (https://phabricator.wikimedia.org/T274463) (owner: 10Jcrespo) [13:12:59] (03Merged) 10jenkins-bot: client: Bring back using the client setting for langlink group [extensions/Wikibase] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700036 (https://phabricator.wikimedia.org/T284854) (owner: 10Ladsgroup) [13:20:09] (03PS1) 10Ayounsi: Allow ignoring LibreNMS devices [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700201 [13:21:42] (03CR) 10Ayounsi: "To prevent https://netbox.wikimedia.org/extras/reports/librenms.LibreNMS/ with "ignore alert tag" in https://librenms.wikimedia.org/device" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700201 (owner: 10Ayounsi) [13:28:04] !log add prometheus-jmx-exporter to bullseye-wikimedia [13:28:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:28:12] 10SRE, 10netbox: netbox: User's groups not updated - https://phabricator.wikimedia.org/T220004 (10ayounsi) @jbond Is that still relevant with the recent switch to SSO? [13:28:28] !log ladsgroup@deploy1002 Synchronized php-1.37.0-wmf.9/extensions/Wikibase/client/includes/ClientHooks.php: Backport: [[gerrit:700036|client: Bring back using the client setting for langlink group (T284854)]] (duration: 00m 58s) [13:28:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:28:34] T284854: Cannot add language versions of Wikipedia on Commons: "Page" (wikibase-linkitem-input-page) remains disabled in the "Link with page" dialog - https://phabricator.wikimedia.org/T284854 [13:28:53] joe: done [13:29:17] Amir1: great [13:30:47] 10SRE, 10netbox: netbox: User's groups not updated - https://phabricator.wikimedia.org/T220004 (10jbond) @ayounsi no this is covered by the cas plugin (not the switch should happen monday) [13:32:30] 10SRE, 10netbox: netbox: User's groups not updated - https://phabricator.wikimedia.org/T220004 (10ayounsi) 05Open→03Resolved a:03ayounsi Great :) [13:33:33] (03PS1) 10Elukey: profile::kubernetes::deployment_server: add istioctl package [puppet] - 10https://gerrit.wikimedia.org/r/700203 (https://phabricator.wikimedia.org/T278192) [13:35:17] 10SRE, 10SRE-Access-Requests: Access to ptwikinews Search Console for Edu - https://phabricator.wikimedia.org/T285091 (10Aklapper) 05Open→03Stalled @Edu: Hi, per https://wikitech.wikimedia.org/wiki/Google_Search_Console_access , do you have a valid NDA on file with [WMF Legal](https://meta.wikimedia.org/wi... [13:37:36] (03PS2) 10Elukey: profile::kubernetes::deployment_server: add istioctl package [puppet] - 10https://gerrit.wikimedia.org/r/700203 (https://phabricator.wikimedia.org/T278192) [13:39:35] (03CR) 10Elukey: [V: 03+1] "PCC SUCCESS (DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29918/console" [puppet] - 10https://gerrit.wikimedia.org/r/700203 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey) [14:06:43] 10SRE, 10netbox: Error in postgres puppettization for new installation (was Netbox: postgres cannot be restarted w/ current config) - https://phabricator.wikimedia.org/T184634 (10ayounsi) [14:06:48] PROBLEM - SSH on wdqs2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [14:09:58] (03CR) 10Jelto: "> Patch Set 5:" [puppet] - 10https://gerrit.wikimedia.org/r/700183 (https://phabricator.wikimedia.org/T274463) (owner: 10Jcrespo) [14:18:28] (03PS1) 10JMeybohm: docker::baseimages: Push images with legacy names [puppet] - 10https://gerrit.wikimedia.org/r/700204 [14:27:00] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] openstack: prometheus-cloudvirt-ceph-network: account for all ceph nodes [puppet] - 10https://gerrit.wikimedia.org/r/700193 (owner: 10Arturo Borrero Gonzalez) [14:27:56] (03CR) 10Volans: [C: 03+1] "LGTM" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700201 (owner: 10Ayounsi) [14:28:18] 10SRE, 10SRE-Access-Requests: Access to ptwikinews Search Console for Edu - https://phabricator.wikimedia.org/T285091 (10ssingh) p:05Triage→03Medium a:03ssingh [14:28:50] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "I would not push all the tags, but just the :latest to the old names, as we want people to transition progressively. Otherwise, good catch" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700204 (owner: 10JMeybohm) [14:29:45] (03PS1) 10Jbond: C:locales: Add ability to customise the installed locales [puppet] - 10https://gerrit.wikimedia.org/r/700206 (https://phabricator.wikimedia.org/T285086) [14:30:21] (03CR) 10jerkins-bot: [V: 04-1] C:locales: Add ability to customise the installed locales [puppet] - 10https://gerrit.wikimedia.org/r/700206 (https://phabricator.wikimedia.org/T285086) (owner: 10Jbond) [14:30:36] (03PS1) 10Giuseppe Lavagetto: mediawiki: further fix to the logic of vhosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/700207 [14:30:40] (03CR) 10Jbond: [V: 03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29919/console" [puppet] - 10https://gerrit.wikimedia.org/r/700206 (https://phabricator.wikimedia.org/T285086) (owner: 10Jbond) [14:30:54] (03PS2) 10JMeybohm: docker::baseimages: Push images with legacy names [puppet] - 10https://gerrit.wikimedia.org/r/700204 [14:31:23] (03CR) 10JMeybohm: docker::baseimages: Push images with legacy names (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700204 (owner: 10JMeybohm) [14:31:41] (03PS2) 10Jbond: C:locales: Add ability to customise the installed locales [puppet] - 10https://gerrit.wikimedia.org/r/700206 (https://phabricator.wikimedia.org/T285086) [14:34:05] (03CR) 10Jbond: C:locales: Add ability to customise the installed locales [puppet] - 10https://gerrit.wikimedia.org/r/700206 (https://phabricator.wikimedia.org/T285086) (owner: 10Jbond) [14:36:13] (03PS3) 10Jbond: C:locales: Add ability to customise the installed locales [puppet] - 10https://gerrit.wikimedia.org/r/700206 (https://phabricator.wikimedia.org/T285086) [14:36:29] (03CR) 10Giuseppe Lavagetto: [C: 03+2] mediawiki: further fix to the logic of vhosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/700207 (owner: 10Giuseppe Lavagetto) [14:37:07] (03CR) 10jerkins-bot: [V: 04-1] C:locales: Add ability to customise the installed locales [puppet] - 10https://gerrit.wikimedia.org/r/700206 (https://phabricator.wikimedia.org/T285086) (owner: 10Jbond) [14:38:59] (03Merged) 10jenkins-bot: mediawiki: further fix to the logic of vhosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/700207 (owner: 10Giuseppe Lavagetto) [14:47:13] 10SRE, 10netops: Cloud IPv6 subnets - https://phabricator.wikimedia.org/T187929 (10aborrero) Ok, so the plan would be to have: * `2a02:ec80:0::/48` - cloud eqiad1 * `2a02:ec80:1::/48` - cloud codfw1dev Please confirm and request approvals as required. [14:47:56] (03PS1) 10Giuseppe Lavagetto: mediawiki: bump chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/700211 [14:50:16] 10SRE, 10netbox: Error in postgres puppettization for new installation (was Netbox: postgres cannot be restarted w/ current config) - https://phabricator.wikimedia.org/T184634 (10ayounsi) 05Open→03Resolved a:03ayounsi Talked with John who is working on Postgres for PuppetDB, the last issue is not happeni... [14:53:31] (03CR) 10Giuseppe Lavagetto: [C: 03+2] mediawiki: bump chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/700211 (owner: 10Giuseppe Lavagetto) [14:54:58] (03PS4) 10Jbond: C:locales: Add ability to customise the installed locales [puppet] - 10https://gerrit.wikimedia.org/r/700206 (https://phabricator.wikimedia.org/T285086) [14:56:56] (03Merged) 10jenkins-bot: mediawiki: bump chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/700211 (owner: 10Giuseppe Lavagetto) [14:59:29] 10SRE, 10SRE-Access-Requests: Access to ptwikinews Search Console for Edu - https://phabricator.wikimedia.org/T285091 (10Edu) @Aklapper Hi! Yes, I already signed the non-disclosure agreement [15:00:27] (03PS1) 10Arturo Borrero Gonzalez: prometheus: node_cloudvirt_ceph_network: sort node list [puppet] - 10https://gerrit.wikimedia.org/r/700214 [15:00:53] 10SRE, 10IDS-extension, 10Wikimedia Taiwan, 10Wikimedia-Extension-setup, and 2 others: Deploy IDS rendering engine to production - https://phabricator.wikimedia.org/T148693 (10Aklapper) 05Open→03Declined p:05Medium→03Low https://www.mediawiki.org/wiki/Extension:IDSextension does not exist plus nume... [15:02:01] (03CR) 10Arturo Borrero Gonzalez: [C: 03+1] "PCC https://puppet-compiler.wmflabs.org/compiler1002/29921/" [puppet] - 10https://gerrit.wikimedia.org/r/700214 (owner: 10Arturo Borrero Gonzalez) [15:02:06] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] prometheus: node_cloudvirt_ceph_network: sort node list [puppet] - 10https://gerrit.wikimedia.org/r/700214 (owner: 10Arturo Borrero Gonzalez) [15:07:28] RECOVERY - SSH on wdqs2001.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [15:20:10] PROBLEM - Check systemd state on mwmaint1002 is CRITICAL: CRITICAL - degraded: The following units failed: mediawiki_job_wikibase_repo_prune_test.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [15:31:08] RECOVERY - Check systemd state on mwmaint1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [15:36:23] (03CR) 10BryanDavis: grid: php config don't rely on php being installed by puppet (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700186 (owner: 10David Caro) [15:38:55] (03PS1) 10Ayounsi: Check DNS name match device name [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700218 (https://phabricator.wikimedia.org/T237464) [15:39:35] (03CR) 10jerkins-bot: [V: 04-1] Check DNS name match device name [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700218 (https://phabricator.wikimedia.org/T237464) (owner: 10Ayounsi) [15:43:12] (03PS2) 10Ayounsi: Check DNS name match device name [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700218 (https://phabricator.wikimedia.org/T237464) [15:45:36] (03CR) 10Ayounsi: "Tested in netbox-next." [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700218 (https://phabricator.wikimedia.org/T237464) (owner: 10Ayounsi) [15:47:44] (03CR) 10MSantos: [C: 03+1] osm: create missing imposm directories, add mirror support to import [puppet] - 10https://gerrit.wikimedia.org/r/699044 (https://phabricator.wikimedia.org/T269582) (owner: 10Hnowlan) [16:04:50] (03CR) 10David Caro: grid: php config don't rely on php being installed by puppet (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700186 (owner: 10David Caro) [16:07:46] (03PS2) 10Majavah: metricsinfra: Monitor toolsbeta [puppet] - 10https://gerrit.wikimedia.org/r/700082 [16:15:08] (03PS6) 10Hnowlan: osm: create missing imposm directories, add mirror support to import [puppet] - 10https://gerrit.wikimedia.org/r/699044 (https://phabricator.wikimedia.org/T269582) [16:20:33] (03PS1) 10Majavah: metricsinfra: Reorganize things [puppet] - 10https://gerrit.wikimedia.org/r/700222 [16:24:03] (03PS2) 10Majavah: metricsinfra: Reorganize things [puppet] - 10https://gerrit.wikimedia.org/r/700222 [16:47:58] 10SRE, 10Datacenter-Switchover, 10Performance-Team (Radar): June 2021 Datacenter switchover - https://phabricator.wikimedia.org/T281515 (10Legoktm) [16:48:18] 10SRE, 10DBA, 10Datacenter-Switchover: Check "Days in advance preparation" for databases before DC switchover - https://phabricator.wikimedia.org/T285069 (10Legoktm) [16:48:22] 10SRE, 10Datacenter-Switchover, 10Performance-Team (Radar): June 2021 Datacenter switchover - https://phabricator.wikimedia.org/T281515 (10Legoktm) [16:50:15] 10SRE, 10netops: Cloud IPv6 subnets - https://phabricator.wikimedia.org/T187929 (10cmooney) My own preference would be to allocate larger ranges to each site as mentioned above, and allocate the cloud prefixes from within those geographic aggregates. Doesn't have to be that way of course, I guess we can see w... [17:02:35] (03PS3) 10Majavah: metricsinfra: Monitor toolsbeta [puppet] - 10https://gerrit.wikimedia.org/r/700082 [17:02:37] (03PS3) 10Majavah: metricsinfra: Reorganize things [puppet] - 10https://gerrit.wikimedia.org/r/700222 [17:22:05] 10SRE, 10Wikimedia-Mailing-lists: Figure out mailman3 search index config - https://phabricator.wikimedia.org/T279701 (10Legoktm) 05Open→03Resolved a:03Legoktm xapian seems to be working fine for now. ` root@lists1001:/var/lib/mailman3/web# du -hs fulltext_xapian_index/ 50G fulltext_xapian_index/ ` > O... [17:38:26] (03CR) 10Bstorm: [C: 03+1] "We have typically left out toolsbeta on purpose, but it honestly is really annoying when we discover it is broken from long forgotten chan" [puppet] - 10https://gerrit.wikimedia.org/r/700082 (owner: 10Majavah) [17:40:58] 10SRE, 10MW-on-K8s, 10serviceops: Create a gateway in kubernetes for the execution of our "lambdas" - https://phabricator.wikimedia.org/T261277 (10Legoktm) @joe is everything in this ticket now covered by Shellbox? [17:41:30] (03CR) 10Majavah: "> Patch Set 3: Code-Review+1" [puppet] - 10https://gerrit.wikimedia.org/r/700082 (owner: 10Majavah) [17:46:48] 10SRE, 10Services, 10Wikibase-Quality-Constraints, 10serviceops, 10Service-deployment-requests: Deploy Shellbox instance (shellbox-constraints) for Wikidata constraint regexes - https://phabricator.wikimedia.org/T285104 (10Legoktm) [18:01:31] !log Deployed latest scap code to beta cluster [18:01:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:09:48] (03CR) 10Volans: "Nice! I've added some minor comments, no blockers." (035 comments) [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700218 (https://phabricator.wikimedia.org/T237464) (owner: 10Ayounsi) [18:09:58] PROBLEM - SSH on wdqs2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [18:16:50] 10SRE, 10Wikimedia-Mailing-lists: Add link to list archives in default footer - https://phabricator.wikimedia.org/T284256 (10Legoktm) {P16606} At this scale we probably want some automatic clean up script. [18:20:04] (03PS3) 10Ayounsi: Check DNS name match device name [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700218 (https://phabricator.wikimedia.org/T237464) [18:21:23] (03CR) 10Ayounsi: [C: 03+2] Allow ignoring LibreNMS devices [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700201 (owner: 10Ayounsi) [18:22:20] (03Merged) 10jenkins-bot: Allow ignoring LibreNMS devices [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700201 (owner: 10Ayounsi) [18:24:08] !log T285106 [WDQS] `ryankemper@wdqs2001:~$ sudo depool` [18:24:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:24:13] T285106: hw troubleshooting: SSH failure for wdqs2001.mgmt.codfw.wmnet - https://phabricator.wikimedia.org/T285106 [18:25:03] (03CR) 10Ayounsi: Check DNS name match device name (035 comments) [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700218 (https://phabricator.wikimedia.org/T237464) (owner: 10Ayounsi) [18:25:15] (03CR) 10Ayounsi: [C: 03+2] Check DNS name match device name [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700218 (https://phabricator.wikimedia.org/T237464) (owner: 10Ayounsi) [18:26:17] (03Merged) 10jenkins-bot: Check DNS name match device name [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700218 (https://phabricator.wikimedia.org/T237464) (owner: 10Ayounsi) [18:33:24] (03PS1) 10Ayounsi: Add frack support to test_mgmt_dns_hostname [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700246 (https://phabricator.wikimedia.org/T237464) [18:34:34] (03CR) 10Ayounsi: [C: 03+2] Add frack support to test_mgmt_dns_hostname [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700246 (https://phabricator.wikimedia.org/T237464) (owner: 10Ayounsi) [18:35:15] (03Merged) 10jenkins-bot: Add frack support to test_mgmt_dns_hostname [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/700246 (https://phabricator.wikimedia.org/T237464) (owner: 10Ayounsi) [19:48:46] (03CR) 10Bstorm: [C: 03+2] "> Patch Set 3:" [puppet] - 10https://gerrit.wikimedia.org/r/700082 (owner: 10Majavah) [19:55:50] (03CR) 10Bstorm: "PCC looks right: https://puppet-compiler.wmflabs.org/compiler1001/29923/" [puppet] - 10https://gerrit.wikimedia.org/r/700222 (owner: 10Majavah) [19:56:00] (03CR) 10Bstorm: [C: 03+2] metricsinfra: Reorganize things [puppet] - 10https://gerrit.wikimedia.org/r/700222 (owner: 10Majavah) [21:30:27] 10SRE, 10Traffic, 10HTTPS: en.wikipedia.com [sic] serves an invalid certificate - https://phabricator.wikimedia.org/T214253 (10Aklapper) [21:30:30] 10SRE, 10Traffic: Switch port 80 to nginx on primary clusters - https://phabricator.wikimedia.org/T107236 (10Aklapper) [21:30:36] 10SRE, 10Domains, 10Traffic, 10Wikimedia-Apache-configuration: en-wp.org certificate error - https://phabricator.wikimedia.org/T190244 (10Aklapper) [21:30:42] 10SRE, 10Traffic, 10HTTPS, 10Tracking-Neverending: HTTPS Plans (tracking / high-level info) - https://phabricator.wikimedia.org/T104681 (10Aklapper) [21:30:49] 10SRE, 10Traffic, 10WMF-Legal: Policy decisions for new (and current) DNS domains registered to the WMF - https://phabricator.wikimedia.org/T101048 (10Aklapper) [21:31:26] 10SRE, 10Traffic, 10Goal, 10HTTPS: Create a secure redirect service for large count of non-canonical / junk domains - https://phabricator.wikimedia.org/T133548 (10Aklapper) 05Open→03Resolved @Vgutierrez: No reply; assuming this is resolved. If not, please reopen. [21:39:02] (03PS2) 10Legoktm: mailman3: Don't redirect pipermail messages with duplicate Message-IDs [puppet] - 10https://gerrit.wikimedia.org/r/690391 (https://phabricator.wikimedia.org/T280731) [21:41:02] (03CR) 10Legoktm: [C: 03+2] mailman3: Don't redirect pipermail messages with duplicate Message-IDs [puppet] - 10https://gerrit.wikimedia.org/r/690391 (https://phabricator.wikimedia.org/T280731) (owner: 10Legoktm) [21:49:48] !log regenerating pipermail redirects to skip those with duplicate message-ids (T280731) [21:49:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:49:54] T280731: Implement static redirects from pipermail archives to hyperkitty archives - https://phabricator.wikimedia.org/T280731 [21:56:53] (03PS2) 10Legoktm: decom lists1002/lists-next [puppet] - 10https://gerrit.wikimedia.org/r/689313 (https://phabricator.wikimedia.org/T281548) [21:58:22] (03CR) 10Legoktm: [V: 03+1] "PCC SUCCESS (DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29924/console" [puppet] - 10https://gerrit.wikimedia.org/r/689313 (https://phabricator.wikimedia.org/T281548) (owner: 10Legoktm) [21:59:02] (03CR) 10Legoktm: [V: 03+1 C: 03+2] decom lists1002/lists-next [puppet] - 10https://gerrit.wikimedia.org/r/689313 (https://phabricator.wikimedia.org/T281548) (owner: 10Legoktm) [22:00:05] (03PS2) 10Legoktm: backup: Exclude /var/lib/mailman3/queue [puppet] - 10https://gerrit.wikimedia.org/r/688383 [22:01:51] (03CR) 10Legoktm: [C: 03+2] backup: Exclude /var/lib/mailman3/queue [puppet] - 10https://gerrit.wikimedia.org/r/688383 (owner: 10Legoktm) [22:09:37] (03PS3) 10Legoktm: redis: Get rid of distro-specific config [puppet] - 10https://gerrit.wikimedia.org/r/682901 [22:19:39] (03CR) 10Legoktm: [V: 03+1 C: 03+2] "PCC SUCCESS (NOOP 6): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29925/console" [puppet] - 10https://gerrit.wikimedia.org/r/689313 (https://phabricator.wikimedia.org/T281548) (owner: 10Legoktm) [22:19:59] (03CR) 10Legoktm: [C: 03+2] redis: Get rid of distro-specific config [puppet] - 10https://gerrit.wikimedia.org/r/682901 (owner: 10Legoktm) [22:24:36] (03PS7) 10Legoktm: exim: Drop support for legacy mailing list domains [puppet] - 10https://gerrit.wikimedia.org/r/681242 (https://phabricator.wikimedia.org/T280472) [22:24:38] (03PS4) 10Legoktm: exim: Clean up remnants of legacy_mailing_lists [puppet] - 10https://gerrit.wikimedia.org/r/681724 (https://phabricator.wikimedia.org/T280472) [22:28:22] (03CR) 10Legoktm: "Herron, could you take a look at this since it touches the main MXes?" [puppet] - 10https://gerrit.wikimedia.org/r/681242 (https://phabricator.wikimedia.org/T280472) (owner: 10Legoktm) [22:55:58] 10SRE, 10observability, 10Patch-For-Review, 10Performance-Team (Radar): Fully migrate producers off statsd - https://phabricator.wikimedia.org/T205870 (10Pchelolo) [22:59:17] (03PS5) 10Legoktm: mailman: Drop absented files and packages [puppet] - 10https://gerrit.wikimedia.org/r/697635 (https://phabricator.wikimedia.org/T282303) (owner: 10Ladsgroup) [22:59:19] (03PS5) 10Legoktm: backup: Drop mm2 exclude backups [puppet] - 10https://gerrit.wikimedia.org/r/697637 (https://phabricator.wikimedia.org/T282303) (owner: 10Ladsgroup) [22:59:21] (03PS2) 10Legoktm: mailman: Drop lists3 role [puppet] - 10https://gerrit.wikimedia.org/r/698306 (https://phabricator.wikimedia.org/T282303) (owner: 10Ladsgroup) [23:03:58] (03CR) 10Legoktm: [C: 04-1] "PCC failed, see inline comment" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/697635 (https://phabricator.wikimedia.org/T282303) (owner: 10Ladsgroup) [23:52:10] PROBLEM - Host wdqs2001 is DOWN: PING CRITICAL - Packet loss = 100% [23:56:00] RECOVERY - Host wdqs2001 is UP: PING OK - Packet loss = 0%, RTA = 33.94 ms