[00:30:52] <icinga-wm>	 RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[01:37:47] <jinxer-wm>	 (Processor usage over 85%) firing: Processor usage over 85%   - https://alerts.wikimedia.org
[01:42:47] <jinxer-wm>	 (Processor usage over 85%) resolved: Processor usage over 85%   - https://alerts.wikimedia.org
[01:47:47] <jinxer-wm>	 (Processor usage over 85%) firing: Processor usage over 85%   - https://alerts.wikimedia.org
[01:52:47] <jinxer-wm>	 (Processor usage over 85%) resolved: Processor usage over 85%   - https://alerts.wikimedia.org
[01:57:47] <jinxer-wm>	 (Processor usage over 85%) firing: Processor usage over 85%   - https://alerts.wikimedia.org
[02:02:47] <jinxer-wm>	 (Processor usage over 85%) resolved: Processor usage over 85%   - https://alerts.wikimedia.org
[02:07:47] <jinxer-wm>	 (Processor usage over 85%) firing: Processor usage over 85%   - https://alerts.wikimedia.org
[02:12:47] <jinxer-wm>	 (Processor usage over 85%) resolved: Processor usage over 85%   - https://alerts.wikimedia.org
[02:16:32] <wikibugs>	 (03CR) 10Krinkle: [C: 03+1] scap: Drop never-used 'sqldump' tool [puppet] - 10https://gerrit.wikimedia.org/r/692370 (owner: 10Zabe)
[02:20:18] <wikibugs>	 (03CR) 10Krinkle: [C: 03+1] "I've emailed ops-l just in case." [puppet] - 10https://gerrit.wikimedia.org/r/692370 (owner: 10Zabe)
[02:21:46] <jinxer-wm>	 (Traffic on tunnel link) firing: Traffic on tunnel link   - https://alerts.wikimedia.org
[02:26:46] <jinxer-wm>	 (Traffic on tunnel link) resolved: Traffic on tunnel link   - https://alerts.wikimedia.org
[03:02:10] <icinga-wm>	 PROBLEM - SSH on mw1284.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[04:02:52] <icinga-wm>	 RECOVERY - SSH on mw1284.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[04:20:14] <icinga-wm>	 PROBLEM - rpki grafana alert on alert1001 is CRITICAL: CRITICAL: RPKI ( https://grafana.wikimedia.org/d/UwUa77GZk/rpki ) is alerting: rsync status alert. https://wikitech.wikimedia.org/wiki/RPKI%23Grafana_alerts https://grafana.wikimedia.org/d/UwUa77GZk/
[04:37:47] <jinxer-wm>	 (Processor usage over 85%) firing: Processor usage over 85%   - https://alerts.wikimedia.org
[04:39:09] <wikibugs>	 10SRE, 10serviceops, 10Datacenter-Switchover, 10Performance-Team (Radar): June 2021 Datacenter switchover - https://phabricator.wikimedia.org/T281515 (10wkandek)
[04:39:41] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1099:3311 T163532', diff saved to https://phabricator.wikimedia.org/P16645 and previous config saved to /var/cache/conftool/dbconfig/20210621-043941-marostegui.json
[04:39:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[04:39:47] <stashbot>	 T163532: Drop index rev_page_id (rev_page, rev_id) - https://phabricator.wikimedia.org/T163532
[04:40:36] <marostegui>	 !log Re-add rev_page_id to db1099:3311 T163532 T285149 
[04:40:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[04:40:41] <stashbot>	 T285149: Schema change for dropping rev_page_id index - https://phabricator.wikimedia.org/T285149
[04:42:47] <jinxer-wm>	 (Processor usage over 85%) resolved: Processor usage over 85%   - https://alerts.wikimedia.org
[04:47:46] <jinxer-wm>	 (Processor usage over 85%) firing: Processor usage over 85%   - https://alerts.wikimedia.org
[04:47:55] <kormat>	 marostegui: o/ (FML)
[04:48:42] <logmsgbot>	 !log kormat@cumin1001 START - Cookbook sre.hosts.downtime for 1:00:00 on 21 hosts with reason: Master switchover s3 T284648
[04:48:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[04:48:47] <stashbot>	 T284648: Switchover s3 from db1123 to db1157 - https://phabricator.wikimedia.org/T284648
[04:48:49] <logmsgbot>	 !log kormat@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 21 hosts with reason: Master switchover s3 T284648
[04:48:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[04:49:56] <logmsgbot>	 !log kormat@cumin1001 dbctl commit (dc=all): 'Set db1157 with weight 0 T284648', diff saved to https://phabricator.wikimedia.org/P16646 and previous config saved to /var/cache/conftool/dbconfig/20210621-044955-kormat.json
[04:49:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[04:51:25] <wikibugs>	 (03CR) 10Jcrespo: [C: 03+1] "Not used by backups either." [puppet] - 10https://gerrit.wikimedia.org/r/692370 (owner: 10Zabe)
[04:52:46] <jinxer-wm>	 (Processor usage over 85%) resolved: Processor usage over 85%   - https://alerts.wikimedia.org
[04:54:10] <wikibugs>	 (03PS5) 10Kormat: mariadb: Promote db1157 as s3 primary [puppet] - 10https://gerrit.wikimedia.org/r/698981 (https://phabricator.wikimedia.org/T284648)
[04:55:23] <wikibugs>	 (03CR) 10Kormat: [C: 03+2] mariadb: Promote db1157 as s3 primary [puppet] - 10https://gerrit.wikimedia.org/r/698981 (https://phabricator.wikimedia.org/T284648) (owner: 10Kormat)
[04:55:36] <wikibugs>	 (03PS3) 10Jcrespo: dbbackups: Switchover s3 backup source from db1171 to db1102 (buster) [puppet] - 10https://gerrit.wikimedia.org/r/692845 (https://phabricator.wikimedia.org/T283131)
[04:56:28] <wikibugs>	 (03PS3) 10Kormat: wmnet: Update s3-master to db1157 [dns] - 10https://gerrit.wikimedia.org/r/698982 (https://phabricator.wikimedia.org/T284648)
[04:57:47] <jinxer-wm>	 (Processor usage over 85%) firing: Processor usage over 85%   - https://alerts.wikimedia.org
[04:57:54] <wikibugs>	 (03PS4) 10KartikMistry: Add support for Elia MT to cxserver [deployment-charts] - 10https://gerrit.wikimedia.org/r/699089 (https://phabricator.wikimedia.org/T276059)
[04:58:05] <wikibugs>	 (03PS1) 10Marostegui: mariadb: Promote db1130 to s5 master. [puppet] - 10https://gerrit.wikimedia.org/r/700462 (https://phabricator.wikimedia.org/T284529)
[04:58:26] <wikibugs>	 (03CR) 10Marostegui: [C: 04-2] "Wait for the failover day" [puppet] - 10https://gerrit.wikimedia.org/r/700462 (https://phabricator.wikimedia.org/T284529) (owner: 10Marostegui)
[04:59:24] <wikibugs>	 (03CR) 10Kormat: [C: 03+1] mariadb: Promote db1130 to s5 master. [puppet] - 10https://gerrit.wikimedia.org/r/700462 (https://phabricator.wikimedia.org/T284529) (owner: 10Marostegui)
[05:01:54] <kormat>	 marostegui: alright, here we go.
[05:01:59] <kormat>	 !log Starting s3 eqiad failover from db1123 to db1157 - T284648
[05:02:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[05:02:04] <stashbot>	 T284648: Switchover s3 from db1123 to db1157 - https://phabricator.wikimedia.org/T284648
[05:02:06] <sobanski>	 drumroll
[05:02:17] <jynus>	 around
[05:02:47] <jinxer-wm>	 (Processor usage over 85%) resolved: Processor usage over 85%   - https://alerts.wikimedia.org
[05:03:04] <logmsgbot>	 !log kormat@cumin1001 dbctl commit (dc=all): 'Set s3 eqiad as read-only for maintenance - T284648', diff saved to https://phabricator.wikimedia.org/P16647 and previous config saved to /var/cache/conftool/dbconfig/20210621-050304-kormat.json
[05:03:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[05:03:37] <marostegui>	 confirmed read only on eswikiquote
[05:03:47] <kormat>	 confirmed on ruwikinews
[05:03:54] <marostegui>	 XDDDDD
[05:04:09] <kormat>	 randomly selected, of course. **cough**
[05:04:15] <marostegui>	 totally
[05:04:40] <icinga-wm>	 PROBLEM - Check systemd state on mwmaint1002 is CRITICAL: CRITICAL - degraded: The following units failed: mediawiki_job_wikibase_repo_prune_test.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[05:05:07] <logmsgbot>	 !log kormat@cumin1001 dbctl commit (dc=all): 'Promote db1157 to s3 master and set section read-write T284648', diff saved to https://phabricator.wikimedia.org/P16648 and previous config saved to /var/cache/conftool/dbconfig/20210621-050506-kormat.json
[05:05:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[05:05:28] <jynus>	 rw I can see on my local wiki
[05:06:19] <kormat>	 critical section done
[05:06:24] <marostegui>	 same on eswikiquote
[05:07:09] <jynus>	 no errors on logstash that I can see
[05:07:29] <sobanski>	 Tendril shows the new state
[05:07:46] <jinxer-wm>	 (Processor usage over 85%) firing: Processor usage over 85%   - https://alerts.wikimedia.org
[05:07:47] <wikibugs>	 (03CR) 10Kormat: [C: 03+2] wmnet: Update s3-master to db1157 [dns] - 10https://gerrit.wikimedia.org/r/698982 (https://phabricator.wikimedia.org/T284648) (owner: 10Kormat)
[05:07:55] <marostegui>	 recentchanges seems to be moving on eswikiquote
[05:08:19] <jynus>	 "new" systemd service started/stopped correctly?
[05:08:37] <kormat>	 jynus: yep
[05:08:42] <jynus>	 cool
[05:09:14] <kormat>	 huh. db1171 is still hanging off the old primary, at least on tendril
[05:09:34] <jynus>	 I can see, I thought it was on purpose
[05:09:40] <marostegui>	 kormat: that one has replication stopped
[05:09:47] <kormat>	 ahh, ok
[05:09:47] <marostegui>	 Is that the backup host?
[05:09:59] <jynus>	 it is, but the old one
[05:10:16] <jynus>	 well, I was about to ask if to merge: https://gerrit.wikimedia.org/r/c/operations/puppet/+/692845
[05:10:29] <marostegui>	 jynus: +1 yeah
[05:11:20] <jynus>	 I didn't stop it, did you or is it being backed up right now?
[05:11:28] <kormat>	 marostegui: can confirm that db-switchover did update zarcillo successfully 
[05:11:38] <marostegui>	 I guess the backup is running jynus 
[05:11:41] <marostegui>	 kormat: excellent!
[05:11:44] <jynus>	 cool
[05:11:50] <logmsgbot>	 !log kormat@cumin1001 dbctl commit (dc=all): 'Depool db1123 until it's reimaged to buster T284648', diff saved to https://phabricator.wikimedia.org/P16649 and previous config saved to /var/cache/conftool/dbconfig/20210621-051149-kormat.json
[05:11:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[05:11:54] <stashbot>	 T284648: Switchover s3 from db1123 to db1157 - https://phabricator.wikimedia.org/T284648
[05:11:55] <jynus>	 then the switchover script did the right thing
[05:12:46] <jinxer-wm>	 (Processor usage over 85%) resolved: Processor usage over 85%   - https://alerts.wikimedia.org
[05:13:01] <jynus>	 let's make sure all of heartbeat, query killer is nice and happy everywhere else
[05:14:36] <kormat>	 marostegui: are there any outstanding schema changes that need to be applied to db1123 (old primary)?
[05:15:06] <marostegui>	 yep, let me list them
[05:15:10] <kart_>	 Is it OK to update cxserver now? Anything on deploy1002 that might interfere?
[05:15:46] <icinga-wm>	 RECOVERY - Check systemd state on mwmaint1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[05:15:59] <marostegui>	 kormat: https://phabricator.wikimedia.org/T277123 https://phabricator.wikimedia.org/T268392 https://phabricator.wikimedia.org/T266486 I can apply those
[05:16:22] <marostegui>	 And https://phabricator.wikimedia.org/T276150
[05:16:40] <kormat>	 actually, we can switch back to #-databases now
[05:16:46] <marostegui>	 yep
[05:16:53] <marostegui>	 kart_: you can proceed
[05:17:46] <kart_>	 Thanks.
[05:18:22] <wikibugs>	 (03CR) 10KartikMistry: [C: 03+2] Add support for Elia MT to cxserver [deployment-charts] - 10https://gerrit.wikimedia.org/r/699089 (https://phabricator.wikimedia.org/T276059) (owner: 10KartikMistry)
[05:20:36] <wikibugs>	 (03Merged) 10jenkins-bot: Add support for Elia MT to cxserver [deployment-charts] - 10https://gerrit.wikimedia.org/r/699089 (https://phabricator.wikimedia.org/T276059) (owner: 10KartikMistry)
[05:23:33] <wikibugs>	 (03PS1) 10Kormat: db1123: Disable notifications. [puppet] - 10https://gerrit.wikimedia.org/r/700463 (https://phabricator.wikimedia.org/T283131)
[05:24:31] <wikibugs>	 (03PS4) 10Jcrespo: dbbackups: Switchover s3 backup source from db1171 to db1102 (buster) [puppet] - 10https://gerrit.wikimedia.org/r/692845 (https://phabricator.wikimedia.org/T283131)
[05:24:45] <wikibugs>	 (03CR) 10Kormat: [C: 03+2] db1123: Disable notifications. [puppet] - 10https://gerrit.wikimedia.org/r/700463 (https://phabricator.wikimedia.org/T283131) (owner: 10Kormat)
[05:25:27] <logmsgbot>	 !log kartik@deploy1002 helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' .
[05:25:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[05:26:56] <wikibugs>	 (03CR) 10Jcrespo: [C: 03+2] dbbackups: Switchover s3 backup source from db1171 to db1102 (buster) [puppet] - 10https://gerrit.wikimedia.org/r/692845 (https://phabricator.wikimedia.org/T283131) (owner: 10Jcrespo)
[05:31:19] <kormat>	 !log stopping replication on db1123 T283131
[05:31:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[05:31:23] <stashbot>	 T283131: Upgrade s3 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283131
[05:33:37] <logmsgbot>	 !log kartik@deploy1002 helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' .
[05:33:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[05:36:46] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 25%: Repool db1099:3311 after re-adding rev_page_id index', diff saved to https://phabricator.wikimedia.org/P16650 and previous config saved to /var/cache/conftool/dbconfig/20210621-053645-root.json
[05:36:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[05:41:19] <logmsgbot>	 !log kartik@deploy1002 helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' .
[05:41:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[05:49:01] <icinga-wm>	 PROBLEM - MariaDB Replica Lag: s3 on db1102 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 1067.29 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
[05:50:31] <kart_>	 !log cxserver: Added support for Elia MT  + Updated to 2021-06-10-074331-production (T276059, T275803, T276246, T283513, T255231, T237028)
[05:50:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[05:50:44] <stashbot>	 T237028: Evaluate: <style> tag holding template definition - https://phabricator.wikimedia.org/T237028
[05:50:44] <stashbot>	 T276059: Section from the original article not showing either as missing or present - https://phabricator.wikimedia.org/T276059
[05:50:45] <stashbot>	 T255231: Integrate Elia (formerly itzultzailea) MT system - https://phabricator.wikimedia.org/T255231
[05:50:45] <stashbot>	 T276246: Create Wikipedia Kari Seediq - https://phabricator.wikimedia.org/T276246
[05:50:45] <stashbot>	 T283513: issue with having just one section/paragraph in SX - https://phabricator.wikimedia.org/T283513
[05:50:45] <stashbot>	 T275803: Create Wikipedia Atayal - https://phabricator.wikimedia.org/T275803
[05:51:49] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 50%: Repool db1099:3311 after re-adding rev_page_id index', diff saved to https://phabricator.wikimedia.org/P16651 and previous config saved to /var/cache/conftool/dbconfig/20210621-055149-root.json
[05:51:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:01:03] <wikibugs>	 (03PS1) 10KartikMistry: cxserver: Remove Matxin MT support [deployment-charts] - 10https://gerrit.wikimedia.org/r/700466 (https://phabricator.wikimedia.org/T285199)
[06:06:03] <icinga-wm>	 RECOVERY - rpki grafana alert on alert1001 is OK: OK: RPKI ( https://grafana.wikimedia.org/d/UwUa77GZk/rpki ) is not alerting. https://wikitech.wikimedia.org/wiki/RPKI%23Grafana_alerts https://grafana.wikimedia.org/d/UwUa77GZk/
[06:06:53] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 75%: Repool db1099:3311 after re-adding rev_page_id index', diff saved to https://phabricator.wikimedia.org/P16652 and previous config saved to /var/cache/conftool/dbconfig/20210621-060652-root.json
[06:06:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:20:15] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1135 T163532', diff saved to https://phabricator.wikimedia.org/P16653 and previous config saved to /var/cache/conftool/dbconfig/20210621-062014-marostegui.json
[06:20:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:20:20] <stashbot>	 T163532: Drop index rev_page_id (rev_page, rev_id) - https://phabricator.wikimedia.org/T163532
[06:20:43] <marostegui>	 !log Re-add rev_page_id to db1135 T163532 T285149 
[06:20:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:20:49] <stashbot>	 T285149: Schema change for dropping rev_page_id index - https://phabricator.wikimedia.org/T285149
[06:21:56] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 100%: Repool db1099:3311 after re-adding rev_page_id index', diff saved to https://phabricator.wikimedia.org/P16654 and previous config saved to /var/cache/conftool/dbconfig/20210621-062156-root.json
[06:21:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:22:28] <wikibugs>	 (03PS2) 10Giuseppe Lavagetto: mediawiki: properly support wikibase rewrites [deployment-charts] - 10https://gerrit.wikimedia.org/r/700387
[06:22:59] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mediawiki: properly support wikibase rewrites [deployment-charts] - 10https://gerrit.wikimedia.org/r/700387 (owner: 10Giuseppe Lavagetto)
[06:24:44] <wikibugs>	 (03PS3) 10Giuseppe Lavagetto: mediawiki: properly support wikibase rewrites [deployment-charts] - 10https://gerrit.wikimedia.org/r/700387
[06:25:08] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mediawiki: properly support wikibase rewrites [deployment-charts] - 10https://gerrit.wikimedia.org/r/700387 (owner: 10Giuseppe Lavagetto)
[06:27:29] <wikibugs>	 (03PS1) 10Marostegui: db2079: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/700468
[06:28:05] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db2079: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/700468 (owner: 10Marostegui)
[06:29:51] <wikibugs>	 (03PS4) 10Giuseppe Lavagetto: mediawiki: properly support wikibase rewrites [deployment-charts] - 10https://gerrit.wikimedia.org/r/700387
[06:31:33] <elukey>	 marostegui, kormat - 8:30 AM on Monday and you have already done a master failover with follow ups, light start of the week :D
[06:31:36] <elukey>	 (kudos)
[06:31:48] <kormat>	 elukey: and tomorrow we do it again 😭
[06:32:14] <marostegui>	 elukey: I am trying to get kormat on the habit of starting the day early, so she can enjoy the morning to the full
[06:32:26] <elukey>	 good point!
[06:32:32] <kormat>	 ಠ_ಠ
[06:32:35] <elukey>	 :D
[06:43:47] <wikibugs>	 10SRE, 10MediaWiki-Uploading, 10Traffic, 10Chinese-Sites, 10Performance Issue: Adding an image to zh.wp 长江桥隧列表 article throws HTTP 503 or 504 error - https://phabricator.wikimedia.org/T285160 (10Aklapper) @IN: Please summarize outcomes when adding comments, and please avoid announcements-only. Every new...
[06:49:47] <moritzm>	 !log installing libwebp security updates on buster
[06:49:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:51:33] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [V: 03+1] "PCC SUCCESS (DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29937/console" [puppet] - 10https://gerrit.wikimedia.org/r/700386 (owner: 10Giuseppe Lavagetto)
[06:54:42] <wikibugs>	 (03PS8) 10Elukey: Add the custom_deploy.d directory with basic Istio config [deployment-charts] - 10https://gerrit.wikimedia.org/r/697938 (https://phabricator.wikimedia.org/T278192)
[06:54:44] <wikibugs>	 (03PS8) 10Elukey: Add support for knative serving [deployment-charts] - 10https://gerrit.wikimedia.org/r/699380 (https://phabricator.wikimedia.org/T278194)
[06:55:24] <wikibugs>	 (03PS1) 10Elukey: WIP - Add kubeflow's kfserving chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/700470 (https://phabricator.wikimedia.org/T272919)
[06:55:54] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] WIP - Add kubeflow's kfserving chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/700470 (https://phabricator.wikimedia.org/T272919) (owner: 10Elukey)
[06:56:56] <wikibugs>	 (03CR) 10Elukey: "Just added a global option for Istio's version." [deployment-charts] - 10https://gerrit.wikimedia.org/r/697938 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey)
[07:28:10] <wikibugs>	 10SRE, 10ops-codfw, 10DC-Ops, 10observability, 10User-fgiunchedi: codfw: Testing Out Sample PDUs - https://phabricator.wikimedia.org/T265435 (10fgiunchedi) >>! In T265435#7161900, @wiki_willy wrote: > Hi Filippo - is this following link the one used for phase monitoring? >  > https://librenms.wikimedia.o...
[07:33:24] <wikibugs>	 (03PS1) 10JMeybohm: systemd::timer::job Adapt docs to new defaults [puppet] - 10https://gerrit.wikimedia.org/r/700471
[07:33:46] <wikibugs>	 (03PS1) 10Jcrespo: dbbackups: Increase back snapshot retention now that s7/s8 hw is fixed [puppet] - 10https://gerrit.wikimedia.org/r/700472 (https://phabricator.wikimedia.org/T284980)
[07:36:17] <wikibugs>	 (03CR) 10Jcrespo: [C: 03+2] dbbackups: Increase back snapshot retention now that s7/s8 hw is fixed [puppet] - 10https://gerrit.wikimedia.org/r/700472 (https://phabricator.wikimedia.org/T284980) (owner: 10Jcrespo)
[07:37:47] <jinxer-wm>	 (Processor usage over 85%) firing: Processor usage over 85%   - https://alerts.wikimedia.org
[07:38:13] <wikibugs>	 (03CR) 10Hashar: [C: 03+1] "That got introduced by Sam Reed back in 2013: https://gerrit.wikimedia.org/r/c/operations/puppet/+/43844" [puppet] - 10https://gerrit.wikimedia.org/r/692370 (owner: 10Zabe)
[07:42:17] <icinga-wm>	 RECOVERY - Check systemd state on deneb is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[07:42:47] <jinxer-wm>	 (Processor usage over 85%) resolved: Processor usage over 85%   - https://alerts.wikimedia.org
[07:44:17] <jayme>	 !log started debian-weekly-rebuild.service on deneb (it failed due to 404 on snapshots.debian.org yesterday)
[07:44:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:47:47] <jinxer-wm>	 (Processor usage over 85%) firing: Processor usage over 85%   - https://alerts.wikimedia.org
[07:47:48] <moritzm>	 !log updated buster d-i image for Buster 10.10 point release (which included ABI bump for Linux kernel)
[07:47:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:52:37] <wikibugs>	 (03PS1) 10Jcrespo: dbbackups: Migrate db1171:s2 to db1139, reimage as buster and set s7&s8 [puppet] - 10https://gerrit.wikimedia.org/r/700473 (https://phabricator.wikimedia.org/T280979)
[07:56:42] <wikibugs>	 (03CR) 10Jcrespo: "s2 has not been migrated away yet, this has to happen before reimage." [puppet] - 10https://gerrit.wikimedia.org/r/700473 (https://phabricator.wikimedia.org/T280979) (owner: 10Jcrespo)
[07:57:46] <jinxer-wm>	 (Processor usage over 85%) firing: Processor usage over 85%   - https://alerts.wikimedia.org
[07:59:34] <wikibugs>	 (03PS2) 10Jcrespo: dbbackups: Migrate db1171:s2 to db1139, reimage as buster and set s7&s8 [puppet] - 10https://gerrit.wikimedia.org/r/700473 (https://phabricator.wikimedia.org/T280979)
[08:02:46] <jinxer-wm>	 (Processor usage over 85%) resolved: Processor usage over 85%   - https://alerts.wikimedia.org
[08:05:27] <wikibugs>	 (03PS1) 10Jcrespo: dbbackups: Reenable notifications and remove db1171 future reimages [puppet] - 10https://gerrit.wikimedia.org/r/700474 (https://phabricator.wikimedia.org/T280979)
[08:06:23] <wikibugs>	 (03PS3) 10Jcrespo: dbbackups: Migrate db1171:s2 to db1139, reimage as buster and set s7&s8 [puppet] - 10https://gerrit.wikimedia.org/r/700473 (https://phabricator.wikimedia.org/T280979)
[08:06:48] <wikibugs>	 (03PS2) 10KartikMistry: cxserver: Remove Matxin MT support [deployment-charts] - 10https://gerrit.wikimedia.org/r/700466 (https://phabricator.wikimedia.org/T285199)
[08:07:47] <jinxer-wm>	 (Processor usage over 85%) firing: Processor usage over 85%   - https://alerts.wikimedia.org
[08:08:12] <wikibugs>	 (03PS4) 10Jcrespo: dbbackups: Migrate db1171:s2 to db1139, reimage as buster and set s7&s8 [puppet] - 10https://gerrit.wikimedia.org/r/700473 (https://phabricator.wikimedia.org/T280979)
[08:10:37] <wikibugs>	 (03PS2) 10Jcrespo: dbbackups: Reenable notifications and remove db1171 future reimages [puppet] - 10https://gerrit.wikimedia.org/r/700474 (https://phabricator.wikimedia.org/T280979)
[08:12:47] <jinxer-wm>	 (Processor usage over 85%) resolved: Processor usage over 85%   - https://alerts.wikimedia.org
[08:31:48] <dcausse>	 !log depooling wdqs1005 (lag)
[08:31:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:34:12] <wikibugs>	 10SRE: Integrate Buster 10.10 point update - https://phabricator.wikimedia.org/T285206 (10MoritzMuehlenhoff)
[08:34:14] <wikibugs>	 (03PS4) 10Volans: cumin: add support for Kerberos auth [software/cumin] - 10https://gerrit.wikimedia.org/r/699192 (https://phabricator.wikimedia.org/T244840) (owner: 10Muehlenhoff)
[08:34:16] <wikibugs>	 (03PS1) 10Volans: doc: use add_css_file() instead of add_stylesheet() [software/cumin] - 10https://gerrit.wikimedia.org/r/700479
[08:34:18] <wikibugs>	 (03PS1) 10Volans: setup.py: add types dependencies for mypy [software/cumin] - 10https://gerrit.wikimedia.org/r/700480
[08:36:46] <wikibugs>	 10SRE, 10Wikimedia-Mailing-lists, 10User-Ladsgroup: Close wikimediahk@lists.wikimedia.org - https://phabricator.wikimedia.org/T285194 (10Ladsgroup) p:05Triage→03Medium a:03Ladsgroup I'll do it later today. You want to keep the archive. correct?
[08:41:41] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [C: 03+2] mediawiki::web::vhost: add wikibase_rewrites boolean [puppet] - 10https://gerrit.wikimedia.org/r/700385 (owner: 10Giuseppe Lavagetto)
[08:43:25] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] doc: use add_css_file() instead of add_stylesheet() [software/cumin] - 10https://gerrit.wikimedia.org/r/700479 (owner: 10Volans)
[08:45:49] <wikibugs>	 (03CR) 10Volans: "The CI failure is fixed in the next patch. If I invert them the other one will fail because of this missing patch. Both are needed to make" [software/cumin] - 10https://gerrit.wikimedia.org/r/700479 (owner: 10Volans)
[08:46:09] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [V: 03+1 C: 03+2] mediawiki: convert configurations to use wikibase_rewrites [puppet] - 10https://gerrit.wikimedia.org/r/700386 (owner: 10Giuseppe Lavagetto)
[08:53:25] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [C: 03+2] mediawiki: properly support wikibase rewrites [deployment-charts] - 10https://gerrit.wikimedia.org/r/700387 (owner: 10Giuseppe Lavagetto)
[08:54:23] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "I'm not familiar with the existing postgres installations, but the code LGTM" [cookbooks] - 10https://gerrit.wikimedia.org/r/666113 (https://phabricator.wikimedia.org/T275381) (owner: 10Hnowlan)
[08:55:38] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1135 (re)pooling @ 25%: Repool db1135 after re-adding rev_page_id index', diff saved to https://phabricator.wikimedia.org/P16655 and previous config saved to /var/cache/conftool/dbconfig/20210621-085538-root.json
[08:55:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:56:24] <wikibugs>	 (03Merged) 10jenkins-bot: mediawiki: properly support wikibase rewrites [deployment-charts] - 10https://gerrit.wikimedia.org/r/700387 (owner: 10Giuseppe Lavagetto)
[08:56:31] <marostegui>	 !log Deploy T266486 T268392 T273360 on db1123
[08:56:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:56:36] <stashbot>	 T268392: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392
[08:56:37] <stashbot>	 T273360: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360
[08:56:37] <stashbot>	 T266486: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486
[08:59:48] <wikibugs>	 (03PS1) 10Filippo Giunchedi: alertmanager: bump resolve_timeout to 20m [puppet] - 10https://gerrit.wikimedia.org/r/700482 (https://phabricator.wikimedia.org/T285205)
[09:02:07] <logmsgbot>	 !log oblivian@deploy1002 helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
[09:02:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:05:23] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C: 03+2] alertmanager: bump resolve_timeout to 20m [puppet] - 10https://gerrit.wikimedia.org/r/700482 (https://phabricator.wikimedia.org/T285205) (owner: 10Filippo Giunchedi)
[09:10:42] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1135 (re)pooling @ 50%: Repool db1135 after re-adding rev_page_id index', diff saved to https://phabricator.wikimedia.org/P16656 and previous config saved to /var/cache/conftool/dbconfig/20210621-091041-root.json
[09:10:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:13:38] <logmsgbot>	 !log oblivian@deploy1002 helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
[09:13:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:14:36] <wikibugs>	 (03PS1) 10Ladsgroup: Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700484
[09:14:38] <wikibugs>	 (03CR) 10Ladsgroup: [C: 03+2] Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700484 (owner: 10Ladsgroup)
[09:16:07] <icinga-wm>	 PROBLEM - Prometheus jobs reduced availability on alert1001 is CRITICAL: job=atlas_exporter site=codfw https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets
[09:16:25] <wikibugs>	 (03Merged) 10jenkins-bot: Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700484 (owner: 10Ladsgroup)
[09:17:59] <icinga-wm>	 RECOVERY - Prometheus jobs reduced availability on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets
[09:19:10] <logmsgbot>	 !log ladsgroup@deploy1002 Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 04m 49s)
[09:19:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:20:57] <wikibugs>	 (03PS3) 10Muehlenhoff: Add helper tool for returning a user's current TGT [puppet] - 10https://gerrit.wikimedia.org/r/700389 (https://phabricator.wikimedia.org/T283242)
[09:21:43] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add helper tool for returning a user's current TGT [puppet] - 10https://gerrit.wikimedia.org/r/700389 (https://phabricator.wikimedia.org/T283242) (owner: 10Muehlenhoff)
[09:25:13] <wikibugs>	 (03PS4) 10Muehlenhoff: Add helper tool for returning a user's current TGT [puppet] - 10https://gerrit.wikimedia.org/r/700389 (https://phabricator.wikimedia.org/T283242)
[09:25:46] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1135 (re)pooling @ 75%: Repool db1135 after re-adding rev_page_id index', diff saved to https://phabricator.wikimedia.org/P16657 and previous config saved to /var/cache/conftool/dbconfig/20210621-092545-root.json
[09:25:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:25:58] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add helper tool for returning a user's current TGT [puppet] - 10https://gerrit.wikimedia.org/r/700389 (https://phabricator.wikimedia.org/T283242) (owner: 10Muehlenhoff)
[09:26:24] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'Set db1130 with weight 0 T284529', diff saved to https://phabricator.wikimedia.org/P16658 and previous config saved to /var/cache/conftool/dbconfig/20210621-092623-root.json
[09:26:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:26:28] <stashbot>	 T284529: Switchover s5 from db1100 to db1130 - https://phabricator.wikimedia.org/T284529
[09:27:46] <jinxer-wm>	 (Processor usage over 85%) firing: (4) Processor usage over 85%   - https://alerts.wikimedia.org
[09:29:41] <godog>	 this shouldn't flap now ^
[09:33:56] <wikibugs>	 (03PS5) 10Muehlenhoff: Add helper tool for returning a user's current TGT [puppet] - 10https://gerrit.wikimedia.org/r/700389 (https://phabricator.wikimedia.org/T283242)
[09:34:29] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] "LGTM merging" [puppet] - 10https://gerrit.wikimedia.org/r/700331 (owner: 10Paladox)
[09:36:59] <wikibugs>	 (03PS1) 10Ladsgroup: Set $wmgWikibaseTmpSerializeEmptyListsAsObjects to true everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700491 (https://phabricator.wikimedia.org/T241422)
[09:37:23] <Amir1>	 I'm going to deploy a long set of config changes for wikidata
[09:39:23] <wikibugs>	 (03CR) 10Ladsgroup: [C: 03+2] Set $wmgWikibaseTmpSerializeEmptyListsAsObjects to true everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700491 (https://phabricator.wikimedia.org/T241422) (owner: 10Ladsgroup)
[09:40:49] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1135 (re)pooling @ 100%: Repool db1135 after re-adding rev_page_id index', diff saved to https://phabricator.wikimedia.org/P16659 and previous config saved to /var/cache/conftool/dbconfig/20210621-094049-root.json
[09:40:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:42:09] <icinga-wm>	 PROBLEM - Prometheus jobs reduced availability on alert1001 is CRITICAL: job=atlas_exporter site=eqiad https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets
[09:42:21] <wikibugs>	 (03PS1) 10Ladsgroup: Drop LocalFile::getHistory hook handler [extensions/FlaggedRevs] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700340 (https://phabricator.wikimedia.org/T284777)
[09:42:45] <wikibugs>	 (03CR) 10Ladsgroup: [C: 03+2] Drop LocalFile::getHistory hook handler [extensions/FlaggedRevs] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700340 (https://phabricator.wikimedia.org/T284777) (owner: 10Ladsgroup)
[09:42:59] <wikibugs>	 (03Merged) 10jenkins-bot: Set $wmgWikibaseTmpSerializeEmptyListsAsObjects to true everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700491 (https://phabricator.wikimedia.org/T241422) (owner: 10Ladsgroup)
[09:43:47] <icinga-wm>	 PROBLEM - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is CRITICAL: /{domain}/v1/page/metadata/{title} (retrieve extended metadata for Video article on English Wikipedia) is CRITICAL: Test retrieve extended metadata for Video article on English Wikipedia returned the unexpected status 503 (expecting: 200) https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29
[09:43:59] <icinga-wm>	 RECOVERY - Prometheus jobs reduced availability on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets
[09:44:03] <wikibugs>	 (03CR) 10Volans: [V: 03+2 C: 03+2] Add pip's dependencies to the generated wheels [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/700181 (owner: 10Volans)
[09:44:27] <logmsgbot>	 !log ladsgroup@deploy1002 Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:700491|Set $wmgWikibaseTmpSerializeEmptyListsAsObjects to true everywhere (T241422)]] (duration: 00m 57s)
[09:44:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:44:32] <stashbot>	 T241422: Wikidata forms without statements use empty JSON array instead of empty JSON object - https://phabricator.wikimedia.org/T241422
[09:45:39] <icinga-wm>	 RECOVERY - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29
[09:47:12] <wikibugs>	 (03CR) 10Ladsgroup: [C: 03+2] Enable wikisource group as langlink group of sourcewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/698226 (https://phabricator.wikimedia.org/T275958) (owner: 10Ladsgroup)
[09:48:51] <wikibugs>	 (03Merged) 10jenkins-bot: Enable wikisource group as langlink group of sourcewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/698226 (https://phabricator.wikimedia.org/T275958) (owner: 10Ladsgroup)
[09:49:06] <wikibugs>	 (03Merged) 10jenkins-bot: Drop LocalFile::getHistory hook handler [extensions/FlaggedRevs] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700340 (https://phabricator.wikimedia.org/T284777) (owner: 10Ladsgroup)
[09:49:21] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] "LGTM thanks" [puppet] - 10https://gerrit.wikimedia.org/r/700471 (owner: 10JMeybohm)
[09:50:04] <wikibugs>	 (03CR) 10Volans: [C: 03+2] "@dcaro, I'm sending the patch for the one existing usage that needs to be amended. When would be a good time for the Spicerack release tha" [software/spicerack] - 10https://gerrit.wikimedia.org/r/700076 (owner: 10Volans)
[09:50:15] <wikibugs>	 (03CR) 10JMeybohm: [C: 04-1] add chart for miscweb (039 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/698895 (https://phabricator.wikimedia.org/T281538) (owner: 10Dzahn)
[09:50:55] <wikibugs>	 (03CR) 10JMeybohm: [C: 03+2] systemd::timer::job Adapt docs to new defaults [puppet] - 10https://gerrit.wikimedia.org/r/700471 (owner: 10JMeybohm)
[09:52:22] <logmsgbot>	 !log ladsgroup@deploy1002 Synchronized wmf-config/Wikibase.php: Config: [[gerrit:698226|Enable wikisource group as langlink group of sourcewiki (T275958)]] (duration: 00m 56s)
[09:52:25] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] "LGTM" [software/cumin] - 10https://gerrit.wikimedia.org/r/700480 (owner: 10Volans)
[09:52:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:52:27] <stashbot>	 T275958: Enable interlanguage links from multilingual Wikisource to other Wikisources - https://phabricator.wikimedia.org/T275958
[09:52:54] <wikibugs>	 (03PS1) 10Volans: wmcs.ceph.reboot_node: update icinga_hosts call [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/700492
[09:53:16] <wikibugs>	 (03CR) 10David Caro: [C: 03+1] "> Patch Set 2: Code-Review+2" [software/spicerack] - 10https://gerrit.wikimedia.org/r/700076 (owner: 10Volans)
[09:55:32] <logmsgbot>	 !log ladsgroup@deploy1002 Synchronized php-1.37.0-wmf.9/extensions/FlaggedRevs: Backport: [[gerrit:700340|Drop LocalFile::getHistory hook handler (T284777 T277883)]] (duration: 00m 58s)
[09:55:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:55:39] <stashbot>	 T284777: PHP Deprecated: Passing extra fields (quality) to LocalFile::loadFromRow was deprecated in MediaWiki 1.37. Property assignment will be removed in a later version. [Called from OldLocalFile::loadFromRow] - https://phabricator.wikimedia.org/T284777
[09:55:39] <stashbot>	 T277883: Drop all low-use and unused features of FlaggedRevs to make it more maintainable - https://phabricator.wikimedia.org/T277883
[09:56:15] <wikibugs>	 (03CR) 10David Caro: [C: 03+1] wmcs.ceph.reboot_node: update icinga_hosts call [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/700492 (owner: 10Volans)
[09:58:21] <wikibugs>	 (03PS1) 10Giuseppe Lavagetto: mediawiki: use server_name for ServerName. [deployment-charts] - 10https://gerrit.wikimedia.org/r/700493
[09:58:29] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mediawiki: use server_name for ServerName. [deployment-charts] - 10https://gerrit.wikimedia.org/r/700493 (owner: 10Giuseppe Lavagetto)
[10:00:48] <wikibugs>	 (03Merged) 10jenkins-bot: icinga: rename some IcingaHosts methods [software/spicerack] - 10https://gerrit.wikimedia.org/r/700076 (owner: 10Volans)
[10:00:59] <wikibugs>	 (03CR) 10JMeybohm: [V: 03+1] "PCC SUCCESS (DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29938/console" [puppet] - 10https://gerrit.wikimedia.org/r/696403 (https://phabricator.wikimedia.org/T256762) (owner: 10JMeybohm)
[10:01:26] <wikibugs>	 (03PS2) 10Giuseppe Lavagetto: mediawiki: use server_name for ServerName. [deployment-charts] - 10https://gerrit.wikimedia.org/r/700493
[10:04:14] <wikibugs>	 (03PS2) 10JMeybohm: docker_registry_ha: Enable local nginx cache by default [puppet] - 10https://gerrit.wikimedia.org/r/696403 (https://phabricator.wikimedia.org/T256762)
[10:05:12] <wikibugs>	 (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/696403 (https://phabricator.wikimedia.org/T256762) (owner: 10JMeybohm)
[10:06:17] <wikibugs>	 (03CR) 10Jbond: "lgtm few comments, nits inline" (035 comments) [software/cumin] - 10https://gerrit.wikimedia.org/r/699192 (https://phabricator.wikimedia.org/T244840) (owner: 10Muehlenhoff)
[10:07:19] <wikibugs>	 (03PS2) 10David Caro: nova: collect VPS data from the hypervisor [puppet] - 10https://gerrit.wikimedia.org/r/700393
[10:07:21] <wikibugs>	 (03CR) 10David Caro: nova: collect VPS data from the hypervisor (039 comments) [puppet] - 10https://gerrit.wikimedia.org/r/700393 (owner: 10David Caro)
[10:10:26] <logmsgbot>	 !log jbond@deploy1002 Started deploy [netbox/deploy@c7762f8]: deploy v2.10.4-wmf4
[10:10:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:13:14] <wikibugs>	 (03CR) 10JMeybohm: [V: 03+1] "PCC SUCCESS (NOOP 6 DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29939/console" [puppet] - 10https://gerrit.wikimedia.org/r/696403 (https://phabricator.wikimedia.org/T256762) (owner: 10JMeybohm)
[10:13:37] <logmsgbot>	 !log jbond@deploy1002 Finished deploy [netbox/deploy@c7762f8]: deploy v2.10.4-wmf4 (duration: 03m 10s)
[10:13:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:13:52] <logmsgbot>	 !log jbond@deploy1002 Started deploy [netbox/deploy@c7762f8]: deploy v2.10.4-wmf4 to netbox-next
[10:13:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:15:22] <logmsgbot>	 !log jbond@deploy1002 Finished deploy [netbox/deploy@c7762f8]: deploy v2.10.4-wmf4 to netbox-next (duration: 01m 30s)
[10:15:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:15:36] <logmsgbot>	 !log jbond@deploy1002 Started deploy [netbox/deploy@c7762f8]: deploy v2.10.4-wmf4 to netbox-next
[10:15:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:15:59] <wikibugs>	 (03PS3) 10Giuseppe Lavagetto: mediawiki: use server_name for ServerName. [deployment-charts] - 10https://gerrit.wikimedia.org/r/700493
[10:16:06] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mediawiki: use server_name for ServerName. [deployment-charts] - 10https://gerrit.wikimedia.org/r/700493 (owner: 10Giuseppe Lavagetto)
[10:16:15] <joe>	 wut
[10:16:38] <wikibugs>	 (03PS4) 10Giuseppe Lavagetto: mediawiki: use server_name for ServerName. [deployment-charts] - 10https://gerrit.wikimedia.org/r/700493
[10:16:39] <logmsgbot>	 !log jbond@deploy1002 Finished deploy [netbox/deploy@c7762f8]: deploy v2.10.4-wmf4 to netbox-next (duration: 01m 03s)
[10:16:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:17:30] <logmsgbot>	 !log jbond@deploy1002 Started deploy [netbox/deploy@c7762f8]: deploy v2.10.4-wmf4 to netbox-next
[10:17:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:18:07] <icinga-wm>	 PROBLEM - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The following units failed: netbox_report_accounting_run.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[10:19:25] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [C: 03+2] mediawiki: use server_name for ServerName. [deployment-charts] - 10https://gerrit.wikimedia.org/r/700493 (owner: 10Giuseppe Lavagetto)
[10:19:43] <logmsgbot>	 !log jbond@deploy1002 Finished deploy [netbox/deploy@c7762f8]: deploy v2.10.4-wmf4 to netbox-next (duration: 02m 13s)
[10:19:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:20:10] <logmsgbot>	 !log jbond@deploy1002 Started deploy [netbox/deploy@c7762f8]: deploy v2.10.4-wmf4 to netbox-next
[10:20:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:20:36] <wikibugs>	 (03PS1) 10Volans: Force the installation of pip deps first [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/700495
[10:21:19] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] "LGTM" [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/700495 (owner: 10Volans)
[10:22:19] <wikibugs>	 (03Merged) 10jenkins-bot: mediawiki: use server_name for ServerName. [deployment-charts] - 10https://gerrit.wikimedia.org/r/700493 (owner: 10Giuseppe Lavagetto)
[10:22:26] <wikibugs>	 (03CR) 10Volans: [V: 03+2 C: 03+2] Force the installation of pip deps first [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/700495 (owner: 10Volans)
[10:22:33] <logmsgbot>	 !log jbond@deploy1002 Finished deploy [netbox/deploy@c7762f8]: deploy v2.10.4-wmf4 to netbox-next (duration: 02m 22s)
[10:22:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:24:25] <logmsgbot>	 !log jbond@deploy1002 Started deploy [netbox/deploy@6b69f2c]: deploy v2.10.4-wmf4 to netbox-next
[10:24:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:25:32] <wikibugs>	 (03CR) 10Elukey: [C: 04-1] "self -1, I discovered that I need to add also a VERSION env variable to make. The version is injected into pilot's binary and exposed as /" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/700396 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey)
[10:27:37] <logmsgbot>	 !log jbond@deploy1002 Finished deploy [netbox/deploy@6b69f2c]: deploy v2.10.4-wmf4 to netbox-next (duration: 03m 12s)
[10:27:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:27:49] <wikibugs>	 (03PS1) 10Volans: Fix typo: distribute not present in wheels [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/700496
[10:28:51] <wikibugs>	 (03CR) 10Dzahn: add chart for miscweb (039 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/698895 (https://phabricator.wikimedia.org/T281538) (owner: 10Dzahn)
[10:28:53] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] Fix typo: distribute not present in wheels [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/700496 (owner: 10Volans)
[10:29:15] <wikibugs>	 (03CR) 10Volans: [V: 03+2 C: 03+2] Fix typo: distribute not present in wheels [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/700496 (owner: 10Volans)
[10:29:34] <logmsgbot>	 !log jbond@deploy1002 Started deploy [netbox/deploy@977f7b6]: deploy v2.10.4-wmf4 to netbox-next
[10:29:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:30:05] <jouncebot>	 jan_drewniak: It is that lovely time of the day again! You are hereby commanded to deploy Wikimedia Portals Update. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210621T1030).
[10:30:26] <wikibugs>	 (03CR) 10Muehlenhoff: "Looks good to me, one nit inline." (032 comments) [software/cumin] - 10https://gerrit.wikimedia.org/r/699192 (https://phabricator.wikimedia.org/T244840) (owner: 10Muehlenhoff)
[10:30:31] <logmsgbot>	 !log jbond@deploy1002 Finished deploy [netbox/deploy@977f7b6]: deploy v2.10.4-wmf4 to netbox-next (duration: 00m 56s)
[10:31:10] <wikibugs>	 (03PS2) 10Dzahn: add chart for miscweb [deployment-charts] - 10https://gerrit.wikimedia.org/r/698895 (https://phabricator.wikimedia.org/T281538)
[10:31:38] <wikibugs>	 (03PS2) 10Muehlenhoff: archiva: Switch to nginx-light [puppet] - 10https://gerrit.wikimedia.org/r/699379 (https://phabricator.wikimedia.org/T164456)
[10:31:46] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] add chart for miscweb [deployment-charts] - 10https://gerrit.wikimedia.org/r/698895 (https://phabricator.wikimedia.org/T281538) (owner: 10Dzahn)
[10:32:46] <logmsgbot>	 !log jbond@deploy1002 Started deploy [netbox/deploy@977f7b6]: deploy v2.10.4-wmf4
[10:33:55] <wikibugs>	 (03PS1) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700501 (https://phabricator.wikimedia.org/T128546)
[10:34:52] <wikibugs>	 (03CR) 10Jdrewniak: [C: 03+2] Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700501 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak)
[10:34:59] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+2] archiva: Switch to nginx-light [puppet] - 10https://gerrit.wikimedia.org/r/699379 (https://phabricator.wikimedia.org/T164456) (owner: 10Muehlenhoff)
[10:35:09] <logmsgbot>	 !log jbond@deploy1002 Finished deploy [netbox/deploy@977f7b6]: deploy v2.10.4-wmf4  (duration: 02m 22s)
[10:36:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:36:53] <wikibugs>	 (03Merged) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700501 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak)
[10:37:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:37:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:38:41] <icinga-wm>	 PROBLEM - Check systemd state on netbox2001 is CRITICAL: CRITICAL - degraded: The following units failed: netbox_dump_run.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[10:38:58] <wikibugs>	 (03CR) 10Hnowlan: [C: 03+2] osm: create missing imposm directories, add mirror support to import [puppet] - 10https://gerrit.wikimedia.org/r/699044 (https://phabricator.wikimedia.org/T269582) (owner: 10Hnowlan)
[10:40:16] <logmsgbot>	 !log jdrewniak@deploy1002 Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:692281| Bumping portals to master (T128546)]] (duration: 00m 56s)
[10:40:32] <logmsgbot>	 !log volans@deploy1002 Started deploy [netbox/deploy@977f7b6]: Force re-creation of the virtualenv
[10:40:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:40:50] <stashbot>	 T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546
[10:40:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:41:13] <logmsgbot>	 !log jdrewniak@deploy1002 Synchronized portals: Wikimedia Portals Update: [[gerrit:692281| Bumping portals to master (T128546)]] (duration: 00m 56s)
[10:41:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:41:22] <logmsgbot>	 !log volans@deploy1002 Finished deploy [netbox/deploy@977f7b6]: Force re-creation of the virtualenv (duration: 00m 50s)
[10:41:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:43:21] <logmsgbot>	 !log volans@deploy1002 Started deploy [netbox/deploy@977f7b6]: Force re-creation of the virtualenv
[10:43:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:44:04] <wikibugs>	 (03CR) 10Dzahn: "Thank you, saw the mails about puppet run but you fixed it again already :)" [puppet] - 10https://gerrit.wikimedia.org/r/700331 (owner: 10Paladox)
[10:44:05] * volans doing one host at at time, hence the multiple deployes ^^^
[10:44:15] <logmsgbot>	 !log volans@deploy1002 Finished deploy [netbox/deploy@977f7b6]: Force re-creation of the virtualenv (duration: 00m 54s)
[10:44:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:44:43] <logmsgbot>	 !log volans@deploy1002 Started deploy [netbox/deploy@977f7b6]: Force re-creation of the virtualenv
[10:44:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:45:09] <moritzm>	 !log installing nettle security updates on buster
[10:45:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:45:37] <logmsgbot>	 !log volans@deploy1002 Finished deploy [netbox/deploy@977f7b6]: Force re-creation of the virtualenv (duration: 00m 54s)
[10:45:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:45:49] <wikibugs>	 (03PS1) 10Giuseppe Lavagetto: mediawiki: default to .name if .server_name is not provided [deployment-charts] - 10https://gerrit.wikimedia.org/r/700514
[10:47:50] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] O:netbox: switch netbox production serveres to CAS sso authentication [puppet] - 10https://gerrit.wikimedia.org/r/699712 (owner: 10Jbond)
[10:47:57] <icinga-wm>	 RECOVERY - Check systemd state on netbox1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[10:51:55] <icinga-wm>	 PROBLEM - SSH on mw1303.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[10:53:30] <wikibugs>	 (03PS1) 10Muehlenhoff: Add library hint for nettle [puppet] - 10https://gerrit.wikimedia.org/r/700516
[10:53:31] <icinga-wm>	 PROBLEM - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The following units failed: netbox_dump_run.service,netbox_ganeti_codfw_sync.service,netbox_ganeti_eqiad_sync.service,netbox_ganeti_esams_sync.service,netbox_ganeti_ulsfo_sync.service,netbox_report_network_run.service,rq-netbox.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[10:55:30] <moritzm>	 !log restarting FPM on mw canaries to pick up nettle security updates
[10:55:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:56:29] <icinga-wm>	 PROBLEM - Check unit status of netbox_ganeti_ulsfo_sync on netbox1001 is CRITICAL: CRITICAL: Status of the systemd unit netbox_ganeti_ulsfo_sync https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[10:56:29] <icinga-wm>	 PROBLEM - Check unit status of netbox_ganeti_codfw_sync on netbox1001 is CRITICAL: CRITICAL: Status of the systemd unit netbox_ganeti_codfw_sync https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[10:56:55] <wikibugs>	 (03PS2) 10Giuseppe Lavagetto: mediawiki: default to .name if .server_name is not provided [deployment-charts] - 10https://gerrit.wikimedia.org/r/700514
[10:57:35] <urbanecm>	 jouncebot: next
[10:57:35] <jouncebot>	 In 0 hour(s) and 2 minute(s): European mid-day backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210621T1100)
[10:57:52] <wikibugs>	 (03PS5) 10Volans: cumin: add support for Kerberos auth [software/cumin] - 10https://gerrit.wikimedia.org/r/699192 (https://phabricator.wikimedia.org/T244840) (owner: 10Muehlenhoff)
[10:57:56] <Lucas_WMDE>	 I have some backports coming up but they’re not merged on master quite yet
[10:58:01] <Lucas_WMDE>	 I’ll add them to the calendar once ready
[10:58:12] <wikibugs>	 (03CR) 10Volans: "Addressed/replied comments" (035 comments) [software/cumin] - 10https://gerrit.wikimedia.org/r/699192 (https://phabricator.wikimedia.org/T244840) (owner: 10Muehlenhoff)
[10:59:59] <icinga-wm>	 PROBLEM - Check unit status of netbox_ganeti_esams_sync on netbox1001 is CRITICAL: CRITICAL: Status of the systemd unit netbox_ganeti_esams_sync https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[11:00:05] <jouncebot>	 Amir1, Lucas_WMDE, awight, and Urbanecm: Dear deployers, time to do the European mid-day backport window deploy. Dont look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210621T1100).
[11:00:05] <jouncebot>	 majavah: A patch you scheduled for European mid-day backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[11:00:10] <Lucas_WMDE>	 o/
[11:00:12] <majavah>	 here
[11:00:19] <Lucas_WMDE>	 I can deploy
[11:00:24] <Lucas_WMDE>	 since I’ll also backport later
[11:00:31] <Lucas_WMDE>	 or are you already a deployer majavah? I don’t remember ^^
[11:00:34] <majavah>	 nope
[11:00:36] <Lucas_WMDE>	 ok
[11:00:57] <icinga-wm>	 PROBLEM - Check unit status of netbox_ganeti_eqsin_sync on netbox1001 is CRITICAL: CRITICAL: Status of the systemd unit netbox_ganeti_eqsin_sync https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[11:01:17] * Lucas_WMDE looks at the change
[11:01:45] <logmsgbot>	 !log jbond@deploy1002 Started deploy [netbox/deploy@977f7b6]: deploy v2.10.4-wmf4
[11:01:47] <icinga-wm>	 PROBLEM - Check unit status of netbox_ganeti_eqiad_sync on netbox1001 is CRITICAL: CRITICAL: Status of the systemd unit netbox_ganeti_eqiad_sync https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[11:01:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:02:40] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [C: 03+2] mediawiki: default to .name if .server_name is not provided [deployment-charts] - 10https://gerrit.wikimedia.org/r/700514 (owner: 10Giuseppe Lavagetto)
[11:03:40] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): [C: 03+1] "Hm, codesearch finds some references to 185.15.56.0/24 as cloud-eqiad (e.g. https://gerrit.wikimedia.org/g/operations/puppet/+/cbef5526d72" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700160 (owner: 10Majavah)
[11:04:06] <Lucas_WMDE>	 majavah: how urgent is that change? I’m not sure I feel confident to deploy it
[11:04:11] <majavah>	 Lucas_WMDE: essentially it treats the public cloud ranges (185.15.56.0/24) like how private cloud ranges are treated, earlier this year the network topology was changed a bit and now instances with public ips show up with those on the mediawiki side instead of their private addresses
[11:04:23] <Lucas_WMDE>	 oh right, I remember that announcement
[11:04:25] <majavah>	 not that urgent, I'm happy to reschedule
[11:04:32] * Lucas_WMDE looks for the email
[11:04:35] <majavah>	 that announcement was for a different and now-cancelled change
[11:04:38] <logmsgbot>	 !log jbond@deploy1002 Finished deploy [netbox/deploy@977f7b6]: deploy v2.10.4-wmf4  (duration: 02m 53s)
[11:04:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:04:44] <urbanecm>	 and i'm happy to deploy/test it if Lucas_WMDE wants me to
[11:04:56] <majavah>	 that was a side effect we didn't really think of when making the topology change
[11:05:14] <wikibugs>	 (03Merged) 10jenkins-bot: mediawiki: default to .name if .server_name is not provided [deployment-charts] - 10https://gerrit.wikimedia.org/r/700514 (owner: 10Giuseppe Lavagetto)
[11:05:22] <Lucas_WMDE>	 well, the announcement mentions the IP 185.15.56.1 at least, which checks out so far
[11:07:13] <wikibugs>	 (03PS2) 10Lucas Werkmeister (WMDE): Add WMCS public addresses to $wgSoftBlockRanges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700160 (owner: 10Majavah)
[11:07:21] <icinga-wm>	 RECOVERY - Check unit status of netbox_ganeti_ulsfo_sync on netbox1001 is OK: OK: Status of the systemd unit netbox_ganeti_ulsfo_sync https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[11:07:21] <icinga-wm>	 RECOVERY - Check unit status of netbox_ganeti_codfw_sync on netbox1001 is OK: OK: Status of the systemd unit netbox_ganeti_codfw_sync https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[11:07:30] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): [C: 03+2] "Let’s try it out." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700160 (owner: 10Majavah)
[11:07:55] <Lucas_WMDE>	 hm, can this change be tested on mwdebug?
[11:08:00] <majavah>	 yes
[11:08:02] <Lucas_WMDE>	 ok
[11:08:25] <wikibugs>	 (03Merged) 10jenkins-bot: Add WMCS public addresses to $wgSoftBlockRanges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700160 (owner: 10Majavah)
[11:08:33] <Lucas_WMDE>	 I guess some kind of curl /w/api.php -d action=edit etc. from a bastion shell would be one option
[11:08:55] <majavah>	 `curl -X "X-Wikimedia-Debug: backend=mwdebug100X.eqiad.wmnet" "https://en.wikipedia.org/w/api.php?action=query&format=json&meta=userinfo&uiprop=blockinfo"` from a toolforge bastion, it should show a block with the change but not without it
[11:08:57] <Lucas_WMDE>	 alright, change is pulled to mwdebug1001
[11:09:51] <majavah>	 appears to be working as intended
[11:10:23] <Lucas_WMDE>	 trying it as well…
[11:10:29] <wikibugs>	 (03PS1) 10Giuseppe Lavagetto: mediawiki: bump chart number [deployment-charts] - 10https://gerrit.wikimedia.org/r/700518
[11:10:36] <Lucas_WMDE>	 yup, indeed
[11:10:39] <Lucas_WMDE>	 syncing
[11:10:43] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [V: 03+2] mediawiki: bump chart number [deployment-charts] - 10https://gerrit.wikimedia.org/r/700518 (owner: 10Giuseppe Lavagetto)
[11:10:49] <icinga-wm>	 RECOVERY - Check unit status of netbox_ganeti_esams_sync on netbox1001 is OK: OK: Status of the systemd unit netbox_ganeti_esams_sync https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[11:11:16] <Lucas_WMDE>	 (with -H instead of -X in case anyone wants to bookmark that command or something ^^)
[11:11:36] <majavah>	 yeah, oops :P
[11:11:46] <majavah>	 -X is method, -H is header
[11:11:49] <icinga-wm>	 RECOVERY - Check unit status of netbox_ganeti_eqsin_sync on netbox1001 is OK: OK: Status of the systemd unit netbox_ganeti_eqsin_sync https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[11:11:57] <logmsgbot>	 !log lucaswerkmeister-wmde@deploy1002 Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:700160|Add WMCS public addresses to $wgSoftBlockRanges]] (duration: 00m 56s)
[11:12:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:12:06] <Lucas_WMDE>	 tada
[11:12:11] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [V: 03+2 C: 03+2] mediawiki: bump chart number [deployment-charts] - 10https://gerrit.wikimedia.org/r/700518 (owner: 10Giuseppe Lavagetto)
[11:12:12] <majavah>	 thanks!
[11:12:26] <Lucas_WMDE>	 my backport isn’t ready yet, does anyone else have something to deploy in the meantime?
[11:12:39] <urbanecm>	 probably, lemme see
[11:12:39] <icinga-wm>	 RECOVERY - Check unit status of netbox_ganeti_eqiad_sync on netbox1001 is OK: OK: Status of the systemd unit netbox_ganeti_eqiad_sync https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[11:13:11] <urbanecm>	 https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/700430/, https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/700428/
[11:13:19] <urbanecm>	 Lucas_WMDE: want to push them out, or should i?
[11:13:35] <Lucas_WMDE>	 either way is fine with me
[11:13:43] <urbanecm>	 i'll do it then
[11:13:46] <Lucas_WMDE>	 I assume one needs a cache purge and the other a namespace dupes maintenance?
[11:13:48] <Lucas_WMDE>	 ok
[11:13:50] <wikibugs>	 (03PS2) 10Urbanecm: ptwikinews: Remove NS ID 102,103 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700428 (https://phabricator.wikimedia.org/T285163)
[11:13:54] <wikibugs>	 (03CR) 10Urbanecm: [C: 03+2] ptwikinews: Remove NS ID 102,103 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700428 (https://phabricator.wikimedia.org/T285163) (owner: 10Urbanecm)
[11:14:31] <urbanecm>	 Lucas_WMDE: in theory the NS change should be fine w/o namespaceDupes, as you can't create conflicts when removing namespaces
[11:14:59] <wikibugs>	 (03Merged) 10jenkins-bot: ptwikinews: Remove NS ID 102,103 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700428 (https://phabricator.wikimedia.org/T285163) (owner: 10Urbanecm)
[11:15:05] <wikibugs>	 (03PS3) 10KartikMistry: cxserver: Remove Matxin MT support [deployment-charts] - 10https://gerrit.wikimedia.org/r/700466 (https://phabricator.wikimedia.org/T285199)
[11:16:05] <Lucas_WMDE>	 ah ok
[11:16:14] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] "thanks lgtm" (032 comments) [software/cumin] - 10https://gerrit.wikimedia.org/r/699192 (https://phabricator.wikimedia.org/T244840) (owner: 10Muehlenhoff)
[11:16:21] <wikibugs>	 (03PS1) 10Dzahn: miscweb: add a define for the httpd prometheus exporter and use it [deployment-charts] - 10https://gerrit.wikimedia.org/r/700522 (https://phabricator.wikimedia.org/T281538)
[11:16:43] <logmsgbot>	 !log urbanecm@deploy1002 Synchronized wmf-config/InitialiseSettings.php: 464cc0bb0d9ff6c47e2a90cba5232e9506ff5dec: ptwikinews: Remove NS ID 102,103 (T285163) (duration: 00m 56s)
[11:16:43] <urbanecm>	 (you can still of course still fix conflicts introduced before, for instance because someone forgot to run it)
[11:16:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:16:48] <stashbot>	 T285163: Delete 'Efeméride' namespace in ptwikinews - https://phabricator.wikimedia.org/T285163
[11:16:49] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] miscweb: add a define for the httpd prometheus exporter and use it [deployment-charts] - 10https://gerrit.wikimedia.org/r/700522 (https://phabricator.wikimedia.org/T281538) (owner: 10Dzahn)
[11:17:04] <wikibugs>	 (03PS2) 10Urbanecm: Change vi.wikisource logo to the same logo being used at en.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700430 (https://phabricator.wikimedia.org/T284612)
[11:17:07] <logmsgbot>	 !log oblivian@deploy1002 helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
[11:17:08] <wikibugs>	 (03CR) 10Urbanecm: [C: 03+2] Change vi.wikisource logo to the same logo being used at en.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700430 (https://phabricator.wikimedia.org/T284612) (owner: 10Urbanecm)
[11:17:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:18:03] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] doc: use add_css_file() instead of add_stylesheet() [software/cumin] - 10https://gerrit.wikimedia.org/r/700479 (owner: 10Volans)
[11:18:30] <wikibugs>	 (03PS4) 10Urbanecm: Disable indexing user (sub)pages and draft-related pages on hrwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699295 (https://phabricator.wikimedia.org/T284384) (owner: 10Acamicamacaraca)
[11:18:34] <wikibugs>	 (03CR) 10Urbanecm: [C: 03+2] Disable indexing user (sub)pages and draft-related pages on hrwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699295 (https://phabricator.wikimedia.org/T284384) (owner: 10Acamicamacaraca)
[11:18:51] <wikibugs>	 (03Merged) 10jenkins-bot: Change vi.wikisource logo to the same logo being used at en.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700430 (https://phabricator.wikimedia.org/T284612) (owner: 10Urbanecm)
[11:18:58] <wikibugs>	 (03CR) 10Volans: [V: 03+2 C: 03+2] doc: use add_css_file() instead of add_stylesheet() [software/cumin] - 10https://gerrit.wikimedia.org/r/700479 (owner: 10Volans)
[11:19:13] <wikibugs>	 (03CR) 10Volans: [C: 03+2] setup.py: add types dependencies for mypy [software/cumin] - 10https://gerrit.wikimedia.org/r/700480 (owner: 10Volans)
[11:19:59] <wikibugs>	 (03Merged) 10jenkins-bot: Disable indexing user (sub)pages and draft-related pages on hrwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699295 (https://phabricator.wikimedia.org/T284384) (owner: 10Acamicamacaraca)
[11:20:24] <logmsgbot>	 !log urbanecm@deploy1002 Synchronized wmf-config/logos.php: 1b97376dea470122f9da9ea8dddde7e8fe94231f: Change vi.wikisource logo to the same logo being used at en.wikisource (T284612) (duration: 00m 57s)
[11:20:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:20:29] <stashbot>	 T284612: Change vi.wikisource logo to the same logo being used at en.wikisource - https://phabricator.wikimedia.org/T284612
[11:21:31] <logmsgbot>	 !log urbanecm@deploy1002 Synchronized logos/config.yaml: 1b97376dea470122f9da9ea8dddde7e8fe94231f: Change vi.wikisource logo to the same logo being used at en.wikisource (T284612) (duration: 00m 56s)
[11:21:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:22:35] <wikibugs>	 (03PS3) 10Jbond: rspec: update to use rspec-mock instead of mocha [puppet] - 10https://gerrit.wikimedia.org/r/649684
[11:23:01] <logmsgbot>	 !log urbanecm@deploy1002 Synchronized wmf-config/InitialiseSettings.php: 0bf35e0d3934a2c44e95fc4d7bc30a2a6c74c59a: Disable indexing user (sub)pages and draft-related pages on hrwiki (T284384) (duration: 00m 56s)
[11:23:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:23:05] <stashbot>	 T284384: Disable indexing user (sub)pages and drafts on Croatian Wikipedia - https://phabricator.wikimedia.org/T284384
[11:24:01] * urbanecm is done
[11:24:03] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] rspec: update to use rspec-mock instead of mocha [puppet] - 10https://gerrit.wikimedia.org/r/649684 (owner: 10Jbond)
[11:24:07] <urbanecm>	 Lucas_WMDE: fyi ^
[11:25:55] <Lucas_WMDE>	 ok thanks!
[11:26:44] <Lucas_WMDE>	 (still waiting for CI)
[11:32:19] <icinga-wm>	 RECOVERY - Check systemd state on netbox1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[11:32:27] <icinga-wm>	 RECOVERY - Check systemd state on netbox2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[11:33:37] <wikibugs>	 (03Merged) 10jenkins-bot: setup.py: add types dependencies for mypy [software/cumin] - 10https://gerrit.wikimedia.org/r/700480 (owner: 10Volans)
[11:36:33] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+1] nova: collect VPS data from the hypervisor [puppet] - 10https://gerrit.wikimedia.org/r/700393 (owner: 10David Caro)
[11:38:12] <wikibugs>	 (03PS1) 10Lucas Werkmeister (WMDE): Rewrite SerializationModifier to be more efficient [extensions/Wikibase] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700528
[11:39:11] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): [C: 03+2] Rewrite SerializationModifier to be more efficient [extensions/Wikibase] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700528 (owner: 10Lucas Werkmeister (WMDE))
[11:39:18] <Lucas_WMDE>	 alright, let’s hope CI is happy with it
[11:39:36] <Lucas_WMDE>	 (might take a bit longer than 20 minutes but the deployment calendar is pretty free after this)
[11:40:15] <wikibugs>	 (03PS1) 10Jbond: O:netbox: Always install the ldap and cas_configueration files [puppet] - 10https://gerrit.wikimedia.org/r/700529
[11:41:10] <wikibugs>	 (03PS2) 10Jbond: O:netbox: Always install the ldap and cas_configueration files [puppet] - 10https://gerrit.wikimedia.org/r/700529
[11:48:11] <wikibugs>	 (03CR) 10Hnowlan: postgres::slave: remove problematic auto-replicate (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/700071 (owner: 10Hnowlan)
[11:49:16] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+2] Add library hint for nettle [puppet] - 10https://gerrit.wikimedia.org/r/700516 (owner: 10Muehlenhoff)
[11:51:44] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1135 T283499', diff saved to https://phabricator.wikimedia.org/P16662 and previous config saved to /var/cache/conftool/dbconfig/20210621-115143-marostegui.json
[11:51:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:51:49] <stashbot>	 T283499: Schema change for renaming page_timestamp index on revision table to rev_page_timestamp - https://phabricator.wikimedia.org/T283499
[11:52:08] <wikibugs>	 (03PS3) 10Dzahn: add chart for miscweb [deployment-charts] - 10https://gerrit.wikimedia.org/r/698895 (https://phabricator.wikimedia.org/T281538)
[11:53:56] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1135 (re)pooling @ 25%: Repool db1135 after schema change', diff saved to https://phabricator.wikimedia.org/P16663 and previous config saved to /var/cache/conftool/dbconfig/20210621-115355-root.json
[11:53:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:54:42] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1099:3311 T283499', diff saved to https://phabricator.wikimedia.org/P16664 and previous config saved to /var/cache/conftool/dbconfig/20210621-115441-marostegui.json
[11:54:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:55:20] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 25%: Repool db1099:3311 after schema change', diff saved to https://phabricator.wikimedia.org/P16665 and previous config saved to /var/cache/conftool/dbconfig/20210621-115519-root.json
[11:55:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:56:08] <wikibugs>	 (03CR) 10JMeybohm: [C: 04-1] Add the custom_deploy.d directory with basic Istio config (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/697938 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey)
[11:56:39] <wikibugs>	 (03CR) 10JMeybohm: [C: 03+1] Add support for istioctl 1.9.5 [debs/istioctl] - 10https://gerrit.wikimedia.org/r/700397 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey)
[11:57:34] <logmsgbot>	 !log jmm@cumin2002 START - Cookbook sre.hosts.reboot-single for host maps2008.codfw.wmnet
[11:57:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:58:05] <wikibugs>	 (03CR) 10Dzahn: "https://phabricator.wikimedia.org/T151680" [puppet] - 10https://gerrit.wikimedia.org/r/692370 (owner: 10Zabe)
[11:59:53] <wikibugs>	 (03CR) 10Jelto: [C: 03+2] "looks good to me now. In my opinion daily full backups of the GitLab backup .tar file is the best solution. We also have to reconfigure Gi" [puppet] - 10https://gerrit.wikimedia.org/r/697850 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[12:03:13] <wikibugs>	 (03PS1) 10Jbond: C:netbox: code clean up [puppet] - 10https://gerrit.wikimedia.org/r/700534
[12:06:04] <kart_>	 Need help in https://phabricator.wikimedia.org/T285219 from ServiceOps.
[12:08:12] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Rewrite SerializationModifier to be more efficient [extensions/Wikibase] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700528 (owner: 10Lucas Werkmeister (WMDE))
[12:08:38] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): [C: 03+2] "random browser test failure :(" [extensions/Wikibase] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700528 (owner: 10Lucas Werkmeister (WMDE))
[12:09:00] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1135 (re)pooling @ 50%: Repool db1135 after schema change', diff saved to https://phabricator.wikimedia.org/P16667 and previous config saved to /var/cache/conftool/dbconfig/20210621-120859-root.json
[12:09:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:09:14] <wikibugs>	 (03PS4) 10Dzahn: add chart for miscweb [deployment-charts] - 10https://gerrit.wikimedia.org/r/698895 (https://phabricator.wikimedia.org/T281538)
[12:09:16] <wikibugs>	 (03PS2) 10Dzahn: miscweb: add a define for the httpd prometheus exporter and use it [deployment-charts] - 10https://gerrit.wikimedia.org/r/700522 (https://phabricator.wikimedia.org/T281538)
[12:09:39] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] add chart for miscweb [deployment-charts] - 10https://gerrit.wikimedia.org/r/698895 (https://phabricator.wikimedia.org/T281538) (owner: 10Dzahn)
[12:09:56] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] miscweb: add a define for the httpd prometheus exporter and use it [deployment-charts] - 10https://gerrit.wikimedia.org/r/700522 (https://phabricator.wikimedia.org/T281538) (owner: 10Dzahn)
[12:10:01] <godog>	 !log bump space for k8s and ops prometheus on prometheus1004 (prometheus1003 has been expanded previously but not logged)
[12:10:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:10:04] <wikibugs>	 (03CR) 10Jbond: postgres::slave: remove problematic auto-replicate (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/700071 (owner: 10Hnowlan)
[12:10:23] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 50%: Repool db1099:3311 after schema change', diff saved to https://phabricator.wikimedia.org/P16668 and previous config saved to /var/cache/conftool/dbconfig/20210621-121023-root.json
[12:10:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:12:57] <logmsgbot>	 !log jmm@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2008.codfw.wmnet
[12:12:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:13:31] <wikibugs>	 (03CR) 10Dzahn: [C: 03+1] "lgtm, afaict. Brennen, do you agree and would run the ansible commands for this?" [gitlab-ansible] - 10https://gerrit.wikimedia.org/r/700084 (https://phabricator.wikimedia.org/T274463) (owner: 10Jelto)
[12:14:40] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "LGTM, if compiler is happy" [puppet] - 10https://gerrit.wikimedia.org/r/700529 (owner: 10Jbond)
[12:16:07] <wikibugs>	 (03PS2) 10Jbond: C:netbox: code clean up [puppet] - 10https://gerrit.wikimedia.org/r/700534
[12:16:48] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "LGTM if compiler is happy" [puppet] - 10https://gerrit.wikimedia.org/r/700534 (owner: 10Jbond)
[12:19:47] <logmsgbot>	 !log jmm@cumin2002 START - Cookbook sre.hosts.reboot-single for host maps1010.eqiad.wmnet
[12:19:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:20:03] <wikibugs>	 (03CR) 10Jcrespo: [C: 03+1] "> Patch Set 5:" [puppet] - 10https://gerrit.wikimedia.org/r/692370 (owner: 10Zabe)
[12:22:36] <wikibugs>	 (03PS3) 10Jbond: C:netbox: code clean up [puppet] - 10https://gerrit.wikimedia.org/r/700534
[12:24:03] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1135 (re)pooling @ 75%: Repool db1135 after schema change', diff saved to https://phabricator.wikimedia.org/P16669 and previous config saved to /var/cache/conftool/dbconfig/20210621-122403-root.json
[12:24:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:25:27] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 75%: Repool db1099:3311 after schema change', diff saved to https://phabricator.wikimedia.org/P16670 and previous config saved to /var/cache/conftool/dbconfig/20210621-122526-root.json
[12:25:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:25:49] <logmsgbot>	 !log jmm@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1010.eqiad.wmnet
[12:25:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:30:26] <wikibugs>	 (03Merged) 10jenkins-bot: Rewrite SerializationModifier to be more efficient [extensions/Wikibase] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700528 (owner: 10Lucas Werkmeister (WMDE))
[12:31:02] <Lucas_WMDE>	 alright, my backport finally merged
[12:31:14] <Lucas_WMDE>	 can I still deploy it or is anyone else deploying something right now? ^^
[12:31:47] <wikibugs>	 (03PS1) 10Dzahn: gitlab::backup: create backup paths with wmflib::dir::mkdir_p [puppet] - 10https://gerrit.wikimedia.org/r/700595 (https://phabricator.wikimedia.org/T274463)
[12:32:14] <wikibugs>	 (03PS2) 10Dzahn: gitlab::backup: create backup paths with wmflib::dir::mkdir_p [puppet] - 10https://gerrit.wikimedia.org/r/700595 (https://phabricator.wikimedia.org/T274463)
[12:32:16] <Lucas_WMDE>	 alright, I’ll go ahead
[12:32:16] <Amir1>	 Lucas_WMDE: go ahead, the window is empty
[12:32:19] <Lucas_WMDE>	 ok thanks
[12:32:50] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] gitlab::backup: create backup paths with wmflib::dir::mkdir_p [puppet] - 10https://gerrit.wikimedia.org/r/700595 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[12:32:58] <Lucas_WMDE>	 pulled to mwdebug1001, testing
[12:33:18] <Lucas_WMDE>	 seems to work fine, I’ll sync it
[12:33:26] <Lucas_WMDE>	 there may be a brief spike in API errors on Wikidata and Commons
[12:33:48] <wikibugs>	 (03PS12) 10Filippo Giunchedi: Sketch of Performance team alerts [alerts] - 10https://gerrit.wikimedia.org/r/670230 (https://phabricator.wikimedia.org/T281358)
[12:34:53] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C: 03+2] "> Patch Set 11: Code-Review+1" [alerts] - 10https://gerrit.wikimedia.org/r/670230 (https://phabricator.wikimedia.org/T281358) (owner: 10Filippo Giunchedi)
[12:35:09] <logmsgbot>	 !log lucaswerkmeister-wmde@deploy1002 Synchronized php-1.37.0-wmf.9/extensions/Wikibase: Backport: [[gerrit:700528|Rewrite SerializationModifier to be more efficient]] (duration: 01m 02s)
[12:35:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:35:39] <Lucas_WMDE>	 yup, spike shows up in logspam-watch
[12:35:41] <Lucas_WMDE>	 checking
[12:36:08] <Lucas_WMDE>	 (the SerializationModifier stuff)
[12:36:11] <wikibugs>	 (03PS1) 10Dzahn: wmflib: replace extlib with wmflib in code examples [puppet] - 10https://gerrit.wikimedia.org/r/700598
[12:38:22] <Lucas_WMDE>	 I *think* it’s recovered on its own
[12:38:28] <Lucas_WMDE>	 though I confess I didn’t expect the spike to be that large
[12:38:31] <Lucas_WMDE>	 sorry
[12:38:49] <Amir1>	 how many it was?
[12:39:07] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1135 (re)pooling @ 100%: Repool db1135 after schema change', diff saved to https://phabricator.wikimedia.org/P16671 and previous config saved to /var/cache/conftool/dbconfig/20210621-123906-root.json
[12:39:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:39:13] <Lucas_WMDE>	 summing the first logspam-watch column… some 2600 or so
[12:40:01] <Lucas_WMDE>	 logstash says 1636+1228
[12:40:15] <Lucas_WMDE>	 (so 2864)
[12:40:30] <logmsgbot>	 !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 100%: Repool db1099:3311 after schema change', diff saved to https://phabricator.wikimedia.org/P16672 and previous config saved to /var/cache/conftool/dbconfig/20210621-124030-root.json
[12:40:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:42:45] <Amir1>	 Lucas_WMDE: that's okay if you ask me, it's in our error budget
[12:43:16] <Lucas_WMDE>	 (and just confirming again, on logstash it’s also clearly recovered on its own)
[12:43:38] <wikibugs>	 (03PS3) 10Dzahn: gitlab::backup: create backup paths with wmflib::dir::mkdir_p [puppet] - 10https://gerrit.wikimedia.org/r/700595 (https://phabricator.wikimedia.org/T274463)
[12:47:28] <wikibugs>	 (03CR) 10Elukey: Add the custom_deploy.d directory with basic Istio config (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/697938 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey)
[12:47:49] <wikibugs>	 (03PS9) 10Elukey: Add the custom_deploy.d directory with basic Istio config [deployment-charts] - 10https://gerrit.wikimedia.org/r/697938 (https://phabricator.wikimedia.org/T278192)
[12:47:53] <wikibugs>	 (03PS9) 10Elukey: Add support for knative serving [deployment-charts] - 10https://gerrit.wikimedia.org/r/699380 (https://phabricator.wikimedia.org/T278194)
[12:48:03] <wikibugs>	 (03PS2) 10Elukey: WIP - Add kubeflow's kfserving chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/700470 (https://phabricator.wikimedia.org/T272919)
[12:49:07] <wikibugs>	 (03CR) 10Elukey: "I am also trying to add some more details to the initial istio settings, namely:" [deployment-charts] - 10https://gerrit.wikimedia.org/r/697938 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey)
[12:49:27] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] WIP - Add kubeflow's kfserving chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/700470 (https://phabricator.wikimedia.org/T272919) (owner: 10Elukey)
[12:53:29] <icinga-wm>	 RECOVERY - SSH on mw1303.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[12:53:32] <wikibugs>	 (03CR) 10Jbond: [V: 03+1] "PCC SUCCESS (DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29949/console" [puppet] - 10https://gerrit.wikimedia.org/r/700534 (owner: 10Jbond)
[12:53:49] <wikibugs>	 (03PS1) 10Dzahn: gitlab: ensure backup dirs exist, add parameter for config backup [puppet] - 10https://gerrit.wikimedia.org/r/700601 (https://phabricator.wikimedia.org/T274463)
[12:53:57] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] O:netbox: Always install the ldap and cas_configueration files [puppet] - 10https://gerrit.wikimedia.org/r/700529 (owner: 10Jbond)
[12:54:02] <wikibugs>	 (03CR) 10Jbond: [V: 03+1 C: 03+2] C:netbox: code clean up [puppet] - 10https://gerrit.wikimedia.org/r/700534 (owner: 10Jbond)
[12:54:25] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] gitlab: ensure backup dirs exist, add parameter for config backup [puppet] - 10https://gerrit.wikimedia.org/r/700601 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[12:55:21] <godog>	 !log move librenms alerts with "max alerts" == -1 to "interval" being 15m - T285205
[12:55:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:55:26] <stashbot>	 T285205: Make sure LibreNMS -> AM alerts don't flap due to lack of notifications - https://phabricator.wikimedia.org/T285205
[12:56:09] <wikibugs>	 (03PS2) 10Dzahn: gitlab: ensure backup dirs exist, add parameter for config backup [puppet] - 10https://gerrit.wikimedia.org/r/700601 (https://phabricator.wikimedia.org/T274463)
[12:56:45] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] gitlab: ensure backup dirs exist, add parameter for config backup [puppet] - 10https://gerrit.wikimedia.org/r/700601 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[12:57:20] <wikibugs>	 (03PS3) 10David Caro: nova: collect VPS data from the hypervisor [puppet] - 10https://gerrit.wikimedia.org/r/700393
[12:57:22] <wikibugs>	 (03PS1) 10Ladsgroup: Replace class_exists with ExtensionRegistry [extensions/MobileFrontend] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700343 (https://phabricator.wikimedia.org/T274275)
[12:57:41] <wikibugs>	 (03CR) 10Ladsgroup: [C: 03+2] Replace class_exists with ExtensionRegistry [extensions/MobileFrontend] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700343 (https://phabricator.wikimedia.org/T274275) (owner: 10Ladsgroup)
[12:58:03] <wikibugs>	 (03CR) 10David Caro: "Added tests, and found and fixed a typo." [puppet] - 10https://gerrit.wikimedia.org/r/700393 (owner: 10David Caro)
[12:59:40] <wikibugs>	 (03PS1) 10Ladsgroup: Avoid loading the whole entity when it only needs description. [extensions/MobileFrontend] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700344 (https://phabricator.wikimedia.org/T269960)
[12:59:52] <wikibugs>	 (03CR) 10Ladsgroup: [C: 03+2] Avoid loading the whole entity when it only needs description. [extensions/MobileFrontend] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700344 (https://phabricator.wikimedia.org/T269960) (owner: 10Ladsgroup)
[13:06:58] <wikibugs>	 (03CR) 10Elukey: [V: 03+2 C: 03+2] Add support for istioctl 1.9.5 [debs/istioctl] - 10https://gerrit.wikimedia.org/r/700397 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey)
[13:07:25] <wikibugs>	 (03CR) 10David Caro: [C: 03+2] nova: collect VPS data from the hypervisor [puppet] - 10https://gerrit.wikimedia.org/r/700393 (owner: 10David Caro)
[13:09:05] <klausman>	 !log starting update of ML team's etcd machines in codfw
[13:09:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:12:05] <logmsgbot>	 !log dcaro@cumin1001 START - Cookbook sre.hosts.downtime for 2:00:00 on 40 hosts with reason: Merged broken patch
[13:12:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:12:20] <logmsgbot>	 !log dcaro@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 40 hosts with reason: Merged broken patch
[13:12:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:12:24] <elukey>	 !log upload istioctl 1.9.5 to {buster,stretch}-wikimedia
[13:12:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:14:39] <logmsgbot>	 !log klausman@cumin2001 START - Cookbook sre.hosts.reboot-single for host ml-etcd2001.codfw.wmnet
[13:14:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:17:01] <logmsgbot>	 !log klausman@cumin2001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2001.codfw.wmnet
[13:17:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:19:13] <logmsgbot>	 !log klausman@cumin2001 START - Cookbook sre.hosts.reboot-single for host ml-etcd2002.codfw.wmnet
[13:19:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:21:31] <logmsgbot>	 !log klausman@cumin2001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2002.codfw.wmnet
[13:21:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:21:44] <logmsgbot>	 !log klausman@cumin2001 START - Cookbook sre.hosts.reboot-single for host ml-etcd2003.codfw.wmnet
[13:21:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:23:31] <wikibugs>	 (03Merged) 10jenkins-bot: Replace class_exists with ExtensionRegistry [extensions/MobileFrontend] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700343 (https://phabricator.wikimedia.org/T274275) (owner: 10Ladsgroup)
[13:23:35] <wikibugs>	 (03Merged) 10jenkins-bot: Avoid loading the whole entity when it only needs description. [extensions/MobileFrontend] (wmf/1.37.0-wmf.9) - 10https://gerrit.wikimedia.org/r/700344 (https://phabricator.wikimedia.org/T269960) (owner: 10Ladsgroup)
[13:24:09] <logmsgbot>	 !log klausman@cumin2001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2003.codfw.wmnet
[13:24:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:25:47] <addshore>	 Amir1: is there a dashboard i can / should follow along on? :P
[13:26:02] <Amir1>	 basically everything
[13:26:07] <Amir1>	 memcached, RED
[13:27:20] <Amir1>	 wow https://performance.wikimedia.org/xhgui/run/view?id=60d093885a3ad27f29801907 and https://performance.wikimedia.org/xhgui/run/view?id=60d092aed1cbb77144102d31
[13:27:26] <wikibugs>	 (03PS3) 10Dzahn: gitlab: ensure backup dirs exist, add parameter for config backup [puppet] - 10https://gerrit.wikimedia.org/r/700601 (https://phabricator.wikimedia.org/T274463)
[13:27:32] <wikibugs>	 (03PS1) 10David Caro: node_cloudvirt_libvirt_stats: use ExecStartPre instead of redirection [puppet] - 10https://gerrit.wikimedia.org/r/700608
[13:27:46] <jinxer-wm>	 (Processor usage over 85%) firing: (4) Processor usage over 85%   - https://alerts.wikimedia.org
[13:28:10] <wikibugs>	 (03CR) 10Elukey: [V: 03+1 C: 03+2] profile::kubernetes::deployment_server: add istioctl package [puppet] - 10https://gerrit.wikimedia.org/r/700203 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey)
[13:28:49] <logmsgbot>	 !log klausman@cumin2001 START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl2002.codfw.wmnet
[13:28:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:28:58] <logmsgbot>	 !log ladsgroup@deploy1002 Synchronized php-1.37.0-wmf.9/extensions/MobileFrontend/includes/ExtMobileFrontend.php: Backport: [[gerrit:700344|Avoid loading the whole entity when it only needs description. (T269960)]] (duration: 00m 58s)
[13:29:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:29:03] <stashbot>	 T269960: Schema properties in client code loads the whole item in every page view - https://phabricator.wikimedia.org/T269960
[13:29:36] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] node_cloudvirt_libvirt_stats: use ExecStartPre instead of redirection [puppet] - 10https://gerrit.wikimedia.org/r/700608 (owner: 10David Caro)
[13:31:07] <wikibugs>	 (03CR) 10Jelto: gitlab: ensure backup dirs exist, add parameter for config backup (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/700601 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[13:32:47] <wikibugs>	 (03PS2) 10David Caro: node_cloudvirt_libvirt_stats: use ExecStartPre instead of redirection [puppet] - 10https://gerrit.wikimedia.org/r/700608
[13:34:28] <wikibugs>	 (03CR) 10David Caro: [C: 04-1] ceph: add latency monitoring stats (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700182 (https://phabricator.wikimedia.org/T281254) (owner: 10David Caro)
[13:35:52] <logmsgbot>	 !log klausman@cumin2001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl2002.codfw.wmnet
[13:35:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:37:25] <wikibugs>	 (03CR) 10David Caro: node_cloudvirt_libvirt_stats: use ExecStartPre instead of redirection (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700608 (owner: 10David Caro)
[13:37:46] <jinxer-wm>	 (Processor usage over 85%) firing: (4) Processor usage over 85%   - https://alerts.wikimedia.org
[13:38:23] <logmsgbot>	 !log klausman@cumin2001 START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl2001.codfw.wmnet
[13:38:26] <wikibugs>	 (03PS1) 10Volans: CHANGELOG: add changelogs for release v0.0.54 [software/spicerack] - 10https://gerrit.wikimedia.org/r/700611
[13:38:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:38:28] <wikibugs>	 (03PS3) 10David Caro: node_cloudvirt_libvirt_stats: use ExecStartPre instead of redirection [puppet] - 10https://gerrit.wikimedia.org/r/700608
[13:38:50] <wikibugs>	 (03CR) 10David Caro: node_cloudvirt_libvirt_stats: use ExecStartPre instead of redirection (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700608 (owner: 10David Caro)
[13:40:32] <logmsgbot>	 !log klausman@cumin2001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl2001.codfw.wmnet
[13:40:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:42:26] <wikibugs>	 (03PS1) 10Kormat: install_server: switch db1123 to buster. [puppet] - 10https://gerrit.wikimedia.org/r/700613 (https://phabricator.wikimedia.org/T283131)
[13:42:28] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] "LGTM (as far as the generic systemd::timer::job bits are concerned)" [puppet] - 10https://gerrit.wikimedia.org/r/700608 (owner: 10David Caro)
[13:43:25] <wikibugs>	 (03CR) 10Kormat: [C: 03+2] install_server: switch db1123 to buster. [puppet] - 10https://gerrit.wikimedia.org/r/700613 (https://phabricator.wikimedia.org/T283131) (owner: 10Kormat)
[13:45:00] <wikibugs>	 (03CR) 10Volans: [C: 03+2] CHANGELOG: add changelogs for release v0.0.54 [software/spicerack] - 10https://gerrit.wikimedia.org/r/700611 (owner: 10Volans)
[13:45:04] <logmsgbot>	 !log klausman@cumin2001 START - Cookbook sre.hosts.reboot-single for host ml-serve2002.codfw.wmnet
[13:45:07] <wikibugs>	 (03CR) 10David Caro: [C: 03+2] node_cloudvirt_libvirt_stats: use ExecStartPre instead of redirection [puppet] - 10https://gerrit.wikimedia.org/r/700608 (owner: 10David Caro)
[13:45:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:48:25] <XioNoX>	 !log reboot scs-ulsfo
[13:48:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:50:41] <icinga-wm>	 PROBLEM - MariaDB Replica IO: s3 on db1171 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl@db1123.eqiad.wmnet:3306 - retry-time: 60 maximum-retries: 86400 message: Cant connect to MySQL server on db1123.eqiad.wmnet (111 Connection refused) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
[13:50:41] <logmsgbot>	 !log jmm@cumin2002 START - Cookbook sre.hosts.reboot-single for host maps1006.eqiad.wmnet
[13:50:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:51:23] <logmsgbot>	 !log klausman@cumin2001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2002.codfw.wmnet
[13:51:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:51:50] <logmsgbot>	 !log klausman@cumin2001 START - Cookbook sre.hosts.reboot-single for host ml-serve2001.codfw.wmnet
[13:51:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:52:16] <wikibugs>	 (03Merged) 10jenkins-bot: CHANGELOG: add changelogs for release v0.0.54 [software/spicerack] - 10https://gerrit.wikimedia.org/r/700611 (owner: 10Volans)
[13:52:35] <icinga-wm>	 ACKNOWLEDGEMENT - MariaDB Replica IO: s3 on db1171 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl@db1123.eqiad.wmnet:3306 - retry-time: 60 maximum-retries: 86400 message: Cant connect to MySQL server on db1123.eqiad.wmnet (111 Connection refused) Kormat db1123 going down for reimage. https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
[13:55:16] <jynus>	 !log stopping replication at db1171:s3 at db1123-bin.004363:906878073
[13:55:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:55:29] <wikibugs>	 (03PS1) 10Volans: Upstream release v0.0.54 [software/spicerack] (debian) - 10https://gerrit.wikimedia.org/r/700616
[13:56:15] <logmsgbot>	 !log otto@cumin1001 START - Cookbook sre.kafka.roll-restart-mirror-maker
[13:56:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:56:28] <logmsgbot>	 !log jmm@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1006.eqiad.wmnet
[13:56:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:56:45] <icinga-wm>	 RECOVERY - MariaDB Replica IO: s3 on db1171 is OK: OK slave_io_state not a slave https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
[13:57:09] <logmsgbot>	 !log klausman@cumin2001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2001.codfw.wmnet
[13:57:11] <jynus>	 ^ kormat I disconnected it here
[13:57:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:58:28] <logmsgbot>	 !log klausman@cumin2001 START - Cookbook sre.hosts.reboot-single for host ml-serve2004.codfw.wmnet
[13:58:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:58:49] <XioNoX>	 !log reboot scs-eqsin - T285229
[13:58:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:58:54] <stashbot>	 T285229: SCS CPU monitoring issue - https://phabricator.wikimedia.org/T285229
[14:00:22] <wikibugs>	 (03CR) 10Jbond: "With the current script all audit files passed on the CLI will be parsed, even if the data required is already found.  i.e. if we find a T" (039 comments) [puppet] - 10https://gerrit.wikimedia.org/r/700389 (https://phabricator.wikimedia.org/T283242) (owner: 10Muehlenhoff)
[14:02:13] <wikibugs>	 (03CR) 10Volans: [C: 03+2] Upstream release v0.0.54 [software/spicerack] (debian) - 10https://gerrit.wikimedia.org/r/700616 (owner: 10Volans)
[14:02:46] <jinxer-wm>	 (Processor usage over 85%) firing: (3) Processor usage over 85%   - https://alerts.wikimedia.org
[14:04:45] <logmsgbot>	 !log klausman@cumin2001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2004.codfw.wmnet
[14:04:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:05:11] <logmsgbot>	 !log klausman@cumin2001 START - Cookbook sre.hosts.reboot-single for host ml-serve2003.codfw.wmnet
[14:05:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:06:24] <logmsgbot>	 !log otto@cumin1001 END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0)
[14:06:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:07:53] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Upstream release v0.0.54 [software/spicerack] (debian) - 10https://gerrit.wikimedia.org/r/700616 (owner: 10Volans)
[14:08:58] <wikibugs>	 (03CR) 10Volans: [V: 03+2 C: 03+2] Upstream release v0.0.54 [software/spicerack] (debian) - 10https://gerrit.wikimedia.org/r/700616 (owner: 10Volans)
[14:11:00] <logmsgbot>	 !log jmm@cumin2002 START - Cookbook sre.hosts.reboot-single for host maps1008.eqiad.wmnet
[14:11:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:11:30] <logmsgbot>	 !log klausman@cumin2001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2003.codfw.wmnet
[14:11:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:11:43] <icinga-wm>	 PROBLEM - rpki grafana alert on alert1001 is CRITICAL: CRITICAL: RPKI ( https://grafana.wikimedia.org/d/UwUa77GZk/rpki ) is alerting: rsync status alert. https://wikitech.wikimedia.org/wiki/RPKI%23Grafana_alerts https://grafana.wikimedia.org/d/UwUa77GZk/
[14:12:41] <volans>	 XioNoX, topranks FYI ^^^
[14:13:16] <XioNoX>	 eh
[14:13:21] <XioNoX>	 we need to remove that alert
[14:13:23] <XioNoX>	 or improve it
[14:13:44] <XioNoX>	 as it's not really actionable for us
[14:14:16] <volans>	 !log uploaded spicerack_0.0.54 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia
[14:14:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:14:28] <klausman>	 !log starting update of ML team's etcd machines in eqiad
[14:14:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:14:57] <XioNoX>	 godog: can we alert on https://grafana-rw.wikimedia.org/d/UwUa77GZk/rpki?editPanel=51&orgId=1&forceLogin=true only if more than X RSYNC are failing?
[14:15:06] <topranks>	 it's alerting that it hasn't been able to rsync over the complete list of ROAs?
[14:15:47] <XioNoX>	 yeah, there are 2 rsync servers unreachable or something
[14:16:13] <XioNoX>	 it would be better if we only alert if more than X are unreachable (showing an issue on our side)
[14:16:27] <logmsgbot>	 !log klausman@cumin1001 START - Cookbook sre.hosts.reboot-single for host ml-etcd1001.eqiad.wmnet
[14:16:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:16:41] <topranks>	 agreed
[14:16:43] <logmsgbot>	 !log jmm@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1008.eqiad.wmnet
[14:16:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:17:13] <icinga-wm>	 RECOVERY - rpki grafana alert on alert1001 is OK: OK: RPKI ( https://grafana.wikimedia.org/d/UwUa77GZk/rpki ) is not alerting. https://wikitech.wikimedia.org/wiki/RPKI%23Grafana_alerts https://grafana.wikimedia.org/d/UwUa77GZk/
[14:17:16] <topranks>	 or if it would only alert after a given time period if one source wasn't working maybe
[14:17:30] <XioNoX>	 !log reboot scs-a1-codfw - T285229
[14:17:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:17:34] <stashbot>	 T285229: SCS CPU monitoring issue - https://phabricator.wikimedia.org/T285229
[14:18:20] <XioNoX>	 topranks: yeah, but that would not be actionable as it would most likely be an issue on the remote side 
[14:18:43] <logmsgbot>	 !log klausman@cumin1001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1001.eqiad.wmnet
[14:18:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:19:30] <XioNoX>	 !log reboot scs-c1-codfw - T285229
[14:19:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:19:43] <wikibugs>	 10SRE: Integrate Buster 10.10 point update - https://phabricator.wikimedia.org/T285206 (10jbond) p:05Triage→03Medium
[14:21:06] <wikibugs>	 10SRE, 10MediaWiki-Uploading, 10Traffic, 10Chinese-Sites, 10Performance Issue: Adding an image to zh.wp 长江桥隧列表 article throws HTTP 503 or 504 error - https://phabricator.wikimedia.org/T285160 (10jbond) p:05Triage→03Medium
[14:21:12] <XioNoX>	 that way we could do the same with the RRDP graph
[14:21:13] <volans>	 !log deployed spicerack release v0.0.54 on the cumin hosts
[14:21:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:21:47] <wikibugs>	 (03CR) 10Volans: [C: 03+2] wmcs.ceph.reboot_node: update icinga_hosts call [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/700492 (owner: 10Volans)
[14:21:51] <logmsgbot>	 !log kormat@cumin1001 START - Cookbook sre.hosts.downtime for 2:00:00 on db1123.eqiad.wmnet with reason: REIMAGE
[14:21:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:22:08] <logmsgbot>	 !log klausman@cumin1001 START - Cookbook sre.hosts.reboot-single for host ml-etcd1002.eqiad.wmnet
[14:22:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:22:46] <jinxer-wm>	 (Processor usage over 85%) resolved: (2) Processor usage over 85%   - https://alerts.wikimedia.org
[14:23:00] <wikibugs>	 (03PS2) 10David Caro: ceph: add latency monitoring stats [puppet] - 10https://gerrit.wikimedia.org/r/700182 (https://phabricator.wikimedia.org/T281254)
[14:23:53] <godog>	 XioNoX: for sure, as part of T282806 I'd suggest moving to prometheus rules as opposed to grafana, since we're moving to AM anyways (see other patches on the task for example)
[14:23:53] <stashbot>	 T282806: Port traffic/netops grafana alerts to AlertManager - https://phabricator.wikimedia.org/T282806
[14:23:59] <logmsgbot>	 !log kormat@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1123.eqiad.wmnet with reason: REIMAGE
[14:24:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:24:18] <logmsgbot>	 !log klausman@cumin1001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1002.eqiad.wmnet
[14:24:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:25:12] <wikibugs>	 (03Merged) 10jenkins-bot: wmcs.ceph.reboot_node: update icinga_hosts call [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/700492 (owner: 10Volans)
[14:27:21] <XioNoX>	 godog: sure, is there doc on how to convert one to the other?
[14:27:37] <wikibugs>	 10SRE, 10MediaWiki-General, 10Platform Engineering, 10Pybal, and 3 others: SELECT query arriving to wikidatawiki db codfw hosts causing pile ups during schema change - https://phabricator.wikimedia.org/T284981 (10jbond) p:05Triage→03Medium
[14:27:54] <wikibugs>	 (03PS2) 10Elukey: Add istio 1.9.5 images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/700396 (https://phabricator.wikimedia.org/T278192)
[14:28:27] <logmsgbot>	 !log klausman@cumin1001 START - Cookbook sre.hosts.reboot-single for host ml-etcd1003.eqiad.wmnet
[14:28:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:29:33] <godog>	 XioNoX: yes the entry point is https://wikitech.wikimedia.org/wiki/Alertmanager#Create_alerts and other reviews like https://gerrit.wikimedia.org/r/c/operations/alerts/+/697710/ should provide enough context, although please LMK feedback, happy to review patches too of course
[14:30:39] <logmsgbot>	 !log klausman@cumin1001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1003.eqiad.wmnet
[14:30:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:30:50] <XioNoX>	 godog: can I test an alert somewhere?
[14:31:19] <XioNoX>	 like in Grafana I can play with the alert in real time to see if the syntax is correct
[14:32:12] <godog>	 XioNoX: yeah you can run tests locally with 'tox' (and/or let CI do it), and for the expression you can use https://thanos.wikimedia.org
[14:33:54] <XioNoX>	 godog: any idea what the syntax should be to alert on the sum of the RRDP/rsync endpoints being unreachable? (for that specific alert)
[14:34:57] <logmsgbot>	 !log klausman@cumin1001 START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl1001.eqiad.wmnet
[14:35:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:35:23] <XioNoX>	 godog: something like sum(routinator_rsync_status{site="codfw"} > 0) > 5 ?
[14:36:56] <godog>	 XioNoX: yeah sth like that, but using count() instead of sum()
[14:37:00] <logmsgbot>	 !log klausman@cumin1001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl1001.eqiad.wmnet
[14:37:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:37:25] <godog>	 sum would work too heh, I think count is more obvious
[14:37:35] <logmsgbot>	 !log klausman@cumin1001 START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl1002.eqiad.wmnet
[14:37:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:37:40] <XioNoX>	 thx!
[14:38:33] <godog>	 sure no problem! send the review my way and I'll take a deeper look XioNoX 
[14:38:53] <XioNoX>	 will give it a try now
[14:39:41] <logmsgbot>	 !log klausman@cumin1001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl1002.eqiad.wmnet
[14:39:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:40:03] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] "LGTM" [software/cumin] - 10https://gerrit.wikimedia.org/r/699192 (https://phabricator.wikimedia.org/T244840) (owner: 10Muehlenhoff)
[14:40:03] <logmsgbot>	 !log klausman@cumin1001 START - Cookbook sre.hosts.reboot-single for host ml-serve1001.eqiad.wmnet
[14:40:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:40:31] <wikibugs>	 10SRE, 10netops: Adjust egress buffer allocations on ToR switches - https://phabricator.wikimedia.org/T284592 (10jbond) p:05Triage→03Medium
[14:40:57] <wikibugs>	 10SRE, 10ops-codfw, 10DC-Ops: (Need By: TBD) rack/setup/install ganeti202[56] - https://phabricator.wikimedia.org/T282603 (10Papaul)
[14:41:04] <wikibugs>	 10SRE, 10Maps, 10Product-Infrastructure-Team-Backlog, 10SRE-swift-storage: tegola-vector-tiles load testing and Swift throughput experiments - https://phabricator.wikimedia.org/T284440 (10jbond) p:05Triage→03Medium
[14:43:20] <wikibugs>	 (03PS6) 10Muehlenhoff: Add helper tool for returning a user's current TGT [puppet] - 10https://gerrit.wikimedia.org/r/700389 (https://phabricator.wikimedia.org/T283242)
[14:44:54] <logmsgbot>	 !log klausman@cumin1001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1001.eqiad.wmnet
[14:44:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:45:30] <wikibugs>	 (03CR) 10Dzahn: gitlab: ensure backup dirs exist, add parameter for config backup (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/700601 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[14:45:41] <wikibugs>	 10SRE, 10DC-Ops, 10netops: Collect and archive KML/KMZ fiber path files for new and existing network circuits - https://phabricator.wikimedia.org/T285136 (10jbond) p:05Triage→03Medium
[14:45:42] <wikibugs>	 (03PS4) 10Dzahn: gitlab: ensure backup dirs exist, add parameter for config backup [puppet] - 10https://gerrit.wikimedia.org/r/700601 (https://phabricator.wikimedia.org/T274463)
[14:46:37] <wikibugs>	 10SRE, 10Services, 10Wikibase-Quality-Constraints, 10Wikidata, and 3 others: Deploy Shellbox instance (shellbox-constraints) for Wikidata constraint regexes - https://phabricator.wikimedia.org/T285104 (10jbond) p:05Triage→03Medium
[14:47:56] <logmsgbot>	 !log klausman@cumin1001 START - Cookbook sre.hosts.reboot-single for host ml-serve1002.eqiad.wmnet
[14:47:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:52:12] <XioNoX>	 godog: can I do something like `routinator_valid_roas{site="$externalLabels.site"}` ?
[14:52:13] <wikibugs>	 (03PS1) 10Dzahn: profile::gitlab: ensure backup dirs exist in production [puppet] - 10https://gerrit.wikimedia.org/r/700622 (https://phabricator.wikimedia.org/T274463)
[14:52:32] <logmsgbot>	 !log klausman@cumin1001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1002.eqiad.wmnet
[14:52:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:52:52] <logmsgbot>	 !log klausman@cumin1001 START - Cookbook sre.hosts.reboot-single for host ml-serve1003.eqiad.wmnet
[14:52:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:53:02] <godog>	 XioNoX: mmhh not sure, what are you trying to achieve ?
[14:53:38] <wikibugs>	 (03PS2) 10Dzahn: profile::gitlab: ensure backup dirs exist in production [puppet] - 10https://gerrit.wikimedia.org/r/700622 (https://phabricator.wikimedia.org/T274463)
[14:53:45] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] profile::gitlab: ensure backup dirs exist in production [puppet] - 10https://gerrit.wikimedia.org/r/700622 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[14:54:14] <XioNoX>	 godog: I want to duplicated the alert of https://grafana.wikimedia.org/d/UwUa77GZk/rpki?viewPanel=29&orgId=1
[14:54:33] <XioNoX>	 so alert if each routinator ROAs go under X
[14:54:58] <XioNoX>	 I feel like if I do routinator_valid_roas > X it's going to sum up both routinator instances?
[14:55:10] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] profile::gitlab: ensure backup dirs exist in production [puppet] - 10https://gerrit.wikimedia.org/r/700622 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[14:55:28] <wikibugs>	 10SRE, 10SRE-Access-Requests: Access to ptwikinews Search Console for Edu - https://phabricator.wikimedia.org/T285091 (10jbond) @KFrancis can you confirm NDA status for @Edu
[14:57:36] <logmsgbot>	 !log klausman@cumin1001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1003.eqiad.wmnet
[14:57:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:57:55] <logmsgbot>	 !log klausman@cumin1001 START - Cookbook sre.hosts.reboot-single for host ml-serve1004.eqiad.wmnet
[14:57:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:57:59] <godog>	 XioNoX: ah ok got it, yeah I'll need to expand on this bit in the documentation, however by default each alert gets evaluated in its own site, with routinator_valid_roas < 300 you'll get an alert per site
[14:58:14] <XioNoX>	 nice! ok
[14:58:24] <wikibugs>	 (03CR) 10Dzahn: "Should I put it in main or in backup.pp?" [puppet] - 10https://gerrit.wikimedia.org/r/700601 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[14:59:18] <wikibugs>	 (03PS3) 10Dzahn: profile::gitlab: ensure backup dirs exist in production [puppet] - 10https://gerrit.wikimedia.org/r/700622 (https://phabricator.wikimedia.org/T274463)
[15:00:53] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] profile::gitlab: ensure backup dirs exist in production [puppet] - 10https://gerrit.wikimedia.org/r/700622 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[15:01:05] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] "lgtm" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700595 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[15:01:19] <logmsgbot>	 !log hnowlan@cumin1001 START - Cookbook sre.hosts.reboot-single for host maps1009.eqiad.wmnet
[15:01:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:01:42] <XioNoX>	 godog: is the description field mandatory?
[15:02:03] <logmsgbot>	 !log klausman@cumin1001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1004.eqiad.wmnet
[15:02:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:03:29] <godog>	 XioNoX: yes that's correct
[15:03:56] <XioNoX>	 godog: what's $labels.operation ?
[15:05:11] <icinga-wm>	 PROBLEM - Prometheus jobs reduced availability on alert1001 is CRITICAL: job=redis_maps site=eqiad https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets
[15:05:21] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps1007 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 505065416 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[15:05:30] <godog>	 XioNoX: I think I need a bit more context, if the expression's metric has a label 'operation' then you can use $labels.operation in e.g. summary/description
[15:06:30] <icinga-wm>	 RECOVERY - Prometheus jobs reduced availability on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets
[15:06:33] <logmsgbot>	 !log hnowlan@cumin1001 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1009.eqiad.wmnet
[15:06:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:06:48] <XioNoX>	 godog: I was looking at how I could pimp the description field, and looking at https://wikitech.wikimedia.org/wiki/Alertmanager#Create_alerts there is a variable $labels.operation
[15:07:08] <XioNoX>	 but no idea what it references
[15:07:50] <godog>	 XioNoX: ah ok got it, thank you I'll expand on that in the docs, but yeah essentially you can reference the metric's labels that way
[15:08:07] <godog>	 so say $labels.tal for valid_roas
[15:09:20] <XioNoX>	 ohhh
[15:09:21] <XioNoX>	 ok
[15:10:49] <XioNoX>	 godog: next question, let me know when you're tired of them: "sum(routinator_valid_roas)" in thanos.w.o shows 168k so it sums everything in eqiad/codfw
[15:11:31] <XioNoX>	 is it going to separate the two eqiad/codfw instances with sum() ?
[15:11:36] <godog>	 XioNoX: sorry in a meeting for the next hour
[15:11:48] <XioNoX>	 ok :)
[15:17:31] <wikibugs>	 (03CR) 10Jbond: "> Patch Set 4:" [puppet] - 10https://gerrit.wikimedia.org/r/700601 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[15:25:24] <hashar>	 !log Updated operations-puppet-tests-buster-docker Jenkins job to use latest Docker image https://gerrit.wikimedia.org/r/c/integration/config/+/700648 
[15:25:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:26:38] <wikibugs>	 10SRE, 10ops-codfw, 10DC-Ops: (Need By: TBD) rack/setup/install ganeti202[56] - https://phabricator.wikimedia.org/T282603 (10Papaul)
[15:29:33] <wikibugs>	 (03PS1) 10Ayounsi: Move RPKI alerts to Prometheus/AM [alerts] - 10https://gerrit.wikimedia.org/r/700649 (https://phabricator.wikimedia.org/T282806)
[15:31:45] <XioNoX>	 godog: https://gerrit.wikimedia.org/r/c/operations/alerts/+/700649
[15:39:33] <wikibugs>	 (03CR) 10Ayounsi: [C: 03+2] Unify netbox tokens [puppet] - 10https://gerrit.wikimedia.org/r/700314 (https://phabricator.wikimedia.org/T241259) (owner: 10Ayounsi)
[15:44:37] <wikibugs>	 (03PS1) 10DCausse: [wdqs] fix prometheus_jmx_exporter class selection [puppet] - 10https://gerrit.wikimedia.org/r/700653 (https://phabricator.wikimedia.org/T270245)
[15:46:21] <wikibugs>	 (03PS1) 10Ayounsi: Remov duplicated fake netbox keys [labs/private] - 10https://gerrit.wikimedia.org/r/700655 (https://phabricator.wikimedia.org/T241259)
[15:46:48] <papaul>	 !log poweroff elastic2043 for maintenance 
[15:46:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:47:59] <icinga-wm>	 PROBLEM - Host elastic2043 is DOWN: PING CRITICAL - Packet loss = 100%
[15:49:10] <wikibugs>	 10SRE, 10Analytics-Clusters: Backport kafkacat 1.6.0 from bullseye to buster-backports or buster-wikimedia - https://phabricator.wikimedia.org/T268936 (10Ottomata) 05Open→03Resolved
[15:50:07] <icinga-wm>	 PROBLEM - Host elastic2043.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[15:50:37] <wikibugs>	 (03PS7) 10Muehlenhoff: Add helper tool for returning a user's current TGT [puppet] - 10https://gerrit.wikimedia.org/r/700389 (https://phabricator.wikimedia.org/T283242)
[15:50:39] <wikibugs>	 (03PS2) 10Ayounsi: Remove duplicated fake netbox keys [labs/private] - 10https://gerrit.wikimedia.org/r/700655 (https://phabricator.wikimedia.org/T241259)
[15:51:17] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add helper tool for returning a user's current TGT [puppet] - 10https://gerrit.wikimedia.org/r/700389 (https://phabricator.wikimedia.org/T283242) (owner: 10Muehlenhoff)
[15:51:29] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "LGTM" [labs/private] - 10https://gerrit.wikimedia.org/r/700655 (https://phabricator.wikimedia.org/T241259) (owner: 10Ayounsi)
[15:52:43] <wikibugs>	 (03CR) 10Ayounsi: [C: 03+2] Remove duplicated fake netbox keys [labs/private] - 10https://gerrit.wikimedia.org/r/700655 (https://phabricator.wikimedia.org/T241259) (owner: 10Ayounsi)
[15:52:55] <wikibugs>	 (03PS8) 10Muehlenhoff: Add helper tool for returning a user's current TGT [puppet] - 10https://gerrit.wikimedia.org/r/700389 (https://phabricator.wikimedia.org/T283242)
[15:53:01] <wikibugs>	 (03CR) 10Ayounsi: [V: 03+2 C: 03+2] Remove duplicated fake netbox keys [labs/private] - 10https://gerrit.wikimedia.org/r/700655 (https://phabricator.wikimedia.org/T241259) (owner: 10Ayounsi)
[15:53:50] <wikibugs>	 (03PS4) 10Dzahn: profile::gitlab: ensure backup dirs exist in production [puppet] - 10https://gerrit.wikimedia.org/r/700622 (https://phabricator.wikimedia.org/T274463)
[15:54:08] <wikibugs>	 (03CR) 10Muehlenhoff: "> Patch Set 5:" (038 comments) [puppet] - 10https://gerrit.wikimedia.org/r/700389 (https://phabricator.wikimedia.org/T283242) (owner: 10Muehlenhoff)
[15:54:44] <wikibugs>	 (03CR) 10Jelto: [C: 03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/700595 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[15:55:45] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] profile::gitlab: ensure backup dirs exist in production [puppet] - 10https://gerrit.wikimedia.org/r/700622 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[15:55:55] <godog>	 XioNoX: neat! LGTM from a quick look, I'll review tomorrow
[15:59:03] <XioNoX>	 thx!
[16:02:14] <wikibugs>	 10SRE, 10ops-codfw, 10DC-Ops: (Need By: TBD) rack/setup/install ganeti202[56] - https://phabricator.wikimedia.org/T282603 (10Papaul)
[16:04:57] <wikibugs>	 (03CR) 10Dzahn: gitlab::backup: create backup paths with wmflib::dir::mkdir_p (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700595 (https://phabricator.wikimedia.org/T274463) (owner: 10Dzahn)
[16:05:07] <wikibugs>	 (03PS4) 10Dzahn: gitlab::backup: create backup paths with wmflib::dir::mkdir_p [puppet] - 10https://gerrit.wikimedia.org/r/700595 (https://phabricator.wikimedia.org/T274463)
[16:05:31] <icinga-wm>	 RECOVERY - Host elastic2043 is UP: PING OK - Packet loss = 0%, RTA = 31.72 ms
[16:07:57] <icinga-wm>	 RECOVERY - Host elastic2043.mgmt is UP: PING OK - Packet loss = 0%, RTA = 33.73 ms
[16:19:56] <wikibugs>	 (03PS2) 10Cwhite: Add metric group. [software/ecs] - 10https://gerrit.wikimedia.org/r/699428
[16:21:11] <papaul>	 me ^ fixing
[16:21:19] <wikibugs>	 (03PS3) 10Cwhite: Add metrics group. [software/ecs] - 10https://gerrit.wikimedia.org/r/699428
[16:23:13] <icinga-wm>	 PROBLEM - Host elastic2043 is DOWN: PING CRITICAL - Packet loss = 100%
[16:27:41] <icinga-wm>	 PROBLEM - Host elastic2043.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[16:30:13] <icinga-wm>	 RECOVERY - Host elastic2043.mgmt is UP: PING OK - Packet loss = 0%, RTA = 33.71 ms
[16:31:23] <icinga-wm>	 RECOVERY - Host elastic2043 is UP: PING WARNING - Packet loss = 71%, RTA = 31.74 ms
[16:32:34] <logmsgbot>	 !log pt1979@cumin2002 START - Cookbook sre.dns.netbox
[16:32:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:37:06] <wikibugs>	 (03PS4) 10Jbond: rspec: update to use rspec-mock instead of mocha [puppet] - 10https://gerrit.wikimedia.org/r/649684
[16:37:08] <wikibugs>	 (03PS1) 10Jbond: rake_modules: update the dynamic spec test to use ParallelTests [puppet] - 10https://gerrit.wikimedia.org/r/700660
[16:37:57] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] rspec: update to use rspec-mock instead of mocha [puppet] - 10https://gerrit.wikimedia.org/r/649684 (owner: 10Jbond)
[16:38:15] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] rake_modules: update the dynamic spec test to use ParallelTests [puppet] - 10https://gerrit.wikimedia.org/r/700660 (owner: 10Jbond)
[16:47:21] <logmsgbot>	 !log pt1979@cumin2002 END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
[16:47:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:59:28] <wikibugs>	 10SRE, 10ops-codfw, 10DC-Ops: (Need By: TBD) rack/setup/install ganeti202[56] - https://phabricator.wikimedia.org/T282603 (10Papaul)
[17:00:04] <jouncebot>	 ryankemper: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Wikidata Query Service weekly deploy . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210621T1700).
[17:03:08] <wikibugs>	 10SRE, 10ops-codfw, 10DC-Ops, 10Discovery-Search: hw troubleshooting: failure to power up for elastic2043.codfw.wmnet - https://phabricator.wikimedia.org/T281327 (10Papaul) 05Open→03Resolved Dell Tech replaced the back plane and cables. server is backup online. Closing this task for now. Please re-open...
[17:27:46] <logmsgbot>	 !log ebernhardson@deploy1002 Started deploy [wikimedia/discovery/analytics@2337592]: airflow: expect eventgate canary events in all dcs
[17:27:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:28:51] <wikibugs>	 (03PS2) 10Brennen Bearnes: copy latest backup to dedicated folder [gitlab-ansible] - 10https://gerrit.wikimedia.org/r/700084 (https://phabricator.wikimedia.org/T274463) (owner: 10Jelto)
[17:32:10] <logmsgbot>	 !log ebernhardson@deploy1002 Finished deploy [wikimedia/discovery/analytics@2337592]: airflow: expect eventgate canary events in all dcs (duration: 04m 24s)
[17:32:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:40:27] <ebernhardson>	 !log post-deploy restart airflow-webserver and airflow-scheduler on an-airflow1001
[17:40:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:44:23] <wikibugs>	 10SRE, 10SRE-Access-Requests: Access to ptwikinews Search Console for Edu - https://phabricator.wikimedia.org/T285091 (10KFrancis) @jbond I haven't been able to locate a record of the NDA.  Is the user an employee, contractor, or volunteer?
[17:51:58] <wikibugs>	 10SRE, 10Wikimedia-Mailing-lists, 10Patch-For-Review: Implement static redirects from pipermail archives to hyperkitty archives - https://phabricator.wikimedia.org/T280731 (10Legoktm) 05Open→03Resolved a:03Legoktm We currently have 620,937 redirects in place, which is much lower than my original estima...
[17:52:00] <wikibugs>	 (03CR) 10Brennen Bearnes: [C: 03+2] "> Patch Set 1: Code-Review+1" [gitlab-ansible] - 10https://gerrit.wikimedia.org/r/700084 (https://phabricator.wikimedia.org/T274463) (owner: 10Jelto)
[18:00:04] <jouncebot>	 RoanKattouw, Niharika, and Urbanecm: I, the Bot under the Fountain, allow thee, The Deployer, to do Morning backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210621T1800).
[18:00:04] <jouncebot>	 ebernhardson and Zabe: A patch you scheduled for Morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[18:00:37] <zabe>	 o/
[18:01:13] <urbanecm>	 i can deploy today
[18:02:00] <wikibugs>	 10SRE, 10SRE-Access-Requests: Access to ptwikinews Search Console for Edu - https://phabricator.wikimedia.org/T285091 (10Urbanecm) >>! In T285091#7166970, @KFrancis wrote: > @jbond I haven't been able to locate a record of the NDA.  Is the user an employee, contractor, or volunteer?  The user is a volunteer. B...
[18:02:23] <urbanecm>	 ebernhardson: around too?
[18:03:08] <wikibugs>	 (03PS4) 10Urbanecm: eswiki AbuseFilter config changes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699494 (https://phabricator.wikimedia.org/T284797) (owner: 10Zabe)
[18:03:12] <wikibugs>	 (03CR) 10Urbanecm: [C: 03+2] eswiki AbuseFilter config changes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699494 (https://phabricator.wikimedia.org/T284797) (owner: 10Zabe)
[18:04:05] <wikibugs>	 (03Merged) 10jenkins-bot: eswiki AbuseFilter config changes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699494 (https://phabricator.wikimedia.org/T284797) (owner: 10Zabe)
[18:05:01] <Pchelolo>	 urbanecm: mind pinging me when you're done? I'll squeeze in one more
[18:05:08] <urbanecm>	 not at all
[18:05:10] <urbanecm>	 will ping you
[18:05:43] <Pchelolo>	 thank you!
[18:06:14] <urbanecm>	 zabe: pulled onto mwdebug1001, if you can test it there
[18:07:34] <wikibugs>	 (03PS4) 10Urbanecm: Add 'managechangetags' to the 'abusefilter' group on eswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700433 (https://phabricator.wikimedia.org/T285167) (owner: 10Zabe)
[18:07:38] <wikibugs>	 (03CR) 10Urbanecm: [C: 03+2] Add 'managechangetags' to the 'abusefilter' group on eswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700433 (https://phabricator.wikimedia.org/T285167) (owner: 10Zabe)
[18:08:14] <zabe>	 urbanecm: seems to be working, I don't realy know if/how I can test the increased thresholds
[18:08:34] <urbanecm>	 i think that's only possible via making edits
[18:08:39] <urbanecm>	 let's skip that part and believe it :)
[18:08:40] <urbanecm>	 syncing
[18:09:26] <wikibugs>	 (03Merged) 10jenkins-bot: Add 'managechangetags' to the 'abusefilter' group on eswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700433 (https://phabricator.wikimedia.org/T285167) (owner: 10Zabe)
[18:10:13] <logmsgbot>	 !log urbanecm@deploy1002 Synchronized wmf-config/abusefilter.php: 219dd5b79d59b98d13e34f7d6e22a41ae1dd9cbb: eswiki AbuseFilter config changes (T284797; 1/2) (duration: 01m 07s)
[18:10:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:10:19] <stashbot>	 T284797: AbuseFilter config changes for eswiki - https://phabricator.wikimedia.org/T284797
[18:10:45] <wikibugs>	 (03PS2) 10Urbanecm: Disable Education Program namespaces in enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700455 (https://phabricator.wikimedia.org/T285193) (owner: 10Zabe)
[18:10:49] <wikibugs>	 (03CR) 10Urbanecm: [C: 03+2] Disable Education Program namespaces in enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700455 (https://phabricator.wikimedia.org/T285193) (owner: 10Zabe)
[18:11:20] <logmsgbot>	 !log urbanecm@deploy1002 Synchronized wmf-config/InitialiseSettings.php: 219dd5b79d59b98d13e34f7d6e22a41ae1dd9cbb: eswiki AbuseFilter config changes (T284797; 2/2) (duration: 00m 56s)
[18:11:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:11:42] <wikibugs>	 (03CR) 10Arlolra: Bump envoy timeout for parsoid-php (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/699425 (https://phabricator.wikimedia.org/T244609) (owner: 10Arlolra)
[18:11:45] <urbanecm>	 zabe: ` Add 'managechangetags' to the 'abusefilter' group on eswiki` is at mwdebug1001, please test
[18:12:30] <wikibugs>	 (03Merged) 10jenkins-bot: Disable Education Program namespaces in enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700455 (https://phabricator.wikimedia.org/T285193) (owner: 10Zabe)
[18:12:34] <zabe>	 urbanecm: works, I see the added 'managechangetags' permission
[18:12:39] <urbanecm>	 great, snycing
[18:14:01] <wikibugs>	 (03PS4) 10Urbanecm: Rename Portal and Portal talk namespaces on viwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700065 (https://phabricator.wikimedia.org/T284868) (owner: 10Zabe)
[18:14:04] <wikibugs>	 (03CR) 10Urbanecm: [C: 03+2] Rename Portal and Portal talk namespaces on viwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700065 (https://phabricator.wikimedia.org/T284868) (owner: 10Zabe)
[18:14:09] <logmsgbot>	 !log urbanecm@deploy1002 Synchronized wmf-config/abusefilter.php: 5a51dd2540cd620cec60b9921b57c8ab185df029: Add `managechangetags` to the `abusefilter` group on eswiki (T285167) (duration: 00m 56s)
[18:14:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:14:14] <stashbot>	 T285167: Add managechangetags right to abusefilter group in eswiki - https://phabricator.wikimedia.org/T285167
[18:14:28] <urbanecm>	 zabe: `Disable Education Program namespaces in enwiki` is now at mwdebug1001, please test
[18:14:51] <wikibugs>	 (03CR) 10Elukey: "Ready for review" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/700396 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey)
[18:15:14] <ebernhardson>	 urbanecm: howdy, little late but here :)
[18:15:20] <zabe>	 urbanecm: works the supposed way
[18:15:35] <urbanecm>	 ebernhardson: ack :). Do you prefer self-service, or should I deploy for you?
[18:15:39] <urbanecm>	 zabe: thanks, syncing
[18:15:53] <ebernhardson>	 urbanecm: you can if you want, this patch doesn't do anything on its own. Sets up config for the next train deploy
[18:15:59] <wikibugs>	 (03Merged) 10jenkins-bot: Rename Portal and Portal talk namespaces on viwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700065 (https://phabricator.wikimedia.org/T284868) (owner: 10Zabe)
[18:16:19] <ebernhardson>	 well, i guess it shrinks a prod pool counter a little, but it has plenty of overhead
[18:16:45] <urbanecm>	 ebernhardson: you linked https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/699311 there, which is a patch for CirrusSearch master
[18:16:52] <logmsgbot>	 !log urbanecm@deploy1002 Synchronized wmf-config/InitialiseSettings.php: 5d8b9df1d6c77cae8fa18c343e5f1ec139d603b0: Disable Education Program namespaces in enwiki (T285193) (duration: 00m 58s)
[18:16:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:16:57] <ebernhardson>	 urbanecm: doh, one sec i have an actual config patch :)
[18:16:57] <stashbot>	 T285193: Remove education program namespaces from enwiki - https://phabricator.wikimedia.org/T285193
[18:17:23] <urbanecm>	 thanks ebernhardson 
[18:17:44] <ebernhardson>	 urbanecm: updated wikitech, https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/699257
[18:17:59] <urbanecm>	 zabe: `Rename Portal and Portal talk namespaces on viwiki` is at mwdebug1001, please test
[18:18:22] * ebernhardson shouldn't have named two patches to two different repos the same thing...oh well
[18:18:37] <urbanecm>	 hehe :)
[18:18:56] <urbanecm>	 ebernhardson: anyway, i'll deploy it (once i'm done with zabe's patches). i assume no testing needed, right? 
[18:19:04] <ebernhardson>	 urbanecm: correct
[18:19:06] <ebernhardson>	 thanks!
[18:19:09] <urbanecm>	 ack, thanks
[18:20:28] <wikibugs>	 (03PS3) 10Brennen Bearnes: copy latest backup to dedicated folder [gitlab-ansible] - 10https://gerrit.wikimedia.org/r/700084 (https://phabricator.wikimedia.org/T274463) (owner: 10Jelto)
[18:21:00] <zabe>	 urbanecm: seems to be working
[18:21:07] <urbanecm>	 zabe: thanks, syncing
[18:21:18] <wikibugs>	 (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] "Checked against test box, running against gitlab1001." [gitlab-ansible] - 10https://gerrit.wikimedia.org/r/700084 (https://phabricator.wikimedia.org/T274463) (owner: 10Jelto)
[18:22:07] <wikibugs>	 (03PS2) 10Urbanecm: Enable wikilove on hewikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699495 (https://phabricator.wikimedia.org/T284864) (owner: 10Zabe)
[18:22:12] <wikibugs>	 (03CR) 10Urbanecm: [C: 03+2] Enable wikilove on hewikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699495 (https://phabricator.wikimedia.org/T284864) (owner: 10Zabe)
[18:22:41] <logmsgbot>	 !log urbanecm@deploy1002 Synchronized wmf-config/InitialiseSettings.php: dd0fecb89da39611cdf18584626508741cf369e9: Rename Portal and Portal talk namespaces on viwiki (T284868) (duration: 00m 56s)
[18:22:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:22:46] <stashbot>	 T284868: Rename Portal and Portal talk namespaces on viwiki - https://phabricator.wikimedia.org/T284868
[18:22:57] <logmsgbot>	 !log ebernhardson@deploy1002 Started deploy [wikimedia/discovery/analytics@40b4b2f]: T273854 Airflow dag to extract and process sparql queries
[18:23:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:23:01] <stashbot>	 T273854: Automate regular WDQS query parsing and data-extraction - https://phabricator.wikimedia.org/T273854
[18:23:08] <wikibugs>	 (03Merged) 10jenkins-bot: Enable wikilove on hewikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699495 (https://phabricator.wikimedia.org/T284864) (owner: 10Zabe)
[18:23:10] <urbanecm>	 !log [urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=hiwikisource # T284864
[18:23:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:23:15] <stashbot>	 T284864: Install wikilove in hewikisource - https://phabricator.wikimedia.org/T284864
[18:23:27] <urbanecm>	 !log Correction: [urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=hiwikisource wikilove # T284864
[18:23:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:23:55] <urbanecm>	 and it's *he*wikisource...
[18:23:56] <urbanecm>	 meh
[18:24:19] <urbanecm>	 !log [urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=hewikisource wikilove # T284864
[18:24:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:24:37] <urbanecm>	 zabe: wikilove patch is at mwdebug1001, please test
[18:26:29] <brennen>	 !log gitlab1001: running ansible for copying latest backup to dedicated folder (T274463)
[18:26:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:26:34] <stashbot>	 T274463: Backups for GitLab - https://phabricator.wikimedia.org/T274463
[18:27:12] <wikibugs>	 10SRE, 10SRE-Access-Requests: Access to ptwikinews Search Console for Edu - https://phabricator.wikimedia.org/T285091 (10KFrancis) @Urbanecm @jbond Okay, no problem!  I will need to set up an NDA.  I will need the volunteer's:  Full legal name Mailing address Email address  Specifics about the type of sensitiv...
[18:27:43] <zabe>	 urbanecm: works, tested with one barnstar
[18:27:48] <urbanecm>	 thanks, syncing
[18:28:37] <wikibugs>	 (03CR) 10Urbanecm: [C: 03+2] Add pool counter for automated search requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699257 (https://phabricator.wikimedia.org/T284479) (owner: 10Ebernhardson)
[18:29:10] <logmsgbot>	 !log urbanecm@deploy1002 Synchronized wmf-config/InitialiseSettings.php: f7db2b94d62c01cd62ec88d25675d3349d9c1b98: Enable wikilove on hewikisource (T284864) (duration: 00m 56s)
[18:29:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:29:16] <stashbot>	 T284864: Install wikilove in hewikisource - https://phabricator.wikimedia.org/T284864
[18:29:26] <wikibugs>	 (03Merged) 10jenkins-bot: Add pool counter for automated search requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699257 (https://phabricator.wikimedia.org/T284479) (owner: 10Ebernhardson)
[18:30:02] <urbanecm>	 ebernhardson: syncing yours
[18:30:08] <logmsgbot>	 !log ebernhardson@deploy1002 Finished deploy [wikimedia/discovery/analytics@40b4b2f]: T273854 Airflow dag to extract and process sparql queries (duration: 07m 11s)
[18:30:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:30:13] <stashbot>	 T273854: Automate regular WDQS query parsing and data-extraction - https://phabricator.wikimedia.org/T273854
[18:30:58] <logmsgbot>	 !log urbanecm@deploy1002 Synchronized wmf-config/PoolCounterSettings.php: af61f1a9ffc4703ad9514eac96655c3dd9d491d8: Add pool counter for automated search requests (T284479) (duration: 00m 59s)
[18:31:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:31:02] <stashbot>	 T284479: Cirrussearch: spike in pool counter rejections related to full_text and entity_full_text queries - https://phabricator.wikimedia.org/T284479
[18:31:05] <urbanecm>	 and...synced
[18:31:16] <ebernhardson>	 urbanecm: excellent
[18:31:22] <urbanecm>	 Pchelolo: that should be all from me, floor is yours.
[18:31:29] <Pchelolo>	 great, thank you
[18:33:21] <icinga-wm>	 PROBLEM - Disk space on an-airflow1001 is CRITICAL: DISK CRITICAL - free space: / 827 MB (2% inode=79%): /tmp 827 MB (2% inode=79%): /var/tmp 827 MB (2% inode=79%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=an-airflow1001&var-datasource=eqiad+prometheus/ops
[18:34:04] <wikibugs>	 (03PS1) 10Bartosz Dziewoński: Offer the DiscussionTools reply tool as opt-out setting at Wikimania wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700675 (https://phabricator.wikimedia.org/T284339)
[18:37:04] <wikibugs>	 (03PS2) 10Ppchelko: Replace uses of AbstractBlock::getTarget() [mediawiki-config] - 10https://gerrit.wikimedia.org/r/697786 (https://phabricator.wikimedia.org/T284141) (owner: 10Vlad.shapik)
[18:37:40] <wikibugs>	 (03CR) 10Ppchelko: [C: 03+2] Replace uses of AbstractBlock::getTarget() [mediawiki-config] - 10https://gerrit.wikimedia.org/r/697786 (https://phabricator.wikimedia.org/T284141) (owner: 10Vlad.shapik)
[18:38:41] <wikibugs>	 (03Merged) 10jenkins-bot: Replace uses of AbstractBlock::getTarget() [mediawiki-config] - 10https://gerrit.wikimedia.org/r/697786 (https://phabricator.wikimedia.org/T284141) (owner: 10Vlad.shapik)
[18:41:20] <logmsgbot>	 !log ppchelko@deploy1002 Synchronized wmf-config/wikitech.php: Replace uses of AbstractBlock::getTarget() T284141 (duration: 00m 58s)
[18:41:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:41:26] <stashbot>	 T284141: operations / mediawiki-config: replace uses of AbstractBlock ::getTargetAndType() and ::getTarget() - https://phabricator.wikimedia.org/T284141
[18:43:47] <wikibugs>	 10SRE, 10vm-requests: Site: 2 VM request for an-airflow100{2,3} - https://phabricator.wikimedia.org/T284934 (10razzi) 05Open→03Resolved VMS are created, further actions will happen on https://phabricator.wikimedia.org/T284225
[18:46:28] <wikibugs>	 (03PS1) 10Brennen Bearnes: remove when condition on root pass variable [gitlab-ansible] - 10https://gerrit.wikimedia.org/r/700676
[18:47:51] <wikibugs>	 10SRE, 10ops-codfw, 10DC-Ops: (Need By: TBD) rack/setup/install ganeti202[56] - https://phabricator.wikimedia.org/T282603 (10Papaul) @MoritzMuehlenhoff are we using ganeti-raid1.cfg on those nodes or ganeti-raid5.cfg?
[18:48:25] <wikibugs>	 (03PS1) 10Bartosz Dziewoński: Offer the DiscussionTools reply tool as opt-out setting at ptwikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700677 (https://phabricator.wikimedia.org/T285162)
[18:52:26] <wikibugs>	 (03PS1) 10Papaul: DHCP, Site.pp: Add ganeti202[56] to site.pp and it's MAC address [puppet] - 10https://gerrit.wikimedia.org/r/700678 (https://phabricator.wikimedia.org/T282603)
[19:00:24] <wikibugs>	 10SRE, 10conftool, 10serviceops, 10Datacenter-Switchover: Disable maintenance scripts via conftool - https://phabricator.wikimedia.org/T266717 (10RLazarus) a:05RLazarus→03Legoktm
[19:02:57] <Trey314159>	  !log reindexing German wikis on elastic@eqiad, elastic@codfw, and cloudelastic (T284185)
[19:02:57] <stashbot>	 T284185: Reindex German, Dutch, and Portugese Wikis - https://phabricator.wikimedia.org/T284185
[19:08:22] <wikibugs>	 10SRE, 10serviceops, 10Datacenter-Switchover: Updates to warmup script - https://phabricator.wikimedia.org/T269179 (10Legoktm) > As written, the warmup script only covers the app servers, and doesn't warm up the API servers at all. It's less harmful to have a spike of latency in the API servers than the app...
[19:10:03] <wikibugs>	 (03PS1) 10Legoktm: mediawiki-cache-warmup: Add one more Action API URL [puppet] - 10https://gerrit.wikimedia.org/r/700683 (https://phabricator.wikimedia.org/T269179)
[19:10:20] <wikibugs>	 (03PS1) 10Legoktm: sre.switchdc.services: Fix comment on 02-restore-ttl [cookbooks] - 10https://gerrit.wikimedia.org/r/700684
[19:12:37] <icinga-wm>	 RECOVERY - WDQS high update lag on wdqs1005 is OK: (C)4.32e+04 ge (W)2.16e+04 ge 2.151e+04 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen
[19:12:49] <wikibugs>	 (03CR) 10Papaul: [C: 03+2] DHCP, Site.pp: Add ganeti202[56] to site.pp and it's MAC address [puppet] - 10https://gerrit.wikimedia.org/r/700678 (https://phabricator.wikimedia.org/T282603) (owner: 10Papaul)
[19:18:29] <icinga-wm>	 PROBLEM - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The following units failed: netbox_report_accounting_run.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[19:19:03] <logmsgbot>	 !log sukhe@cumin1001 START - Cookbook sre.hosts.downtime for 0:05:00 on doh1001.wikimedia.org with reason: temporarily depooling host
[19:19:04] <logmsgbot>	 !log sukhe@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on doh1001.wikimedia.org with reason: temporarily depooling host
[19:19:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:19:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:26:35] <Reedy>	 jouncebot: now
[19:26:35] <jouncebot>	 No deployments scheduled for the next 0 hour(s) and 33 minute(s)
[19:26:36] <Reedy>	 jouncebot: next
[19:26:37] <jouncebot>	 In 0 hour(s) and 33 minute(s): Services – Graphoid / ORES (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210621T2000)
[19:44:01] <icinga-wm>	 RECOVERY - Check systemd state on netbox1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[19:48:53] <icinga-wm>	 PROBLEM - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/transform/html/to/mobile-html/{title} (Get preview mobile HTML for test page) is CRITICAL: Test Get preview mobile HTML for test page returned the unexpected status 503 (expecting: 200) https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29
[19:50:43] <icinga-wm>	 RECOVERY - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29
[20:00:04] <jouncebot>	 chrisalbon and accraze: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for Services – Graphoid / ORES. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210621T2000).
[20:28:04] <wikibugs>	 (03CR) 10Volans: [C: 03+2] "Thanks!" [cookbooks] - 10https://gerrit.wikimedia.org/r/700684 (owner: 10Legoktm)
[20:31:13] <wikibugs>	 (03Merged) 10jenkins-bot: sre.switchdc.services: Fix comment on 02-restore-ttl [cookbooks] - 10https://gerrit.wikimedia.org/r/700684 (owner: 10Legoktm)
[20:39:46] <wikibugs>	 10SRE, 10observability, 10Datacenter-Switchover: Switchover thanos-query and thanos-swift services as part of DC switchover - https://phabricator.wikimedia.org/T285273 (10Legoktm)
[20:40:41] <wikibugs>	 10SRE, 10Wikimedia-Mailing-lists, 10User-Ladsgroup: Close wikimediahk@lists.wikimedia.org - https://phabricator.wikimedia.org/T285194 (10Ladsgroup) 05Open→03Resolved Done. The mailing list is now closed. The archives have not been deleted.
[20:40:42] <wikibugs>	 (03PS1) 10Legoktm: sre.switchdc.services: Don't exclude thanos-query & thanos-swift services [cookbooks] - 10https://gerrit.wikimedia.org/r/700695 (https://phabricator.wikimedia.org/T285273)
[20:44:44] <wikibugs>	 (03CR) 10Ebernhardson: [C: 03+1] "patch seems appropriate, pcc looks reasonable: https://puppet-compiler.wmflabs.org/compiler1001/29952/" [puppet] - 10https://gerrit.wikimedia.org/r/700653 (https://phabricator.wikimedia.org/T270245) (owner: 10DCausse)
[20:53:20] <sbassett>	 Hey all - some secteam folks are going to deploy a sec patch for T285190 soon.
[21:00:05] <jouncebot>	 Reedy and sbassett: Dear deployers, time to do the Weekly Security deployment window deploy. Dont look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210621T2100).
[21:04:18] <wikibugs>	 10SRE, 10ops-codfw, 10DC-Ops: (Need By: TBD) rack/setup/install ganeti202[56] - https://phabricator.wikimedia.org/T282603 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2002.codfw.wmnet for hosts: ` ganeti2025.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/202...
[21:10:23] <wikibugs>	 (03CR) 10Krinkle: [C: 04-1] "Does this need to be a per-server URL, as opposed to a perwiki/clusterwide url? Also, given we already have 1 api.php url, and the task sa" [puppet] - 10https://gerrit.wikimedia.org/r/700683 (https://phabricator.wikimedia.org/T269179) (owner: 10Legoktm)
[21:11:17] <sbassett>	 !log Deployed security patch for T285190
[21:11:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:13:48] <wikibugs>	 (03PS1) 10Legoktm: sre.switchdc.mediawiki: Warm up caches in api_appserver cluster too [cookbooks] - 10https://gerrit.wikimedia.org/r/700704 (https://phabricator.wikimedia.org/T269179)
[21:14:41] <wikibugs>	 (03CR) 10Legoktm: "> Patch Set 1: Code-Review-1" [puppet] - 10https://gerrit.wikimedia.org/r/700683 (https://phabricator.wikimedia.org/T269179) (owner: 10Legoktm)
[21:14:55] <wikibugs>	 (03Abandoned) 10Legoktm: mediawiki-cache-warmup: Add one more Action API URL [puppet] - 10https://gerrit.wikimedia.org/r/700683 (https://phabricator.wikimedia.org/T269179) (owner: 10Legoktm)
[21:23:44] <wikibugs>	 10SRE, 10serviceops, 10Datacenter-Switchover, 10Patch-For-Review: Updates to warmup script - https://phabricator.wikimedia.org/T269179 (10Legoktm) >>! In T269179#7167263, @Legoktm wrote: >> As written, the warmup script only covers the app servers, and doesn't warm up the API servers at all. It's less harm...
[21:30:18] <wikibugs>	 10SRE, 10ops-codfw, 10DC-Ops: (Need By: TBD) rack/setup/install ganeti202[56] - https://phabricator.wikimedia.org/T282603 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['ganeti2025.codfw.wmnet'] `  Of which those **FAILED**: ` ['ganeti2025.codfw.wmnet'] `
[21:31:18] <wikibugs>	 (03PS2) 10Cwhite: rsyslog: try to parse the msg field as json before shipping [puppet] - 10https://gerrit.wikimedia.org/r/694758
[21:33:52] <wikibugs>	 (03CR) 10Cwhite: [C: 03+2] rsyslog: try to parse the msg field as json before shipping [puppet] - 10https://gerrit.wikimedia.org/r/694758 (owner: 10Cwhite)
[21:34:59] <wikibugs>	 (03PS3) 10Krinkle: [Beta Cluster] mc-labs.php: Enable onHostRoutingPrefix for WAN cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692729 (https://phabricator.wikimedia.org/T264604)
[21:35:18] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [Beta Cluster] mc-labs.php: Enable onHostRoutingPrefix for WAN cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692729 (https://phabricator.wikimedia.org/T264604) (owner: 10Krinkle)
[21:37:02] <wikibugs>	 (03PS3) 10Krinkle: Simplify mc.php (2/7): Remove mc-labs.php overrides that match prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692723
[21:43:41] <wikibugs>	 (03PS1) 10Jdrewniak: Enable new Vector Languages-in-header feature & AB test for pilot wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/700705 (https://phabricator.wikimedia.org/T269093)
[21:55:45] <wikibugs>	 (03PS1) 10Jdrewniak: launchULS: Add context to interface.language.change hook [extensions/UniversalLanguageSelector] (wmf/1.37.0-wmf.10) - 10https://gerrit.wikimedia.org/r/700633 (https://phabricator.wikimedia.org/T280770)
[22:06:41] <wikibugs>	 (03PS3) 10Krinkle: Simplify mc.php (3/7): Remove $wgMemCachedTimeout setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692724
[22:06:49] <wikibugs>	 (03PS3) 10Krinkle: Simplify mc.php (4/7): Move wgMemCachedServers to mc.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692725
[22:06:56] <wikibugs>	 (03PS3) 10Krinkle: Simplify mc.php (5/7): Move wgMemCachedServers to mc.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692726
[22:07:03] <wikibugs>	 (03PS3) 10Krinkle: Simplify mc.php (6/7): Move mc.php and db.php inclusion a few lines up [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692727
[22:07:09] <wikibugs>	 (03PS3) 10Krinkle: Simplify mc.php (7/7): Define 'wancache-main-mcrouter' unconditionally [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692728
[22:08:27] <wikibugs>	 (03CR) 10Aaron Schulz: Simplify mc.php (2/7): Remove mc-labs.php overrides that match prod (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692723 (owner: 10Krinkle)
[22:08:56] <wikibugs>	 (03CR) 10Aaron Schulz: Simplify mc.php (2/7): Remove mc-labs.php overrides that match prod (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692723 (owner: 10Krinkle)
[22:09:01] <wikibugs>	 (03CR) 10Aaron Schulz: [C: 03+1] Simplify mc.php (2/7): Remove mc-labs.php overrides that match prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692723 (owner: 10Krinkle)
[22:09:32] <wikibugs>	 (03CR) 10Aaron Schulz: [C: 03+1] Simplify mc.php (3/7): Remove $wgMemCachedTimeout setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692724 (owner: 10Krinkle)
[22:11:28] <wikibugs>	 (03CR) 10Aaron Schulz: [C: 03+1] Simplify mc.php (4/7): Move wgMemCachedServers to mc.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692725 (owner: 10Krinkle)
[22:11:36] <wikibugs>	 (03CR) 10Aaron Schulz: [C: 03+1] Simplify mc.php (5/7): Move wgMemCachedServers to mc.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692726 (owner: 10Krinkle)
[22:23:18] <wikibugs>	 10SRE, 10serviceops, 10Datacenter-Switchover, 10Patch-For-Review: Updates to warmup script (2020-2021) - https://phabricator.wikimedia.org/T269179 (10Krinkle)
[22:37:57] <wikibugs>	 (03CR) 10Aaron Schulz: [C: 03+1] Simplify mc.php (6/7): Move mc.php and db.php inclusion a few lines up [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692727 (owner: 10Krinkle)
[22:38:18] <wikibugs>	 (03CR) 10Aaron Schulz: [C: 03+1] Simplify mc.php (7/7): Define 'wancache-main-mcrouter' unconditionally [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692728 (owner: 10Krinkle)
[22:39:47] <wikibugs>	 (03PS1) 10Krinkle: mediawiki-cache-warmup: Improve url docs [puppet] - 10https://gerrit.wikimedia.org/r/700712 (https://phabricator.wikimedia.org/T269179)
[22:41:46] <urbanecm>	 !log [urbanecm@mwmaint1002 ~]$ mwscript namespaceDupes.php --wiki=viwiki --fix # T284868 # P16674
[22:41:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[22:41:52] <stashbot>	 T284868: Rename Portal and Portal talk namespaces on viwiki - https://phabricator.wikimedia.org/T284868
[22:43:30] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] Simplify mc.php (2/7): Remove mc-labs.php overrides that match prod (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692723 (owner: 10Krinkle)
[22:44:10] <wikibugs>	 (03Merged) 10jenkins-bot: Simplify mc.php (2/7): Remove mc-labs.php overrides that match prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692723 (owner: 10Krinkle)
[22:44:25] * Krinkle testing on mwdebug1002
[22:48:33] <Krinkle>	 hm.. I'm getting nothing in logstash
[22:48:39] <Krinkle>	 incl the stuff I'd expect to get
[22:51:18] <Krinkle>	 hm.. I guess scap logs are still broken / missing from the index where they used to be
[22:51:33] <Krinkle>	 and mw no longer logs anything during an edit. It used to.
[22:51:39] <Krinkle>	 verbose logging does show a bunch of stuff, okay
[22:51:45] <Krinkle>	 alright, rolling out
[22:53:06] <logmsgbot>	 !log krinkle@deploy1002 Synchronized wmf-config/CommonSettings.php: Iea94283c53 (duration: 00m 57s)
[22:53:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[22:56:52] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] Simplify mc.php (3/7): Remove $wgMemCachedTimeout setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692724 (owner: 10Krinkle)
[22:57:32] <wikibugs>	 (03Merged) 10jenkins-bot: Simplify mc.php (3/7): Remove $wgMemCachedTimeout setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692724 (owner: 10Krinkle)
[22:57:44] <logmsgbot>	 !log krinkle@deploy1002 Synchronized wmf-config/mc.php: Iea94283c53 (duration: 00m 57s)
[22:57:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:00:05] <jouncebot>	 RoanKattouw, Niharika, and Urbanecm: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Evening backport window . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210621T2300).
[23:00:05] <jouncebot>	 No GERRIT patches in the queue for this window AFAICS.
[23:03:13] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] Simplify mc.php (4/7): Move wgMemCachedServers to mc.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692725 (owner: 10Krinkle)
[23:03:58] <wikibugs>	 (03Merged) 10jenkins-bot: Simplify mc.php (4/7): Move wgMemCachedServers to mc.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692725 (owner: 10Krinkle)
[23:04:01] <logmsgbot>	 !log krinkle@deploy1002 Synchronized wmf-config/mc.php: Icc2676c23b808d (duration: 00m 56s)
[23:04:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:05:01] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] Simplify mc.php (5/7): Move wgMemCachedServers to mc.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692726 (owner: 10Krinkle)
[23:05:50] <wikibugs>	 (03Merged) 10jenkins-bot: Simplify mc.php (5/7): Move wgMemCachedServers to mc.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692726 (owner: 10Krinkle)
[23:05:54] <logmsgbot>	 !log krinkle@deploy1002 Synchronized wmf-config/mc.php: I877a3ee1bbffb1e0 (duration: 00m 57s)
[23:05:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:07:00] <wikibugs>	 (03PS4) 10Krinkle: [Beta Cluster] mc-labs.php: Enable onHostRoutingPrefix for WAN cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692729 (https://phabricator.wikimedia.org/T264604)
[23:07:04] <wikibugs>	 (03CR) 10Legoktm: [C: 03+2] mediawiki-cache-warmup: Improve url docs [puppet] - 10https://gerrit.wikimedia.org/r/700712 (https://phabricator.wikimedia.org/T269179) (owner: 10Krinkle)
[23:08:29] <logmsgbot>	 !log krinkle@deploy1002 Synchronized wmf-config/CommonSettings.php: Idcac4d0345059f84 (duration: 00m 56s)
[23:08:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:09:01] <wikibugs>	 (03PS4) 10Krinkle: Simplify mc.php (7/7): Define 'wancache-main-mcrouter' unconditionally [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692728
[23:09:11] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] Simplify mc.php (6/7): Move mc.php and db.php inclusion a few lines up [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692727 (owner: 10Krinkle)
[23:09:54] <wikibugs>	 (03Merged) 10jenkins-bot: Simplify mc.php (6/7): Move mc.php and db.php inclusion a few lines up [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692727 (owner: 10Krinkle)
[23:12:02] <wikibugs>	 (03CR) 10Aaron Schulz: [C: 03+1] [Beta Cluster] mc-labs.php: Enable onHostRoutingPrefix for WAN cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692729 (https://phabricator.wikimedia.org/T264604) (owner: 10Krinkle)
[23:12:14] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] Simplify mc.php (7/7): Define 'wancache-main-mcrouter' unconditionally [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692728 (owner: 10Krinkle)
[23:12:53] <logmsgbot>	 !log krinkle@deploy1002 Synchronized wmf-config/CommonSettings.php: I302a71d19604a45c54c (duration: 00m 56s)
[23:12:55] <wikibugs>	 (03Merged) 10jenkins-bot: Simplify mc.php (7/7): Define 'wancache-main-mcrouter' unconditionally [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692728 (owner: 10Krinkle)
[23:12:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:15:20] <wikibugs>	 (03PS5) 10Krinkle: [Beta Cluster] mc-labs.php: Enable onHostRoutingPrefix for WAN cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692729 (https://phabricator.wikimedia.org/T264604)
[23:15:24] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] [Beta Cluster] mc-labs.php: Enable onHostRoutingPrefix for WAN cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692729 (https://phabricator.wikimedia.org/T264604) (owner: 10Krinkle)
[23:16:05] <logmsgbot>	 !log krinkle@deploy1002 Synchronized wmf-config/mc.php: I13646a5557c9 (duration: 00m 55s)
[23:16:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:16:16] <wikibugs>	 (03Merged) 10jenkins-bot: [Beta Cluster] mc-labs.php: Enable onHostRoutingPrefix for WAN cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/692729 (https://phabricator.wikimedia.org/T264604) (owner: 10Krinkle)
[23:23:37] <wikibugs>	 (03PS1) 10Legoktm: mediawiki-cache-warmup: Add api.php meta=siteinfo request to per-server list [puppet] - 10https://gerrit.wikimedia.org/r/700716 (https://phabricator.wikimedia.org/T269179)
[23:24:30] <wikibugs>	 (03CR) 10Krinkle: mediawiki-cache-warmup: Add api.php meta=siteinfo request to per-server list (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700716 (https://phabricator.wikimedia.org/T269179) (owner: 10Legoktm)
[23:25:28] <wikibugs>	 (03PS2) 10Legoktm: mediawiki-cache-warmup: Add api.php meta=siteinfo request to per-server list [puppet] - 10https://gerrit.wikimedia.org/r/700716 (https://phabricator.wikimedia.org/T269179)
[23:25:36] <wikibugs>	 (03CR) 10Legoktm: mediawiki-cache-warmup: Add api.php meta=siteinfo request to per-server list (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/700716 (https://phabricator.wikimedia.org/T269179) (owner: 10Legoktm)
[23:29:18] <wikibugs>	 (03CR) 10Krinkle: [C: 03+1] mediawiki-cache-warmup: Add api.php meta=siteinfo request to per-server list [puppet] - 10https://gerrit.wikimedia.org/r/700716 (https://phabricator.wikimedia.org/T269179) (owner: 10Legoktm)
[23:30:20] <wikibugs>	 (03CR) 10Legoktm: [C: 03+2] mediawiki-cache-warmup: Add api.php meta=siteinfo request to per-server list [puppet] - 10https://gerrit.wikimedia.org/r/700716 (https://phabricator.wikimedia.org/T269179) (owner: 10Legoktm)