[00:04:55] (SystemdUnitFailed) resolved: rsync-aptrepo-apt2001.wikimedia.org.service on apt1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:08:25] (SystemdUnitFailed) firing: rsync-aptrepo-apt2001.wikimedia.org.service on apt1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:33:32] (03CR) 10Tim Starling: [C: 03+1] "Approved" [puppet] - 10https://gerrit.wikimedia.org/r/1010590 (https://phabricator.wikimedia.org/T357877) (owner: 10Ahmon Dancy) [00:38:41] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1010669 [00:38:43] (03CR) 10TrainBranchBot: [C: 03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1010669 (owner: 10TrainBranchBot) [00:41:54] (03PS1) 10Tim Starling: migrateBlocks.php: Fix infinite loop [core] (wmf/1.42.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1010571 [00:42:19] (03PS1) 10Tim Starling: migrateBlocks.php: Fix infinite loop [core] (wmf/1.42.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1010572 [01:01:25] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1010669 (owner: 10TrainBranchBot) [01:28:21] (03CR) 10TrainBranchBot: [C: 03+2] "Approved by tstarling@deploy2002 using scap backport" [core] (wmf/1.42.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1010571 (owner: 10Tim Starling) [01:28:24] (03CR) 10TrainBranchBot: [C: 03+2] "Approved by tstarling@deploy2002 using scap backport" [core] (wmf/1.42.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1010572 (owner: 10Tim Starling) [01:46:47] (03Merged) 10jenkins-bot: migrateBlocks.php: Fix infinite loop [core] (wmf/1.42.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1010571 (owner: 10Tim Starling) [01:49:06] (03Merged) 10jenkins-bot: migrateBlocks.php: Fix infinite loop [core] (wmf/1.42.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1010572 (owner: 10Tim Starling) [01:50:23] !log tstarling@deploy2002 Started scap: Backport for [[gerrit:1010571|migrateBlocks.php: Fix infinite loop]], [[gerrit:1010572|migrateBlocks.php: Fix infinite loop]] [01:52:44] !log tstarling@deploy2002 tstarling: Backport for [[gerrit:1010571|migrateBlocks.php: Fix infinite loop]], [[gerrit:1010572|migrateBlocks.php: Fix infinite loop]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [01:55:35] (03CR) 10Tim Starling: "I am here because the modelines failed with an error in vim 8.2. I am not some sort of human modeline validator." [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1010664 (owner: 10Tim Starling) [01:55:55] !log tstarling@deploy2002 tstarling: Continuing with sync [02:01:34] PROBLEM - Check whether ferm is active by checking the default input chain on mw1464 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm [02:02:06] PROBLEM - Check whether ferm is active by checking the default input chain on kubernetes2046 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm [02:07:05] !log tstarling@deploy2002 Finished scap: Backport for [[gerrit:1010571|migrateBlocks.php: Fix infinite loop]], [[gerrit:1010572|migrateBlocks.php: Fix infinite loop]] (duration: 16m 42s) [02:08:31] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [02:08:38] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [02:10:13] !log on mwmaint2002: running migrateBlocks.php on all wikis [02:10:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:12:02] (03PS2) 10Tim Starling: Add procps to base images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1010690 [02:17:12] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:26:49] (03PS3) 10Tim Starling: Fix broken vim modelines [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1010664 [02:31:34] RECOVERY - Check whether ferm is active by checking the default input chain on mw1464 is OK: OK ferm input default policy is set https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm [02:32:06] RECOVERY - Check whether ferm is active by checking the default input chain on kubernetes2046 is OK: OK ferm input default policy is set https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm [02:37:14] (JobUnavailable) firing: (3) Reduced availability for job ldap in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:41:04] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin1002 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [02:41:44] (SwiftTooManyMediaUploads) firing: Too many codfw mediawiki originals uploads - https://wikitech.wikimedia.org/wiki/Swift/How_To#mediawiki_originals_uploads - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=codfw - https://alerts.wikimedia.org/?q=alertname%3DSwiftTooManyMediaUploads [02:42:25] (SystemdUnitFailed) firing: httpbb_kubernetes_mw-api-ext_hourly.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:50:16] PROBLEM - Host asw1-eqsin is DOWN: PING CRITICAL - Packet loss = 100% [02:50:48] PROBLEM - OSPF status on cr2-eqsin is CRITICAL: OSPFv2: 2/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [02:50:52] PROBLEM - OSPF status on cr3-eqsin is CRITICAL: OSPFv2: 2/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [02:51:02] PROBLEM - Host ps1-604-eqsin is DOWN: PING CRITICAL - Packet loss = 100% [02:51:14] PROBLEM - Host ps1-603-eqsin is DOWN: PING CRITICAL - Packet loss = 100% [02:51:24] PROBLEM - Host mr1-eqsin IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [02:57:14] (JobUnavailable) firing: (4) Reduced availability for job ldap in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:07:56] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: CRITICAL - Host Unreachable (2403:b100:3001:9::2) [03:11:45] (SwiftTooManyMediaUploads) resolved: Too many codfw mediawiki originals uploads - https://wikitech.wikimedia.org/wiki/Swift/How_To#mediawiki_originals_uploads - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=codfw - https://alerts.wikimedia.org/?q=alertname%3DSwiftTooManyMediaUploads [03:35:30] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [03:35:37] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [03:41:04] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin1002 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [03:42:25] (SystemdUnitFailed) resolved: httpbb_kubernetes_mw-api-ext_hourly.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:49:50] RECOVERY - Router interfaces on cr2-eqord is OK: OK: host 208.80.154.198, interfaces up: 46, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [03:50:04] RECOVERY - Router interfaces on cr2-codfw is OK: OK: host 208.80.153.193, interfaces up: 122, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [03:54:07] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [03:54:13] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [04:08:25] (SystemdUnitFailed) firing: rsync-aptrepo-apt2001.wikimedia.org.service on apt1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:08:45] (03CR) 10Tim Starling: "I updated the commit message to clarify that this is a real error, not a quibble about the spec." [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1010664 (owner: 10Tim Starling) [04:34:02] (03CR) 10KartikMistry: [C: 03+2] Update cxserver to 2024-03-12-113634-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1010508 (https://phabricator.wikimedia.org/T350773) (owner: 10KartikMistry) [04:34:55] (03Merged) 10jenkins-bot: Update cxserver to 2024-03-12-113634-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1010508 (https://phabricator.wikimedia.org/T350773) (owner: 10KartikMistry) [04:37:52] !log kartik@deploy2002 helmfile [staging] START helmfile.d/services/cxserver: apply [04:38:17] !log kartik@deploy2002 helmfile [staging] DONE helmfile.d/services/cxserver: apply [04:44:52] !log kartik@deploy2002 helmfile [codfw] START helmfile.d/services/cxserver: apply [04:45:24] !log kartik@deploy2002 helmfile [codfw] DONE helmfile.d/services/cxserver: apply [04:48:52] !log kartik@deploy2002 helmfile [eqiad] START helmfile.d/services/cxserver: apply [04:49:27] !log kartik@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [04:50:27] !log Updated cxserver to 2024-03-12-113634-production (T350773, T359525) [04:50:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:50:32] T350773: Remove preq and use node fetch - https://phabricator.wikimedia.org/T350773 [04:50:33] T359525: MinT: Translation with MinT/Apertium are failing: fetch failed - https://phabricator.wikimedia.org/T359525 [06:00:04] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20240313T0600) [06:08:25] (SystemdUnitFailed) resolved: rsync-aptrepo-apt2001.wikimedia.org.service on apt1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:12:11] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [06:12:18] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [06:13:55] (SystemdUnitFailed) firing: rsync-aptrepo-apt2001.wikimedia.org.service on apt1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:17:12] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:21:45] (SwiftTooManyMediaUploads) firing: (2) Too many eqiad mediawiki originals uploads - https://wikitech.wikimedia.org/wiki/Swift/How_To#mediawiki_originals_uploads - https://alerts.wikimedia.org/?q=alertname%3DSwiftTooManyMediaUploads [06:35:24] RECOVERY - Host ps1-603-eqsin is UP: PING OK - Packet loss = 0%, RTA = 223.47 ms [06:35:24] RECOVERY - Host ps1-604-eqsin is UP: PING OK - Packet loss = 0%, RTA = 223.30 ms [06:35:26] RECOVERY - Host asw1-eqsin is UP: PING OK - Packet loss = 0%, RTA = 224.43 ms [06:35:28] RECOVERY - OSPF status on cr3-eqsin is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:35:38] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 237.18 ms [06:36:16] RECOVERY - OSPF status on cr2-eqsin is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:37:14] (JobUnavailable) firing: (3) Reduced availability for job ldap in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:39:58] RECOVERY - Host mr1-eqsin IPv6 is UP: PING OK - Packet loss = 0%, RTA = 222.96 ms [06:47:39] 10SRE-Access-Requests: Not able to access Gerrit - https://phabricator.wikimedia.org/T360006 (10cchen) 03NEW [07:00:05] Amir1 and Urbanecm: Time to do the UTC morning backport window deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20240313T0700). [07:00:05] No Gerrit patches in the queue for this window AFAICS. [07:00:40] 06SRE, 10SRE-Access-Requests: Not able to access Gerrit - https://phabricator.wikimedia.org/T360006#9625620 (10RhinosF1) [07:01:45] (SwiftTooManyMediaUploads) resolved: (2) Too many eqiad mediawiki originals uploads - https://wikitech.wikimedia.org/wiki/Swift/How_To#mediawiki_originals_uploads - https://alerts.wikimedia.org/?q=alertname%3DSwiftTooManyMediaUploads [07:08:36] (03CR) 10Volans: "And the actual solution is Id41ff6d1ce41447a315dee2886771b382eaaebdc, that is pending some additional verification/testing to ensure is no" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1010507 (owner: 10Arturo Borrero Gonzalez) [07:10:41] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [07:10:48] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [07:24:19] (03CR) 10Marostegui: [C: 03+2] admin: add rkhan to group 'restricted' (mwmaint access) [puppet] - 10https://gerrit.wikimedia.org/r/1010514 (https://phabricator.wikimedia.org/T359490) (owner: 10Dzahn) [07:24:49] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to mwmaint for rkhan / Himejijo - https://phabricator.wikimedia.org/T359490#9625647 (10Marostegui) [07:25:40] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: 14Requesting access to mwmaint for rkhan / Himejijo - 14https://phabricator.wikimedia.org/T359490#9625649 (10Marostegui) 05Open→03Resolved 14This has been done. Give it 30-45 minutes for the change to get spread across the infra. [07:35:44] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [07:35:50] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [07:46:02] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [07:46:09] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [07:49:30] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [07:49:33] OH I got it [07:49:36] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [07:49:41] I know why I was confused yesterday for the train deployment [07:50:02] all the deployments slots have shifted one hour earlier cause they are defined relatively to a USA time zone [07:50:14] but are intended for non-US consumption [07:50:15] fun [07:50:36] "when=2024-03-13 01:00 SF" [07:50:37] aka PST [07:53:33] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [07:53:39] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [07:59:36] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [07:59:43] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [07:59:47] 06SRE, 10SRE-Access-Requests, 10Gerrit: Not able to access Gerrit - https://phabricator.wikimedia.org/T360006#9625673 (10Peachey88) [08:00:04] hashar and jnuche: Deploy window MediaWiki train - Utc-0 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20240313T0800) [08:01:32] jouncebot: refresh [08:01:32] I refreshed my knowledge about deployments. [08:01:35] jouncebot: now [08:01:36] For the next 0 hour(s) and 58 minute(s): UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20240313T0800) [08:03:00] !log Moved "UTC morning backport window" and "MediaWiki train" deployment windows from PST to UTC effectively shifting them one hour later. That is due to daylight saving time kicking it at different time. With the change, the windows are now at their usual time relatively to CET [08:03:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:03:50] urbanecm: Amir1: jnuche: I have moved the backport window to 9:00 CET and the MediaWiki train at 10:00 CET [08:04:19] the deployment calendar has them defined with San Francisco timezone so they ended up one hour earlier this week [08:04:29] and that explains why I was confused yesterday [08:07:42] ack [08:20:21] 06SRE, 10SRE-Access-Requests, 10Gerrit: Not able to access Gerrit - https://phabricator.wikimedia.org/T360006#9625691 (10hashar) Please please give your login account next time which saves a bit of time figuring out which ones has to be diagnosed. Eventually I found it via the parent task on T356645#9522774... [08:20:25] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [08:20:31] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [08:29:49] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [08:29:55] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [08:30:52] (03PS1) 10JHathaway: jhathaway: update dotfiles [puppet] - 10https://gerrit.wikimedia.org/r/1010848 [08:31:53] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [08:31:59] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [08:32:49] 06SRE, 10SRE-Access-Requests, 10Gerrit: 14Not able to access Gerrit - 14https://phabricator.wikimedia.org/T360006#9625700 (10hashar) 05Open→03Resolved a:03hashar 14I have manually reactivated the account, you should be able to login now :) [08:33:23] (03CR) 10JHathaway: [C: 03+2] jhathaway: update dotfiles [puppet] - 10https://gerrit.wikimedia.org/r/1010848 (owner: 10JHathaway) [08:34:33] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [08:34:39] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [08:38:02] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [08:38:09] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [08:45:06] !log Disable GTID on eqiad es5 master T358199 [08:45:09] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [08:45:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:45:12] T358199: Database pre-switchover tasks March 2024 - https://phabricator.wikimedia.org/T358199 [08:45:16] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [08:47:50] !log Disable GTID on eqiad es4 master T358199 [08:47:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:53:03] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [08:53:09] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [08:57:13] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [08:57:19] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [09:00:04] hashar and jnuche: Deploy window MediaWiki train - Utc-0 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20240313T0900) [09:01:11] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [09:01:18] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [09:04:21] I will run the train in a few minutes [09:04:26] I am finishing filing a bug report [09:15:01] 06SRE, 10SRE-Access-Requests, 10Gerrit: 14Not able to access Gerrit - 14https://phabricator.wikimedia.org/T360006#9625758 (10hashar) 14To address the root cause of the Gerrit user not being reactivated when the accounts is unblocked from Wikitech, I have reopened a past task and posted my analysis on it... [09:15:04] ok train [09:16:04] !log isaranto@deploy2002 helmfile [ml-serve-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [09:16:23] (03PS1) 10TrainBranchBot: group1 wikis to 1.42.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010849 (https://phabricator.wikimedia.org/T354440) [09:16:26] (03CR) 10TrainBranchBot: [C: 03+2] group1 wikis to 1.42.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010849 (https://phabricator.wikimedia.org/T354440) (owner: 10TrainBranchBot) [09:17:10] !log isaranto@deploy2002 helmfile [ml-serve-eqiad] 'sync' command on namespace 'ores-legacy' for release 'main' . [09:17:14] (03Merged) 10jenkins-bot: group1 wikis to 1.42.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010849 (https://phabricator.wikimedia.org/T354440) (owner: 10TrainBranchBot) [09:23:01] (03CR) 10Ilias Sarantopoulos: httpbb: add ores-legacy tests (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1010245 (https://phabricator.wikimedia.org/T359871) (owner: 10Ilias Sarantopoulos) [09:30:13] !log hashar@deploy2002 rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.22 refs T354440 [09:30:19] T354440: 1.42.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T354440 [09:31:06] * hashar whistles [09:32:15] (MediaWikiLatencyExceeded) firing: p75 latency high: eqiad mw-parsoid (k8s) 1.139s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-parsoid - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [09:37:15] (MediaWikiLatencyExceeded) resolved: p75 latency high: eqiad mw-parsoid (k8s) 1.139s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-parsoid - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [09:42:32] !log hashar@deploy2002 Synchronized php: group1 wikis to 1.42.0-wmf.22 refs T354440 (duration: 12m 18s) [09:42:39] T354440: 1.42.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T354440 [09:44:06] did the train just hit Wikidata? [09:44:13] yese [09:44:20] common.js stopped loading [09:44:22] for several users [09:45:36] filing a bug now [09:45:37] site common.js or their user common.js? [09:45:39] ok [09:46:14] user common.js [09:46:37] T360014 extremely bare bones, will add more [09:46:38] T360014: common.js not loading on Wikidata - https://phabricator.wikimedia.org/T360014 [09:47:36] ok, can confirm I think [09:49:00] i boldly set that to UBN [09:49:50] what projects should i add to that task? [10:00:17] should the train be rolled back? [10:01:53] o/ [10:03:52] IMHO yes [10:04:06] the error seems easy enough to reproduce locally, so I don’t think we need to keep the site broken to investigate it [10:04:11] let me commute to my office (which is 30 seconds away) [10:04:15] and user JS is a pretty important feature [10:09:25] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [10:09:31] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [10:10:17] Jhs: Lucas_WMDE: there was another bug in that code yesterday which got fixed by matmarex [10:10:24] i am rolling back [10:11:24] I guess the issue can be reproduced on a group0 wiki meanwhile (eg mediawiki.org or test.wikipedia.org ) [10:14:10] (SystemdUnitFailed) firing: rsync-aptrepo-apt2001.wikimedia.org.service on apt1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:15:12] Jhs: thank you for the bug report and flagging it for attention! :) [10:15:28] hashar, np :) [10:15:55] it takes a bit of time to rollback nowadays unfortunately, but the old versions will be rolled in a few minutes [10:16:27] I think we end up doing a full deployment rather than rolling back [10:17:12] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:17:59] fwiw i can reproduce on mediawiki.org too [10:19:00] my guess is that is less disruptive there [10:19:05] compared to commons/wikidata [10:21:42] yeah [10:26:36] !log hashar@deploy2002 rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.42.0-wmf.22" - T354440 T360014 [10:26:42] T354440: 1.42.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T354440 [10:26:42] T360014: User JavaScript not loading on 1.42.0-wmf.22 (Wikidata, Commons, other group1 wikis) - https://phabricator.wikimedia.org/T360014 [10:37:14] (JobUnavailable) firing: (2) Reduced availability for job ldap in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:39:14] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [10:39:20] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [10:43:03] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [10:43:10] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [10:47:25] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [10:47:32] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [10:51:54] lunch & [11:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20240313T1100) [11:03:55] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:04:02] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [11:12:28] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:12:35] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [11:15:32] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:15:39] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [11:17:45] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:17:52] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [11:22:47] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:22:54] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [11:26:59] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:27:06] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [11:27:59] (03PS1) 10Hashar: wikitech: allow unblocking inactive accounts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010853 (https://phabricator.wikimedia.org/T307558) [11:28:35] (03CR) 10Hashar: "That is a minor follow up to the otherwise excellent *wikitech: Update Gerrit blocking logic* change ( https://gerrit.wikimedia.org/r/c/op" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010853 (https://phabricator.wikimedia.org/T307558) (owner: 10Hashar) [11:31:24] (03PS1) 10Hashar: Revert "group1 wikis to 1.42.0-wmf.22" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010854 (https://phabricator.wikimedia.org/T354440) [11:31:32] (03CR) 10Hashar: [C: 03+2] Revert "group1 wikis to 1.42.0-wmf.22" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010854 (https://phabricator.wikimedia.org/T354440) (owner: 10Hashar) [11:31:40] (03Merged) 10jenkins-bot: Revert "group1 wikis to 1.42.0-wmf.22" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010854 (https://phabricator.wikimedia.org/T354440) (owner: 10Hashar) [11:33:50] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:33:56] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [11:38:35] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:38:42] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [11:41:53] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:41:59] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [11:43:27] (03PS1) 10Dreamy Jazz: Use iterator_to_array when calling ::assertCount [extensions/MediaModeration] (wmf/1.42.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1010575 (https://phabricator.wikimedia.org/T360017) [11:43:43] (03PS1) 10Dreamy Jazz: Use iterator_to_array when calling ::assertCount [extensions/MediaModeration] (wmf/1.42.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1010576 (https://phabricator.wikimedia.org/T360017) [11:44:24] (03PS1) 10Dreamy Jazz: Use wgCanonicalServer instead of wgSitename in intro text of email [extensions/MediaModeration] (wmf/1.42.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1010577 (https://phabricator.wikimedia.org/T359979) [11:44:32] (03PS1) 10Dreamy Jazz: Use wgCanonicalServer instead of wgSitename in intro text of email [extensions/MediaModeration] (wmf/1.42.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1010578 (https://phabricator.wikimedia.org/T359979) [11:44:48] (03PS2) 10Dreamy Jazz: Use wgCanonicalServer instead of wgSitename in intro text of email [extensions/MediaModeration] (wmf/1.42.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1010577 (https://phabricator.wikimedia.org/T359979) [11:44:56] (03PS2) 10Dreamy Jazz: Use wgCanonicalServer instead of wgSitename in intro text of email [extensions/MediaModeration] (wmf/1.42.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1010578 (https://phabricator.wikimedia.org/T359979) [11:44:58] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:45:05] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [11:45:20] (03CR) 10Klausman: [C: 03+1] Remove unecessary regexes from Lift Wing metrics [grafana-grizzly] - 10https://gerrit.wikimedia.org/r/1010193 (owner: 10Elukey) [11:45:44] (03CR) 10Klausman: [C: 03+1] Add Dragonfly 2p2 cache to ml-serve k8s [puppet] - 10https://gerrit.wikimedia.org/r/1010535 (https://phabricator.wikimedia.org/T359416) (owner: 10Elukey) [11:47:33] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:47:39] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [11:47:53] (03Abandoned) 10Arturo Borrero Gonzalez: spicerack: avoid pyyaml 5.4.1 [software/spicerack] - 10https://gerrit.wikimedia.org/r/1010507 (owner: 10Arturo Borrero Gonzalez) [11:50:38] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:50:45] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [11:52:16] (03PS7) 10Majavah: P:toolforge::proxy: drop grid engine dynamicproxy support [puppet] - 10https://gerrit.wikimedia.org/r/1010503 (https://phabricator.wikimedia.org/T314664) [11:52:24] (03PS7) 10Majavah: dynamicproxy: cleanup after removing toolforge support [puppet] - 10https://gerrit.wikimedia.org/r/1010504 (https://phabricator.wikimedia.org/T314664) [11:52:32] (03PS8) 10Majavah: dynamicproxy: add support for per-project zones [puppet] - 10https://gerrit.wikimedia.org/r/1010505 (https://phabricator.wikimedia.org/T342398) [11:52:40] (03PS10) 10Majavah: dynamicproxy: allow specifying different certs for each zone [puppet] - 10https://gerrit.wikimedia.org/r/1010506 (https://phabricator.wikimedia.org/T342398) [11:52:48] (03PS6) 10Majavah: dynamicproxy: add spec test for API [puppet] - 10https://gerrit.wikimedia.org/r/1010509 [11:52:49] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:52:56] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [11:53:28] (03CR) 10Daniel Kinzler: "I guess this was lost, is it still relevant?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/937090 (owner: 10Alexandros Kosiaris) [11:53:44] (03PS11) 10Majavah: dynamicproxy: allow specifying different certs for each zone [puppet] - 10https://gerrit.wikimedia.org/r/1010506 (https://phabricator.wikimedia.org/T342398) [11:53:52] (03PS7) 10Majavah: dynamicproxy: add spec test for API [puppet] - 10https://gerrit.wikimedia.org/r/1010509 [11:55:10] (03PS12) 10Majavah: dynamicproxy: allow specifying different certs for each zone [puppet] - 10https://gerrit.wikimedia.org/r/1010506 (https://phabricator.wikimedia.org/T342398) [11:55:18] (03PS8) 10Majavah: dynamicproxy: add spec test for API [puppet] - 10https://gerrit.wikimedia.org/r/1010509 [11:55:34] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [11:55:41] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [12:07:17] (03PS13) 10Majavah: dynamicproxy: allow specifying different certs for each zone [puppet] - 10https://gerrit.wikimedia.org/r/1010506 (https://phabricator.wikimedia.org/T342398) [12:07:25] (03PS9) 10Majavah: dynamicproxy: add spec test for API [puppet] - 10https://gerrit.wikimedia.org/r/1010509 [12:08:34] (03PS2) 10KartikMistry: Enable Content/Section translation on some Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010226 (https://phabricator.wikimedia.org/T353510) [12:13:55] (SystemdUnitFailed) resolved: rsync-aptrepo-apt2001.wikimedia.org.service on apt1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:14:39] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [12:14:46] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [12:17:25] (SystemdUnitFailed) firing: rsync-aptrepo-apt2001.wikimedia.org.service on apt1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:24:23] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [12:24:30] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [12:27:00] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [12:27:07] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [12:42:14] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [12:42:21] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [12:49:34] (03PS1) 10Majavah: Remove labtesttoolsadmin [dns] - 10https://gerrit.wikimedia.org/r/1010884 [12:51:15] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [12:51:21] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [12:52:07] (03PS4) 10GergesShamon: Set ShowRollbackConfirmationDefaultUserOptions on arwiki to false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010645 (https://phabricator.wikimedia.org/T355213) [12:52:40] Hi [12:54:46] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [12:54:53] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [12:57:12] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [12:57:19] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [13:00:05] RoanKattouw, Lucas_WMDE, Urbanecm, awight, and TheresNoTime: I, the Bot under the Fountain, call upon thee, The Deployer, to do UTC afternoon backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20240313T1300). [13:00:05] mo_abualruz, _Gerges, and Dreamy_Jazz: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:13] \o [13:00:16] I can self-deploy [13:00:22] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [13:00:29] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [13:00:32] Hi [13:01:38] mo_abualruz: You around? [13:02:03] (03PS1) 10Majavah: hieradata: remove SGE nodes from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1010887 [13:02:03] Hello I am around I want to deploy the ticket please [13:02:42] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [13:02:45] Dreamy_Jazz: I think you can go ahead [13:02:48] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [13:02:52] * Lucas_WMDE hasn’t had lunch yet and hopes someone else can also deploy the other changes [13:03:03] (03CR) 10Dom Walden: [C: 03+1] Switch block schema to read-new/write-both mode [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1006180 (https://phabricator.wikimedia.org/T355034) (owner: 10Tim Starling) [13:03:16] I can deploy [13:03:33] (03CR) 10Majavah: [V: 03+1] "PCC SUCCESS (DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet5-compiler-node/1636/console" [puppet] - 10https://gerrit.wikimedia.org/r/1010887 (owner: 10Majavah) [13:03:54] I will deploy 1010698 [13:04:02] Okay [13:04:09] Do you want to go ahead now then [13:04:28] Thanks 🥰 [13:04:35] yea [13:04:42] (03CR) 10TrainBranchBot: [C: 03+2] "Approved by mabualruz@deploy2002 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010698 (https://phabricator.wikimedia.org/T359183) (owner: 10Jdlrobson) [13:05:25] (03Merged) 10jenkins-bot: Disable night mode on history pages [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010698 (https://phabricator.wikimedia.org/T359183) (owner: 10Jdlrobson) [13:06:07] !log mabualruz@deploy2002 Started scap: Backport for [[gerrit:1010698|Disable night mode on history pages (T359183)]] [13:06:24] T359183: Exclude non-functional pages from night mode - https://phabricator.wikimedia.org/T359183 [13:08:22] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [13:08:29] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [13:08:33] !log mabualruz@deploy2002 jdlrobson and mabualruz: Backport for [[gerrit:1010698|Disable night mode on history pages (T359183)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [13:09:25] I'm busy right now, please postpone my deploy for half an hour [13:09:29] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1010645/4 [13:09:31] Checking teh changes with mwdebug [13:09:51] Gerges: Okay. I will proceed with my backports before your change then. [13:10:10] (after this change has been deployed ofc) [13:11:16] All good proceeding [13:11:19] !log mabualruz@deploy2002 jdlrobson and mabualruz: Continuing with sync [13:12:09] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1010676 [13:12:15] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1010677 [13:12:22] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [13:12:29] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [13:13:01] (03CR) 10Arturo Borrero Gonzalez: [C: 03+1] "LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/1010887 (owner: 10Majavah) [13:14:28] (03PS1) 10Majavah: Archive repository [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/1010891 (https://phabricator.wikimedia.org/T359935) [13:14:35] (03CR) 10Majavah: [V: 03+1 C: 03+2] hieradata: remove SGE nodes from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1010887 (owner: 10Majavah) [13:14:38] (03CR) 10CI reject: [V: 04-1] Archive repository [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/1010891 (https://phabricator.wikimedia.org/T359935) (owner: 10Majavah) [13:15:10] (03CR) 10Majavah: [V: 03+2 C: 03+2] Archive repository [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/1010891 (https://phabricator.wikimedia.org/T359935) (owner: 10Majavah) [13:15:34] (03CR) 10CI reject: [V: 04-1] Archive repository [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/1010891 (https://phabricator.wikimedia.org/T359935) (owner: 10Majavah) [13:17:09] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [13:17:22] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [13:21:55] !log mabualruz@deploy2002 Finished scap: Backport for [[gerrit:1010698|Disable night mode on history pages (T359183)]] (duration: 15m 48s) [13:22:00] T359183: Exclude non-functional pages from night mode - https://phabricator.wikimedia.org/T359183 [13:22:17] (03CR) 10TrainBranchBot: [C: 03+2] "Approved by dreamyjazz@deploy2002 using scap backport" [extensions/MediaModeration] (wmf/1.42.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1010575 (https://phabricator.wikimedia.org/T360017) (owner: 10Dreamy Jazz) [13:22:18] (03CR) 10TrainBranchBot: [C: 03+2] "Approved by dreamyjazz@deploy2002 using scap backport" [extensions/MediaModeration] (wmf/1.42.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1010576 (https://phabricator.wikimedia.org/T360017) (owner: 10Dreamy Jazz) [13:24:25] Hi [13:24:43] (03Merged) 10jenkins-bot: Use iterator_to_array when calling ::assertCount [extensions/MediaModeration] (wmf/1.42.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1010575 (https://phabricator.wikimedia.org/T360017) (owner: 10Dreamy Jazz) [13:24:49] (03Merged) 10jenkins-bot: Use iterator_to_array when calling ::assertCount [extensions/MediaModeration] (wmf/1.42.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1010576 (https://phabricator.wikimedia.org/T360017) (owner: 10Dreamy Jazz) [13:25:04] thanks all the back port of 1010698 is done and changes are checked [13:25:11] 👍 [13:25:17] !log dreamyjazz@deploy2002 Started scap: Backport for [[gerrit:1010575|Use iterator_to_array when calling ::assertCount (T360017)]], [[gerrit:1010576|Use iterator_to_array when calling ::assertCount (T360017)]] [13:25:22] T360017: MediaModeration CI failing with "Passing an argument of type Generator for the $haystack parameter is deprecated." - https://phabricator.wikimedia.org/T360017 [13:26:10] Gerges: Are you back now? [13:26:14] Yes [13:26:28] Okay. I'll wait until these backports are done and then proceed with your change. [13:26:35] (03PS8) 10Majavah: P:toolforge::proxy: drop grid engine dynamicproxy support [puppet] - 10https://gerrit.wikimedia.org/r/1010503 (https://phabricator.wikimedia.org/T314664) [13:26:36] (03PS1) 10Majavah: Remove Toolforge grid engine [puppet] - 10https://gerrit.wikimedia.org/r/1010892 (https://phabricator.wikimedia.org/T314664) [13:27:35] !log dreamyjazz@deploy2002 dreamyjazz: Backport for [[gerrit:1010575|Use iterator_to_array when calling ::assertCount (T360017)]], [[gerrit:1010576|Use iterator_to_array when calling ::assertCount (T360017)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [13:27:38] !log dreamyjazz@deploy2002 dreamyjazz: Continuing with sync [13:28:19] (03CR) 10Majavah: [V: 03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet5-compiler-node/1637/co" [puppet] - 10https://gerrit.wikimedia.org/r/1010892 (https://phabricator.wikimedia.org/T314664) (owner: 10Majavah) [13:29:47] !log Disable GTID on eqiad s8 master T358199 [13:29:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:29:51] T358199: Database pre-switchover tasks March 2024 - https://phabricator.wikimedia.org/T358199 [13:30:27] 06SRE, 06Data-Persistence, 06Infrastructure-Foundations: Integrate dbctl IP changes as part of VLAN changes. - https://phabricator.wikimedia.org/T360029 (10Marostegui) 03NEW [13:31:57] !log Disable GTID on eqiad s7 master T358199 [13:32:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:35:05] !log Disable GTID on eqiad s6 master T358199 [13:35:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:35:22] T358199: Database pre-switchover tasks March 2024 - https://phabricator.wikimedia.org/T358199 [13:36:46] !log Disable GTID on eqiad s5 master T358199 [13:36:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:36:54] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [13:37:01] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [13:38:18] !log dreamyjazz@deploy2002 Finished scap: Backport for [[gerrit:1010575|Use iterator_to_array when calling ::assertCount (T360017)]], [[gerrit:1010576|Use iterator_to_array when calling ::assertCount (T360017)]] (duration: 13m 01s) [13:38:26] T360017: MediaModeration CI failing with "Passing an argument of type Generator for the $haystack parameter is deprecated." - https://phabricator.wikimedia.org/T360017 [13:39:09] !log Disable GTID on eqiad s4 master T358199 [13:39:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:39:31] (03CR) 10Elukey: [V: 03+1 C: 03+2] Add Dragonfly 2p2 cache to ml-serve k8s [puppet] - 10https://gerrit.wikimedia.org/r/1010535 (https://phabricator.wikimedia.org/T359416) (owner: 10Elukey) [13:40:08] (03CR) 10TrainBranchBot: [C: 03+2] "Approved by dreamyjazz@deploy2002 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010645 (https://phabricator.wikimedia.org/T355213) (owner: 10GergesShamon) [13:40:31] !log Disable GTID on eqiad s3 master T358199 [13:40:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:40:45] T358199: Database pre-switchover tasks March 2024 - https://phabricator.wikimedia.org/T358199 [13:41:27] (03Merged) 10jenkins-bot: Set ShowRollbackConfirmationDefaultUserOptions on arwiki to false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1010645 (https://phabricator.wikimedia.org/T355213) (owner: 10GergesShamon) [13:41:50] !log dreamyjazz@deploy2002 Started scap: Backport for [[gerrit:1010645|Set ShowRollbackConfirmationDefaultUserOptions on arwiki to false (T355213)]] [13:41:54] T355213: Set ShowRollbackConfirmationDefaultUserOptions on arwiki to true - https://phabricator.wikimedia.org/T355213 [13:43:06] !log Disable GTID on eqiad s2 master T358199 [13:43:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:44:09] !log dreamyjazz@deploy2002 dreamyjazz and gergesshamon: Backport for [[gerrit:1010645|Set ShowRollbackConfirmationDefaultUserOptions on arwiki to false (T355213)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [13:44:43] Gerges: Please test [13:44:58] Ok [13:45:18] !log jclark@cumin1002 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1246.eqiad.wmnet [13:45:18] !log Disable GTID on eqiad s1 master T358199 [13:45:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:45:36] !log jclark@cumin1002 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1246.eqiad.wmnet [13:46:06] !log jclark@cumin1002 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1246.eqiad.wmnet [13:46:38] !log jclark@cumin1002 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1246.eqiad.wmnet [13:46:47] !log jclark@cumin1002 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1246.eqiad.wmnet [13:46:49] Dreamy_Jazz: done [13:46:52] Thanks. [13:46:54] !log jclark@cumin1002 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts db1246.eqiad.wmnet [13:46:55] !log dreamyjazz@deploy2002 dreamyjazz and gergesshamon: Continuing with sync [13:47:34] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [13:47:41] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [13:48:58] Dreamy_Jazz: Are there any changes published on the main server? [13:49:09] It should be done soon. [13:49:11] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: hw troubleshooting: Unidentified for db1246.eqiad.wmnet - https://phabricator.wikimedia.org/T359940#9626616 (10Marostegui) For what is worth, this error is also present on the HW logs, even though it is a month old, it might be an indication of something el... [13:49:37] Okey [13:51:18] Thanks [13:51:22] !log @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [13:51:35] !log @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [13:52:18] (03CR) 10Herron: [C: 03+1] Remove unecessary regexes from Lift Wing metrics [grafana-grizzly] - 10https://gerrit.wikimedia.org/r/1010193 (owner: 10Elukey) [13:52:47] !log Stopping MediaModeration scanning scripts for backport [13:52:50]