[00:39:26] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/921313 [00:39:32] (03CR) 10TrainBranchBot: [C: 03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/921313 (owner: 10TrainBranchBot) [00:55:31] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/921313 (owner: 10TrainBranchBot) [00:58:00] (NodeTextfileStale) firing: Stale textfile for bast2003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [02:06:32] (JobUnavailable) firing: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:11:32] (JobUnavailable) firing: (2) Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:26:32] (JobUnavailable) resolved: (2) Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:50:01] (NodeTextfileStale) firing: (2) Stale textfile for lists1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [04:58:00] (NodeTextfileStale) firing: Stale textfile for bast2003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [06:50:01] (NodeTextfileStale) firing: (2) Stale textfile for lists1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [06:55:37] (LogstashKafkaConsumerLag) firing: Too many messages in kafka logging - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-codfw&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [07:01:59] (03CR) 10Jelto: [C: 03+2] "merging this and running httpbb tests afterwards:" [puppet] - 10https://gerrit.wikimedia.org/r/918424 (https://phabricator.wikimedia.org/T336217) (owner: 10Jelto) [07:05:37] (LogstashKafkaConsumerLag) resolved: Too many messages in kafka logging - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-codfw&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [07:12:37] (LogstashKafkaConsumerLag) firing: Too many messages in kafka logging - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-codfw&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [07:17:37] (03PS1) 10Jelto: httpbb: fix http status for miscweb annualreport [puppet] - 10https://gerrit.wikimedia.org/r/921661 (https://phabricator.wikimedia.org/T336217) [07:22:33] (03CR) 10Jelto: [C: 03+2] httpbb: fix http status for miscweb annualreport [puppet] - 10https://gerrit.wikimedia.org/r/921661 (https://phabricator.wikimedia.org/T336217) (owner: 10Jelto) [07:22:37] (LogstashKafkaConsumerLag) resolved: Too many messages in kafka logging - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-codfw&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [07:28:03] 10SRE, 10ops-codfw: Degraded RAID on backup2010 - https://phabricator.wikimedia.org/T337174 (10ops-monitoring-bot) [07:40:16] !log jelto@deploy1002 helmfile [staging] START helmfile.d/services/miscweb: apply [07:41:08] !log jelto@deploy1002 helmfile [staging] DONE helmfile.d/services/miscweb: apply [07:42:42] !log jelto@deploy1002 helmfile [codfw] START helmfile.d/services/miscweb: apply [07:43:17] !log jelto@deploy1002 helmfile [codfw] DONE helmfile.d/services/miscweb: apply [07:44:33] !log jelto@deploy1002 helmfile [eqiad] START helmfile.d/services/miscweb: apply [07:45:06] !log jelto@deploy1002 helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [08:13:31] (03PS1) 10Alexandros Kosiaris: Git template: Clean up git commit template message [deployment-charts] - 10https://gerrit.wikimedia.org/r/921668 [08:35:55] 10SRE, 10ops-codfw, 10Data-Persistence-Backup: Degraded RAID on backup2010 - https://phabricator.wikimedia.org/T337174 (10Peachey88) [08:58:00] (NodeTextfileStale) firing: Stale textfile for bast2003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:11:37] (03CR) 10Lydia Pintscher: [C: 04-1] "We're unfortunately not ready yet for this from the product side. The API needs to mature a bit more on Wikidata and then we need to see i" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/921612 (https://phabricator.wikimedia.org/T337141) (owner: 10Simon04) [09:37:02] (03PS3) 10Lucas Werkmeister: Restart Kubernetes webservices more cleanly [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) [10:37:14] (03PS4) 10Lucas Werkmeister: Restart Kubernetes webservices more cleanly [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) [10:38:12] (03CR) 10CI reject: [V: 04-1] Restart Kubernetes webservices more cleanly [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) (owner: 10Lucas Werkmeister) [10:38:43] (03CR) 10Lucas Werkmeister: Restart Kubernetes webservices more cleanly (031 comment) [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) (owner: 10Lucas Werkmeister) [10:39:05] (03PS5) 10Lucas Werkmeister: Restart Kubernetes webservices more cleanly [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) [10:43:18] (03CR) 10Majavah: [C: 04-1] "+1 to the idea, a few comments inside." [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) (owner: 10Lucas Werkmeister) [10:48:27] (03PS6) 10Lucas Werkmeister: Restart Kubernetes webservices more cleanly [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) [10:48:35] (03CR) 10Lucas Werkmeister: Restart Kubernetes webservices more cleanly (033 comments) [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) (owner: 10Lucas Werkmeister) [10:49:25] (03CR) 10CI reject: [V: 04-1] Restart Kubernetes webservices more cleanly [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) (owner: 10Lucas Werkmeister) [10:49:55] (03PS7) 10Lucas Werkmeister: Restart Kubernetes webservices more cleanly [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) [10:50:01] (NodeTextfileStale) firing: (2) Stale textfile for lists1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [10:50:13] (03CR) 10Lucas Werkmeister: Restart Kubernetes webservices more cleanly (031 comment) [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) (owner: 10Lucas Werkmeister) [10:51:05] (03CR) 10Majavah: Restart Kubernetes webservices more cleanly (031 comment) [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) (owner: 10Lucas Werkmeister) [10:51:13] (03PS8) 10Lucas Werkmeister: Restart Kubernetes webservices more cleanly [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) [10:52:46] (03CR) 10CI reject: [V: 04-1] Restart Kubernetes webservices more cleanly [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) (owner: 10Lucas Werkmeister) [10:56:13] (03PS9) 10Lucas Werkmeister: Restart Kubernetes webservices more cleanly [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) [10:56:20] (03Abandoned) 10Robertsky: change wikimaniawiki logo to 2023 version. T337044 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/921372 (owner: 10Robertsky) [10:56:51] (03PS2) 10Robertsky: going through the tox as stated in the readme T337044 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/921610 [10:57:41] (03CR) 10CI reject: [V: 04-1] Restart Kubernetes webservices more cleanly [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) (owner: 10Lucas Werkmeister) [10:58:23] (03PS10) 10Lucas Werkmeister: Restart Kubernetes webservices more cleanly [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) [11:11:44] (03PS1) 10Giuseppe Lavagetto: profile::configmaster: dump a json data structure of the pools [puppet] - 10https://gerrit.wikimedia.org/r/921692 (https://phabricator.wikimedia.org/T330705) [11:12:47] (03CR) 10CI reject: [V: 04-1] profile::configmaster: dump a json data structure of the pools [puppet] - 10https://gerrit.wikimedia.org/r/921692 (https://phabricator.wikimedia.org/T330705) (owner: 10Giuseppe Lavagetto) [11:13:28] (03CR) 10Lucas Werkmeister: Restart Kubernetes webservices more cleanly (031 comment) [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/921620 (https://phabricator.wikimedia.org/T337182) (owner: 10Lucas Werkmeister) [11:17:32] (03PS2) 10Giuseppe Lavagetto: profile::configmaster: dump a json data structure of the pools [puppet] - 10https://gerrit.wikimedia.org/r/921692 (https://phabricator.wikimedia.org/T330705) [11:34:48] (03CR) 10FNegri: [C: 03+1] "LGTM, Arturo spotted it must also be removed from Netbox: https://netbox.wikimedia.org/ipam/ip-addresses/4792/" [dns] - 10https://gerrit.wikimedia.org/r/907136 (https://phabricator.wikimedia.org/T333477) (owner: 10Majavah) [11:35:05] (03PS1) 10Effie Mouzeli: ipoid: Create iPoid chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/921700 (https://phabricator.wikimedia.org/T336163) [11:36:15] (03CR) 10CI reject: [V: 04-1] ipoid: Create iPoid chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/921700 (https://phabricator.wikimedia.org/T336163) (owner: 10Effie Mouzeli) [11:37:36] (03PS1) 10Matthias Mullie: [WikibaseMediaInfo] Add 'main subject of' property [mediawiki-config] - 10https://gerrit.wikimedia.org/r/921701 [11:43:39] (03PS1) 10Effie Mouzeli: admin_ng: Add iPoid namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/921704 (https://phabricator.wikimedia.org/T336163) [11:44:50] (03CR) 10CI reject: [V: 04-1] admin_ng: Add iPoid namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/921704 (https://phabricator.wikimedia.org/T336163) (owner: 10Effie Mouzeli) [12:41:51] (03PS1) 10Effie Mouzeli: ipoid: add helmfile.d config [deployment-charts] - 10https://gerrit.wikimedia.org/r/921707 (https://phabricator.wikimedia.org/T336163) [12:42:37] (03CR) 10CI reject: [V: 04-1] ipoid: add helmfile.d config [deployment-charts] - 10https://gerrit.wikimedia.org/r/921707 (https://phabricator.wikimedia.org/T336163) (owner: 10Effie Mouzeli) [12:58:00] (NodeTextfileStale) firing: Stale textfile for bast2003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [13:22:23] (03CR) 10Jelto: [C: 04-1] "review in line" [cookbooks] - 10https://gerrit.wikimedia.org/r/919057 (https://phabricator.wikimedia.org/T336490) (owner: 10Jelto) [13:43:25] (03CR) 10Superpes15: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/921610 (owner: 10Robertsky) [14:06:32] (JobUnavailable) firing: (2) Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:16:32] (JobUnavailable) resolved: (2) Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:50:01] (NodeTextfileStale) firing: (2) Stale textfile for lists1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [15:04:07] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [15:16:29] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [15:53:18] (KubernetesAPILatency) firing: High Kubernetes API latency (PATCH nodes) on k8s-staging@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/000000435?var-site=codfw&var-cluster=k8s-staging - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [15:58:18] (KubernetesAPILatency) resolved: High Kubernetes API latency (PATCH nodes) on k8s-staging@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/000000435?var-site=codfw&var-cluster=k8s-staging - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [16:24:44] (HaproxyUnavailable) firing: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [16:29:44] (HaproxyUnavailable) resolved: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [16:58:00] (NodeTextfileStale) firing: Stale textfile for bast2003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [17:23:03] (ProbeDown) firing: (2) Service centrallog2002:6514 has failed probes (tcp_rsyslog_receiver_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#centrallog2002:6514 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:28:03] (ProbeDown) resolved: (2) Service centrallog2002:6514 has failed probes (tcp_rsyslog_receiver_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#centrallog2002:6514 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:50:01] (NodeTextfileStale) firing: (2) Stale textfile for lists1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [19:58:19] 10SRE, 10SRE-tools, 10Infrastructure-Foundations, 10netops: Setup zero touch provisioning (ZTP) for network devices - https://phabricator.wikimedia.org/T336485 (10ayounsi) From previous tests, DNS resolution is not available at this stage on Junos devices, so better to go directly with the IP. [20:58:00] (NodeTextfileStale) firing: Stale textfile for bast2003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [22:50:01] (NodeTextfileStale) firing: (2) Stale textfile for lists1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [23:41:45] (03PS1) 10Superpes15: [ruwiki] Add 'abusefilter log/view private' to ArbCom [mediawiki-config] - 10https://gerrit.wikimedia.org/r/921764 (https://phabricator.wikimedia.org/T336625) [23:43:23] (03PS2) 10Superpes15: [ruwiki] Add 'abusefilter log/view private' flags to ArbCom [mediawiki-config] - 10https://gerrit.wikimedia.org/r/921764 (https://phabricator.wikimedia.org/T336625) [23:59:01] (03PS1) 10Superpes15: [kaawiki] Enable SandboxLink extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/921765 (https://phabricator.wikimedia.org/T336648)