[09:46:39] 10serviceops, 10SRE: Too many mw versions caused out of disk space on ~30 mw hosts - https://phabricator.wikimedia.org/T355117 (10jnuche) 05Open→03Resolved a:03jnuche After my recent changes to the staging code, failures during the presync shouldn't cause old versions to pile up anymore. Last night presy... [10:34:54] 10serviceops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [11:17:53] 10serviceops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [11:18:21] 10serviceops, 10Content-Transform-Team, 10Parsoid, 10Wikimedia-production-error: TypeError: Argument 4 passed to Wikimedia\Parsoid\Utils\Title::__construct() must be of the type string, null given, called in /srv/mediawiki/php-1.42.0-wmf.15/vendor/wikimedia/parsoid... - https://phabricator.wikimedia.org/T356024 [15:25:21] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin2002 for host mw2366.codfw.wmnet with OS bullseye [15:26:00] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin2002 for host mw2368.codfw.wmnet with OS bullseye [15:26:20] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin2002 for host mw2370.codfw.wmnet with OS bullseye [15:28:01] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1440.eqiad.wmnet with OS bullseye [15:28:39] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1457.eqiad.wmnet with OS bullseye [15:28:45] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1482.eqiad.wmnet with OS bullseye [15:28:49] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1459.eqiad.wmnet with OS bullseye [15:28:54] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1466.eqiad.wmnet with OS bullseye [16:03:25] 10serviceops, 10MW-on-K8s, 10SRE: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1459.eqiad.wmnet with OS bullseye completed: - mw1459 (**PASS**) - Downtimed on Icinga/Alertma... [16:08:19] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin2002 for host mw2368.codfw.wmnet with OS bullseye completed: - mw2368 (**PASS**) - Downtimed on... [16:09:25] 10serviceops, 10MW-on-K8s, 10SRE: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1482.eqiad.wmnet with OS bullseye completed: - mw1482 (**PASS**) - Downtimed on Icinga/Alertma... [16:13:18] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin2002 for host mw2370.codfw.wmnet with OS bullseye completed: - mw2370 (**PASS**) - Downtimed on... [16:15:07] 10serviceops, 10MW-on-K8s, 10SRE: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1440.eqiad.wmnet with OS bullseye completed: - mw1440 (**PASS**) - Downtimed on Icinga/Alertma... [16:18:53] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin2002 for host mw2366.codfw.wmnet with OS bullseye completed: - mw2366 (**PASS**) - Downtimed on... [16:21:27] 10serviceops, 10MW-on-K8s, 10SRE: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1457.eqiad.wmnet with OS bullseye completed: - mw1457 (**PASS**) - Downtimed on Icinga/Alertma... [16:25:23] 10serviceops, 10MW-on-K8s, 10SRE: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1466.eqiad.wmnet with OS bullseye completed: - mw1466 (**PASS**) - Downtimed on Icinga/Alertma... [17:17:26] Hi, i am trying to debug page summary timeouts on mobileapps/staging k8s for T339865 [17:17:49] Can we change the log levels temporarily only on staging? [17:18:22] We can revert after: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/994209 [17:22:27] cc hnowlan ^ [17:34:59] 10serviceops, 10Content-Transform-Team, 10Parsoid, 10Wikimedia-production-error: TypeError: Argument 4 passed to Wikimedia\Parsoid\Utils\Title::__construct() must be of the type string, null given, called in /srv/mediawiki/php-1.42.0-wmf.15/vendor/wikimedia/parsoid... - https://phabricator.wikimedia.org/T356024 [20:03:08] 10serviceops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [20:31:41] 10serviceops, 10Content-Transform-Team, 10Parsoid, 10Wikimedia-production-error: TypeError: Argument 4 passed to Wikimedia\Parsoid\Utils\Title::__construct() must be of the type string, null given, called in /srv/mediawiki/php-1.42.0-wmf.15/vendor/wikimedia/parsoid... - https://phabricator.wikimedia.org/T356024 [20:58:28] 10serviceops, 10Content-Transform-Team, 10Parsoid, 10Wikimedia-production-error: TypeError: Argument 4 passed to Wikimedia\Parsoid\Utils\Title::__construct() must be of the type string, null given, called in /srv/mediawiki/php-1.42.0-wmf.15/vendor/wikimedia/parsoid... - https://phabricator.wikimedia.org/T356024 [22:06:16] 10serviceops, 10Content-Transform-Team, 10Parsoid, 10Wikimedia-production-error: TypeError: Argument 4 passed to Wikimedia\Parsoid\Utils\Title::__construct() must be of the type string, null given, called in /srv/mediawiki/php-1.42.0-wmf.15/vendor/wikimedia/parsoid... - https://phabricator.wikimedia.org/T356024 [23:20:47] 10serviceops, 10Content-Transform-Team, 10Parsoid, 10Wikimedia-production-error: TypeError: Argument 4 passed to Wikimedia\Parsoid\Utils\Title::__construct() must be of the type string, null given, called in /srv/mediawiki/php-1.42.0-wmf.15/vendor/wikimedia/parsoid... - https://phabricator.wikimedia.org/T356024