[08:05:06] !log creating deployment-logstash05 and configure it like 04, looks like elasticsearch does not like clusters with only one host T283013 [08:05:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:05:09] T283013: Migrate beta cluster to ELK7 - https://phabricator.wikimedia.org/T283013 [09:07:17] looks like there are some mediawiki logs flowing on beta cluster elk7 setup! https://logstash-beta.wmcloud.org/app/discover [10:07:27] 10Beta-Cluster-Infrastructure, 10User-Majavah: Migrate beta cluster to ELK7 - https://phabricator.wikimedia.org/T283013 (10Majavah) https://logstash-beta.wmcloud.org is now receiving events from mediawiki and others. Next steps are importing dashboards from somewhere (not sure where, betas ELK5 is fully broken... [11:59:42] 10Beta-Cluster-Infrastructure: deployment-logstash03 puppet errors - https://phabricator.wikimedia.org/T276521 (10Majavah) 05Open→03Declined This machine will be decommissioned shortly (T283013) [11:59:45] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Jessie Deprecation): Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch) - https://phabricator.wikimedia.org/T238707 (10Majavah) [12:00:16] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10cloud-services-team (Kanban): Puppet failing on deployment-logstash03.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T280324 (10Majavah) 05Open→03Declined This machine will be decommissioned shortly (T283013) [12:00:49] 10Beta-Cluster-Infrastructure, 10SRE: deployment-logstash03: UDP listener died EADDRINUSE, logstash port conflict with rsyslogd - https://phabricator.wikimedia.org/T241481 (10Majavah) 05Open→03Declined This machine will be decommissioned shortly (T283013) [12:00:53] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Jessie Deprecation): Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch) - https://phabricator.wikimedia.org/T238707 (10Majavah) [12:02:28] 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash, 10observability, 10User-Majavah: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10Majavah) a:05herron→03Majavah I'm hoping this was fixed for good with the ELK7 upgrade (T283013) [12:03:09] 10Beta-Cluster-Infrastructure, 10Puppet: MIssing hiera settings for deployment-parsoid11.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T259533 (10Majavah) 05Open→03Declined This VM was removed at some point in the past. [12:05:06] 10Beta-Cluster-Infrastructure: Request access to deployment-prep and beta-cluster logstash - https://phabricator.wikimedia.org/T231684 (10Majavah) Hi @xSavitar, is this still wanted? [12:05:34] 10Beta-Cluster-Infrastructure: Packages broken on deployment-mediawiki-07 (libpcre3) - https://phabricator.wikimedia.org/T223801 (10Majavah) 05Open→03Declined Machine was deleted at some point [12:22:22] 10Beta-Cluster-Infrastructure, 10User-Majavah: Migrate beta cluster to ELK7 - https://phabricator.wikimedia.org/T283013 (10Majavah) New issue: `/var/log/logstash/logstash-json.log` keeps filling up rather quickly with various events, it runs the logstash nodes frequently out of disk space on the root partition... [12:31:24] 10Beta-Cluster-Infrastructure, 10User-Ladsgroup: image sizes not displayed on beta - https://phabricator.wikimedia.org/T285125 (10Majavah) @dancy Hi, do you imagine the recent `scap` python3 migration might be causing these? [13:44:46] !log remove deployment-deploy02 T278689 [13:44:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:44:49] T278689: Upgrade deployment-prep deploy* hosts to Buster - https://phabricator.wikimedia.org/T278689 [15:15:17] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic: Rename deployment-cache-(text|upload)0x to deployment-cp0x - https://phabricator.wikimedia.org/T280393 (10Majavah) One more issue: given cloud vps does not have per-role hiera keys, we need to rely on instance prefix based hiera (which is different for text an... [17:41:45] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap, 10User-Ladsgroup: image sizes not displayed on beta - https://phabricator.wikimedia.org/T285125 (10Majavah) https://logstash-beta.wmcloud.org (works now, if not clean up `/var/log/logstash/logstash-json.log` and restart `logstash.service` o... [18:25:12] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap, 10User-Ladsgroup: image sizes not displayed on beta - https://phabricator.wikimedia.org/T285125 (10Reedy) Scrap just calls a MW maintenance script to make them… [19:29:10] (03PS1) 10Ostrzyciel: Remove MediaUploader's dependency on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/700436 (https://phabricator.wikimedia.org/T275157)