[09:21:57] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10collaboration-services, and 3 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10Jelto) [09:22:36] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10collaboration-services, and 3 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10Jelto) p:05Mediumā†’03High [09:22:52] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10collaboration-services, and 3 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10Jelto) I deployed the change above which adds this two lines to the `gitlab.rb` file: ` gitlab_rails['omniauth_auto_link_user'] =... [10:01:02] Iā€™m getting my shutters mounted, and I had to share it (one of my favorites so far) [10:01:13] https://usercontent.irccloud-cdn.com/file/4kxiYThA/IMG_2855.JPG [10:02:48] hahhaha [10:03:23] recently someone from Dell called me "zarzuela" on an email thread [10:03:30] <3 [10:06:43] :D hehehe [11:58:24] (SystemdUnitFailed) firing: uwsgi-puppetdb-microservice.service Failed on puppetdb1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:17:48] ^this is me will be fixed soon [12:23:24] (SystemdUnitFailed) resolved: uwsgi-puppetdb-microservice.service Failed on puppetdb1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:25:04] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin1001 for host sretest1002.eqiad.wmnet wit... [12:43:25] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin1001 for host sretest1002.eqiad.wmnet with OS... [12:43:58] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin1001 for host sretest1002.eqiad.wmnet wit... [12:56:52] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin1001 for host sretest1002.eqiad.wmnet with OS... [13:00:20] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin1001 for host sretest1002.eqiad.wmnet wit... [13:12:52] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin1001 for host sretest1002.eqiad.wmnet with OS... [13:13:54] 10SRE-tools, 10Infrastructure-Foundations: Cumin fails to get uptime in debian installer - https://phabricator.wikimedia.org/T342345 (10jbond) p:05Triageā†’03Medium [13:17:21] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin1001 for host sretest1002.eqiad.wmnet wit... [13:47:07] slyngs are you around to help w/makevm cookbook? The bookworm installer keeps booting into interactive mode...tried many times so far (ref https://phabricator.wikimedia.org/T341705 ) [13:47:39] if there's a manual way to provision a VM I'd be OK w/that too [13:59:19] 10SRE-tools, 10Infrastructure-Foundations: Cumin fails to get uptime in debian installer - https://phabricator.wikimedia.org/T342345 (10jbond) > typing go completes the installation correctly correction typing go and then doing a reboot via install_console allows things to complete [14:01:38] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin1001 for host sretest1002.eqiad.wmnet with OS... [14:09:28] Reimage is having the same problem...maybe an issue with partman recipe? I assume that's in puppet somewhere? [14:10:15] inflatador: modules/install_server/files/autoinstall/netboot.cfg [14:10:22] check here? [14:13:52] Thanks sukhe . Any idea if this needs to be explicitly set for every host pattern? The first 2 VMs I provisioned don't match any of these patterns but did eventually build [14:14:39] inflatador: yep, as long as the pattern matches host. so for zookeeper, it should be zookeeper* here but I only see test? [14:15:10] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin1001 for host sretest1002.eqiad.wmnet wit... [14:15:31] inflatador: but yeah I think try adding it here and then running the cookbook [14:16:15] sukhe ack...worth a shot, right? ;) [14:16:23] inflatador: most certainly :) [14:16:36] IIRC I had a similar issue once and this was it [14:16:43] and in any case, the entry is required here [14:17:33] I guess there's no way to build a VM from an image? [14:17:59] not sure... [14:18:18] oh well, I'll get the puppet patch up and worry about that later [14:25:35] 10SRE-tools, 10Infrastructure-Foundations: Cumin fails to get uptime in debian installer - https://phabricator.wikimedia.org/T342345 (10jbond) I have rebuild sretest1002 again and things now work. I have build things a few times (logs in cumin1001:~cookbook-testing/logs) and so far it seems a bit random when... [14:53:55] inflatador: sorry i missed this earlier but i see sukhe got you sorted :) cheers sukhe [14:54:42] jbond np, also just updated https://wikitech.wikimedia.org/wiki/Ganeti#Create_partman_config [14:55:23] thanks <3 [15:06:58] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin1001 for host sretest1002.eqiad.wmnet with OS... [15:07:33] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin1001 for host sretest1002.eqiad.wmnet wit... [15:20:35] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin1001 for host sretest1002.eqiad.wmnet with OS... [15:46:33] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin1001 for host sretest1002.eqiad.wmnet wit... [15:54:06] 10SRE-tools, 10Infrastructure-Foundations: Cumin fails to get uptime in debian installer - https://phabricator.wikimedia.org/T342345 (10jbond) i wonder iff this is because im running this over and over again in quick succession and possibly fingerprint checking is getting in the way as i get the following from... [16:21:46] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin1001 for host sretest1002.eqiad.wmnet with OS... [16:38:32] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin1001 for host sretest1002.eqiad.wmnet wit... [16:52:26] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin1001 for host sretest1002.eqiad.wmnet with OS... [16:54:47] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review: improvements to firmware upgrade cookbook - https://phabricator.wikimedia.org/T329722 (10BTullis) I don't know if you want a new ticket for this, but I'm seeing errors from the `upgrade-firmware` cookbook when run against some hosts with the old... [17:10:02] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin1001 for host sretest1002.eqiad.wmnet wit... [17:20:39] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review: improvements to firmware upgrade cookbook - https://phabricator.wikimedia.org/T329722 (10Papaul) @BTullis the firmware cookbook will failed if the IDRAC version is too old. You need to get it to a minimum version for the API to work. [17:41:16] 10SRE-tools, 10netbox, 10Infrastructure-Foundations, 10Puppet-Infrastructure, and 3 others: update systems to use new puppetdb instance - https://phabricator.wikimedia.org/T342214 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin1001 for host sretest1002.eqiad.wmnet with OS...