[00:03:49] <brett>	 I went ahead and added it to the prefix since the other designate values were there
[11:07:01] <Krinkle>	 > Puppet is having issues on the "gerrit.wikimedia.org (208.80.154.151)" instance in project
[11:07:01] <Krinkle>	 integration in Wikimedia Cloud VPS.
[11:07:16] <Krinkle>	 Got a few of these overnight
[11:08:18] <Krinkle>	 That is either an unlikely instance name, or... that email subject/body has lost a variable and now says this for everyone
[11:08:39] <Krinkle>	 I'm guessing the latter?
[11:10:39] <taavi>	 huh
[11:10:54] <taavi>	 Krinkle: can you share the email headers? that should reveal which instance that's actually coming from
[11:13:00] <Krinkle>	 It's not consistent there either
[11:13:43] <Krinkle>	 Received: from root by gerrit.wikimedia.org with local (Exim 4.94.2)
[11:13:43] <Krinkle>	 	(envelope-from <root@wmcloud.org>)
[11:13:57] <Krinkle>	 Subject: [Cloud VPS alert][integration] Puppet failure on gerrit.wikimedia.org
[11:14:07] <Krinkle>	 From: root <root@gerrit.wikimedia.org>
[11:14:19] <Krinkle>	 Received: from integration-agent-docker-1047.integration.eqiad1.wikimedia.cloud ([172.16.3.56]:40278 helo=gerrit.wikimedia.org)
[11:14:36] <Krinkle>	 https://www.irccloud.com/pastebin/zgRdT6Zu
[11:14:42] <Krinkle>	 smtp.mailfrom=root@wmcloud.org;
[11:14:42] <Krinkle>	        dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=wikimedia.org
[11:15:12] <Krinkle>	 </dump>
[11:15:55] <Krinkle>	 Quite a few functional values there have gotten messed up too, including dmarc fail as result
[11:16:07] <Krinkle>	 Although spf fine so still delivered?
[11:16:16] <taavi>	 so integration-agent-docker-1047.integration.eqiad1.wikimedia.cloud does think it is gerrit.wikimedia.org for some reason (per `hostname -f`)
[11:16:24] <Krinkle>	 lol
[11:16:32] <Krinkle>	 Okay so a more localised issue
[11:18:07] <Krinkle>	 Although if there's a way to get the correct value maybe we don't need to include this one. So it is at least t correct/consistent in those kind of email
[11:20:20] <Krinkle>	 The email includes a diff from last puppet run on that instance saying:
[11:20:45] <Krinkle>	 --- /etc/resolv.conf	2025-02-18 14:01:53.773054774 +0000 search integration.eqiad1.wikimedia.cloud 
[11:20:45] <Krinkle>	  options timeout:1 attempts:3 ndots:1
[11:20:45] <Krinkle>	 -nameserver 172.16.5.62
[11:20:45] <Krinkle>	 +nameserver 172.20.255.1
[11:21:34] <Krinkle>	 Might have something that do with it. Also converged_environment: production initial_environment: production, looks sus
[11:26:59] <taavi>	 I'm going to try to reboot this host
[11:27:53] <taavi>	 now it's back to being integration-agent-docker-1047
[11:28:39] <Pumbpsi>	 Hei, there is a severe downtime on tool https://geohack.toolforge.org/ . Shows "404 Not found" in 9 of 10 reloads. Who can help?
[11:30:00] <taavi>	 Krinkle: yeah, that resolv.conf change is very suspiciously timed, the hostname in syslog changes just seconds after.. I think that was related to some experiment andrewbogott was doing
[12:04:20] * andrewbogott here, catching up...
[12:06:32] <andrewbogott>	 the resolv.conf change was just switching 'integration' back to the normal dns recursor after the network instability was fixed.
[12:06:59] <andrewbogott>	 It is definitely the case that resolv.conf is also where the host's fqdn is set, but... that part didn't change did it?
[12:09:20] <andrewbogott>	 soooo Krinkle I have no theory of how we got there :(  Is everything working properly now?
[12:13:36] * andrewbogott prepares to depart
[12:27:58] <wm-bot>	 !log maurelio@tools-bastion-13 tools.mabot Update mabot to latest pywkikibot-core stable 62fbca0
[12:28:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mabot/SAL
[15:31:39] <wm-bot>	 !log bsadowski1@tools-bastion-13 tools.stewardbots Restarted StewardBot/SULWatcher because of a connection loss
[15:31:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[17:50:06] <owuh>	 if i run a webservice on toolforge are ssl certificates automatically handled for me?
[17:50:43] <lucaswerkmeister>	 yes, TLS is handled by the toolforge.org proxy
[17:50:51] <lucaswerkmeister>	 you’ll just see incoming HTTP requests
[17:51:29] <owuh>	 amazing
[17:52:09] <lucaswerkmeister>	 (on the port specified by the $PORT environment variable, usually 8000 https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Other_/_generic_web_servers)
[22:39:30] <wm-bb>	 <İ> Giriş
[22:50:35] <wm-bb>	 <forzagreen> Hello, is there any directive against using Google Analytics or other third-party analytics tools? What is the recommended approach for analytics?
[23:07:08] <wm-bb>	 <bd808> My recommendation is you don’t need visitor analytics.
[23:10:46] <wm-bb>	 <bd808> The Toolforge front proxy deliberately hides the client’s ip address from your tool to increase privacy. Adding a third-party tracking system to your application actively violates the spirit of that protection.
[23:12:15] <wm-bb>	 <bd808> See also: https://phabricator.wikimedia.org/T133919
[23:39:43] <wm-bb>	 <forzagreen> Clear, thanks!
[23:39:44] <wm-bb>	 <forzagreen> Is there a way for monitoring the tool usage while respecting privacy? For example, tracking the number of requests per day or per API endpoint?
[23:51:46] <AntiComposite>	 https://toolviews.toolforge.org/ counts every request to a tool (the unique daily visitor count isn't working). You can also keep your own logs and analyze them, but without collecting personal data like IP addresses https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use
[23:58:16] <owuh>	 just want to confirm im doing this right before i actually do it so i dont blow something up: putting "<?php if(isset($_SERVER['HTTP_X_GITLAB_EVENT'])) { `git -C .. pull`; } ?>" in ~/public_html/git-pull.php will pull the contents of the git repo to ~ when i push to gitlab, right?
[23:58:43] <owuh>	 (provided i set up the webhook in gitlab ofc)