[00:04:08] log tools deploy a version of the php3.7 web image that includes the python2 package with tag :testing T287421 [00:04:08] T287421: Latest Toolforge docker images don't always have python2 installed at all - https://phabricator.wikimedia.org/T287421 [00:04:21] !log tools deploy a version of the php3.7 web image that includes the python2 package with tag :testing T287421 [00:04:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:05:19] !log tools.parliamentdiagram changed deployment to use the image docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web:testing T287421 [00:05:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.parliamentdiagram/SAL [00:05:32] That fixes the problem, of course [00:05:39] I'll put up a patch for review [00:53:02] bd808: a few days ago someone asked if data put in toolhub now will be carried over to the official launch. my assumption was yes [01:44:55] @harej ugh. [01:45:04] I can tell them "no" [01:47:21] I really hoped that naming it “toolhub-demo” would be a strong hint that this is not a long term anything. I can certainly try to write some export/import system, but I’m not sure when I will find time for that. [01:50:18] Or we could not add data to it. [01:52:41] I think there are fewer than 10 total added entries [02:09:40] I am happy that folks are playing with it. And I think editing is a good thing for folks to test. I’m just not super excited about migrating data to the eventual prod system as an automatic action. [02:13:15] As a last resort I can manually copy from demo toolhub to prod toolhub by hand; there isn't much data. [02:14:56] I think scripts would also work. This wouldn't copy revision history, whatever that's worth. [08:35:16] my packagist-mirror1.packagist-mirror.eqiad.wmflabs VM is failing puppet because the jessie-wikimedia/openstack-mitaka-jessie component 404s now [08:35:31] but my VM is stretch, why is it trying to load jessie packages? [08:35:46] can I just delete /etc/apt/sources.list.d/openstack-mitaka-jessie.list ? [08:36:44] I imagine yes [08:37:20] * majavah wonders where did that come from [08:43:51] * legoktm does [09:01:16] it worked :) [09:01:26] I moved it to openstack-mitaka-jessie.list.bak just in case it was actually important [11:02:10] !log migr sudo mv openstack-mitaka-jessie.list openstack-mitaka-jessie.list.bak [11:02:11] wm-bot: Unknown project "migr" [11:03:07] This was supposed to be about misc-01.wikibase-registry.eqiad.wmflabs 🤔 [11:08:24] but it worked insofar as that `sudo run-puppet-agent` exited with 0. The server itself is still 502 but that's seemingly unrelated [11:15:52] o_O [11:15:55] !log migr [11:15:56] wm-bot: Missing project or message? Expected !log [11:16:07] did you run dologmsg on a Toolforge host or on a Cloud VPS system? [11:17:02] MichaelG_WMDE: the project name should be wikibase-registry, based on the DNS name so the !log line should be !log wikibase-registry [11:17:10] !log wikibase-registry on libcanada-01.wikibase-registry.eqiad.wmflabs to remove outdated and now unavailable apt source [11:17:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [11:17:31] !log wikibase-registry `sudo mv openstack-mitaka-jessie.list openstack-mitaka-jessie.list.bak` on libcanada-01.wikibase-registry.eqiad.wmflabs to remove outdated and now unavailable apt source [11:17:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [11:17:37] oh, the log line itself came from the bot, ACK [11:17:47] and it works [11:17:59] mutante: thank you :) [11:18:20] yw [11:21:21] Lucas_WMDE: on the cloud vps system itself. In my last attempt above, I just did it manually and that seemed to work better [11:21:37] ok, I don’t know much about the dologmsg command on cloud VPS [11:21:50] I’m more confident in the Toolforge version [11:21:53] looking back, I should probably have added the ticket ID as well [11:21:59] Lucas_WMDE: [11:22:23] Lucas_WMDE: there is no dologmsg on cloud VPS, I copied the one from toolforge [11:22:49] didn't do exactly what I wanted, but reading the code helped in figuring it out :) [11:23:12] ok ^^ [11:23:29] yeah it assumes the $USER is the project name because that’s how Toolforge works [11:23:41] I think there might be a patch somewhere on Gerrit for a cloud VPS version but it didn’t go anywhere [11:23:50] (there’s a file somewhere in /etc that could provide the project name) [11:24:12] /etc/wmcs-project [11:26:10] /etc/wmflabs-project seems to also work [11:27:29] https://wikitech.wikimedia.org/wiki/Help:Labs_labs_labs, tl;dr don't use "labs" anywhere if at all possible, too confusing term [11:28:04] 👍 [12:25:33] Hi, is node12 available for web in toolforge? https://gerrit.wikimedia.org/r/c/wikidata/query-builder/+/705353/13#message-8f2791bb0cb331ae2a9d289958fde43194deba9c [12:25:43] docker-registry.tools.wmflabs.org/toolforge-node12-sssd-web:latest failed [12:26:08] (node12 should come with bullseye) [12:28:33] Amir1: no, we're waiting for bullseye official release before publishing the bullseye images :/ [12:28:50] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Container_images lists all available images [12:29:05] noted [12:29:21] thanks [12:29:42] I have tested building some toolforge images from pre-releases locally and they seem to work fine, so I don't expect it to take long after official release [12:31:11] awesome [12:47:49] !log wikidata-dev wb-reconcile Edited the mediawiki user’s crontab to disable all automatic updates (T287459, T286292) [12:47:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [12:47:54] T286292: The ansible-role setup fills up the disk - https://phabricator.wikimedia.org/T286292 [12:47:54] T287459: wikidata-dev instances causing git "Internal error during upload-pack" every 5 minutes - https://phabricator.wikimedia.org/T287459 [15:15:58] !log admin "rm /etc/apt/sources.list.d/openstack-mitaka-jessie.list" cloud-wide [15:16:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:33:01] addshore, Amir1, wbregistry-01.wikibase-registry has a 100% full drive so everything is failing there. Is that a sign of an abandoned instance/project or should I try to clean up some disk space? [15:34:09] (the disk usage is all in /var/lib/docker) [15:34:34] tarrow, Lucas_WMDE, same question :) [15:40:05] !log wikidata-dev wb-reconcile deleted instance [15:40:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [15:40:23] andrewbogott: I think we still want that instance to some extent [15:40:29] I remember some discussion about it a few days ago [15:40:35] not sure if there’s a task for it [15:45:59] MichaelG_WMDE was working on a different instance in the project https://phabricator.wikimedia.org/T287452 [15:48:20] yes, I looked into the misc-01 and libcanada-01 instances. But we should probably take care of that as well [15:57:13] andrewbogott: sorry I was afk, it seems it's already handled. Ping me if I can be of any service [16:00:55] MichaelG_WMDE: maybe you can just delete some docker images? [16:01:43] andrewbogott: yes, I'm looking at that right now. It seems there are some old docker container still running and stuck [16:01:53] sounds good, thank you for looking! [16:02:07] Probably you were getting 'puppet is broken' emails about this for the last while [16:02:48] yes, I did get emails about those other instances, but not for the one that we actually care about :/ [16:03:43] hm, maybe a full disk meant it was too broken to email :) [16:05:10] 😭 [16:14:35] Ok, we're back to 11% free space. Not great but enough breathing space for now, so that we can look into this more throughly [16:14:42] * MichaelG_WMDE forgot about SAL, will fix [16:20:14] !log tools built new php images with python2 on board T287421 [16:20:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:20:18] T287421: Latest Toolforge docker images don't always have python2 installed at all - https://phabricator.wikimedia.org/T287421 [16:22:02] !log wikibase-registry `sudo docker kill 36a5b061acfe` stopping duplicate old docker container T287492 [16:22:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [16:22:36] !log wikibase-registry `sudo docker kill 9e0309ff1120` stopping duplicate old docker container T287492 [16:22:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [16:22:43] !log wikibase-registry `sudo docker kill dcd0b3d98c51` stopping duplicate old docker container T287492 [16:22:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [16:23:15] !log wikibase-registry `sudo docker container prune` removing stopped containers to free up space T287492 [16:23:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [16:23:23] !log wikibase-registry `sudo docker image prune` removing dangling docker images to free up space T287492 [16:23:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [16:31:56] !log wikidata-dev wb-reconcile created instance, installed MediaWiki (using in-VM MariaDB, not the cool new Cloud VPS database thing) [16:31:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [19:19:56] !log wikicite create project with default quota + 200 gigabytes storage [19:19:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikicite/SAL [20:02:38] balloons: I think that's record time for project approval, thank you! [20:03:17] harej, you are most welcome. We [20:03:43] harej, we'd love it if you tried out trove. And if you're curious, I can grant more trove quota to help out [20:15:20] Trove looks like exactly the kind of thing I want [20:22:35] I tried creating a DB with a 120 GB volume and I was told I couldn't do more than 80. [20:43:19] harej, it's a bit obscure, trove quotas are separate from project quotas [20:43:49] although if your quota is 80 that suggests that someone already adjusted it... [20:45:55] balloons, maybe we should swap some of wikicite's project quota for db quota? ^^^ [20:47:44] andrewbogott: that sounds reasonable to me. I think @harej asked for the space to hold his planned db, but obviously he can shout if that's a wrong assumption. [20:48:06] I'll make the switch [20:48:34] The main space requirement is for the database [20:50:22] Thanks Andrew [20:50:25] !log wikicite reduced project quota to 100Gb and increased Trove volume quota to 120Gb [20:50:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikicite/SAL [20:50:46] harej, there might be some stealth 80Gb limit in there too so lmk if that helped at all [20:52:05] !log admin draining VMs off of cloudvirt1012 so we can replace the battery for T286748 [20:52:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:52:09] T286748: hw troubleshooting: Raid battery stuck in recharging for cloudvirt1012.eqiad.wmnet - https://phabricator.wikimedia.org/T286748 [21:04:15] Looks like I am still running into the 80 GB limit [21:11:21] ok, I'm investigating [21:19:50] heh, the trove config has this setting: "max_accepted_volume_size = 80" [21:19:55] I wonder if that could have anything to do with this? [21:30:36] harej, this should help https://gerrit.wikimedia.org/r/c/operations/puppet/+/708353 [21:32:31] !log admin putting cloudvirt1012 back into service T286748 [21:32:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:32:35] T286748: hw troubleshooting: Raid battery stuck in recharging for cloudvirt1012.eqiad.wmnet - https://phabricator.wikimedia.org/T286748