[07:25:41] What do i have to do to make the webservice work? [07:25:49] https://bawolff.toolforge.org/ is giving me an error that there is no webservice [07:25:55] but when i run "webservice status" [07:26:02] I get "Your webservice of type php7.4 is running on backend kubernetes" [07:28:13] Are there logs somewhere i can read? [07:28:34] I'm not sure how to debug this situation. Assume i am toolforge n00b ;) [07:31:06] try `webservice --backend=kubernetes logs` from https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#View_web_service_logs [07:31:30] ty :) [07:32:36] It looks like it didn't like my .lighttpd.conf file. I guess mimetype.assign += ( ".txt" => "text/plain; charset=utf-8" ) is conflicting with some defaults that changed since i wrote the file [07:33:51] All fixed now :) [08:35:54] Wrong certificate for https://pm20-search.toolforge.org. Apparently, the delivered certificate is for toolforge.org. So the connection is rejected by my institutes firewall. https://pm20-filmlink.toolforge.org with a specific certificate for that site works well. [08:40:51] https://pm20-search.toolforge.org/ works for me, with a certificate for *.toolforge.org [08:41:17] (and https://pm20-filmlink.toolforge.org gives me the exact same certificate) [08:44:32] sounds kind of like the firewall is parsing only cn and not san fields, but if it did that half the internet would be broken [08:44:44] That's strange - it now works for me again, too. [08:48:47] Thanks for the hint re. cn and san field. The "firewall" is indeed some intransparent Cisco "Secure Endpoint" / "Umbrella" thing. If the error hits again, I know what to get whitelisted. [08:51:11] yes, please complain to your firewall admin, we serve correct and valid certificates [09:06:30] Thanks again for the quick help and explanation - I'll introduce the function today in the Hamburg Wikipedia Kontor, and it would have been too bad to have my own notebook blocking the access without me knowing what is going on [09:41:28] hi! my bot at toolforge has problems to use the db replica since >24h. has anything changed recently? [09:46:24] lustiger_seth: hi, what kind of problems? [09:48:06] DBD::mysql::st execute failed: Lost connection to MySQL server during query [09:48:36] at "SELECT\x{a}\x{9}\x{9}\x{9}\x{9}\x{9}`rc_timestamp` AS `timestamp`,\x{a}\x{9}\x{9}\x{9}\x{9}\x{9}`actor_name` "... [09:50:33] the strange thing is that the query is unchanged for months now. but since yesterday the query seems to fail. [09:50:55] how often is that happening? which wiki and replica type? what does your code look like? [09:51:58] my code: https://gitlab.com/wp-seth/camelbot/-/blob/master/lib/CamelBot.pm?ref_type=heads#L1460-1484 [09:53:41] how often: since 2023-11-27 13:30 UTC mor than 150 times [09:55:14] which wiki: dewiki [09:55:24] replica type: ? [09:57:19] correction: since 2023-11-29 13:30 UTC more than 150 times [09:57:47] so >24h is not correct. it's ~20h now. [10:03:29] ok, that roughly matches with some maintenance that would have caused existing connections to the wiki replicas to be reset. the most likely cause of the issue you're seeing is that your code is still trying to use the now-closed connection instead of retrying and creating a new one [10:12:58] how can i force to retry a new one? (the whole process is restarted every time the db conn fails.) [10:45:42] strangely sometimes the queries seem to work and sometimes not. because the bot still does its work: https://de.wikipedia.org/wiki/Special:Contributions/CamelBot. however the jobs fail every 5 to 20 minutes. [10:55:46] ok, I might have realized what's going on. the connection timeouts seem to be much lower than usual by accident. one moment [11:11:35] lustiger_seth: just applied a fix, can you report if it happens again after this? [11:13:07] ok, thanks. let's see, if i still get cronjob failure mails every few minutes. ;-) [11:26:37] no mails anymore. seems to be fixed. thanks! :-) [11:27:39] great, sorry about that [16:19:43] Heya, I feel dumb because "SHOW INDEX IN revision;" on "frwiki_p" returns an empty set. I do not know why. The only specificity stated in the wikitechdoc is that "the revision and logging tables do not have indexes on user columns". Any idea? [16:35:01] Lofhi: the "revision" you see is a view and not a table. I don't think it is actually possible for an end user of the Wiki Replicas databases to enumerate the indexes on the underlying tables. It would be great if someone proves me wrong about that though. [16:37:41] :-( [16:39:13] How can I be sure that I am using any indexes when using BETWEEN on... "rev_timestamp" by example? [16:43:33] For a more detailed context, I am working on community project for frwiki. I need to retrieve revisions by looking for a #hashtag in the revisions comments. I think I will limit the range of the search to 3 months max, but even with this limit the DBMS still needs 30s to execute the query. The LIKE operator is clearly not helping. [16:43:57] Tried https://sql-optimizer.toolforge.org/ but I think I can't do more. [16:51:28] Lofhi: https://wikitech.wikimedia.org/wiki/News/Actor_storage_changes_on_the_Wiki_Replicas#special-views [16:51:53] The alternative I can imagine is to just cache the resulting set on server-side since the revisions are just historical data, but we still have the delay for the first request and until invalidation [16:52:07] Thanks HolidayRhino, I am already using comment_revision ! [16:53:37] But I use comment_revision for the set join... I will try the opposite. [16:54:45] MariaDB [frwiki_p]> SELECT * [16:54:46]     -> FROM comment_revision AS com [16:54:46]     -> INNER JOIN revision AS rev ON com.comment_id = rev.rev_comment_id [16:54:47]     -> WHERE rev_timestamp BETWEEN '20220301000000' AND '20221230000000' [16:54:47]     -> AND com.comment_text LIKE '%test%'; [16:54:49] 8925 rows in set (36.923 sec) [16:55:10] Can you paste that into a pastebin? [16:55:14] yep [16:55:23] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/%2B/production/modules/profile/templates/wmcs/db/wikireplicas/index-conf.yaml should be all of the non-standard indexes built for the Wiki Replicas. [16:56:19] https://pastebin.com/EVrhCqVa [16:56:45] Thanks bd808 [16:56:58] a double wildcard LIKE is always going to be a table scan [16:58:10] If you can do anything more to narrow the number of scanned rows that's the only way to speed up such a query [16:58:15] Yea, but I don't think I can use a trim or something else :-( [16:58:42] probably not. the concept of edit comment hashtags is basically cursed [16:58:53] I was betting on narrowing with the namespaces from "page" table, but it didn't help [17:00:38] The problem is [17:00:53] It's scanning over a million rows so that's going to take time [17:01:05] I also had the idea to add a functional constraint (specify a list of users for the revisions to avoid a global lookup) [17:01:25] But it seems that the doc is saying that there is no index on the user column [17:01:35] So it would not even help [17:01:48] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Tables_for_revision_or_logging_queries_involving_user_names_and_IDs [17:03:08] And you tried `revision_userindex` instead of the normal `revision`? [17:03:24] Nope [17:04:04] "user" is really "actor" in the modern tables. but rev_actor should be indexed in the revision_userindex view [17:05:12] And I think I just said something dumb, query will always be quickier with an equal/in operator and a list of users [17:05:41] will try revision_userindex and narrowing with contributors names [17:06:19] It is ok to misspeak :) [17:06:23] I have another idea, but I think mediawiki devs will kill me (using rev tags hahaha) [17:07:47] https://www.mediawiki.org/wiki/Manual:Change_tag_table [17:09:00] !log deployment-prep added Dreamy_Jazz to members [17:09:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [17:09:23] eeeh, maybe not a dumb idea "If a tag is no longer wanted, you may delete it. This removes the definition of the tag from the database, and removes the tag from all revisions and log entries to which it is applied." [17:09:42] Lofhi: those are pretty commonly used. any tool that uses OAuth authentication when making an edit for example applies an "OAuth CID: nnnnn" tag to the revision. [17:09:43] it should work for editathons [17:10:33] yep but [17:10:41] If your use case is tracking participation in editathons, https://outreachdashboard.wmflabs.org/ might be interesting to you. [17:10:55] these tags are handled by MediaWiki, no? I mean, it is not something specified by the editor? [17:11:18] Can it replace a #hashtag in the revision comment? [17:11:44] Oh, wil bookmark outreachdashboard, thanks [17:12:42] For context, Wikipedia in French organizes monthly events: to clean up articles, to neutralize them... And they list these changes by adding a hashtag to the review comment. [17:14:31] Somehow, I am conceptualizing a tool merging outreachdashboard and fountain.https://fountain.toolforge.org [17:17:24] Lofhi: I know there have been a number of discussions about making some API that would allow bots and humans to add tags to their edits, but I don't know of an actual implementation. I think they all do come from backend PHP code at this time either in MediaWiki core itself or added by extensions as in the OAuth case. [17:19:14] A tool with an OAuth grant could be used as a "hack" for this, but the tool would have to be used to make the actual edits which probably limits the usefulness of that hack. [17:22:19] Do you think that a MediaWiki:Gadget could add a tag on a revision? [17:22:25] Didn't check [17:23:21] So, I tested, with a global search, it executes in ~15s. Specifying an user reduces it to ~5s. https://quarry.wmcloud.org/query/78402# [17:23:37] (looking for a keywork on 3 months) [17:23:57] Guess they will just wait, I just don't want to stress the replicas somehow [17:24:25] A gadget would only have javascript access to the MediaWiki API, so no I don't think that would help. [17:24:28] The OAuth hack could be cool, but it adds complexity to the project [17:24:39] Meh, I am unlucky [17:26:32] bd808: I thought that's what api.php?action=tag was for? Am I misunderstanding the use of that API module? [17:27:01] It exists, yea, but does not the endpoint need managechangetags right or something? [17:27:52] Didn't look in MediaWiki codebase for months, already crying if I need to check CommonSettings & co [17:27:57] stw: you are teaching me, and yeah it looks like the thing. [17:28:15] Lofhi: oh, possibly. I don't see anything documented on the API help about that though [17:28:45] https://www.mediawiki.org/w/api.php?action=help&modules=managetags looks like the bit to create one too? [17:28:52] oh [17:29:03] I think [17:29:06] It is lying to us [17:29:17] "This module requires read rights. This module requires write rights." [17:29:21] Surely this is not enough [17:29:40] and user should have/in a usergroup with managechangetags right? [17:29:52] be in* [17:30:47] But a gadget adding tags for monthly wikiproject and a webapp working on revisions with a new temporary specific tag could be an attractive idea [17:32:07] You could also think about things like a bot that found edits via comment hashtag once per hour and added tags to them or something like that. [17:33:16] We have a lot of options yea, but then it means handling a database for the webapp... I try to conceptualize something simple... So If someday I got erased by a car, the community can take back the project easily [17:33:17] :D [17:34:03] Working with the API/SQL and replicas gives some stability to the project, adding a database for the project add a bit of complexity [17:35:02] my user did get a permissiondenied error trying to create a new tab on testwiki. it looks like the managechangetags right is only granted to sysop there. [17:35:10] I may sound demanding, but there are already abandoned gadgets on frwiki, robots running while we've lost contact with their creators... Perl scripts running in non-reproducible environments. [17:35:39] bd808: I think we can skip this step and assume that an admin would manage the tags [17:35:51] But the changetags & applychangetags rights are granted to all users I think [17:36:51] Lofhi: tool rot is very real. One of the goals of Toolforge is to make it easier for the community to collaborate on tools and adopt tools who's maintainers have wandered away from the projects. [17:37:38] https://wikitech.wikimedia.org/wiki/User:BryanDavis/Developing_community_norms_for_critical_bots_and_tools is an essay/talk I have given a number of times about the general problem and some suggestions to make things better. [17:37:53] Will read, thanks [17:40:34] stw: Thanks for jumping it to point out the API endpoint. I had somehow found old discussion on enwiki of making such an API but not the API itself. [17:40:55] np :) [17:42:40] hi there, we've been having some mail receipt issues for the wikipedia library (hosted on a vps instance). Our django app is reporting that emails are getting sent (it's configured to use mx-out03), but we've been getting scattered reports of no delivery since september. Most messages are getting delivered just fine, but enough are not getting to inboxes that it's become a workflow issue. [17:42:41] https://phabricator.wikimedia.org/T347512 [17:42:41] Because our app is sending directly to mx-out and it says everything is fine, I'm at a bit of a loss. It seems like my next step would be to use localhost as an mta so I could at least have some smtp logs. Does that seem like a reasonable next step? I'm happy to troubleshoot anything that I have access to, and I'm happy to take any advice or pointers you may have for me, including exim config. I've worked with postfix quite [17:42:41] a bit, but never had to touch exim before. [17:46:24] JSherman: You would probably be helped a bit by getting a Cloud VPS root to take a look at logs on mx-out03, especially if you have known date, sender, and recipient data for the emails that didn't show up. [17:46:56] I don't have time right this second, but I could look later today [17:51:44] bd808: that would be awesome, I will send some info to you on slack to avoid publishing it to the irc archives [17:52:53] JSherman: +1. We can make a security ticket too if needed (to protect PII) [18:01:31] hotnews: a frwiki admin is not against using a tag for my problem, it could help greatly! [18:33:53] Jdlrobson: https://skins.wmcloud.org/#/skin/eveskin is coming up with an error about reaching enwp with preview [18:53:51] !log metricsinfra no longer send quarry alerts to cloud services team [18:53:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Metricsinfra/SAL [19:26:43] tired, thanks for help, good day/night!! [19:54:30] !log admin reimaged cloudrabbit100[23] after https://gerrit.wikimedia.org/r/c/operations/puppet/+/979127. I didn't reimage 1001 because that will require rebuilding the whole cluster but I did remove the related packages. [19:54:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:27:29] !log devtools - phabricator-bullseye - attempting to fix mariadb/mysql server, apt-get remove mariadb-server, running puppet, debugging why it wont start [20:27:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [20:29:41] !log devtools - phabricator-bullseye - running 'mariadb-install-db' [20:29:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [20:33:20] !log devtools - phabricator-bullseye - running 'mariadb-secure-installation' interactive script - this fixed mysql shell which previously exited with "bash: /nonexistent: No such file or directory" [20:33:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [20:44:41] !log admin generating to application credentials for the tests that run on tf-infra-test [20:44:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL