Page MenuHomePhabricator

[Epic][X1] Move Wikibase's entity usage database tables to the extension database cluster
Open, Needs TriagePublic

Description

In order to improve the database space usage, wbc_entity_usage tables of all wikis are to be moved to the "extension" database cluster (x1).

Steps involved:
Part 1: Code changes

  • Adjust Wikibase logic to have a separate database configuration for the entity usage storage, using virtual domain: T424185
  • Query migrations

Part 2: Tables/backfilling

  • wbc_entity_usage tables of all wikis are moved to x1 cluster: T424440
  • switch the Wikibase config to point to x1 for entity usage storage
    • Details of rollout strategy to be defined, might want some gradual rollout etc

Related Objects

StatusSubtypeAssignedTask
OpenNone
Resolvedseanleong-WMDE
Resolvedseanleong-WMDE
ResolvedSpikeNicholusmuwonge_wmde
Resolvedseanleong-WMDE
ResolvedSuzanneWood-WMDE
ResolvedSuzanneWood-WMDE
ResolvedDanny_Benjafield_WMDE
Resolvedseanleong-WMDE
Resolvedseanleong-WMDE
Openseanleong-WMDE
OpenZabe
ResolvedZabe
ResolvedZabe
OpenZabe
ResolvedZabe
OpenZabe
OpenNone

Event Timeline

Regarding your question: It depends on how it is being implemented. The tables are too big to be moved manually. So you have to either 1- just create them empty and let them repopulate organically 2- Write a maint script to populate it even if it's slightly outdated. Either way, the only thing SRE will do is just creating the tables.

Thanks for confirming @Ladsgroup. I have been (naively) hopeful we could get around "backfilling" the tables given as we got lucky with that when moving Wikidata's term-related tables in T351820. But I see that even if those tables were potentially bigger, there was just a dozen of tables from one cluster. Here we have several hundred tables, many of them fairly big, and from a half a dozen of clusters.
We'll need to plan for that then.

The relevant part is probably this:

We acknowledge that third party tools on Cloud VPS and Toolforge may rely on the Wikibase database schema. Because of this, changes to the available tables and fields are subject to the above notification policy. However, note that the database schema is not designed to be a public API, and less consideration is given to backwards compatibility.

OTOH, I assume the usage of this table to be rather low (specially compared to other tables). Is there a way to poll for usage via email to mailing lists or similar? Maybe noone really use it and we can get away with not exposing it to the wikireplicas at all.

The relevant part is probably this:

We acknowledge that third party tools on Cloud VPS and Toolforge may rely on the Wikibase database schema. Because of this, changes to the available tables and fields are subject to the above notification policy. However, note that the database schema is not designed to be a public API, and less consideration is given to backwards compatibility.

Agreed

OTOH, I assume the usage of this table to be rather low (specially compared to other tables). Is there a way to poll for usage via email to mailing lists or similar? Maybe noone really use it and we can get away with not exposing it to the wikireplicas at all.

I assume it is low too but I am not certain enough. Asking around is probably up to @Danny_Benjafield_WMDE. (I can help.)

SuzanneWood-WMDE renamed this task from Move Wikibase's entity usage database tables to the extension database cluster to [X1][Epic] Move Wikibase's entity usage database tables to the extension database cluster.May 20 2026, 12:01 PM

It would be nice if the data was still available via the wikireplicas. Right now I can write a toolforge tool to run through all the main wiki shards and determine which pages on every WMF wiki is using a particular Wikidata entity. Forcing us to go through an API will be very slow by comparison. I understand this would be in a different database so we can no longer do server side SQL joins but if the extension databases were also available under the wikireplicas then one could query these and at least do manual application side joins. That would also allow one to see the wikibase terms in x3 again too. I might not have a correct understanding of things but that is my take on it if I understand them correctly.

So can the extension database clusters be added to the wikireplicas? I would be good with them being in a different set of 'x' shards vs. the current 's' shards.

EDIT: It looks like this is being handled by: T395881: Set up x1 replication to Wiki Replicas.

seanleong-WMDE renamed this task from [X1][Epic] Move Wikibase's entity usage database tables to the extension database cluster to [Epic][X1] Move Wikibase's entity usage database tables to the extension database cluster.Sep 1 2026, 12:19 PM
seanleong-WMDE updated the task description. (Show Details)