port items from old core

[Felk]

Mostly a tracking issue for me, and a place to dump intermediate results and thoughts.

Old core is currently handling items, and all that is in dire need to be modernized. There are a few challenges that go beyond just porting the code as-is:

• We are quite unsatisfied with the current items situation in old core, so we definetely want to take a fresh look at what we have any make a bunch of simplifications where possible.
• The concept of "crates" was overhauled a few years back, meaning new crates are distinct items instead of an item of type "crate" with its actual content specified by "data.content_id".
  Nontheless, these old-style crates still exist in the database and cannot be ignored.
• The fact that we have stuff in the DB goes beyond the old-style crates, so all existing stuff must either be migrated or kept in mind in a backwards-compatible manner

[Felk]

I started by getting an overview of the amounts and kinds of old-style crates we still have in the database. I ran this query:

[
  {
    $match: { type_id: "crate" } },
  {
    $group:
      {
        _id: {
          name: "$name",
          content_id: "$data.content_id",
          data_keys: {
            $sortArray: {
              input: {
                $map: {
                  input: { $objectToArray: "$data" },
                  as: "kvPair",
                  in: "$$kvPair.k"
                }
              },
              sortBy: 1
            }
          }
        },
        count: { $sum: 1 },
        count_usernotnull: {
          $sum: {
            $cond: [ { $ne: ["$user_id", null] }, 1, 0 ]
          }
        }
      }
  },
  {
    $project:
      {
        _id: false,
        name: "$_id.name",
        content_id: { $ifNull: ["$_id.content_id", null] },
        data_keys: "$_id.data_keys",
        count: "$count",
        count_usernotnull: "$count_usernotnull"
      }
  },
  {
    $sort: { content_id: 1 }
  }
]

and here's the prettified results:

name	content_id	count	count_usernotnull	other data keys
Rare Badge Crate	null	180	39
Gen 3-7 Badge Crate	null	39	0
Token Crate	null	29	17
Crate Crate	null	12	2
Probably Empty Crate	null	7612	7165
Uncommon Badge Crate	null	250	71
Token Crate	null	991	595	max_amount, min_amount
Fun Crate	null	377	120
Small Fun Crate	null	419	109
Gen 2-7 Badge Crate	null	1	1
Common Badge Crate	badge_common	50	0
Gen 2-7 Badge Crate	badge_gen2-7	76	8
Gen 3-7 Badge Crate	badge_gen3-7	103	3
badge_rare	badge_rare	4	1
Rare Badge Crate	badge_rare	259	50
Common Badge Crate	badge_trash	97	9
Uncommon Badge Crate	badge_uncommon	880	196
badge_uncommon	badge_uncommon	2	0
Crate Crate	crate_crate	1536	184
Fun Crate	funstuff	6592	2749
funstuff	funstuff	7	0
Probably Empty Crate	mostly_nothing	15	0	max_amount, min_amount
Probably Empty Crate	mostly_nothing	36951	32916
mostly_nothing	mostly_nothing	32	14
Token Crate	one_token	498	277
Small Fun Crate	small_funstuff	2643	1341
tokens	tokens	37	19	max_amount, min_amount
Token Crate	tokens	10195	6276	max_amount, min_amount

[Felk]

First step is filling in the missing content_ids:

db.items.updateMany(
  { name: 'Rare Badge Crate', 'data.content_id': null, type_id: 'crate' },
  { $set: { 'data.content_id': "badge_rare" } }
);
db.items.updateMany(
  { name: 'Gen 3-7 Badge Crate', 'data.content_id': null, type_id: 'crate' },
  { $set: { 'data.content_id': "badge_gen3-7" } }
);
db.items.updateMany(
  { name: 'Crate Crate', 'data.content_id': null, type_id: 'crate' },
  { $set: { 'data.content_id': "crate_crate" } }
);
db.items.updateMany(
  { name: 'Probably Empty Crate', 'data.content_id': null, type_id: 'crate' },
  { $set: { 'data.content_id': "mostly_nothing" } }
);
db.items.updateMany(
  { name: 'Uncommon Badge Crate', 'data.content_id': null, type_id: 'crate' },
  { $set: { 'data.content_id': "badge_uncommon" } }
);
db.items.updateMany(
  { name: 'Fun Crate', 'data.content_id': null, type_id: 'crate' },
  { $set: { 'data.content_id': "funstuff" } }
);
db.items.updateMany(
  { name: 'Small Fun Crate', 'data.content_id': null, type_id: 'crate' },
  { $set: { 'data.content_id': "small_funstuff" } }
);
db.items.updateMany(
  { name: 'Gen 2-7 Badge Crate', 'data.content_id': null, type_id: 'crate' },
  { $set: { 'data.content_id': "badge_gen2-7" } }
);
db.items.updateMany(
  { name: 'Token Crate', 'data.content_id': null, type_id: 'crate', 'data.min_amount': {$ne: null} },
  { $set: { 'data.content_id': "tokens" } }
);
db.items.updateMany(
  { name: 'Token Crate', 'data.content_id': null, type_id: 'crate', 'data.min_amount': {$eq: null} },
  { $set: { 'data.content_id': "one_token" } }
);

This assumes that token crates without min_amount and max_amount were meant to mean "1 token", which is the case judging from the current code.

Next, remove the min_amount and max_amount from mostly_nothing crates, as that makes no sense.

db.items.updateMany(
  { 'data.content_id': 'mostly_nothing', type_id: 'crate', 'data.min_amount': {$ne: null} },
  { $unset: {'data.min_amount': true} }
);
db.items.updateMany(
  { 'data.content_id': 'mostly_nothing', type_id: 'crate', 'data.max_amount': {$ne: null} },
  { $unset: {'data.max_amount': true} }
);

After this, all crates have a content_id, and for each content_id the set of fields present in data is unambiguous:

content_id	count	count_usernotnull	other data keys
badge_common	50	0
badge_gen2-7	77	9
badge_gen3-7	142	3
badge_rare	443	90
badge_trash	97	9
badge_uncommon	1132	267
crate_crate	1548	186
funstuff	6976	2869
mostly_nothing	44610	40095
one_token	527	294
small_funstuff	3062	1450
tokens	11223	6890	max_amount, min_amount

[Felk]

Next is finding each old-style crate's new-style counterpart. I went through them and made a mapping:

old-style crate	new-style crate item
badge_trash	trashbadgecrate
badge_common	commonbadgecrate
badge_uncommon	uncommonbadgecrate
badge_rare	rarebadgecrate
badge_gen2-7	gen2-7badgecrate
badge_gen3-7	gen3-7badgecrate
crate_crate	cratecrate
funstuff	funcrate
small_funstuff	smallfuncrate
mostly_nothing	probablyemptycrate
one_token	1tokencrate
tokens	tokencrate

Note that although the respective trash/common/uncommon/rare new-style crates still have the same rarity name, in the code those get shifted up a rarity level since we no longer have trash but we do have ultrarare. So the trash crate turns into a trashbadgecrate, but which will actually drop common badges, and rare will drop ultrarare etc., so everything's fine here.

So all we need to do is perform that translation and drop the content_id:

db.items.updateMany(
  { 'data.content_id': 'badge_trash', type_id: 'crate' },
  { $set: {type_id: 'trashbadgecrate'}, $unset: {'data.content_id': true} }
);
db.items.updateMany(
  { 'data.content_id': 'badge_common', type_id: 'crate' },
  { $set: {type_id: 'commonbadgecrate'}, $unset: {'data.content_id': true} }
);
db.items.updateMany(
  { 'data.content_id': 'badge_uncommon', type_id: 'crate' },
  { $set: {type_id: 'uncommonbadgecrate'}, $unset: {'data.content_id': true} }
);
db.items.updateMany(
  { 'data.content_id': 'badge_rare', type_id: 'crate' },
  { $set: {type_id: 'rarebadgecrate'}, $unset: {'data.content_id': true} }
);
db.items.updateMany(
  { 'data.content_id': 'badge_gen2-7', type_id: 'crate' },
  { $set: {type_id: 'gen2-7badgecrate'}, $unset: {'data.content_id': true} }
);
db.items.updateMany(
  { 'data.content_id': 'badge_gen3-7', type_id: 'crate' },
  { $set: {type_id: 'gen3-7badgecrate'}, $unset: {'data.content_id': true} }
);
db.items.updateMany(
  { 'data.content_id': 'crate_crate', type_id: 'crate' },
  { $set: {type_id: 'cratecrate'}, $unset: {'data.content_id': true} }
);
db.items.updateMany(
  { 'data.content_id': 'funstuff', type_id: 'crate' },
  { $set: {type_id: 'funcrate'}, $unset: {'data.content_id': true} }
);
db.items.updateMany(
  { 'data.content_id': 'small_funstuff', type_id: 'crate' },
  { $set: {type_id: 'smallfuncrate'}, $unset: {'data.content_id': true} }
);
db.items.updateMany(
  { 'data.content_id': 'mostly_nothing', type_id: 'crate' },
  { $set: {type_id: 'probablyemptycrate'}, $unset: {'data.content_id': true} }
);
db.items.updateMany(
  { 'data.content_id': 'one_token', type_id: 'crate' },
  { $set: {type_id: '1tokencrate'}, $unset: {'data.content_id': true} }
);
db.items.updateMany(
  { 'data.content_id': 'tokens', type_id: 'crate' },
  { $set: {type_id: 'tokencrate'}, $unset: {'data.content_id': true} }
);

After that, no more items with type_id equal to crate exist anymore 🎉

[Felk]

The next step is to get rid of any still existing multi-use items (having a data.uses field), which were deprecated in 2018 (https://github.com/TwitchPlaysPokemon/tpp/commit/da0fbb0f6859cda69e58bdd42ebbbd8510125c4d). Instead of handing out items with X uses, we are just handing out X of those item.

These are all the items we still have in the DB which have a data.uses:

type_id	count	count_usernotnull	data_keys0	data_keys1
badgerain	8231	0	uses	null
bloodymail	1036	0	uses	null
chatter	22748	198	uses	null
emotedrip	5432	0	uses	null
emoterain	506	202	cents	uses
emoterain	6118	0	uses	null
emoterainween	1069	0	uses	null
mail	6223	262	uses	null
noisemaker	60064	526	uses	null
spookynoisemaker	6219	0	uses	null
whisper	3606	0	uses	null

This leads us into a small excourse: If we get rid of those cents first, all our items with uses would be easy to migrate: just duplicate them n times and then get rid of the data.

The emoterains with cents (ignoring all "uses: 0") look like this:

type_id	cents	data_keys0	data_keys1	count	count_usernotnull
emoterain	200	cents	uses	141	140
emoterain	10	cents	uses	66	62

The emoterain's default cents were increased from 200 to 400 on 2023-10-21 (https://github.com/TwitchPlaysPokemon/tpp/commit/74a6066996a7b4dab1a15c5081e31b3ff4bfe5ac), so for those 200-cent emoterains it would probably be acceptable to "upgrade" those by just removing their cents.
For the 10-cent emoterains, those can just be migrated to "emotedrip", which already is a 10-cent emoterain.

db.items.updateMany(
  {type_id: "emoterain", "data.cents": 200},
  {$unset: {'data.cents': true}}
);
db.items.updateMany(
  {type_id: "emoterain", "data.cents": 10},
  {$set: {type_id: "emotedrip"}, $unset: {'data.cents': true}}
);

[Felk]

Now onto getting rid of multi-use items. One think we need to keep in mind now is that we have an itemlog collection, which is supposed to act as a papertrail where each individual item comes from. So if we duplicate items with uses, we should also generate itemlog entries for those.

First, let's create the duplicates. This aggregation query makes data.uses-1 copies of each item, keeping the original item:

db.items.aggregate([
  {
    $match: {
      "data.uses": {
        $gt: 0
      }
    }
  },
  {
    $set: {
      copies: {
        $range: [1, "$data.uses"]
      }
    }
  },
  {
    $unwind: "$copies"
  },
  {
    $addFields: {
      duplicate_of: "$_id"
    }
  },
  {
    $project: {
      _id: false,
      copies: false,
      data: false
    }
  },
  {
    $merge: "items"
  }
]);

Then all the data.uses can be removed:

db.items.updateMany(
  {'data.uses': {$exists: true}},
  {$unset: {'data.uses': true}}
);

Finally, create itemlog entries for the newly created items:

db.items.aggregate([
  {
    $match: {
      duplicate_of: { $exists: true }
    }
  },
  {
    $project: {
      item_id: "$_id",
      cloned_from_item_id: "$duplicate_of",
      event: "cloned_migrated_from_multiuse",
      timestamp: new Date(),
      old_user_id: null,
      user_id: "$user_id"
    }
  },
  {
    $merge: "itemlog"
  }
]);

After that, no more items with data.uses exist anymore 🎉

[Felk]

Okay, now let's check where we stand. What items with data still exist now? This is the pipeline query I ran:

[
  {
    $match: {
      data: {
        $exists: true
      }
    }
  },
  {
    $group: {
      _id: {
        type_id: "$type_id",
        data_keys: {
          $sortArray: {
            input: {
              $map: {
                input: {
                  $objectToArray: "$data"
                },
                as: "kvPair",
                in: "$$kvPair.k"
              }
            },
            sortBy: 1
          }
        }
      },
      count: {
        $sum: 1
      },
      count_usernotnull: {
        $sum: {
          $cond: [
            {
              $ne: ["$user_id", null]
            },
            1,
            0
          ]
        }
      }
    }
  },
  {
    $project: {
      _id: false,
      type_id: "$_id.type_id",
      data_keys: "$_id.data_keys",
      count: "$count",
      count_usernotnull: "$count_usernotnull"
    }
  },
  {
    $sort: {
      content_id: 1
    }
  }
]

And this is the result:

type_id	data_keys0	data_keys1	count	count_usernotnull
tokencrate	max_amount	min_amount	11296	6893
giftcrate	max_amount	min_amount	1575	0

The giftcrate having min_amount and max_amount makes no sense. I'm not sure how that happened, but let's just fix it:

db.items.updateMany(
  { 'type_id': 'giftcrate', 'data.min_amount': {$ne: null} },
  { $unset: {'data.min_amount': true} }
);
db.items.updateMany(
  { 'type_id': 'giftcrate', 'data.max_amount': {$ne: null} },
  { $unset: {'data.max_amount': true} }
);

As an intermediate cleanup-step, let's remove data from all items where it's an empty object:

db.items.updateMany(
  {'data': {$eq: {}}},
  {$unset: {'data': true}}
);

Note that the current code may still write empty data objects, so this is just for easier analysis on a db copy that isn't live. It's not a necessary step.

At this point, the only items with data are tokencrate. Here's a pipeline query to analyze their contents:

[
  {
    $match: {
      type_id: "tokencrate"
    }
  },
  {
    $group: {
      _id: {
        min_amount: "$data.min_amount",
        max_amount: "$data.max_amount"
      },
      count: {
        $sum: 1
      },
      count_usernotnull: {
        $sum: {
          $cond: [
            {
              $ne: ["$user_id", null]
            },
            1,
            0
          ]
        }
      }
    }
  },
  {
    $project: {
      _id: false,
      min_amount: "$_id.min_amount",
      max_amount: "$_id.max_amount",
      count: "$count",
      count_usernotnull: "$count_usernotnull"
    }
  },
  {
    $sort: {
      min_amount: 1
    }
  }
]

This gives us:

count	count_usernotnull	min_amount	max_amount
2	0	null	null
10095	6551	1	1
493	200	2	2
250	84	3	3
121	23	4	4
75	17	5	5
55	5	6	6
30	2	7	7
19	2	8	8
20	2	9	9
11	2	10	10
5	0	11	11
8	1	12	12
3	1	13	13
10	0	14	14
11	0	15	15
3	0	16	16
7	1	17	17
2	0	18	18
3	1	19	19
11	0	20	20
4	0	21	21
3	0	23	23
2	0	24	24
2	0	25	25
3	0	26	26
2	1	27	27
1	0	28	28
1	0	29	29
8	0	30	30
4	0	31	31
2	0	33	33
1	0	34	34
1	0	35	35
1	0	37	37
7	0	40	40
1	0	44	44
6	0	45	45
1	0	46	46
1	0	47	47
1	0	49	49
12	0	60	60

Looking at the code, the tokencrate without min_amount and max_amount default to 1 token, so we can easily fix those:

db.items.updateMany(
  { 'type_id': 'tokencrate', 'data.min_amount': {$eq: null} },
  { $set: {'data.min_amount': 1} }
);
db.items.updateMany(
  { 'type_id': 'tokencrate', 'data.max_amount': {$eq: null} },
  { $set: {'data.max_amount': 1} }
);

So what about the remaining tokencrate items? Should they be migrated somehow so the concept of data can be completely abolished? If yes, how?

[Felk]

A quick survey on discord showed that we want to keep the data. So let's call it a day regarding migrating items and continue porting items to new core with what we have.