Error creating item Invalid index name [items_Randy], must be lowercase

[pedro-cf]

Describe the bug
In a previously working environment, when trying to create a STAC item in a collection with id Randy, I get the following error:

{  
  "code": "BadRequestError",  
  "description": "BadRequestError(400, 'invalid_index_name_exception', 'Invalid index name [items_Randy], must be lowercase')"  
}  

This was previously working but now fails with an invalid index name exception.

To Reproduce
I am unable to reproduce this in a fresh environment.

Expected behavior
The STAC item should be successfully created, as it was before. If uppercase names are now restricted, the API should automatically convert them to lowercase or provide a clearer validation error earlier in the process.

Additional context

• SFEOS v3.2.3 deployed on k8s
• Elasticsearch v8.11.0 deployed on k8s

[jonhealy1]

@rhysrevans3 Hi. Can you help out here?

[rhysrevans3]

@pedro-cf sorry just so I'm clear. Previously when you tried to create an index name item_Randy would would receive an this error:

{
"code": "BadRequestError",
"description": "BadRequestError(400, 'invalid_index_name_exception', 'Invalid index name [items_Randy], must be lowercase')"
}

And now you still can't create the index but also don't get an exception?

[pedro-cf]

@pedro-cf sorry just so I'm clear. Previously when you tried to create an index name item_Randy would would receive an this error:

{
"code": "BadRequestError",
"description": "BadRequestError(400, 'invalid_index_name_exception', 'Invalid index name [items_Randy], must be lowercase')"
}

And now you still can't create the index but also don't get an exception?

Greetings @rhysrevans3 , In the past I created a collection with the id Randy and I was able to create items in that collection just fine.

But after some time with a couple of environment restarts for both the API and elasticsearch, I was unable to create any more items for that collection due to always getting the error:

{  
  "code": "BadRequestError",  
  "description": "BadRequestError(400, 'invalid_index_name_exception', 'Invalid index name [items_Randy], must be lowercase')"  
}  

Eventually I was forced to migrate all items in the collection to a new collection in order to work around the issue.

I assume this issue is related? #328

I believe this issue occurred after PR #329 was merged and I updated the SFEOS API to v3.2.3

[rhysrevans3]

#329 should allow collection_ids to include uppercase characters now but this has meant a change to the syntax for the item indices it is now {ITEMS_INDEX_PREFIX}{lower case collection_id with ES_INDEX_NAME_UNSUPPORTED_CHARS removed}_{hex encoded collection_id} with an alias of {ITEMS_INDEX_PREFIX}{collection_id with ES_INDEX_NAME_UNSUPPORTED_CHARS removed}

You can generate the new syntaxes with these functions from the elasticsearch or opensearch database logic

from stac_fastapi.elasticsearch.database_logic import index_alias_by_collection_id, index_by_collection_id
index_alias_by_collection_id(collection_id)
index_by_collection_id(collection_id)

This might be having unexpected effects on existing indices? And causing the error you're seeing. Am I right in saying you now have a collection is now called randy all lowercase? And are not able to add new items to it?

You might need to migrate your indexes again. There are some notes on reindexing in the readme #reindexing

But these are the steps I'd take (the script in the reindex is different)

Create the new collection record:

curl -X "POST" "{STAC API}/collections" \
     -H 'Content-Type: application/json' \
     -d $'{
  "id": "{new collection with uppercase}"
}'

you might also want to copy over any meta data from the existing collection.

You can then reindex the old collection items into the new index.

curl -X "POST" "{ELASTICSEARCH SERVER}:9200/_reindex" \
  -H 'Content-Type: application/json' \
  -d $'{
    "source": {
      "index": "{old index}"
    }, 
    "dest": {
      "index": "{new index}-000001"
    },
    "script": {
      "source": "ctx._id = ctx._id.split('|')[0] + '|{new collection with uppercase}';ctx._source.collection = {new collection with uppercase}",
	}
  }'

Create the new alias

curl -X "POST" "{ELASTICSEARCH SERVER}:9200/_aliases" \
  -h 'Content-Type: application/json' \
  -d $'{
    "actions": [
      {
        "add": {
          "index": "{new index}-000001",
          "alias": "{new alias}"
        }
      }
    ]
  }'

If you're happy with the new collection you can then remove the old one.

For your case moving from collection_id = randy to Randy the values should be:

new collection with uppercase = Randy
old index = items_randy
new index = items_randy_52616e6479-000001
new alias = items_Randy

[pedro-cf]

Thank you @rhysrevans3 for the help!

Since I migrated the items to a new lowercase collection and deleted the old collection I'm not sure I will be able to re-create this issue.

It would be cool to alert of "(schema) migrations" / re-indexing if needed in new releases @jonhealy1