[BUG]Multi-modal embedding issue on 2.17

[b4sjoo]

What is the bug?

Multimodal ingestion and queries are not working as expected with Bedrock image v1 connector.

How can one reproduce the bug?

Create connector and register the model

{
    "name": "Amazon Bedrock Connector: multi-modal embedding",
    "description": "The connector to bedrock Titan multi-modal embedding model",
    "version": 1,
    "protocol": "aws_sigv4",
    "parameters": {
        "region": "us-east-1",
        "service_name": "bedrock",
        "model": "amazon.titan-embed-image-v1",
        "input_docs_processed_step_size": 2
    },
    "credential": {
        "access_key": "{{AWS_ACCESS_KEY_ID}}",
        "secret_key": "{{SECRET_ACCESS_KEY}}",
        "session_token": "{{AWS_SESSION_TOKEN}}"
    },
    "actions": [
        {
            "action_type": "predict",
            "method": "POST",
            "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke",
            "headers": {
                "content-type": "application/json",
                "x-amz-content-sha256": "required"
            },
            "request_body": "{\"inputText\": \"${parameters.inputText:-null}\", \"inputImage\": \"${parameters.inputImage:-null}\"}",
            "pre_process_function": "connector.pre_process.bedrock.multimodal_embedding",
            "post_process_function": "connector.post_process.bedrock.embedding"
        }
    ]
}

Create ingest pipeline

{
    "description": "An example neural search pipeline",
    "processors": [
        {
            "text_image_embedding": {
                "model_id": "{{titanimage_modelid}}",
                "embedding": "vector_embedding",
                "field_map": {
                    "image": "image_field",
                    "text": "text_field"
                }
            }
        }
    ]
}

Create index

{
    "settings": {
        "index": {
            "default_pipeline": "multi_modal_embedding_pipeline",
            "knn": true,
            "knn.algo_param.ef_search": 100
        }
    },
    "mappings": {
        "properties": {
            "id": {
                "type": "text"
            },
            "text": {
                "type": "text"
            },
            "text_field": {
                "type": "knn_vector",
                "dimension": 1024,
                "method": {
                    "name": "hnsw",
                    "space_type": "l2",
                    "engine": "nmslib",
                    "parameters": {
                        "ef_construction": 128,
                        "m": 24
                    }
                }
            },
            "image": {
                "type": "text"
            },
            "image_field": {
                "type": "knn_vector",
                "dimension": 1024,
                "method": {
                    "name": "hnsw",
                    "space_type": "l2",
                    "engine": "nmslib",
                    "parameters": {
                        "ef_construction": 128,
                        "m": 24
                    }
                }
            }
        }
    }
}

Ingest doc

{
    "text": "Say this is a test",
    "image": "/9j...7/",
    "id": "s1"
}

Get doc, found not embedded.

{
    "_index": "multi_modal_index",
    "_id": "1",
    "_version": 1,
    "_seq_no": 0,
    "_primary_term": 1,
    "found": true,
    "_source": {
        "image": "/9j/.../",
        "text": "Say this is a test",
        "id": "s1"
    }
}

Do a multimodal query, return exception, no log trace

# Request
GET /multi_model_index/_search
{
    "query": {
        "neural": {
            "vector_embedding": {
                "query_text": "table lamp",
                "query_image": "base64_for_image_abcdef12345678",
                "model_id": {{titanimage_modelid}}
            }
        }
    }
}

# Response
{
    "error": {
        "root_cause": [
            {
                "type": "json_parse_exception",
                "reason": "Unexpected character ('F' (code 70)) in numeric value: expected digit (0-9) to follow minus sign, for valid numeric value\n at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 7, column: 30]"
            }
        ],
        "type": "json_parse_exception",
        "reason": "Unexpected character ('F' (code 70)) in numeric value: expected digit (0-9) to follow minus sign, for valid numeric value\n at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 7, column: 30]"
    },
    "status": 400
}

What is the expected behavior?

Multimodal fields should be correctly embedded and indexed and multimodal query should work.

What is your host/environment?

OS 2.17

Do you have any additional context?

Prediction with the model directly is working, indicating the connector itself and ML-commons are working as expected.

# Request
POST /_plugins/_ml/models/{{titanimage_modelid}}/_predict
{
  "parameters": {
    "inputText": "Say this is a test",
    "inputImage": "/9j/4...="
  }
}

# Response
{
    "inference_results": [
        {
            "output": [
                {
                    "name": "sentence_embedding",
                    "data_type": "FLOAT32",
                    "shape": [
                        1024
                    ],
                    "data": [
                        0.017202578,
...
                        0.011084469
                    ]
                }
            ],
            "status_code": 200
        }
    ]
}

[martin-gaievski]

@b4sjoo thanks for reporting the issue.
I've checked your steps, it's a setup problem.

in your ingest pipeline configuration you need to have fields on your doc that do have text and image info, and embedding field should point to vector field that will store the embeddings that we retrieve from inference call. Note that we assume single embedding for both image and text.

Let's extend your index mapping and add vector_embedding field that will store the embeddings

{
    "settings": {
        "index": {
            "default_pipeline": "multi_modal_embedding_pipeline",
            "knn": true,
            "knn.algo_param.ef_search": 100
        }
    },
    "mappings": {
        "properties": {
            "id": {
                "type": "text"
            },
            "text": {
                "type": "text"
            },
            "image": {
                "type": "text"
            },
            "vector_embedding": {
                "type": "knn_vector",
                "dimension": 1024,
                "method": {
                    "name": "hnsw",
                    "space_type": "l2",
                    "engine": "nmslib",
                    "parameters": {
                        "ef_construction": 128,
                        "m": 24
                    }
                }
            }
        }
    }
}

If you change ingest pipeline to config to something like below it should work

{
    "description": "An example neural search pipeline",
    "processors": [
        {
            "text_image_embedding": {
                "model_id": "{{titanimage_modelid}}",
                "embedding": "**vector_embedding**",
                "field_map": {
                    "image": "**image**",
                    "text": "**text**"
                }
            }
        }
    ]
}

I tested it with local model, this returns the doc with embeddings

{
    "took": 3,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "my-index",
                "_id": "1",
                "_score": 1.0,
                "_source": {
                    "image": "/9j...7/",
                    "vector_embedding": [
                        0.19196375,
                        -0.2834456,

                    ],
                    "text": "Say this is a test",
                    "id": "s1"
                }
            }
        ]
    }
}