VLM table conversion quality is not yet great

[geoHeil]

When using https://docling-project.github.io/docling/examples/compare_vlm_models/

with the example file (see here in the attachment:

for models like:

vlm_model_specs.GRANITE_VISION_TRANSFORMERS.torch_dtype = 'float16'
vlm_model_specs.SMOLDOCLING_TRANSFORMERS.torch_dtype = 'float16'
vlm_model_specs.PIXTRAL_12B_TRANSFORMERS.torch_dtype = 'float16'


vlm_models = [

    vlm_model_specs.SMOLDOCLING_MLX, 
    vlm_model_specs.SMOLDOCLING_TRANSFORMERS,
    ## Markdown models (using MLX framework)

    vlm_model_specs.PIXTRAL_12B_MLX,
    vlm_model_specs.GEMMA3_12B_MLX,

    vlm_model_specs.GRANITE_VISION_TRANSFORMERS, # works rather well
    vlm_model_specs.PIXTRAL_12B_TRANSFORMERS, 
]


# with

pipeline_options.accelerator_options.device = AcceleratorDevice.CUDA
pipeline_options.accelerator_options.cuda_use_flash_attention2 = True

The result is interesting. I so far never saw the table parsed correctly.

Smoldocling

• columns confused
• columns missing

including the G12W mutation predicted to result from a single nucleotide substitution from the cysteine 12 codon.

## Table 5

IC50 Values of the KRas G12C Inhibitor MR1X849 and the Pan KRas Inhibitor Example 5 in a 5-day Viability Assay in 3T3 Cells Engineered to Express MRTX849 Resistance Mutations

| MRTX849 (nM)   | Example 5 (nM)   |
|----------------|------------------|
| G12A           | >3000            |
| G12C           | 16.62            |
| G12D           | >3000            |
| G12R           | >3000            |
| G12V           | >3000            |
| G12W           | >3000            |
| G13D           | >3000            |
| Q61H           | >3000            |

granite

• text missing
• column content missing
• but overall structure OK

|    | MRTX849  (nM)   |   Example 5  (nM) |
|----|-----------------|-------------------|
|  0 | >3000           |             32    |
|  1 | 16.62           |             28.1  |
|  2 | >3000           |             20.25 |
|  3 | >3000           |           1742    |
|  4 | >3000           |             94    |
|  5 | >3000           |             50    |
|  6 | >3000           |            610    |
|  7 | >3000           |             58    |

pixtral

• columns confused but per row

including the G12W mutation predicted to result from a single nucleotide substitution from the cysteine 12 codon.

Table 5

ICS0 Values of the KRas GI2C Inhibitor MRTX849 and the Pan KRas Inhibitor Example 5 in a 5-day Viability Assay in 3T3 Cells Engineered to Express MRTX849 Resistance Mutations

|       |   MRTX849 (nM) | Example 5 (nM)   |
|-------|----------------|------------------|
| G12A  |          32    | >3000            |
| 16.62 |          28.1  | G12C             |
| >3000 |          20.25 | G12D             |
| >3000 |        1742    | G12R             |
| >3000 |          94    | G12V             |
| >3000 |          50    | G12W             |
| >3000 |         610    | G13D             |
| >3000 |          58    | Q61H             |

how to get correct conversion with a VLM approach?

[NabuchodonosorII]

Hello, have you add anything to pipeline_options_vlm_model.py or vlm_model_specs.py ? Not sure why i am getting a Syntax error.

Cell In[1], line 115 vlm_model_specs.SMOLDOCLING_TRANSFORMERS.torch_dtype = 'float16' ^ SyntaxError: cannot assign to attribute here. Maybe you meant '==' instead of '='?

[geoHeil]

Interesting,

also the new paddle V5 models on Rapicocr still behave weirdly here:

[geoHeil]

current version of docling 2.75.0 seems to tackle this well