Training Shape Error: Shape must be rank 1 but is rank 4 for 'batch_normalization

[AlexMikhalev]

latest keras 2.2.0 with tensorflow 1.8.0 backend:
After successfull pre-processing results in Shape error.

h 10830 MB memory) -> physical GPU (device: 0, name: TITAN V, pci bus id: 0000:01:00.0, compute capability: 7.0)
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1567, in _create_c_op
    c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape must be rank 1 but is rank 4 for 'batch_normalization_1/cond/FusedBatchNorm' (op: 'FusedBatchNorm') with input shapes: [?,94,21009,32], [1,94,1,1], [1,94,1,1], [1,94,1,1], [1,94,1,1].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_network.py", line 63, in <module>
    val_split=args.val, tile=args.tile)
  File "train_network.py", line 32, in train_network
    model, serial_model = setup_model(X_train, class_names, weights_file=weights_file)
  File "/data/code/panotti/panotti/models.py", line 210, in setup_model
    serial_model = MyCNN_Keras2(X.shape, nb_classes=len(class_names), nb_layers=nb_layers)
  File "/data/code/panotti/panotti/models.py", line 47, in MyCNN_Keras2
    model.add(BatchNormalization(axis=1))
  File "/src/keras/engine/sequential.py", line 187, in add
    output_tensor = layer(self.outputs[0])
  File "/src/keras/engine/base_layer.py", line 460, in __call__
    output = self.call(inputs, **kwargs)
  File "/src/keras/layers/normalization.py", line 204, in call
    training=training)
  File "/src/keras/backend/tensorflow_backend.py", line 3069, in in_train_phase
    x = switch(training, x, alt)
  File "/src/keras/backend/tensorflow_backend.py", line 3004, in switch
    else_expression_fn)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2072, in cond
    orig_res_f, res_f = context_f.BuildCondBranch(false_fn)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1913, in BuildCondBranch
    original_result = fn()
  File "/src/keras/layers/normalization.py", line 165, in normalize_inference
    epsilon=self.epsilon)
  File "/src/keras/backend/tensorflow_backend.py", line 1894, in batch_normalization
    is_training=False
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py", line 904, in fused_batch_norm
    name=name)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 3429, in _fused_batch_norm
    is_training=is_training, name=name)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
    op_def=op_def)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1734, in __init__
    control_input_ops)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1570, in _create_c_op
    raise ValueError(str(e))
ValueError: Shape must be rank 1 but is rank 4 for 'batch_normalization_1/cond/FusedBatchNorm' (op: 'FusedBatchNorm') with input shapes: [?,94,21009,32], [1,94,1,1], [1,94,1,1], [1,94,1,1], [1,94,1,1].

Seems reshape step missing or did I make a mistake of adding --phase during pre-processing?
P.S. Impressive library, please keep going.

[AlexMikhalev]

Update:
seems to be working fine on Keras 2.1.5(6), breaking changes in Keras 2.2.0.
My solution was to roll back to 2.1.6

[satels]

With keras==2.1.5(6):

Traceback (most recent call last):
  File "train_network.py", line 63, in <module>
    val_split=args.val, tile=args.tile)
  File "train_network.py", line 32, in train_network
    model, serial_model = setup_model(X_train, class_names, weights_file=weights_file)
  File "/opt/data/panotti/panotti/models.py", line 210, in setup_model
    serial_model = MyCNN_Keras2(X.shape, nb_classes=len(class_names), nb_layers=nb_layers)
  File "/opt/data/panotti/panotti/models.py", line 51, in MyCNN_Keras2
    model.add(Conv2D(nb_filters, kernel_size))
  File "/usr/local/lib/python3.5/dist-packages/keras/models.py", line 522, in add
    output_tensor = layer(self.outputs[0])
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py", line 619, in __call__
    output = self.call(inputs, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/keras/layers/convolutional.py", line 168, in call
    dilation_rate=self.dilation_rate)
  File "/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py", line 3341, in conv2d
    data_format=tf_data_format)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 782, in convolution
    return op(input, filter)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 870, in __call__
    return self.conv_op(inp, filter)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 522, in __call__
    return self.call(inp, filter)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 206, in __call__
    name=self.name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 953, in conv2d
    data_format=data_format, dilations=dilations, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3292, in create_op
    compute_device=compute_device)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3332, in _create_op_helper
    set_shapes_for_outputs(op)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2496, in set_shapes_for_outputs
    return _set_shapes_for_outputs(op)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2469, in _set_shapes_for_outputs
    shapes = shape_func(op)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2399, in call_with_requiring
    return call_cpp_shape_fn(op, require_shape_fn=True)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/common_shapes.py", line 627, in call_cpp_shape_fn
    require_shape_fn)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/common_shapes.py", line 691, in _call_cpp_shape_fn_impl
    raise ValueError(err.message)
ValueError: Negative dimension size caused by subtracting 3 from 1 for 'conv2d_4/convolution' (op: 'Conv2D') with input shapes: [?,22,1,32], [3,3,32,32].

[drscotthawley]

Hi. It's been a little while since I upgraded by Keras. This could be an error in the image channel ordering, which is set to either channels_first or channels_last in your Keras config file. Here's mine:

$ cat ~/.keras/keras.json 
{
    "image_data_format": "channels_last",
    "epsilon": 1e-07,
    "backend": "tensorflow",
    "floatx": "float32"
}

...Hmmm. although I do override the setting to make it channels_last in models.py as well:
K.set_image_data_format('channels_last')
Not sure. I'll try to take a look soon.

[AlexMikhalev]

bump. I have exactly the same error as @satels.

[wylie-r3pi]

Update:
seems to be working fine on Keras 2.1.5(6), breaking changes in Keras 2.2.0.
My solution was to roll back to 2.1.6

Fixed it for me, thanks !