augmentedManifestFile + PipeModeDataset example

[vlordier]

It would really help to have a full end to end example of, say, image classification with augmentedManifestFile + PipeModeDataset

as I keep getting errors of formats like
tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Could not parse example input, value: '����

I build a jsonl augumentedManifest with

{'image-ref': s3://path/to/image, 'label': 3} 
{'image-ref': s3://path/to/image, 'label': 1}  
{'image-ref': s3://path/to/image, 'label': 2}  

then preparing training channel as

train_data = sagemaker.session.s3_input(augmented_manifest_file_on_s3,
                                        distribution 	= 'FullyReplicated',
                                        content_type 	= 'image/jpeg',
                                        s3_data_type 	= 'AugmentedManifestFile',
                                        attribute_names	= ['image-ref', 'label'],
					input_mode 		= 'Pipe',
                                        record_wrapping = 'RecordIO') 

and launching the .fit as

data_channels = {'train': train_data}

# Train a model.
tf_estimator.fit(inputs=data_channels, logs=True)

in my entry script, I have

	dataset = PipeModeDataset(channel = channel)
	dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)
	dataset = dataset.batch(2)
	dataset = dataset.map(combine)
	dataset = dataset.map(example_parser, num_parallel_calls=batch_size)
	dataset = dataset.repeat(epochs)
	dataset = dataset.batch(batch_size, drop_remainder=True)
	image_batch, label_batch = next(iter(dataset))

and as a modified example parser, I have

`def example_parser(exemple1, exemple2):

feat1 = tf.io.parse_single_example(
	exemple1,
	features={
		'image-ref'		: tf.io.FixedLenFeature([], tf.string),
	})

feat2 = tf.io.parse_single_example(
	exemple2,
	features={
		'label'			: tf.io.FixedLenFeature([], tf.int64),
	})

image 					= feat1['image-ref']
image = tf.image.decode_jpeg(image, channels=3)
image = tf.image.convert_image_dtype(image, tf.float32)
label 					= tf.cast(feat2['label'], tf.int32)
return image, label

`

What am I doing wrong ?
The documentation here is not clear about using augmented manifest files

[laurenyu]

could you show your code for creating the TF estimator and launching the training job as well?

[vlordier]

creating the TF estimator

tf_estimator = TensorFlow( entry_point = train.py, 
					train_use_spot_instances = False,
					train_max_wait = 36000,
					train_max_run= 36000,
					role 	= role,
					train_instance_count 	= 1, 
					train_instance_type = 'ml.p2.8xlarge',
					framework_version = '2.0.0', 
					py_version 	= 'py3',
					input_mode 	= 'Pipe',
					source_dir 	= source_dir,
					hyperparameters 	= hyperparameters,
					distributions 	= distributions,
					)

with distributions as

distributions={
			'mpi': {
					'enabled': True,
					'processes_per_host': 8,
					'custom_mpi_options': '--NCCL_DEBUG INFO'
			}
	}

since I am looking to

• stream a dataset of compressed jpg from S3 using Pipemodedataset and Augmented Manifest File
• decode and augment it on the fly
• feed it to multiple GPU with horovod for training

Launching the training job with
tf_estimator.fit(inputs=data_channels, logs=True)

with data_channels as
data_channels = {'training': train_data, 'testing' : test_data, 'validation' : val_datal}

and train_data defined as

train_data = sagemaker.session.s3_input(augmented_manifest_file_on_s3,
                                        distribution 	= 'FullyReplicated',
                                        content_type 	= 'image/jpeg',
                                        s3_data_type 	= 'AugmentedManifestFile',
                                        attribute_names	= ['image-ref', 'label'],
					input_mode 		= 'Pipe',
                                        record_wrapping = 'RecordIO') 

This notebook here is a good starting point to put together a little demo of how to use Pipe Mode on Augmented Manifest Files

[nadiaya]

Have you seen the documentation we have on using augmented manifest files with PipeModeDataset?
https://github.com/aws/sagemaker-tensorflow-extensions#using-the-pipemodedataset-with-sagemaker-augmented-manifest-files
https://docs.aws.amazon.com/sagemaker/latest/dg/augmented-manifest.html

[nadiaya]

Marking it with a 'feature request' label for the notebook with a better explanation.

[matthewhardern]

I am also encountering the above issue when trying to decode PNG images.

OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Could not parse example input

Have we got any idea what is causing the above issue?

[matthewhardern]

@vlordier Update for you if you have not found the answer, I removed the features and fed in the example1 / 2 in your case directly although you will need to tf.strings.to_number() on your label. This then worked for me