Skip to content

odtp-org/odtp-huggingface-dataloader

 
 

Repository files navigation

odtp-component-example

This is a component used to test ODTP components. It uses Hugging face datasets, which you can download and save as .csv.

Tool Info Links
Original https://github.com/odtp-org/tool-example
Current Tool Version 88f953f0aabbfb08189bdc3212aa1c209fbedb2d

ODTP command

odtp new odtp-component-entry \
--name odtp-example \
--component-version 0.1.0 \
--repository https://github.com/odtp-org/odtp-component-example \
--commit a30858fe0997c839130f0e31583232caa143c79c

Tutorial

Testing the component.

This component can be tested in isolation with the following instructions.

  1. Prepare manually a folder called volume containing the input files/folder needed:

  2. Create your .env file with the parameters.

HF_DATASET=rooten_tomatoes
  1. Build the dockerfile.
docker build -t odtp-component-example .
  1. Create an execution folder outside this repository and place the .env inside
mkdir odtp-component-example-execution
cd odtp-component-example-execution
mkdir odtp-input
mkdir odtp-output
  1. Run the following command.
docker run -it --rm -v [Absolute Path to your input folder]:/odtp/odtp-input -v [Absolute Path to your output folder]:/odtp/odtp-output --env-file .env odtp-component-example

Changelog

  • v0.1.2: Update odtp-component-client to v0.1.0

  • v0.1.1: Change Dockerfile setup: use fixed versions for python and ubuntu

  • v0.1.0: Basic functionality

Development

Developed by SDSC

About

This is an odtp dataloder to huggingface datasets

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Shell 52.6%
  • Dockerfile 47.4%