Skip to content

Commit

Permalink
update
Browse files Browse the repository at this point in the history
  • Loading branch information
deependujha committed Feb 8, 2025
1 parent 6ffb1fa commit cd31285
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 3 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,7 @@ hf_uri = "hf://datasets/leonardPKU/clevr_cogen_a_train/data"
ds = ld.StreamingDataset(hf_uri)

for _ds in ds:
print(f"{_ds[1]=}; {_ds[2]}")
print(f"{_ds[1]}; {_ds[2]}")
```

You don’t need to worry about indexing the dataset or any other setup. **LitData** will **handle all the necessary steps automatically** and `cache` the `index.json` file, so you won't have to index it again.
Expand Down Expand Up @@ -314,7 +314,7 @@ hf_uri = "hf://datasets/open-thoughts/OpenThoughts-114k/data"
ds = ld.StreamingDataset(hf_uri, item_loader=ParquetLoader(), index_path="hf-index-dir")

for _ds in ds:
print(f"{_ds[0]=}; {_ds[1]}\n")
print(f"{_ds[0]}; {_ds[1]}\n")
```

 
Expand Down
2 changes: 1 addition & 1 deletion src/litdata/utilities/parquet.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ def delete_thread(rmq: Queue) -> None:
file_path = rmq.get()
if file_path is None: # Sentinel value to exit
break
if os.path.exists(file_path):
with suppress(FileNotFoundError):
os.remove(file_path)

# before exiting, just sleep for some time, to complete any pending deletions
Expand Down

0 comments on commit cd31285

Please sign in to comment.