Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding a pipeline file example notebook #12

Closed
jlvdb opened this issue May 16, 2024 · 2 comments · Fixed by #18
Closed

Adding a pipeline file example notebook #12

jlvdb opened this issue May 16, 2024 · 2 comments · Fixed by #18
Labels
documentation Improvements or additions to documentation help wanted Extra attention is needed question Further information is requested

Comments

@jlvdb
Copy link
Collaborator

jlvdb commented May 16, 2024

Since the yet_another_wizz wrapper is composed of five stages, I would like to include a pipeline.yaml file that can reproduce the currently existing notebook examples/full_example.ipynb.

I have created a simple notebook that is supposed to generate the pipeline file and then execute it. However I'm struggling to even get the first stage to connect to its input data: https://github.com/LSSTDESC/rail_yaw/blob/9dc44159c7ba38411c55302a5221380243d6bfcc/examples/pipeline.ipynb

When running the generated pipeline file, get this error, but I do not understand what I am doing wrong:

KeyError: 'cache_ref failed to get data by handle data_ref, associated to data'

@jlvdb jlvdb added documentation Improvements or additions to documentation help wanted Extra attention is needed question Further information is requested labels May 16, 2024
@eacharles
Copy link
Contributor

see the branch I added. Basically, you can used DataHandles to establish the connections between stages, but you need to be consistent by using them all the way through. In your notebook you made the input data in memory and didn't use a DataHandle to wrap it and that was causing the problem you saw.

@jlvdb jlvdb mentioned this issue Jul 11, 2024
4 tasks
@jlvdb jlvdb linked a pull request Jul 11, 2024 that will close this issue
4 tasks
@jlvdb
Copy link
Collaborator Author

jlvdb commented Jul 11, 2024

Fixes will be addressed in #16

@jlvdb jlvdb closed this as completed in #18 Jul 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation help wanted Extra attention is needed question Further information is requested
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants