Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add partial eval platform #252

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

add partial eval platform #252

wants to merge 1 commit into from

Conversation

jackaldenryan
Copy link

@jackaldenryan jackaldenryan commented Jan 10, 2025

Structure of this is as follows:

  • The snippetization notebook is for loading the longmemeval data set and doing a one time reformatting of the LME data set into "snippets". This notebook saves all of this data, which is very large, into a newly created folder, which is git-ignored
  • The mini-dataset creation notebook is for filtering and formatting some specific "mini-dataset" which might be for instance 10 snippets. This notebook is also responsible for calling the utility function for ingesting and labeling, since part of the mini data set itself includes GPT-4o labels. The mini-dataset will have two forms: the internal eval CSV, used for the actual CLI test, and then the CSV used for human labeling, which will ultimately be imported into a Google sheet
  • The eval_extract_nodes file is for the CLI test. It is currently outdated and will eventually be updated to use the same ingestion and labeling utility function.

Copy link


Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant