-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Change ingestion input to JSON lines format #639
Conversation
Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: Tyler Ohlsen <[email protected]>
@ohltyler Can we add a learn more external link in the helper text? Users who are not familiar with JSON lines can learn more about it. |
Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: Tyler Ohlsen <[email protected]>
Yes, good idea :) |
Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: Tyler Ohlsen <[email protected]>
@ohltyler Nice, does the list update dynamically if users fixes an error or a new error came up? |
The list is cleared until users click out of focus again - this keeps any performance or lag issues from occurring, in addition to low-level details like line numbers becoming stale as users make changes. |
@ohltyler Do you have a quick demo video on multiple error messages? Inline validations feels much more clear and intuitive to me. Although this coder editor doesn't support inline validation, we should invest into an editor that support modern code editing patterns. However, this shouldn't be a blocker to JSON lines. |
screen-capture.27.webmSure. And strongly agree, which is why I've brought this up again in opensearch-project/OpenSearch-Dashboards#2875 |
Signed-off-by: Tyler Ohlsen <[email protected]> (cherry picked from commit a484199) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
(cherry picked from commit a484199) Signed-off-by: Tyler Ohlsen <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Description
This PR updates the ingestion input format to JSON lines format: https://jsonlines.org/. This is a more standard data file format for persisting JSON data, and makes it easier for importing.
More details:
JsonLinesField
component, which is similar toJsonField
but has specific logic around error handling and formatting to be specific to JSON lines.*.jsonl
instead of JSON inputsprocessorConfigsToTemplateProcessors
. Externally, this has no impact.Demo video, showing the new input type, validation, error handling, and successful ingestion:
screen-capture.26.webm
Check List
--signoff
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.