Semantic model perf lab #426
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eisber
requested changes
Jan 30, 2025
| return str(self._properties) | ||
|
|
||
|
|
||
| def _get_or_create_workspace( |
There was a problem hiding this comment.
should move to common utils.
| create_abfss_path, | ||
| ) | ||
|
|
||
| class PropertyBag: |
| raise ValueError("For new workspaces, the workspace parameter must be string, not a Guid. Please provide a workspace name.") | ||
| except ValueError: | ||
| # OK, it's not a Guid. But also make sure the workspace parameter isn't empty. | ||
| if workspace == "" or workspace is None: |
There was a problem hiding this comment.
just use if workspace:
https://stackoverflow.com/questions/9573244/how-to-check-if-the-string-is-empty-in-python
| print(f"{icons.green_dot} Workspace '{workspace_name}' created.") | ||
| return (workspace,workspace_id) | ||
|
|
||
| def _get_or_create_lakehouse( |
| """ | ||
|
|
||
| # Treat empty strings as None. | ||
| if lakehouse == "": |
There was a problem hiding this comment.
not sure why this is needed.
|
|
||
| try: | ||
| (target_workspace_name, target_workspace_id) = resolve_workspace_name_and_id(workspace=target_workspace) | ||
| except: |
There was a problem hiding this comment.
catch specific exception, otherwise we mask errors
|
|
||
| return spark.createDataFrame(rows, schema=schema).dropDuplicates() | ||
|
|
||
| def _get_min_max_keys( |
| def get_storage_table_column_segments( | ||
| test_cycle_definitions: DataFrame, | ||
| tables_info: DataFrame | ||
| )->DataFrame: |
There was a problem hiding this comment.
run black to get formatting
|
|
||
| try: | ||
| (target_workspace_name, target_workspace_id) = resolve_workspace_name_and_id(workspace=target_workspace) | ||
| except: |
| ] | ||
|
|
||
|
|
||
| def _get_test_definitions( |
There was a problem hiding this comment.
the dataframe seems small, so I'd suggest not to use a pandas dataframe.
since you want to subsequently persist it (and maybe re-hydrate), I think a separate class would make it more convenient.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Functions to provision and run a Semantic Model Perf Lab for continuous testing and ad-hoc investigations.