Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doc: Document usage of dynamic log links #6285

Merged
merged 1 commit into from
Feb 27, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,95 @@ task_logs:
- "https://..."
```

### Configure dynamic log links

Dynamic log links are links which are 1. not shown by default for all tasks and 2. which can use template variables provided during task registration.


Configure dynamic log links in the flytepropeller configuration the following way:

```yaml
configmap:
task_logs:
plugins:
logs:
dynamic-log-links:
- log_link_a: # Name of the dynamic log link
displayName: Custom dynamic log link A
templateUris: 'https://some-service.com/{{ .taskConfig.custom_param }}'
```

In `flytekit`, dynamic log links are activated and configured using a so-called `ClassDecorator`.
You can define such a custom decorator for controlling dynamic log links for instance as follows:

```py
from flytekit.core.utils import ClassDecorator
class configure_log_links(ClassDecorator):
"""
Task function decorator to configure dynamic log links.
"""
def __init__(
self,
task_function: Optional[Callable] = None,
enable_log_link_a: Optional[bool] = False,
custom_param: Optional[str] = None,
**kwargs,
):
"""
Configure dynamic log links for a task.
Args:
task_function (function, optional): The user function to be decorated. If the decorator is called
with arguments, task_function will be None. If the decorator is called without arguments,
task_function will be function to be decorated.
enable_log_link_a (bool, optional): Activate dynamic log link `log_link_a` configured in the backend.
custom_param (str, optional): Custom parameter for log link templates configured in the backend.
"""
self.enable_log_link_a = enable_log_link_a
self.custom_param = custom_param
super().__init__(
task_function,
enable_log_link_a=enable_log_link_a,
custom_param=custom_param,
**kwargs,
)
def execute(self, *args, **kwargs):
output = self.task_function(*args, **kwargs)
return output
def get_extra_config(self) -> dict[str, str]:
"""Return extra config for dynamic log links."""
extra_config = {}

log_link_types = []
if self.enable_log_link_a:
log_link_types.append("log_link_a")

if self.custom_param:
extra_config["custom_param"] = self.custom_param
# Activate other dynamic log links as needed

extra_config[self.LINK_TYPE_KEY] = ",".join(log_link_types)
return extra_config


@task
@configure_log_links(
enable_log_link_a=True,
custom_param="test-value",
)
def my_task():
...

```
For inspiration, consider how the flytekit [wandb](https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-wandb/flytekitplugins/wandb/tracking.py), [neptune](https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-neptune/flytekitplugins/neptune/tracking.py) or [vscode](https://github.com/flyteorg/flytekit/blob/master/flytekit/interactive/vscode_lib/decorator.py) plugins make use of dynamic log links.
## Datadog integration
To send your Flyte workflow logs to Datadog, you can follow these steps:
Expand Down
Loading