Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use with x.get_frame instead of x.get_frame globally #171

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

LightArrowsEXE
Copy link
Member

This should help prevent some minor memory leaks. Since it only happens upon init, the memory leaks ultimately shouldn't get worse over time (unless called in i.e. a frame eval). However, since a lot of functions call get_prop, this might negatively impact memory for scripts that depend on a lot of these functions.

This should help prevent some minor memory leaks. Since it only happens upon init, the memory leaks ultimately shouldn't get worse over time (unless called in i.e. a frame eval). However, since a lot of functions call `get_prop`, this might negatively impact memory for scripts that depend on a lot of these functions.
vstools/utils/info.py Outdated Show resolved Hide resolved
Copy link

@petzku petzku left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The with blocks without any processing (just a pass) look weird, consider using explicit try/finally instead? I'm not fully sure what their purpose is in the first place, though.

@@ -60,7 +60,8 @@ def check(func: FuncExceptT, to_check: vs.VideoNode) -> None:
"""

try:
to_check.get_frame(0)
with to_check.get_frame(0):
pass
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considering this is already inside a try block, it might be cleaner to use an explicit finally instead?

        try:
            to_check.get_frame(0)
        except vs.Error as e:
        # -- snip --
        finally:
            to_check.close()

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't work as suggested. Works if I do something like this instead, though:

        try:
            frame = to_check.get_frame(0)
        except vs.Error as e:
            if 'no path between colorspaces' in str(e):
                raise InvalidColorspacePathError(func, e)
            raise
        finally:
            frame.close()

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, whoops. yeah, that's true

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, close needs to be on the returned frame, not the clip.

But: the adjusted snippet fails if get_frame actually throws, because then the assignment never happens.

@@ -169,10 +169,12 @@ def clear_cache() -> None:
try:
for output in get_outputs().values():
if isinstance(output, VideoOutputTuple):
output.clip.get_frame(0)
with output.clip.get_frame(0):
pass
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same try/finally idea?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing, though I found out this is apparantly valid

                if isinstance(output, VideoOutputTuple):
                    output.clip.get_frame(0).close()
                    break

Copy link

@petzku petzku Feb 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The (typical) problem with not using with or finally is that close() might not get called if an exception is raised. I'm not sure if this can happen with get_frame() alone, though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There’s nothing between the get_frame call and the close call, so no opportunities for other exceptions. If get_frame throws, there’s nothing to be closed. If it succeeds, you immediately close it anyway.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, get_frame().close() should be perfectly fine then

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ironically, I believe this is actually safer than a with or a tryfinally with an assignment if the code is running in the main thread of command-line python, because those constructs have extra bytecode ops (for the __exit__ call, the frame= assignment, etc.), each of which is an additional opportunity for a KeyboardInterrupt to occur.

break
except Exception:
core.std.BlankClip().get_frame(0)
with core.std.BlankClip().get_frame(0):
pass
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if applicable here. What even is the purpose of getting (and immediately discarding) a blank clip's first frame?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure. Figuring out why and whether there's a better method is out of the scope of this PR though imo

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function modifies VapourSynth’s cache limit, so perhaps this is just an arbitrary VapourSynth call to trigger an actual cache update.

@astiob
Copy link
Contributor

astiob commented Feb 3, 2025

The with blocks without any processing (just a pass) look weird, consider using explicit try/finally instead?

No try needed btw; just clip.get_frame(0).close()

@jsaowji
Copy link

jsaowji commented Feb 3, 2025

I don't think there is a memory leak here because a frame going out of scope is freed/closed automatically in its __dealloc__.
Copying the props is a good idea overall but even that at a glance looks like its not being kept around anywhere where it wouldn't be freed automatically.

@astiob
Copy link
Contributor

astiob commented Feb 3, 2025

I do see the code you mention in __dealloc__, but without the close calls, people are in fact seeing warnings like these upon exit:

Warning: Core freed but 63 filter instance(s) still exist
Warning: Core freed but 259850240 bytes still allocated in framebuffers
Warning: Core freed but 5 function instance(s) still exist

And personally, I feel I got an improvement in RAM usage when I replaced frames() calls with frames(close=True) in some batch-processing scripts I have that load and iterate over many clips in sequence, although admittedly, I didn’t rigorously measure this.

So I suspect the __dealloc__ doesn’t actually trigger or work!

@astiob
Copy link
Contributor

astiob commented Feb 3, 2025

…although apparently those warnings are present even with the close calls. Probably a red herring, then. Sorry.

Still, explicitly closing resources seems like good hygiene.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants