Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittently Errno::ENOENT errors #98

Closed
f1sherman opened this issue May 16, 2024 · 10 comments · Fixed by #121
Closed

Intermittently Errno::ENOENT errors #98

f1sherman opened this issue May 16, 2024 · 10 comments · Fixed by #121

Comments

@f1sherman
Copy link

We intermittently (a few times a week) see Errno::ENOENT errors when calling CodeOwnership.for_file where code ownership claims that a file we're looking up ownership for doesn't exist. It happens for different paths, not always the same path. There doesn't seem to be any obvious pattern for when this happens. The file exists and is found on 99.9% of requests. I've included the gem-specific portion of the trace below. Any ideas on how to track this down? For now we've added a begin/rescue block around it, but obviously that's not an ideal solution. It's really weird given that I can pass a path that definitely doesn't exist to CodeOwnership.for_file and it doesn't error, just returns nil.

The error has the signature Errno::ENOENT: No such file or directory @ rb_sysopen - path/to/file/that/exists.rb.

/app/vendor/bundle/ruby/3.3.0/gems/code_ownership-1.36.2/lib/code_ownership/private/ownership_mappers/file_annotations.rb:86:in `foreach'
,
/app/vendor/bundle/ruby/3.3.0/gems/code_ownership-1.36.2/lib/code_ownership/private/ownership_mappers/file_annotations.rb:86:in `each'
,
/app/vendor/bundle/ruby/3.3.0/gems/code_ownership-1.36.2/lib/code_ownership/private/ownership_mappers/file_annotations.rb:86:in `first'
,

/app/vendor/bundle/ruby/3.3.0/gems/code_ownership-1.36.2/lib/code_ownership/private/ownership_mappers/file_annotations.rb:86:in `file_annotation_based_owner'
,
/app/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11372/lib/types/private/methods/call_validation_2_7.rb:968:in `bind_call'
,
/app/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11372/lib/types/private/methods/call_validation_2_7.rb:968:in `block in create_validator_method_medium1'
,
/app/vendor/bundle/ruby/3.3.0/gems/code_ownership-1.36.2/lib/code_ownership/private/ownership_mappers/file_annotations.rb:29:in `map_file_to_owner'
,
/app/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11372/lib/types/private/methods/call_validation_2_7.rb:968:in `bind_call'
,
/app/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11372/lib/types/private/methods/call_validation_2_7.rb:968:in `block in create_validator_method_medium1'
,
/app/vendor/bundle/ruby/3.3.0/gems/code_ownership-1.36.2/lib/code_ownership.rb:41:in `block in for_file'
,
/app/vendor/bundle/ruby/3.3.0/gems/code_ownership-1.36.2/lib/code_ownership.rb:40:in `each'
,
/app/vendor/bundle/ruby/3.3.0/gems/code_ownership-1.36.2/lib/code_ownership.rb:40:in `for_file'
,
/app/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11372/lib/types/private/methods/call_validation_2_7.rb:968:in `bind_call'
,
/app/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11372/lib/types/private/methods/call_validation_2_7.rb:968:in `block in create_validator_method_medium1'
@tstannard
Copy link
Contributor

Hi Brian!

It looks like the Errno::ENOENT error you are getting is indicating a file does not exist even though you can see it in your directory.

This also seems weird since on line 78 we check that the file exists: return unless File.file?(filename)

Do you think there is something in the background that is moving or renaming files?

@f1sherman
Copy link
Author

Yes it is super weird. I can't think of anything that would move or rename files, and we don't see anything like this anywhere else in our error monitoring. Any tips on how we could troubleshoot it?

@ashleywillard
Copy link
Contributor

Hi @f1sherman! I want to check in and see if you're still experiencing this issue. Recently, one of the folks on our team made some improvements to this repo, and those changes may or may not impact this issue. We haven't been able to reproduce this issue, but we'll continue to keep an eye out for it.

@f1sherman
Copy link
Author

@ashleywillard thanks for the heads up! I haven't seen this crop up in at least 2 weeks. I'll try removing our code that swallows the error and report back here if it comes back. Thank you!

@f1sherman
Copy link
Author

@ashleywillard unfortunately this error came back after I removed our patch :(.

@ashleywillard
Copy link
Contributor

Darn! We still have yet to see this error but I will let you know if we do. It seems like this issue might be tricky to reproduce on our end.

@timlkelly
Copy link
Contributor

I recently started experiencing this issue as well. It happened when I reorganized some of our internal gems within our mono-repo. We haven't had this problem at all previously.

When it started, we had CodeOwnership on version 1.36.2, and I upgraded to 1.38.3, but the error persisted. It seems intermittent but happens at least once a day on weekdays, and I've seen as many as seven errors in a single day. I'm assuming the lower weekend traffic is why I see zero or only one error on Saturday/Sunday.

Is there anything you might recommend to look into to help debug?

martinemde added a commit that referenced this issue Feb 28, 2025
Errno::ENOENT is occuring intermittently for some users.
This can happen even when the file seems to exist at the time.
Since we were already returning quietly if a file did not exist, just
try to read the file and rescue exceptions, returning silently.

Fixes #98
@martinemde
Copy link
Contributor

Hi @f1sherman and @timlkelly,

Are you seeing these errors on CI builds? In containers? The list of reasons why we'd see an ENOENT for a file that does exist, but for different files each time, is a pretty small set of possible causes (race conditions, filesystem IO problems, etc).

Are the files that are raising ones that you would expect to have annotations?

We pushed #121 to solve this by rescuing for now since it's hard to diagnose exactly why this is happening.

@f1sherman
Copy link
Author

I have not seen it in CI, only in running Kubernetes pods in Production.

@timlkelly
Copy link
Contributor

Same, I have not seen this in CI. Only in production, also Kubernetes pods.

Do you mean the File Annotation ownership? No, the files I've seen so far in the error reports are Package-based Ownership.

martinemde added a commit that referenced this issue Mar 1, 2025
Errno::ENOENT is occuring intermittently for some users.
This can happen even when the file seems to exist at the time.
Since we were already returning quietly if a file did not exist, just
try to read the file and rescue exceptions, returning silently.

Fixes #98
martinemde added a commit that referenced this issue Mar 1, 2025
Errno::ENOENT is occuring intermittently for some users.
This can happen even when the file seems to exist at the time.
Since we were already returning quietly if a file did not exist, just
try to read the file and rescue exceptions, returning silently.

Fixes #98
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants