fix: _get_folder_size fn

[deependujha]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

This PR addresses comment: #468 (comment)

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[tchaton]

The idea is not use os.stat which is expensive and not walk the folder.

Instead, we check only the cache folder, not its parents, list the file and use the config to estimate the size

[bhimrazy]

Instead, we check only the cache folder, not its parents, list the file and use the config to estimate the size

yes, while debugging to understand the sizes, I could see it was considering the other dirs(for eg: remote_dir) in the tests

[tchaton]

Note

[deependujha]

sorry that was just to make the pr, I was working on using config dict.

[bhimrazy]

@deependujha, It seems the failing tests are correct because, previously, the size calculation included the remote_dir as well, whereas now it only considers the cache_dir size.
Since the max_cache_size remains unchanged, this results in more chunks being stored in the cache directory.

To address this, I think we could either reduce the max_cache_size to around 1300(or less) or update the number of files we check.

Incase we donot consider the index.json file then it would be just 48-50 bytes. The size of index.json file itself seems to be around 1317 bytes