Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce impact of scan for a db large tables #3

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

GarethSpace48
Copy link

uses a query to generate a query to run an RLIKE on all fields in the table, so that you only get rows return with encrypted values (and not every row), thus reducing the memory impact

uses an query to generate a query to run an RLIKE on all fields in the table, so that you only get rows return with encrypted values (and not every row), thus reducing the memory impact
@VladyslavSikailo
Copy link

worked for me in the same way as the original solution ✅

@vitaliy-golomoziy
Copy link
Collaborator

vitaliy-golomoziy commented Jul 25, 2024

In a new version we got rid of fetchAll and scanning the whole database record by record (see #7). I'm afraid that for large tables, a series of RLIKE queries might be very slow and introduce an extra load on the database (like querying search terms, or something like that). We would like to minimise the impact of the scan operation on a live environment (assuming that the script can be executed on a separate/cron/admin instance and won't impact production operations).
The proposed solution is interesting, we may incorporate it as an algorithm of choice, something like --algorithm=php or --algorithm=sql.

Do you have any measurements on time/memory/cpu of a php-based version versus sql-based?

It would be very interesting to see such measurements.

@GarethSpace48
Copy link
Author

Hey, sorry, been a wild couple of weeks @vitaliy-golomoziy

I didn't have any metrics, I can put some together though if you want.

I did spot a problem with this, I couldn't use the SQL in this PR on one of the clients because of an different MySQL version (I haven't gone back and worked out why, but for that client I ran the version without fetchAll)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants