How can I reset pending changes?

[hicksjacobp]

We have a component that seems to be having issues.

Originally, the git repository had diverged due to concurrent translation commits and development commits. In an attempt to resolve this, I generally took the following steps:

1. Locked the component
2. Cloned/fetched the weblate and origin/upstream repositories locally.
3. Performed the merge.
4. Pushed to origin/upstream.
5. Fetched/reset weblate to the latest from origin/upstream.

NOTE: about generally - I tried various steps with the Weblate Web UI, wlc, and eventually shell access. At various times, operations would time out or fail. I wouldn't be surprised if some of this affects my current situation.

So now I have a component which is currently locked, has 0 outgoing commits, and 0 missing commits. Yet, Weblate indicates there are pending changes. Early on this in process, I noticed this number as being much larger than the actual translation changes that have been made by translators (only a handful of changes were made, but the pending changes are >10,000). After trying to just get repos in sync, I now noticed the pending changes >1,000,000. All this while, the component has been locked and no one has been able to contribute translation changes.

I've tried performing the "cleanup" and "reset" operations via wlc (really the REST API since wlc is having some certificate issues due to my environment). They seem to operate successfully, yet the pending changes still sit >1,000,000. I'm not confident that this component is ready to be unlocked - I would expect the pending changes to be at 0 if I've performed a cleanup/reset.

Like noted above, at various times some operations time out or end up returning a bad gateway [when being done via the Web UI]. Sometimes on the shell, performing a checkgit command fails to acquire the lock for the component by timing out. Other times, it works fine.

I'm betting there are some settings or logs that may help give some insight. If anyone could identify what specifically to change and gather, I'd appreciate it. Ultimately what I'm looking for is if someone knows a way to really get this component back to a good state.

Thanks!

[nijel]

Maybe you did click on "Force synchronization"? That could mark all strings as pending.

The question is: Why these were not written out - do you see any errors in the logs? Is Celery process running?

The cleanup and reset operate on the repository only, so these do not reset changes in the database.

[hicksjacobp]

I fixed the permissions to match what the other files/directories have and now the Performance Report doesn't have any messages regarding file ownership issues.

I tried committing via the REST API, but it ended up returning 502 Bad Gateway after waiting a very long time. I still occasionally get an Internal Server Error when attempting to access the Repository maintenance screen in the UI.

It looks like we don't have logging being handled with any provider that I can inspect, so we'll have to enable it and then analyze appropriately.

Thank you so far for the assistance. I'll report back once we have more information or have it resolved.

[hicksjacobp]

Sooo I found that our logs weren't enabled and we'd have to enable and restart the docker container. Without having some others involved at the moment, I can't pursue this today.

I tried to give removing the component a shot. After doing so, Weblate is now complaining that celery is not processing or too slow (E019). The celery queue has a growing number, while all others remain at 0. Via the shell, running celery status doesn't seem to return. I'm not an expert with celery so any guidance would be helpful.

[nijel]

The Docker container logs in a usual way, see https://docs.weblate.org/en/latest/contributing/debugging.html#weblate-logs

[hicksjacobp]

Thank you, the docker-compose logs definitely outputs some information. Will this docker logs output include information regarding celery? Since I wasn't able to run this until just recently, a lot of the information is likely stale and difficult to parse through for when I was actually encountering issues.

After re-adding the component, the state is good now, but we'll keep an eye on it to see if anything happens again. It is the largest component we have and I know we've had performance issues with it before.

I'm not sure if I should really mark this thread as resolving the answer because we never really figured out the root cause. But if it's the only way to close out this "Q&A", then I understand.

[nijel]

Yes, it contains logs from both uwsgi and celery processes.