Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid the multiprocessing forkserver method in Python 3.14 #3442

Merged
merged 1 commit into from
Mar 12, 2025

Conversation

befeleme
Copy link
Contributor

Pull Request check-list

Please make sure to review and check all of these items:

  • Do tests and lints pass with this change?
  • Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
  • Is the new or changed code fully tested?
  • Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
  • Is there an example added to the examples folder (if applicable)?
  • Was the change added to CHANGES file?

NOTE: these things are not required to open a PR and can be done
afterwards / while the PR is open.

Description of change

This change is essentially a copy of: giampaolo/pyftpdlib#656

Python 3.14 has changed the default multiprocessing method from fork to forkserver which causes failures like this one:

____________________ TestMultiprocessing.test_redis_client _____________________

self = <tests.test_multiprocessing.TestMultiprocessing object at 0x7f646b43d250>
r = <redis.client.Redis(<redis.connection.ConnectionPool(<redis.connection.Connection(host=localhost,port=6379,db=0)>)>)>

    def test_redis_client(self, r):
        "A redis client created in a parent can also be used in a child"
        assert r.ping() is True
    
        def target(client):
            assert client.ping() is True
            del client
    
        proc = multiprocessing.Process(target=target, args=(r,))
>       proc.start()

tests/test_multiprocessing.py:163: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.14/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/usr/lib64/python3.14/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/usr/lib64/python3.14/multiprocessing/context.py:300: in _Popen
    return Popen(process_obj)
/usr/lib64/python3.14/multiprocessing/popen_forkserver.py:35: in __init__
    super().__init__(process_obj)
/usr/lib64/python3.14/multiprocessing/popen_fork.py:20: in __init__
    self._launch(process_obj)
/usr/lib64/python3.14/multiprocessing/popen_forkserver.py:47: in _launch
    reduction.dump(process_obj, buf)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <Process name='Process-9' parent=996 initial>
file = <_io.BytesIO object at 0x7f646bbee390>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       _pickle.PicklingError: Can't pickle local object <function TestMultiprocessing.test_redis_client.<locals>.target at 0x7f646bb0f530>
E       when serializing dict item '_target'
E       when serializing multiprocessing.context.Process state
E       when serializing multiprocessing.context.Process object

I verified this makes tests pass with Python 3.14.0a2 in Fedora's build environment

@petyaslavova
Copy link
Collaborator

@befeleme please check linters failure

@befeleme befeleme force-pushed the py314-compat branch 2 times, most recently from c6b9eb1 to 5faa54e Compare March 11, 2025 12:33
@befeleme
Copy link
Contributor Author

Hopefully mitigated.

@befeleme
Copy link
Contributor Author

One more amended change: ' -> "

@petyaslavova
Copy link
Collaborator

Hi @befeleme, the linters are failing. You can run them locally to check if there is something else than the one reported from the pipeline, the instructions are available in the CONTRIBUTING.md file

@petyaslavova petyaslavova merged commit ea01a30 into redis:master Mar 12, 2025
41 checks passed
@befeleme befeleme deleted the py314-compat branch March 12, 2025 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants