Skip to content

[Bug]: java.io.IOException: Cannot run program ... Argument list too long when generating pex #542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
phoenixuprising opened this issue Mar 29, 2025 · 0 comments · May be fixed by #543
Open

[Bug]: java.io.IOException: Cannot run program ... Argument list too long when generating pex #542

phoenixuprising opened this issue Mar 29, 2025 · 0 comments · May be fixed by #543
Labels
bug Something isn't working

Comments

@phoenixuprising
Copy link

phoenixuprising commented Mar 29, 2025 •
edited
Loading

What happened?

I had an existing py_binary I was originally building with rules_python and I switched to using aspect_rules_py to generate a pex file. The py_binary depends on just google-api-core, google-auth, google-cloud-secret-manager, bazel-runfiles, and pyyaml packages, and contained a total of 5 source files. When compiling the pex target, it fails with:

ERROR: /Users/chayes/Development/project_root/foo/bar/BUILD:75:21: Building PEX binary //foo/bar:baz_pex failed: (Exit -1): pex failed: error executing PyPex command (from target //foo/bar:baz_pex) bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/aspect_rules_py+/py/tools/pex/pex ... (remaining 19428 arguments skipped)
...
Action failed to execute: java.io.IOException: Cannot run program "/usr/bin/sandbox-exec" (in directory "/private/var/tmp/_bazel_chayes/8fa303648e57c6deaed2bf40b3eacda1/sandbox/darwin-sandbox/3/execroot/_main"): error=7, Argument list too long

When inspecting the logs, the arguments fed into the _pex executable is full of:

'--source=external/+http_archive+gcloud_darwin_arm64/google-cloud-sdk/platform/gsutil/third_party/urllib3/docs/reference/urllib3.fields.rst=+http_archive+gcloud_darwin_arm64/google-cloud-sdk/platform/gsutil/third_party/urllib3/docs/reference/urllib3.fields.rst' '--source=external/+http_archive+gcloud_darwin_arm64/google-cloud-sdk/platform/gsutil/third_party/urllib3/docs/reference/urllib3.poolmanager.rst=+http_archive+gcloud_darwin_arm64/google-cloud-sdk/platform/gsutil/third_party/urllib3/docs/reference/urllib3.poolmanager.rst' '--source=external/+http_archive+gcloud_darwin_arm64/google-cloud-sdk/platform/gsutil/third_party/urllib3/docs/reference/urllib3.request.rst=+http_archive+gcloud_darwin_arm64/google-cloud-sdk/platform/gsutil/third_party/urllib3/docs/reference/urllib3.request.rst'

There were 4.4mb worth of args being passed in total. There is probably another bug in here with regards to how it is passing --source for every file of every package (and transitive package) which is ultimately led to the argument list too long error. I'll see if I can sus out that bug in the next few days and provide a repro case.

Version

Development (host) and target OS/architectures:
MacOS
Darwin chayes.local 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:05:23 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6031 arm64

Output of bazel --version:
bazel 9.0.0-pre.20250307.1

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:
1.3.2
Language(s) and/or frameworks involved:
Python

How to reproduce



Any other information?


No response

      		
    

      
  





        



            
            

              
            

        

      


      
    








      

    



      

  
            

    

      
    

    


      

          @phoenixuprising
phoenixuprising




          added
  the 

  bug

  Something isn't working
 label


      Mar 29, 2025

    

  



  


  
    
  
  

    
  phoenixuprising 

      added a commit
        to phoenixuprising/rules_py
      that referenced
      this issue

    
      Mar 29, 2025
    
      
        
      
    

      


  

    

      


  

      
        @phoenixuprising
  





      

        
          fix `java.io.IOException: Cannot run program ... Argument list too lo…
        

          
            
          

      


      

          

    
    
    

  

      


      

      


      
      

        
          32bb41d
        
      

    

  

    

      
…ng` during pex generation.

In certain circumstance, a large number of `--source` args are passed to
the `_pex` script which causes an IOException to be thrown by bazel.
This refactors the rule to write the args to a file and pass that single
path as an arg to the `_pex` script. The script then opens that file and
feeds it into the arg parser instead.

fixes aspect-build#542

    




    

  




  

  

    
  


  

      

    @phoenixuprising
phoenixuprising



    linked a pull request
    
      Mar 29, 2025
    

    that will
    close
    this issue





  


  

      
   Open

  







  













  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

    No one assigned







      


  


  

    Labels
  



    

      

    bug

  Something isn't working








      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      No milestone





    
      
          


  

    

      
        

          
    

    Development
  




            
  
Successfully merging a pull request may close this issue.





    


  
          
      fix java.io.IOException: ... Argument list too long during pex generation.

    
      phoenixuprising/rules_py
    





      
    

  




      

    

  
      

  

    

      1 participant
    

    

        
          @phoenixuprising