Rubygems is installing the precompiled C extensions for MRI on TruffleRuby

[rafaelfranca]

When installing a gem that provides precompiled binaries, as for example google-protobuf, RubyGems is installing the MRI binary even though it only works for MRI and not TruffleRuby.

It seems that it should be aware of the Ruby engine being used and not download the compiled gem if the engine used to compile it is not the same as the local engine.

I'm willing to fix the problem but I could not find the place where RubyGems detect if the platform the gem was built is the same as the one it is being installed, so any pointers would be appreciated.

Also, the Gem::Platform object lacks information about the RUBY_ENGINE being used in the local platform and in the spec platform, so I'm not sure if we will need a change to rubygems.org to provide this information on the spec that were already pushed in order to fix this issue.

To reproduce:

Using TruffleRuby do:

$ gem install google-protobuf
Fetching google-protobuf-3.9.2-universal-darwin.gem
Successfully installed google-protobuf-3.9.2-universal-darwin
1 gem installed

You will see the universal-darwin or any OS specific version being installed if your OS is one of the supported by that gem.

I also searched for, but I could not find a way to tell RubyGems to ignore binary gems and always compile the C extensions. Do you know if we have one?

This issue is related to:

• Network problems
• Installing a library
• Publishing a library
• The command line gem
• Other

Here are my current environment details:

$ gem env version
RubyGems Environment:
  - RUBYGEMS VERSION: 3.0.3
  - RUBY VERSION: 2.6.2 (2019-10-03 patchlevel 0) [x86_64-darwin]
  - INSTALLATION DIRECTORY: /Users/rafaelfranca/.gem/truffleruby/2.6.2
  - USER INSTALLATION DIRECTORY: /Users/rafaelfranca/.gem/truffleruby/19.3.0-dev-7d425d48
  - RUBY EXECUTABLE: /Users/rafaelfranca/src/truffleruby/mxbuild/truffleruby-jvm/jre/languages/ruby/bin/truffleruby
  - GIT EXECUTABLE: /opt/dev/bin/user/git
  - EXECUTABLE DIRECTORY: /Users/rafaelfranca/.gem/truffleruby/2.6.2/bin
  - SPEC CACHE DIRECTORY: /Users/rafaelfranca/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /Users/rafaelfranca/src/truffleruby/etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-darwin
  - GEM PATHS:
     - /Users/rafaelfranca/.gem/truffleruby/2.6.2
     - /Users/rafaelfranca/src/truffleruby/mxbuild/truffleruby-jvm/jre/languages/ruby/lib/gems
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
     - "gem" => "--no-doc"
     - :benchmark => false
     - :sources => ["https://rubygems.org"]
  - REMOTE SOURCES:
     - https://rubygems.org
  - SHELL PATH:
     - /Users/rafaelfranca/.gem/truffleruby/2.6.2/bin
     - /Users/rafaelfranca/src/truffleruby/mxbuild/truffleruby-jvm/jre/languages/ruby/lib/gems/bin
     - /opt/rubies/truffleruby/bin
     - /Users/rafaelfranca/.nix-profile/bin
     - /Users/rafaelfranca/.yarn/bin
     - /opt/dev/bin/user
     - /Users/rafaelfranca/.nix-profile/bin
     - /Users/rafaelfranca/.rbenv/shims
     - /usr/local/bin
     - /usr/bin
     - /bin
     - /usr/sbin
     - /sbin
     - /Library/Apple/usr/bin
     - /Library/Apple/bin
     - /usr/local/MacGPG2/bin
     - /opt/X11/bin

I will abide by the code of conduct.

[duckinator]

I, uh, may have found another bug while investigating this one: gem install $X; gem uninstall $X claims it successfully installed $X, then claims it wasn't installed.

console logs

~$ gem install google-protobuf                                                                                         
Successfully installed google-protobuf-3.9.2-x86_64-linux
1 gem installed                                            
~$ gem uninstall google-protobuf                                                                                       
Gem 'google-protobuf' is not installed
~$ gem install inq                                         
Fetching faraday-0.17.0.gem                                
Fetching descendants_tracker-0.0.4.gem    
Fetching thread_safe-0.3.6.gem          
Fetching multi_json-1.13.1.gem          
Fetching multi_xml-0.6.0.gem                               
Fetching jwt-2.2.1.gem                                     
Fetching rack-2.0.7.gem                                    
Fetching multipart-post-2.1.1.gem     
Fetching hashie-3.6.0.gem                                  
Fetching public_suffix-4.0.1.gem                           
Fetching addressable-2.7.0.gem                                                                                         
Fetching github_api-0.18.2.gem                             
Fetching openssl-better_defaults-0.0.2.gem
Fetching cacert-0.5.0.gem
Fetching okay-11.0.0.gem
Fetching json_pure-2.2.0.gem
Fetching oauth2-1.4.2.gem
Fetching inq-26.0.0.gem
Successfully installed thread_safe-0.3.6
Successfully installed descendants_tracker-0.0.4
Successfully installed multipart-post-2.1.1
Successfully installed faraday-0.17.0
Successfully installed jwt-2.2.1
Successfully installed multi_json-1.13.1
Successfully installed multi_xml-0.6.0
Successfully installed rack-2.0.7
Successfully installed oauth2-1.4.2
Successfully installed hashie-3.6.0
Successfully installed public_suffix-4.0.1
Successfully installed addressable-2.7.0
Successfully installed github_api-0.18.2
Successfully installed openssl-better_defaults-0.0.2
Successfully installed cacert-0.5.0
Successfully installed okay-11.0.0
Successfully installed json_pure-2.2.0
Successfully installed inq-26.0.0
18 gems installed
~$ gem uninstall inq                                        
Gem 'inq' is not installed
~$ gem install cacert
Successfully installed cacert-0.5.0
1 gem installed
~$ gem uninstall cacert
Gem 'cacert' is not installed
~$

[duckinator]

Unfortunately, after like 3 hours of digging, I've also been unable to do much more than confirm what you said.

1. It seems like it needs to be aware of the Ruby engine used to build it.
2. If it is aware of this, this information doesn't seem to be being used.
3. Gem::Platform.local doesn't include information about RUBY_ENGINE

However, I found a possible workaround until this is properly fixed: if you pass --platform <some platform> and specify a platform that there isn't a binary for (e.g. gem install --platform ruby google-protobuf), it will try to build it.

I was unable to get it to successfully build google-protobuf, but I'm not sure if that's a RubyGems problem, a limitation of my workaround, or a proper google-protobuf+TruffleRuby incompatibility.

[rafaelfranca]

Yeah, TruffleRuby is not able to build google-protobuf yet, but that work around already help me a lot. Thank you for that.

[eregon]

@rafaelfranca Thank you for opening this issue, I'm looking at that as well as part of truffleruby/truffleruby#1837

As a short term workaround we can probably just ignore precompiled gems on TruffleRuby, but it would be good to consider RUBY_ENGINE because different Ruby implementations often (always?) have different ABI (and macro definitions).

I wondered how Rubinius handles this, apparently they don't:

$ chruby rbx-4.5
$ gem install google-protobuf
Fetching: google-protobuf-3.11.2-x86_64-linux.gem (100%)
Successfully installed google-protobuf-3.11.2-x86_64-linux
1 gem installed
$ ruby -rgoogle/protobuf -e 'p Google.constants'
...
Could not open library ~/.rubies/rbx-4.5/gems/gems/google-protobuf-3.11.2-x86_64-linux/lib/google/2.3/protobuf_c.so -
~/.rubies/rbx-4.5/gems/gems/google-protobuf-3.11.2-x86_64-linux/lib/google/2.3/protobuf_c.so:
undefined symbol: rb_cObject (LoadError)

[eregon]

I found a workaround by setting the Gem.platforms in the TruffleRuby-specific RubyGems setup to only [Gem::Platform::RUBY]. That seems to work OK, but might break a few RubyGems tests.

[eregon]

truffleruby/truffleruby@6da9d7c is the workaround as mentioned above, it seems to work well.

It would still be better to consider RUBY_ENGINE in Gem::Platform though, as that would allow having gems precompiled for TruffleRuby.