Support new OpenSSH format private keys with passphrase

[akito19]

Hi, all

I faced a problem when connecting my SFTP server with new OpenSSH format.

Environment

% embulk gem list | grep embulk
embulk (0.9.18 java)
embulk-input-sftp (0.3.2)

Overview

Connecting to SFTP server failed when executing embulk guess seed.yml -o config.yml by using OpenSSH private key accompanying passphrase.

Problem

OpenSSH 6.5 released new private key format when ssh-keygen and the format has been default in OpenSSH 7.8 since last year.

And then, if new default format is set, embulk processes are failed.
For example, when I setup SFTP server and tried executing Embulk, I received rg.apache.commons.vfs2.FileSystemException: Could not connect to SFTP server and Could not connect to SFTP server.

% embulk guess seed.yml -o config.yml

java.lang.RuntimeException: org.apache.commons.vfs2.FileSystemException: Could not connect to SFTP server at "sftp://prev_header@{host}/".
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.embulk.input.sftp.SftpFileInput.listFilesByPrefix(SftpFileInput.java:303)
        at org.embulk.input.sftp.SftpFileInputPlugin.transaction(SftpFileInputPlugin.java:23)
        at org.embulk.spi.FileInputRunner.transaction(FileInputRunner.java:62)
        at org.embulk.exec.SamplingParserPlugin.runFileInputSampling(SamplingParserPlugin.java:47)
        at org.embulk.spi.FileInputRunner.guess(FileInputRunner.java:80)
        at org.embulk.exec.GuessExecutor.doGuess(GuessExecutor.java:108)
        at org.embulk.exec.GuessExecutor.access$000(GuessExecutor.java:32)
        at org.embulk.exec.GuessExecutor$1.run(GuessExecutor.java:81)
        at org.embulk.exec.GuessExecutor$1.run(GuessExecutor.java:78)
        at org.embulk.spi.Exec.doWith(Exec.java:22)
        at org.embulk.exec.GuessExecutor.guess(GuessExecutor.java:78)
        at org.embulk.EmbulkEmbed.guess(EmbulkEmbed.java:183)
        at org.embulk.EmbulkRunner.guessInternal(EmbulkRunner.java:203)
        at org.embulk.EmbulkRunner.guess(EmbulkRunner.java:60)
        at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:427)
        at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
        at org.embulk.cli.Main.main(Main.java:64)
Caused by: org.apache.commons.vfs2.FileSystemException: Could not connect to SFTP server at "sftp://prev_header@{host}/".
        at org.apache.commons.vfs2.provider.sftp.SftpFileProvider.doCreateFileSystem(SftpFileProvider.java:86)
        at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.getFileSystem(AbstractOriginatingFileProvider.java:92)
        at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:71)
        at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:55)
        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:711)
        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:648)
        at org.embulk.input.sftp.SftpFileInput$1.call(SftpFileInput.java:226)
        at org.embulk.input.sftp.SftpFileInput$1.call(SftpFileInput.java:205)
        at org.embulk.spi.util.RetryExecutor.run(RetryExecutor.java:81)
        at org.embulk.spi.util.RetryExecutor.runInterruptible(RetryExecutor.java:62)
        at org.embulk.input.sftp.SftpFileInput.listFilesByPrefix(SftpFileInput.java:205)
        ... 16 more
Caused by: org.apache.commons.vfs2.FileSystemException: Could not load private key from "org.apache.commons.vfs2.provider.sftp.IdentityInfo@59cde35".
        at org.apache.commons.vfs2.provider.sftp.SftpClientFactory.addIndentity(SftpClientFactory.java:174)
        at org.apache.commons.vfs2.provider.sftp.SftpClientFactory.addIdentities(SftpClientFactory.java:157)
        at org.apache.commons.vfs2.provider.sftp.SftpClientFactory.createConnection(SftpClientFactory.java:83)
        at org.apache.commons.vfs2.provider.sftp.SftpFileProvider.doCreateFileSystem(SftpFileProvider.java:79)
        ... 26 more
Caused by: com.jcraft.jsch.JSchException: invalid privatekey: [B@c29b0dc
        at com.jcraft.jsch.KeyPair.load(KeyPair.java:664)
        at com.jcraft.jsch.KeyPair.load(KeyPair.java:561)
        at com.jcraft.jsch.IdentityFile.newInstance(IdentityFile.java:40)
        at com.jcraft.jsch.JSch.addIdentity(JSch.java:423)
        at org.apache.commons.vfs2.provider.sftp.SftpClientFactory.addIndentity(SftpClientFactory.java:172)
        ... 29 more

Error: org.apache.commons.vfs2.FileSystemException: Could not connect to SFTP server at "sftp://prev_header@{host}/".

The in directive of seed.yml file used at that time is below:

in:
  type: sftp
  host: HOST
  port: PORT
  user: USER
  secret_key_file:
    content: |
      -----BEGIN OPENSSH PRIVATE KEY-----
      ...
      -----END OPENSSH PRIVATE KEY-----
  secret_key_passphrase: *****
  user_directory_is_root: false
  timeout: 300
  path_prefix: /path/to/file

Thus, we will be happy to new key format available in this plugin.

Workaround

I succeeded to generate config.yml by using previous default format, PEM style keys.
The previous style format can be generated by adding -m PEM option to ssh-keygen command or using OpenSSH before version 7.7.

in:
  type: sftp
  host: HOST
  port: PORT
  user: USER
  secret_key_file:
    content: |
      -----BEGIN RSA PRIVATE KEY-----
      Proc-Type: 4,ENCRYPTED
      ...
      -----END RSA PRIVATE KEY-----
  secret_key_passphrase: *****
  user_directory_is_root: false
  timeout: 300
  path_prefix: /path/to/file

Thanks,