SSH (Secure Shell) is a network protocol that enables secure remote connections between two systems. System admins use SSH utilities to manage machines, copy, or move files between systems. Because SSH transmits data over encrypted channels, security is at a high level. This article will guide you through the most popular SSH commands. Secure Shell (SSH) is commonly used to execute administrative commands remotely from a command-line terminal. App Service on Linux provides SSH support into the app container. App Service on Linux provides SSH support into the app container. For this task, you need to create an empty ssh file on another machine. We will use this file on the Raspbian boot SD card to enable SSH. If you do not feel like creating the file yourself, we made the blank ssh boot file available for download. To enable SSH on Raspberry Pi in headless mode, follow these steps: 1. Ssh-add program - Usage, ssh-agent, SSH keys ssh-add is a command for adding SSH private keys into the SSH authentication agent for implementing single sign-on with SSH. The agent process is called ssh-agent; see that page to see how to run it. WinSCP is open source and does not come with any support. Many organizations have massive amounts of SSH keys that must be properly managed. If left unmanaged, they pose a major risk and compliance issue. SSH tunneling, unless properly controlled, can allow backdoor access from the Internet into internal networks.

SSH is ubiquitous. It’s the de-facto solution for remote administration of *nix systems. But SSH has some pretty gnarly issues when it comes to usability, operability, and security.

You’re probably familiar with these issues:

• SSH user experience is terrible. SSH user on-boarding is slow and manual. Connecting to new hosts produces confusing security warnings. You’re left with weird new credentials to manage with little guidance on how to do so.
• Operating SSH at scale is a disaster. Key approval & distribution is a silly waste of time. Host names can’t be reused. Homegrown tools scatter key material across your fleet that must be cleaned up later to off-board users.
• SSH encourages bad security practices. Rekeying is hard, so it’s not done. Users are exposed to key material and encouraged to reuse keys across devices. Keys are trusted permanently, so mistakes are fail-open.

The good news is this is all easy to fix.

None of these issues are actually inherent to SSH. They’re actually problems with SSH public key authentication. The solution is to switch to certificate authentication.

SSH certificate authentication makes SSH easier to use, easier to operate, and more secure.

Background

At smallstep, certificates are kind of our jam. We build open source software that lets you run your own private certificate authority and manage X.509 (TLS/HTTPS) certificates.

SSH certificates have been on our radar for a while. From our perspective they’re all pros, no cons. But no one uses them. Why not? We’ve asked hundreds of people that question. Here’s what we found out:

• People do use SSH certificates. In fact, pretty much everyone operating at scale that knows what they’re doing does (Facebook, Uber, Google, Netflix, Intercom, Lyft, etc), but…
• Certificates and public key infrastructure (PKI) are hard to grok. People don’t immediately understand the benefits.
• There’s a (small) tooling gap that exacerbates this knowledge gap. It’s not hard to fill, but people are wary to do so themselves without a deeper understanding of PKI concepts.
• More than anything, SSH certificates haven’t gotten the press they deserve. Most people we asked hadn’t heard of them at all.

We’re convinced that SSH certificates are the right way to do SSH. They’re not that hard to understand, and it’s well worth the effort. SSH certificates deserve more press, and broader use.

Public key authentication

Most SSH deployments use public key authentication, which uses asymmetric (public key) cryptography with a public / private key pair generated for each user & host to authenticate.

The magic of asymmetric cryptography is the special correspondence between a public and private key. You can sign data with your private key and someone else can verify your signature with the corresponding public key. Like a hash, it’s computationally infeasible to forge a signature. Thus, if you can verify a signature, and you know who owns the private key, you know who generated the signature.

Simple authentication can be implemented by challenging someone to sign a big random number. If I open a socket to you and send a random number, and you respond with a valid signature over that number, I must be talking to you.

This is an oversimplification, but it’s more or less how SSH public key authentication works. Certificate authentication works the same way, but with an important twist that we’ll get to in a moment.

To SSH to a host using public key authentication the host needs to know your public key. By default, your public key must be added to ~/.ssh/authorized_keys. Maintaining this file for every user across a fleet is operationally challenging and error prone.

SSH user onboarding with public key authentication usually starts with some baroque incantation of ssh-keygen, hopefully pulled from a runbook, but more likely cribbed from stack overflow. Next you’ll be asked to submit your public key for approval and distribution. This process is typically manual and opaque. You might be asked to email an administrator or open a JIRA ticket. Then you wait. While you’re doing that, some poor operator gets interrupted and told to add your key to a manifest in some repo and trigger a deploy. Once that’s done you can SSH. Since key bindings are permanent, your SSH access will continue in perpetuity until someone reverses this process.

Certificate authentication

Certificate authentication eliminates key approval and distribution. Instead of scattering public keys across static files, you bind a public key to a name with a certificate. A certificate is just a data structure that includes a public key, name, and ancillary data like an expiration date and permissions. The data structure is signed by a certificate authority (CA).

To enable certificate authentication simply configure clients and hosts to trust any certificates signed by your CA’s public key.

On each host, edit /etc/ssh/sshd_config, specifying the CA public key for verifying user certificates, the host’s private key, and the host’s certificate:

On each client, add a line to ~/.ssh/known_hosts specifying the CA public key for verifying host certificates:

That’s it. That’s literally all that you need to do to start using certificate authentication. You can even use it alongside public key authentication to make transitioning easier.

Static keys in ~/.ssh/authorized_keys are no longer needed. Instead, peers learn one another’s public keys on demand, when connections are established, by exchanging certificates. Once certificates have been exchanged the protocol proceeds as it would with public key authentication.

Certificate authentication improves usability

With public key authentication, when you SSH to a remote host for the first time, you’ll be presented with a security warning like this:

You’ve probably seen this before. If you’re like most people, you’ve been trained to ignore it by just typing “yes”. That’s a problem because this is a legitimate security threat. It’s also a pretty horrendous user experience. I’d wager the vast majority of SSH users don’t actually understand this warning.

When you SSH to a host, the host authenticates you. Your SSH client also attempts to authenticate the host. To do so your client needs to know the host’s public key. Host public keys are stored in a simple database in ~/.ssh/known_hosts. If your client can’t find the host’s public key in this database you get this warning. It’s telling you that the host can’t be authenticated!

What you’re supposed to do is verify the key fingerprint out-of-band by asking an administrator or consulting a database or something. But no one does that. When you type “yes” the connection proceeds without authentication and the public key is permanently added to ~/.ssh/known_hosts. This is the trust on first use (TOFU) anti-pattern.

Since certificate authentication uses certificates to communicate public key bindings, clients are always able to authenticate, even if it’s the first time connecting to a host. TOFU warnings go away.

Certificate authentication also offers a convenient place to gate SSH with custom authentication: when the certificate is issued. This can be leveraged to further enhance SSH usability. In particular, it let’s you extend single sign-on (SSO) to SSH. SSO for SSH is certificate authentication’s biggest party trick. We’ll return to this idea and see how it further enhances usability and security later. For now, let’s move on to operability.

Certificate authentication improves operability

Eliminating key approval and distribution has immediate operational benefits. You’re no longer wasting ops cycles on mundane key management tasks, and you eliminate any ongoing costs associated with monitoring and maintaining homegrown machinery for adding, removing, synchronizing, and auditing static public key files across your fleet.

The ability to issue SSH user certificates via a variety of authentication mechanisms also facilitates operational automation. If a cron job or script needs SSH access it can obtain an ephemeral SSH certificate automatically, when it’s needed, instead of being pre-provisioned with a long-lived, static private key.

SSH public key authentication introduces some weird operational constraints around host names that certificate authentication eliminates. As we’ve seen, when an SSH client connects to a host for the first time it displays a TOFU warning to the user. When the user types “yes” the host’s public key is added locally to ~/.ssh/known_hosts. This binding between the host name and a specific public key is permanent. If the host presents a different public key later, the user gets an even scarier host key verification failure error message that looks like this:

This makes it operationally challenging to reuse host names. If prod01.example.com has a hardware failure, and it’s replaced with a new host using the same name, host key verification failures will ensue. This usually results in a bunch engineers contacting secops to tell them they’re being hacked.

Ignoring host key verification failures has the exact same attack surface area as not knowing the key at all. Curiously, OpenSSH chooses to soft-fail with an easily bypassed prompt when the key isn’t known (TOFU), but hard-fails with a much scarier and harder to bypass error when there’s a mismatch.

In any case, certificates fix all of this since a current name-to-public-key binding is communicated when a connection is established. Changing the host’s public key is fine, as long as the host also gets a new certificate. You can safely reuse host names and even run multiple hosts with the same name. You’ll never see a host key verification failure again. Beyond name reuse, we’ll soon see that eliminating host key verification failures is one of the many ways certificate authentication facilitates good security hygiene.

Certificate authentication improves security

While the SSH protocol itself is secure, public key authentication encourages a bunch of bad security practices and makes good security hygiene hard to achieve.

With public key authentication, keys are trusted permanently. A compromised private key or illegitimate key binding may go unnoticed or unreported for a long time. Key management oversight (e.g., forgetting to remove an ex-employee’s public keys from hosts) results in SSH failing open: unauthorized access without end.

Certificates, on the other hand, expire. In an incident — a mistake, theft, misuse, or key exfiltration of any form — compromised SSH credentials will expire automatically, without intervention, even if the incident goes unnoticed or unreported. SSH certificates are fail-secure. Access expires naturally if no action is taken to extend it. And when SSH users and hosts check in periodically with your CA to renew their credentials, a complete audit record is produced as a byproduct.

We’ve already seen how public key authentication trains users to ignore serious security warnings (TOFU) and triggers spurious security errors. This is more than an operational nuisance. Confusion caused by host key verification failure discourages host rekeying (i.e., replacing a host’s key pair). Host private keys aren’t very well protected, so periodic rekeying is good practice. Rekeying may be required after a breach or after offboarding a user. But, to avoid disruption from ensuing host key verification failures, it’s often not done. Certificate authentication makes rekeying hosts trivial.

Public key authentication also makes rekeying difficult for users. Key approval and distribution is annoying enough that users are reluctant to rekey, even if you’ve built tools to make it possible. Worse, frustrated users copy private keys and reuse them across devices, often for many years. Key reuse is a serious security sin. Private keys are never supposed to be transferred across a network. But SSH public key authentication exposes users directly to sensitive private keys, then fails to give them usable tools for key management. It’s a recipe for misuse and abuse.

An SSH CA, coupled with a simple command-line client for users, can streamline key generation and insulate users from a lot of unnecessary detail. Certificate authentication can’t completely eliminate all security risks, but it does facilitate SSH workflows that are more intuitive, easier to use, and harder to misuse.

An ideal SSH flow

SSH certificate authentication is the foundation of what I think is the ideal SSH flow.

To SSH, users first run a login command in their terminal (e.g., step ssh login):

A browser is opened and an SSO flow is initiated at your organization’s identity provider:

A web-based SSO flow makes it easy to leverage strong MFA (e.g., FIDO U2F) and any other advanced authentication capabilities your identity provider offers. Users login with a familiar flow, and removing a user from your canonical identity provider ensures prompt termination of SSH access.

Once the user completes SSO, a bearer token (e.g., an OIDC identity token) is returned to the login utility. The utility generates a new key pair and requests a signed certificate from the CA, using the bearer token to authenticate and authorize the certificate request.

The CA returns a certificate with an expiry long enough for a work day (e.g., 16-20 hours). The login utility automatically adds the signed certificate and corresponding private key to the user’s ssh-agent.

Users needn’t be aware of any of this detail. All they need to know is that, in order to use SSH, they must first run step ssh login. Once that’s done they can use SSH like normal:

Like browser cookies, short-lived certificates issued by this flow are ephemeral credentials, lasting just long enough for one work day. Like logging into a website, logging into SSH creates a session. It’s a simple process that must be completed, at most, once per day. This is infrequent enough that strong MFA can be used without frustrating or desensitizing users.

New private keys and certificates are generated automatically every time the user logs in, and they never touch disk. Inserting directly into ssh-agent insulates users from sensitive credentials. If a user wants to connect from a different device it’s easier for them to run step ssh login there than it is to exfiltrate keys from ssh-agent and reuse them.

There are lots of possible variations of this flow. You can adjust the certificate expiry, use PAM authentication at the CA instead of SSO, generate the private key on a smart card or TPM, opt not to use ssh-agent, or move MFA to the actual SSH connection. Personally, I think this combination offers the best balance of security and usability. Indeed, relative to most existing SSH deployments it’s operationally simpler, more secure, and more usable.

Critics of SSH certificate authentication say that it’s new, not well supported, and the tooling doesn’t exist to use certificates in practice. The truth is, certificate authentication was added inOpenSSH 5.4almost a decade ago. It’s battle tested and used in production by massive operations. And the tooling required to build this ideal SSH flow is available today.

Tools

There are lots of existing tools for managing SSH certificates. Here are a few:

• ssh-keygen can generate root certificates and sign user & host certificates
• netflix/bless is Netflix’s SSH CA that runs in AWS Lambda and uses IAM
• nsheridan/cashier is Intercom’s SSH CA
• uber/pam-ussh lets you use certificates to authorize sudo use
• hashicorp/vault has an SSH secrets engine

For our part, the most recent release of step & step-ca (v0.12.0) adds basic SSH certificate support. In other words:

• step-ca is now an SSH CA (in addition to being an X.509 CA)
• step makes it easy for users and hosts to get certificates from step-ca

SSH workflows aren’t fully fleshed out yet, but these tools already do everything you need for the ideal flow.

With the appropriate configuration of step-ca you can use step to:

Get a host certificate automatically at startup

To demonstrate, let’s create a new EC2 instance with the aws CLI tool. The interesting bits are tucked in some light configuration (using a user-data startup script) that gets a host certificate and enables certificate authentication for users:

Note: you should be able to use our instance identity document support here, but we’ve got a few kinks to work out. Stay tuned.

Get a user certificate using SSO (OAuth OIDC)

Now we’ll use step ssh certificate locally (you can brew install step) to generate a new key pair, get a certificate from the CA using SSO, and automatically add the certificate and private key to ssh-agent.

That sounds like a lot, but it’s just one command:

Once that’s done we can SSH to the instance we just created, using certificate authentication, with no TOFU!

For more info check out our getting started guide and SSH example repo. Make sure you pass the --ssh flag to step ca init when you’re setting up the CA (the getting started guide doesn’t do this).

There’s a lot more that can be done to make SSH certificate authentication even more awesome. We’re working on that. If you have any ideas, let us know!

Use SSH certificates

SSH certificate authentication does a lot to improve SSH. It eliminates spurious TOFU warnings and host key verification failures. It lets you drop complex key approval & distribution processes and extend SSO to SSH. It makes rekeying possible for hosts and easier than key reuse for users. It makes SSH keys ephemeral, making key management oversights fail-secure.

You can deploy an SSH CA and reconfigure hosts in a matter of minutes. It’s easy to transition — you can continue supporting public key authentication at the same time.

SSH certificate authentication is the right way to do SSH.

At smallstep, we’re looking forward to improving our SSH story. We’re building out infrastructure and streamlined workflows to make SSH better for everyone. And keep an eye on our blog because we have a lot more to say about SSH!

Most authentication in Windows environments is done with a username-password pair.This works well for systems that share a common domain.When working across domains, such as between on-premise and cloud-hosted systems, it becomes vulnerable to brute force intrusions.

By comparison, Linux environments commonly use public-key/private-key pairs to drive authentication which doesn't require the use of guessable passwords.OpenSSH includes tools to help support this, specifically:

• ssh-keygen for generating secure keys
• ssh-agent and ssh-add for securely storing private keys
• scp and sftp to securely copy public key files during initial use of a server

This document provides an overview of how to use these tools on Windows to begin using key authentication with SSH.If you are unfamiliar with SSH key management, we strongly recommend you review NIST document IR 7966 titled 'Security of Interactive and Automated Access Management Using Secure Shell (SSH).'

About key pairs

Key pairs refer to the public and private key files that are used by certain authentication protocols.

Winscp Import Openssh Key

SSH public-key authentication uses asymmetric cryptographic algorithms to generate two key files – one 'private' and the other 'public'. The private key files are the equivalent of a password, and should stay protected under all circumstances. If someone acquires your private key, they can log in as you to any SSH server you have access to. The public key is what is placed on the SSH server, and may be shared without compromising the private key.

When using key authentication with an SSH server, the SSH server and client compare the public keys for username provided against the private key. If the server-side public key cannot be validated against the client-side private key, authentication fails.

Multi-factor authentication may be implemented with key pairs by requiring that a passphrase be supplied when the key pair is generated (see key generation below).During authentication the user is prompted for the passphrase, which is used along with the presence of the private key on the SSH client to authenticate the user.

Host key generation

Public keys have specific ACL requirements that, on Windows, equate to only allowing access to administrators and System.To make this easier,

• The OpenSSHUtils PowerShell module has been created to set the key ACLs properly, and should be installed on the server
• On first use of sshd, the key pair for the host will be automatically generated. If ssh-agent is running, the keys will be automatically added to the local store.

To make key authentication easy with an SSH server, run the following commands from an elevated PowerShell prompt:

Since there is no user associated with the sshd service, the host keys are stored under ProgramDatassh.

User key generation

To use key-based authentication, you first need to generate some public/private key pairs for your client.From PowerShell or cmd, use ssh-keygen to generate some key files.

This should display something like the following (where 'username' is replaced by your user name)

You can hit Enter to accept the default, or specify a path where you'd like your keys to be generated.At this point, you'll be prompted to use a passphrase to encrypt your private key files.The passphrase works with the key file to provide 2-factor authentication.For this example, we are leaving the passphrase empty.

Now you have a public/private ED25519 key pair(the .pub files are public keys and the rest are private keys):

Remember that private key files are the equivalent of a password should be protected the same way you protect your password.To help with that, use ssh-agent to securely store the private keys within a Windows security context, associated with your Windows login.To do that, start the ssh-agent service as Administrator and use ssh-add to store the private key.

Winscp Openssh

After completing these steps, whenever a private key is needed for authentication from this client, ssh-agent will automatically retrieve the local private key and pass it to your SSH client.

Note

It is strongly recommended that you back up your private key to a secure location,then delete it from the local system, after adding it to ssh-agent.The private key cannot be retrieved from the agent.If you lose access to the private key, you would have to create a new key pairand update the public key on all systems you interact with.

Deploying the public key

To use the user key that was created above, the public key needs to be placed on the server into a text file called authorized_keys under usersusername.ssh.The OpenSSH tools include scp, which is a secure file-transfer utility, to help with this.

To move the contents of your public key (~.sshid_ed25519.pub) into a text file called authorized_keys in ~.ssh on your server/host.

This example uses the Repair-AuthorizedKeyPermissions function in the OpenSSHUtils module which was previously installed on the host in the instructions above.

These steps complete the configuration required to use key-based authentication with SSH on Windows.After this, the user can connect to the sshd host from any client that has the private key.