online posts 16
linux and the command line 101

task 6.2 · chapter 6 of 8: ssh and keys · 25 min

Short names, a copy, a changed key

objective

Give ws1, srv1 and r1 short names in ~/.ssh/config, copy a file to srv1 with scp, then meet the warning every engineer meets one day: the host key has changed. Find out why, and fix it the right way.

you edit, in the lab~/.ssh/config~/.ssh/known_hosts

open a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter

given
LessonSSH and keys
Loginsws1 and srv1: netops / netops-lab. r1: admin / capeos-lab
Rebuild srv1make break T=6.2: same name, same address, new host keys
new to this? step by stepbeginner
  1. nano ~/.ssh/config: a Host block for each of ws1, srv1 (User netops) and r1 (User admin).
  2. ssh-copy-id srv1, then ssh srv1 and ssh ws1 work with no user name and no password. ssh r1 asks for r1’s password.
  3. scp files/notes/handover.txt srv1:, then ssh srv1 ls.
  4. make break T=6.2, then ssh srv1: read the warning. Remove srv1’s old key with ssh-keygen -R srv1, connect, check the new fingerprint, yes.
  5. make check T=6.2.

more in meet the lab

1 explain

ssh’s own settings, and a key that changed

~/.ssh/config

Typing netops@ every time gets old, and real hosts have long names, odd ports and their own keys. ~/.ssh/config lets you give each host its settings once:

Host web
    HostName srv1
    User netops
    Port 22

ssh web now means ssh -p 22 netops@srv1. Host is the short name you type; HostName is the real name or address (leave it out and the Host name is used). Everything that uses ssh reads this file: scp, Git, Ansible.

in one sentence A Host block in ~/.ssh/config turns a short name into a user, a real host name and options.

scp copies over SSH

scp FROM TO copies a file, and either end can be host:path. scp notes.txt srv1: copies notes.txt into your home folder on srv1 (nothing after the colon means home); scp srv1:notes.txt . copies it back into the folder you’re in. Same logins, same keys, same ~/.ssh/config.

“Remote host identification has changed”

ssh remembers every host key you’ve said yes to. If a host offers a different key next time, ssh refuses to connect at all, and says so loudly. There are two reasons that can happen:

  • the server was rebuilt or reinstalled, and made new host keys. Common, and harmless;
  • something between you and the server is pretending to be it: a man-in-the-middle. Rare, and the reason ssh refuses instead of asking.

You can’t tell which from the warning alone. Find out (here, you know: you just rebuilt srv1), then remove the old key, connect, and check the new fingerprint the way you did in task 6.1. Never turn the check off to make the warning go away.

in one sentence A changed host key means a rebuilt server or an impostor: find out which, then replace the one old line.

2 show

The warning

After make break T=6.2:

$ ssh srv1
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:D5T0vLnK5aqJqyXkl0nNrxpCcACGUqPLTgSGR0KO4Eo.
Please contact your system administrator.
Add correct host key in /home/vscode/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/vscode/.ssh/known_hosts:4
  remove with:
  ssh-keygen -f "/home/vscode/.ssh/known_hosts" -R "srv1"
Host key for srv1 has changed and you have requested strict checking.
Host key verification failed.

ssh tells you which line of known_hosts holds the old key, and the command that removes it. You can’t just read the file to find it: Debian hashes the names in known_hosts, so each line starts with |1| and a scramble instead of srv1. ssh-keygen -R and ssh-keygen -F work by name anyway.

Your line number may differ. A server has more than one host key (ED25519, ECDSA, RSA), and after you log in ssh saves all of them, so each machine you’ve logged in to takes up three lines. ssh-keygen -R srv1 removes all of srv1’s.

3 follow along

Short names

  1. Open ssh’s config file. It doesn’t exist yet, so nano makes it:

    run in the terminal
    nano ~/.ssh/config
    
  2. Type a block for ws1, save (Ctrl+O, Enter) and leave (Ctrl+X). The indentation is just for you; ssh doesn’t mind it.

    edit~/.ssh/configCtrl+P → config
    Host ws1
        User netops
    
  3. Try it: no user name, no password.

    run in the terminal
    ssh ws1 hostname
    
  4. See every setting ssh would use for a host, from all its config files together, without connecting:

    run in the terminal
    ssh -G ws1 | head -n 4
    

    user netops is there.

in one sentence One Host block, and ssh ws1 means ssh netops@ws1; ssh -G shows what ssh will really do.

4 challenge

Three names, one copy, one changed key

  1. Add blocks for srv1 (user netops) and r1 (user admin) to ~/.ssh/config. Put your key on srv1 too, so ssh srv1 needs no password. r1 is a router, so its login stays a password; ssh r1 should just not need the user name. (exit leaves r1.)
  2. Copy files/notes/handover.txt into netops’s home folder on srv1 with scp.
  3. make break T=6.2 rebuilds srv1. ssh srv1 now refuses. Make it work again: replace srv1’s old host key, checking the new fingerprint first (click srv1 on the Topology tab, then the ssh-keygen -lf from task 6.1).

Then make check T=6.2.

hint

ssh-copy-id srv1 works with the short name as soon as srv1’s block is in the config.

hint

scp FILE srv1: copies into the home folder. ssh srv1 cat handover.txt checks it arrived.

hint

The warning gives you the command. Then connect again: ssh asks about srv1’s key as if it were the first time.

show the solution
Host ws1
    User netops
Host srv1
    User netops
Host r1
    User admin
ssh-copy-id srv1
scp files/notes/handover.txt srv1:
ssh srv1 cat handover.txt
make break T=6.2
ssh srv1                       # REMOTE HOST IDENTIFICATION HAS CHANGED
ssh-keygen -R srv1             # removes srv1's old line from known_hosts
ssh srv1                       # asks about the new key: check it, then yes

make reset puts srv1’s original host key back, so after a reset you’ll see this warning once more, for the same harmless reason. Keys you’ve put in authorized_keys go with a reset too: ssh-copy-id again.

5 check: run it, then prove it

$ make check T=6.2

when it's right, you'll see

╭─ task 6.2 · short names, a copy, a changed key
│  ✓ ~/.ssh/config: ssh ws1 logs in as netops (challenge): it would use netops
│  ✓ ~/.ssh/config: ssh srv1 logs in as netops (challenge): it would use netops
│  ✓ ~/.ssh/config: ssh r1 logs in as admin (challenge): it would use admin
│  ✓ ssh ws1 works, with no user name and no password (challenge)
│  ✓ ssh srv1 works too (challenge)
│  ✓ files/notes/handover.txt is on srv1, in netops's home folder (challenge)
│  ✓ make break T=6.2 rebuilt srv1, and you fixed known_hosts (challenge)
╰─ 7/7 checks passed  verified