Jeff's comment points in the right direction, but there are many more and better safeguards that are readily available and easy to set up. The ultimate is a "knock server" using fwknopd, which puts Jeff's general concept on a cryptographically strong footing. This employs Single Packet Authentication (SPA), so it cannot inadvertently be triggered by a scan or so.
The idea is that a daemon (fwknopd) is (completely passively) listening for incoming packets on a designated port. It's conceptually like sending encrypted and signed email and actually uses the same underlying gnupg technology - with email, you encrypt your email for the intended recipient using the recipient's key, and sign it with yours. It works the same way with SPA, you use the daemon's key (that you set up, or course) to encrypt the single packet and your key to sign it, so the demon can verify that the request is really coming from you. Then the daemon takes the requested action, which usually is to open a short-lived conduit for the ssh port for you to log in (but you can set the system up to execute any other command if you like).
There are many good tutorials just a google search away. I consider this the ultimate protection for your home setup. No one can log in without your multi-layered credentials (the SPA port-opening, and then requiring ssh keys to log in - you should disable passwords), but even more importantly, you do not advertise your IP as "active" and worth a closer look.
If you set the firewall up properly (no open ports to the outside world), no scanner can even tell that a system is listening. It looks like a "dead" IP address. This is important because I see, on average, about two scan attempts per minute on my home system. The scanners typically hit some 50 standard ports and, seeing no reply on any, move on. However, if you give away the presence of a live system by just one open port, even if it cannot be exploited, you usually unleash a much more aggressive scan and subsequent attacks on your system.
fwknopd is readily available on virtually all linux systems, including the Pi. In addition, standard clients are available on Windows, MacOS, and Android.
Let's go dark...