A comprehensive guide to SFTP (SSH/Secure File Transfer Protocol)

SFTP (also known as Secure File Transfer Protocol or SSH File Transfer Protocol) is a secure network protocol used for transferring files between systems over an encrypted SSH connection. SFTP was designed to be universal, secure, and fast, while solving a lot of the limitations around the older and widely supported FTP protocol. 

What is SFTP used for?

SFTP is widely supported as a protocol and method of transferring files to and from servers or systems. Commonly used as a sysadmin and developer tool to quickly upload and download data from production and/or test servers, SFTP has become a common tool in the toolbox for moving data. 

But limiting SFTP to just that would be a bit insulting.

Today, SFTP has all but replaced FTP as the standardized method for automating the exchange and transfer of files to and from services and servers. SFTP exists in tech, healthcare, logistics, and finance to exchange files in a secure and robust way.

Much like FTP was a pillar in the 90s, SFTP has taken on that role today.

The SFTP Protocol

SFTP operates as a binary protocol over a secure SSH connection. You can think of SSH as the secure pipe between two systems, with SFTP providing the commands and structure used to move and manage files through that pipe.

One of the nice things about SFTP is that security isn't something bolted onto the side. SSH handles the secure connection, including server authentication, encryption, and data integrity, before the SFTP session gets down to the business of moving files.

During the SSH connection, the client and server negotiate the cryptographic algorithms they both support. Once that secure channel is established, SFTP traffic travels through it encrypted.

This single-channel design also gets around a lot of the headaches associated with traditional FTP. SFTP doesn't need a separate data connection for each transfer, which generally makes it much easier to work with firewalls and NAT.  

Protocol Versions

SFTP has a slightly messy version history, so it's worth clearing up one common misconception: there isn't really a simple progression where everybody moved from SFTP version 1 through to version 6.

SFTP version 3 became the common interoperability baseline and is still the version implemented by OpenSSH, which makes it the version you'll run into most often in the real world.

Later versions of the protocol, including versions 4, 5, and 6, were developed in subsequent Internet-Drafts and added features and changes to the protocol. They were never universally adopted across SFTP implementations, so seeing a later version number doesn't automatically mean you're dealing with a newer or more secure SFTP server.

Fortunately, SFTP includes version negotiation. When a session starts, the client tells the server which protocol version it supports and the server responds with the version it will use.

For most people using SFTP, none of this needs to keep you awake at night. The important bit is making sure your client and server are compatible and that the SSH configuration underneath them is using modern, secure algorithms.

SFTP Authentication and Encryption

Since SFTP runs over SSH, the secure SSH connection is established before the SFTP session begins.

There are a couple of different pieces involved here that are easy to lump together as "encryption."

First, the client needs to establish that it is talking to the correct server. SSH does this using the server's host key.

The user then authenticates. Depending on how the server is configured, that can include methods such as:

• Password-based authentication
• Public-key authentication using an SSH key pair
• GSSAPI
• Multi-factor authentication

SSH also negotiates the algorithms used to protect the connection. Public-key cryptography and key-exchange algorithms are used as part of establishing and authenticating the secure session, while symmetric encryption is normally used to encrypt the actual data moving across it.

In plain English: the two systems establish a trusted, encrypted SSH connection first, and then SFTP sends your files through it.

Modern SSH implementations support strong encryption algorithms such as AES, along with integrity or authenticated-encryption mechanisms that protect the data from being read or modified while it is in transit.

This is fundamentally different from traditional FTP, where usernames, passwords, commands, and file data can be sent without encryption unless another security layer is added.

SFTP Protocol Structure

Keeping things simple, SFTP uses a request-and-response messaging protocol with extensibility baked in. Once the SSH connection has been established, SFTP messages are sent between the client and server through that secure channel.

Unlike FTP, SFTP doesn't create separate external data connections for file transfers. Everything travels over the SSH connection, which is one of the reasons it tends to play much more nicely with NAT and firewalls.

If you're curious, an SFTP packet generally contains:

• Length: A 32-bit integer describing the packet length.
• Type: An 8-bit value identifying the type of message, such as init, version, open, close, read, or write.
• Payload: The data required for that particular message.

When the SFTP session begins, the client sends an init packet containing the protocol version it supports. The server responds with a version packet identifying the version the session will use, along with any supported protocol extensions.

For most users, you'll never need to look at an SFTP packet directly. Your client and server take care of all of this behind the scenes, but this relatively simple message structure is part of what has made SFTP so widely implemented.

SFTP Ports and Firewall Considerations

SFTP typically operates over SSH port 22, although an SSH server can be configured to listen on pretty much any available port.

The useful part is that SFTP uses the SSH connection for both commands and file data. Unlike traditional FTP, you don't need to manage a separate set of data connections every time you transfer something.

That makes SFTP considerably easier to deal with when firewalls and NAT are involved.

If your SSH server uses the default configuration, you'll normally connect on port 22. If your administrator has configured a different port, your SFTP client simply needs to be told which one to use.

One small security note: moving SSH from port 22 to another port can reduce some of the automated internet noise hitting the service, but changing the port isn't a replacement for proper authentication, access controls, patching, and a secure SSH configuration.

SFTP vs Other Protocols

Here's a comparison table of SFTP with other popular file transfer protocols and solutions:

Read more about protocols at What is SFTP and how are FTP, SFTP, SCP, and Rsync different? 

How To Transfer Files with SFTP

Using the SFTP protocol to transfer files is one of those things that can be done a few different ways and how you get it done will depend a lot on what problem you are trying to solve. There are of course a few common methods. In general, you would use an SFTP client or an SFTP server.

An SFTP Client is the piece of software that you are using to upload or download files. This client connects to the SFTP server with a hostname, username, password, and port. If you don't have these details from your "server provider", grab them quick, real quick.

When transferring files, you normally need a client. You have three main options when it comes to clients.  You can use the built-in client from the terminal, use an external GUI-based client, or use a proprietary connector built into the software that you want to upload or download files to or from.

The SFTP Server is the part on the other end. If you think you need a server, you can configure it manually or use Couchdrop and set one up in minutes, with Couchdrop doing the difficult parts for you automatically.

Using SFTP from the terminal to transfer files

All modern operating systems support SFTP out of the box. This is part of what makes SFTP so useful as a protocol. However, using the SFTP client built into your operating system will require some familiarity with the terminal and command line tools. If this does not sound like your cup of tea, then jump down to the next section on using SFTP from the GUI to transfer files.

Uploading a file using the SFTP from the command line:

1. Open a terminal;
2. In the terminal, type 'sftp <username>@<hostname>:/'
3. Enter your password if prompted
4. Upload the file by typing 'put <filename and path>'

Downloading a file using the SFTP from the command line:

1. Open a terminal;
2. In the terminal, type 'sftp <username>@<hostname>:/'
3. Enter your password if prompted
4. Download the file by typing 'get <filename and path>'

Other commands:

You can navigate the SFTP server using the following commands as well:

• ls [<path>]: List the contents of a remote directory.

• cd <path>: Change the current directory on the remote server

• mkdir <path>: Create a new directory on the remote server.

• rmdir <path>: Remove a directory from the remote server.

• rm <path>: Remove a file from the remote server.

• rename <oldpath> <newpath>: Rename a remote file or directory.

Using SFTP from the GUI to transfer files 

If using the command line or terminal is not what you are familiar with, then a GUI SFTP client is going to make sense and be the fastest way to get up and running with SFTP. Luckily there are a lot of great GUI SFTP clients available, both for free and for a small license.

At Couchdrop, we are big fans of Transmit and Filezilla, but most SFTP clients work roughly the same. 

Common Clients Available

The beauty of a GUI client is that it's point-and-click. For most clients, you can simply:

1. Download the SFTP client and install it
2. Create a new connection
3. Provide the hostname, port, username, and password
4. Drag and drop files between the SFTP server and your local machine

Using SFTP to transfer files inside an application (Salesforce/Zapier/Etc.)

SFTP is so well supported as a standardized method for transferring files, that support for it is baked into many applications. This is a major advantage for SFTP over proprietary methods of data transfer and means you can import and export files into various different software applications automatically from any location. 

The configuration steps all vary slightly, but generally, you need to follow the steps below to use SFTP inside an application.

1. Configure a new connection
2. Provide the connection details, hostname, port, username and password
3. Provide a folder name

Troubleshooting and available options are normally well documented on the software provider's knowledge base. 

Automating File Transfers with SFTP

It's important to remember that SFTP is not a collaboration tool. Where it really shines is in backend file transfers and automated file transfers. With this in mind, it's remarkably easy to automate, allowing you to schedule and automate recurring file transfer tasks. Automating file transfers can save time, reduce manual effort, and ensure consistent and reliable data exchange between systems.

Automating file transfers is normally done in one of three ways.

1. By creating a bash or PowerShell script 
2. By creating a small application using an SFTP SDK (Like Python or Golang)
3. By using an automation platform like Zapier or Couchdrop

For more details about automating file transfers with SFTP, see our article Streamline Your File Transfer Process with Automated SFTP.

How-to: Uploading a file from a Bash Script using SFTP

You can write scripts using shell scripting languages like Bash or PowerShell to automate SFTP file transfers. These scripts can include SFTP commands to connect to the server, navigate directories, and transfer files. Here's an example of a simple Bash script that automates file upload:

#!/bin/bash

SFTP_SERVER="sftp.example.com"
SFTP_USER="username"
SFTP_PASSWORD="password"
LOCAL_FILE="/path/to/local/file.txt"
REMOTE_DIR="/path/to/remote/directory"

sshpass -p "$SFTP_PASSWORD" sftp "$SFTP_USER@$SFTP_SERVER" << EOF
put "$LOCAL_FILE" "$REMOTE_DIR"
EOF

How-to: Uploading a file with Python using SFTP

For more advanced automation scenarios, you can use programming languages and SFTP libraries or SDKs to develop custom SFTP automation scripts or applications. Many programming languages have SFTP client libraries that allow you to programmatically interact with SFTP servers. Some popular SFTP libraries include:

• Python: Paramiko, pysftp
• Java: JSch, Apache Commons VFS
• .NET: SSH.NET, Renci.SshNet
• Node.js: ssh2, node-ssh

These libraries provide APIs to establish SFTP connections, authenticate, navigate directories, and perform file transfer operations programmatically.

At Couchdrop, we are Python Fan Boys, so here's an example of using the Python Paramiko library to automate file downloads:

import paramiko

# Create and authenticate the SSH transport
transport = paramiko.Transport(("sftp.example.com", 22))
transport.connect(username="username", password="password")

# Create the SFTP client
sftp = paramiko.SFTPClient.from_transport(transport)

try:
    # Download a file
    sftp.get(
        "/path/to/remote/file.txt",
        "/path/to/local/file.txt"
    )
finally:
    # Close both connections
    sftp.close()
    transport.close()

By automating SFTP file transfers, you can streamline data exchange processes, reduce manual intervention, and ensure consistent and timely transfer of files between systems. Automation also helps minimize errors and provides a more reliable and efficient way to handle recurring file transfer tasks.

Using a file transfer automation platform

If you'd rather avoid writing code, there are plenty of platforms that make automating the transfer of files to and from an SFTP server really easy. These platforms offer reliability and simplicity and provide a myriad of scheduling and advanced options. 

Some of the services in the market at the moment:

• FileZilla offers the FileZilla Server, which supports event-based actions and file transfer automation.
• WinSCP provides a scripting and automation interface that allows you to create and execute SFTP automation scripts.
• Couchdrop provides file transfer automation and an SFTP server as a SaaS platform
• Zapier provides a very flexible automation platform that can upload and download files to SFTP

These automation platforms enable you to define file transfer tasks, schedule them, and set up event-triggered actions.

In some circles, they are also called Managed File Transfer Solutions (MFT). 

SFTP and Managed File Transfer (MFT) Solutions

Managed File Transfer (MFT) solutions often include SFTP as one of the supported protocols. MFT solutions provide a centralized platform for automating, managing, and monitoring file transfers across an organization. They offer features like:

• Workflow automation. Define complex file transfer workflows with conditional logic, error handling, and data transformations.
• Scheduling. Schedule file transfers to run at specific times or intervals.
• Event-driven transfers. Trigger file transfers based on specific events, such as the arrival of a new file or a change in a database.
• Auditing and reporting. Maintain detailed audit trails and generate reports on file transfer activities.

How to set up an SFTP server

When it comes to setting up an SFTP server, you have two main options:

1. Manually installing an SFTP server on your own infrastructure
2. Use a cloud SFTP service like Couchdrop

Setting up an SFTP Server Manually

To set up an SFTP server manually, you'll need to follow these steps:

1. Choose an SFTP Server Software: Select an SFTP server software that suits your needs. One popular option is OpenSSH, which comes pre-installed on most Linux distributions.

2. Install the SFTP Server Software: Install the chosen SFTP server software on your server machine. The installation process may vary depending on the operating system and the specific software you are using.

3. Configure SSH Server Settings: Modify the server configuration file (usually located at /etc/ssh/sshd_config on Linux) to enable SFTP and set the desired settings. Some common settings to configure include:

   • Port: Specify the port number on which the server should listen (default is 22).

   • PermitRootLogin: Determines whether root login is allowed (recommended to set to "no" for security reasons).

   • PubkeyAuthentication: Enables public key authentication (set to "yes" for key-based authentication).

   • PasswordAuthentication: Enables password-based authentication (set to "no" if only key-based authentication is desired).

   • Subsystem sftp: Specifies the path to the SFTP server subsystem (e.g., /usr/lib/openssh/sftp-server).

4. Create SFTP User Accounts: Set up user accounts for SFTP access. You can use existing system user accounts or create new ones specifically for SFTP.

5. Configure User Permissions: Set appropriate file and directory permissions for SFTP users. Use tools like chmod and chown to control access rights.

6. Configure Firewall: Open the necessary ports in your firewall to allow SFTP connections. By default, SFTP uses port 22. If you configured a different port, make sure to open that port as well.

7. Restart the SSH Server: Restart the server to apply the configuration changes. On Linux, you can typically use the command sudo systemctl restart ssh or sudo service ssh restart.

8. Test SFTP Connection: Use an SFTP client to test the connection to your newly set up SFTP server. Verify that you can successfully authenticate and perform file transfer operations.

By following these steps, you can set up a basic SFTP server. However, depending on your specific requirements, you may need to perform additional configuration, such as setting up chrooted directories, enabling logging, or configuring security features like IP whitelisting or rate limiting.

It's important to regularly update your SFTP server software to ensure you have the latest security patches and features. Additionally, monitoring SFTP server logs can help you detect and investigate any suspicious activities or potential security breaches.

Use a hosted, managed SaaS platform like Couchdrop

Couchdrop is a managed SaaS SFTP solution that simplifies the process of setting up and managing an SFTP server and integrates with your Cloud Storage (Dropbox/Sharepoint/S3). Couchdrop takes the pain out of server management away and offers a simple, integrated, and scalable SFTP platform.

Setting up and configuring a cloud SFTP server is incredibly simple with Couchdrop:

1. Create an SFTP Server. Sign up for a Couchdrop account and create a new SFTP server with just a few clicks. Choose a server location, specify the storage capacity, and set up user accounts.

2. Configure your SFTP Server. Setting up Couchdrop is done via an intuitive web interface. 

3. Customer Support. All customers benefit from dedicated customer support provided by Couchdrop. Get assistance with setup, configuration, and troubleshooting if you ever need help.

By using Couchdrop, you can offload the complexities of setting up and managing an SFTP server. Couchdrop takes care of the underlying infrastructure, security, and maintenance, allowing you to focus on your core business activities.

Couchdrop is particularly suitable for businesses that require a reliable, scalable, and easy-to-use SFTP solution without the overhead of managing the server themselves.

Case Study: Inzata's SFTP Solution with Couchdrop

Inzata, a data analytics platform, faced challenges in receiving data files from their customers in various formats like Excel files and CSVs. Their existing solution using FileZilla was cumbersome and required significant manual effort.

Alex Durante, the Product Director at Inzata, explains, "Our developer would have ended up keeping track of 50 SFTPs himself, which wasn't feasible."

Inzata discovered Couchdrop, an SFTP solution that streamlined their file transfer process. Instead of managing separate SFTPs for each customer, Couchdrop provided a single SFTP with individual folders for each customer, making it easy for Inzata to manage.

Alex highlights the benefits of Couchdrop:

• Simplified SFTP setup and management for customer support team
• Reduced turnaround time for onboarding new customers
• User-friendly drag-and-drop interface for less technical customers
• Enhanced privacy and security by acting as a transfer portal without storing data
• Improved customer experience and conversion rates

Alex also praised Couchdrop's exceptional customer service, stating, "Any time we had an issue with anything, they went above and beyond to help us and even put in place custom items specifically for us. There's nothing about Couchdrop I dislike or would change."

Inzata's success story demonstrates how SFTP solutions like Couchdrop can simplify and automate file transfer processes, enhance security, and improve the overall customer experience for data-driven businesses.

SFTP: a secure and reliable file transfer protocol

SFTP is a secure and reliable protocol for transferring files between systems, offering strong encryption, authentication, and data integrity. Its single-port design and firewall-friendly nature make it an ideal choice for automated file transfers in various business scenarios.