How to Run Screaming Frog SEO Spider on a VPS

Screaming Frog SEO Spider can run on a VPS as a remote crawl workstation. You connect to its desktop, start a crawl and reconnect later to review the results. The crawl can keep running while your laptop is disconnected, provided the server and desktop session stay running.

This guide covers Ubuntu 24.04, an XFCE desktop with xrdp, installation of SEO Spider and a small test crawl. It also explains how to choose memory settings and keep copies of crawl data outside the VPS.

Prepare the server and licence

You will need:

• A 64-bit Ubuntu 24.04 server with enough free storage for the desktop, application, crawl databases, exports and temporary files.
• A non-root account with SSH and sudo access. Keep a working SSH session open while changing remote-access or firewall settings.
• A package matching the server's amd64 or arm64 architecture.
• A Screaming Frog licence for the operator if you need more than 500 URLs, saved crawls or other licensed features.
• Permission to crawl the target and an agreed crawl speed.
• Storage outside the VPS for independent copies of crawl files and exports.

The platform FAQ confirms the 64-bit requirement and bundled Java runtime. The licensing FAQ assigns each licence to one authorised person. A shared server does not make that licence available to a whole team.

Size memory and storage for your crawls

Memory use depends on the pages and features you crawl. JavaScript rendering, stored HTML, screenshots, custom extraction, integrations and multiple application instances can all increase resource use. Database storage also needs room for crawl data and temporary files.

Screaming Frog's memory guidance recommends allocating at least 2 GB less than the machine's physical RAM so the operating system and other applications have room to run. On a 6 GB VPS, that puts the ceiling at 4 GB under this rule. Use less if the desktop and other processes need more than the remaining 2 GB.

Cloud VPS S is one configuration you can test:

* Unlimited bandwidth is subject to fair use, acceptable-use requirements, network integrity and service limits. Keep independent backups; the included snapshot does not replace a separate copy of your crawl data.

Start with a small crawl and measure peak memory, CPU use, disk growth, duration and errors. Choose more capacity if those results leave too little headroom for the intended crawl. Confirm the resize or migration procedure before relying on a later upgrade.

Set up the remote desktop

This example uses XFCE and xrdp from Ubuntu's package repositories. The xrdp project supports RDP clients and uses TLS by default. Its standard listening port is TCP 3389.

Before installing xrdp, restrict TCP 3389 to your administrative source address using the applicable firewall. Packages may start services during installation, so set this restriction first. Keep SSH explicitly allowed and confirm a second SSH login works before enabling or changing a firewall policy.

From the non-root operator's SSH session, install the desktop and configure that user's session:

sudo apt update
sudo apt install --yes xfce4 xfce4-goodies xrdp
printf '%s\n' 'startxfce4' > "$HOME/.xsession"
sudo systemctl enable --now xrdp
systemctl is-active xrdp

The last command should return active. Connect with your RDP client as the same operator, check that XFCE opens and launch a terminal. Disconnect and reconnect to confirm that access works. Keep the original SSH session open until those checks pass so you can correct the configuration if needed.

Disconnecting from RDP and logging out of the desktop are different actions. Logging out can stop applications in that session, including an active crawl.

Install the official Ubuntu package

The official download page lists SEO Spider 24.3 with Ubuntu packages for both amd64 and arm64. Check dpkg --print-architecture on the VPS before choosing the file.

For an x86-64 Ubuntu server, download and inspect the amd64 package, then install it:

mkdir -p "$HOME/Downloads"
curl -fL --proto '=https' --tlsv1.2 \
  --output "$HOME/Downloads/screamingfrogseospider_24.3_amd64.deb" \
  "https://download.screamingfrog.co.uk/products/seo-spider/screamingfrogseospider_24.3_amd64.deb"
dpkg-deb --info "$HOME/Downloads/screamingfrogseospider_24.3_amd64.deb"
sudo apt-get install "$HOME/Downloads/screamingfrogseospider_24.3_amd64.deb"

If the architecture command returns arm64, choose the official arm64 download and update both the URL and local filename in the commands. When using a newer release, take its exact package URL from the official download page.

SEO Spider 24.3 includes Java, so a separate Java installation is unnecessary. You can inspect the application's runtime under Help > Debug in Java Info. Follow the vendor's prerequisites if a later package changes this requirement.

Verify the installed package and executable:

dpkg-query -W -f='${Status} ${Version}\n' screamingfrogseospider
command -v screamingfrogseospider

The package status should begin with install ok installed, and the second command should print the executable path. Resolve any package or architecture error before continuing.

Open the application and activate your licence

In the remote desktop, open SEO Spider from the application menu or run screamingfrogseospider in a terminal. Review the end-user licence agreement on first launch.

The free edition can crawl up to 500 URLs, with restrictions on configuration and features such as saving and reopening crawls. For licensed features, open Licence > Enter Licence Key, enter the assigned username and key, and restart when prompted. Check that the licence is valid and the expiry date is correct.

Keep licence keys and any crawl credentials out of shell history, shared screenshots and shared exports. Give each operator the licence and access their use requires.

Configure storage and memory

Open File > Settings > Storage Mode. Database Storage is the default and stores crawl data on disk, which makes it a useful starting point on an NVMe-backed VPS. Licensed crawls can be reopened from File > Crawls.

Under File > Settings > Memory Allocation, apply the vendor's headroom rule and allow extra room for any other services. Save the setting and restart SEO Spider when prompted. Allocating all the server's RAM can leave the desktop and operating system short of memory.

Before crawling, record:

1. Free memory after the desktop and SEO Spider have started.
2. Free space on the filesystem holding the crawl database.
3. SEO Spider's configured memory allocation.

Use free -h, df -h and pgrep -af screamingfrog in the desktop terminal to check the system. Resolve heavy swapping or low disk space before starting the test.

Run a small test crawl

Use a site you own or have permission to crawl. With a licensed installation, set Configuration > Spider > Limits > Limit Crawl Total to 100. This walkthrough uses that limit for its first test; the free edition restricts configuration and has its own 500-URL ceiling.

Enter the site's canonical HTTPS URL in Enter URL to spider, select Start and watch the run. Keep the crawl speed within the target site's agreed limits.

Record URLs completed and queued, peak memory, swap activity, database growth and free disk space. Review HTTP errors, timeouts, blocked requests and unexpected external hosts. Note the duration and any effect on the target site's response time.

Check that the results are plausible and stay within the agreed resource and site limits. Disconnect and reconnect to the desktop without logging out, then confirm the crawl or completed results remain accessible. Increase the crawl scope gradually after the small test passes.

Export results and keep independent copies

Export the tabs and reports needed for the audit to a dated project folder. Open the exports and check their contents and row counts before copying them off the server.

Licensed database-storage crawls appear under File > Crawls. Use File > Export to create a portable .dbseospider file for another licensed installation. Ordinary CSV or spreadsheet exports are useful for people who only need the findings. The free edition can export available result data, but saving and reopening crawl projects requires a licence.

Copy the crawl file, required exports and a short configuration note to storage outside the VPS. Restrict access where crawls contain authenticated pages, personal information or client data, and set a retention period appropriate to the project. Test that the intended user can open the independent copy.

Maintain access and the crawl workstation

After the first successful crawl:

• Keep RDP restricted to the required source addresses, or disable xrdp when you no longer need interactive access.
• Use the non-root operator account and review sudo permissions when the team changes.
• Schedule OS and SEO Spider updates, then repeat a small crawl after significant changes.
• Limit access to licence keys, API credentials, authenticated crawl data and exports.
• Monitor memory, disk space, failed logins, application failures and crawl duration.
• Keep testing that independent crawl copies can be opened or restored.

Virtarix provides self-managed infrastructure. Include desktop administration, firewall configuration, updates, monitoring and recovery in the time and cost of running the workstation.

Compare a VPS with a local machine or cloud desktop

A VPS is useful for a persistent remote workstation that you can administer. A local machine may be simpler for occasional small crawls. A managed cloud desktop may save maintenance time if its service includes the desktop tasks you need.

Compare the same test crawl and include these costs and responsibilities:

Virtarix's uptime SLA is 99.99%; application sessions and crawl recovery still need their own checks. Choose the setup that meets the crawl, access and recovery requirements at an acceptable total cost.

Official sources

• Screaming Frog SEO Spider download page: Ubuntu package version and architecture links.
• Screaming Frog general user guide: installation, launching, crawling and exports.
• Screaming Frog configuration guide: crawl limits, storage mode and memory allocation.
• Screaming Frog FAQ: bundled Java, licensing and operating-system requirements.
• xrdp project documentation: RDP transport, port and sessions.