Running Screaming Frog in the Cloud using a Virtarix Server

Running Screaming Frog SEO Spider on a VPS gives one authorised operator a remote desktop that can continue a crawl after their laptop disconnects. This guide takes an Ubuntu server from a fresh login to an installed, licensed, test-crawled setup, then shows how to preserve the resulting crawl data.

The worked path assumes Ubuntu 24.04 because that is the Linux release covered by Screaming Frog's current installation guide. Confirm the server architecture, operating-system release, software licence, crawl permission, and recovery plan before you begin.

Last checked: 11 August 2026.

What you need before you start

Prepare these items before provisioning or changing the server:

• A 64-bit Ubuntu 24.04 server whose CPU architecture matches the official amd64 or arm64 package.
• SSH access through a non-root administrative account, with a second working session kept open while changing remote-access or firewall settings.
• A Screaming Frog licence assigned to the person who will use the software when the crawl will exceed the free 500-URL limit or needs licensed features such as saving and opening crawls.
• Written permission to crawl each target and a conservative crawl-speed policy that will not overload the site.
• Enough free NVMe storage for the application, desktop, database-mode crawl files, exports, logs, and temporary data.
• An independent destination for crawl files and exports. A copy on the same VPS is not an independent backup.

Screaming Frog's current platform FAQ says the application requires a 64-bit operating system and includes its own Java runtime. Its licensing FAQ states that a licence is assigned to one authorised user, so a shared remote server does not turn one licence into a multi-user licence.

Choose the VPS size from crawl memory requirements

Size from a measured crawl, not from a universal URL count. Page complexity, JavaScript rendering, stored HTML, screenshots, custom extraction, integrations, storage mode, and concurrent application instances can all change memory and disk demand.

The vendor's memory-allocation guidance says to keep at least 2 GB below the machine's physical memory available for the operating system and other applications. On a 6 GB server, that makes 4 GB the maximum allocation allowed by that rule, not a guaranteed or automatically safe target. The desktop session and other processes still have to fit in the remaining memory.

Virtarix Cloud VPS S is a starting configuration to benchmark, not a crawl-capacity promise.

*Unlimited bandwidth is subject to fair use, acceptable-use requirements, network integrity, and service limits. The included backup and snapshot do not replace customer-owned copies or an independent recovery plan.

Start with a short crawl and record peak memory, CPU, disk growth, completion time, and errors. If the acceptance test does not fit, choose a separately published plan with more headroom before the production crawl; this guide does not claim an automatic or one-click resize path.

Install a desktop environment or remote-access method

Screaming Frog has a graphical interface, so the Ubuntu VPS needs a customer-managed desktop and a controlled way to reach it. This worked example uses XFCE and xrdp from Ubuntu's package repositories. The xrdp project documents that xrdp accepts RDP clients, uses TLS for transport by default, and listens on TCP port 3389.

Keep the current SSH session open. Run the package commands from a non-root account with sudo:

sudo apt update
sudo apt install --yes xfce4 xfce4-goodies xrdp
printf '%s\n' 'startxfce4' > "$HOME/.xsession"
sudo systemctl enable --now xrdp
systemctl is-active xrdp

The final command must return active. Restrict TCP 3389 to a known administrative source address in the customer-configured firewall before connecting. Do not publish the port to the whole internet, do not close the working SSH session until a second login succeeds, and do not enable a new firewall policy until SSH has an explicit tested allowance.

Connect with an RDP client as the non-root operator. Confirm that the XFCE desktop loads, the session can disconnect and reconnect, and the operator can open a terminal. If any check fails, restore the previous access rule from the existing SSH session instead of continuing with the application install.

Install Java only if the current Screaming Frog package requires it

Do not install a separate Java package for SEO Spider 24.3. Screaming Frog's current platform FAQ says the Linux application includes a bundled Java runtime, and the application exposes the active runtime under Help > Debug as Java Info.

This rule is deliberately version-bound: if a future official package or installation guide explicitly removes the bundled runtime or names a prerequisite, follow that primary documentation for that package. Do not guess a JDK version from an old tutorial or add a system-wide Java repository pre-emptively.

Install Screaming Frog SEO Spider from the official package

The official download page listed SEO Spider 24.3 for Ubuntu on the check date. For an x86-64 VPS, download the amd64 package directly from Screaming Frog's HTTPS download host, inspect the package metadata, and install that exact file:

mkdir -p "$HOME/Downloads"
curl -fL --proto '=https' --tlsv1.2 \
  --output "$HOME/Downloads/screamingfrogseospider_24.3_amd64.deb" \
  "https://download.screamingfrog.co.uk/products/seo-spider/screamingfrogseospider_24.3_amd64.deb"
dpkg-deb --info "$HOME/Downloads/screamingfrogseospider_24.3_amd64.deb"
sudo apt-get install "$HOME/Downloads/screamingfrogseospider_24.3_amd64.deb"

Do not use the amd64 file on an ARM server. Select the official arm64 link from the same download page when dpkg --print-architecture returns arm64, and keep the filename and command consistent with that choice.

Verify installation without starting a crawl:

dpkg-query -W -f='${Status} ${Version}\n' screamingfrogseospider
command -v screamingfrogseospider

The package query must begin with install ok installed, and the second command must print the executable path. If either check fails, read the apt-get error, correct that exact dependency or architecture mismatch, and rerun the verification before continuing.

Launch and activate Screaming Frog

Open SEO Spider from the XFCE application menu or launch it from the remote desktop's terminal with screamingfrogseospider. On first use, read and accept the vendor's end-user licence agreement only if it is acceptable for your organisation.

The free edition can crawl up to 500 URLs but does not provide every configuration, scheduling, save/open, rendering, or integration feature. For a licensed setup, choose Licence > Enter Licence Key, paste the assigned username and key into their matching fields, and restart the application after the licence is accepted. The vendor FAQ says the licence view should show that the key is valid and display its expiry date.

Never put a licence key, site password, API token, or session cookie in a shell history, shared screenshot, exported configuration, or team-wide crawl folder. Each person using the application needs licensing that permits their use.

Configure crawl memory and storage

Open File > Settings > Storage Mode. Database Storage is the current default and stores crawl data on disk; it is the sensible starting mode for an NVMe-backed VPS because it can handle more crawl data for a given memory allocation and keeps licensed crawls available under File > Crawls.

Then open File > Settings > Memory Allocation. Apply the vendor's 2 GB headroom rule to physical RAM, subtract additional headroom for any other workload you intentionally run, save the change, and restart SEO Spider when prompted. Do not allocate all 6 GB of a Cloud VPS S to the application.

Before the first crawl, record these three values:

1. Free memory after the desktop and SEO Spider have started.
2. Free disk space on the filesystem holding the database-storage directory.
3. The configured memory allocation shown inside SEO Spider.

Use free -h, df -h, and pgrep -af screamingfrog from the desktop terminal to capture the operating-system view. If the server is already swapping heavily or the crawl filesystem is close to full before the test, stop and change the plan rather than treating the crawl as a valid benchmark.

Run a test crawl

Use a site you own or have explicit permission to crawl. In Configuration > Spider > Limits, set Limit Crawl Total to 100 for the first run. Enter the site's canonical HTTPS URL in Enter URL to spider, select Start, and watch the crawl rather than leaving it unattended.

Record:

• URLs completed and URLs queued.
• Peak application memory and whether the VPS swaps.
• Crawl-database growth and remaining disk space.
• HTTP errors, timeouts, blocked requests, and unexpected external hosts.
• Crawl duration and any effect on the target site's response time.

The test passes only when the crawl completes inside the agreed resource and target-site limits, the result set is plausible, no credentials were exposed, and the operator can disconnect and reopen the remote session without losing access. Increase crawl scope in measured steps; do not jump from 100 URLs to an unbounded production crawl.

Export results and persist crawl files

Export the specific result tabs and reports needed by the audit to a dated project directory. Check that each exported file opens and contains the expected row count before copying it off the VPS.

Licensed database-storage crawls are available through File > Crawls. Use File > Export when another licensed installation needs a portable .dbseospider crawl, and keep ordinary CSV or spreadsheet exports for people who only need the findings. The free edition can export available result data, but saving and reopening crawl projects is a licensed feature.

Copy the crawl file, required exports, and a short configuration note to customer-controlled storage outside the VPS. Define retention based on client contracts, personal-data requirements, and operational need. Every Virtarix VPS/VDS includes one backup and one snapshot, but there is no approved claim here about schedule, retention, storage location, consistency, restore mechanism, or charge; those recovery points do not replace the independent copy.

Security and remote-access cleanup

Treat the VPS as production infrastructure: patch the operating system, protect credentials, restrict administrative access, review installation packages, and test recovery procedures.

After the first successful crawl:

• Keep RDP limited to the smallest practical source-address range, or disable xrdp when interactive access is not required.
• Keep the crawl operator separate from root; review sudo access and remove access for former operators.
• Apply operating-system and SEO Spider updates in a planned window, then rerun a bounded crawl before relying on the new version.
• Store licence keys, API credentials, authenticated crawl data, and exports under least-privilege access.
• Monitor memory, disk space, failed logins, application failures, and crawl duration.
• Test that an independent crawl export can be restored or opened by the intended authorised user.

Virtarix supplies the allocated server infrastructure and access. Credentials, firewall and network configuration, operating-system and application updates, installed software, monitoring, data retention, independent copies, recovery planning, and the legal use of crawl data remain customer responsibilities.

When a VPS is better than a local machine or cloud desktop

A VPS is useful when one operator needs a persistent remote crawl workstation, wants the crawl isolated from their laptop, can administer Linux safely, and prefers a fixed monthly server allocation. A local machine is often simpler for occasional small crawls, while a managed cloud desktop may be a better fit when the team wants desktop administration included in a separate service.

Compare the same authorised test crawl before choosing:

Add the Screaming Frog licence, operator time, desktop administration, independent storage, monitoring, and recovery testing to both options. The winner is the setup that passes the same crawl, access, recovery, and total-cost criteria—not the one with the broadest infrastructure label.

Official sources checked

• Screaming Frog SEO Spider download page: current Ubuntu package version and architecture links.
• Screaming Frog general user guide: Ubuntu installation, launch, crawl, save/open, and export workflow.
• Screaming Frog configuration guide: crawl limits, storage mode, and memory allocation.
• Screaming Frog FAQ: bundled Java runtime, licensing, and operating-system requirements.
• xrdp project documentation: RDP transport, port, packages, and session behaviour.