Screaming Frog SEO Spider is a popular tool among SEO professionals, digital marketers, and web developers. It helps analyse and optimise website performance, identify technical issues, and provide valuable insights that can assist in improving your SEO rankings.
However, running this powerful desktop software on a local machine can cause significant issues. Users often face high resource consumption, performance limitations, and system interference.
Imagine scheduling a comprehensive weekly website health check only to dread the inevitable slowdowns that make multitasking on your local machine nearly impossible. Fortunately, there’s an easy to implement solution: the Virtual Private Server (VPS).
Running Screaming Frog on a VPS ensures continuous operation without straining your local computer or network resources. It offers enhanced performance, solves bandwidth and scalability issues, and provides reliable data storage and management. Plus, you can access it remotely, making it easier to manage your SEO projects anytime, anywhere.
Why Run Screaming Frog on a VPS?
Running Screaming Frog on a VPS offers several compelling advantages. A VPS provides dedicated and scalable resources, ensuring smooth operation without overloading your local machine. This results in superior performance, faster crawl times and the potential to perform MASSIVE crawls due to enhanced hardware capabilities.
Additionally, using a VPS leverages independent network bandwidth, preventing local internet slowdowns. The redundancy and uptime provided by a VPS far outmatch what you can achieve when running locally, making it ideal for long or scheduled crawls that require continuous operation.
The flexibility to adjust resources based on your crawl size and frequency offers scalable options tailored to your specific needs. A VPS also excels in data storage and management, handling large datasets more effectively and providing improved backup options. By running Screaming Frog on a VPS, you minimise interference with your local system, keeping your machine free for other tasks. This setup not only enhances efficiency but also optimises your overall workflow.
Big 3 Cloud Providers vs. VPS: Which is Better for Screaming Frog?
Considering its resource-intensive processing, many users seek alternatives to running Screaming Frog on local machines. Two popular options are deploying it with major public Cloud providers or on a VPS. Each has its pros and cons, so let’s explore which might be the better choice.
Using Major Cloud Providers
Running Screaming Frog with major Cloud providers like Google Cloud, AWS, Microsoft Azure, IBM Cloud, and Alibaba Cloud provides easy scalability, allowing users to adjust resources for any crawl size. This ensures optimal performance without hardware limitations. These Cloud services also enable global accessibility, making it easy to manage SEO projects from anywhere with an internet connection. Built-in redundancy and failover mechanisms ensure high availability and reliability, minimising downtime during peak times.However, using Cloud solutions from these major providers comes with ongoing costs, such as subscription fees and usage-based pricing, which can fluctuate and potentially increase expenses over time. This unpredictability in cost can be a significant consideration for long-term projects or larger crawls. .
On a VPS
On the other hand, running Screaming Frog on a VPS provides dedicated resources, including CPU, RAM, and storage, ensuring consistent performance and faster crawl times. This setup offers improved stability and uptime, ideal for long or scheduled crawls, with fixed pricing plans that offer budget control, avoiding surprises from usage-based billing. For a monthly cost from $50 to $100 (or more, depending on your VPS size), you can have a Screaming Frog VPS running around the clock and with no unexpected charges.
VPS solutions also offer high levels of control and customisation, allowing you to install and customise any software you need. The predictable costs and control over the environment are beneficial for users with specific requirements and technical expertise. However, this comes with the responsibility of managing and maintaining the server, which may require additional time and effort. For agencies, companies, or individuals with in-house technical expertise, running on a VPS is an easy choice. They can handle the necessary tasks themselves, with support from Virtarix available when needed.
| Aspect | Running Screaming Frog with Virtarix | Running Screaming Frog on a VPS | Running Screaming Frog with Major Cloud Providers |
|---|---|---|---|
| Backups | Included in the VPS price at no additional cost. | Manual setup required and usually with an additional charge, or can require third-party backup software. |
May require additional setup or incur extra costs for backup services. |
| Control | Full control over the VPS, including an easy-to-use client management console. | Control of the Virtual Machine. | Some limitations and configurations are managed by the provider. |
| Customisation | Customisable to your precise needs. |
High level of customisation with the ability to install and configure any software. |
Customisation is limited by the services and configurations offered by the provider. |
| Security | Users are responsible for VPS security. We provide off host firewall controls that allow you to lockdown your machine to ensure no unwanted access. | Users are responsible for most security measures. |
Advanced security features and compliance certifications available from the provider. |
| Compliance | Users must ensure compliance with specific regulatory standards. |
Users must ensure compliance with specific regulatory standards. |
Many Cloud providers offer compliance with various international standards, which can simplify adherence. |
Deployment Speed |
Can be up and running your first crawl in as little as 15 minutes. This time will be halved with our soon to be launched Screaming Frog template. |
Slower, as it requires manual configuration and setup. |
Faster, with automated tools and templates available for quick provisioning and scaling. |
| Uptime and Availability | Reliable and redundant across multiple nodes. |
High availability minimises downtime, ideal for long or scheduled crawls. |
Built-in redundancy and failover mechanisms ensure continuous operation. |
Pricing |
Fixed monthly bill at a highly attractive price point per month. |
Fixed monthly pricing means no surprise bills. |
Usage-based pricing can lead to unpredictable monthly costs. |
Real-World Application: Running Screaming Frog on a Virtarix VPS
" Running Screaming Frog on a Virtarix VPS allows us to conduct extensive website crawls without slowing down our local machines during processing. This setup enables seamless project sharing without the hassle of exporting and importing for reviews, enhancing collaboration and providing flexible access to data when needed.
We can run intensive crawls on Screaming Frog while handling other tasks, preventing the wasted hours our local machines used to endure. Automation is streamlined through the VPS, eliminating the need to keep local machines running.
We opted for VPS over Compute due to cost-efficiency, especially with our in-house technical expertise. It’s worth mentioning that BILL SHOCK was a reality: one of our team members once left a Google Compute instance running, resulting in a $5,000 USD bill for the month due to the power setup. Compared to a VPS running for a month, it’s a no-brainer. While this approach saves costs, it's crucial to maintain proper licensing compliance with tools like Screaming Frog to support their development efforts and strategy."
Chris Avery
Founding Partner at Mickey Llew
Setup and Implementation
If you were to run Screaming Frog on Google Compute, there are numerous lengthy and confusing steps involved (see for yourself in this installation guide). In comparison, setting it up on a VPS involves only a few critical steps detailed below.
However, with Virtarix, the process is even simpler! We provide an instance of Screaming Frog on a VPS that's ready to go. Avoid the complexities and enjoy a streamlined, efficient setup process with only a few clicks.
If you want to do the heavy lifting and set up Screaming Frog on a VPS yourself, these are the steps you would need to take:
- Choosing the Best VPS Plan for Screaming Frog : Firstly, choose a VPS plan that aligns with Screaming Frog's hardware recommendations. Opt for an option slightly above the base recommendation to handle your typical crawl sizes efficiently, without requiring extra RAM.
Our Windows VPS L or XL, equipped with NVMe disks for optimal performance, is recommended. We also offer expanded disk space options, ideal for crawling sites with over 5 million URLs.
- Installing Screaming Frog on Your VPS : Once your VPS is ordered and provisioned, log in via Remote Desktop and follow Screaming Frog’s steps for installation on Windows. For Linux or command-line versions, consider ordering a Linux-based VPS and follow the respective instructions.
- Configuration Tips : Ensure your server is secured with a complex password. Additionally, optimise security performance by configuring firewall settings like changing RDP to a custom port number scheduling regular updates, and allocating Screaming Frog resources according to their best-practices guide. For seamless data sharing across your VPS and office, install a file sync tool like Google Drive or Z1 Storage.
- Licensing Compliance : Ensure you maintain proper licensing compliance when utilising tools like Screaming Frog on your VPS. This ensures you support the developers and adhere to legal requirements. Remember, licences are per user, so ensure you have one for each user accessing the tool to maintain compliance and maximise its benefits effectively.
Leveraging a VPS to run Screaming Frog can offer major advantages for SEO audits and website optimisation. With dedicated resources, reliable performance, and seamless remote access, managing extensive crawls and fostering collaboration becomes effortless. By implementing optimal setup, configuration, and ensuring licensing compliance, users can maximise Screaming Frog's capabilities on a VPS, empowering efficient workflow management and robust digital marketing strategies.
