Running your own mail server puts you in full control of your email infrastructure. Mailcow is an open-source mail server suite built for Docker that bundles everything you need into one solution. This guide covers hosting your own email server on a VPS using Mailcow.
Requirements
Your server needs adequate resources to run Mailcow smoothly. Let’s review what you should have in place before we begin the installation.
Hardware Requirements
You’ll need at least a dual-core processor (or a quad-core processor for higher volumes). Mailcow needs 6GB of RAM to run all components comfortably. The system can run on 2GB with ClamAV disabled, but 6GB is recommended for production. Disk space starts at 20GB minimum. For multiple domains and users, start with 50GB or more. A static IP address is required for DNS records.
Software Requirements
Install a modern Linux distribution as your foundation. Debian 12 and Ubuntu 24.04 LTS work excellently and get regular testing from the Mailcow team. You need Docker Engine version 24.0.0 or newer. Don’t use Docker packages from your distribution’s default repositories – install directly from Docker’s official repositories.
Docker Compose version 2.0 or higher is required. Modern Docker installations include Compose as a plugin. Your system also needs these command-line tools: git, openssl, curl, awk, grep, cut, and jq. Most distributions include these by default, except jq.
DNS Preparation
Set up basic DNS records before installation:
- A record pointing your mail hostname to the server IP
- MX record for your domain pointing to the mail server hostname
- SPF record: v=spf1 mx ~all (authorizes your MX host to send mail)
You’ll add DKIM and DMARC records after installation. For detailed DNS configuration help, see Setting up DNS with Cloudflare.
Mailcow Installation Steps
System Preparation
Update your system packages to the latest versions:
sudo apt update && sudo apt upgrade -y
Install required packages:
sudo apt install -y git openssl curl gawk coreutils grep jq
Check that no other services are using the ports Mailcow needs:
ss -tlpn | grep -E -w '25|80|110|143|443|465|587|993|995|4190'
If any ports show up, stop those services before continuing. Port conflicts will prevent Mailcow from starting. Set the correct umask:
umask 0022
Firewall Configuration
Configure your firewall to allow the ports Mailcow needs. If you’re using UFW, here’s how to set it up properly.
sudo ufw allow 25/tcp
sudo ufw allow 80/tcp
sudo ufw allow 110/tcp
sudo ufw allow 143/tcp
sudo ufw allow 443/tcp
sudo ufw allow 465/tcp
sudo ufw allow 587/tcp
sudo ufw allow 993/tcp
sudo ufw allow 995/tcp
sudo ufw allow 4190/tcp
These ports handle everything from SMTP mail transfer to IMAP access and webmail. Don’t skip any of them or you’ll run into connection problems later.
Docker Installation
Download and run the official Docker installation script:
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
Enable Docker to start automatically:
sudo systemctl enable --now docker
Verify Docker is installed correctly:
sudo docker --version
Modern Docker installations include Compose as a plugin. Verify with docker compose version.
Repository Setup
Clone the Mailcow repository:
cd /opt
sudo git clone https://github.com/mailcow/mailcow-dockerized
cd mailcow-dockerized
Run the configuration generator script:
./generate_config.sh
Enter your mail server hostname when prompted (like mail.yourdomain.com). The script validates your input and sets up the initial configuration. The system might ask if you want to disable ClamAV on servers with limited memory. Choose based on your available RAM.
Download and Launch Mailcow
Pulling Images
Pull all required Docker images:
sudo docker compose pull
This downloads multiple containers, including Postfix, Dovecot, SOGo, and Rspamd. The process takes several minutes.
Starting Services
Launch all containers in detached mode:
sudo docker compose up -d
Give it about three to five minutes to stabilize. Monitor startup progress:
sudo docker compose logs -f
Press Ctrl+C to stop following logs once services report they’re ready.
Verification
Check that all containers are running:
sudo docker compose ps
Look for “Up” status for each container. If any show “Restarting” or “Exit”, check their logs:
sudo docker compose logs containername
Initial Mailcow Setup
Web Interface Access
Navigate to https://mail.yourdomain.com. Certificate warnings on first access are normal. Log in with: Username admin, Password moohoo.
Security Configuration
Change the default password immediately (click admin username → Edit). Enable two-factor authentication.
Reverse DNS Configuration
Configure reverse DNS (PTR records) – critical for deliverability. Contact your hosting provider to set up PTR records pointing your server IP to your mail hostname.
Example: IP 1.2.3.4 → mail.yourdomain.com
Verify your PTR record:
dig -x YOUR_SERVER_IP +short
It should return your mail hostname. Set up PTR records for both IPv4 and IPv6 if you have both.
DNS Authentication Records
Go to Configuration → Mail setup → ARC/DKIM Keys.
Generate a new DKIM key for your domain. Select your domain, leave “dkim” as selector, and choose 2048 as key length. Click Add.
Copy the public key (starts with v=DKIM1). Add this to your DNS as a TXT record:
- Hostname: dkim._domainkey.yourdomain.com
- Value: The entire public key
Add a DMARC record. Create a TXT record:
- Hostname: _dmarc.yourdomain.com
- Value: v=DMARC1; p=none; rua=mailto:[email protected]
Start with policy “none” for testing. After a week or two, change to “quarantine” or “reject”.
Domain Setup
Go to Configuration → Mail setup → Domains → Add Domain.
Enter your domain name without a subdomain prefix (yourdomain.com, not mail.yourdomain.com). Set the domain description and configure quotas based on your storage. Defaults are 10GB per domain, 3GB per mailbox.
After adding, click the DNS button next to the domain. Verify all A, MX, SPF, DKIM, and DMARC records are configured.
Creating Mailboxes
Select Mailboxes from Configuration → Add Mailbox. Choose a username, pick your domain, and set a secure password. Add the owner’s full name (appears in the From field). Click Add.
Testing Your Mail Server
Verify everything works before production use.
Webmail Testing
Click SOGo, log in with your credentials. Send test emails to external addresses – verify delivery to inbox (not spam) and that SPF/DKIM pass in headers.
Email Client Configuration
Configure a desktop or mobile email client with these settings:
Incoming: mail.yourdomain.com, IMAP port 993, SSL/TLS
Outgoing: mail.yourdomain.com, SMTP port 587, STARTTLS
Username: Full email address for both
Authentication Verification
Send a test email to [email protected]. Verify SPF, DKIM, and DMARC pass. Use mail-tester.com for deliverability scoring (aim for 9/10+).
Log Review
Check Postfix logs for errors:
sudo docker compose logs postfix-mailcow | tail -100
Look for authentication failures or rejected messages. Successful deliveries show “status=sent”.
Check the mail queue:
sudo docker exec -it postfix-mailcow postqueue -p
A healthy server has an empty or very small queue.
Troubleshooting Common Issues
Even with careful setup, you might run into problems. Here’s how to diagnose and fix the most common issues you’ll encounter.
Container Won’t Start
If a container fails to start or keeps restarting, check its logs first. Replace containername with the actual container that’s having problems.
sudo docker compose logs containername
Port conflicts are most common – stop the conflicting service.
Permission issues happen when file ownership gets messed up. Run this command from the Mailcow directory to fix permissions.
sudo chown -R root:root .
Memory problems cause containers to restart unexpectedly. Check your available RAM with the free -h command. If you’re running low, consider disabling ClamAV or upgrading your server memory.
Email Delivery Failures
Check if emails are stuck in queue:
sudo docker exec -it postfix-mailcow postqueue -p
Connection timeout errors indicate port 25 is blocked – verify the firewall allows outbound SMTP. “Relay access denied” errors mean your SMTP authentication isn’t working. Verify you’re using the correct username and password. The username should be your full email address, not just the part before the @ symbol.
If major providers like Gmail are rejecting your emails, check your reverse DNS. Run dig -x YOUR_IP +short and verify it returns your mail hostname. Missing or incorrect PTR records are deal-breakers for deliverability.
SSL Certificate Issues
Certificate errors in the browser usually mean Let’s Encrypt provisioning failed. Check the acme-mailcow container logs to see what went wrong.
sudo docker compose logs acme-mailcow
Common causes: incorrect DNS or port 80 blocked. Ensure port 80 is open for Let’s Encrypt verification. If you’re behind a proxy or load balancer, Let’s Encrypt might not work automatically. You’ll need to manually configure certificates or use DNS validation instead of HTTP validation.
Authentication Problems
Verify the mailbox exists and is active in the web interface. Verify the Dovecot container is running properly. This handles all IMAP and POP3 authentication.
sudo docker compose ps dovecot-mailcow
If users can receive but not send email, check the Postfix logs for authentication errors. The credentials for incoming and outgoing should be the same, but sometimes email clients are configured differently.
Web Interface Not Accessible
If you can’t access the Mailcow web interface, check if the nginx-mailcow container is running. This serves the web interface.
sudo docker compose ps nginx-mailcow
Verify nothing else is using port 443. The web interface needs this port for HTTPS. If another service is using it, you’ll need to stop that service or reconfigure Mailcow to use a different port. Verify A record points to the correct IP. DNS propagation takes up to 48 hours.
Spam Filter Too Aggressive
If legitimate emails are being marked as spam, you need to tune Rspamd. Check the spam scores in the Rspamd web interface at https://mail.yourdomain.com/rspamd. Train the filter: move false positives to the Inbox, false negatives to the Junk.
Whitelist trusted senders if needed. Go to Configuration, then Routing in the Mailcow interface. Add whitelisted domains or email addresses that should never be marked as spam. Adjust the spam threshold if the filter is too aggressive across the board. The default threshold is 15 points. Emails scoring above this go to spam. You can raise it to 20 for less aggressive filtering.
Best Practices
Follow these practices to keep your mail server secure, reliable, and performing well. These aren’t optional extras but essential maintenance that prevents problems down the road.
Regular Updates
Update your Linux system monthly for security patches. Update Mailcow whenever new releases come out. The project releases updates monthly with security fixes and new features. Updating is straightforward with Docker.
cd /opt/mailcow-dockerized
sudo docker compose pull
sudo docker compose up -d
Docker volumes preserve data across updates. Check the Mailcow blog before updating for breaking changes.
Backup Strategy
Implement automated backups. Mailcow stores data in Docker volumes and config files. Use the built-in backup script to create consistent backups while the system is running. Run it from the Mailcow directory.
./helper-scripts/backup_and_restore.sh backup all
This creates a complete backup including email data, databases, and configuration. The backup file gets saved in the backup directory with a timestamp. Store backups off-server using cloud storage or a dedicated backup server.
Test the restore process quarterly to verify backup integrity.
Resource Monitoring
Monitor CPU, memory, and disk usage. Watch for trends indicating you need to scale up. Check disk space regularly – set alerts at 80% usage. Running out of disk space causes mail delivery failures and can corrupt databases.
ClamAV uses 1-2GB RAM. Upgrade or disable non-critical services if low on memory. Monitor mail queue sizes through the Mailcow web interface. A growing queue indicates delivery issues that need investigation. A healthy mail server has an empty or very small queue most of the time.
Access Security
Use strong passwords. Enforce password complexity in Mailcow settings. Enable 2FA for all administrative accounts.
Limit administrative access to specific IP addresses if possible. Configure your firewall to restrict access to the management interface from known safe locations. This dramatically reduces your attack surface. Review accounts regularly. Remove mailboxes inactive for 6+ months.
Log Management
Review logs weekly for suspicious activity. Watch for repeated authentication failures from the same IP. Configure log retention based on your compliance requirements. Mailcow rotates logs automatically, but you might need longer retention for audit purposes. Adjust retention settings in the mailcow.conf file.
Set up fail2ban integration for automatic IP blocking. Mailcow includes fail2ban support that blocks IPs after repeated failed authentication attempts. This stops brute force attacks automatically.
Conclusion
You now have a fully functional mail server running on Docker with Mailcow. Your system handles incoming and outgoing email with proper authentication, spam filtering, and security.
Users can access email through SOGo webmail or standard IMAP/SMTP clients. The Docker-based architecture makes maintenance straightforward – pull new images and restart containers for updates.
Monitor your server regularly and keep up with updates. A properly maintained mail server provides reliable service for years. For advanced configuration, consult the official Mailcow documentation at docs.mailcow.email.
