Self-Hosted Email Servers: Setup, Security and Maintenance

Self-hosting email puts you in charge of the mail software, accounts, storage and recovery process. It also puts delivery failures, compromised accounts and out-of-hours maintenance on your team. Installing the software is only the beginning.

This guide covers the decisions behind a working mail service: where to run it, which components to use, how to configure authentication, and what to monitor after launch. Start by confirming that your intended network permits mail-server traffic and gives you the DNS and IP controls the service needs.

Decide whether self-hosting fits your team

Self-hosting can suit an organisation that needs specific routing rules, integrations or control over mailbox storage. It requires someone who can maintain the operating system, investigate queues and authentication failures, and restore service when a component breaks.

A hosted mail provider takes on more of the infrastructure work. You still manage accounts, access policies and retention, but the provider operates the mail platform. Compare the actual contract, export options, data locations and recovery arrangements of each service.

Neither approach guarantees privacy or compliance. Self-hosted messages still pass through recipients' servers and may be handled by a hosting provider, relay or backup service. Location control depends on where every relevant copy is stored, including logs and archives.

Compare total costs using your own mailbox count and retention requirements. Include infrastructure, backup storage, filtering, relay charges, licences and administrator time. Avoid choosing from a headline server price or a fixed cost-per-user assumption alone.

Understand the main components

SMTP transfers mail between servers and accepts outgoing messages from authenticated clients. An SMTP service needs rules that distinguish local delivery, authorised submission and unauthorised relaying.

IMAP gives clients access to server-side mailboxes and synchronises folders across devices. POP3 downloads messages and can retain or delete the server copy, depending on client settings. Webmail adds a browser interface; it does not replace the underlying delivery and storage services.

A complete installation also needs spam filtering, malware controls appropriate to the workload, certificates, DNS authentication, logs and backups. Check which pieces your chosen package includes and which you must operate separately.

Check network and DNS prerequisites

Before installing software, confirm:

• You control the domain and can publish its DNS records.
• The hosting network permits the required inbound and outbound mail traffic.
• You have a stable address and can configure reverse DNS through the IP provider.
• The address is suitable for mail use and you have a process for investigating reputation problems.
• The selected hostname resolves to the address you intend to use.
• A recovery contact and alert destination remain available if this mail server fails.

Direct delivery between Internet mail servers uses TCP port 25. Client submission on ports 587 or 465 does not substitute for direct server-to-server delivery. An authorised outbound relay is a separate design choice and adds its own terms, authentication and sending limits.

Publish an MX record pointing to the receiving server's hostname. Its address record must resolve correctly. Configure the PTR record for the sending IP through the provider, and keep the hostname, forward lookup and SMTP identity consistent. Publish an IPv6 address only if routing, filtering, reverse DNS and mail delivery work over IPv6 too.

Configure SPF, DKIM and DMARC

SPF identifies the senders authorised for a domain's envelope-sender identity. Inventory every legitimate sending service before publishing the record. Keep one SPF record per domain and observe the protocol's DNS-lookup limit. A strict policy added before that inventory can break legitimate mail.

DKIM adds a digital signature that receiving systems can verify using a public key in DNS. It authenticates the signature and signed message content; it does not encrypt the message body. Use the key sizes supported by your software and receiving providers, protect private keys and plan rotation.

DMARC checks whether a passing SPF or DKIM identity aligns with the visible From domain. Start with reporting, investigate legitimate senders that fail alignment, then move towards quarantine or rejection when the results support it. A DMARC policy is not a guarantee that a receiver will accept a message or place it in the inbox.

Read the receiving provider's requirements. For example, Google's sender guidelines distinguish ordinary senders from bulk senders. Do not treat one provider's threshold as a universal protocol requirement.

BIMI is optional branding support, separate from basic mail delivery. Its certificate and display requirements depend on the mailbox provider. It is not a prerequisite for sending production email.

Choose your mail software

Mailcow

Mailcow packages mail transfer, mailbox access, filtering, webmail and administration in a Docker-based stack. Its integrated interface can reduce the work of assembling those services yourself.

You still configure the domain's DNS, secure administrative access, manage users and keep the host and stack updated. Mailcow can generate information such as DKIM records, but it does not automatically control your external DNS zone. Capacity depends on mailbox activity, filtering, indexing and enabled components, not a universal user limit.

Check the project's system requirements before selecting hardware. Use its supported update and backup procedures; pulling a few container images is not a complete maintenance workflow.

A modular mail stack

Postfix handles SMTP transfer and routing. Dovecot provides mailbox access through IMAP and POP3 and can participate in authentication and local delivery. They are often combined, but neither alone supplies every part of a complete mail service.

This approach gives an experienced administrator control over routing, storage and integration. It also requires explicit decisions about virtual users, certificates, authentication, filtering, webmail and backups. Document how the components connect so another administrator can troubleshoot them.

Exim

Exim is another mail transfer agent with configurable routing and access controls. It can suit a team already familiar with its configuration or maintaining an existing deployment. Evaluate the surrounding mailbox, filtering and administration tools alongside the MTA.

Microsoft Exchange

Exchange Online is Microsoft's hosted service. Self-hosted Exchange is a separate product: Microsoft's Exchange Server documentation describes that deployment path.

Choose between them based on licensing, supported platforms, identity integration, client features and operational capacity. Do not assume that a generic IMAP stack provides every Exchange collaboration feature or that self-hosted Exchange has disappeared.

Secure the service before adding users

Certificates and encryption

Use trusted certificates for the names clients connect to. Automate renewal where supported and monitor the result, including whether each mail service has loaded the renewed certificate.

Require encryption for authenticated client access. Disable obsolete TLS versions using the supported settings for your software, while checking client and peer compatibility. Client submission and Internet server-to-server delivery have different interoperability requirements; a single blanket TLS rule can interrupt mail.

Transport encryption protects a connection. It does not make stored mail end-to-end encrypted or prevent the recipient's service from reading it.

Firewalls and exposed ports

Open only the services you intend to provide. The common choices are:

Ports 143 and 110 can support STARTTLS in suitable configurations; they are not inherently limited to plaintext. Expose them only if required and enforce secure authentication. Restrict administrative and database interfaces separately.

For Docker deployments, verify forwarding and published-port rules. A host firewall's ordinary input rules may not control container traffic as expected. Test the reachable ports from outside the host.

Accounts and abuse prevention

Require strong, unique credentials and enable multifactor authentication on administrative interfaces. Confirm how each client protocol authenticates: a webmail MFA setting does not automatically protect every IMAP or SMTP login. App passwords and legacy authentication need their own controls.

Limit relaying to authorised users or systems. Test that an unauthenticated external client cannot relay mail between unrelated domains. Add appropriate rate limits, monitor login failures and use tools such as fail2ban where they fit the stack. A compromised account can send abuse even when the server is not an open relay.

Updates

Track security advisories for the operating system and mail components. Before an update, review release notes, confirm a recoverable backup and plan any restart. Afterwards, test submission, external delivery, mailbox access, filtering and certificate status.

For Mailcow, follow its documented updater. For a package-based installation, use the distribution's supported update process and review configuration changes. Automated updates still need failure alerts and a way to confirm that services restarted successfully.

Plan storage, accounts and client access

Measure existing mailbox data where possible. Allow space for projected growth, indexes, queues, temporary files, logs and the working space needed during backups or maintenance. Use mailbox quotas with alerts before users reach them.

Storage performance depends on concurrent activity, search, filtering and backup I/O. SSDs can help active mailboxes, but size the complete workload rather than declaring one disk type sufficient for every deployment. Test under expected load and leave room for maintenance spikes.

If you archive older mail, verify that users can search or recover it and that retention rules apply consistently. Moving an archive to object storage does not by itself satisfy a legal requirement or make it a usable backup.

Give each administrator an individual account and only the permissions needed. Where the software supports it, separate system administration, domain administration and mailbox access. Define an offboarding process covering sessions, app passwords, forwarding rules, shared mailboxes and retained business records.

Test the actual clients your users rely on. Thunderbird, Apple Mail, Outlook and webmail differ in account discovery, authentication and groupware support. Record the server names, ports and encryption modes rather than assuming automatic configuration works everywhere. JMAP is another option when both the chosen server and client support it.

Choose a deployment model

A VPS places physical hardware and network operation with a provider while leaving guest administration with you. Confirm mail-use permission, address controls, storage performance and recovery options before buying. Resource upgrades may require downtime, and an advertised CPU or RAM allocation does not establish mailbox capacity.

An on-premise server gives you responsibility for hardware, power, cooling, connectivity and replacement parts. It can suit existing infrastructure or location requirements, but it still depends on upstream networks and external mail systems.

A hybrid setup may combine local mailboxes with hosted filtering, an outbound relay or off-site backups. Map the dependencies and data flows. More components can improve a particular recovery design, but they also create more authentication, monitoring and troubleshooting work.

Monitor delivery and practise recovery

Watch queue age and size, rejected and deferred messages, authentication failures, unusual sending activity, disk space, CPU, memory and certificate expiry. Investigate changes against normal traffic rather than relying only on whether the server process is running.

Centralised logs help trace a message across components. Keep timestamps consistent and limit access to logs containing addresses or message details. Send critical alerts to a channel outside the mail service being monitored.

Back up mailbox data, relevant databases, configuration and the keys needed for recovery. Store independent copies with controlled access. Test restoration to an isolated system, including account access and message retrieval, and record how long recovery takes.

Before launch, test incoming and outgoing mail with external providers, inspect SPF/DKIM/DMARC results, check client login, verify relay restrictions and exercise recovery. Authentication success is one part of deliverability; content, recipient consent, reputation and sending patterns matter too.

Review privacy and retention obligations

Hosting email yourself does not settle which laws apply. Assess the organisation's activities, the people whose data it processes, where providers operate and any cross-border transfers. The European Commission's data protection guidance and the California Attorney General's CCPA guidance are useful starting points for a jurisdiction-specific review.

For GDPR, establish the appropriate lawful basis for processing; consent is not the only possible basis. Define access controls, retention, handling of rights requests and an incident response process. Breach-notification duties depend on the facts and applicable rules.

CCPA applicability also depends on statutory criteria, rather than applying automatically to every server that holds a Californian address. Review consumer rights and any sale or sharing of personal information with the people responsible for your organisation's privacy obligations.

Turn those decisions into practical procedures: account access reviews, documented retention, protected audit logs, tested exports and deletion processes, and trained incident owners. Keep recovery copies in that review so backup retention does not undermine the intended policy.

Make an operating plan before launch

Choose self-hosting when its control is worth the ongoing work and someone can own that work. Record who handles updates, delivery problems, abuse reports and recovery. A successful pilot should demonstrate those responsibilities as well as sending and receiving a test message.