DNS Security: DNSSEC, DoH/DoT and Filtering

DNS security is the discipline of protecting the name resolution layer so that a domain always points to the legitimate IP, travels encrypted and is never used as an attack channel. This article is a deliberately defensive guide: it does not describe how DNS is attacked (our pharming, spoofing and Man in the Middle pieces already cover that), but how it is hardened. It walks through DNSSEC and its real limits, encrypted transport DoH versus DoT, resolver hardening, protective DNS as a control against malware C2 and phishing, and closes with an actionable checklist for SMBs.

Why DNS is a defensive blind spot

DNS was designed in the 1980s to work, not to be secure. The base protocol travels over UDP on port 53, with no encryption and no origin authentication. Any device in the path can read which domains a user looks up and, under the right conditions, forge the answer. For decades this was accepted as a lesser evil, which is exactly why DNS became the classic blind spot of many architectures: HTTP traffic, email and endpoints get monitored, but nobody watches name lookups.

That blindness has a cost. DNS resolution is the phase an attacker manipulates in a pharming attack, it is the vector for spoofing techniques against name resolution, and it is often the first step of a Man in the Middle. At the same time, roughly 90% of modern malware touches DNS at some point: to locate its command and control (C2) server, to exfiltrate data through tunnelling or to resolve algorithmically generated domains (DGA). Defending DNS closes an entry vector and an exit channel at once.

DNSSEC: what it signs and what it does not protect

DNSSEC (DNS Security Extensions, defined in RFC 4033 to 4035) adds origin authentication and integrity to DNS answers through digital signatures. Each zone signs its records and publishes the signature in RRSIG records, the public key in a DNSKEY record and a hash of that key (DS) in the parent zone. This builds a chain of trust that climbs up to the root, whose key is anchored globally. A validating resolver checks that chain and rejects any answer whose signature does not add up, which neutralises cache poisoning and most answer forgery.

The point most often misunderstood: DNSSEC provides no confidentiality. It signs, it does not encrypt. A network observer still sees in the clear which domain is queried and which IP is returned. DNSSEC guarantees that the answer is authentic and untampered, nothing more. Query privacy is a job for encrypted transport, covered in the next section. Another nuance: the NSEC records that prove the non existence of a name allow zone enumeration (zone walking); NSEC3 with salt and iterations mitigates that problem, though it does not fully remove it.

How to deploy DNSSEC

1. Pick a modern algorithm. ECDSA P-256 (algorithm 13) or Ed25519 (algorithm 15) produce small, fast signatures. Avoid RSA/SHA-1, which is obsolete.
2. Sign the zone. With BIND you use dnssec-signzone or inline signing; Knot DNS and PowerDNS offer automated signing and rollover, which greatly reduce operational risk.
3. Publish the DS record at the registrar. This step closes the chain of trust with the parent zone. Without the DS published, the signature does not validate upward.
4. Verify. Confirm validation with dig +dnssec example.com (look for the ad flag, authenticated data), with delv or with the graphical tool DNSViz.
5. Automate key rollover. Separate the KSK (Key Signing Key) and ZSK (Zone Signing Key) and schedule rotation. An expired signature makes the zone unreachable for validating resolvers, one of the most common failures.

Encrypted transport: DoH versus DoT

Encrypted transport solves what DNSSEC does not touch: query confidentiality. There are two main standards.

• DoT (DNS over TLS, RFC 7858). Wraps DNS inside TLS on a dedicated port, 853. Because it uses its own port, an organisation can tell it apart clearly: it can allow it toward its trusted resolvers, monitor it or block it.
• DoH (DNS over HTTPS, RFC 8484). Sends queries as HTTPS requests over port 443, blended with the rest of web traffic. This maximises privacy against an external observer, but from the enterprise point of view it hurts visibility: DNS is no longer distinguishable from normal HTTPS traffic.

The enterprise trade-off is clear. On a corporate network, DoT toward a trusted internal resolver is usually the better option: it encrypts the query up to the resolver, keeps visibility and respects filtering policy. The real risk is browser DoH, which can bypass the corporate resolver and defeat all DNS filtering. Firefox and Chrome allow it to be disabled by policy (ADMX/GPO) and honour the canary domain use-application-dns.net: if its resolution fails, the browser disables its automatic DoH. For maximum-privacy scenarios there is also ODoH (Oblivious DoH), which separates the client identity from the query content through an intermediary proxy.

Resolver hardening

The recursive resolver is the heart of internal DNS security. Hardening it rests on several concrete controls, in the same spirit as systems and network hardening:

• Trusted upstreams. Forward only to known resolvers (Quad9, Cloudflare, a managed service) and, preferably, over DoT. Avoid open recursion to the Internet.
• Close open recursion. A resolver that answers recursive queries for anyone is an amplification weapon in reflection DDoS attacks. Restrict recursion to internal networks.
• Response Rate Limiting (RRL). Caps the rate of identical answers and curbs both amplification and part of the poisoning brute force.
• Cache poisoning defences. Source port randomisation, case randomisation (0x20 encoding) and DNS Cookies (RFC 7873). These answer families such as the Kaminsky attack (CVE-2008-1447) and the more recent SAD DNS (CVE-2020-25705). Keeping software current also closes critical server flaws such as SIGRed (CVE-2020-1350) in Windows DNS.
• QNAME minimisation (RFC 9156). The resolver sends each authoritative server only the part of the name it needs, reducing exposure of the full query.
• Split-horizon (views). Keep the internal view separate from the external one so that private infrastructure never leaks into public answers. BIND (views), Unbound and Knot Resolver implement this pattern.

Protective DNS: filtering against C2 and phishing

Protective DNS (PDNS) turns the resolver into an active security control: before resolving, it checks the domain against threat intelligence and blocks or redirects (sinkholes) anything malicious. It is one of the best controls by cost to impact ratio because it acts before the connection is established, fits naturally into a Zero Trust architecture and covers several scenarios at once:

• Malware C2. If a host gets infected, blocking resolution of the command and control domain cuts communication with the attacker even after the binary runs.
• Phishing. Blocking known phishing domains and newly registered domains (a strong abuse signal) reduces the successful click.
• DGA and tunnelling domains. Pattern analysis detects algorithmically generated domains and anomalous TXT or NULL queries typical of DNS tunnel exfiltration (iodine, dnscat2).

In practice it is implemented with Response Policy Zones (RPZ) fed by threat feeds, or with managed services such as Cisco Umbrella, Cloudflare Gateway, NextDNS or Quad9. In the public sphere, DNS4EU in Europe and PDNS services from agencies such as the NCSC offer reference filtering. Both NIST and CISA recommend PDNS as a baseline control.

DNS security checklist for SMBs

A small business does not need a dedicated DNS team to raise its posture significantly. This is the priority order:

1. Sign your own domains with DNSSEC and publish the DS at the registrar.
2. Pin a trusted resolver across the whole organisation (for example 9.9.9.9 with filtering) and use DoT toward it.
3. Turn on protective DNS that blocks C2, phishing and newly registered domains.
4. Disable browser DoH by policy so it cannot bypass corporate filtering.
5. Close open recursion and apply rate limiting on any resolver you operate.
6. Log and review DNS telemetry: it is one of the most cost-effective signals for spotting a compromise.
7. Keep DNS software patched against critical server flaws.
8. Monitor DNSSEC signature expiry to avoid outages from an expired RRSIG.

If your organisation lacks the staff to operate these controls, Secra's managed cybersecurity service integrates DNS filtering, query monitoring and incident response into a single layer.

Frequently asked questions

Does DNSSEC encrypt my DNS queries?

No. DNSSEC signs answers to guarantee their authenticity and integrity, but it provides no confidentiality: a network observer still sees the domain and IP in the clear. To encrypt the query you need encrypted transport such as DoT or DoH. They are complementary controls, not alternatives.

Which is better for a business, DoH or DoT?

For a corporate network, DoT toward a trusted internal resolver is usually preferable: it encrypts the query while keeping visibility and filtering. The risk is browser DoH, which can bypass the corporate resolver; it is best disabled by policy and used only in a controlled way.

Does protective DNS replace antivirus or the firewall?

No, it is an additional layer. Protective DNS acts before the connection is established, cutting resolution of malicious C2 or phishing domains. It complements EDR, the firewall and email security, but does not replace them: it is part of defence in depth.

Does DNSSEC stop pharming and cache poisoning?

DNSSEC neutralises cache poisoning and forged answers when both the zone signs and the resolver validates. It does not protect against pharming based on the local hosts file or against compromised routers, because those attacks happen outside the DNS chain of trust.

How do I check whether a domain has DNSSEC enabled?

With dig +dnssec domain.com and checking the ad flag (authenticated data) in the answer from a validating resolver, or with the delv tool. For a visual diagnosis of the full chain of trust, DNSViz shows each link and points out where validation breaks.