After signing .AT,  providing registrars an opportunity to test DNSSEC enhancements, and performing their own testing, nic.at began accepting DNSSEC delegations on February 29.

[This is an updated version of a blog post from about 2 years ago to reflect current situation and fix links].

According to various surveys a large majority of recursive resolvers and authoritative servers are DNSSEC ready.  This is measured by looking at the resolver  that sends queries with the ENDS0 and DO bit set.  Similarly, resolvers can measure this by seeing how many authoritative servers return answers with EDNS0 option.

When .GOV was signed some sites noticed that many queries for .GOV took a longer time and resulted in TCP queries.  When .ORG was signed there was a large spike in TCP queries to .ORG servers. Both of these spikes can be attributed to large DNSSEC answers that did not get through “narrow” DNS links. To be fair, in both cases the volume of TCP queries was amplified by “mistakes” in signing the zones.

Both zones are signed by NSEC3 (which is fine) but NSEC3 has certain properties that make some of the answers larger than if NSEC was used. GOV originally used 2048 bit RSA zone signing key; this means each signature was 287 bytes long.  In an NSEC3 signed zone each negative answer requires 4 signed RRsets, the SOA record plus 3 NSEC3 answers, resulting in answers over 1500 bytes long.  ORG’s mistake was to set the TTL of all records at the zone apex to 0, which effectively killed all negative caching for that zone. Once this was fixed, the TCP traffic volume fell. At the height of the TCP flood ORG’s servers were answering about 15% of all queries over TCP.

Both mistakes are actually a good thing, as they highlight certain important issues for future DNSSEC deployment and can help us avoid them in the future. GOV’s spike of TCP connections was primarily caused by “narrow” pipes; i.e. links that cannot handle DNS packets over certain size.  In this discussion we will focus on the size issue and how to address it.  ORG’s mistake further amplified TCP traffic from resolvers that are sitting behind “narrow” pipes and have ill-chosen fallback mechanisms when big packets cannot get through.

Some technical background: DNS, as defined 25+ years ago, specified a maximum payload size of 512 bytes over UDP.  This was appropriate at the time given the links available.  RFC2671, issued in August 1999, specifies a mechanism to extend DNS packets in various ways including specifying larger messages.  The RFC recommended larger message size of 4096 has become quite common, but because one of the ugly truths of the Internet is that no path is wider than the narrowest link, the large DNS answer does not get through all the time.

Question: “Why are there narrow links?”

There are number of reasons why large UDP messages cannot pass through links.  Most links close to edges use Ethernet frames that are about 1500 bytes long.  Thus, large TCP and UDP messages are broken up into units (fragments) smaller than 1500 bytes.  A large number of links do not allow UDP fragments to pass through or only the first fragment is passed on.  There are multiple reasons why this is the case:

– Some implementations in routers/NAT boxes do not know what to do with UDP fragments

– Some firewalls outlaw fragments as a security risk.

– Some firewalls “know” that all DNS are less than N bytes, thus any DNS packet larger that that is bad and is dropped.

If your site wants to use DNSSEC either as a consumer (in resolution) or as a producer (answering for signed zones) it is important that you know how large a message will get through your links, and if the path is restricted, tune your DNS systems to avoid transmission of packets that will not get through. The leading DNSSEC capable DNS resolver implementations – Bind, Unbound, CNS and MS DNS – have to set/change the advertised UDP size.  When DNS resolution problems occur it is human nature to want to point fingers at the remote site, but it is just as likely that the problem is local.  Fortunately, there are a number of tools that can be used to check resolvers and their paths:

–  DNS-OARC has a  tool and this Perl script (dns-path) can be used to check all the local resolvers to my code

– DNSFUNNEL is a sub tool in Vantage test tools

– NetAnzlr from UCB is a good network testing tool

Configuring DNS resolvers to ask for smaller answer (using 1480 as example):

Bind-9:

in options set edns-udp-size=1480

Unbound: 1.4.0+ (older versions do not support) set parameter

edns-buffer-size: 1480

Update: 18 months after the root was signed, there are no reports about  big problems related to link sizes, this is a result of people checking their links and zone publishers being aware of the issue and keep their RRset’s small enough that the packet size does not become an issue.

In order to experiment with validation and provide as simple a working example as possible to others, I have written vsResolver, a stub DNSSEC validator that can be run from the command line of any Windows, Mac or Linux end system. I hope you find it useful and that it helps you understand the complexities of DNSSEC validation.

vsResolver is a validating stub resolver that works in conjunction with a DNSSEC-aware recursive resolver to answer a DNS query with a validated result. vsResolver implements the DNSSEC validation protocol described in RFC’s RFC4033, RFC4034 and RFC4035.  vsResolver is not in itself a recursive resolver — it issues DNS queries to a specified DNS recursive resolver to do name resolution. vsResolver does however issue multiple additional queries that are necessary to perform DNSSEC validation. vsResolver works well with a validating recursive resolver such as the BIND recursive resolver from ISC or the Unbound recursive resolver from NLnet labs.

The audience for vsResolver is software developers and system administrators who are interested in an example of a DNSSEC validation algorithm, or who might be interested in a command line domain validation script. vsResolver is comparable to the tools available from Sandia Labs and Verisign Labs. vsResolver differs in that it is usable from the command line, and that it comes as BSD-licensed source code.

You might wonder why you would need vsResolver in addition to a validating recursive resolver. The simple answer is — you don’t. Unbound does validating recursive resolution all by itself. The point of vsResolver is to provide an example of DNSSEC validation that is as simple and straightforward as possible. vsResolver is written in python, a high-level language. (To use vsResolver, you need to have python, dnspython and pycrypto installed on a computer.) vsResolver leverages the dnspython DNS name resolution library and the pycrypto cryptography library and so does not re-implement any code unnecessarily. vsResolver is synchronous, so there are no confusing asynchronous interactions going on. vsResolver can generate diagnostic output that directly reflects the steps the algorithm takes to validate and determine the security outcome of a query. This output is directly comparable to the output from Verisign Labs’ DNSSEC debugger. This provides a way to verify the vsResolver’s security outcome or diagnose problems in the vsResolver validation algorithm.

vsResolver can function as either a utility program or as a software library. That is, vsResolver can run as a command line utility to determine the DNSSEC status of specific domains and resource record types. Alternately, the vsResolver API can be called from python code to do a DNS query and provide the results of that query along with its DNSSEC status, called a Security Outcome, in a variety of ways.

A DNS query is a request for a resource of a particular type from a particular domain. For example, a DNS query might ask for the A data (e.g., the IPV4 Address data) from the comcast.net domain. The result of such a DNS query is the resource data contained in the resource records that resulted from the query. vsResolver provides an additional Security Outcome that indicates the answer’s validity.

The Security Outcome has one of three values: Secure, Provably Insecure, or Bogus. A Secure outcome means that the DNS data returned in the result has a cryptographically validated chain of trust back to a trust anchor. A Provably Insecure outcome means that the result has been verifiably proven to not have a chain of trust back to a trust anchor (in other words, the result is proven to not be signed). A Bogus outcome means that the result is verifiably proven to have a secure chain of trust back to a trust anchor, but the chain of trust fails validation for one reason or another (in other words, the result is proven to be signed but the chain of trust is not valid).

vsResolver provides results within the context of one of three behaviors. In SecureOnly behavior mode, vsResolver only provides a normal result (e.g., a function return) if the answer cryptographically validates to be secure. Otherwise, vsResolver raises a BadResult exception. The BadResult exception contains information to determine whether the answer was actually insecure or bogus. In NoBogus behavior mode, vsResolver provides a normal result if the answer validates to be secure or provably insecure. In Permissive behavior mode, vsResolver provides a normal result regardless of the security outcome.

SecureOnly would be the preferred mode of operation except that many domains are not yet signed. For that reason, NoBogus is the recommended mode of operation for the near term until domains are signed as a matter of course. Operating in Permissive mode is not recommended at all, except for debug, as there is no reason to accept a bogus DNS answer.

vsResolver can run with three modes of trust anchors specified:  root trust anchor only, root trust anchor plus additional trust anchors, or additional trust anchors only.  Root trust anchor only is the normal mode of operation.  In this mode vsResolver uses the root trust anchor as the root of the validation chain of trust.   Root trust anchor plus additional trust anchors allows for the validation of domains that validate to disconnected trust anchors as well as validation of domains that validate to the root trust anchor.  For example, this mode allows for the validation of a signed subdomain of a domain that is itself unsigned. Finally, additional trust anchors only mode only validates domains that have chains of trust to specific trust anchors; for example, organization A only trusts organization B and no other.

vsResolver can determine the key of the root trust anchor on its own, at least for the near term, as it contains the signature of the present root key. The user can specify alternate signatures for the root key, for use in cases such as when the root key changes, or the user can specify that the root key should not be used. In addition, the user can specify the keys for additional trust anchors, which can be used.

We hope that vsResolver will be helpful as an example of a DNSSEC validation algorithm that can be used directly, or that can serve as an outline for the implementation of DNSSEC validation in other applications.

vsResolver is available at SourceForge.

We try to keep ourselves and you informed about all things related to the deployment of DNSSEC, but we can’t do it alone. There are other sites that also publish useful information from a variety of points of view, and we try to bring those to the attention of the community, but they too are incomplete. One source of information that is often overlooked in these blog posts and web sites is you. We appreciate that everyone involved in deploying DNSSEC is on the leading edge (yes, you are) and that you may have as many useful bits of information as questions.

Our solution, which hasn’t been given the spotlight that it deserves, is our wiki. Wander over to https://www.dnssec-deployment.org/wiki/, create an account, and help us document the DNSSEC world.  You’ll help others and may learn something in the process.

If you reached this site without the benefit of DNSSEC validation, you may have noticed that the website banner contains an icon declaring that “DNSSEC is Off.”  If you see a green check mark instead, you can pat yourself (or your ISP) on the back for protecting your hosts against DNS-based attacks, and for a job well done.

The difference is on account of a name that is resolved while trying to render the site.  The web browser tries to pull in a Cascading Style Sheet (CSS) file from a domain name (badsign-a.testsub.dnssec-deployment.org) that has deliberately broken DNSSEC signatures. This CSS file’s only purpose is to set the “DNSSEC is off” image on the DNSSEC Deployment website banner. With DNSSEC validation enabled, the browser does not load the CSS from badsign-a.testsub.dnssec-deployment.org, and the default header image with the green check mark shows up instead.

It shouldn’t come as a surprise to you that your browser was trying to load content from badsign-a.testsub.dnssec-deployment.org although you had not typed that in the address bar. More generally, it shouldn’t be surprising that it requires more than a single DNS lookup to fill the contents of a page. In fact, as the query trace from loading a relatively simple page such as www.dnssec-deployment.org illustrates below, an un-primed resolver easily performs in excess of a hundred lookups before the browser renders the complete page. Some of these queries are not even for names under the dnssec-deployment.org domain. For more content-packed sites the number of names looked up is even higher.

You might be somewhat surprised, though, that content that was pulled remotely (from badsign-a.testsub.dnssec-deployment.org in this example, but really could have been from anywhere) was able to modify the contents of the www.dnssec-deployment.org page so easily. Javascript and CSS are extraordinarily powerful in that respect. Would you even notice if only a portion of the website you were looking at was actually modified using one of these vectors? You might think that if you were validating all your DNS responses you would probably have been safe from this type of attack. However, consider widely-used software such as google-analytics, which relies on the browser fetching remote javascript, but does so from under a zone that is not signed (google-analytics.com is not signed). The zone is “provably insecure” so there is no way for your resolver to validate that name. There is no doubt that for the benefits of DNSSEC to be truly realized, we need zone owners to continue to sign their zones in earnest in order to allow DNSSEC-capable clients to protect themselves against different types of name-based attacks.

When zone owners sign their zones, enabling validation either within the application or through a trusted recursive name server mitigates the threats discussed above.  Moving DNSSEC validation into the application (in this case, the browser) provides three additional advantages:

1. It shifts the responsibility of determining the validity of DNS answers to the entity that will use the answers, so the DNSSEC last mile shrinks considerably.
2. It allows the application to provide the user with intuitive error messages, thus enabling the user to recognize that the problem lies with the zone in question and not an intermediary, such as an ISP.
3. It allows the application to make its own determination on what types of answers it is willing to trust. For instance, DANE-related lookups will require answers to be validated (and nothing short of it), while answers that have been shown to be provably insecure (but not bogus!) would be perfectly acceptable for normal browsing activity.

Moving validation into the application leads to the possibility that applications vendors will then want to expose the ability to enable and disable the DNSSEC validation function to the end-user. The point about “click-through insecurity” is well-made; however this is functionally no different than a user knowingly adding a security exception for certificates in current browser implementations. The browser vendor must make it hard for the user to add exceptions, without doubt, but the functionality to circumvent certain problems (such as ignoring the signature inception/expiration times on zones that remain published with expired signatures) is useful, especially while many zone operators are still fine-tuning their DNSSEC workflow.

It is also important, given that web pages are typically composed of a number of discrete elements, that validation be performed for all lookups initiated by the browser and not just for the name typed in the address bar. Many browser plugins for DNSSEC support will validate only the latter; while that capability is certainly useful, the real benefit of local validation is realized only when the browser (or the OS) completely integrates DNSSEC validation capability into its internal resolver library and enables validation for all queries.

The good news is that browser vendors (and their user communities) have been showing increased interest in seeing DNSSEC capability extended to the end-applications. Proof-of-concept implementations of browsers with DNSSEC validation support (e.g., the DNSSEC-Tools Firefox patch) have been available for a while, and with DNSSEC validation capability being continuously extended to new platforms and devices, there is hope that DNSSEC capability in browsers will eventually become more commonplace.

The 43rd ICANN meeting, to be held in  San José, Costa Rica from the 11th to the 16th of March, will include a DNSSEC Workshop on Wednesday the 14th.

Tentative schedule is for the meeting to run from 08:30 a.m. to 1:00 p.m. with an agenda that will cover:

• DNSSEC Deployment Around the World
• Challenges and Benefits of DNSSEC for Latin America and Regional Update
• DNSSEC Operational Best Practices and Lessons from the Trenches
• Using DNSSEC to Protect Reputations
• DNSSEC Signer Implementation Hardware

The final agenda, speakers, and room will be posted on the meeting’s web site in the next couple weeks.