Sam Pointer

Time and again I hear the following: “I’ve updated DNS with the new [A record, CNAME], it’ll take a couple of hours to propagate.”

No it won’t.

If you’ve previously tried to look up the non-existent name it’ll take as long as the minimum TTL value in the last field of your SOA record (see section #5 of RFC2308) for any caches to expire the NXDOMAIN response. If you’ve not then it will most likely be instant, depending on a few factors.

 

Image credit: xesc
 

I think this myth most likely comes from three places.

Firstly, most people don’t run their own DNS, they rely on a 3rd party provider. Most cheap providers do not use an authoritative nameserver that is database driven, such as PowerDNS or BIND with the DLZ (“dynamically loadable zones”) patches compiled in (I think this ships with modern 9.x versions). Instead, when you submit a form through their web UI you queue up some process behind the scenes that will produce a plain flat zonefile which will be reloaded on a schedule: usually every 15 or 30 minutes.

Secondly, back in the bad old days AXFR - a full zone transfer at the expiration of a timeout - was the only option for updating slave servers with the changes to a given zone. Depending on the value set in the SOA this could be quite some time. These days we have IXFR (“incremental zone transfer”), where the master server takes the initative to update all of its slaves when it detects a change, rather than waiting for them to time out and pull the changes. This delay, and the NXDOMAIN caching it would produce then slaves were queried before an AXFR would introduce a compound delay that certainly looks like propagation, but isn’t.

Thirdly propagation does need to take place in some instances. When you create a new domain you need to wait for the glue records to be circulated between the TLD authoritative servers, for instance. For most people for whom DNS is a once-a-year deal I suspect this idea of “a bunch of servers on the Internet need to sync up” gets a bit ingrained, doubly so if you’re mainly registering new domains more than you’re moving or mucking with existing ones beyond repointing the odd www. A record.

The O’Reilly “Cricket” DNS book, whilst nominally about BIND, is well worth the read for anyone who has more than a passing interest in DNS, or administers any operating system on a network. The first few chapters on DNS theory are worth the price of the book alone.
 

Edit: I’ve either proven my point or have not been clear by seeing most of reddit.com/r/sysadmin miss the point completely by discussing the expiration of positive caches, non-adherence to TTLs, etc. This is not what I am trying to discuss here.

What most people misunderstand is that when they add a new record to a zone (‘admin.nowhere.com’, for example - which has not existed before) they think that there is some propagation time. There is not, unless you’ve tried to look up that non-existent record before or your slaves are doing AXFRs with huge timeouts, or are distant in latency terms and doing IXFRs. When you register a new domain there is propagation time as the various TLD or cTLD root servers swap glue and NS records about. The mistake is thinking that the latter process has to occur for new records in an existing domain.

As for “propagation” meaning the gradual expiration of positive caches and the disemation of the information (an IP or MX change, for example), this isn’t propagation in the common DNS parlance. Propagation, in this context, is a “transmission” and is actively performed by the authoritative DNS server, not your local DNS resolver cache expiring.

I perhaps should have added a fourth reason: people confuse the cache TTL time-out of existing records with the lookup of a new record. There is nothing to expire with a new record, save the SOA value for NXDOMAIN as discussed above, and nothing to propagate.

I recently had cause to investigate a production problem related to the error message:

java.io.ioexception: insufficient system resources exist to complete the request

and noticed that through the course of my investigation there was quite a bit of cruft floating around this issue. In particular, if you enter the error message in your favourite generic internet search engine, you will most likely hit a bug logged with Sun in 2003 that offers nothing in the way of help in the slightest.

Before I start, let me try and get some of the background in first.

In our production setup we had a 64 bit Windows Server 2008 box running a long-lived Java-based daemon/service. This process is the man in the middle for an SFTP and bulk load process; all day it does something like this:

1. Poll UNC share advertised by Samba on a Solaris box for files uploaded via SFTP.
2. Determine that those files are not still being written to and some other unimportant (to this explanation!) validation.
3. Copy files to database server (another similar box)
4. Send message to database server that files have arrived and bulk upload into database can begin.

Of course, all of this had been tested in the various stages of our production release cycle and passed with flying colors. In production, it was now failing to write to the database server.

In light of that, if we now add UNC to our search query we get this Sun bug ticket, which is a bit more enlightening:

I have seen similar when a java application on Windows tries a "write" over a UNC path. 
When this is just a "write" there is an application limit of 64MB, "writes" cannot use Windows cache manager if the destination is a remote file systems. If you change this to a "read/write" it will work. The restriction of 64MB is not configurable and is set to prevent Applications starting by issuing a lot of large IO. This issue only happens for remote file systems, local systems will work. 

For local cases, MS can cache the data because they can enforce the “write-only” logic in the OS. They don’t allow a “write only” handle to read, so its OK if the cache manager caches the write. They can’t do this for remote files because the server can’t trust the client to enforce that logic.

Hmm. This seemed more promising to me, but the 64MB file size just didn’t gel with what we were seeing: the files it was failing to copy were well below 64MB, so even if the code had been terrible nasty heap-abusing things, failing because the files were over 64MB wasn’t one of them in this case.

After much digging about (this Microsoft support article about failing backups turned out to be the key to understanding this) we determined that the following was happening:

• java.nio was being used in the application code, and it wasn’t using any sort of chunking or buffering - it was just passing a whacking great multi-megabyte array of bytes to Windows.
• When a file is copied to a UNC share in this fashion (this doesn’t occur for local copies) the client machine will allocate a region of memory approximately the size of (never less than) the file to be copied from the paged kernel memory. You can see your current paged kernel memory value in the Windows task manager.
• The sized of the kernel paged memory buffer will vary depending on a number of factors, including but not limited to, the uptime of the machine, the amount of RAM in the machine, the usage and I/O profile of the machine over it’s uptime - I’m no Windows internals guru but let’s just say the thing can change ;)

What we discovered is that is the current size of the paged kernel memory, plus the size of the buffer that Windows is attempting to allocate out of it (which is itself derived from the size of the file you’re trying to copy) exceed 160MB then the "insufficient system resources" exception will be raised.

It turned out for us that all of our internal and production release machines had a suitably small paged kernel memory size and as such didn’t pop this seemingly arbitrary 160MB limit. Our production machines, much longer lived and more trafficked, did. The file in question was 47MB in size and our test machines had 22MB of paged kernel memory, whereas our production machines at 120MB. 47 + 22 was a flying pass in the tests, 47 + 120 popped the limit by 7MB and it was game over for files of that size or above.

There is a registry key that can be edited to raise the 160MB upper limit for the paged kernel memory buffer and therefore allow drivers to allocate more memory for buffers, but that isn’t really the way to fix problems such as these. In my case the solution was to explain all of the above to the developers and have them configure their java.nio class to properly "chunk" the copy of the file in suitably small pieces.