It’s a wee ironic that a day after I tell a group of students that DNSSEC is very stable today compared to what it was like years ago (grandpa telling tales at the fireplace), I get a mail from the .CH registry:

My first thought was “they must be mistaken; I have only one dummy zone there”, and my second though was “wow, that’s pretty neat that they monitor customer zones!”. Still convinced it must be a case of mixed up identity, I click on the link which takes me to

OK, that’s definitely my zone; it’s the one I set up for testing DNSSEC provisioning automation with CDS/CDNSKEY .

I query the NS RRset for the domain and get a … SERVFAIL . What?!

Next up: DNSviz , of course, and there’s no doubt: a cryptographic signature on the RRSIG of the NS RRset does not compute:

I’m getting a bit nervous because the third day of training commences in an hour, but let me see what I see. Two of the four NS are under my control, so I force a transfer from the primary and notify all secondaries. The SOA serials match across the board. I contact the operator of the other two secondaries and request him to force transfer; no change – the kaputtness remains.

There’s no way for me to force re-signing of the zone, so I simply don’t know how to “fix” that one signature. I copy the zone file aside on primary and secondary, and then un-sign the primary zone, bump the serial, reload. This will now of course completely break validation as there still is a DS in the parent zone, but kaputt is kaputt and I can’t kaputt it further.

I bump the SOA serial again and sign the zone and see that the new zone is now on all servers.

Green . Relief.

post mortem

My training ends at 21:00 UTC, and I can’t switch off – I need to know what happened.

Using ldns-verify-zone on the kaputt zone, I see it too reports errors:

% ldns-verify-zone -V5 tcp53.ch.kaputt >/dev/null
Error: Bogus DNSSEC signature for tcp53.ch.	NS
RRSet:
tcp53.ch.	3600	IN	NS	ns1.dnspartner.de.
tcp53.ch.	3600	IN	NS	ns2.dnspartner.de.
tcp53.ch.	3600	IN	NS	lumpy.jpmens.net.
tcp53.ch.	3600	IN	NS	woozle.jpmens.net.
Signature:
tcp53.ch.	3600	IN	RRSIG	NS 13 2 3600 20221120102241 20221021101736 44444 tcp53.ch. TPfWAl4GffhEcyX50bZ4z43dtsrjL3dj/i+sSAMnJPXTuYmMCtQvLM8Hr/TbracpOPjymPrgvSQ+8wfBLeZgxw==

There were errors in the zone

I then load the zone into a fresh server, and see what delv has to say:

% delv +vtrace +root=tcp53.ch +multiline +trust +rrcomments +crypto +rtrace -d 99 -a tcp53.keys  @::1 tcp53.ch NS
...
;; validating tcp53.ch/NS: verify rdataset (keyid=44444): RRSIG failed to verify
;; validating tcp53.ch/NS: failed to verify rdataset
;; validating tcp53.ch/NS: verify failure: success
;; validating tcp53.ch/NS: no valid signature found
...

OK, I know of the failure, I was hoping for a few more details.

It occurred to me to use Perl Net::DNS to see if I could obtain more details. I asked for a bit of help, and Oli Schacher came to the rescue. First I verify that the current zone is OK so I use the current NS RRset and its RRSIG:

#!/usr/bin/env perlusestrict;useNet::DNS;useNet::DNS::SEC;my$dnskey="tcp53.ch. 3600 IN DNSKEY  256 3 13 egTvRrsMdaMjapWI4pC2M5dq6s0W6gpsLT4LwiwXvYs66CqPu+N+JgbO kLVIAwm8PGnPDEIDcAcHViYSvFbHpg==";# current one: verifiesmy$rrsig="tcp53.ch. 3600 IN RRSIG NS 13 2 3600 20221122031533 20221109133528 44444 tcp53.ch. xxW7tx5fIMUiIOIYrjfCq4h/T28rLlR6NSa0NOZC5NFalz/ShKPkpL3K KgsBKjTD0lleKUd5cqGCtyM4vIFm1Q==";# this one is kaputt# $rrsig = "tcp53.ch. 3600 IN RRSIG NS 13 2 3600 20221120102241 20221021101736 44444 tcp53.ch. TPfWAl4GffhEcyX50bZ4z43dtsrjL3dj/i+sSAMnJPXTuYmMCtQvLM8Hr/TbracpOPjymPrgvSQ+8wfBLeZgxw==";my@data=();push(@data,Net::DNS::RR->new("tcp53.ch.  3600 IN NS woozle.jpmens.net."));push(@data,Net::DNS::RR->new("tcp53.ch.  3600 IN NS ns2.dnspartner.de."));push(@data,Net::DNS::RR->new("tcp53.ch.  3600 IN NS lumpy.jpmens.net."));push(@data,Net::DNS::RR->new("tcp53.ch.  3600 IN NS ns1.dnspartner.de."));my$dnskeyrr=Net::DNS::RR->new($dnskey);my$nssig=Net::DNS::RR->new($rrsig);my$v=$nssig->verify([@data],[$dnskeyrr]);print"verifies\n"if$vordie$nssig->vrfyerrstr;

% ./sig.pl
verifies

I then run the program with the kaputt RRSIG, and voila, we’re on the right track:

% ./sig.pl
key 44444: signature verification failed at ./sig.pl line 28.

I say “on the right track”, but I’m not really – I’ve simply verified what the registry’s email informed me of that morning.

But why is this signature broken?

Oli checks the logs from their scanner tool that actually caused this email to see if they have more details from the Extended DNS Error, but it also just reports “(DNSSEC Bogus)”.

It’s now quite late, and Oli thinks the kaputt RRSIG is probably due to bit flip, cosmic rays, a bug in the signer, or the Pentium FDIV, and I am starting to think he’s right about that…

Time to sleep.

I rise, get coffee, and look through the repository which gets a copy of all zone activity upon notify .

The (now broken) signature was introduced on 2022-10-21. Then, on 2022-10-31T11:21:05, I replaced one of the nameservers, retiring “kanga” and introducing “woozle”, and due to cosmic whatever, the signature was not updated. (It must be a bug, but how on earth do I report that?!)

Can I prove that’s the reason? Yes, I can. In our Perl program, if I s/woozle/kanga/ in the NS RRset the (reportedly broken) RRSIG validates!

I deduce it’s a problem in the signer, but have no idea why it occurred. However, my biggest question is: why did this take so long to be noticed?

While getting the data for this blogticle, I note that DNSviz noticed the failure on 2022-10-31 15:13:10Z which was the time at which the zone went insecure because of the NS RRset change. (But that’s a story for another day.)

I was lucky that this is currently a toy zone.

And kudos to our friends at SWITCH for the excellent service; thank you!

Značky: #Network