It’s a wee ironic that a day after I tell a group of students that DNSSEC is very stable today compared to what it was like years ago (grandpa telling tales at the fireplace), I get a mail from the .CH registry:

My first thought was “they must be mistaken; I have only one dummy zone there”, and my second though was “wow, that’s pretty neat that they monitor customer zones!”. Still convinced it must be a case of mixed up identity, I click on the link which takes me to

OK, that’s definitely my zone; it’s the one I set up for testing DNSSEC provisioning automation with CDS/CDNSKEY .

I query the NS RRset for the domain and get a … SERVFAIL . What?!

Next up: DNSviz , of course, and there’s no doubt: a cryptographic signature on the RRSIG of the NS RRset does not compute:

I’m getting a bit nervous because the third day of training commences in an hour, but let me see what I see. Two of the four NS are under my control, so I force a transfer from the primary and notify all secondaries. The SOA serials match across the board. I contact the operator of the other two secondaries and request him to force transfer; no change – the kaputtness remains.

There’s no way for me to force re-signing of the zone, so I simply don’t know how to “fix” that one signature. I copy the zone file aside on primary and secondary, and then un-sign the primary zone, bump the serial, reload. This will now of course completely break validation as there still is a DS in the parent zone, but kaputt is kaputt and I can’t kaputt it further.

I bump the SOA serial again and sign the zone and see that the new zone is now on all servers.

Green . Relief.

post mortem

My training ends at 21:00 UTC, and I can’t switch off – I need to know what happened.

Using ldns-verify-zone on the kaputt zone, I see it too reports errors:

% ldns-verify-zone -V5 tcp53.ch.kaputt >/dev/null
Error: Bogus DNSSEC signature for tcp53.ch.	NS
RRSet:
tcp53.ch.	3600	IN	NS	ns1.dnspartner.de.
tcp53.ch.	3600	IN	NS	ns2.dnspartner.de.
tcp53.ch.	3600	IN	NS	lumpy.jpmens.net.
tcp53.ch.	3600	IN	NS	woozle.jpmens.net.
Signature:
tcp53.ch.	3600	IN	RRSIG	NS 13 2 3600 20221120102241 20221021101736 44444 tcp53.ch. TPfWAl4GffhEcyX50bZ4z43dtsrjL3dj/i+sSAMnJPXTuYmMCtQvLM8Hr/TbracpOPjymPrgvSQ+8wfBLeZgxw==

There were errors in the zone

I then load the zone into a fresh server, and see what delv has to say:

% delv +vtrace +root=tcp53.ch +multiline +trust +rrcomments +crypto +rtrace -d 99 -a tcp53.keys  @::1 tcp53.ch NS
...
;; validating tcp53.ch/NS: verify rdataset (keyid=44444): RRSIG failed to verify
;; validating tcp53.ch/NS: failed to verify rdataset
;; validating tcp53.ch/NS: verify failure: success
;; validating tcp53.ch/NS: no valid signature found
...

OK, I know of the failure, I was hoping for a few more details.

It occurred to me to use Perl Net::DNS to see if I could obtain more details. I asked for a bit of help, and Oli Schacher came to the rescue. First I verify that the current zone is OK so I use the current NS RRset and its RRSIG:

#!/usr/bin/env perlusestrict;useNet::DNS;useNet::DNS::SEC;my$dnskey="tcp53.ch. 3600 IN DNSKEY  256 3 13 egTvRrsMdaMjapWI4pC2M5dq6s0W6gpsLT4LwiwXvYs66CqPu+N+JgbO kLVIAwm8PGnPDEIDcAcHViYSvFbHpg==";# current one: verifiesmy$rrsig="tcp53.ch. 3600 IN RRSIG NS 13 2 3600 20221122031533 20221109133528 44444 tcp53.ch. xxW7tx5fIMUiIOIYrjfCq4h/T28rLlR6NSa0NOZC5NFalz/ShKPkpL3K KgsBKjTD0lleKUd5cqGCtyM4vIFm1Q==";# this one is kaputt# $rrsig = "tcp53.ch. 3600 IN RRSIG NS 13 2 3600 20221120102241 20221021101736 44444 tcp53.ch. TPfWAl4GffhEcyX50bZ4z43dtsrjL3dj/i+sSAMnJPXTuYmMCtQvLM8Hr/TbracpOPjymPrgvSQ+8wfBLeZgxw==";my@data=();push(@data,Net::DNS::RR->new("tcp53.ch.  3600 IN NS woozle.jpmens.net."));push(@data,Net::DNS::RR->new("tcp53.ch.  3600 IN NS ns2.dnspartner.de."));push(@data,Net::DNS::RR->new("tcp53.ch.  3600 IN NS lumpy.jpmens.net."));push(@data,Net::DNS::RR->new("tcp53.ch.  3600 IN NS ns1.dnspartner.de."));my$dnskeyrr=Net::DNS::RR->new($dnskey);my$nssig=Net::DNS::RR->new($rrsig);my$v=$nssig->verify([@data],[$dnskeyrr]);print"verifies\n"if$vordie$nssig->vrfyerrstr;

% ./sig.pl
verifies

I then run the program with the kaputt RRSIG, and voila, we’re on the right track:

% ./sig.pl
key 44444: signature verification failed at ./sig.pl line 28.

I say “on the right track”, but I’m not really – I’ve simply verified what the registry’s email informed me of that morning.

But why is this signature broken?

Oli checks the logs from their scanner tool that actually caused this email to see if they have more details from the Extended DNS Error, but it also just reports “(DNSSEC Bogus)”.

It’s now quite late, and Oli thinks the kaputt RRSIG is probably due to bit flip, cosmic rays, a bug in the signer, or the Pentium FDIV, and I am starting to think he’s right about that…

Time to sleep.

I rise, get coffee, and look through the repository which gets a copy of all zone activity upon notify .

The (now broken) signature was introduced on 2022-10-21. Then, on 2022-10-31T11:21:05, I replaced one of the nameservers, retiring “kanga” and introducing “woozle”, and due to cosmic whatever, the signature was not updated. (It must be a bug, but how on earth do I report that?!)

Can I prove that’s the reason? Yes, I can. In our Perl program, if I s/woozle/kanga/ in the NS RRset the (reportedly broken) RRSIG validates!

I deduce it’s a problem in the signer, but have no idea why it occurred. However, my biggest question is: why did this take so long to be noticed?

While getting the data for this blogticle, I note that DNSviz noticed the failure on 2022-10-31 15:13:10Z which was the time at which the zone went insecure because of the NS RRset change. (But that’s a story for another day.)

I was lucky that this is currently a toy zone.

And kudos to our friends at SWITCH for the excellent service; thank you!

Značky: #Network

There are two adaptations of Johannes Mario Simmel’s “Es muss nicht immer Kaviar sein” as TV series: the first with O.W. Fischer in 1961 , and the second with Siegfried Rauch in 1977 . I think I saw the second, but I digress: the name of the book occurred to me when thinking about this blogticle.

NLnet Labs make DNS software (and more), and amongst their best-known utilities are probably the recursive Unbound server and the authoritative NSD server, one which was originally written to drive a root DNS server.

I’m occasionally asked whether NSD can be used to serve a DNSSEC-signed zone (the answer is ‘yes’), but it isn’t a signer; it requires other utilities to actually sign zones which it can then serve. I’ve occasionally mentioned LDNS , particularly when I wrote about using a SmartCard-HSM for DNSSEC and again when I wrote about DNSSEC signing with an offline KSK .

But let’s look specifically at the task of signing one or more zones and serving them with NLnet Labs programs.

signing

We start with an unsigned zone file, here for the domain example.aa ( aa is a user-assigned code element .)

$TTL 3600
@	SOA	nsd  jp ( 1 3H 1H 1W 1H )
	NS      nsd
nsd	A	192.0.2.43
	AAAA	2001:0db8:0:0:0202:b3ff:fe1a:8329

I’d like to sign this zone with a single Combined Signing Key (CSK), also called a Single Signing Key (SSK), and I chose algorithm ECDSA Curve P-256 with SHA-256 (number 13) for the key, and I specify the key will have flags 257 on it (KSK), though that is not strictly necessary:

$ldns-keygen -a13-k example.aa
Kexample.aa.+013+22967

$ls-l K*-rw-r--r--  1 jpm  staff   94 Nov  6 13:16 Kexample.aa.+013+22967.ds
-rw-r--r--  1 jpm  staff  153 Nov  6 13:16 Kexample.aa.+013+22967.key
-rw-------  1 jpm  staff  114 Nov  6 13:16 Kexample.aa.+013+22967.private

The key filenames begin with a capital K (for “key”), are followed by the zone name (“example.aa”, the key algorithm (13), the key ID or key tag, and the file extension:

• the file containing the Delegation Signer record(s)
• the public key (later queryable via the DNSKEY resource record)
• the private key

I keep the private key safely (ldns has removed group and other permissions, and I tend to chmod 400 the file to ensure I don’t overwrite it accidentally).

The *.ds file contains the Delegation Signer (DS) record which must be added to the parent zone. I can reproduce this file at will with a different ldns utility, and with the -n option I decide whether I want the DS printed to stdout or not:

$ldns-key2ds -2 Kexample.aa.+013+22967.key
Kexample.aa.+013+22967
$cat Kexample.aa.+013+22967.ds
example.aa.	3600	IN	DS	22967 13 2 cfb29a4218cf608d93c08c0d14c3e315b9ab85797bf20269b820ed3a2c1d8f47

$ldns-key2ds -n-2 Kexample.aa.+013+22967.key
example.aa.	3600	IN	DS	22967 13 2 cfb29a4218cf608d93c08c0d14c3e315b9ab85797bf20269b820ed3a2c1d8f47

Having generated the required key(s), I can now sign the zone. I specify -u to have the SOA serial number set to the Unix epoch time, the -o origin (i.e. zone name) of the zone I’m signing, and the key filename of the key ldns should use to sign the zone (without its file extension).

$ldns-signzone -u\-o example.aa \
		example.aa \
		Kexample.aa.+013+22967
$ls-l*.signed
-rw-r--r--  1 jpm  staff  1762 Nov  6 13:22 example.aa.signed

$ldns-read-zone -E SOA example.aa.signed
example.aa.	3600	IN	SOA	nsd.example.aa. jp.example.aa. 1667737361 10800 3600 604800 3600

The ldns-read-zone utility is very practical. I can use it to strip DNSSEC data from a zone file – something I do when I want to concentrate on the actual zone data. Here I’ve used it to print out just one record type.

Finally, to ensure the signed zone is valid, I can use another utility to verify that, though I prefer using something written by a different developers to ensure they agree that the zone is valid.

$ldns-verify-zone -V4 example.aa.signed
Zone is verified and complete

$validns -p all example.aa.signed
example.aa.signed:3: there should be at least two NS records per name
example.aa.signed:5: No KSK found

validns is a bit of an older utility but still quite useful. Here it complains that my zone has only one NS record (correct, it does), and that no KSK was found. This latter has to do with validns not realizing that a zone can be signed with a single key. I ignore this warning because I know the zone is otherwise valid.

serving

I configure NSD to load my zone from the *.signed file and serve it:

zone:
    name: "example.aa"
    zonefile: "example.aa.signed"
    provide-xfr: 127.0.0.1/8 NOKEY

After reloading the server I query it and see the signatures (I use +nocrypto to omit the dreary base64-encoded stuff):

$dig @::1 example.aa SOA +dnssec +multi +nocrypto +norec
;; Got answer:
;; ->>HEADER<<-opcode: QUERY, status: NOERROR, id: 65367
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
;EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;example.aa.		IN SOA
;; ANSWER SECTION:
example.aa.		3600 IN	SOA nsd.example.aa. jp.example.aa. (
				1667737361 ;serial
				10800      ;refresh (3 hours)				3600       ;retry (1 hour)				604800     ;expire (1 week)				3600       ;minimum (1 hour)				)
example.aa.		3600 IN	RRSIG SOA 13 2 3600 (
				20221204122241 20221106122241 22967 example.aa.
				[omitted] )

;; Query time: 0 msec
;; SERVER: ::1#53(::1);; WHEN: Sun Nov 06 12:37:16 UTC 2022
;; MSG SIZE  rcvd: 188

Signatures (RRSIG) have a default validity of 30 days, but I can easily change that. I’ve created eitime which prints a time stamp in UTC YYYYMMDDHHMMSS for RRSIG expiration/inception times and makes my life easier:

$ldns-signzone -e$(eitime -z 180) ...

Don’t forget to re-sign and reload the zone in a timely manner before its signatures expire. Cron is your friend.

parent

There’s one thing missing in order to complete the chain of trust to my zone: submit the DS record to the parent, and all I can do here, unfortunately, is to say “that’s your problem”. The method by which you do this varies widely.

Regarding caviar: signing with ldns and serving with NSD might well be very appealing to you, and I know people for whom this is true.

Značky: #Network

Ever since thinking “that’s quite nifty”, I have the devil of a time spelling ntfy without transposing the ‘t’ and the ‘f’, but it’s called notify so the ‘t’ comes before the ‘f’. :-)

With ntfy I can send push notifications to a phone or desktop via a simple HTTP-based pub/sub service. Philipp C. Heckel created ntfy and open-sourced its components.

ntfy is the backend server (written in Golang), Android and iOS apps, and a desktop UI which runs in a Web browser. There’s also a command-line client.

Messages are published via either HTTP PUT or POST, and something as simple as this suffices to transmit a message. Note that there’s no authentication so use your own server or chose a topic which is hard to guess.

$curl -d"Hola mundo" ntfy.sh/my-topic

Subscribing via the API is equally easy:

$curl -s ntfy.sh/admin-alerts/json
{"id":"AQJQ0ThDMuda","time":1667123190,"event":"open","topic":"admin-alerts"}
{"id":"7osug0iTf31n","time":1667123207,"event":"message","topic":"admin-alerts","message":"deployment on alice is complete. 🐄","priority":1,"tags":["heavy_check_mark"]}
{"id":"by9EicvNYMsp","time":1667123235,"event":"keepalive","topic":"admin-alerts"}

All manner of features can be added to a message: priorities, tags, title, attachments, and messages can even contain an action to send a HTTP request from the client when an action button is tapped.

Ansible

I was thinking about Ansible action plugins the other day and decided to implement one which notifies via ntfy . Why an action plugin and not a module? Because I’d expect notification to be emitted from the Ansible controller (also possible with a module delegated to it) so that’s how I did it. (Let me know if you prefer a module; we could do both.)

-ntfy:msg:"deploymenton{{inventory_hostname}}iscomplete.🐄"

ansible-ntfy notifies ntfy.sh by default with a topic configured in a play var. Topic, URL, and additional publishing features can be configured in a dict passed as attrs :

-ntfy:topic:"admin-alerts"url:"https://nfty.sh"msg:"that'sawrap"attrs:tags:[rotating_light,heavy_check_mark]priority:4actions:-action:viewlabel:"OpenMastodon"url:"https://mastodon.social/@jpmens"

Aside from this Ansible module, there are lots of existing integrations .

ntfy is nifty. :-)

Značky: #Network

Here I am again with Ansible ’s local facts , but this time on Windows nodes, where the local facts behave differently.

Local fact files on Unix/Linux must be named *.fact irrespective of whether they contain INI or JSON or whether they are executable programs which emit JSON, and they are searched for in a default fact_path which is typically /etc/ansible/facts.d (or /usr/local/etc/ansible/facts.d on BSD systems).

There’s no default fact directory for Windows nodes, but we specify one with the fact_path parameter to the setup module or in an ansible.cfg file:

[defaults]nocows=1fact_path=c:/users/r1/facts

That directory must exist and it MUST contain PowerShell ( *.ps1 ) files which output objects which will be formatted as JSON by Ansible . Here’s an example containing a static object:

@{username='janej'lat=48.856826lon=2.292713}

When Ansible’s setup module runs, we obtain the following fact from the Windows node:

"ansible_os_family":"Windows","ansible_osm":{"lat":48.856826,"lon":2.292713,"username":"janej"}

Note that Ansible places these objects alongside other Ansible facts, whereas on Unix, if I have a JSON file /etc/ansible/facts.d/osm.fact , Ansible returns osm nested under ansible_local :

"ansible_os_family":"Darwin","ansible_local":{"osm":{"lat":48.856826,"lon":2.292713,"username":"janej"}}

dynamic

On Unix/Linux *.fact files may be executable and return JSON, so I install my program as /etc/ansible/facts.d/hungry.fact

$/etc/ansible/facts.d/hungry.fact
{"id":4,"meal":"dinner","dish":"tikka masala"}

The utility is executed during setup and the result is:

"ansible_local":{"hungry":{"dish":"tikka masala","id":4,"meal":"dinner"},"osm":{"lat":48.856826,"lon":2.292713,"username":"janej"}}

On Windows nodes fact files MUST be *.ps1 so I figure out^W^W google how to invoke a program from a PowerShell script and produce this masterpiece in a file called currybeer.ps1 :

Invoke-Expression"& `"C:\Users\r1\facts\hungry.exe`"  /run /exit /SilentMode"

Sadly, that doesn’t do what I thought it would: the PowerShell script is returning a string:

"ansible_currybeer":"{\"id\":4,\"meal\":\"dinner\",\"dish\":\"tikka masala\"}"

A bit more figuring out later, I add | ConvertFrom-Json to the .ps1 script, and Ansible correctly interprets its output:

"ansible_currybeer":{"dish":"tikka masala","id":4,"meal":"dinner"},"ansible_osm":{"lat":48.856826,"lon":2.292713,"username":"janej"},

I can now use these facts like any others:

-hosts:wingather_facts:truetasks:-debug:msg="{{ ansible_currybeer.dish }} for {{ ansible_currybeer.meal }}"

and look forward to this evening:

TASK [debug] *************************
ok: [192.168.1.163] => {
    "msg": "tikka masala for dinner"
}

I don’t know why Ansible nests local facts differently on Windows nodes, and I don’t see an advantage in doing so.

And if you need inspiration, we’re collecting ideas for using local facts

Značky: #Network

I was asked today whether I’ve a list of ideas for using Ansible ’s local facts , and my answer was, sadly, ‘no’. I thought I’d start one with the help of my Ansible followers.

• I often say data from special sensors can be reported on by an executable fact file.
• Andreas reports they use executable facts to report on nginx vhosts, MySQL databases, Docker containers, installed software such as PHP and Node versions, and last patch date of the node. They use this data with ansible-cmdb , and they MOTD (message of the day) files with Ansible.
• Nicolai uses executable fact files to report on custom hardware information for their RevPi HAT EEPROM and detected IO modules for automatic configuration templates.
• An idea I once heard was to use fact files to “enable” and/or “disable” nodes or services on the nodes. These were hot-standby nodes which used fact files to report they were available (or not).

Do let me know, please, how you use local facts (files or executables); I’m really interested.

Značky: #Network

When calling Ansible modules frequently with similar parameters, module_defaults can save on typing and, almost more importantly, improve on clarity by defining default values for modules I use in a play.

Let’s look at a small example in which I want to clear records for a host from a dynamic DNS server and then add it in again. In all invocations of the nsupdate module I would need to specify key name, algorithm, server, etc., but through the use of module_defaults I can set default values and no longer have to specify these repetitively on individual tasks.

-hosts:allvars_files:dns-params.ymlmodule_defaults:community.general.nsupdate:key_algorithm:"hmac-sha256"key_name:"{{tsig_key_name}}"key_secret:"{{tsig_key_secret}}"server:"{{dns_server}}"protocol:"tcp"ttl:60zone:"example.org"tasks:-name:Clean old records from DNScommunity.general.nsupdate:record:"www.example.org."state:absent-name:Add IPv4 addresses to DNScommunity.general.nsupdate:record:"www.example.org."type:"A"value:"192.0.2.43"state:present

What I wasn’t aware of and learned today via Ton , is that the module defaults can also be provided via a lookup plugin. I whipped up a simple example to illustrate this.

The lookup plugin ( md.py ) returns a dictionary of values. It doesn’t use terms passed to it but easily could, e.g. to have the plugin return classes of values, etc.

fromansible.errorsimportAnsibleError,AnsibleParserErrorfromansible.plugins.lookupimportLookupBaseclassLookupModule(LookupBase):defrun(self,terms,variables=None,**kwargs):data={"content":"Rijsttafel","mode":"0400",}ret=[data]returnret

The playbook shows how this plugin is used to feed module_defaults for the copy module, and it then invokes copy twice:

-hosts:localhostmodule_defaults:copy:"{{lookup('md')}}"tasks:-copy:content="Hello" dest="a" mode=0444-copy:dest="b"

The result: two files with distinct permissions and content:

$ls-l ?
-r--r--r--  1 jpm  staff   5 Oct  5 18:32 a
-r--------  1 jpm  staff  10 Oct  5 18:32 b

$cat a
Hello

$cat b
Rijsttafel

The second file b has permissions 0400 and content “Rijsttafel” set from the lookup, and both have these defaults overridden for file a .

Značky: #Network

I pretty much stumbled over sematicon and their N200 Series on my quest for Hardware Security Modules (HSM) of almost any shape and size, and after a very pleasant phone call with one of their managers, I was quickly given documentation to study and access to a dedicated device over a Wireguard VPN.

Every appliance features two or more independently configurable crypto processors, which work with a variety of keys. Therefore, test and production can be operated separately of each other on only one platform.

Although the N200 series isn’t specifically tailored to PKCS#11, it’s what I’m mainly interested in so what I focus on. I pretty much ignored the actual PKI capabilities of the device. It can be a certification authority (CA), produce X.509 certificates, perform symmetric encryption, and a load of other features. I toyed with creating a user and assigning access rules. The Web-based UI is neat and functional. There were some labels and captions which appeared confusing to me, but I got used to them.

PKCS#11 support is relatively new, and the documentation doesn’t yet mention about it, but I’m told that will change for the upcoming new version. A small interim PDF gave some tips, but I had a bit trouble actually using the interface because after initializing the device, I wasn’t aware that I had to create an access rule for a user who wants to use the PKCS#11 interface without which I kept getting an access denied. A brief meeting with the person who’s blog article initially led me to sematicon got me set up.

I copy the PKCS#11 sematicon-provided shared object library to the directory I want it in and configure the n200.conf file, adding a high log level to begin with, but it turns out the level is ignored: setting it to 0 doesn’t change much. Beware the log: it will contain the user and password in clear, but I’ve reported this.

% echo "10.195.10.11 seSAM-nserie-101" >> /etc/hosts    # avoid TLS verification errors on hostname                                                        #the appliance has a cert with subject CN= this name

% install -m 555 pkcs11-sesam-x64-3.so /usr/local/lib

% export p=/usr/local/lib/pkcs11-sesam-x64-3.so

% cat $N200_CONF 
{
   "url": "https://10.195.10.11/n200/web/postv1",
   "tlscert": "/etc/tls/server63.crt",
   "user": "pkmaster",
   "pass": "<password>",
   "core": 1,
   "keygen_pin": "12345678",
   "log_level": 10,
   "log_output": "/tmp/seSAM_p11.log"
}

% pkcs11-tool --module $p --show-infoCryptoki version 3.0
Manufacturer     sematiconAG
Library          se.SAM N200 pkcs11 (ver 1.0)
Using slot 0 with a present token (0xc13cc61)

The pkcs11-sesam.so speaks to the N200 via their REST API using curl , and judging by the number of CURL_GLOBAL_INIT occurrences, there is plenty of opportunity to improve performance of the shared library. (I will gloss over performance, because the network connection I had to the HSM was so flaky as to not be funny.)

I’ve already mentioned the N200 has a REST API which I want to test. After learning how to do so in the Web UI, I use the API to generate a key-pair on the device:

% curl \
        --cacert server63.crt \
        -u $user:$pass\--no-progress-meter\-dcommand=genecckey \-dcore=1 \-dcurve=NIST \-dkeysize=256 \-dacl=FFFF \-dpin=12345678 \-dnice_name=JP.example \
        https://seSAM-nserie-101/n200/web/postv1
{       
  "command": "genecckey",
  "core": 1,
  "clustersync": "n/a",
  "nice_name": "JP.example",
  "publickey": "FACD4CF599AD0CBD43A5C38B294AF3DDC06FBC0883D419EEBCBF8039C132FC61FE972236B38A1C236E5EF0F1E78B64B31C9632C0D5B9CFA49AECDB89144A2C3C",
  "result": "hsm010123F13C56131E1EEE0000000000000025"
}

% pkcs11-tool --module $p --token-label hsm010123F13C56131E1EEE00000025 --list-objectsPublic Key Object;EC  EC_POINT 256 bits
  EC_POINT:   044104facd4cf599ad0cbd43a5c38b294af3ddc06fbc0883d419eebcbf8039c132fc61fe972236b38a1c236e5ef0f1e78b64b31c9632c0d5b9cfa49aecdb89144a2c3c
  EC_PARAMS:  06082a8648ce3d030107
  label:      JP.example
  ID:         aaab68606be2
  Usage:      encrypt, verify
  Access:     local
Private Key Object;EC
  label:      JP.example
  ID:         aaab68606be2
  Usage:      decrypt, sign, non-repudiation
  Access:     sensitive, always sensitive, never extractable, local
  Allowed mechanisms: ECDSA,ECDSA-SHA1,ECDSA-SHA224,ECDSA-SHA256,ECDSA-SHA384,ECDSA-SHA512

The result from the key generation has an identifier (also visible in the UI) which sadly has eight more zeroes in it than the token label in the slot; I don’t know if that’s on purpose. (A limit of 32 characters in a token slot name maybe?) I say sadly because it requires an extra step ( pkcs11-tool --module $p -T ) to find the correct token label.

It’s time to sign our zone with our Combined Signing Key or Single Signing Key, both euphemisms for “I’m too poor to afford a second key pair”.

% cat $OPENSSL_CONF
openssl_conf = openssl_init

[openssl_init]
engines=engine_section

[engine_section]
pkcs11 = pkcs11_section

[pkcs11_section]
engine_id = pkcs11
dynamic_path = /usr/lib64/engines-3/libpkcs11.so
MODULE_PATH = /usr/local/lib/pkcs11-sesam-x64-3.so
init = 1

After configuring OpenSSL’s engine to use our pkcs11-sesam.so module, I generate the pair of key files referencing the key material in the HSM. Use of the PIN value here depends on the setting in the HSM proper: the N200 can store the PIN on the device in order to not require it on the client, IMO six of one and half-a-dozen of the other in terms of security.

% cat jp.example 
$TTL 3600
@ SOA  ns  jpmens  1 3H 1H 1W 1H
  NS   @
  A    192.0.2.43

% dnssec-keyfromlabel -E pkcs11 -a13 -l "token=hsm010123F13C56131E1EEE00000025;object=JP.example;pin-value=12345678" -fKSK jp.example 
Kjp.example.+013+29467

% dnssec-signzone -E pkcs11 -t -z -S -o jp.example jp.example 
Fetching jp.example/ECDSAP256SHA256/29467 (KSK) from key repository.
Verifying the zone using the following algorithms:
- ECDSAP256SHA256
Zone fully signed:
Algorithm: ECDSAP256SHA256: KSKs: 1 active, 0 stand-by, 0 revoked
                            ZSKs: 0 active, 0 stand-by, 0 revoked
jp.example.signed
Signatures generated:                        5
Signatures retained:                         0
Signatures dropped:                          0
Signatures successfully verified:            0
Signatures unsuccessfully verified:          0
Signing time in seconds:                 5.302
Signatures per second:                   0.942
Runtime in seconds:                      5.984

I wrote earlier that I would gloss over performance, so I will. I’ve no idea where the device is (network-wise) so I’m taking these numbers with spoon-fulls of salt. (To be fair: I wouldn’t expect miracles if the device were close to me because of the seemingly large amount of HTTP requests I see in the log; recall that each DNS and NSEC{3} record needs to be signed and causes a HTTP request.)

I see in the Web UI that I can import a key to the device and try a PKCS#8-wrapped key, but that doesn’t work for me, but the API should work, and it does:

#!/usr/bin/env python3
importrequestsimportjsonurl="https://seSAM-nserie-101/n200/web/postv1"data={"command":"importecckey","core":1,"curve":"NIST","acl":"FFFF","pin":"12345678","nice_name":"example-dot-com-EC","key_to_import":"443A5A0F630559075B8ED453E69FB134E73BED96B7E7695D55AFBBFC085278CD"}r=requests.post(url,data,verify='server63.crt')print("status: %s, %s"%(r.status_code,json.dumps(json.loads(r.text),indent=4)))

The RAW key data for key_to_import I obtained from the ECDSA key I want to import:

% dnssec-keygen -a13 -fksk example.com
Generating key pair.
Kexample.com.+013+37354

% awk '/PrivateKey:/ { print $2 }' Kexample.com.+013+37354.private | base64 -d | ./hexlify
443A5A0F630559075B8ED453E69FB134E73BED96B7E7695D55AFBBFC085278CD

% ./jpimport.py
status: 200, {
    "command": "importecckey",
    "core": 1,
    "result": "hsm010123F13C56131E1EEE0000000000000018"
}

There’s a so-called “TCP-RAW-API” which allows direct access to the crypto processor as well as an HTTP GET API which uses the URL to transport a request to the appliance; the response to this is JSON. Here are a few examples including symmetric encryption and decryption. (I’ve not yet tried the factoryreset …) In the following examples I omit specifying the certificate verification option in the interest of brevity.

% curl https://seSAM-nserie-101/n200/timestamp/  
{"command": "timestamp", "result": "1664657599"}

% curl https://seSAM-nserie-101/n200/getrandom/5/
{"command": "getrandom", "numbers": "5", "core": 1, "result": "EBFB604875"}

% curl https://seSAM-nserie-101/n200/getkeyattributes/k01/
{
  "command": "getkeyattributes",
  "result": {
    "hsm010123F13C56131E1EEE0000000000000026": {
      "key_id": "hsm010123F13C56131E1EEE0000000000000026",
      "key_type": "SYM",
      "key_len": 256,
      "key_acl": "FFFF",
      "key_owner": "pkmaster",
      "nice_name": "k01"
    }
  }
}

% curl https://seSAM-nserie-101/n200/encrypt/k01/12345678/CBC/hello-world/
{
  "command": "encrypt",
  "core": 1,
  "result": "A84663E043099BC74C11628DAFB69B6D8FAABCFBF4FC4AF33657B719C96617B772D229AB8A9A0E7B3903C6E404121980"
}

% curl https://seSAM-nserie-101/n200/decrypt/k01/12345678/CBC/A84663E043099BC74C11628DAFB69B6D8FAABCFBF4FC4AF33657B719C96617B772D229AB8A9A0E7B3903C6E404121980/
{
  "command": "decrypt",
  "core": 1,
  "result": "hello-world"
}

% curl https://seSAM-nserie-101/n200/listkeys/NIST/
{
  "command": "listkeys",
  "core": 0,
  "result": {
    ...
    "hsm010123F13C56131E1EEE0000000000000025": {
      "slot": "hsm010123F13C56131E1EEE00000025",
      "key_id": "hsm010123F13C56131E1EEE0000000000000025",
      "id": "AAAB68606BE2",
      "key_type": "NIST",
      "key_len": 256,
      "key_acl": "FFFF",
      "nice_name": "JP.example",
      "date_of_creation": "2022/10/01 20:15",
      "date_of_last_use": "2022/10/01 20:32",
      "date_planned_lifetime_until": null,
      "token_name": "hsm010123F13C56131E1EEE00000025",
      "key_owner": "pkmaster",
      "public_key": "FACD4CF599AD0CBD43A5C38B294AF3DDC06FBC0883D419EEBCBF8039C132FC61FE972236B38A1C236E5EF0F1E78B64B31C9632C0D5B9CFA49AECDB89144A2C3C"
    }
  }
}

The sematicon N200 is a network-attached HSM and is positioned as an IoT-specialized device, and I felt comfortable using it after a bit of study. The documentation is good, and the chapter on APIs describes each call in detail for the three access methods (RAW, GET, POST).

My specific use-case (signing DNS records via a PKCS#11 interface) is not a forte of the appliance, but I was told that from the outset. In spite of this, I was positively surprised that all the PKCS#11 functions required by the utilities I used were supported.

As an all-purpose HSM this appears to be a very interesting device at a price far below what the market “leaders” demand for a networked appliance.

Značky: #Network

The level of complexity caused by KSK/ZSK split in DNSSEC isn’t necessary for most purposes; you can in fact just have one key for a zone and sign everything with it such. co.uk. does this , and I also make frequent use of Combined Signing Key (or Single Signing Key).

However, the KSK/ZSK split has one advantage: it allows us to keep the KSK offline and only bring it out occasionally to sign the DNSKEY RRset. This key can be kept stable (i.e. it need to be changed frequently or at all) and thus the interaction with the parent (DS submission) be kept at a minimum.

Jaromir Talir held a presentation about offline KSK with Knot DNS , and from the terminology he used in 2019 I made this diagram a few years ago to help me better understand the flow of keys.

The idea is that the KSK (depicted on the left) is managed completely offline. Only public keys are carried around – on floppy disks or USB keys, say, and as such there is an extremely low risk of the KSK being compromised.

The ZSK and the zone proper are maintained on the right, online and/or on a hidden primary. Since the ZSK can easily be replaced there is little risk of compromise, and the zone data is public anyway.

Once created, the ZSK (or ZSKs – plural) are copied to the KSK machine for signing. The KSK signer uses the KSK to sign the DNSKEY RRset and the resulting RRset and its signatures (RRSIG) are transported back to the zone signer on the right. This data is now public: the DNSKEY RRset contains the public keys for the zone.

The zone is now signed (on the right) with the ZSK only, so this signs all of the RRsets in the zone, and as a last step we enrich the newly-signed zone with the public DNSKEY of the KSK and its signatures, and this zone (named example.com.final in the diagram) we then load and serve.

The diagram shows “key requests” being passed left and right. These could be signed (PGP?) containers carrying public key material, but for our purposes simple files containing the public DNSKEY records suffice.

In the example which follows, the offline KSK will be managed with tools from the ldns toolchain, and the zone signer with tools from BIND and optionally from ldns . Why? Just to introduce diversity.

creating the keys

We begin by preparing our environment. Everything in the directory left contains files which we will have on the offline KSK machine; conversely, the directory right contains files on our actual zone signer.

% mkdir left right
% touch left/KSK.data                       #this file will contain public data

% cd right/
right% dnssec-keygen -a13 example.com       #generate ZSK
Generating key pair.
Kexample.com.+013+54147

right% ln -s Kexample.com.+013+54147.key ZSK.key

We copy the public ZSK.key to the left . In the following script, I have to replace $INCLUDE.*$ by the actual public ZSK DNSKEY RR because ldns-read-zone and friends don’t understand $INCLUDE .

% cd left/
left% ldns-keygen -a13 -k example.com       #generate KSK
Kexample.com.+013+11238

left% cat example.com
$TTL 3600
@ SOA  unused.invalid. jp.invalid.  1 3H 1H 1W 1H
  NS   unused.invalid.

;$INCLUDE"../zone/ZSK.key";This is a ZONE-SIGNING key, keyid 54147, for example.com.
example.com. IN DNSKEY 256 3 13 ah+p9L1ydOMAWsHTW9sLokRDrA1by6TV6ZPLKPDhkowEMAN7o36Vs1Bt db+obk6h22D548XWTWV6cWEkbNYHPQ==

left% cat Kexample.com.+013+11238.ds
example.com.	IN	DS	11238 13 2 2d0d834fe6aea4c80ec3bcbf5af753653c9d9ea6dc997ea86e90c2615edef19f

Note the DS record which we will later submit to the parent. This is public data, so we can safely copy it to the right for further submission.

On the left, the content of the zone doesn’t matter; I’ve used invalid. to underline the fact. We will now sign that zone with the KSK, and I set the signature validity to a date I’ll recognize in the final zone. The only records from this signed zone we are interested in are the DNSKEY and RRSIG over DNSKEY.

left% ldns-signzone -e 20221224 -o example.com example.com Kexample.com.+013+11238

#now grab the KSK DNSKEY record and its RRSIG
left% ldns-read-zone example.com.signed |
	awk '($4 =="DNSKEY"&&$5== 257)||($4=="RRSIG"&&$5=="DNSKEY"){ print }' | tee KSK.data
example.com.	3600	IN	DNSKEY	257 3 13 cMW3t5/jWcESqrzNgNjksnfUZdS4TtqO1gOae0cBRxgopoT/FNIa5GATNQAg64TJw3tgECVOq1beXpbJGSjG7Q== ;{id = 11238 (ksk), size = 256b}
example.com.	3600	IN	RRSIG	DNSKEY 13 2 3600 20221224000000 20220922073938 11238 example.com. 35J+toX0FyIyXRTNpamM6UVyivLFV1deUSWrJl+3DCVH2feolbDP45i/myLDnnywAS4NzQIZqjdkZQMTyVSxfg==

left% wc -l KSK.data
       2 KSK.data

The file KSK.data contains the public key and signature which we transfer (floppy disk, remember?) back to the right .

signing the zone

Back on the right we have the zone, the ZSK, and the public KSK in the file KSK.data , so we can now sign this:

right% ./bsig.sh
*** sign the zone;set SOA serial to epoch
example.com.signed
*** remove dsset file as it contains ZSK only
*** replace RRSIG over DNSKEY created by ZSK with that created by KSK
*** verify signed zone using ldns-verify-zone
Zone is verified and complete
*** verify signed zone using dnssec-verify
Loading zone 'example.com' from file 'example.com.signed'
Verifying the zone using the following algorithms: ECDSAP256SHA256.
Zone fully signed:
Algorithm: ECDSAP256SHA256: KSKs: 1 active, 0 stand-by, 0 revoked
                            ZSKs: 1 active, 0 stand-by, 0 revoked
*** verify signed zone using validns
records found:       17
skipped dups:        0
record sets found:   10
unique names found:  3
delegations found:   0
    nsec3 records:   0
not authoritative names, not counting delegation points:
                     0
validation errors:   0
signatures verified: 8
time taken:          0.003s

The b in bsig.sh is for signing with the BIND utilities, and there’s an lsig.sh which uses ldns for signing.

Both scripts work in a similar way: they first sign the zone using only our ZSK (BIND’s smart signing finds the key by itself, with ldns I have to specify it). Then they remove the RRSIG over DNSKEY which was created by the ZSK (we don’t need that; our DNSKEY RRset is signed by the KSK), and add the public DNSKEY of the KSK and its RRSIG, both obtained from left . Finally, we use three distinct tools to verify the validity of the zone.

The signed zone file is the one we (re-)load on our (hidden) primary server.

finally

As the diagram above suggests, there’s no reason why you couldn’t add complexity security by adding a bunch of HSMs here and there.

And if you don’t like using floppy disks to transport the public keys from right to left and back, but want to make life difficult for intruders, a serial cable and the likes of UUCP or Kermit might be employed. :-)

Značky: #Network

I had to abort an exercise I gave students last night during a BIND training, because after adding

allow-new-zones yes;

and reconfiguring a running BIND 9.16 server, the process complained it couldn’t open _default.nzd . (If compiled with LMDB support, BIND uses the memory-mapped database for storing new zone definitions instead of the “old” .nzf file.)

We run these training systems with SELinux enabled (I wouldn’t, but my colleague likes it :-), and that’s the reason I aborted the lab: I couldn’t tell students how to solve the cause other than by disabling SELinux entirely, but there wasn’t enough time for that.

After getting a good pointer on Twitter (thank you Howard, Stef, and Evgeni), I saw this:

#!!!! This avc can be allowed using the boolean 'domain_can_mmap_files'
allow named_t named_zone_t:file map;

The comment contained “boolean”, which I recognized, so I was able to fix the issue an hour after the training. The command

setsebool -P domain_can_mmap_files 1

has now found its way into the setup playbooks for these machines.

Značky: #Network