close
  • Pl chevron_right

    Returning to DebConf

    pubsub.slavino.sk / planetdebian · Tuesday, 31 August - 01:29 · 1 minute

I first started using Debian sometime in the mid 90s and started contributing as a developer and package maintainer more than two decades years ago. My first very first scholarly publication, collaborative work led by Martin Michlmayr that I did when I was still an undergrad at Hampshire College , was about quality and the reliance on individuals in Debian. To this day, many of my closest friends are people I first met through Debian. I met many of them at Debian’s annual conference DebConf .

Given my strong connections to Debian, I find it somewhat surprising that although all of my academic research has focused on peer production , free culture, and free software, I haven’t actually published any Debian related research since that first paper with Martin in 2003!

So it felt like coming full circle when, several days ago, I was able to sit in the virtual DebConf audience and watch two of my graduate student advisees—Kaylea Champion and Wm Salt Hale—present their research about Debian at DebConf21 .

Salt presented his masters thesis work which tried to understand the social dynamics behind organizational resilience among free software projects. Kaylea presented her work on a new technique she developed to identifying at risk software packages that are lower quality than we might hope given their popularity (you can read more about Kaylea’s project in our blog post from earlier this year ).

If you missed either presentation, check out the blog post my research collective put up or watch the videos below. If you want to hear about new work we’re doing—including work on Debian—you should follow our research group blog , and/or follow or engage with us in the Fediverse (@communitydata@social.coop ), or on Twitter (@comdatasci) .

And if you’re interested in joining us—perhaps to do more research on FLOSS and/or Debian and/or a graduate degree of your own?— please be in touch with me directly !

Wm Salt Hale’s presentation plus Q&A. ( WebM available )
Kaylea Champion’s presentation plus Q&A. ( WebM available )

Značky: #Debian

  • Pl chevron_right

    A Candid explainer: Safe higher-order upgrades

    pubsub.slavino.sk / planetdebian · Monday, 30 August - 21:21 · 5 minutes

This is the second post in a series about the inteface description language Candid .

Safe upgrades

A central idea behind Candid is that services evolve over time, and so also their interfaces evolve. As they do, it is desirable to keep the interface usable by clients who have not been updated. In particular on a blockchainy platform like the Internet Computer, where some programs are immutable and cannot be changed to accommodate changes in the interface of the services they use, this is of importance.

Therefore, Candid defines which changes to an interface are guaranteed to be backward compatible. Of course it’s compatible to add new methods to a service, but some changes to a method signature can also be ok. For example, changing

service A1 : { get_value : (variant { current; previous : nat }) -> (record { value : int; last_change : nat }) }

to

service A2 : { get_value : (variant { current; previous : nat; default }) -> (record { value : int; last_change : nat; committed : bool }) }

is fine: It doesn’t matter that clients that still use the old interface don’t know about the new constructor of the argument variant. And the extra field in the result record will silently be ignored by old clients.

In the Candid spec, this relation is written as A2 <: A1 (note the order!), and the formal footing this builds on is subtyping . We thus say that “it is safe to upgrade a service to a subtype”, and that A2 ’s interface is a subtype of A1 ’s.

In small examples, I often use nat and int , because every nat is also a int , but some int values are not not nat values, namely the negative ones. We say nat is a subtype of int , nat <: int . So a function that in the first returns a int can in the new version return a nat without breaking old clients. But the other direction doesn’t work: If the old version’s method had a return type of nat , then changing that to int would be a breaking change, as old clients would not be prepared to handle negative numbers.

Note that arguments of function types follow different rules. In fact, the rules are simply turned around, and in argument position (also called “negative position”), we can evolve the interface to accept supertypes . Concretely, a function that originally expected an nat can be changed to expect an int , but the other direction doesn’t work, because there might still be old clients around that send negative numbers. This reversing-of-the-rules is called contravariance .

The vision is that the developer’s tooling will warn the developer before they upgrade the code of a running service to a type that is not a subtype of the old interface.

Let me stress, though, that all this is about not breaking existing clients that use the old interface. It does not mean that a client developer who fetches the new interface for your service won’t have to change their code to make his programs compile again.

Higher order composition

So far, we considered the simple case of one service with a bunch of clients, i.e. the “first order” situation. Things get much more interesting if we have multiple services that are composed, and that pass around references to each other, and we still want everything to be nicely typed and never fail even as we upgrade services.

Therefore, Candid’s type system can express that a service’s method expects or a returns a reference to another service or method, and the type that this service or method should have. For example

service B : { add_listener : (text, func (int) -> ()) -> () }

says that the service with interface B has a method called add_listener which takes two arguments, a plain value of type text and a reference to a function that itself expects a int -typed argument.

The contravariance of the subtyping rules explained above also applies to the types of these function reference. And because the int in the above type is on the left of two function arrows, the subtyping rule direction flips twice, and it is therefore safe to upgrade the service to accept the following interface:

service B : { add_listener : (text, func (nat) -> ()) -> () }

Soundness theorem and Coq proof

The combination of these higher order features and the safe upgrade mechanism is maybe the unique selling point of Candid, but also what made its design quite tricky sometimes. And although the conventional subtyping rules work well, we wanted to do some less conventional things (more on that below), and more than once thought we had a good solution, only to notice that it did not hold water in complicated higher-order cases.

But what does it mean to hold water? I felt the need to precisely define a soundness criteria for higher order interface description languages, which you can find in the document An IDL Soundness Proposition . This is a general framework which you can instantiate with your concrete IDL language and rules, and then it tells you what you have to prove to consider your language to be sound. Intuitively, it simulates all possible ways in which services can be defined and upgraded, and in which they can pass around references to each other, and the soundness property is that then that for all messages sent between services, they can always be understood.

The document also shows, in general, that any system that builds on “canonical subtyping” in sound, as expected. That is still a generic theorem that you can instantiate with a concrete system, but now there is less to do.

Because these proofs can get tricky in corner cases, it is valuable to mechanize them in an interactive theorem prover and have the computer check the proofs. So I have created a Coq formalization that defines the soundness criteria , including the reduction to canonical subtyping. It also defines MiniCandid, a greatly simplified variant of Candid , proves various properties about it (transitivity etc.) and finally instantiates the soundness theorem.

I am particularly fond of the use of coinduction to model the equirecursive types, and the use of named cases , as we know them from Isabelle, to make the proofs a bit more readable and maintainable.

I am less fond of how much work it seems to be to extend MiniCandid with more of Candid’s types. Already adding vec was more work than it’s worth it, and I defensively blame Coq’s not-so-great support for nested recursion.

The soundness relies on subtyping, and that is all fine and well as long as we stick to canonical rules. Unfortunately, practical considerations force us to deviate from these a bit, as we will see in the next post of this series.


Značky: #Debian

  • Pl chevron_right

    Oh, my goodness, where's the fantastic barbeque [OMGWTFBBQ 2021]

    pubsub.slavino.sk / planetdebian · Monday, 30 August - 18:52 · 2 minutes

I'm guessing the last glasses will be through the dishwasher (again) and Pepper the dog can settle down without having to cope with so many people.

For those who don't know - Steve and his wife Jo (Sledge and Randombird) hold a barbeque in their garden every August Bank Holiday weekend [UK Bank Holiday on the last Monday in August]. The barbeque is not small - it's the dominating feature in the suburban garden, brick built, with a dedication stone, lights, electricity. The garden is small, generally made smaller by forty or so Debian friends and allies standing and sitting around. People are talking, arguing, hugging people they've not seen for (literal) years and putting the world to rights.

This is Debian central point - with large quantities of meat and salads, an amount of beer/alcohol and "Cambridge gin" and general goodwill. This year was more than usually atmospheric because for some of us it was the first time with a large group of people in a while. Side conversations abound: for me it was learning something about the high energy particle physics community, how to precision build helicopters, fly quadcopters and precision 3D print anything, the maths of Isy counting crochet stitches to sew together randomly sized squares ... and, of course, obligatory things like how random is random and what's good enough entropy. And a few sessions of the game of our leader.

This is also a place for stuff to get done: I was unashamedly using this to upgrade the storage in my laptop while there were sensible engineers around. A corner of the table, a RattusRattus and it was quickly sorted - then a discussion around the internals of Thinkpads as he took his apart. Then getting a full install - Gb Ethernet to the Debian mirror in the cupboard six feet away is faster bandwidth than a jumbo jet full of tapes. Then getting mail to work again - it's handy when the mailserver owner is next to you, having come in from the garden to help, and finally IRC. And not just me: "You need a GPG key signed - there's three DPLs here, there's a release manager - but you've just missed one of the DAMs." plus an in-depth GPG how-to session on the other side of the table.

I was the luckiest one with the most comfortable bed in the house overnight but I couldn't stay for last night. Thanks once again to all involved but especially Steve and Jo who do this for the love of it, and the fun, and the community and the family. Oh, and thanks to Lenovo - not just for being a platinum sponsor of Debconf but also for providing the official laptop of this and most Debian occasions 😀


Značky: #Debian

  • Pl chevron_right

    developers-reference needs love

    pubsub.slavino.sk / planetdebian · Monday, 30 August - 09:52 · 1 minute

During Debconf, Holger, who’s one of the developers-reference maintainers, made a quick presentation that was explaining the developers-reference needs some love. Indeed, it has gathered dust, and some useful refresh would be very welcome. Holger pointed at the list of bugs:
https://bugs.debian.org/src:developers-reference

After having a quick look into that list, after Holger’s Debconf presentation, I wrote to him on IRC:

<zigo> Many of the bugs you refered are indeed easily actionable, if all of us just try to help for one bug, that’d be a huge improvement of that doc.

Then, as I was waiting for the closing ceremony of Debconf, I thought I shouldn’t just say it, but actually do something about it. I decided to address https://bugs.debian.org/793633 as I thought it was easy. In just a few minutes, I was able to do a first patch, as seen here:

https://salsa.debian.org/debian/developers-reference/-/merge_requests/27

I wrote about it on IRC, and a few people helped with rephrasing what was there (thanks to Fil for correcting my English mistakes, and others for the content).

Today, which is 2 days after the MR was opened, I have decided it was long enough and actually merged it, as I considered it was enough time to gather comments. So we now have a brand new shiny chapter about Backports and how to handle them. I’m sure that new part is perfectible, so do not hesitate, and do patch what I just wrote if you feel like you can do better.

If I’m writing this blog post, this is not to promote myself. The goal is to promote the developers-reference manual and push others in Debian to do the same. Please do what Holger suggested, and what I just did: contribute to the document by addressing just one of the currently opened bugs. If all DDs do it, we’ll get a much nicer document, and help others to contribute to Debian.

This is going to take less than 30 minutes of your time, and it is very much ok if you do this only once. It is really easy: just clone https://salsa.debian.org/debian/developers-reference/ and write a patch. If you’re a DD, you can even merge your patch yourself once you’re satisfied with it.


Značky: #Debian

  • Pl chevron_right

    Links August 2021

    pubsub.slavino.sk / planetdebian · Monday, 30 August - 08:31 · 3 minutes

Sciencealert has an interesting article on a game to combat misinformation by “microdosing” people [1] . The game seemed overly simplistic to me, but I guess I’m not the target demographic. Research shows it to work.

Vice has an interesting and amusing article about mass walkouts of underpaid staff in the US [2] . The way that corporations are fighting an increase in the minimum wage doesn’t seem financially beneficial for them. An increase in the minimum wage means small companies have to increase salaries too and the ratio of revenue to payroll is probably worse for small companies. It seems that companies like McDonalds make oppressing their workers a higher priority than making a profit.

Interesting article in Vice about how the company Shot Spotter (which determines the locations of gunshots by sound) forges evidence for US police [3] . All convictions based on Shot Spotter evidence should be declared mistrials.

BitsNBites has an interesting article on the “fundamental flaws” of SIMD (Single Instruction Multiple Data) [4] .

The Daily Dot has a disturbing article anbout the possible future of the QAnon movement [5] . Let’s hope they become too busy fighting each other to hurt many innocent people.

Ben Taylor wrote an interesting blog post suggesting that Web Assembly should be a default binary target [6] . I don’t support that idea but I think that considering it is useful. Web assembly could be used more for non-web things and it would be a better option than Node.js for some things. There are also some interesting corner cases like games, Minecraft was written in Java and there’s no reason that Web Assembly couldn’t do the same things.

Vice has an interesting article about the Phantom encrypted phone service that ran on Blackberry handsets [7] . Australia really needs legislation based on the US RICO law!

Vice has an interesting article about an encrypted phone company run by drug dealers [8] . Apparently after making an encrypted phone system for their own use they decided to sell it to others and made millions of dollars. They could have run a successful legal business.

Salon has an insightful interview with Michael Petersen about his research on fake news and people who share it because they need chaos [9] . Apparently low status people who are status seeking are a main contributor to this, they share fake news knowingly to spread chaos. A society with less inequality would have less problems with fake news.

Salon has another insightful interview with Michael Petersen, about is later research on fake news as an evolutionary strategy [10] . People knowingly share fake news to mobilise their supporters and to signal allegiance to their group. The more bizarre the beliefs are the more strongly they signal allegiance. If an opposing group has a belief then they can show support for their group by having the opposite belief (EG by opposing vaccination if the other political side supports doctors). He also suggests that lying can be a way of establishing dominance, the more honest people are opposed by a lie the more dominant the liar may seem.

Vice has an amusing article about how police took over the Encrochat encrypted phone network that was mostly used by criminals [11] . It’s amusing to read of criminals getting taken down like this. It’s also interesting to note that the authorities messed up by breaking the wipe facility which alerted the criminals that their security was compromised. The investigation could have continued for longer if they hadn’t changed the functionality of compromised phones. A later vice article mentioned that the malware installed on Encrochat devices recorded MAC addresses of Wifi access points which was used to locate the phones even though they had the GPS hardware removed.

Cory Doctorow wrote an insightful article for Locus about the insufficient necessity of interoperability [12] . The problem if monopolies is not just an inability to interoperate with other services or leave it’s losing control over your life. A few cartel participants interoperating will be able to do all the bad things to us tha a single monopolist could do.


Značky: #Debian

  • Pl chevron_right

    A Candid explainer: The rough idea

    pubsub.slavino.sk / planetdebian · Sunday, 29 August - 15:51 · 4 minutes

One of the technologies that was created by DFINITY as we built the “Internet Computer” is Candid . Candid is

  • a language to describe the interface of a service, with named methods, arguments and results,
  • a type system for these arguments and results that maps to many possible implementation languages, thus enabling interoperability between these languages,
  • a wire format (encoding) of values of these types,
  • a notion of “safe upgrade” that allows service developers to evolve their interface with the assurance that no clients break, even in the presence of higher-order composition,
  • a meta-theory of safe interface description languages and a formal proof that Candid has these properties, and
  • a bunch of tools and libraries.

In this in-depth blog post series I want to shed some light onto these aspects of Candid. The target audience is mainly anyone who wants to deeply understand Candid, e.g. researchers, implementors of Candid tools, anyone who wants to builds a better alternative, but no particular prior Candid knowledge is expected. Also, much of what is discussed here is independent of the Internet Computer. Some posts may be more theoretical or technical than others; if you are lost in one I hope you’ll rejoin for the subsequent post

Much in these posts is not relevant for developers who just want to use Candid in their projects, and also much that they want to know is missing. The Internet Computer dev docs will hopefully cater to that audience.

Blog post series overview

  1. The rough idea (this post)

    Announcing the blog post series, and very briefly outlining what an Interface Description Language even is.

  2. Safe higher-order upgrades (to be published)

    Every can do first-order interface description languages. What makes Candid special is its support for higher-order service composition. This post also contains some general meta-theory and a Coq proof!

  3. Opt is special (to be published)

    Extending records in argument position is surprisingly tricky. Lean why, and how we solved it.

  4. Language integration (to be published)

    The point of Candid is inter-op, so let’s look the various patterns of connecting Candid to your favorite programming language.

  5. Quirks (to be published)

    The maybe most interesting post in this series: Candid has a bunch of quirks (hashed field names, no tuples but sequnces, references that nobody used, and more). I’ll explain some of them, why they are there, and what else we could have done. Beware: This post will be opinionated.

The rough idea

The idea of an interface description language is easy to begin with. Something like

service A : {
  add : (int) -> ();
  get : () -> (int);
}

clearly communicates that a service called A provides two methods add and get , that add takes an integer as an argument, and that get returns such a number. Of course, this only captures the outer shape of the service, but not what it actually does – we might assume it implements a counter of sorts, but that is not part of the Candid interface description.

In order to now use the service, we also need a (low-level) transport mechanism. Candid itself does not specify that, but merely assumes that it exists, and is able to transport the name of the invoked method and a raw sequence of bytes to the service, and a response (again a raw sequence of bytes) back.

Already on the Internet Computer we have two distinct transport mechanisms used are external calls via an HTTP-based RPC interface and on-chain inter-canister calls . Both are handled by the service in the same way. Here we can see that Candid succeeds in abstracting over differences in the low-level transport mechanism. Because of this abstraction, it is possible to use Candid over other transportation mechanisms as well (conventional HTTPS, E-Mail, avian carrier). I think Candid has potential there as well, so even if you are not interested in the Internet Computer, this post may be interesting to you.

The translation of argument and result values (e.g. numbers of type int ) to the raw sequence of bytes is specified by Candid , defining a wire format . For example, the type int denotes whole numbers of arbitrary size, and they are encoded using the LEB128 scheme . The Candid wire format also contains a magic number ( DIDL , which is 0x4449444c in hex) and a type description (more on that later). So when passing the value 2342 to the add , the raw bytes transported will be 0x4449444c00017da612 .

Of course we want to transfer more data than just integers, and thus Candid supports a fairly complete set of common basic types ( nat , int , nat8 , nat16 , nat32 , nat64 , int8 , int16 , int32 , int64 , float32 , float64 , bool , text , blob ) and composite types ( vec tors, opt ional values , record s and variant s). The design goal is to provide a canonical set of types – enough to express most data that you might want to pass, but no more than needed, so that different host languages can support these types easily.

The Candid type system is structural . This means that two types are the same when they are defined the same way. Imagine two different services defining the same

type User = record { name : text; user_id : nat }

then although User is defined in two places, it’s still the same type. In other words, these name type definitions are always just simple aliases , and what matters is their right-hand side.

Because Candid types can be recursive (e.g. type Peano = opt Peano ), this means we have an equirecursive type system, which makes some things relatively hard.


Značky: #Debian

  • Pl chevron_right

    Latest topics about fabre.debian.net

    pubsub.slavino.sk / planetdebian · Sunday, 29 August - 11:46

Aug 28, 2021, I gave a short talk about "Latest topics about fabre. debian .net" at DebConf21.

You can see the presentation slide here:

slide.rabbit-shocker.org

Short talk explanation is here:

debconf21.debconf.org

In this talk , I've explained mainly 3 topics

I've sent a pre-recorded video in advance, it is ok, but there is trouble with the mic during the session. :-(

The recorded video will be available on the talk detail page soon.


Značky: #Debian

  • Pl chevron_right

    2021/08, FLOSS activity

    pubsub.slavino.sk / planetdebian · Saturday, 28 August - 18:00 · 3 minutes

LTS

This is my sixth month of working for LTS. I was assigned 12 hrs and worked all of them.

Released DLAs

  1. DLA 2742-1 ffmpeg_7:3.2.15-0+deb9u3

    • CVE-2020-22036 : A heap-based Buffer Overflow vulnerability in filter_intra at libavfilter/vf_bwdif.c, which might lead to memory corruption and other potential consequences.
    • CVE-2020-22032 : A heap-based Buffer Overflow vulnerability in gaussian_blur, which might lead to memory corruption and other potential consequences.
    • CVE-2020-22031 : A Heap-based Buffer Overflow vulnerability in filter16_complex_low, which might lead to memory corruption and other potential consequences.
    • CVE-2020-22028 : Buffer Overflow vulnerability in filter_vertically_8 at libavfilter/vf_avgblur.c, which could cause a remote Denial of Service.
    • CVE-2020-22026 : Buffer Overflow vulnerability exists in the config_input function at libavfilter/af_tremolo.c, which could let a remote malicious user cause a Denial of Service.
    • CVE-2020-22025 : A heap-based Buffer Overflow vulnerability exists in gaussian_blur at libavfilter/vf_edgedetect.c, which might lead to memory corruption and other potential consequences.
    • CVE-2020-22023 : A heap-based Buffer Overflow vulnerabililty exists in filter_frame at libavfilter/vf_bitplanenoise.c, which might lead to memory corruption and other potential consequences.
    • CVE-2020-22022 : A heap-based Buffer Overflow vulnerability exists in filter_frame at libavfilter/vf_fieldorder.c, which might lead to memory corruption and other potential consequences.
    • CVE-2020-22021 : Buffer Overflow vulnerability at filter_edges function in libavfilter/vf_yadif.c, which could let a remote malicious user cause a Denial of Service.
    • CVE-2020-22020 : Buffer Overflow vulnerability in the build_diff_map function in libavfilter/vf_fieldmatch.c, which could let a remote malicious user cause a Denial of Service.
    • CVE-2020-22016 : A heap-based Buffer Overflow vulnerability at libavcodec/get_bits.h when writing .mov files, which might lead to memory corruption and other potential consequences.
    • CVE-2020-22015 : Buffer Overflow vulnerability in mov_write_video_tag due to the out of bounds in libavformat/movenc.c, which could let a remote malicious user obtain sensitive information, cause a Denial of Service, or execute arbitrary code.
    • CVE-2020-21041 : Buffer Overflow vulnerability exists via apng_do_inverse_blend in libavcodec/pngenc.c, which could let a remote malicious user cause a Denial of Service
    • CVE-2021-3566 : The tty demuxer did not have a ‘read_probe’ function assigned to it. By crafting a legitimate “ffconcat” file that references an image, followed by a file the triggers the tty demuxer, the contents of the second file will be copied into the output file verbatim (as long as the -vcodec copy option is passed to ffmpeg).
    • CVE-2021-38114 : libavcodec/dnxhddec.c does not check the return value of the init_vlc function. Crafted DNxHD data can cause unspecified impact.
  2. DLA 2742-2 ffmpeg_7:3.2.15-0+deb9u4 During the backporting of one of patches in CVE-2020-22021 one line was wrongly interpreted and it caused the regression during the deinterlacing process. Thanks to Jari Ruusu for the reporting the issue and for the testing of prepared update.

Other LTS-related work

  • Analyzed CVE-2020-22027 and marked as ignored for stretch. Original patch is not appliable. The issue is reproducible though.

  • Analyzed CVE-2020-22019 and marked as ignored for stretch.

  • Analyzed CVE-2020-22017 and marked as ignored for stretch.

  • Updated tomcat8-LTS-salsa-repo to tne newer 8.5.54-0+deb9u7 version.

  • Minor WIKI update.

  • Frontdesk duties CW33/2021. It was my fist attempt to do such kind of task. I triaged apache2 and gitit.

  • Analyzed CVEs in firmware-nonfree .

  • Started to work on rustc .

LTS-Meeting

  • I attended the Debian LTS team Jitsi-meeting (though the connection was extremely bad).
  • Partly participated in preparation of Debconf21 BoF “Funding Projects to Improve Debian”.

Debian Science Team

  • Partly participated in Debconf21 Debian Science BoF.

Other FLOSS activities

  • Reviewed many merge requests in Yade open source project, merge some of them.

Značky: #Debian