close
  • chevron_right

    Movim 0.20 - Skiff

    Timothée Jaussoin · pubsub.movim.eu / Movim · Saturday, 19 February - 10:25 edit · 3 minutes

I was used to #release a new version of #Movim twice a year. Skiff is an exception. One year of work was required to release the 20th major version of the project.

The main reason is mostly based on the amount of work and adjustments required to integrate the main feature of this release: the support of end-to-end #encryption through the implementation of OMEMO.

So let's dive in all the new exciting features that you will discover in this major release.

OMEMO

The technical part was already extensively covered by the dedicated article End to end encryption in Movim - OMEMO is (finally) there!.

The user experience and flow is not very different than on other XMPP clients, if Movim detects that you can start an encrypted conversation with a contact a small lock icon will be displayed next to the chatbox. You can always choose to toggle it back to have a non-encrypted discussion.

The new redesigned chatbox

It is also possible to see all the encryption fingerprints in the Contact drawer under the dedicated "Fingerprints" tab. You can also enable and disable encryption to each fingerprint manually there. Movim is displaying the last message sent or received and the client linked to the fingerprint to help you with your configuration. But rest assured, those settings are only for those that wants to configure in detail their encryption levels.

OMEMO Fingerprints

End-to-end encryption is also available for group chats, the flow is exactly the same as for single contacts.

There is some chances that you encounter encryption issues in some cases, even after a lot of debug and refactoring end-to-end encryption is a really complex beast that is difficult to handle. Feel free to open a ticket with all the details to reproduce the issue if you encounter one.

I'd like to thank again NLNet for their help on this project ! With the funding I was able to free-up time to finally integrate end-to-end encryption in Movim.

NLNet Logo

Posts

A few changes were made regarding the posts and their integration within Movim.

The post publication form was slightly redesigned and now allows several images, files or links to be attached. Linked to that change, post cards were also redesigned with a more compact design.

Multiple attached pictures

The public Communities and Blog pages now have the same 2-columns design as their private version. The displayed Communities and Contacts information are also now more compact.

Two column design for the public pages

The tags were redesigned and are now more clearly visible and navigable.

Now design for the tags

Chat

The contacts and chatrooms drawers were redesigned and now include some really useful information. Pictures and links sent in conversations are now quickly available in dedicated tabs.

Redesigned chat drawer

Chat bubbles are now properly displaying quotes and support message styling.

Chat bubble with styling

A big refactoring was also done regarding how the edited messages are handled in Movim. This refactoring allowed messages to be edited in Group Chats and the support of several edits on a single message (which caused some weird message duplication bug).

Chatrooms

Chatrooms administrators can now manage affiliations and ban/unban users.

Changing affiliation for a user

You can now prioritize your most important chatrooms on top of the list with the pin feature.

Pinned chatrooms

...and many other things

The old Movim API code was fully removed. It had been left untouched for years and not really used nor up-to-date anymore.

When you are in a chat conversation, the other chats counter is displayed on the back button.

The internal picture library was rewritten and simplified, it now supports transparent avatars. All pictures are now compressed in WebP by default.

Admins can now fully disable the registration feature. It is quite useful if you have a dedicated Movim setup and a specific separated flow to register your users (using an internal LDAP in a company or school for example).

Plenty of new emojis were integrated with the support of Unicode 13.0.

Movim is now a Progressive Web App

Movim used to have some "native" apps, on desktop and Android. All those app are now deprecated and replaced by work that was done to make Movim a full Progressive Web App. From any browsers you can now install Movim as an app on your phone or desktop in a single click.

Conclusion

Lots of other small improvements and features were integrated in this release but not listed there, it's time for you to discover them. Enjoy this new version!

That's all folks!

  • chevron_right

    Movim is available in Basque! HOT PEPPER

    Timothée Jaussoin · pubsub.movim.eu / Movim · Thursday, 20 January - 22:17 edit

Once in a while I download and synchronize the #Movim #translations from the Transiflex platform.

Thanks to the awesome community, Movim is now translated in 57 different languages.

I was really surprised that the project was fully translated in Basque. A language spoken in the beautiful #Basque Country that sits between France and Spain. I personnally lived there for a few years (in Bayonna) and really loved those beautiful lands!

I'd like to thank again all the people that are working hard on the Movim translations!

Mila esker (Thank you in Basque) !

  • chevron_right

    End to end encryption in Movim - OMEMO is (finally) there!

    Timothée Jaussoin · pubsub.movim.eu / Movim · Wednesday, 15 December - 22:01 edit · 7 minutes

A few days ago I finally closed the OMEMO encryption ticket on Github. Opened in 2015 it had many twists and turns along the years but I finally found a proper way of integrating it in Movim.

In this article I'll explain why adding #E2EE (End to End Encryption) was not as easy as with the other #XMPP clients (and more generaly all the chat clients that are using a similar encryption protocol) and how I addressed the issue.

But before going in the details I'd like to thanks the NLNet Fundation for its financial support in this project. With their help I was able to free-up some time to work on the problem and propose a proper architecture (detailled bellow) for it.

NLNet Foundation logo

The result of this work will be released with the upcoming 0.20 version of #Movim. There is still some quirks and whims about it but the base is there and works pretty well.

End to End encryption in XMPP, a quick overview

The introduction of Signal in 2015 brought a small revolution into the encryption protocols in the IM ecosystem. The Double Ratchet Algorythm (see the dedicated technical documentation on the Signal website) allowed users to exchange messages between different clients in an “end to end encrypted” way (only user devices themselves know how to encrypt and decrypt messages) with some technical improvements (not detailed here) that made the new protocol a “must have” for all the others IM solutions.

Today the Double Ratchet Algorithm is used in applications such as WhatsApp or Matrix.

In the XMPP ecosystem it was primarily pushed by Daniel Gultsch in the Conversations.im client and standardized along the way in the OMEMO XMPP Extension XEP-0384: OMEMO Encryption. Throughout the years many XMPP clients implemented OMEMO, their status can be tracked on the following website Are we OMEMO yet?.

The OMEMO architecture

Without going too deep into the technical details the general idea about OMEMO is to generate some keys on each of the user's devices and publish the public ones on their account server.

Using the keys published on the XMPP user's servers, anyone can then start an encrypted session at any time (the servers are always available) and start to send messages to the desired contact without having to wait.

Publishing keys and building sessions with OMEMO

If one of the user's contacts wants to start an encrypted discussion they will first start to get those keys, then build sessions with their secret one and encrypt a message using the freshly built sessions.

If a user receives a new encrypted message and doesn't have an encryption session to that device, their device will then retrieve the contact keys, build the encryption sessions and start decrypting messages.

This can be done automatically if the contact trusts blindly the key used or in a more “trusted” way by accepting manually each keys to build the encryption sessions on.

All the existing XMPP clients are using this simple architecture. XMPP servers are storing their users' #OMEMO public keys and the users are connecting directly using their different devices to build their encrypted sessions.

The Movim particularity

But Movim is kind of special. The XMPP connection is actually not maintained on user devices but by the Movim server (built in PHP and running behind a web server such as Apache or nginx, see Movim General architecture on the Wiki). Movim is then processing everything server side, saving the information (articles, contacts and messages) in a SQL Database (PostgreSQL or MySQL) and then showing the result to the Movim users through a lightweight website.

If a user is connecting on the same Movim instance through several browsers using the same XMPP account all the browsers are then “merged” into one unique XMPP session (called "resource") and all the browsers are synchronized in real time by the Movim server. This is pretty useful to save memory and to prevent Movim to maintain several XMPP connections at the same time for a unique user. This also allows quick disconnection/reconnections, the users can close and reopen their tabs without having to reload the whole XMPP state when they come back after a while (Movim is closing the XMPP session after a day of inactivity).

End to end encryption actually requires to encrypt and decrypt messages on the user device, this brings several issues:

  • For Movim, the user device is actually a “dumb” browser that only display the messages pre-processed by the Movim server, there is no logic whatsoever browser side
  • A user can use simultaneously several browsers with the same XMPP connection on the same Movim instance
  • All the message processing logic is done server side

This unique architecture requires a very unique way of adressing the E2EE situation. Hopefully OMEMO offers all the tools needed to handle those cases.

Split the logic

The OMEMO extension is actually talking about devices, for a large majority of the XMPP clients a device is connected through a unique XMPP session (one device equal one current XMPP resource in those cases).

Publishing keys with Movim

The fact that Movim is sharing a unique session (resource) with several devices (browsers) is actually not an issue in the end. Each browser will then be considered as a unique device on its own, with its own key and its own OMEMO encrypted sessions.

Building encrypted sessions with Movim

This brings some interesing results. When a user is connected using the same XMPP account using two different browsers on the same Movim server (also called instance, or pod), an encrypted message sent by the browser Firefox will then directly be decrypted by the browser Chrome without even having to travel through the XMPP network.

The term “browser” is also defining more than actual browsers (like Firefox, Chrome or Opera). Since we can have private navigation or containers (in Firefox) each time it is seen as a different “browser” on the Movim side (because each context is separated, with a different cookie and different local data).

So the global idea is to continue to handle the messages server side, push the encrypted message object to the browser, and then implement only the key handling and message encryption-decryption flow browser side. When doing this implementation I actually looked at the Converse.js and JSXC OMEMO implementations, the Movim implementation is really close to the one done on those two clients (I am also re-using the libsignal JavaScript implementation).

This architecture actually works for the current version of OMEMO (0.3.0) where only the body is encrypted. The upcoming versions are looking to encrypt a larger part of the XML stanza. This will be way more difficult to handle for Movim, as it will require to decrypt messages browser side and then implement a second parser, this time in JavaScript (everything is parsed in PHP using libxml at the moment).

if (textarea.dataset.encryptedstate == 'yes') {
    // Try to encrypt the message
    let omemo = ChatOmemo.encrypt(jid, text, Boolean(textarea.dataset.muc));
    if (omemo) {
        omemo.then(omemoheader => {
            ...
            xhr = Chat_ajaxHttpDaemonSendMessage(jid, tempId, muc, null, replyMid, mucReceipts, omemoheader);
            ...
        });
    }
} else {
    xhr = Chat_ajaxHttpDaemonSendMessage(jid, text, muc, null, replyMid, mucReceipts);
    ...
}

This little JavaScript Movim code extract presents the differences in handling encrypted and unencrypted messages. The text variable is containing the clear text version of the message. When the body is encrypted it is then calling the same method as for a clear text message.

This method is actually a wrapper generated by the RPC (Remote Procedure Call) Movim server core. Once this function is called an Ajax called is made and the rest of the flow is handled server side. The encrypted body, and generated OMEMO headers passed will be injected in a freshly generated XMPP XML <message>.

Keep the messages in the local database

With the separation of the logic it was then required to keep a copy of the decrypted messages browser side.

To do that an IndexedDB database is used. This database is quite simple and only contains a key-value store, where the key is the message id (the same as the one in the Movim server SQL server databse) and the value the plaintext message.

  • When a message is decrypted the plaintext body is then stored in this database.
  • When the user sends an encrypted message, the original text is also saved in this database.
  • If a message cannot be decrypted, the message key is still saved in the browser database with a false value. This prevents Movim to try to decrypt several times a message, knowing that the decryption will fail each time in the end.

Using this database, when a chat is loaded, all the messages are then sent chronologically from the server, passed trough a little bit of code that will lookup the state of all messages and then decrypt the ones that are not decrypted yet, the already decrypted messages are then shown, or an error is displayed for those that cannot be decrypted.

To sum up

In this article I tried to present you what limitations I faced when trying to implement end to end encryption within Movim and what architectural and technical solutions were used to address them.

The current solution seems to fit and bring all the desired features to Movim without too much downsides. The feature can now be considered as done and will be released soon. And as always, lots of small fixes and adjustments will be integrated to polish it afterward.

That's all folks!

edhelas

  • favorite

    8 Like

    Sal, Kris, Matt, Jorge Luis, U, matlag, Marzanna, Alexander

  • 2 Comments

  • person

    23 December stefan

    Great work! Does OMEMO also work for group chats?

    I'm however a little worried if the Javascript handling the keys is served by the same server which e2ee protect against. If I understand correctly, a malicious server operator could easily extract my private key by (temporarily) serving a modified, malicious version of the Movim frontend.

  • 31 December Alexander

    Bon boulot

  • chevron_right

    A little update from the Movim project

    Timothée Jaussoin · pubsub.movim.eu / Movim · Tuesday, 16 November, 2021 - 21:29 edit · 1 minute

It's been a while since I have told you about Movim. I was quite busy the past few months with many other things in my personal life, but this doesn't mean that I haven't worked on Movim!

So in short, here is a few things that have been changed recently:

api.movim.eu

api.movim.eu has been upgraded and refreshed with a new Laravel backend, Google ReCAPTCHA has been replaced with hCaptcha

The Legals page will be soon upgraded. A specific ticket has been opened to allow you to report and bring ideas on how to improve it.

End of the Android application

Following our previous blog post about the state of the Android application I decided to officially archive the Android app and remove it from F-Droid.

Some work has been done to integrate Movim better as a #PWA (Progressive Web App) to still allow you to enjoy it from your mobile phone.

Security Audit

We had an extensive #SecurityAudit by Radically Open Security that did an amazing job.

The audit covered Movim itself but also api.movim.eu and all the related projects, including movim/movim_docker. The result of the audit is currently restricted but we are actively using the finding to improve the general project security:

An finally on Movim itself

  • I added a basic support of OMEMO for MUC Groups, you can already try it out on the official pod
  • And as always, bugs fixes!

That's all folks!

edhelas

  • chevron_right

    Element and Movim Messengers Comparison Made Simple - ubuntubuzz.com

    Timothée Jaussoin · pubsub.movim.eu / Movim · Sunday, 3 October, 2021 - 20:05 edit

A nice #comparison between #Movim (XMPP) and Element (Matrix) from similar features. Thanks !

  • chevron_right

    We are looking for some help for the Movim Android application

    Timothée Jaussoin · pubsub.movim.eu / Movim · Monday, 31 May, 2021 - 20:17 edit · 2 minutes

Hi,

I am writing this little post today to ask for (a bit) of help on the official Movim #Android application that is currently available there.

I am maintaining #Movim for more than 10 years now, mostly without asking for any help from the hundreds of daily users that we have now (on the servers we're aware of, at least). I am also maintaining and updating the infrastructure hosting the website, the official pod and the #XMPP server and related services.

However Movim is a little more than that, there is also an Android app, and a desktop app (currently abandoned).

I am maintaining the Android app with the really thin knowledge that I have on this platform. Most of the code of the application is basically copy/pasted from StackOverflow and a bit hacky. Fixing and developing this app is always really time consuming as the environment, and the way of dealing with it, is really different than the knowledge that I have on the web development side.

So basically today I'm asking from some help for the official Movim Android application. If you have a little bit of experience in Android development (or if you want to learn !) and if you're willing to help me to improve the application do not hesitate to contact me through our official chatroom, directly on Github or as a comment of this post.

The app is quite simple. It is basically a WebView that shows one of the configured pods. And… that's mostly it.

What would be really helpful would be to have (non exhaustive list):

  • Another pair of eye to check and maybe refactor the few classes that are contained in the application
  • Create two flavors of the app, one "Play Store ready" and one "free of Google Play" integration (the current app cannot be published on F-Droid because there is some strong dependencies there)
  • Fix the mic/camera support and authorizations
  • See if the current notification system can be improved
  • Send an event to Movim when the app is "put in background" in a chat conversation to ensure that the notifications are re-enabled in this specific case
  • And any other nice feature that you would like to see integrated

There is no pressure or deadline, any pull requests that could help improving or fixing things in the application would be really appreciated.

I am currently dealing with Google to re-enable the application on the Play Store as well, I will keep you updated about that.

Regards,

edhelas

  • Pictures 1 image

  • visibility
  • favorite

    8 Like

    arie, thuraht, adbenitez, hentamatsu, valeg, Arne, bung, kefah

  • 2 Comments

  • person

    24 August, 2021 kefah

    First of all: Hats off for the amazing accomplishment. I just came across movim today and I'm pretty impressed.

    How can I learn more about the codebase? What technologies did you chose? I can see that the android app is almost identical to the web, so I assume that the web version is embedded into the android app.

    Do you have the code/issues on github?

  • person

    24 August, 2021 kefah

    My bad! just noticed the github link on your post https://github.com/movim/movim_android
    Will look into it and come back. Thanks!

  • chevron_right

    Movim 0.19 - Ikeya is out!

    Timothée Jaussoin · pubsub.movim.eu / Movim · Saturday, 20 February, 2021 - 21:58 edit · 7 minutes

This version is a major step for Movim with a lot of improvements on the performances and many (many!) small fixes and new features.

Chat and chatrooms

A criticism that was coming once in a while is how difficult it was to join and/or create a chatroom in Movim. The 0.19 version follow the Modern XMPP Multi-user Chats recommendations that is already implemented in several other XMPP clients. This standard is bringing a common naming convention and flow to create and configure chatrooms in XMPP.

Add chatroom dialog

On top of that the join/create chatroom flow was completely redesigned. The new one is inspired by the Telegram one, with one unique "+" button to start a new one-to-one or one-to-many conversation. The global chatroom search was also directly integrated in the flow to allow you to quickly search and join one of the few thousands already listed ones.

The one-to-one chat conversations views are now cached server side. This simple trick is saving some precious milliseconds during the page load. The performance boost allowed us to simply remove the chat list placeholder (you don't need a placeholder if the content is loading instantly).

Conversations improvements

The encrypted messages are now displayed in the bubbles.

An encrypted message

You can receive messages read confirmation in group chats and in small channels (when there are less than 10 persons connected).

XEP-0201: "Best Practices for Message Threads" has been implemented. Now Movim handles replies on received messages. Useful to follow multiple topics during a conversation.

A replied message

Like images before, now videos are also embedded in conversations with the support of the WebM and H264 video format.

With this awesome feature, the Tenor API was also integrated. You can now search in millions of GIFs and publish them instantly in your conversations. The Tenor API is disabled by default, the server administrator can enable it by adding its custom API key in the Movim admin panel.

The GIF picker

All the published message URLs are also now embedded in the discussions. Now you can also easily browse the URL related pictures directly in the Movim picture preview gallery widget.

An embeded URL

Chat rooms with unread messages are put on top of the chat rooms list now.

A package of ~1500 new emojis were added to the emoji picker.

A couple of new emojis

Communities and blogs

The explore page was reorganized with a few new features. You can now filter the explore panel to return only the communities or blog articles (the same way you can filter the articles on the news page). A "More" button is also now allowing you to retrieve older articles.

The new discover page

The communities on the servers are now ordered by last updated.

But also…

The preview widget is now having a "copy to clipboard" button. Useful!

Preview and copy easily the picture link

There is a first integration of the Firebase Push notifications in Movim. This is currently in test with the Android app and will require some more improvements and adjustments before being released as a stable feature.

You can now directly drag & drop files or copy & paste pictures in Movim to trigger the upload dialog box.

The drag and drop box

Performances

Batch insertion

Lets now dive in the more technical aspects with the performances improvements.

Most of the performances bump in Ikeya are actually regarding how he database is requested. In the previous Movim version, a presence buffer was introduced to save "batches" of incoming requests in one database query and not hundred of them.

The presence is a core concept of XMPP (eXtensible Messaging and Presence Protocol). During the connection, you will receive a presence from each online contact from your contact list, similarly, when you join a chatroom, you will as well receive a presence for each member connected to the room.

So now imagine that you have a big account, with hundreds of contacts and a few dozen chat rooms with hundreds of connected users in each of them. During the login, your XMPP client will then receive thousands of presences. The presence buffer is there to "stack them" when they are received and save them in batches in the database and then notifies the UI (your browser) only once that "all those presences were saved".

This presence buffer element was greatly improved and will not only save the presences in batches now, but also handle some related data like capabilities requests (those information are used to know what your contacts XMPP clients are capable of) as well as vcard requests (your contact personal information and avatars).

In a similar aspect, the retrieved bookmarked conferences and community subscriptions are also now saved in batches in the database, saving dozens of requests.

Preload

Another important optimisation that can be done when dealing with databases is to preload some information when retrieving list of items. Lets imagine that you need to retrieve 50 messages from the database. Those messages might have related data, such as reactions, replies or attached files. It is often good to load in parallel all those data "in batches".

The wrong way
1. Get the messages A, B, E and F
2. Process the messages
  - Check if A has an attached file
  - Check if B has an attached file
  …

The good way
1. Get the messages A, B, E and F
2. Try to get files for the A, B, E and F messages
3. Once the two lists are retrieved, see if some files matches the messages

This was already done in many places in Movim (hopefully), but even after years we are still finding some small improvements that were missing. All those small improvements are now allowing Movim to retrieve full conversations and scroll the history in a few hundred milliseconds, even if the database is filled with millions of messages.

You should know that each time you open a discussion in Movim, your browser is actually asking the server about all the contact that will be displayed, there is no local cache. If you have a backend that is fast enough, you don't need complex Javascript frontends to process and store things ;)

Database connection auto-close

Each connected user on a Movim instance is having it's own specific process launched on the server. This is bringing some nice isolation and performances distribution, especially on servers with many parallel available threads (this also brings some memory consumption issues that will be improved in the upcoming versions).

For each of those launched "user-session-processes" a database connection was opened to allow them to perform database queries quickly.

The main issue there is that database servers (such as MySQL and PostgreSQL, the two supported servers for Movim) can only handle a certain limit of parallel connections. Movim is then now automatically closing unused connections after a few seconds and resume them once a new query is sent. This allows Movim to handle hundreds of parallel sessions without overloading the database server anymore.

Translations cache

The languages translations are now cached when the daemon is launched once for all. This brings a few milliseconds gain during page load. It's not much but it's an easy win!

The translations can also be manually recompiled using the following command

php daemon.php compileLanguages

Picture proxyfying and caching

Most of the Movim pictures are now proxyfied by Movim. This way the Movim pod will hide its users personal IPs and browser information when the pictures are retrieved. The proxy is also moving all the pictures URL under a specific picture/ directory. This simple trick allows server admins to create a nice caching system directly on the web-server level.

For example, if a friend in a chatroom is sharing a nice picture that is a bit heavy. The first person that will receive the picture will ask Movim, through the Movim web-server to download it and return an optimized version of it. The web-server will then keep a copy of the optimized picture. All the other users will then ask the same resource, and the web-server will simply return the cached version, without even asking anything to Movim anymore.

A specific section in the Movim setup tutorial was added to explain you how to setup and configure this cache system during the Movim deployment.

To conclude

This was quite a big version indeed, we hope that you will enjoy it. The upcoming one will be a special release and will focus on one specific feature. We will come back to it in the upcoming weeks, be patient.

A clue about the upcoming release?

As always, if you like Movim please share it around. You can always help us by directly contributing to the project and help funding us on our Patreon.

That's all folks!

  • Pictures 1 image

  • visibility
  • favorite

    10 Like

    hentamatsu, valeg, typhoon, adbenitez, Arne, archerships, odysseuslibre, Xabi, Stefan, debacle

  • chevron_right

    Movim | Basic Review & Beginner's Guide

    Timothée Jaussoin · pubsub.movim.eu / Movim · Sunday, 14 February, 2021 - 11:17 edit

Thanks for this awesome guide by UbuntuBuzz. With a lot of details and some ideas of improvements for the future. #Movim #review #guide

  • favorite

    9 Like

    wxyz, fedext, Yannv, U, Xabi, Matt, ericbuijs, debacle, eyome

  • 1 Comments

  • 10 April, 2021 preptorrent

    Please can you put Movim back so html tables can be written in our posts the old way?

    Sadly, Movim have now changed the ability to write in html so your tables end up garbled. What a terrible shame. I wish Movim had just left it as it was before. It was fine as html! Nowadays it forces your markup to be their Movim markdown which is simply nowhere as neatly laid out as html with tables. Also, people have long understood html.

    I feel gutted about this. I'm not a fan of markdown being mandatory in place of html. Please can you put Movim back so html can be written the old way? I used to show people how cool movim was for that. It used to be so cool.