close
  • Pl chevron_right

    How to rsync files between two remotes?

    pubsub.slavino.sk / planetdebian · Saturday, 6 November - 10:41 · 3 minutes

scp -3 can copy files between two remote hosts through localhost. This comes in handy when the two servers cannot communicate directly or if they are unable to authenticate one to the other. 1 Unfortunately, rsync does not support such a feature. Here is a trick to emulate the behavior of scp -3 with SSH tunnels.

When syncing with a remote host, rsync invokes ssh to spawn a remote rsync --server process. It interacts with it through its standard input and output. The idea is to recreate the same setup using SSH tunnels and socat , a versatile tool to establish bidirectional data transfers.

The first step is to connect to the source server and ask rsync the command-line to spawn the remote rsync --server process. The -e flag overrides the command to use to get a remote shell: instead of ssh , we use echo .

$ ssh web04
$ rsync -e 'sh -c ">&2 echo $@" echo' -aLv /data/. web05:/data/.
web05 rsync --server -vlogDtpre.iLsfxCIvu . /data/.rsync: connection unexpectedly closed (0 bytes received so far) [sender]rsync error: error in rsync protocol data stream (code 12) at io.c(228) [sender=3.2.3]

The second step is to connect to the destination server with local port forwarding. When connecting to the local port 5000, the TCP connection is forwarded through SSH to the remote port 5000 and handled by socat . When receiving the connection, socat spawns the rsync --server command we got at the previous step and connects its standard input and output to the incoming TCP socket.

$ ssh -L 127.0.0.1:5000:127.0.0.1:5000 web05
$ socat tcp-listen:5000,reuseaddr exec:"rsync --server -vlogDtpre.iLsfxCIvu . /data/."

The last step is to connect to the source with remote port forwarding. socat is used in place of a regular SSH connection and connects its standard input and output to a TCP socket connected to the remote port 5000. Thanks to the remote port forwarding, SSH forwards the data to the local port 5000. From there, it is relayed back to the destination, as described in the previous step.

$ ssh -R 127.0.0.1:5000:127.0.0.1:5000 web04
$ rsync -e 'sh -c "socat stdio tcp-connect:127.0.0.1:5000"' -aLv /data/. remote:/data/.
sending incremental file listhaproxy.debian.net/haproxy.debian.net/dists/buster-backports-1.8/Contents-amd64.bz2haproxy.debian.net/dists/buster-backports-1.8/Contents-i386.bz2[…]media.luffy.cx/videos/2021-frnog34-jerikan/progressive.mp4sent 921,719,453 bytes  received 26,939 bytes  7,229,383.47 bytes/sectotal size is 7,526,872,300  speedup is 8.17

This little diagram may help understand how everything fits together:

Diagram showing how all processes are connected together: rsync, socat and ssh
How each process is connected together. Arrows labeled “stdio” are implemented as two pipes connecting the process to the left to the standard input and output of the process to the right. Don't be fooled by the apparent symmetry!

The rsync manual page prohibits the use of --server . Use this hack at your own risk!

The options --server and --sender are used internally by rsync, and should never be typed by a user under normal circumstances. Some awareness of these options may be needed in certain scenarios, such as when setting up a login that can only run an rsync command. For instance, the support directory of the rsync distribution has an example script named rrsync (for restricted rsync) that can be used with a restricted ssh login.

Addendum

I was hoping to get something similar with a one-liner. But this does not work!

$ socat \>  exec:"ssh web04 rsync --server --sender -vlLogDtpre.iLsfxCIvu . /data/."\>  exec:"ssh web05 rsync --server -vlogDtpre.iLsfxCIvu /data/. /data/."\over-long vstring received (511 > 255)over-long vstring received (511 > 255)rsync error: requested action not supported (code 4) at compat.c(387) [sender=3.2.3]rsync error: requested action not supported (code 4) at compat.c(387) [Receiver=3.2.3]socat[878291] E waitpid(): child 878292 exited with status 4socat[878291] E waitpid(): child 878293 exited with status 4

  1. And SSH agent forwarding is dangerous . Don’t use it if you can. ↩︎


Značky: #Debian

  • Pl chevron_right

    Sal Mubarak 2078!

    pubsub.slavino.sk / planetdebian · Saturday, 6 November - 05:08

Diwali Fireworks

Wishing everyone in Debian good health and prosperity in the Gujarati New Year Vikram Samvat 2078 called Pramadi.


Značky: #Debian

  • Pl chevron_right

    25 things I would like to 3D print

    pubsub.slavino.sk / planetdebian · Friday, 5 November - 21:47 · 2 minutes

Last year I started collecting ideas of things I would like to 3D print one day, on Twitter. Twitter is fundamentally ephemeral, so I'll collect it here instead. I got up to 14 items on Twitter, and now I'm up to 25.

I don't own a 3D printer, but I have access to one at the work office. Perhaps this list is my subconcious trying to convince me to buy one.

What am I missing? What else should I be thinking of printing? Let me know!

  1. Some kind of 45° leaning prong to dry bottles and flasks on
  2. A tea tray and coasters
  3. a replacement prop arm/foot for my computer keyboard ( something like this but for the Lenovo Ultranav)
  4. some attempted representation of Borges’ Library of Babel, a la @jwz The Library of Babel, again
  5. an exploration of the geometry of Susanna Clarke’s “Piranesi”
  6. further iterations of my castle
  7. Small tins to keep loose-leaf tea in
  8. Who am I kidding, bound to be a map from DOOM. E1M1 perhaps, or something more regular ( MAP07 ? E2M8 ?)

    See also this amazing print of Quake 3: Arena's "Camping Grounds"

  9. replacement bits for kids toy sets, e.g. a bolt with long shank from Early Learning Centre Build It Deluxe Set , without all 4 of which you can't build much of anything

  10. A stand for a decorative Christmas bauble (kid's hand print on it)

    A roll of cellotape works pretty well in the mean time

  11. DIY bits-and-bobs sorter/storage (nuts and bolts etc)

  12. A space ship from Elite/Frontier. Probably a Cobra mk3 or maybe a Viper mk2 . In a glow-in-the-dark PLA which i’d overpaint with gunmetal except for the fuselage

  13. A watch stand/holder/storage thing

    Except it would look nicer in wood

    (And I’m more inclined to get rid of all but one watch instead)

  14. Little tabbed 7” dividers for vinyl records, with A-Z cut into the tabs

    12” ones might be a stretch

    ( something like this )

  15. A low-profile custom trackpoint cover like the ones by SaotoTech ( e.g. )

  16. A vinyl record. (Not sure that any 3D printer I would have access to would have the necessary resolution. I haven't done any research yet.)

  17. A free-standing inclined vinyl record display stand ( e.g. )

  18. A "Settlers of Catan" set. I've got the travel edition which is great but it would be nicer to have a larger-sized set. There are some things I really like about the travel set that the full-size set lacks; so designing and printing a larger set myself could incorporate them. Also I don't feel inclined to buy the full-size set for £50 or so to end up with essentially the same game I already bought. No doubt I'd spent at least that much in PLA.

  19. Little kids trinkets. Pacman ghosts, that sort-of thing. Whatever my daughters come up with next.

  20. Lego storage/sorters.

  21. Some kind of lenticular picture. Perhaps a gift or Christmas card combined in one.

  22. A bracket to install a Gotek drive in my Amiga 500 ( e.g. ). I've managed without but the fit isn't great.

  23. An attempt at using the 3D printer for 2D drawing. I would never get the same kind of quality results you can get from a proper plotter, but still… Take a look at some proper plotter art!

  24. Garden decorations. I like the idea of porous geometric shapes that you can plant mosses or ferns into, but also things which might be taken over and "used" by nature in ways I hadn't thought of.

  25. Floor plans / 3D plans of my house (including variations if I remodelled)


Značky: #Debian

  • Pl chevron_right

    Tea(GL) Sandwiches

    pubsub.slavino.sk / planetdebian · Friday, 5 November - 21:41 · 1 minute

For SeaGL 2021 I’m making some tea sandwiches. I know nothing about tea sandwiches, but read a Wikipedia article . If you want to make them along with me, I recommend getting the ingredients ahead of time.

Smoked Not-salmon Sandwiches

I guess you could make these with smoked salmon, though I think it’s cleverer this way.

Ingredients

  • carrots
  • marinade (I used soy sauce, sesame oil, and a sheet of nori. If you don’t have a smoke infuser, I recommend also adding liquid smoke, but I like that flavor. I also think it’d be good with horseradish, but that’s a different conversation.)
  • Cream cheese (vegan or otherwise) – room temperature
  • Dill + other herbs
  • Bread

Procedure

  • THE DAY BEFORE : Boil the carrot(s) for a few minutes, until just barely soft. Pour out the hot water and in some cold water. This will help it to cool down quickly. Also, when something boils (or cooks in general) it continues to cook for a bit after you take it off the heat. This will stop that.
  • Slice the carrots thinly
  • Put the carrots into a jar with the marinade ingredients. Seal and stick in the fridge.
  • THE DAY OF: Take the cream cheese out before hand so it gets soft.
  • Take out some carrots and pat dry.
  • If you’re using a smoke infuser, smoke infuse the carrots. We’ll see how this goes. I am very skeptical it will work.
  • Finely chop herbs and add to the soft cream cheese. Mix it up. Yum.
  • Take two slices of bread. Spread a thin layer of cream cheese on either side. This will help everything stay together.
  • Put a layer of carrots on a piece of bread. Put the other piece on top. Viola!

Apple and Brie Sandwiches

This sounded good to me. Let’s see how it goes.

Ingredients

  • Bread
  • Apple
  • Brief
  • Butter (soft, optional)

Procedure

  • Slice the apple thinly
  • Slice the brie thinly
  • Take a piece of bread and stack on it: brie, apple, brie.
  • Top with another slice of bread.
  • Optional: Butter the OUTSIDE of the sandwich. Put it in a pan, griddle, or panini press. Grill it a bit. I think the cheese should soften, but the apple will stay pretty crunchy.
  • Slice elegantly.

Značky: #Debian

  • Pl chevron_right

    Mastodon again

    pubsub.slavino.sk / planetdebian · Friday, 5 November - 16:18 · 1 minute

I've been an occasional Mastodon user for (apparently) four years. I experimented with running my own instance for a while before giving that up, then I shopped around for an interesting little instance and settled on one, which was lovely until the maintainer decided (quite reasonably) it wasn't worth the trouble any more and turned it off.

So I'm in need of a new instance! The first account I had was @Jmtd@mastodon.cloud , but that instance was too busy for my liking. I'd like to find something with small but vibrant local traffic, ideally matching to my interests. In fact one of the problems with my last instance 1 was it was too quiet, and the local timeline was mostly empty. So something between "empty" and "overwhelmingly busy" as mastodon.cloud is, or was. Recommendations welcome!

I'd set @jmtd@mastodon.cloud to forward to my instance for most of the last four years. I've turned that off now, so I am (at least temporarily) back to that instance. That's got me thinking: perhaps it's less onerous to run a local instance that only forwards, and to use that as a sort-of "permalink", allowing me to migrate from real instance-to-instance when required, but giving me a Mastodon "perma-link". Something to look at.

(It's also worth considering that you should back up the list of accounts you are following off whichever instance you are on. I'm starting again with almost a clean slate.)


  1. I'm deliberately omitting the name of my last instance to give the former maintainer a break.

Značky: #Debian

  • Pl chevron_right

    diffoscope 190 released

    pubsub.slavino.sk / planetdebian · Friday, 5 November - 00:00

The diffoscope maintainers are pleased to announce the release of diffoscope version 190 . This version includes the following changes:

[ Chris Lamb ]
* Don't raise a traceback if we cannot de-marshal Python bytecode to support
  Python 3.7 loading newer .pyc files.
  (Closes: reproducible-builds/diffoscope#284)
* Fix Python tests under Python 3.7 with file 5.39+.

[ Vagrant Cascadian ]
* Skip Python bytecode testing when "file" is older than 5.39.

[ Roland Clobus ]
* Detect whether the GNU_BUILD_ID field has been modified.

You find out more by visiting the project homepage .


Značky: #Debian

  • Pl chevron_right

    Call for Translations: Ayatana Indicators 0.9.x Release Series

    pubsub.slavino.sk / planetdebian · Thursday, 4 November - 14:38 · 1 minute

We (Robert Tari, the UBports developers team, myself) are very close to releasing Ayatana Indicators 0.9.x. The work on Ayatana Indicators is currently nearly completed funded by the UBports Foundation and over the past half year, many many changes, improvements and clean-ups have been added to the code.

Ayatana Indicators 0.9.x will be the first release series to be in the development tree of Ubuntu Touch 20.04 (which is currently under very heavy development).

Ayatana Indicators 0.9.x will also be used in various other desktop environments available in upcoming Ubuntu 22.04 LTS, such as Ubuntu MATE, Xubuntu, (optionally in) Ubuntu Budgie (please correct my wording, if you know better), (send me a note, if I forgot your desktop env), etc.

So, to all Ubuntu Touch, Ubuntu MATE, Xubuntu, etc. users. If you not already are a translator of Ayatana Indicators and you are good in English and fluent in at least one other language, please consider helping out with translating or improving translations of Ayatana Indicators.

The translation work needs to be done on Hosted Weblate [1], please sign up for an account (if you haven't done so, yet) and chime in.

Thanks so much for your contributions!

light+love
Mike

https://hosted.weblate.org/projects/ayatana-indicators/


Značky: #Debian

  • Pl chevron_right

    Python: send emails with embedded images

    pubsub.slavino.sk / planetdebian · Thursday, 4 November - 13:45 · 1 minute

to send emails with images you need to use MIMEMultipart , but the basic approach:

import smtplib

from email.mime.multipart import MIMEMultipart
from email.mime.image import MIMEImage

msg = MIMEMultipart('alternative')
msg['Subject'] = "subject"
msg['From'] = from_addr
msg['To'] = to_addr

part = MIMEImage(open('/path/to/image', 'rb').read())

s = smtplib.SMTP('localhost')
s.sendmail(from_addr, to_addr, msg.as_string())
s.quit()

will produce an email with empty body and the image as an attachment.

The better way, ie to have the image as part of the body of the email, requires to write an HTML body that refers to that image:

import smtplib

from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.image import MIMEImage

msg = MIMEMultipart('alternative')
msg['Subject'] = "subject
msg['From'] = from_addr
msg['To'] = to_addr

text = MIMEText('<img src="cid:image1">', 'html')
msg.attach(text)

image = MIMEImage(open('/path/to/image', 'rb').read())

# Define the image's ID as referenced in the HTML body above
image.add_header('Content-ID', '<image1>')
msg.attach(image)

s = smtplib.SMTP('localhost')
s.sendmail(from_addr, to_addr, msg.as_string())
s.quit()

The trick is to define an image with a specific Content-ID and make that the only item in an HTML body: now you have an email with contains that specific image as the only content of the body, embedded in it.

Bonus point : if you want to take a snapshot of a webpage (which is kinda the reason i needed the code above) i found it extremely useful to use the Google PageSpeed Insights API ; a good description on how to use that API with Python is available at this StackOverflow answer.

UPDATE (2020-12-26): I was made aware via email that some mail providers may not display images inline when the Content-ID value is too short (say, for example, Content-ID: 1 ). A solution that seems to work on most of the providers is using a sequence of random chars prefixed with a dot and suffixed with a valid mail domain.

Značky: #Debian