Low Bandwidth Matrix: An implementation guide

2021-06-10T17:08:54+00:00

Disclaimer: Low bandwidth Matrix is experimental, not yet standardised, and subject to change without notice.

🔗</a>Setting up a low bandwidth proxy for your homeserver</h2>
Prerequisites:

Go 1.13+</li>
`openssl</code> to generate a self-signed DTLS certificate, or an existing certificate you want to use.</li>`
`Linux or Mac user</li> </ul> Steps:`

Clone the repo: `git clone https://github.com/matrix-org/lb.git</code> </li>`
`Build the low bandwidth proxy: go build ./cmd/proxy</code> </li>`
`Generate a elliptic curve DTLS key/certificate: (we use curve keys as they are smaller than RSA keys, but both work.)`
`openssl ecparam -name prime256v1 -genkey -noout -out private-key.pem openssl req -new -x509 -key private-key.pem -out cert.pem -days 365 # you now have cert.pem and private-key.pem </code></pre> </li>`

Run it pointing at matrix.org:
`./proxy -local 'https://matrix-client.matrix.org' \ --tls-cert cert.pem --tls-key private-key.pem \ --advertise "http://127.0.0.1:8008" \ --dtls-bind-addr :8008 </code></pre> </li>`

You should see something like this:
INFO[0000] Listening on :8008/tcp to reverse proxy from http://127.0.0.1:8008 to https://matrix-client.matrix.org - HTTPS enabled: false INFO[0000] Listening for DTLS on :8008 - ACK piggyback period: 5s </code></pre> </li> </ul> Mac users: If you are having trouble generating EC certificates, make sure you are using OpenSSL and not LibreSSL which comes by default: openssl version</code>. To use OpenSSL, brew install openssl</code> which then dumps the binary to /usr/local/opt/openssl/bin/openssl</code>. To test it is working correctly: # build command line tools we can use to act as a low bandwidth client go build ./cmd/jc go build ./cmd/coap # do a CoAP GET request to matrix.org via the proxy ./coap -X GET -k 'http://localhost:8008/_matrix/client/versions' | ./jc -c2j '-' {"unstable_features":{"io.element.e2ee_forced.private":false,"io.element.e2ee_forced.public":false,"io.element.e2ee_forced.trusted_private":false,"org.matrix.e2e_cross_signing":true,"org.matrix.label_based_filtering":true,"org.matrix.msc2432":true,"org.matrix.msc3026.busy_presence":false,"uk.half-shot.msc2666":true},"versions":["r0.0.1","r0.1.0","r0.2.0","r0.3.0","r0.4.0","r0.5.0","r0.6.0"]} </code></pre> If this doesn't work: Check the proxy logs for errors (e.g bad hostname)</li> Try adding -v</code> to ./coap</code> (e.g bad method or path)</li> Run the proxy with SSLKEYLOGFILE=ssl.log</code> and inspect the decrypted traffic using Wireshark.</li> </ul> Otherwise, congratulations! You now have a low bandwidth proxy! You can connect to your proxy just like you would to matrix.org or any other homeserver. 🔗</a>Security considerations</h3> The proxy acts as a man in the middle and can read all non-E2EE traffic, including login credentials. DO NOT USE UNTRUSTED LOW BANDWIDTH PROXY SERVERS. Only use proxy servers run by yourself or the homeserver admins.</li> </ul> 🔗</a>Further reading</h3> The proxy</code> README</a></li> coap</code> README</a> and jc</code> README</a></li> </ul> 🔗</a>Setting up a custom Element Android</h2> We'll add low bandwidth matrix to Element Android and iOS by default once it's standardised - but while things are still experimental, here's a guide for how to build Element Android to do it yourself if you feel the urge. This can be used as inspiration for other Matrix clients too. Prerequisites: Android Studio</li> </ul> Steps: Clone the repo: git clone https://github.com/vector-im/element-android.git</code> </li> Checkout kegan/lb</code>: git checkout kegan/lb</code>. This branch replaces all HTTP traffic going to /_matrix/client/*</code> with LB traffic. /_matrix/media</code> traffic is left untouched. This branch also disables TLS checks entirely so self-signed certificates will work. </li> Clone the low bandwidth repo if you haven't already: git clone https://github.com/matrix-org/lb.git</code> </li> In the low bandwidth repo, build the mobile bindings: go get golang.org/x/mobile/cmd/gomobile cd mobile # if gomobile isn't on your path, then ~/go/bin/gomobile gomobile bind -target=android </code></pre> </li> Copy the output files to a directory in the Element Android repo which Gradle will pick up: mkdir $PATH_TO_ELEMENT_ANDROID_REPO/matrix-sdk-android/libs cp mobile-sources.jar $PATH_TO_ELEMENT_ANDROID_REPO/matrix-sdk-android/libs cp mobile.aar $PATH_TO_ELEMENT_ANDROID_REPO/matrix-sdk-android/libs </code></pre> </li> Open the project in Android Studio. </li> Build and run on a device/emulator. </li> Configure the proxy's --advertise</code> address. If you are running on a local device, restart the proxy with an --advertise</code> of your machines LAN IP e.g 192.168.1.2 instead of 127.0.0.1. If you are running on an emulator, restart the proxy with an --advertise</code> of the host IP</a>: 10.0.2.2. The URL scheme should be https</code> not http</code>, else image loading won't work as Element Android won't download media over http</code>. </li> Login to your matrix.org account via the proxy with the --advertise</code> address as the HS URL e.g https://192.168.1.2:8008</code> or https://10.0.2.2:8008</code>. The port is important. </li> </ul> To verify it is running via low bandwidth: Install Wireshark.</li> Restart the proxy with the environment variable SSLKEYLOGFILE=ssl.log</code>.</li> Run tcpdump on the right interface e.g: sudo tcpdump -i en0 -s 0 -v port 8008 -w lb.pcap</code></li> Force stop the android app to forcibly close any existing DTLS connections.</li> Re-open the app.</li> Open lb.pcap</code> in Wireshark and set ssl.log</code> as the Pre-Master Secret log filename via Preferences -> Protocols -> TLS -> Pre-Master Secret log filename.</li> Check there is DTLS/CoAP traffic.</li> </ul> 🔗</a>Performance</h2> To send a single 'Hello World' message to /room/$room_id/send/m.room.message/$txn_id</code> and receive the response, including connection setup: Protocol</th> Num packets</th> Total bytes</th></tr></thead> HTTP2+JSON</td> 43</td> 6533</td></tr> CoAP+CBOR</td> 6</td> 1440</td></tr> </tbody></table> 🔗</a>Limitations</h2> CoAP OBSERVE</a> is not enabled by default. This extension allows the server to push data to the client so the client doesn't need to long-poll. It is not yet enabled because of the risk of state synchronisation issues between the proxy and the client. If the proxy gets restarted, the client will not receive sync updates until it refreshes its subscription, which happens infrequently. During this time the client is not aware that anything is wrong.</li> CoAP uses Blockwise Transfer</a> to download large responses. Each block must be ACKed before the next block can be sent. This is less efficient than TCP which has a Receive Window which allows multiple in-flight packets at once. This means CoAP is worse at downloading large responses, requiring more round trips to completely send the data.</li> The current version of /sync</code> sends back much more data than is strictly necessary. This means the initial sync can be slower than expected. On a low kbps link this can flood the network with so much data that the sync stream begins to fall behind. Future work will look to optimise the sync API.</li> The proxy currently doesn't implement the low bandwidth response</a> in /versions</code>.</li> </ul> Usage of matrix-nio (Python Sans IO) 2019-07-03T00:00:00+00:00 Canonical version of this article at https://matrix.org/docs/guides/usage-of-matrix-nio</a> This article concerns matrix-nio</a>, and asyncio</a>. We'll build a simple "echo bot", meaning a bot which replies to messages with the text it has just read. Note that this article does not cover E2EE with matrix-nio. 🔗</a>Instantiation and Login</h2> First create a new venv, and install matrix-nio via pip</code>. On the command line, run: python3 -m venv env source env/bin/activate pip install matrix-nio </code></pre> Next, create a new Python file, and open it for editing. We'll import everything we require for this tutorial: from importlib import util import asyncio from nio import (AsyncClient, SyncResponse, RoomMessageText) </code></pre> We're importing asyncio so we can use the AsyncClient</code> class from matrix-nio. Create a new instance of AsyncClient</code> by passing the homeserver and username as arguments: async_client = AsyncClient( "https://matrix.org", "%%YOUR-USERNAME-HERE%%" ) </code></pre> Then login, and await the response: response = await async_client.login("%%YOUR-PASSWORD-HERE%%") print(response) </code></pre> Of course, we are using an async client, and awaiting the response. Because of this, we must call the async_client.login()</code> from an async method, like so: async def main(): response = await async_client.login("%%YOUR-PASSWORD-HERE%%") print(response) asyncio.run(main()) </code></pre> Note that for versions of Python before 3.7 the asyncio syntax must be: async def main(): response = await async_client.login("%%YOUR-PASSWORD-HERE%%") print(response) loop = asyncio.get_event_loop() loop.run_until_complete(main()) </code></pre> The remainder of this tutorial assumes you are running everything from an async</code> method. The response string should look like: Logged in as @pyconweb-bot:matrix.org, device id: ZBLAJHLKVP. </code></pre> 🔗</a>Get into a /sync</code> loop</h2> To get updates from a Matrix homeserver to the client, the client makes a request to the /sync</code> endpoint. In the matrix-nio AsyncClient, this is wrapped by the sync()</code> method. We can get the latest updates: sync_response = await async_client.sync(30000) </code></pre> 30000 means we will wait up to 30 seconds before returning. sync_response</code> will now contain a Python object containing a mapping of the (JSON) response from the Matrix homeserver</a>. We'll inspect this response in the next section. In fact, we expect there to be updates regularly, so let's create a very simple loop: while (True): sync_response = await async_client.sync(30000) print(sync_response) # note that this could be LARGE! # do some reading from sync_response </code></pre> In this way, every time there is a response (i.e. new events) from the homeserver, they are made available in sync_response</code> for processing, and we loop again. 🔗</a>Explore the sync response object</h2> sync_response</code> can contain multitudes</a>, depending on the rooms this user is part of, or has been part of. sync_response.rooms.join</code> contains updates for the rooms which the current user is "joined to" (meaning, is a member of.) Of these joined rooms, we are (perhaps!) most interested in the events on the timeline. These are stored in timeline.events</code>, see below: if len(sync_response.rooms.join) > 0: joins = sync_response.rooms.join for room_id in joins: for event in joins[room_id].timeline.events: print(event) </code></pre> Message events are a specific type of event which contain an Instant Messenger message. We can check the type before proceeding: for event in joins[room_id].timeline.events: if isinstance(event, RoomMessageText): print (event.body) </code></pre> In these cases, where the event is a message to a room, the body</code> field will contain the message text. 🔗</a>Isolate specific message event objects</h2> Knowing that we can get the message text from an event, we can read it to determine a response. Let's make a new variable and have it store some string we'll check for: response_string = "!replybot" </code></pre> Now let's suppose we're in our /sync</code> loop, and just received an event. We can filter messages that are meant for our bot as follows: if len(sync_response.rooms.join) > 0: joins = sync_response.rooms.join for room_id in joins: for event in joins[room_id].timeline.events: if hasattr(event, 'body') and event.body.startswith(response_string): print(event) </code></pre> 🔗</a>Use room_send</code></h2> To send messages, matrix-nio provides a room_send()</code> method. There are three arguments: the room_id</li> the message type, we will use "m.room.message"</li> a JSON object representing the content of the message</li> </ul> Let's improve the example above, by sending back a message to echo the ones we isolated above: joins = sync_response.rooms.join for room_id in joins: for event in joins[room_id].timeline.events: if hasattr(event, 'body') and event.body.startswith(response_string): response_body = event.body.replace(response_string, "").strip() content = { "body": response_body, "msgtype": "m.text" } await async_client.room_send(room_id, 'm.room.message', content) </code></pre> Now whenever the bot receives a message "!replybot some message" it will send back "some message". 🔗</a>Use of /sync next_batch tokens</h2> Finally, let's consider the importance of next_batch</code> tokens. Whenever you receive a response from the /sync</code> endpoint, the response will contain a "next_batch" field, which you then pass on the next request to ensure you have the latest messages. matrix-nio keeps track of this automatically, so it doesn't get repeated messages. However, when you stop the program and call the .sync()</code> method again, how can you tell it where to start from? First let's get the latest next_batch</code> token: async def main(): response = await async_client.login("%%YOUR-USERNAME-HERE%%", "") while (True): sync_response = await async_client.sync(30000) print(sync_response.next_batch) # this is the token </code></pre> Then we'll write the token to a file: async def main(): response = await async_client.login("%%YOUR-USERNAME-HERE%%", "") while (True): sync_response = await async_client.sync(30000) # we write the token to a file here with open("next_batch","w") as next_batch_token: next_batch_token.write(sync_response.next_batch) </code></pre> Once that token is written, we know we can re-use it for the first /sync/</code> request next time: async def main(): response = await async_client.login("%%YOUR-USERNAME-HERE%%", "") # we read the previously-written token... with open ("next_batch","r") as next_batch_token: # ... and well async_client to use it async_client.next_batch = next_batch_token.read() while (True): sync_response = await async_client.sync(30000) with open("next_batch","w") as next_batch_token: next_batch_token.write(sync_response.next_batch) </code></pre> 🔗</a>Conclusion</h2> With this, you can see that in very few lines, it's possible to write a working Matrix bot in Python, using matrix-nio</a>. Bridging Matrix with WhatsApp running on a VM 2019-02-26T00:00:00+00:00 This guide will live with the documentation at https://matrix.org/docs/guides/whatsapp-bridging-mautrix-whatsapp</a>, but you can find the text below. Matrix is: an open standard for interoperable, decentralised, real-time communication. </blockquote> In this article we'll benefit from all three of these attributes: interoperable: we'll see how Matrix can be made to interact with WhatsApp</li> decentralised: you can perform this on your own server while still enjoying the benefits of being connected to the rest of the Matrix federation</li> real-time communication: we'll see how to send and receive messages in real-time</li> </ul> 🔗</a>Install your homeserver and install mautrix-whatsapp, the WhatsApp bridge</h2> Firstly, you need to have a Matrix homeserver installed. If you don't currently have one, take a look at the instructions at Installing Synapse</a>, and also in the Synapse README</a>. Next, install mautrix-whatsapp</a> by following the instructions at mautrix-whatsapp/wiki</a>. If you are starting from scratch, I suggest you take a look at matrix-docker-ansible-deploy</a>, as this project will enable you to deploy Synapse, mautrix-whatsapp</a> and other components easily. For example, if you have an existing deployment using matrix-docker-ansible-deploy</a>, you can add mautrix-whatsapp</a> simply by adding the following line to your configuration file: matrix_mautrix_whatsapp_enabled: true </code></pre> ... and re-running the setup: ansible-playbook -i inventory/hosts setup.yml --tags=setup-all </code></pre> Either way, you will soon have a functioning Matrix Synapse homeserver and mautrix-whatsapp</a> installed with it. Next, we will set up an Android VM. 🔗</a>Set up an Android VM</h2> The best way to run an Android Virtual Machine is to use the Android Studio tools from Google. First, install Android Studio</a>, making sure to follow the post-install steps, as they will install additional tools we need, including AVD Manager. Once installed, run AVD manager by choosing Tools -> AVD Manager</code> from the menu. Follow the steps to create a new virtual machine, in this example I have a Nexus 5X running Android 9, but almost any configuration is fine here. Make sure that you give the device access to the Play Store. 🔗</a>Install WhatsApp and sign-in</h2> Launch the Virtual Device, the open the Play Store and sign in. Now use the Play Store to install WhatsApp on the Virtual Device. You will be asked to verify your phone number, use your number on another device to complete this step. </center> 🔗</a>Setup mautrix-whatsapp bridge</h2> Now that you have WhatsApp working in a VM, and Matrix working on your server, it's time to bridge them together! Per the instructions at mautrix-whatsapp/wiki</a>, you must start a new chat with @whatsappbot:<yourdomain>. Type login</code> to begin the authentication process. mautrix-whatsapp operates by using the WhatsApp Web feature of WhatsApp - which means it uses a QR code that you must now scan on the device running WhatsApp - which in your case is the AVD. In order to scan the presented QR code, set your AVD camera to passthrough the camera device on your host machine - see the images below. Once this is complete, you can type sync</code>, to start bridging contacts, and sync --create</code> to automatically create room invites. And that's it! You may need to take a little time to watch the sync happen, particularly if you have a very large number of chats on the WhatsApp side, but there is no further configuration needed. 🔗</a>Demo!</h2>

Matrix.org - Tutorials

Low Bandwidth Matrix: An implementation guide

Usage of matrix-nio (Python Sans IO)

Bridging Matrix with WhatsApp running on a VM