Penn Mackintosh

Make it fast, then make it right

Fri, 30 Oct 2026 10:30:00 +0000

Abstract

I challenge the common wisdom of “make it work, make it right, make it fast”. Sometimes, you have to make it fast, then make it work. When working on machine learning, you have to process huge amounts of data, and code errors are almost impossible to debug or even detect without analysing that data at scale. As with any software engineering task, bugs are fixed through repeated cycles of finding and fixing a bug, then moving on to the next one. If you write slow code, each cycle of bug-fixing (“make it right”) takes a long time. But if you first focus on the performance of the program (“make it fast”) and then start fixing bugs, the amount of time spent in the bug fixing phase could be lower. I present an anecdote with absolutely no statistical evidence whatsoever.

Introduction

I couldn’t possibly write a blog post today without mentioning LLMs in some way.

In today’s software engineering culture there seems to be a huge focus on churning out code as fast as possible. That means less review of the code, and perhaps more importantly, less thoughtful structuring. In all the “slop” codebases I’ve worked on, a common pattern is that the same task will be solved in three different ways in three different places. LLMs do love tricolon, after all. More different places solving the same problem means more chances that one of them has a bug, or worse, that you don’t even remember to fix the bug in all three at once! If you’re writing a data-intensive pipeline, with this less-structured code also comes less thought about the flow of data through the pipeline. While there is a good chance that the LLM approximates an existing efficient pipeline structure from training data, that’s unlikely to remain the case through many agent debugging cycles. Of course, the agent can detect when things are running slowly and try to optimise it, but that usually involves weird compiler-level hacks rather than thoughtful structure for efficiency. The end result is that by the time you, the human, need to debug the code, it’s a nonsensical mess that runs slowly. Much the same as if you had a team of undergraduates writing it, which is how I got into this situation.

For a coursework, we were asked to write a deep reinforcement learning agent. Not a particularly difficult task in general, but we chose an unusual problem space so we had a fair bit of reward shaping to do, and we weren’t allowed to use RL libraries. At the point where we were beginning to debug our broken code, I started trying to optimise our code, for example by using numpy functions rather than pure-python maths, and by batching operations together.

One team member said there was no point optimising until we finished fixing the bugs, and that’s what I’m refuting.

My hypothesis

For us to fix the bugs, we needed to run the full pipeline and check what was happening at each stage (often by looking at the standard deviation of various inputs and outputs to see if data was getting lost or rescaled). Then we needed to tweak each section, either by fixing a bug or tuning a hyperparameter.

I think that if we had not spent time optimising the pipeline early on, we would have taken much longer to debug it. Of course, there were overheads introduced by the optimisation, for instance when a numpy method gives an error it can often be much more obscure than the corresponding pure-Python messages. However, I suspect that it was worth it.

That’s all

I don’t think this post really deserves a conclusion since I didn’t do any research. I suppose you could call it inconclusive.

Fixing downloads in DecisionTime

Sat, 09 May 2026 14:00:00 +0000

Summary

When I try to download files from Decision Time, it just downloads corrupt docx files. I am not sure whether this is user error (probably by the document uploader) or a system error, but I fixed it.

Issue

When downloading a .docx file from Decision Time as a meeting attendee, the file is corrupt and cannot be opened (or opens with warnings). The document can still be viewed in the web interface. This means that people who rely on a screen reader cannot view the file with their preferred software, and also prevents sharing the documents with others.

Other document types such as PDF are also affected but PDF viewers ignore the leading newline, so it does not actually show an error.

Solution

I investigated the file and found that a newline character is erroneously inserted at the start of the file. I have created a Firefox extension to fix it by automatically removing the extra newline when you download the file (if you downloaded a corrupt copy before, you will need to download it again before it works).

If you do not want to use the extension, follow these steps:

Download the file.
Rename it to end in .txt.
Open it in a plain text editor such as Notepad.
The file will start with a blank line, then the letters PK, then lots of nonsense characters. Remove the blank line so that the file starts immediately with PK.
Save your changes and rename the file to end with .docx again.

Jupyter as Markdown for screen readers

Thu, 19 Feb 2026 00:40:00 +0000

Abstract

You can edit a Jupyter notebook as markdown without ever leaving text mode. This can work around accessibility issues or be used on non-graphical machines.

Introduction

Trying to use Orca with Jupyter is nigh on impossible. It doesn’t even let you open a file without using the mouse, and mouse routing buttons don’t seem to work for me. The same applies to JupyterLab, and I’ve heard similar stories for them under JAWS.

A lot of great work was started to improve this¹, but at the moment I wasn’t able to make any headway using it, and I couldn’t find any sign of ongoing work with a quick search.

Using VSCodium may be an alternative but didn’t work for me. I heard it was heavy and slow to navigate, but your mileage may vary.

I wanted to find a way to use and author Jupyter notebooks using plain text, to make it accessible to screen reader users. This is likely to be less accessible to many people, but by introducing more options, more people will find something that suits them.

I found a lot of tooling for Jupyter² which led to the discovery of Bookbook, a tool that sounded cool but one-way and unmaintained. JupyText is a very cool two-way converter, but it doesn’t seem to include cell outputs by default so would preclude easy access to results. I also found Notedown and its de facto successor nbconvert.

Solution

This covers the end-to-end workflow of downloading and modifying a notebook from a JupyterLab instance, using nbconvert and pandoc.

Install dependencies
Download all the files from the JupyterLab server
Convert notebook to markdown
Edit and convert back again to execute
Iterate as usual

Dependency installation

Downloading notebooks

Update: I have written some JavaScript that automatically downloads a zip file from a JupyterLab instance. To use it, open the JupyterLab instance and press F12 to open developer tools. Press Escape to open the Console, then paste the code. After about 2 minutes, it will download a zip file which contains the entire notebook so that you can use it locally in your preferred software. I have only tested this in Firefox so your mileage may vary. If it doesn’t work, try the old annoying way below.

(() => {
  document.evaluate('//*[@class="jp-LauncherCard"][@title="Python (Pyodide)"][@data-category="Console"]', document, null, XPathResult.ANY_UNORDERED_NODE_TYPE, null).singleNodeValue.click();
  console.log("created console");
  const d = new DataTransfer();
  const f = "Jupyter "+new Date().toISOString()+".zip";
  const t = 'import zipfile, os\n'                      +
             'zf = zipfile.ZipFile("'+f+'", "w")\n'   +
             'for dp, _, fns in os.walk("."):\n'        +
             '    for fn in fns:\n'                     +
             '        if fn == "'+f+'":\n'            +
             '            continue\n'                   +
             '        zf.write(os.path.join(dp, fn))\n' +
             'zf.close()'                               ;
  d.setData("text/plain", t);
  const i = setInterval((() => {
    if (!document.activeElement.classList.contains("cm-content")) {
      return;
    }
    document.activeElement.dispatchEvent(new ClipboardEvent("paste", { clipboardData: d, dataType: "text/plain", data: t }));
    console.log("pasted script");
    document.getElementsByClassName("jp-CodeConsole-promptCell")[0].getElementsByClassName("cm-content")[0].dispatchEvent(new FocusEvent("focus"));
    clearInterval(i);
    console.log("focused input");
    const j = setInterval((() => {
      const e = document.evaluate('//*[@data-command="console:run-forced"]', document, null, XPathResult.ANY_UNORDERED_NODE_TYPE, null).singleNodeValue;
      if (e === null) {
        return;
      }
      if ("disabled" in e.attributes) {
        return;
      }
      if (!document.getElementsByClassName("jp-CodeConsole-promptCell")[0].getElementsByClassName("cm-content")[0].innerText) {
        return;
      }
      e.dispatchEvent(new MouseEvent("mousedown", { bubbles: true, cancelable: true, view: window }));
      e.dispatchEvent(new MouseEvent("mouseup", { bubbles: true, cancelable: true, view: window }));
      clearInterval(j);
      console.log("clicked run");
      const k = setInterval((() => {
        const s = document.evaluate('//*[contains(concat(" ",normalize-space(@class)," ")," jp-InputPrompt ")][text()="[*]:"]', document, null, XPathResult.ANY_UNORDERED_NODE_TYPE, null).singleNodeValue;
        if (s !== null) {
          console.log("waiting for execution");
          return;
        }
        const e = document.evaluate('//*[@class="jp-DirListing-itemText"]//*[text()="'+f+'"]', document, null, XPathResult.ANY_UNORDERED_NODE_TYPE, null).singleNodeValue;
        if (e === null) {
          return;
        }
        const e2 = e.closest(".jp-DirListing-item");
        e2.dispatchEvent(new PointerEvent("mousedown", { bubbles: true, cancelable: true }));
        e2.dispatchEvent(new PointerEvent("mouseup", { bubbles: true, cancelable: true }));
        e.dispatchEvent(new PointerEvent("contextmenu", { bubbles: true, cancelable: true }));
        clearInterval(k);
        console.log("right-clicked file");
        const l = setInterval((() => {
          const e = document.evaluate('//*[@data-command="filebrowser:download"]', document, null, XPathResult.ANY_UNORDERED_NODE_TYPE, null).singleNodeValue;
          if (e === null) {
            return;
          }
          e.dispatchEvent(new FocusEvent("focus"));
          console.log("focused download button");
          e.dispatchEvent(new PointerEvent("mousedown", { bubbles: true, cancelable: true }));
          e.dispatchEvent(new PointerEvent("mouseup", { bubbles: true, cancelable: true }));
          console.log("clicked download button");
          clearInterval(l);
        }), 200);
      }), 200);
    }), 200);
  }), 200);
})();

The old annoying way to download the notebook

This code converts an online Jupyter notebook into a zip file that you can download. To run it, you need to launch a Python interpreter in the inaccessible web interface and paste this code, then press Shift+Enter. There will probably be no output that a screen reader can understand, but a file named “lab.zip” will appear in the “file explorer” in the web interface, which you can download by right-clicking and choosing Download.

import zipfile, os
zf = zipfile.ZipFile("lab.zip", "w")
for dp, _, fns in os.walk("."):
 for fn in fns:
   if fn == "lab.zip":
     continue
   zf.write(os.path.join(dp, fn))
zf.close()

Now decompress the files by double-clicking them (or using a terminal as below).

mkdir lab
cd lab
unzip ../lab.zip

Note that if you don’t entirely trust the files, you’ll need to remove the --execute in the next section. If you really don’t trust them at all, don’t download them!

Converting to markdown

The magic command is:

jupyter nbconvert --execute --allow-errors --inplace notebook.ipynb && pandoc --extract-media=images notebook.ipynb -o notebook.md

This will run the code (--execute) and save the readable file into notebook.md which you can edit using your favourite text editor. Images such as graphs will be saved into the images directory.

If you don’t entirely trust the code, remove the --execute flag (technically you can skip the entire first part of the pipeline but that’s harder to explain)

If you run into an error about NoSuchKernel³, try the following command, which will tell it to use the default kernel. This will fix the error but may change behaviour. You will still need to run the conversion command again afterwards.

jq -e '.metadata.kernelspec.name = "python"' notebook.ipynb > notebook.ipynb.tmp && mv notebook.ipynb.tmp notebook.ipynb

Converting back to Jupyter format

To test your code, first turn it back into a Jupyter notebook:

pandoc notebook.md -o notebook.ipynb

Then just turn it back into Markdown like before, and any errors will show up below the corresponding code (you can search .output .error to find them). You could also remove the --allow-errors flag to have the errors show up without updating the Markdown file, but I think this would be more confusing.

Limitations

The Jupyter web interfaces allow re-running individual blocks of code. I haven’t looked into this for nbconvert, but this feature can accelerate development, so finding a way to add it would be useful. One can imagine that only cells that have changed since the last run would be evaluated by default by a tool incorporating this feature.

The process described is quite clunky and requires considerable technical knowledge. Markdown is a fairly easy language to learn, so with the advent of less expensive multi-line Braille displays⁴ there is scope for a simple application that ties the whole workflow together into a simple editor with sensible keyboard shortcuts and search functionality.

Conclusion

This workflow can help with Jupyter usage for some people whose accessibility needs are not met by the existing solutions, but it’s not perfect. People who just love their machines in text-mode may prefer jut. People who want to commit plain-text versions of Jupyter files may fare better with Jupytext.

P.S.

Wondering why the writing style is so weird? Reading too many academic papers while trying to write casually at one in the morning don’t mix well.

All code in this post is licensed under CC BY 4.0

GnuPG in Thunderbird in Flatpak in Silverblue

Wed, 13 Aug 2025 11:29:00 +0000

Thunderbird’s already flaky GPG smartcard support falls down when using Flatpak. But fear not - I managed to get it working!

Summary

To get your Yubikey working on Fedora Silverblue with Thunderbird, follow these steps:

Fix hotplug¹:

$ cat >> ~/.gnupg/scdaemon.conf <<EOF
disable-ccid
pcsc-shared
EOF
$

Enable gpg-agent.socket²:

systemctl --user enable --now gpg-agent.socket

Grant Thunderbird access to the system gpg-agent instead of running its own agent:

flatpak override --user --socket=gpg-agent --nosocket=pcsc '--nofilesystem=~/.gnupg' net.thunderbird.Thunderbird
flatpak override --user '--filesystem=~/.gnupg:ro'

Fix the session DBus³⁴:

sudo tee /opt/pinentry.sh <<EOF
#!/bin/env -S --ignore-environment /bin/sh

# The shebang using env ensures that all (potentially malicious) environment variables are cleared

# Set the correct DBus session bus path - this is hardcoded because we don't trust the environment we receive if called from a sandboxed caller
export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/$(/usr/bin/id -ru)/bus"
# Set a placeholder display, otherwise the curses fallback is sometimes used.
# In particular this makes a difference if the caller doesn't specify --display.
# This might break if you use X11, but if you are, stop, and use Wayland instead.
export DISPLAY="__stub"

# And finally pass everything else through
exec /usr/bin/pinentry "$@"
EOF
sudo chown root:root /opt/pinentry.sh
sudo chmod 0755 /opt/pinentry.sh
sudo chcon -t pinentry_exec_t /opt/pinentry.sh
echo "pinentry-program /opt/pinentry.sh" >> ~/.gnupg/gpg-agent.conf

Reboot (I mean it!)
Follow the normal setup instructions for Thunderbird

Other useful steps you might want to take are found in the community guide.

Thunderbird debugging

I started off looking at the Thunderbird error log (Ctrl+Shift+J), but it was rather unhelpful:

mailnews.send: NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS: [JavaScript Error: "encryptMessageStart FAILED: -1" {file: "chrome://openpgp/content/modules/mimeEncrypt.sys.mjs" line: 455}]'[JavaScript Error: "encryptMessageStart FAILED: -1" {file: "chrome://openpgp/content/modules/mimeEncrypt.sys.mjs" line: 455}]' when calling method: [nsIMsgComposeSecure::finishCryptoEncapsulation]

Annoyingly that code was quite tricky to trace but I eventually found that it called into a library called GPGME.

GPGME debugging

Thankfully, there is pdocumentation on how to debug GPGME!

flatpak run --env=GPGME_DEBUG=9 net.thunderbird.Thunderbird

And suddenly I was able to see the actual error while trying to send signed mail!

2025-08-13 11:05:34 gpgme[2.2]     gpgme_op_keylist_next:1367: error: End of file <GPGME>\n
2025-08-13 11:05:34 gpgme[2.2]     gpgme_release: call: ctx=0x00007f633fdfeac0 
2025-08-13 11:05:34 gpgme[2.2]     gpgme_data_release: call: dh=0x0000000000000000 
2025-08-13 11:05:34 gpgme[2.2]     gpgme_data_release: call: dh=0x00007f633e165000 
2025-08-13 11:05:34 gpgme[2.2]   gpgme_get_key:1495: error: End of file <GPGME>\n

Okay, not a super helpful error. But scrolling up a bit (a lot, these logs are super verbose), I found something more useful:

2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: gpg: error running '/usr/bin/gpg-agent': exit st
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: atus 2<LF>
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: gpg: failed to start gpg-agent '/usr/bin/gpg-age
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: nt': General error<LF>
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: gpg: can't connect to the gpg-agent: General err
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: or<LF>
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: gpg: keydb_search failed: No agent running<LF>
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: gpg: error reading key: No agent running<LF>

The GPG agent

So it looks like we’re struggling to connect to the GPG agent. Looking at the pid of the GPG agent from outside the flatpak sandbox, I was able to confirm this:

$ pidof gpg-agent
$

Initially I decided to postpone fixing this issue and just manually start a GPG agent to see if we could make any other progress, so I just started an agent manually by calling a GPG command on the host.

$ gpg --card-status
<snip>
$ pidof gpg-agent
24067
$

Great, let’s try sending another email.

2025-08-13 11:08:52 gpgme[2.2]   gpgme_op_sign:522: error: pinentry error <Pinentry>\n

Pinentry inside the container

A new error! My first instinct here was that the sandbox was blocking us from requesting the PIN, so I decided to test it out inside the Flatpak:

$ flatpak enter net.thunderbird.Thunderbird bash
bash-5.2$ pinentry
OK Pleased to meet you
GETPIN
<snip>

Unfortunately, this opened a Curses interface.

bash-5.2$ pinentry --help
pinentry-curses (pinentry) 1.3.1-unknown

Wrong pinentry… but there’s a pinentry-gnome3 binary installed in the Flatpak.

bash-5.2$ pinentry-gnome3 --debug
No Gcr System Prompter available, falling back to curses
OK Pleased to meet you

It complained that there was something wrong with the connection to gcr, but wouldn’t say what.

At this point I went down a rabbithole and compiled my own version of pinentry-gnome3 with the fallback disabled, which showed me the actual error message. Basically it was complaining that a session-bus DBus call was failing (although it didn’t say what).

DBus allowlisting

Luckily, Flatpak has documentation on monitoring DBus traffic. Following this, I saw the following:

Filtering message due to arg0 org.gnome.keyring.SystemPrompter, policy: 0 (required 1)
C4: -> org.gnome.keyring.SystemPrompter call org.gnome.keyring.internal.Prompter.BeginPrompting at /org/gnome/keyring/Prompter

At this point, I tried granting org.gnome.keyring.* to the Flatpak to see if it would fix pinentry, and it did!

bash-5.2$ pinentry-gnome3 --debug
OK Pleased to meet you
GETPIN
D testing
OK
^C
bash-5.2$

This got me excited so I tried sending an email, but I still got the exact same error:

gpgme_op_sign:522: error: pinentry error <Pinentry>\n

This was most discouraging as it suggested I had fixed the wrong problem, especially as there was nothing in the DBus logs.

Pinentry logging

I decided to take a new approach and try calling gpg manually inside the flatpak container. To my surprise, this worked! And what’s more, after manually signing a message (which correctly prompted for my PIN), I was able to send emails through Thunderbird!

This was definitely progress but after restarting Thunderbird it was broken again. I decided to try to get access to the actual error returned by pinentry in real operation.

I created a bash wrapper script:

#!/bin/sh
echo "pinentry called with:" "$@" >> ~/pinentry.log
tee -a ~/pinentry.in.log | pinentry --debug 2>>~/pinentry.err.log | tee -a ~/pinentry.out.log

I configured it with pinentry-program in gpg-agent.conf. At this point after failing to send an email I had full logs from the pinentry.

Most interesting was the stderr output:

failed to connect to user session D-Bus (1): Could not connect: No such file or directory
Timeout: the Gcr system prompter was already in use.

This initially surprised me as I had been able to connect to the session bus in my manual tests of pinentry. In order to try to reproduce the issue, I decided to have my script print the entire env and /proc/$$/status into my log files.

I noticed that the PPid was one of two gpg-agent processes running on my system!

At this point, I wanted to ensure that there was only one gpg-agent, and I wanted it outside the sandbox so that I could use it from the console. A quick search online told me that it was a one-liner!

I did this and rebooted. I also noticed in Flatseal that I could enable gpg-agent socket sharing but that this was currently disabled. I enabled this and disabled the pcsc socket to avoid conflicts.

At this point I was still encountering the same error but at least I knew that the pinentry was meant to be running outside the sandbox, and the PPid confirmed that. However, the same error was still occurring.

I noticed in the logs that there was a DBus environment variable set:

DBUS_SESSION_BUS_ADDRESS=unix:path=/run/flatpak/bus

Containment confusion

This was most odd: we know that pinentry is not running in Flatpak, so it definitely can’t access a Flatpak DBus proxy.

At this point, I was pretty sure I had my culprit. After some extensive searching I found that gpg agent clients are allowed to send certain environment variables to pinentry using OPTION putenv.

Sadly, there doesn’t seem to be a way to disable putenv. It sounds like a sandbox escape waiting to happen. To test my hypothesis, I grabbed my unsandboxed DBus address:

$ echo $DBUS_SESSION_BUS_ADDRESS
unix:path=/run/user/1000/bus
$

and I entered this manually into my pinentry.sh wrapper. Lo and behold, I was able to send signed emails!

However, now that I knew about the putenv option, I realised that the wrapper would need some additional security measures to avoid vulnerabilities. For example, I wanted to use /run/user/$(id -ru)/bus as the session bus, but calling id could be subject to PATH Interception. To avoid this, I decided to use env -i:

#!/bin/env -S --ignore-environment /bin/sh

which invokes the shell script with an entirely clean environment.

Update: It turns out that putenv only lets you edit a restricted set of environment variables. Nevertheless, clearing out the environment if we don’t need it is a good idea for Defence in Depth.

Finally I dumped the script in /opt and configured it.

#!/bin/env -S --ignore-environment /bin/sh

# The shebang using env ensures that all (potentially malicious) environment variables are cleared

# Set the correct DBus session bus path - this is hardcoded because we don't trust the environment we receive if called from a sandboxed caller
export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/$(/usr/bin/id -ru)/bus"

# And finally pass everything else through
exec /usr/bin/pinentry "$@"

pinentry-program /opt/pinentry.sh

At this point, I had working emails. After some testing, it seems that this also fixed decryption for free.

This broke some other programs - turns out that the curses fallback is activated if DISPLAY is unset and --display is not passed. Setting DISPLAY=__stub fixed this.

Alternatives

I decided to run the GPG agent outside the sandbox. In theory, it could run inside the Flatpak (and that seems to be the design intended by the Thunderbird team), but this would come with some tradeoffs:

This means that pinentry would run inside the sandbox which means a compromised container could read your smartcard PIN, potentially allowing other credentials to be compromised.
On the other hand, running pinentry outside the sandbox increases the attack surface for a sandbox escape. This is arguably mitigated by removing access to pcscd which is no longer required.
Most importantly, running gpg-agent outside the sandbox means that ~/.gnupg can be exposed to the sandbox in read-only mode. When this is read-write, sandbox escape is trivial as it can just edit the configuration file to make the system-wide GPG agent invoke a malicious pinentry program.

I also wasn’t able to get the agent working reliably inside the sandbox, which is the main reason for running it outside.

Conclusion

At the start of this process I knew very little about GPG’s internal workings and now I feel like I do at least understand some of the PIN entry architecture and the purpose of the agent.

It also reminded me of some of my Flatpak knowledge that was getting rusty. I should probably have noticed faster that the pinentry call was in the wrong environment when it didn’t show up in the DBus logs during email sending attempts.

Thoughts on Golang

Sun, 20 Jul 2025 14:08:00 +0000

Over the last year, I’ve had the opportunity to work a lot with Golang code. It has certainly been an interesting experience, as I’ve never seriously used Go before.

Quick overview

Golang is a programming language created by Google. It’s fairly low-level and compiles to machine code, but is designed for general-purpose use and has automatic memory management.

Paradigm

Golang feels like a piece of software designed with system-driven design rather than user-centred design. The stated design goal is to keep the language simple. This has both positive and negative results.

Firstly, I disagree with the implied statement that a simple language is an imperative language. It’s possible to write a language that is no more complex than Golang which supports, at the very least, easy use of functional patterns mixed in with imperative code.

Tooling

The package manager

go.mod is great. For those who are unfamiliar with it, Golang will simply refuse to link any packages into your code unless they are listed in the go.mod residing in the root of your codebase. More languages should do this. Why? It prevents you accidentally introducting unwanted dependencies into your project.

go.sum, meanwhile. I’m not a big fan of. It lists the checksums of all packages in the go.mod file. This is great for security reasons as it provides Trust On First Use security for all your dependencies. This means that if an upstream source becomes compromised and retags a version tag to some malicious code, you’re safe. However, it’s quite frustrating to have to run a command to update it every time you modify your dependencies. It feels like this could be done automatically, and is only this way due to system-centred design. Of course you can call go get but it’s slow with non-obvious syntax for getting a specific version.

Golang also compiles all dependencies locally. This is good although it can feel frustrating when it takes a long time to compile. But it’s worth it to avoid the likes of the xz-utils backdoor. I have no complaints here.

Containerisation

There are various approaches to building Golang code into a container.

You can go with the simple approach of making a Dockerfile that calls go build, but then you end up needing to download the dependencies during every build process where a cache miss occurs. Getting the Dockerfile to build quickly is rather tricky as Docker doesn’t make explicit caching sharing very easy, especially if you build multiple containers from the same source (or with common dependencies).

Since a Golang project contains (by definition, in the go.sum) a hash of every dependency, the build process is quasi-reproducible¹. This means that we can safely bypass most of the complicated work that Docker does to protect us from external influences during build.

And that’s exactly what Ko does. You simply declare the arguments for the compiler and it handles turning it into a Docker image, basically by running go build on the host machine then copying into a container. It’s a very nice tool and speeds up the build process by an order of magnitude (in my case, over 3x speedup).

Linting

A wise man once told me “programmers should be lazy and stupid”². The logic is that a “lazy” programmer won’t over-complicate things, and a good programmer should write code that “stupid” people can understand. What with agentic LLMs writing code, I suspect that code authors are going to get less lazy and more stupid. Since perhaps the most useful input to an agentic system is the criticism, a good linter is essential. Some of the linters in Golang are fragmented, slow and often wrong, which makes it difficult to use AI for development, but also slows down human work.

thelper

A common mistake I’ve seen (in human- and AI-written code) is to use t.Helper() (from testing.T) in test helpers. T.Helper has very precise criteria for correct use³, but the thelper lint rule complains every time any function accepts a testing.T parameter. This leads people to think they are making a mistake by not including t.Helper() at the start of helper functions.

Test discovery

A common practice in Golang is to have dynamically-declared subtests (table-based testing). That’s a good thing in most languages, because it avoids code duplication. However, in Golang, the test names aren’t known until after you have started running them. In Java, for example, table-driven tests are brought up at class initialisation time which means they are discoverable. This means that Go subtests are a pain to discover⁴.

Memory management

Go uses a garbage collector for memory management, and as GCs go, it’s pretty good. It’s not very stop-the-world and can be tuned.

Race detector

The race detector is very helpful when testing. It’s very nice to get a warning when your code has a data race, and can save a lot of time debugging in production.

Code patterns

Golang’s language design brings about a number of code patterns that I dislike. Some of these are avoidable (and should be avoided!), but others we’re stuck with.

Slices

Golang has arrays (fixed length) and slices (variable length) which is slightly more complex than necessary but not bad (although I’ve seen it cause problems for new developers).

In the interests of keeping the language simple, Go has sacrificed the ability to cast slices. That’s quite good because it means that every piece of syntax is constant time. Except when casting between string and []byte⁵. Go has a very frustrating habit of making sensible choices and then giving them weird exceptions for convenience.

Another thing that annoys me is that a slice exposes its capacity all over the place. Surely the underlying capacity of a slice should be an implementation detail? If you need to ensure there exists an Nth element of a slice, you should request a specific length. If you want to request a specific capacity, fine. But you shouldn’t have to consider the capacity of the slice when slicing it. It’s simpler for Go but much more complex for everyone who uses Go. Another example of system-centric design.

Goroutines

I used to think goroutines were the scourge of the entire language. Now, I’m not so sure. I still think that structured concurrency is the way to go (no pun intended), but when used correctly, goroutines are fine.

Using errgroups and channels correctly can make goroutines less horrible to work with, but there are still many situations where it would just be so much eaiser to write correct code if we had structured concurrency.

Deferring

I strongly dislike the defer statement. It makes it very hard to see where the code flow goes. There’s no alternative, sadly. I think that Python’s with blocks are exemplary, as they effectively do the same thing but with two key advantages:

you can use a with block to scope the cleanup code to something other than a function
a with block declares exactly what it scopes at the top, and always executes the same cleanup code at the end

Error handling

Go tries to be too simple here. These days, when I write error handling code in Go, I always wrap the error at every level. That means that I’m effectively creating a stack trace as the error unwinds. Just I have to write the whole thing myself. Every single time. If Go had automatic stack traces and automatic error propegation, it would be infinitely more pleasant to use. Once again this is an area where Rust excels.

Type system-related issues

Go has both typed and untyped constants. Once again it’s a simple concept that turns out to be quite annoying. An untyped constant has no type, so it implicitly converts to whatever type is needed, but it has a default type that it falls back to. This means that any function that implements a comparison gets messed up⁶.

Or how about taking a pointer to a value. In most languages (on the same level as Golang, that is) you can take a pointer of almost any value, but Go decides that all pointers must be writable. On the face of it this is a very attractice proposition because it eliminates a whole class of errors where we write to a pointer in rodata. But Go also doesn’t include any sort of Optional container⁷, so any optional field becomes a pointer. Which means that all optional fields are pass-by-reference and mutable.

And that brings me to my next point, the language has no concept of optional parameters. This can sometimes be worked around with pointers but it makes it very painful to call the function because all arguments must be declared as variables beforehand. So most people use the options pattern to get around this. Unfortunately this pattern requires loads of code, reduces discoverability of arguments and adds a level of indirection. If Go supported a simple, universal Optional type, that would mean we didn’t need to use pointers or options.

Still on pointers (wow I hate them), Golang supports nesting structs, and it can implicitly copy fields and methods from the inner struct to the outer one. Very neat. Here’s an example:

type Animal struct {
	Species string
	Name string
}

func (a Animal) PrintName() {
	fmt.Sprintf("My name is %s", a.Name)
}

type Collar struct {
	Label string
}

type Cat struct {
	Animal
	*Collar // pointer to make it optional; not all cats have collars.
}

Let’s ignore the convoluted example and focus on the fact that every Cat contains an Animal struct, which provides the cat’s name and species. This means that we can call cat.PrintName() and it will Just Work™. There is also an optional Collar.

At this point, the cat may have a collar, which then contains a label. Golang, in an effort to be helpful, lets us call things like cat.PrintName(), cat.Species or (crucially) cat.Label.

If we try to access Cat{}.Label, the entire program panics with a null pointer deference error, because .Label is inside the .Collar which is nil. That’s despite the fact that the type that we accessed is not a pointer!

I’ll be the first to admit that this is quite a convoluted example because you should never have an embedded struct that is a nullable reference. It’s a very bad idea. But Golang permits it and raises no warnings about it, and I’ve seen it crop up in the wild.

Standard library

Due to the long-standing lack of generics (now resolved), Go’s standard library is very sparse, and what it does do is often overspecialised. I think generics is a very good case of when less is actually less – without it you have no chance of making useful, safe, reusable functions.

Documentation

And don’t even get me started on how poorly Golang is documented. If you search virtually any Go-related query, the first result is always some website called golangdocs. I don’t trust it.

Conclusion

While I have mainly focused on the downsides of Go, most of these can be avoided if code is written to sensible standards. It’s unfortunate that these standards have no specification or documentation, and mostly have no linters available (because I’ve skipped the mistakes that have linters available).

On the whole I think Go is a decent language although it can be very frustrating at times.

Telegram bug in terminated sessions

Fri, 24 Sep 2021 07:52:52 +0000

Do you think that Telegram servers are coded by monkeys?

TL;DR

A terminated session or a deleted account was still able to receive messages from active connections. Telegram fixed it as of 15~16 September 2021.

Introduction

The Telegram MTProto protocol is tricky, like its backend. Sometimes Telegram developers forget to implement some critical security controls when adding new features, in this case kicking out existing sessions. The same thing has happened before (or after) with missing rate-limiting in chat imports, although it was fixed very quickly after release.

The vulnerability

As the ability to invalidate logged in sessions or to kick out users from the application after their account got deleted are very old features, we can assume this vulnerability has been there for a long time.

But what is it about? Well, as you can read in the TL;DR, after a session is invalidated, if the connection has not been closed yet, Telegram will continue sending channels messages updates. Note that official clients always do close the connection, so the behaviour wasn’t obvious - everything appeared to work fine if you used an official client to test.

Note that channel can mean a broadcast channel, a supergroup, a gigagroup or a local group, as they are all the same at API level. Therefore, you would not receive updates from private chats, bots, basic groups, or most importantly, the Telegram service account which sends login codes.

POC

When a session gets kicked out, Telegram sends an updatesTooLong constructor, which tells clients that they are supposed to call updates.getDifference, which will then give a 401 AUTH_KEY_UNREGISTERED RPC error, prompting the client to close the TCP connection.

However, if it is ignored by the client, and no more TL functions are called, Telegram will just continue sending you channel updates until connection is closed.

To exploit this, I used the Telethon library. First, the script connected to the Telegram testmode environment and created an account for itself. Next, it immediately logged out of this account, meaning that it should be unable to read any new messages. Finally, it waited for incoming updates from Telegram (bypassing the built-in Telethon code, which made some extra RPC requests, breaking the code) and printed them.

The proof of concept code is very small (note that it no longer works, as the issue is resolved):

import asyncio
import telethon
from telethon.sync import TelegramClient
from telethon.tl.functions.auth import LogOutRequest
from telethon.tl.functions.updates import GetStateRequest


client = TelegramClient(None, 12345, "0123456789abcdef0123456789abcdef")
client.session.set_dc(2, '149.154.167.40', 80)
client.start(phone='9996621234', code_callback=lambda:'22222')

@client.on(telethon.events.NewMessage())
async def raw(e):
    print(e.text)

with client:
    client.start()
    print(client(GetStateRequest()))
    print(client(LogOutRequest()))
    asyncio.get_event_loop().run_until_complete(asyncio.wait_for(client.disconnected, None))

Here is an extract of the logs of what happened:

DEBUG:telethon.network.mtprotosender:Handling update UpdatesTooLong
DEBUG:telethon.network.mtprotosender:Receiving items from the network...
UpdatesTooLong()
DEBUG:telethon.extensions.messagepacker:Assigned msg_id = 7003973880031307500 to PingRequest (7fea21e29fa0)
DEBUG:telethon.network.mtprotosender:Encrypting 1 message(s) in 28 bytes for sending
DEBUG:telethon.network.mtprotosender:Encrypted messages put in a queue to be sent
DEBUG:telethon.network.mtprotosender:Waiting for messages to send...
DEBUG:telethon.extensions.messagepacker:Assigned msg_id = 7003973880037489216 to MsgsAck (7fea21e29e80)
DEBUG:telethon.network.mtprotosender:Encrypting 1 message(s) in 60 bytes for sending
DEBUG:telethon.network.mtprotosender:Encrypted messages put in a queue to be sent
DEBUG:telethon.network.mtprotosender:Waiting for messages to send...
DEBUG:telethon.network.mtprotosender:Handling container
DEBUG:telethon.network.mtprotosender:Handling pong for message 7003973880031307500
DEBUG:telethon.network.mtprotosender:Handling update UpdateShort
DEBUG:telethon.network.mtprotosender:Handling update Updates
DEBUG:telethon.network.mtprotosender:Handling update Updates
DEBUG:telethon.network.mtprotosender:Receiving items from the network...
UpdateChannelUserTyping(channel_id=10812878, from_id=PeerUser(user_id=925104), action=SendMessageTypingAction(), top_msg_id=None)
UpdateNewChannelMessage(message=Message(id=2, peer_id=PeerChannel(channel_id=10812878), date=datetime.datetime(2021, 9, 4, 7, 15, 33, tzinfo=datetime.timezone.utc), message='here is a sensitive message', out=False, mentioned=False, media_unread=False, silent=False, post=False, from_scheduled=False, legacy=False, edit_hide=False, pinned=False, from_id=PeerUser(user_id=925104), fwd_from=None, via_bot_id=None, reply_to=None, media=None, reply_markup=None, entities=[], views=None, forwards=None, replies=MessageReplies(replies=0, replies_pts=3, comments=False, recent_repliers=[], channel_id=None, max_id=None, read_max_id=None), edit_date=None, post_author=None, grouped_id=None, restriction_reason=[], ttl_period=None), pts=3, pts_count=1)

Conclusion

The flaw was reported to Telegram on 2021/09/04, and the bug was fixed by 2021/09/15.

I was offered a bounty but didn’t accept it because it came with an NDA, which would’ve forced me to abide by a series of rules that would’ve severely limited my freedom to disclose future vulnerabilities (whether responsibly or not) as well as trap me in an unnecessary, not to mention purposefully ambiguous, legal rat’s nest that could have been easily exploited to silence me in the future. I understand that Telegram needs to protect themselves from irresponsible security experts that disclose vulnerabilities in ways that dishonor the field, but forcing this kind of restriction onto people who spend their own time trying to find problems into their infrastructure and who attempt to proactively help fix them is an oxymoron, especially given that the economical compensation for such precious time, which Telegram never fails to boast, is locked behind the wall of signing this NDA. This sounds like a way of telling people “Thanks for reporting this! Now take this money, sign this paper here and shut up or we’ll sue you if you ever say a word about it”, which is counterintuitive given Telegram’s CEO Pavel Durov has always been very open about his despise for oppressive governments and organizations.

Credits

Me for realising the bug and the PoC
@ShalmonAnandMate for reminding me of the issue
@DavideGalilei and @nocturn9x for helping with this write-up

Turning off the RPi3B power LED in Fedora

Wed, 31 Mar 2021 10:42:50 +0000

The problem

Fedora doesn’t support the sysfs GPIO interface, and since it isn’t running the official raspbian kernel, you can’t use the GPU driver as listed on the RPi forums.

What I tried first

At first I tried enabling the configuration options in /boot/efi/config.txt, but this had no effect, presumably due to the switch of the LEDs to the GPIO expander.

Enabling GPIO

Since the LED is on a GPIO extender, I wondered if Fedora would be able to see it via the GPIO. First follow Using the GPIO on Fedora Wiki to install libgpiod, and run sudo gpioinfo. At the end you should see this output:

gpiochip1 - 8 lines:
	line   0:      "BT_ON"   "shutdown"  output  active-high [used]
	line   1:      "WL_ON"      "reset"  output   active-low [used]
	line   2: "STATUS_LED"        "ACT"  output  active-high [used]
	line   3:    "LAN_RUN"       unused  output  active-high 
	line   4: "HDMI_HPD_N"       unused   input  active-high 
	line   5:  "CAM_GPIO0"       unused  output  active-high 
	line   6:  "CAM_GPIO1"       unused  output  active-high 
	line   7:  "PWR_LOW_N"       unused   input  active-high

Success! On line 7 you see the power LED.

Turning off the LED

It’s as simple as setting that pin to 1 - sudo gpioset gpiochip1 7=1 will turn off the LED.

Welcome to my new blog

Sun, 06 Sep 2020 14:31:59 +0000

Yeah, this is me now!

I spent a lot of money on this website, so I might actually use it.