Hestia's Blog

SQLX - WHERE-IN queries. The good, the bad and the ugly.

Sun, 28 Dec 2025 00:00:00 +0000

I love SQLX for many reasons, but one in particular is the possibility to validate syntax and return types at compile-time with the query! macros. At least that’s what I thought. For a project recently I wanted to perform a query to get some role data from from a OAUTH 2.0 custom role claim. The query in and off itself was quite simple, but it lead me down a rabbit hole of the internals of sqlx and some limitations of MariaDB. Let’s have a look at the query first:

SELECT *
FROM roles
WHERE roles.idp_group IN ?;

Looks simple, right? Testing it out with DataGrip even works, because in the prompt you can just insert the list you want. So now we just pass our slice of strings to the query and:

error: error returned from database: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '?' at line 1
 --> src/main.rs:3:5
 |
3 | sqlx::query!("SELECT * FROM roles WHERE idp_group IN ?;", l);
 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 |
 = note: this error originates in the macro `$crate::sqlx_macros::expand_query` which comes from the expansion of the macro `sqlx::query` (in Nightly builds, run with -Z macro-backtrace for more info)

error: could not compile `sqlx_list` (bin "sqlx_list") due to 1 previous error

Weird. Why is MariaDB suddenly complaining here. The syntax should be right, if it just replaces the ? with the list I just passed. After some experimentation, to make sure I was not missing something trivial, I reached out to the developers of SQLX on their discord. They responded me promptly that MariaDB (and also MySQL) do not support binding arrays like that.

So it doesn’t work. Now we effectively have two option: Run one query for each group and pray to god no user has so many groups that this would cause performance issues, or use the QueryBuilder, but forego the compile-time type and query checking from SQLX.

I quickly learned that the second option wasn’t feasible. Rust type system was super strict and I ran into several problems when testing it. And considering that I wanted to use similar queries in the future (in much less traversed parts than the auth code), this was a no-go.

So it’s the first option then? Well, no. This query will fetch the user auth on every request, any slowness in this part of the code will haunt the user on every single request. I do not want to have any surprises coming my way from there.

So the only solution left: build a solution that does both myself!

The implementation

The solution that I came up with was essentially another macro, that generates the code for the QueryBuilder, while generating a separate “example” query that gets validated against the database at compile-time the way SQLX does it. I’ll quickly guide you through the steps to implement such a macro, but first we need to decide what syntax to use.

The syntax I came up with had the following constraints: You specify a list of either strings or for-loops in the query. All the strings will be appended to the query, while the for-loops will use the QueryBuilder::separated function to append the items. The trick here is, that the body of the for-loop functions as a block, returning the last statement and therefor effectively working like a list-comprehension from python.

let groups = vec![String::new("/root")];
let roles = query_in_list!{
 RoleType,
 "SELECT *"
 "FROM groups"
 "WHERE idp_group IN"
 for group in groups { group }
};

Now we just need our macro to actually parse this and generate new valid Rust source code. The parsing code took huge inspiration from the instrument macro of the tracing crate as I did not know how to do this prior to actually doing it. The logic is actually quite simple, like tracing, I just used the syn crate to parse the actual source-code.

use syn::{
 Block, Ident, LitStr, Stmt, Token, Type,
 parse::{Parse, ParseStream, Result},
};

struct QueryInList {
 query_result_type: Type,
 query: Vec<QueryFragment>,
}

enum QueryFragment {
 Lit(LitStr),
 Loop { name: Ident, list: Ident, expr: Vec<Stmt> },
}

impl Parse for QueryInList {
 fn parse(input: ParseStream) -> Result<Self> {
 let query_result_type = input.parse::<Type>()?;
 input.parse::<Token![,]>()?;

 let mut fragments = vec![];
 while !input.is_empty() {
 if let Ok(literal) = input.parse::<LitStr>() {
 let mut content = literal.value();
 content.push(' ');
 fragments.push(QueryFragment::Lit(LitStr::new(&content, lit.span())));
 continue;
 }

 input.parse::<Token![for]>()?;
 let name = input.parse::<Ident>()?;
 input.parse::<Token![in]>()?;
 let list = input.parse::<Ident>()?;
 let block = input.parse::<Block>()?;
 fragments.push(QueryFragment::Loop {
 name,
 list,
 expr: block.stmts,
 });
 }

 Ok(QueryInList {
 query_result_type,
 query: fragments,
 })
 }
}

First we simply parse a type, followed by a comma from the input stream. Then we move on to the items of the query.

As you can see, in the parse loop at line 24, I try to parse a string literal from the input stream. If that does not work, I just assume it contains the for-loop and parse one after the other: the keyword for, an identifier that is used in the loop, the keyword in, another identifier for the iterable that is used and lastly a block expression that gives us our value for the query. I package everything up into a Vec and with that, we have our data prepared.

(I also push a space after each string literal, so you don’t have to remember to have the trailing whitespace after each like. You’re welcome.)

The next step now is to turn this data into an example query that we use to validate against the database. For that we just take our list of elements. If we encounter a string-literal, we simply concatenate it, if it is a for-loop, we add the literal (?, ?) as an example, and keep track of how many placeholder we inserted:

impl QueryInList {
 fn example_query(&self) -> (usize, String) {
 let mut query = String::new();
 let mut placeholder = 0;
 for fragment in &self.query {
 match fragment {
 QueryFragment::Lit(lit) => query += &lit.value(),
 QueryFragment::Loop { .. } => {
 query += " ( ?, ? ) ";
 placeholder += 2;
 }
 }
 }

 query.push(';');

 (placeholder, query)
 }
}

With that done, all that’s left to do is assemble all the pieces into valid Rust code, that we can output from our macro for the compile to process. The generated code should look something like this:

let roles = {
 let _ = query_as!(RoleType, "SELECT * FROM roles WHERE idp_group IN (?, ?)", true, true);
 let mut builder = sqlx::QueryBuilder::new("");
 builder.push("SELECT role_id, BIT_OR(permissions) AS permissions ");
 builder.push("FROM roles ");
 builder.push("WHERE idp_group IN ");
 builder.push(" ( ");
 let mut separated = builder.separated(", ");
 for group in groups {
 separated.push_bind({ group });
 }
 builder.push(" ) ");
 builder
};

So I used the quote crate to generate the code. The great thing about quote is, that we can do string-interpolation on our source-code, making it much easier to audit and understand later.

As a quick aside: A lot of Rust detractors might say now: “See you need to pull in so many different crates. This is a huge supply-chain risk”. Interestingly enough though, all the crates used (quote, syn and proc-macro2) come from the same person: David Tolnay (I have sung his praises already, so I’ll spare you from it here). So the supply-chain risk is actually just the same as one crate, we would just degrade compile time and code re-usability. So please don’t just look at “number big” and assume that means the supply-chain risk is directly correlated. There goes much more into that.

Back to the topic – let’s generate the source code:

use proc_macro::TokenStream;
use quote::quote;
use syn::parse_macro_input;

#[proc_macro]
pub fn query_in_list(input: TokenStream) -> TokenStream {
 let query_in_list = parse_macro_input!(input as QueryInList);
 let (placeholders, qe) = query_in_list.example_query();
 let ph = std::iter::repeat_n(true, placeholders);
 let query_result_type = query_in_list.query_result_type;

 let fragments = query_in_list.query.iter().map(|f| match f {
 QueryFragment::Lit(lit) => quote! { builder.push(#lit); },
 QueryFragment::Loop { name, list, expr } => quote! {
 builder.push(" ( ");
 let mut separated = builder.separated(", ");
 for #name in #list {
 separated.push_bind({ #(#expr)* });
 }
 builder.push(" ) ");
 },
 });

 quote! {{
 // We are generating an example query here, but discarding the result.
 // This is only used to run at compile-time against the database to
 // ensure type-safety.
 let _ = sqlx::query_as!(#query_result_type, #qe, #(#ph),*);
 let mut builder = sqlx::QueryBuilder::new("");
 #(#fragments)*
 builder
 }}
 .into()
}

So, what’s happening here? First we transform the fragments we collected from the macro into their respective code-pieces: Just pushing the string in case of a literal, otherwise creating the Separated and pushing all the values based on the for-loop. As you can see, I basically just paste the whole code block inside the curly-braces into the Separated::push_bind function, which actually works great.

As for our example query, we actually need to push the same number of arguments into the codes, as we have bindings in the query, for that I just created an iterator of the value true that yields exactly the right amount of times. Surprisingly, this works and MariaDB does not complain about any type miss-matches either (put a pin in that, we’ll need that later again).

The Good

We have our cake and can eat it too (mostly, we’ll come to that). We can validate our query at compile-time, while leveraging the QueryBuilder to actually avoid the limitations of MariaDB. Nominally I have reached my goal, but the solution is still far from perfect, as I will explain in the next sections.

The Bad

While we actually do have the type now, to actually get the data out we still have to derive FromRow for the struct and pass through that code.

Must worse though, this solution is not 100% air-tight. Let me demonstrate this:

let groups = vec![String::new("/root")];
let roles = query_in_list!{
 RoleType,
 "SELECT *"
 "FROM groups"
 "WHERE idp_group IN"
 for group in groups { group }
};
roles.query_as::<OtherType>(&mut db).await

Since the function query_as takes a mutable reference, we cannot return just the query because the builder, to which it holds a reference, would be dropped at the end of the block. So between the macro and the actual use of the query it could still come to some type confusion.

The other alternative is to accept another parameter into the macro: the buffer itself, hoping that the user (or an especially clever LLM) has not had any ideas of pre-filling it to avoid a compiler error for the type confusion.

The second solution could be the prettier one, as we can re-use the specified type twice, but for now, I opted for solution one. This is definitely a point I need to come back on. But for now I don’t have a solution to actually for this problem, other than hoping that the borrow checker will become smarter about this, but I don’t think that can work, as reordering drops like that can break a bunch of Rust code where the block is used to force e.g. a MutexGuard to be dropped early.

Edit (2025-12-30): Fixing the bad.

This was a real thorn to my side, but I couldn’t think of a good solution. Thankfully I was on the Chaos Communication Congress and my friend Isse suggested that I should talk to the people at the Rust assembly, maybe they know something that can be done. And after some discussion we came up with an idea, that while not the prettiest, resolves this problem for good.

For the uninitiated in Rust macros: Rust proc-macros work by being compiled first, and then being called by the compile to generate more code. Right now they have the limitations that they need to live in a separate crate that can only export the macros themselves.

So first we move all the macro code into its separate crate, e.g. query_macros_impl and then create a separate crate query_macros that re-exports the macros alongside the new code.

Basically, what we came up with is the following: Create a wrapper type for the query builder, that takes the row type as a generic parameter and exposes only one function, which is to get the QueryAs type, with the correct lifetime. The actual code is really simple:

use sqlx::{Database, FromRow, QueryBuilder};
use sqlx::query::QueryAs;

pub struct SealedQueryBuilder<'a, DB: Database, T: FromRow<'a, DB::Row>> {
 builder: QueryBuilder<'a, DB>,
 _marker: PhantomData<T>,
}

impl<'a, DB: Database, T: FromRow<'a, DB::Row>> SealedQueryBuilder<'a, DB, T> {
 pub fn new(builder: QueryBuilder<'a, DB>) -> Self {
 Self {
 builder,
 _marker: PhantomData,
 }
 }

 pub fn query_as(&'a mut self) -> QueryAs<'a, DB, T, DB::Arguments<'a>> {
 self.builder.build_query_as()
 }
}

We basically just create the struct, the generics can be copied almost 1:1 from the QueryAs macro and the QueryAs::fetch_all method. Then we create a function that internally calls build_query_as and… That’s it. Now we can bring the type all the way through to the point where it is being used.

The Ugly

I actually lied to you about the implementation earlier to get to not spoiler the surprise in this section.

In the code that inserts the binding values we have a edge-case we did not cover. If the iterable is empty, we generate an invalid query and have a runtime error and we’d have to implement custom logic before every use of the macro. This is quite counter-intuitive and easy to forget. So we have two options again: Add a custom error that the user is required to handle, or make sure the query can run, even if one of the lists is empty.

I decided to go for the second option and that’s where we’ll reach back to our pin from earlier: remember how I said MariaDB will not complain, even if we have a type-missmatch in our query? That’s were it comes back useful: If we just push a true for each for-loop, we make sure the actual list is always (syntactically) valid and you’ll get an empty list back, which is what you’d probably expect anyways.

I want to also take this moment to formally apologize to any potential future co-workers that look at the queries executed in the database and then have to go into a rabbit-hole on why every list contains a true at the beginning.

While that “feature” of MariaDB to silently ignore type miss-matches is a blessing for fixing that edge-case, it’s also a curse – if you accidentally pass in a list of, say, ids instead of strings, the validation will work just-fine™ while the query can never return an actual row. But since this this is not something I can influence in the moment, I have to (begrudgingly) accept this as part of the deal, not just for my macro, but for all of SQLX. Nevertheless, I’m happy I discovered it, so I can be aware of this behavior going forward.

Understanding the Huffman Table Optimization

Tue, 25 Nov 2025 00:00:00 +0000

A while ago I was studying the webp image format by Google out of curiosity. I had written a ~~very incomplete~~ apng parser in the past and was interested in seeing how the lossless VP8L encoding in particular was working in that library.

While I was using a external Rust library to decode the actual image data in the past, I quickly found myself drawn to trying to implement the decoding myself. Long story short, after several days of trying to wrap my head around their Huffman decoding optimization I had to give up. I just couldn’t understand how it worked, given just the C library implementation.

I moved on, but I couldn’t get the problem out of my head. Only several weeks later, while doing something completely unrelated to programming, it suddenly all clicked into place. Since I still haven’t found a good article that explains it, I’m here now to explain how the Huffman-Coding lookup table works.

But let’s start at the beginning: what even is a Huffman-Coding and what is this optimization I was talking about?

Introduction to Huffman Coding

If you learned about the basics of this algorithm in college, like I did, feel free to skip this section. If you’ve never even heard of this, or if it’s been a while and you want a refresher, keep reading.

According to Wikipedia:

In computer science and information theory, a Huffman code is a particular type of optimal prefix code that is commonly used for lossless data compression. The process of finding or using such a code is Huffman coding, an algorithm developed by David A. Huffman while he was a Sc.D. student at MIT, and published in the 1952 paper “A Method for the Construction of Minimum-Redundancy Codes”.

For normal people: Huffman coding is a (very old in CS timelines) method to encode data more efficiently. It does this by assigning shorter codes to more frequent values, and longer ones to less frequent values. In aggregate, the resulting encoded string is shorter than the initial one, approaching the theoretical minimum size defined by the data’s entropy.

A prefix code in this context just means that there are no control characters in the final string. No code is a prefix of any other code, so the decoder can always uniquely identify each symbol as it reads the bits.

To compress a text with Huffman Coding, we first have to create a Huffman tree, by counting the frequency of each character and then aggregating it into a tree that’s approximately balanced according to their frequency. Once you’ve associated a count with each character, we take the characters with the lowest counts and join them into a tree, inserting the sum of the counts as the new root node. We then continue this, until all characters are inserted into the tree.

This works to compress the data because we are using longer codes for less frequent bytes and shorter codes for more frequent ones. So if a byte appears often, it will get compressed more efficiently, while bytes that are very rare can actually get longer. But since they appear so infrequently, we balance that overhead out with the space saved by or more frequent codes.

As an example, let’s take the string aaaaaaaabbbbbbccccddd. It contains eight a’s, six b’s, four c’s and three d’s. So our initial nodes are just those four. For our first step, we see that c and d have the lowest counts, so we group them into a new node by adding a parent:

Next up, we see that the new c-d node and the b node are the smallest, so once again, we merge them into a single node:

Now only two nodes are left, so in the last merge we create our final Huffman tree:

As you can see, I’ve also already annotated the prefix of each character. To save an a into our new string, it now only needs a single bit, a 0. To encode a b it’s 10 and c and d are 110 and 111. In total that would be 8 bits for the a’s, 12 bits for the b’s and 21 bits for the c’s and d’s, leaving us with 41 bits or 6 bytes in total, much shorter than the initial 21 bytes.

Decoding the Huffman Encoding

To decode the compressed string, we can just walk the tree in reverse, treating it as a decision tree. We start from the root, and for each bit we step one node down. If we hit a leaf node (one that contains a character), we write that character to our output buffer and jump back up to the root to start the next code.

Now while this is efficient to store, it’s not that efficient to decode: walking a tree requires many random accesses to memory and if statements. Both are not ideal for modern CPUs that try to predict our next moves to pre-fetch instructions and memory.

Every if statement is a chance for the CPU to miss-predict, and thus have to flush and re-fetch the instruction cache, throwing away the speculatively pre-computed result. Every jump to the next node risks the memory being flushed out of the CPU cache and thus then need to be fetched from a higher cache or, even worse, from RAM.

So we want to do as few memory operations as possible. Preferably just pick the result from a memory location and immediately continue decoding the input string.

That’s where the Huffman table comes into play. In contrast to the Huffman tree, the table is just a contiguous slice of memory, easily accessible. But we need to do some pre-computation to produce the table from the tree.

The Huffman Table Optimization

Since the Huffman coding guarantees that no code is a prefix of another, we can use that to our advantage. To start, we need to find the greatest code length (in bits) in the whole tree. In our case, that would be three, for both c and d.

Next, we traverse the tree, and for every node that has not yet reached that depth, we push it down to both of its children. To still know the initial length, we will also store the code length together with the decoded character.

Once we’re done with that procedure, we have a tree that’s completely full, all possible nodes are at the same level:

So how does this help us go forward? We now have more branches in the tree, with more jumps through memory and more possibilities for the CPU’s speculative execution to go wrong.

In fact, our tree now has $O(2^n)$ leaf nodes, using much more memory in the process as well. Well, as we can see in this tree, at every node, whichever branch we take, we will always find ourselves in a valid branch, until we hit the tree’s depth. In other words, every bit-pattern of length $n$ is valid in this search tree.

If you take a closer look at the leaf nodes, the first leaf node from the left can be accessed with the bit-pattern 000, the next with 001 and the one after that with 010 until we reach the last one with 111. What we did here is simply to create a lookup table for the prefix of the encoding.

So what we can do here is to simply discard the rest of the tree. Instead of taking the branches every time, we simply list the leaf nodes in a chunk of contiguous memory and then take the next $n$ bit from the stream and index directly into it with a single operation, completely eliminating the need to traverse a tree.

Once we have our character and prefix code length, we can write the character to the output buffer and re-use the rest of the index that is longer than the prefix length.

Now, some of you may wonder if using $O(n^2)$ memory is really any kind of optimization at all. And to that I can clearly say NO. In the worst case, if we are encoding single bytes, we will end up with $2^{256}$ bytes of memory usage, which is far more memory than humanity has produced so far.

But Google was aware of that limitation as well when they produced the webp image standard. So they capped the total tree depth to at most 19, which would use 1MiB of RAM for the lookup table, maintaining a good trade-off between encoding efficiency and memory usage during decoding.

In summary, Huffman coding is a cool way to compress data, and the lookup table optimization makes decoding much faster, at the cost of some extra memory. This helps us improve decoding times, especially for lower-end devices, as we are repeating the same steps sometimes multiple times per pixel in image decoding.

Reconstructing Protected or Hidden Custom Variables in Icinga DB Web

Tue, 04 Nov 2025 00:00:00 +0000

Recently Icinga DB Web had a new security release, fixing a vulnerability where protected or hidden custom variables could be inferred by any user with object visibility by abusing comparative filters on those hidden variables. This led to a 5.3/10 rated Information Disclosure Vulnerability, documented in CVE-2025-61789. In this blog post I’ll go through the discovery and the inner workings of the exploit.

Setting the Scene

Every company has their own specific needs when it comes to monitoring their environments. From specific metrics they’re interested in, to thresholds, workdays, team composition and shifts, and company structure, there’s a lot more that goes into effective monitoring than meets the eye.

Icinga’s approach to this complexity is to provide a framework to specify templates and custom data fields that lets you adapt the monitoring environment to your needs. It can be used to store thresholds, parameters for checks or notifications, or flags for searching objects. But most importantly it can be used to store passwords or API tokens needed for the check itself.

To make the monitoring results visible to those users who aren’t directly managing the infrastructure, Icinga Web 2 allows users to access only the monitoring results, or in this case the Icinga DB view and not Director with the configuration.

Icinga DB Web further allows you to hide specific custom variables, like the aforementioned passwords and API tokens. These can be hidden by giving the user a role that sets the restriction icingadb/protect/variables to give a list of patterns the user is not allowed to access, e.g. *pass*,api-token.

To execute the exploit, an attacker must have at least viewing permissions on the attacked host or service, and needs to know the name of the variable, which can’t be known in hidden variables, but can often still be inferred from the check or other properties of the object.

The Discovery

While I was investigating Icinga DB Web for integration into NetEye, I happened to be working on understanding these restrictions while at the same time helping a colleague look over a PR to support the customization of the search params.

That’s where I made the first connection. So purely out of curiosity for what might happen, I looked up one of the passwords for an object I had created to test the restriction and ran the following query against Icinga DB Web: host.name=test-host&host.vars.password=<password>.

To my surprise the object was actually returned correctly. Weird. So I tried changing the password and it disappeared. Okay, so apparently the protected variables are only protected AFTER the filters have already run, giving us access to some information about them.

If it was just an equality check that in and of itself isn’t the biggest of problems, a normal user wouldn’t know the password to run this query. And if you already know it, you don’t need the query any more.

The worst outcome of this behavior is the possibility of brute-forcing the password in Icinga DB Web instead of the service directly, in some cases circumventing the service’s brute-force protection. But we are only marginally improving our options, fundamentally working in the same time-complexity.

Improving Our Attack

So to actually be able to exploit this vulnerability, we need some way to improve the time complexity, getting more information for each request. But since we cannot see the actual password, the only state we can observe is either true or false, 1 or 0, depending on whether the object is still present. So is there no way to actually exploit this small oversight?

Well, I wouldn’t be writing this blog post if that were the case. A quick look into the UI of Icinga DB Web show that = is not the only operator we can apply to the variables. Other than equality, we can also check for greater or lesser than, i.e. the lexicographical ordering of the words.

For this example, let’s assume the password can be found in a dictionary. We all know that the words there are sorted A-Z. So if we wanted to find it as fast as possible, we could start asking: “does the password start with '”. After we’ve asked that at most 26 times, we’re bound to have found the initial letter of the password, effectively reducing the amount of words we have to look for by 96%.

And it doesn’t stop there. Now we can do the same procedure again for the second letter of the password. Another 26 questions later, and we’ve removed another 96% of the remaining words, with only 0.15% of words remaining. Continuing this strategy, we can find the password in the dictionary in at most 26 times the number of characters in the password, going from exponential to linear.

We can decrease this even further if we simply ask “is the character the password starts with greater than ?”. Then we can start in the middle and with each request exclude half of the remaining letters. With that we need at most 5 queries to find a character, going from linear to logarithmic.

We can use the same strategy for Icinga DB Web. For that we take the known prefix (the part of the password that we already know) and add the character we want to test afterwards, looking for the host with the query host.name=test-host&host.vars.password=(known prefix)(character), and with that, perform a binary search for the next character.

This strategy lets us reduce the overall time complexity from $O(A^n)$ to $O(n * log_2(A))$ . As an example, if we had a strong random password with upper and lowercase letters, digits and special characters, then we have in total $A=100$ options per character, and the difference would look as follows: Randomly guessing a password with 16 characters would take you at most $100^{16} = 10^{32}$ tries (yes, this is not a typo).

Even with 10,000 requests per second it would take over $10^{32} / 10^4 / 60 / 60 / 24 / 365 \approx 3.17e20$ years (or 24 billion times the age of the universe) to crack. On the other hand, with our improved strategy, we could reduce the amount of queries to at most $16 * ceil(log_2(100)) = 16 * 7 = 112$ (since $log_2(100) \approx 6.64$ ).

So with the same 10,000 requests per second it would take just 0.01 seconds before the password is broken. In a realistic scenario, this will likely take much longer, since we need to perform our requests strictly sequentially, reducing the number of queries we can perform per second, but it’ll probably still take less than a minute. Afterwards, to know if you’ve actually found the whole password, you then simply make the query from the beginning using the equality operator.

Of course, passwords are just one example, but this attack can be used on any variable protected in this way, including API tokens, internal IDs or other sensitive information. The credentials could then also be used to pivot to other services to use the access to trigger exploits there.

The Aftermath

Once I had found the vulnerability and verified it actually worked, I immediately reported it to security@icinga.com. They investigated it and came to the same conclusion as me that this was a severe problem.

Their investigations also uncovered a similar problem in Icinga 2, where the fix was released with versions 2.15.1, 2.14.7 and 2.13.13, while the fix for Icinga DB Web was released in versions 1.1.4 and 1.2.3. All versions were released on October 16th 2025. I encourage all users who run versions lower than that to update their systems as soon as possible.

I’ve collaborated with the people from Icinga 2 on the fix for the vulnerability, and hope I can do that in the future as well to keep all Icinga users safe. This has been a fun experience and a valuable lesson in how seemingly small bugs can be abused to obtain additional data.

Plugin Systems and Capabilities

Mon, 13 Jan 2025 00:00:00 +0000

At the 36th Chaos Communication Congresses back before COVID forced a three year break, I attended a talk from the German tech-blogger Fefe. There he talked about the “nützlich-unbedenklich Spektrum” or in English, the useful – harmless spectrum. He argued that all software lies on that spectrum to some degree.

Of course one could argue that it’s actually a triangle, where effort can optimize both usefulness and harmlessness, but for a given complexity budget I think it’s a good enough approximation to the real world to be useful for our analysis. The more capabilities the user has to impact a program’s behavior, the more chances there are for security vulnerabilities.

In this post I’d like to peg on that spectrum three specific plugin-systems, as representatives of a specific approach, and discuss their advantages and associated risk potential.

This blog post isn’t meant to be a conclusive enumeration of all possible plugin-systems with all their advantages and disadvantages, but instead to illustrate the various considerations when planning a plugin system for your software.

In particular I’ll be talking about the following the plugin-systems:

The IcingaWeb 2 plugin system
The current implementation of the Icinga2 Notifications plugin system
The plugin system of the open source video game veloren.

The IcingaWeb 2 Plugin System

The IcingaWeb 2 plugin system is the one with the most capabilities I’ll discuss in this post. It works by loading PHP source code files from the plugin directory and executing them. It’s similar to how GLPI and WordPress work, but since I’m most familiar with IcingaWeb 2, I’ll stick with that.

This plugin system comes in three flavors. First of all there’s the configuration.php file. This file is used to register all the necessary permissions and restrictions the plugin needs, as well as the side-panel entries and submenus to the module configuration. The configuration part mostly just provides some extra data to IcingaWeb 2 that it already knows how to handle, with some checks to only load certain data when multiple plugins are loaded together.

Secondly there are the hooks. Those get registered when the plugin is loaded in the run.php file. They are attached to a hook name and provide a PHP class that implements a specific interface. When a specific hook is triggered, the application will then look up all classes registered to that hook name and then loops over the hooks to call them, triggering their behavior.

Lastly, there’s the “Controller” that allows custom endpoints and “CliCommands” to extend the capabilities of icingacli. These classes are loaded from the directories application/controllers and application/clicommands respectively. Here, the behavior is directly injected into the routing/cli sub-command resolution of IcingaWeb 2, with no additional code required. They can contain any PHP code, and handle a lot of logic through reflection and the respective class and method names.

While those flavors are very different in what they provide, they actually share one crucial commonality: They all run in the same process with the same global access to all application resources. This is great, as it allows the plugin author easy access to all the needed information in a straightforward way.

There’s no need to consider every aspect that the plugin might want to access data or state. Furthermore it allows for an easy way to share dependencies, as they are already loaded in the application. This means that not every plugin has to include a way to handle database connections or HTTP routing; it’s all already provided by the application context.

This pattern is widely used in the PHP world and has certainly helped the huge success of WordPress and their Plugin ecosystem. It’s extremely useful to easily build vastly extensible software, allowing plugin developers to customize the experience of the user in a very granular way. But, as we all know, with great power comes great responsibility.

Since the plugins share the same process and application context, they need to be reviewed very carefully. Any security vulnerability in a plugin has the potential to give an attacker full access to the application, no matter how small the part of the application context that the plugin actually needs. Furthermore a harmless oversight in one plugin combined with a small bug in another can cascade into a larger security issue if an attacker manages to chain them together in a non-trivial way.

Personally I really dislike this approach for several reasons: First of all, all the permissions and capabilities of the plugin are implicit. Whatever the code does is the canonical behavior of the plugin, and there are no further checks in place to prevent misbehaving code from accessing resources outside its control.

Furthermore, I believe that as much behavior as possible should be immutable. The permissions defined should be static in a configuration file instead of being defined in the code. This would reduce the surface area where behavior can be injected, making it easier to audit the code.

Lastly, the functional programming side of my brain really dislikes that hooks can directly mutate data that’s passed in. I know that it’s not that big of an issue in the grand scheme of things, but I personally like to think of hooks as being behavior outside of the application that should not have any impact on the state if the code doesn’t explicitly use the returned data.

The Icinga2 Notifications Plugin System

The icinga2-notifications plugin system is much smaller in scope and its developers therefore opted for a different approach. Instead of running all plugins in the same application, each plugin runs in its own process, where communication with the main application is conducted via a simple JSON-RPC interface over stdin and stdout.

To load the necessary context to run the plugin, the plugin needs to implement two JSON-RPC methods: The GetInfo method is called to retrieve the necessary schema for the application configuration, and then the application makes sure the user can configure the plugin correctly and provide the requested data. If the configuration can be provided, the SetConfig RPC call is made to initialize the plugin with the configuration.

This approach is really elegant in my opinion. It gives a clear, finite point to inspect the data the plugin is actually allowed to use while keeping the state of the plugin as isolated as possible. This also reduces the information exposed to the plugin to the minimum required, reducing the impact of a possible security vulnerability.

After the plugin is initialized, the application can call the SendNotification method for each notification that needs to be sent, passing in the necessary data of the notification as the method parameters as defined in the JSON-RPC standard. The plugin can then autonomously handle the request and must then answer with the result of the action, providing either a result with an arbitrary JSON value, or an error with an error message.

While this approach limits the plugin authors to implementing only a very specific part of the user-experience, it also provides a much clearer interface to the user as a very strict separation of concerns and permissions through the OS process boundary.

In theory the plugins capabilities could even be further restricted by using a container runtime like Docker or just plain old selinux. Depending on the plugin, removing the ability to access local files, networks or services could all help reduce the blast radius of potential vulnerabilities.

The Plugin System of Veloren

So until now I’ve focused on two plugin systems from Icinga 2, so why the sudden switch to a video game? I actually spent quite a bit of time thinking about how to best implement plugin systems securely and how to balance the usefulness of the interface with the security implications, and the plugin system of veloren inspired me to finally write my thoughts down.

I’m writing this blog post at the 38th Chaos Communication Congress in Hamburg, specifically from the Veloren assembly. I had the great honor to be able to talk to Christof Petig, one of the core contributors to veloren and an all around brilliant mind. Besides hundreds of questions about the networking protocols of the game, I also got to extensively talk about the new plugin system he developed for veloren.

Christof actually did some work for the Bytecode Alliance himself; he had a little involvement in the WIT specification and did the majority of the work on the C++ backend of the canonical ABI. He led me through all the code to understand how its plugins are implemented, loaded and executed.

A veloren plugin consists of a .tar archive which must contain at least a plugin.toml file. That archive can then furthermore contain various assets for models, textures and skeletons. All of those are static files with no code attached, so assuming security while loading the files (which is a pretty easy conclusion to come to, as Rust already prevents the majority of memory vulnerabilities) they pose no real danger to users.

However should the plugin want to expose custom behavior in code, it allows the plugin author to include WebAssembly files in the archive. If the plugin file in the archive root specifies any wasm components in the modules section, those will be loaded through the wasmtime runtime. The runtime then limits the application to only perform stdio to provide logs to the game.

To confirm that the plugin is actually compliant with the hooks, the runtime will also check the wasm module against the WIT (WebAssembly Interface Types) file. Since I’m mostly concerned with code execution, in this context I’ll focus on those wasm files.

As already mentioned, the plugins code in veloren is shipped in WebAssembly, a sandboxed execution environment with a formal proof of its soundness and memory-safety. his allows the plugin to run arbitrary code in the same process, while maintaining performance and security.

One interesting decision that was made in the veloren plugin-system is that the plugins have absolutely no access to any operating resources outside stdio. This means that even a malicious plugin could never access the players’ local files or network, crucial for ensuring the security of a plugin for a game where servers can ship their own plugins to run on the users’ machines. The only thing the plugin can do is perform operations and computations on the data passed in by the game.

Speaking of which, that’s where WIT and the canonical ABI come back into play. The contract between the plugin and the game is defined in an WIT file. From that the code for both the server and plugin side are generated, making sure both sides adhere to the same contract.

When first loading the plugin, wasmtime will then make sure all the types and method definitions are actually met, as expected by the game. Then it will compile the WASM down to machine code, exposing the plugin functions over the canonical ABI. The code generated on the game’s side then makes sure that the type layout and calling conventions on the Rust side are the same as the ones in the plugin.

After the plugin is loaded, it’s basically just more native assembly running in the same process as the game. The overhead to call such functions is fairly minimal and the performance of the plugin is also very close to native code, all while maintaining security.

The obvious downside of this approach is that it’s much more limited in its capabilities. While the plugins in PHP can usually hijack a lot of the control flow of the application if they’re inclined to do so, this plugin system only allows you to attach to very specific hooks with very narrowly defined interfaces.

While I think this is the perfect choice for a video game, where the game needs to run arbitrary code from a remote server, this might not be the best choice for code that’s audited and running locally on a server, since the overhead of defining all the interfaces and developing the plugins against them might not be worth the security benefits.

Conclusion

Plugin-systems are in a interesting spot, where they both should allow users to extend the software as much as they like, while balancing it with the inherent security risks of running third-party code in your application. There are many tradeoffs to consider, such as security, performance, stability and ease of use for the plugin authors. The decision between those is often not trivial, and may require a lot of thought be put in in order to balance the different aspects.

I hope this post gave you some insight into some of the different approaches to implementing plugins and how they are used in the wild.

Taking RPITIT ― Rust’s Shiny New Feature ― for a Test Ride with Type-system-level Lambda Calculus.

Fri, 27 Oct 2023 00:00:00 +0000

With the upcoming Rust version 1.75.0, two important new features will be stabilized. These features go by the names return-position impl Trait in trait (RPITIT) and async fn in trait (AFIT).

Let me take a moment here to congratulate and thank everyone working on these features for their incredible work. This really is a big step in the Rust ecosystem. While the latter one is certainly the one more await-ed by most people, I will today focus on RPITIT, which is also the foundation for AFIT.

I’ll provide some code samples, but for the sake of brevity and to not repeat myself, not all of the code is explicitly written out. If you want to take a look at the full version, I’ve uploaded it to my GitHub. Since the feature is not out right now, it has to be compiled with the nightly: $ cargo +nightly run.

impl Trait in the Return Type

So what exactly is RPITIT? Rust has had the possibility to specify the trait bound in the return type instead of the actual type since version 1.26.0. However, one of the limitations was that this syntax couldn’t be used in traits. With the release of Rust 1.75.0 this restriction will be lifted.

But what do we even need this for in the first place? Couldn’t this just be accomplished with generics? To answer that question, let’s first take a look at the function declaration of serde_json’s from_str:

pub fn from_str<'a, T: Deserialize<'a>>(s: &'a str) -> Result<T>

As you can see, this function is generic over T. But the actual concrete type of T is in the hands of the end user. But what if we want to return a value of a distinct type that implements a trait, but that shouldn’t be left up to the user? Then we need to use the impl syntax.

If you think it’s just syntactic sugar to not spell out the whole name of the type, you’re at least partially right – if there weren’t Voldemort types. Voldemort types (types which must not be named) are types that you can’t name in a program. In Rust, both closures and functions are such Voldemort types. Each function or closure has its own unique type determined during compile-time.

This means you can’t know the concrete type before the program actually compiles, and therefore can’t use it to refer to them. But with the return type impl syntax, you can tell the compiler to figure the type out for you. Let’s look at a quick example:

fn get_iter(start: i32, end: i32) -> impl Iterator<Item = (usize, i32)> {
 (start..end)
 .into_iter()
 .map(|x| x * 2 + 1)
 .filter(|x| *x % 3 == 0)
 .enumerate()
}

Besides the obvious error with the Voldemort types of the two closures, this function could have the return type:

Enumerate<Filter<Map<Range<i32>, {{closure}}>, {{closure}}>>

But that wouldn’t really give the user of our function any additional, useful information. And if we ever decided to change the function, we would have to change the return type along with it. With this new feature however, we can pass a lot of the work onto the compiler, so we can be faster when programming.

return-position impl Trait in Trait

Now with Rust 1.75.0 this feature is finally also available for Traits. Now you can specify a trait with impl return types. Each implementation of the trait can then return a different concrete type without violating the Trait bounds. The compiler then figures out (based on the implementation of the Trait) the concrete type that is returned. Since we need to know the concrete implementation at compile-time in order to know the type that is returned, this obviously makes the whole Trait no longer object-safe, meaning we cannot create a Trait object Box<dyn Trait> with it.

Lambda Calculus

Before we can implement Lambda Calculus on the type system, we first need to understand what it actually is. Lambda Calculus is a model of computation introduced in the 1930s by mathematician Alonzo Church. It follows a few simple rules:

The only thing that exists in the basic calculus are functions.
Each function takes exactly one value as an argument and returns exactly one value.
In the function body, we can apply functions to other functions, or construct new functions.

The simplest function like this is the identity function: $id = \lambda x . x$ which is a function that just returns its input. This function however isn’t very helpful.

We can also apply functions, for example the function $\lambda x. x\ x$ applies the function $x$ once to itself. Multiple arguments can be modeled by returning a function that binds the first argument. With that we can construct new functions. $\lambda x . \lambda y . x$ returns a function that takes one argument $y$ and completely ignores it, returning $x$ instead.

In the standard notation, lambda calculus is evaluated from left to right, so the function $\lambda x . x\ x\ x$ is equivalent to $\lambda x . (x\ x)\ x$ . Calculation is then performed by applying the functions until it’s in its minimal form and can no longer be reduced.

Putting Everything Together

So how can we use this to actually calculate things in Rust? We know that we need to store state to return constructed functions, so each function will be a struct. Then, we need to define the common interface for our functions. As we just established, the function needs to take one argument and return one argument, both of which are functions. The function is also allowed a reference to itself, which does not count as a function argument.

trait Lambda: Copy {
 fn eval<T: Lambda>(self, lambda: T) -> impl Lambda;
}

And here we can already see RPITIT in action. We know that each function will return a struct that has the lambda trait and we know that that function will be a concrete type. So we leave the concrete type details up to the compiler.

Boolean Logic

Let’s warm up to the concept by starting from the very basics of computer science: true and false. We can’t have truth values, as lambda calculus only allows for functions to exist, so we have to encode these two as functions, e.g. types that implement Lambda. If we think about how we use true and false, it’s to compare them and change the control flow accordingly. So our two values will simply be functions that take two inputs and call the respective function: $True = \lambda x. \lambda y . x$ and $False = \lambda x. \lambda y . y$ . In plain English: If the value is true, the first function is called, if the value is false, the second function will be called instead.

We also know that we need to store some state here, so we need a second struct to hold it. We will name each function that carries state for its parent function with the same name as the parent function, followed by an underscore, so we’ll know it represents a partial computation. When implemented, this logic would look like this:

#[derive(Copy, Clone)]
struct True;

#[derive(Copy, Clone)]
struct True_<T: Lambda>(T);

impl Lambda for True {
 fn eval<T: Lambda>(self, lambda: T) -> impl Lambda {
 True_(lambda)
 }
}

impl<T: Lambda> Lambda for True_<T> {
 fn eval<U: Lambda>(self, lambda: U) -> impl Lambda {
 self.0
 }
}

The same (just in reverse) is also true for our False implementation. As you can see, the only thing we are doing in these functions is either creating a type or returning a type. The return type of True_<T> will always be T which will be known at compile-time. So no matter which two values we initially put into True, the resulting type will be known at compile-time.

With our boolean values in place, let’s do some logic with logical or and and. If we write them out in lambda calculus, the implementation will also be immediately clear: $lor = \lambda x . \lambda y . x\ x\ y$ . If $x$ is true, then we can return $x$ knowing that it’s true, otherwise the operation can only be true if $y$ is true, so we return $y$ . The same goes for logical and: $land = \lambda x . \lambda y . x\ y\ x$ . If $x$ is false, we can return $x$ , if it’s true, then the expression is true if and only if $y$ is true. Lastly to have a full logic system we’re missing the not operator, which is probably the simplest to understand: $\lambda x . x\ False\ True$ , if it’s true, return false, else return true.

When implemented, it would look something like:

impl Lambda for Lor {
 fn eval<T: Lambda>(self, lambda: T) -> impl Lambda {
 Lor_(lambda)
 }
}

impl<T: Lambda> Lambda for Lor_<T> {
 fn eval<U: Lambda>(self, lambda: U) -> impl Lambda {
 let first = self.0;

 first.eval(first).eval(lambda)
 }
}

And similarly for Land and Not.

The attentive ones of you might have noticed that the $lor$ implementation is actually overly complex. Since we can just partially apply functions, we can reduce the implementation to $\lambda x . x\ x$ if we so choose. This will return a function with the first field bound, and it can equally be applied to the next function. In Haskell this is also called point-free programming.

Helper Functions

Now we want to actually get our hands on the result of the calculations we’ve performed so far. Since the structs / types ARE our data, we can simply print out the type name of the final value to get our answers. Rust provides the std::any::type_name() function to get the string for a type. However to get that, we would have to know that type, which we explicitly don’t want to know ourselves. We can easily build around this though by creating a helper function that does the job for us:

fn type_name<T>(_: &T) -> &'static str {
 std::any::type_name::<T>()
}

fn size_of<T>(_: &T) -> usize {
 std::mem::size_of::<T>()
}

println!("{}", type_name(&Lor.eval(True).eval(False)));

With this we delegate the work to actually know the type we want to print out to the compiler. We can do the same to check the size of the type so we know how much memory it occupies.

Church Numerals

The last thing I want to touch on are numbers. How can we represent numbers in this system? In a system where all you can do is apply functions to other functions, numbers can be represented as a number of function applications. Zero obviously applies the function zero times. This could also be defined as False, e.g. return the last parameter ignoring the first, or as Zero returning the identity function $zero = \lambda x . id$ . Each function greater than zero is then represented as the successor to the previous number. So one is the successor of zero and two is the successor of one (or, the successor of the successor of zero).

The application of successor is then a bit more involved: $succ = \lambda x . \lambda y . \lambda z . y\ (x\ y\ z)$ , i.e. we apply the function $y$ once to the result of $x$ applications of $y$ to $z$ . So each successor applies the function once until it hits zero which does not apply the function, but returns its initial value.

Addition then can be defined as the repeated application of the successor function to a value: $add = \lambda x . \lambda y . x\ succ\ y$ . So we are basically adding one $x$ times to $y$ which is functionally the same as $x + y$ .

Remember the size_of function from before? Now let’s see how much memory our church numeral takes up if we, for example calculate $2 + 3$ :

let two = Succ.eval(Succ.eval(Zero));
let three = Succ.eval(Succ.eval(Succ.eval(Zero)));
let result = Add.eval(three).eval(two);
println!("{}", type_name(&result)); // Succ_<Succ_<Succ_<Succ_<Succ_<Zero>>>>>
println!("{}", size_of(&result)); // 0

Yes, that’s right. Our church numeral does not take up any memory at all! It’s all just type information at compile-time. And the addition we have defined is just type resolution at compile-time. If we execute $ strings on our final binary, we can even see the result of the type name already lying there. So theoretically we have created the perfect program. It computes everything in 0 seconds (not counting compile-time). However, all information must also be known at compile-time, and we have no further control at runtime, so it’s probably not the most flexible program ever written.

Static Field Validation in Serde

Mon, 05 Jun 2023 00:00:00 +0000

I recently had to parse the JSON-RPC 2.0 standard and ran into the following problem: The standard requires the field "jsonrpc": "2.0" in the JSON itself, and I wanted to validate that with Serde to ensure the message conforms to the standard. On the other hand I don’t need the field in the actual struct, since once it’s validated it won’t ever and should never change.

Edit: I now know of the existence of the crate monostate by the creator of serde, David Tolnay. I think it’s still interesting to explore this problem a bit.

Serde Tag

My first idea was to use the tag feature of Serde:

use serde::{Deserialize, Serialize};
use serde_json::Value;

#[derive(Serialize, Deserialize)]
#[derive(Debug, Eq, PartialEq)]
#[serde(tag = "jsonrpc", rename = "2.0")]
struct JsonRpcTagged {
 method: String,
 #[serde(skip_serializing_if = "Option::is_none")]
 params: Option<Value>,
 #[serde(skip_serializing_if = "Option::is_none")]
 id: Option<usize>,
}

so let’s try it out:

let jsonrpc = JsonRpcTagged { method: "my_method".to_owned(), params: None, id: None };

let message = serde_json::to_string(&jsonrpc).unwrap();
assert_eq!(message, r#"{"jsonrpc":"2.0","method":"my_method"}"#);

Wonderful! As expected it adds the jsonrpc tag to the message! Now let’s try de-serializing:

let message = r#"{"jsonrpc":"2.0","method":"my_method"}"#;
let expected = JsonRpcTagged { method: "my_method".to_owned(), params: None, id: None };

let result: JsonRpcTagged = serde_json::from_str(message).unwrap();
assert_eq!(result, expected);

Till now everything worked great and I was optimistic. But what we really want is to fail if the jsonrpc field isn’t given, as that’s what the specification demands. So let’s try that:

let message = r#"{"method":"my_method"}"#;
let expected = JsonRpcTagged { method: "my_method".to_owned(), params: None, id: None };

let result: JsonRpcTagged = serde_json::from_str(message).unwrap();
assert_eq!(result, expected);

Serde will still de-serialize this! Which is bad, actually, because I wanted to use it for validation. This path was obviously a dead end, so I went back to the drawing board.

(Ab)using enums

Now what if the tag actually means something? What if it’s actually an enum variant and therefore is needed for de-serialization? So I constructed my second attempt:

#[derive(Serialize, Deserialize)]
#[derive(Debug, Eq, PartialEq)]
#[serde(tag = "jsonrpc")]
enum JsonRpcEnum {
 #[serde(rename = "2.0")]
 JsonRpc {
 method: String,
 #[serde(skip_serializing_if = "Option::is_none")]
 params: Option<Value>,
 #[serde(skip_serializing_if = "Option::is_none")]
 id: Option<usize>,
 },
}

Running all the previous tests showed that this works! It serializes what it should and more importantly, doesn’t de-serialize what it shouldn’t. Even better, the Rust compiler is smart enough to see that the enum has only one possible variant, and it therefore optimized away the 8-byte tags that are normally associated with an enum.

assert_eq!(std::mem::size_of<JsonRpcTagged>(), std::mem::size_of<JsonRpcEnum>())

Now I should be happy right? I have achieved what I initially set out to do. But my perfectionism wouldn’t let me stop there. First of all, the type looks a bit weird, but more dramatically, we can’t access the fields in an easy way. The easiest way to access the contents of this enum is pretty ugly:

let JsonRpcEnum::JsonRpc{ref method, .. } = some_jsonrpc;

It’s cool to see that Rust actually lets you compile a monstrosity like that, but it’s definitely not something for the weak-spirited. While I try not to judge people based on their appearances, I will judge code for that. So back to the drawing board I went one last time.

Using Custom De-serialization

First I tried implementing both Serialize and De-serialize for an empty struct: struct JsonRpcVersion;, but that meant for all other derives I’d have to add them to both the JsonRpc struct and this empty struct. Furthermore this was not very reusable as I’d need replicate the whole logic the next time I needed something similar.

Enter the empty tuple. One of my favorite Rust features is that it allows some cool code trickery and compiler optimizations. Look at the implementation of the HashSet if you have some time, or read some of the many other great blog posts. It solves my first problem of running (after validation) structs for all derive macros I want to use. It’s also zero-size, meaning we’re not wasting memory space or compute time.

The other savior of my soul was the #[serde(with = "Validator")] attribute. It lets you specify a validator that handles both serialization and de-serialization by calling Validator::serialize(..) and Validator::deserialize(..) on it respectively. Now all I need is a validator struct that implements it, and a trait that does the implementation for me.

So let’s take a look at the Trait I came up with:

use serde::de::{Error, Unexpected, Visitor};
use serde::{Deserializer, Serializer};
use std::fmt::Formatter;
use std::marker::PhantomData;

pub trait Validator: Sized {
 const EXPECTED_VALUE: &'static str;

 fn serialize<S: Serializer>(_version: &(), serializer: S) -> Result<S::Ok, S::Error> {
 serializer.serialize_str(Self::EXPECTED_VALUE)
 }

 fn deserialize<'de, D: Deserializer<'de>>(deserializer: D) -> Result<(), D::Error> {
 deserializer.deserialize_str(ValidatorVisitor::<Self> {
 phantom_data: PhantomData::default(),
 })
 }
}

#[derive(Default)]
struct ValidatorVisitor<T: Validator> {
 phantom_data: PhantomData<T>,
}

impl<'de, T: Validator> Visitor<'de> for ValidatorVisitor<T> {
 type Value = ();

 fn expecting(&self, formatter: &mut Formatter) -> std::fmt::Result {
 formatter.write_str(T::EXPECTED_VALUE)
 }

 fn visit_borrowed_str<E: Error>(self, v: &'de str) -> Result<Self::Value, E> {
 if v == T::EXPECTED_VALUE {
 Ok(())
 } else {
 Err(E::invalid_value(Unexpected::Str(v), &T::EXPECTED_VALUE))
 }
 }
}

It’s a bit of code, but it’s worth understanding, so let me guide you through it.

The trait itself implements the serialize and de-serialize functions that will later be called by Serde. The value for it is provided as an associated constant value. Shout-out to u/SkiFire13 on the Rust user forum for making me aware of that feature. It provides whatever data we need to parse the value later on. For the serialization, we can simply write the string to the formatter. But de-serialization gets a bit more complicated.

To de-serialize data, I need a Visitor. My first instinct was to implement the Visitor for all arbitrary types T: Validator to get quick access to the string slice in the trait bound. Sadly the compiler doesn’t allow blanket implementations of traits that aren’t defined in this crate so I had to create a wrapper type.

The ValidatorVisitor is a zero-sized struct that does nothing but hold the type T for the Visitor to access. Since the compiler doesn’t allow you to compile a struct with an unused generic argument, we need to add one field to the visitor: some phantom data that consumes that type in the background and does some compiler magic that allows this implementation to actually compile.

In the implementation body of the trait we then simply provide the function visit_borrowed_str(..) that returns an empty tuple if it receives the correct value; otherwise it returns an error.

Now what does the final implementation of my JsonRpc struct actually look like? The Validators are just four LOC:

struct JsonRpcVersionValidator;

impl Validator for JsonRpcVersionValidator {
 const EXPECTED_VALUE: &'static str = "2.0";
}

That’s what I’m talking about! That’s what clean code should look like! We have one point that statically controls the value, and it’s very re-usable. After we’ve defined the Validator, we can simply use it in the struct as follows:

#[derive(Serialize, Deserialize, Debug, Eq, PartialEq)]
struct JsonRpcValidated {
 #[serde(with = "JsonRpcVersionValidator", rename = "jsonrpc")]
 version: (),
 method: String,
 #[serde(skip_serializing_if = "Option::is_none")]
 params: Option<Value>,
 #[serde(skip_serializing_if = "Option::is_none")]
 id: Option<usize>,
}

And with that we have a struct that matches all the criteria I originally set out. It serializes correctly, it de-serializes what it should, and doesn’t what it shouldn’t. It’s easy to use and reuse and doesn’t waste any space or computation time.

About Set Theory, the N-queens Problem, and SQL

Mon, 05 Jun 2023 00:00:00 +0000

The n-queens problem is a common exercise in computer science. Legend has it that a mathematician once declared that women are like the queens in chess, you can’t put eight of them in a room without them trying to kill each other. This obviously isn’t true, and since I’m a feminist, but also a nerd, we will disprove him today in an unconventional way: with an SQL query.

Let me give you a quick overview of the problem:

In chess the queen can move any number of squares vertically, horizontally or diagonally. This means, if two queens are in the same row, the same column or diagonal to one another, they can attack each other. The problem requires us to place n queens on a n-by-n board without any of them being able to attack each other, or in more technical terms, we are looking for the set of all sets with cardinality n such that no two queens in that set can attack each other. If that set is non-empty, then we have disproven the claim.

Modeling the problem

Normally this problem would be solved iteratively with an algorithm, however this is not possible in SQL. Therefore we have to reach back to set theory to model the problem in a language that translates well to SQL.

First we need a set of all positions a queen can be in. For that we model a queen as a tuple of two coordinates in the n-by-n grid:

possible\ positions = \lbrace ( x, y )\ |\ x, y \in [ 1 , n ] \rbrace

After that we need a set of all combinations with queens. This set of possible positions grows quadratically with the length of the board, described by the function $((n^2!) / ( n^2 - n )! ) / n!$ . That’s because we have $n^2!$ possible combinations of queen positions on the board without stacking any on top of each other. However $(n^2-n)!$ of those are sets with more than n queens on the board. After that we have all combinations duplicated $n!$ times as permutations of each other. We are, however, only interested in single solutions without considering those permutations. With this we find there exist 1,820 possible combinations for a 4-by-4 board, which is the smallest board with a solution. On a standard 8-by-8 board this number rises to 4,426,165,368.

boards_n = \Bigl\{ \bigl\{ (x_1, y_1), (x_2, y_2) .. (x_n, y_n) \bigr\} | (x, y) \in possible\ positions \Bigr\}

Now that we have that set, we need to filter out all sets that contain queens that can attack each other. To check if they are in the same column or row is trivial. We just have to check if the x or y coordinates of the tuple are equal. However for the diagonals we have to use a trick.

Let’s first list the diagonal fields from the top left to the bottom right out so we get a feel for the data: (1,1), (2,2) .. (8,8). As we can see, in this diagonal both the x and y coordinates grow at the same rate. With that information we can guarantee that the difference between x and y in the same diagonal also stays the same: 1 – 1 = 0, 2 – 2 = 0 .. 8 – 8 = 0. We can use that difference to check weather two queens are in the same diagonal or not.

We can use a similar trick for the other diagonal as well. Let’s list the fields from the top right to the bottom left first: (8,1), (7,2) .. (1,8). Here the x coordinate decreases at the same rate that y increases. Therefore the sum of the coordinates in this diagonal stays the same: 8 + 1 = 9, 7 + 2 = 9 .. 1 + 8 = 9.

With this we now have all our conditions for which queens can attack each other, and we only need to plug them into our final formula to get the expression:

\lbrace\

\hspace{5mm} board \in boards\ |

\hspace{5mm} \forall (x_1, y_1) \in board\ :\ \nexists (x_2, y_2) \in ( board \setminus (x_1, y_1))

\hspace{5mm} such\ that\ (x_1=x_2) \lor (y_1=y_1) \lor (x_1-y_1=x_2-y_2) \lor (x_1+y_1) = (x_2+y_2)

\rbrace

We create the set of all sets of queens with cardinality $n$ , for which it holds that for all queens in that set there does not exist another queen such that they could attack each other horizontally, vertically or diagonally.

Or in plain English: All sets with n queens in them and no two queens can attack one another.

Translating this to SQL

While at first it seems like this logic could be mapped fairly well to SQL, we quickly run into our first problems: this logic works on sets of sets of tuples, however SQL works on sets of tuples. Therefore we have to flatten the sets down to a larger tuple.

However this structure also gives us distinct advantages: We can make assumptions about our data and with that decrease the amount of work that has to be done and stop computation for partial results, if they already don’t fit our criteria.

I will implement the logic here for the smallest board with solutions: the 4-by-4 board, since the implementation get increasingly verbose. However this model can theoretically be increased for all sizes.

But first we need a set of columns on which we can place. We use a recursive common table expression (rCTE) for this to save ourselves a bit work:

WITH RECURSIVE nqueens_columns AS (
 SELECT 4 as queen_column
 UNION
 SELECT queen_column - 1 AS queen_column
 FROM nqueens_columns
 WHERE queen_column > 1
) SELECT * FROM nqueens_columns;

Now we could theoretically create a table for all positions of queens on the board with nested common table expressions to get a set of all possible positions for a queen on a chess board.

WITH RECURSIVE queen_positions AS (
 WITH RECURSIVE nqueens_columns AS (
 SELECT 4 as queen_column
 UNION
 SELECT queen_column - 1 AS queen_column
 FROM nqueens_columns
 WHERE queen_column > 1
 )
 SELECT 4 as queen_row, queen_column
 FROM nqueens_columns
 UNION
 SELECT queen_row - 1 AS queen_row, new_columns.queen_column
 FROM queen_positions JOIN nqueens_columns as new_columns
 WHERE queen_row > 1
) SELECT * FROM queen_positions;

After n cross joins of this table with itself, we would then get a set of all boards with exactly n queens on them, even the boards with queens stacked on top of each other. However this would also result in a lot of wasted work by including many solutions that obviously don’t work. Furthermore it would duplicate all solutions by a factor of n! since the tuple is sorted and therefore all mutations of the same result appear different. This is a tradeoff we have to stomach by mapping the set of tuples, which would avoid such cases, to a tuple that can be processed by an SQL query. The query to filter out duplicates would grow very quickly and would be much more verbose that doing the join by hand. However, by writing the query by hand, we have an alternative.

We can save a lot of work by making assumptions about our data. Remember how I previously mentioned that we have 1,820 possible combinations of queen positions for this board? We know that combinations with queens in the same row can never be in the final result, because they already violate one of the rules. Therefore we can leave them out of the initial set by adding each new queen in a new row. The new set with this simple assumption will still grow exponentially with the function n^n, which is a huge space to search, but it’s still far less than what’s needed by the brute-force approach. By including that assumption in our query (and making it a bit more verbose), this query will only return 256 results, saving the computer a lot of time and resources, especially for larger boards.

WITH RECURSIVE nqueens_columns AS (
 SELECT 4 as columns
 UNION
 SELECT columns - 1 AS columns
 FROM nqueens_columns
 WHERE columns > 1
) SELECT * FROM (
 SELECT 1 AS x1, columns AS y1 
 FROM nqueens_columns
) as queen1
JOIN
(
 SELECT 2 AS x2, columns AS y2
 FROM nqueens_columns
) AS queen2
JOIN
(
 SELECT 3 AS x3, columns AS y3
 FROM nqueens_columns
) AS queen3
JOIN
(
 SELECT 4 AS x4, columns AS y4
 FROM nqueens_columns
) AS queen4;

After this we only need to apply our rules to the query, and filter out all sets that contain queens that attack each other. Here we can make another assumption about our data: While it would be easiest to add a where-clause for all restrictions to the end of the query, we can optimize by throwing out partial results that already violate our rules. We check after each join if it clashes with any of the previous queens and only keep those that don’t, reducing the joins that need to be carried out for the remaining queens. For example, after joining in the second queen, we already check if she can attack the first queen. If that’s possible, we can already ignore that without taking any further queens into account. After the third queen, we check if there are clashes with queens one and two and so on.

WITH RECURSIVE nqueens_columns AS (
 SELECT 4 as columns
 UNION
 SELECT columns - 1 AS columns
 FROM nqueens_columns
 WHERE columns > 1
) SELECT * FROM (
 SELECT 1 AS x1, columns AS y1 
 FROM nqueens_columns
) as queen1
JOIN
(
 SELECT 2 AS x2, columns AS y2
 FROM nqueens_columns
) AS queen2
ON NOT (
 y1 = y2
 OR x1 - y1 = x2 - y2
 OR x1 + y1 = x2 + y2
)
JOIN
(
 SELECT 3 AS x3, columns AS y3
 FROM nqueens_columns
) AS queen3
ON NOT (
 y1 = y3
 OR x1 - y1 = x3 - y3
 OR x1 + y1 = x3 + y3
 OR y2 = y3
 OR x2 - y2 = x3 - y3
 OR x2 + y2 = x3 + y3
)
JOIN
(
 SELECT 4 AS x4, columns AS y4
 FROM nqueens_columns
) AS queen4
ON NOT (
 y1 = y4
 OR x1 - y1 = x4 - y4
 OR x1 + y1 = x4 + y4
 OR y2 = y4
 OR x2 - y2 = x4 - y4
 OR x2 + y2 = x4 + y4
 OR y3 = y4
 OR x3 - y3 = x4 - y4
 OR x3 + y3 = x4 + y4
);

With this, we have at last constructed the final query! Now the only thing remaining is to format our output to get actual chess positions, so that the chess players here won’t hunt me down later.

[..] SELECT
 CONCAT(CHAR(x1 + 64), y1) AS queen1,
 CONCAT(CHAR(x2 + 64), y2) AS queen2,
 CONCAT(CHAR(x3 + 64), y3) AS queen3,
 CONCAT(CHAR(x4 + 64), y4) AS queen4
FROM [..]

The final result is a set with two possible solutions. After a quick check on Wikipedia we can see that we actually found the correct solutions for a 4-by-4 board through a simple SQL query.

+--------+--------+--------+--------+
| queen1 | queen2 | queen3 | queen4 |
+--------+--------+--------+--------+
| A2 | B4 | C1 | D3 |
| A3 | B1 | C4 | D2 |
+--------+--------+--------+--------+

And to answer the question from above, here’s the query for eight queens. We get 92 results, firmly disproving this mathematician of legends.

WITH RECURSIVE nqueens_columns AS (
 SELECT 8 as columns
 UNION
 SELECT columns - 1 AS columns
 FROM nqueens_columns
 WHERE columns > 1
) SELECT 
 CONCAT(CHAR(x1 + 64), y1) AS queen1,
 CONCAT(CHAR(x2 + 64), y2) AS queen2,
 CONCAT(CHAR(x3 + 64), y3) AS queen3,
 CONCAT(CHAR(x4 + 64), y4) AS queen4,
 CONCAT(CHAR(x5 + 64), y5) AS queen5,
 CONCAT(CHAR(x6 + 64), y6) AS queen6,
 CONCAT(CHAR(x7 + 64), y7) AS queen7,
 CONCAT(CHAR(x8 + 64), y8) AS queen8
FROM (
 SELECT 1 AS x1, columns AS y1 
 FROM nqueens_columns
) as queen1
JOIN
(
 SELECT 2 AS x2, columns AS y2
 FROM nqueens_columns
) AS queen2
ON NOT (
 y1 = y2
 OR x1 - y1 = x2 - y2
 OR x1 + y1 = x2 + y2
)
JOIN
(
 SELECT 3 AS x3, columns AS y3
 FROM nqueens_columns
) AS queen3
ON NOT (
 y1 = y3
 OR x1 - y1 = x3 - y3
 OR x1 + y1 = x3 + y3
 OR y2 = y3
 OR x2 - y2 = x3 - y3
 OR x2 + y2 = x3 + y3
)
JOIN
(
 SELECT 4 AS x4, columns AS y4
 FROM nqueens_columns
) AS queen4
ON NOT (
 y1 = y4
 OR x1 - y1 = x4 - y4
 OR x1 + y1 = x4 + y4
 OR y2 = y4
 OR x2 - y2 = x4 - y4
 OR x2 + y2 = x4 + y4
 OR y3 = y4
 OR x3 - y3 = x4 - y4
 OR x3 + y3 = x4 + y4
)
JOIN
(
 SELECT 5 AS x5, columns AS y5
 FROM nqueens_columns
) AS queen5
ON NOT (
 y1 = y5
 OR x1 - y1 = x5 - y5
 OR x1 + y1 = x5 + y5
 OR y2 = y5
 OR x2 - y2 = x5 - y5
 OR x2 + y2 = x5 + y5
 OR y3 = y5
 OR x3 - y3 = x5 - y5
 OR x3 + y3 = x5 + y5
 OR y4 = y5
 OR x4 - y4 = x5 - y5
 OR x4 + y4 = x5 + y5
)
JOIN
(
 SELECT 6 AS x6, columns AS y6
 FROM nqueens_columns
) AS queen6
ON NOT (
 y1 = y6
 OR x1 - y1 = x6 - y6
 OR x1 + y1 = x6 + y6
 OR y2 = y6
 OR x2 - y2 = x6 - y6
 OR x2 + y2 = x6 + y6
 OR y3 = y6
 OR x3 - y3 = x6 - y6
 OR x3 + y3 = x6 + y6
 OR y4 = y6
 OR x4 - y4 = x6 - y6
 OR x4 + y4 = x6 + y6
 OR y5 = y6
 OR x5 - y5 = x6 - y6
 OR x5 + y5 = x6 + y6
)
JOIN
(
 SELECT 7 AS x7, columns AS y7
 FROM nqueens_columns
) AS queen7
ON NOT (
 y1 = y7
 OR x1 - y1 = x7 - y7
 OR x1 + y1 = x7 + y7
 OR y2 = y7
 OR x2 - y2 = x7 - y7
 OR x2 + y2 = x7 + y7
 OR y3 = y7
 OR x3 - y3 = x7 - y7
 OR x3 + y3 = x7 + y7
 OR y4 = y7
 OR x4 - y4 = x7 - y7
 OR x4 + y4 = x7 + y7
 OR y5 = y7
 OR x5 - y5 = x7 - y7
 OR x5 + y5 = x7 + y7
 OR y6 = y7
 OR x6 - y6 = x7 - y7
 OR x6 + y6 = x7 + y7
)
JOIN
(
 SELECT 8 AS x8, columns AS y8
 FROM nqueens_columns
) AS queen8
ON NOT (
 y1 = y8
 OR x1 - y1 = x8 - y8
 OR x1 + y1 = x8 + y8
 OR y2 = y8
 OR x2 - y2 = x8 - y8
 OR x2 + y2 = x8 + y8
 OR y3 = y8
 OR x3 - y3 = x8 - y8
 OR x3 + y3 = x8 + y8
 OR y4 = y8
 OR x4 - y4 = x8 - y8
 OR x4 + y4 = x8 + y8
 OR y5 = y8
 OR x5 - y5 = x8 - y8
 OR x5 + y5 = x8 + y8
 OR y6 = y8
 OR x6 - y6 = x8 - y8
 OR x6 + y6 = x8 + y8
 OR y7 = y8
 OR x7 - y7 = x8 - y8
 OR x7 + y7 = x8 + y8
);