Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character. It may intuitively feel like having 26 more characters to encode with would be hugely more efficient but for the extra risk of what all those characters get us we're only getting just over .5 more bits per character. I need to be either pretty desperate for efficiency, rare in 2026, or very, very sure that those things are only going places where those characters aren't risky or inconvenient, which is pretty difficult to guarantee since even if it is in fact true today who's to say it will be tomorrow, for this to be worth the risk of the extra characters.
Even just the extra characters in base64 are often annoying and we already have multiple common base64 standards because even trying to find just two more characters to fill out a-zA-Z0-9 is hard.
Base64 all the way, but with dot and underscore. Dash is problematic (or, well, let's say you're bound to end up on a script not correctly dealing with those: you definitely can deal correctly with dashes and spacing chars in filenames, but you're guaranteed to encounter a SNAFU at some point when a script you didn't write shall invariably fuxx0rs everything up).
Dot is accepted and common in filenames anyway.
I've got a scheme where I added checksums to my files (in the filename) using dashes but switching to underscore is on my TODO list. Basically I'll change this:
dsc0974987-b3-0a772982c1.jpg
to:
dsc0974987_b3_0a772982c1.jpg
(YMMV as to whether adding the first x bits of whatever checksum you fancy on files you know aren't going to change works or doesn't work for you: it works for me and it's part of a suite of tools I made for myself for file integrity / backuping purpose / classification / etc.)
Meanwhile, Windows uses UTF-16 everywhere internally, so "a" and "あ" are both 2 bytes large. You'd still have to exclude surrogate pairs and the Unicode control characters from a filename.
> It emits groups of five characters. Five is the sweet spot: 84⁵ = 4,182,119,424, only 2.6% short of 2³².
> That leaves enough room for a group to hold 32 bits about 95% of the time on uniformly random input, and 31 bits otherwise.
> The encoder looks at the next 31 bits. If their value is below 84⁵ - 2³¹, there’s room for a 32nd bit. Otherwise, it consumes just those 31 bits. Either way, the value fits in five base-84 digits.
I think bit-aligned encodings (i.e. schemes where the output alphabets have a size that is a power of two) are the sweet-spot for encoding schemes due to simplicity of encoding and decoding compared to non-bit-aligned schemes like base84.
Base16 is verbose, but each symbol in the alphabet carries exactly 4 bits (hence why a single byte in hex is two symbols). This is nice for encoding bytes. Base64 requires two non-alphanumeric symbols such as + and / or - and _ because A-Za-z0-9 only provides 62 characters. Each output symbol encodes exactly 6 bits of the original input. But when we're encoding whole bytes we sometimes need to use padding (when the input is not a multiple of 3 bytes and that ambiguity is harmful in the context that decoding occurs in). Then there's Base32, which is what currently seems like a sweet spot to me and is what I'm considering using for identifiers in my own systems. It only requires 32 characters, so can easily take a range of A-Z0-9 or a-z0-9 excluding visually-similar characters. Like Base64 it can also require padding in some settings.
I quite like the general scheme used in the Bech32 and Bech32m framing idea, which allows for a human-readable prefix before a `1` and then the base32 data follows. This can be done because `1` is excluded from the Bech32 alphabet. This prefix can be used to differentiate between kinds of identifier, for example:
But to address the article directly, I'm not sure the complexity is worth the gains in the table in https://github.com/jedisct1/zig-base84#encoding, which states that we save ~6.5% of characters, and for 128 input bytes, the resulting Base64 string is 171 characters, while the Base84 string is 161-166 characters.
I also dislike that the chosen alphabet breaks text selection; GitHub very carefully picked their current token formats so that the whole token is selected with a double-click:
> One other neat thing about _ is it will reliably select the whole token when you double click on it. Other characters we considered are sometimes included in application word separators and thus will stop highlighting at that character. Try out double clicking this-random-text versus this_random_text!
Regardless though we still have the fundamental issue that bytes encoded as Base16, Base32, Base64, or even Base91/Base84 will be _longer_ when encoded. Base32 can encode up to 159 bytes before hitting a filename limit of 255 bytes. I can't remember whether filenames are 16-bit on Windows (UCS-2? UTF-16?), but I speculate on average that maybe you could save a lot of bytes by first ensuring that filenames are UTF-8 before encrypting them, since filenames on a lot of computeres, especially ones running in the west, likely contain a lot of latin characters. You could even switch between encodings to get the best bit-packing in filenames.
I think turbocrypt could switch to using a surrogate file when an encrypted filename exceeds 255 bytes to allow for encrypting any filename permitted by the filesystem, while still preserving the property that the same filename in a different directory has the same encrypted filename. If an encrypted filename is too long to fit on the filesystem, maybe its hash could be stored as the filename instead (which is extremely unlikely to collide). Then we could store the full encrypted filename in a `.name` file in the same directory. We could do similar for directory names that are too long. We could then use a less dense but bit-aligned encoding scheme like Base32 with an all-lowercase alphabet without any punctuation to make filename text selection straightfoward. We would avoid all case sensitivity traps that can occur with things like Base64 and Base84. Let's pick two filename prefixes, say "tcf1" for "turbocrypt filename" and "tch1" for "turbocrypt filename hash". Then we could have a directory layout like this:
tch1q9x8gf2tvdw0 # a file whose encrypted filename exceeds 255 bytes; this filename is a hash of the encrypted filename, e.g. SHA-256.
tch1q9x8gf2tvdw0.name # a file whose contents are the full encrypted filename of tch1q9x8gf2tvdw0.
tcf1tvdhc6mua7l9x8s3qprz # a file whose encrypted filename does not exceed 255 bytes.
Once a system to support long filenames is implemented, the size of the alphabet used for encoding the filenames (like Base84) becomes less important; Base16, Base32, Base64, or another bit-aligned encoding scheme could be used.
As a very small added bonus, you could even implement a bit-aligned codec such as base16 or base64 using SIMD via a bunch of swizzling, shifting, bitmasking, and AND/XORing, making it very fast should the need arise.
> Here’s the alphabet, in encoding order:
> ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!#$%&'()+,-;=@[]^_`{}~
Yeah I definitely don't want any of those last ~22 characters in file names if I need to do anything with them from a shell
Moreover, I'd be nervous about the fact that filenames on Windows tend to be case insensitive.
Yeah, I'm sympathetic to their desire for compression and efficiency but base64URL is way more shell-friendly.
I quite like base58 for human-presented identifiers also. I tend to use it instead of hex for having ~8-16 character IDs of things.
Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character. It may intuitively feel like having 26 more characters to encode with would be hugely more efficient but for the extra risk of what all those characters get us we're only getting just over .5 more bits per character. I need to be either pretty desperate for efficiency, rare in 2026, or very, very sure that those things are only going places where those characters aren't risky or inconvenient, which is pretty difficult to guarantee since even if it is in fact true today who's to say it will be tomorrow, for this to be worth the risk of the extra characters.
Even just the extra characters in base64 are often annoying and we already have multiple common base64 standards because even trying to find just two more characters to fill out a-zA-Z0-9 is hard.
Even just "-" in a filename is problematic, since you can name actual files with "--" at the beginning, and the name could match a switch.
What you don’t want a file named `reboot`?
The non-alphanumeric characters are just asking for trouble. Agents will fail to escape them correctly, and the $ could expand a variable.
Base62 (or 64, with -_) is totally sufficient.
> Base62 (or 64, with -_) is totally suffic
Base64 all the way, but with dot and underscore. Dash is problematic (or, well, let's say you're bound to end up on a script not correctly dealing with those: you definitely can deal correctly with dashes and spacing chars in filenames, but you're guaranteed to encounter a SNAFU at some point when a script you didn't write shall invariably fuxx0rs everything up).
Dot is accepted and common in filenames anyway.
I've got a scheme where I added checksums to my files (in the filename) using dashes but switching to underscore is on my TODO list. Basically I'll change this:
to: (YMMV as to whether adding the first x bits of whatever checksum you fancy on files you know aren't going to change works or doesn't work for you: it works for me and it's part of a suite of tools I made for myself for file integrity / backuping purpose / classification / etc.)Meanwhile, Windows uses UTF-16 everywhere internally, so "a" and "あ" are both 2 bytes large. You'd still have to exclude surrogate pairs and the Unicode control characters from a filename.
> It emits groups of five characters. Five is the sweet spot: 84⁵ = 4,182,119,424, only 2.6% short of 2³².
> That leaves enough room for a group to hold 32 bits about 95% of the time on uniformly random input, and 31 bits otherwise.
> The encoder looks at the next 31 bits. If their value is below 84⁵ - 2³¹, there’s room for a 32nd bit. Otherwise, it consumes just those 31 bits. Either way, the value fits in five base-84 digits.
Seems pretty awkward.
I think bit-aligned encodings (i.e. schemes where the output alphabets have a size that is a power of two) are the sweet-spot for encoding schemes due to simplicity of encoding and decoding compared to non-bit-aligned schemes like base84.
Base16 is verbose, but each symbol in the alphabet carries exactly 4 bits (hence why a single byte in hex is two symbols). This is nice for encoding bytes. Base64 requires two non-alphanumeric symbols such as + and / or - and _ because A-Za-z0-9 only provides 62 characters. Each output symbol encodes exactly 6 bits of the original input. But when we're encoding whole bytes we sometimes need to use padding (when the input is not a multiple of 3 bytes and that ambiguity is harmful in the context that decoding occurs in). Then there's Base32, which is what currently seems like a sweet spot to me and is what I'm considering using for identifiers in my own systems. It only requires 32 characters, so can easily take a range of A-Z0-9 or a-z0-9 excluding visually-similar characters. Like Base64 it can also require padding in some settings.
I quite like the general scheme used in the Bech32 and Bech32m framing idea, which allows for a human-readable prefix before a `1` and then the base32 data follows. This can be done because `1` is excluded from the Bech32 alphabet. This prefix can be used to differentiate between kinds of identifier, for example:
https://github.com/bitcoin/bips/blob/master/bip-0350.mediawi...
But to address the article directly, I'm not sure the complexity is worth the gains in the table in https://github.com/jedisct1/zig-base84#encoding, which states that we save ~6.5% of characters, and for 128 input bytes, the resulting Base64 string is 171 characters, while the Base84 string is 161-166 characters.
I also dislike that the chosen alphabet breaks text selection; GitHub very carefully picked their current token formats so that the whole token is selected with a double-click:
> One other neat thing about _ is it will reliably select the whole token when you double click on it. Other characters we considered are sometimes included in application word separators and thus will stop highlighting at that character. Try out double clicking this-random-text versus this_random_text!
https://github.blog/engineering/platform-security/behind-git...
For example, compare double-clicking on the article's Base84 alphabet:
> ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!#$%&'()+,-;=@[]^_`{}~
with the Bech32 Base32 alphabet:
> qpzry9x8gf2tvdw0s3jn54khce6mua7l
Regardless though we still have the fundamental issue that bytes encoded as Base16, Base32, Base64, or even Base91/Base84 will be _longer_ when encoded. Base32 can encode up to 159 bytes before hitting a filename limit of 255 bytes. I can't remember whether filenames are 16-bit on Windows (UCS-2? UTF-16?), but I speculate on average that maybe you could save a lot of bytes by first ensuring that filenames are UTF-8 before encrypting them, since filenames on a lot of computeres, especially ones running in the west, likely contain a lot of latin characters. You could even switch between encodings to get the best bit-packing in filenames.
At the moment it looks like turbocrypt forbids encrypted filenames that are too long: https://github.com/jedisct1/turbocrypt/blob/4905241d271e84a0...
I think turbocrypt could switch to using a surrogate file when an encrypted filename exceeds 255 bytes to allow for encrypting any filename permitted by the filesystem, while still preserving the property that the same filename in a different directory has the same encrypted filename. If an encrypted filename is too long to fit on the filesystem, maybe its hash could be stored as the filename instead (which is extremely unlikely to collide). Then we could store the full encrypted filename in a `.name` file in the same directory. We could do similar for directory names that are too long. We could then use a less dense but bit-aligned encoding scheme like Base32 with an all-lowercase alphabet without any punctuation to make filename text selection straightfoward. We would avoid all case sensitivity traps that can occur with things like Base64 and Base84. Let's pick two filename prefixes, say "tcf1" for "turbocrypt filename" and "tch1" for "turbocrypt filename hash". Then we could have a directory layout like this:
gocryptfs follows a similar idea for long filenames: https://github.com/rfjakob/gocryptfs-website/blob/master/doc...Once a system to support long filenames is implemented, the size of the alphabet used for encoding the filenames (like Base84) becomes less important; Base16, Base32, Base64, or another bit-aligned encoding scheme could be used.
As a very small added bonus, you could even implement a bit-aligned codec such as base16 or base64 using SIMD via a bunch of swizzling, shifting, bitmasking, and AND/XORing, making it very fast should the need arise.