David Larlet vor 10 Monaten
Ursprung
Commit
ee2cf76406
Signiert von: David Larlet <david@larlet.fr> GPG-Schlüssel-ID: 3E2953A359E7E7BD

+ 441
- 0
cache/2024/ce5fdc61fd66cdb9ce548fb543eba986/index.html Datei anzeigen

@@ -0,0 +1,441 @@
<!doctype html><!-- This is a valid HTML5 document. -->
<!-- Screen readers, SEO, extensions and so on. -->
<html lang="fr">
<!-- Has to be within the first 1024 bytes, hence before the `title` element
See: https://www.w3.org/TR/2012/CR-html5-20121217/document-metadata.html#charset -->
<meta charset="utf-8">
<!-- Why no `X-UA-Compatible` meta: https://stackoverflow.com/a/6771584 -->
<!-- The viewport meta is quite crowded and we are responsible for that.
See: https://codepen.io/tigt/post/meta-viewport-for-2015 -->
<meta name="viewport" content="width=device-width,initial-scale=1">
<!-- Required to make a valid HTML5 document. -->
<title>Unsigned Commits (archive) — David Larlet</title>
<meta name="description" content="Publication mise en cache pour en conserver une trace.">
<!-- That good ol' feed, subscribe :). -->
<link rel="alternate" type="application/atom+xml" title="Feed" href="/david/log/">
<!-- Generated from https://realfavicongenerator.net/ such a mess. -->
<link rel="apple-touch-icon" sizes="180x180" href="/static/david/icons2/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/static/david/icons2/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/static/david/icons2/favicon-16x16.png">
<link rel="manifest" href="/static/david/icons2/site.webmanifest">
<link rel="mask-icon" href="/static/david/icons2/safari-pinned-tab.svg" color="#07486c">
<link rel="shortcut icon" href="/static/david/icons2/favicon.ico">
<meta name="msapplication-TileColor" content="#f7f7f7">
<meta name="msapplication-config" content="/static/david/icons2/browserconfig.xml">
<meta name="theme-color" content="#f7f7f7" media="(prefers-color-scheme: light)">
<meta name="theme-color" content="#272727" media="(prefers-color-scheme: dark)">
<!-- Is that even respected? Retrospectively? What a shAItshow…
https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/ -->
<meta name="robots" content="noai, noimageai">
<!-- Documented, feel free to shoot an email. -->
<link rel="stylesheet" href="/static/david/css/style_2021-01-20.css">
<!-- See https://www.zachleat.com/web/comprehensive-webfonts/ for the trade-off. -->
<link rel="preload" href="/static/david/css/fonts/triplicate_t4_poly_regular.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)" crossorigin>
<link rel="preload" href="/static/david/css/fonts/triplicate_t4_poly_bold.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)" crossorigin>
<link rel="preload" href="/static/david/css/fonts/triplicate_t4_poly_italic.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)" crossorigin>
<link rel="preload" href="/static/david/css/fonts/triplicate_t3_regular.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: dark)" crossorigin>
<link rel="preload" href="/static/david/css/fonts/triplicate_t3_bold.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: dark)" crossorigin>
<link rel="preload" href="/static/david/css/fonts/triplicate_t3_italic.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: dark)" crossorigin>
<script>
function toggleTheme(themeName) {
document.documentElement.classList.toggle(
'forced-dark',
themeName === 'dark'
)
document.documentElement.classList.toggle(
'forced-light',
themeName === 'light'
)
}
const selectedTheme = localStorage.getItem('theme')
if (selectedTheme !== 'undefined') {
toggleTheme(selectedTheme)
}
</script>

<meta name="robots" content="noindex, nofollow">
<meta content="origin-when-cross-origin" name="referrer">
<!-- Canonical URL for SEO purposes -->
<link rel="canonical" href="https://blog.glyph.im/2024/01/unsigned-commits.html">

<body class="remarkdown h1-underline h2-underline h3-underline em-underscore hr-center ul-star pre-tick" data-instant-intensity="viewport-all">


<article>
<header>
<h1>Unsigned Commits</h1>
</header>
<nav>
<p class="center">
<a href="/david/" title="Aller à l’accueil"><svg class="icon icon-home">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-home"></use>
</svg> Accueil</a> •
<a href="https://blog.glyph.im/2024/01/unsigned-commits.html" title="Lien vers le contenu original">Source originale</a>
<br>
Mis en cache le 2024-01-25
</p>
</nav>
<hr>
<p>I am going to tell you why I don’t think you should sign your Git commits, even
though doing so with SSH keys is now easier than ever. But first, to
contextualize my objection, I have a brief hypothetical for you, and then a bit
of history from the evolution of security on the web.</p>
<hr>
<p><img alt="paper reading “Sign Here:” with a pen poised over it" src="https://blog.glyph.im/images/just-sign-here.jpeg"></p>
<hr>
<p>It seems like these days, everybody’s signing all different kinds of papers.</p>
<p>Bank forms, permission slips, power of attorney; it seems like if you want to
securely validate a document, you’ve gotta sign it.</p>
<p>So I have invented a machine that automatically signs every document on your
desk, just in case it needs your signature. Signing is good for security, so
you should probably get one, and turn it on, just in case something needs your
signature on it.</p>
<p>We also want to make sure that verifying your signature is easy, so we will
have them all notarized and duplicates stored permanently and publicly for
future reference.</p>
<p>No? Not interested?</p>
<hr>
<p>Hopefully, that sounded like a silly idea to you.</p>
<p>Most adults in modern civilization have learned that signing your name to a
document has an <em>effect</em>. It is not merely decorative; the words in the
document being signed have some specific meaning and can be enforced against
you.</p>
<p>In some ways the metaphor of “signing” in cryptography is bad. One does not
“sign” things with “keys” in real life. But here, it is spot on: a
cryptographic signature can have an effect.</p>
<p>It should be an <em>input</em> to some software, one that is acted upon. Software
does a thing differently depending on the presence or absence of a signature.
If it doesn’t, the signature probably shouldn’t be there.</p>
<hr>
<p>Consider the most venerable example of encryption and signing that we all deal
with every day: HTTPS. Many years ago, browsers would happily display
unencrypted web pages. The browser would also encrypt the connection, if the
server operator had paid for an expensive certificate and correctly configured
their server. If that operator messed up the encryption, it would pop up a
helpful dialog box that would tell the user “This website did something wrong
that you cannot possibly understand. Would you like to ignore this and keep
working?” with buttons that said “Yes” and “No”.</p>
<p>Of course, these are not the precise words that were written. The words, as
written, said things about “information you exchange” and “security
certificate” and “certifying authorities” but “Yes” and “No” were the words
that most users <em>read</em>. Predictably, most users just clicked “Yes”.</p>
<p>In the usual case, where users ignored these warnings, it meant that no user
ever got meaningful security from HTTPS. It was a component of the web stack
that did nothing but funnel money into the pockets of certificate authorities
and occasionally present annoying interruptions to users.</p>
<p>In the case where the user carefully read and honored these warnings in the
spirit they were intended, adding any sort of transport security to your
website was a potential liability. If you got everything perfectly correct,
nothing happened except the browser would display a picture of a <a href="https://www.wired.com/2016/11/googles-chrome-hackers-flip-webs-security-model/">small green
purse</a>. If
you made any small mistake, it would scare users off and thereby directly harm
your business. You would only want to do it if you were doing something that
put a big enough target on your site that you became unusually interesting to
attackers, or were required to do so by some contractual obligation like credit
card companies.</p>
<p>Keep in mind that the second case here is the <em>best</em> case.</p>
<p>In 2016, the browser makers noticed this problem and started taking some
<a href="https://security.googleblog.com/2016/09/moving-towards-more-secure-web.html">pretty aggressive
steps</a>
towards actually enforcing the security that HTTPS was supposed to provide, by
fixing the user interface to do the right thing. If your site didn’t have
security, it would be shown as “Not Secure”, a subtle warning that would
gradually escalate in intensity as time went on, correctly incentivizing site
operators to adopt transport security certificates. On the user interface
side, certificate errors would be significantly harder to disregard, making it
so that users who didn’t understand what they were seeing would actually be
stopped from doing the dangerous thing.</p>
<p>Nothing fundamental<sup id="fnref:1:unsigned-commits-2024-1"></sup> changed about the technical aspects of the
cryptographic primitives or constructions being used by HTTPS in this time
period, but <em>socially</em>, the meaning of an HTTP server signing and encrypting
its requests changed a lot.</p>
<hr>
<p>Now, let’s consider signing Git commits.</p>
<p>You may have heard that in some abstract sense you “should” be signing your
commits. GitHub puts a little green “verified” badge next to commits that are
signed, which is neat, I guess. They provide “security”. 1Password provides a
<a href="https://developer.1password.com/docs/ssh/git-commit-signing/">nice UI</a> for
setting it up. If you’re not a 1Password user, GitHub itself recommends you
<a href="https://docs.github.com/en/authentication/managing-commit-signature-verification/telling-git-about-your-signing-key#telling-git-about-your-ssh-key">put in just a few lines of
configuration</a>
to do it with either a GPG, SSH, or even an S/MIME key.</p>
<p>But while GitHub’s documentation quite lucidly tells you <em>how</em> to sign your
commits, its explanation of
<a href="https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification">why</a>
is somewhat less clear. Their purse is the word “Verified”; it’s still green.
If you enable “<a href="https://docs.github.com/en/authentication/managing-commit-signature-verification/displaying-verification-statuses-for-all-of-your-commits">vigilant
mode</a>”,
you can make the blank “no verification status” option say “Unverified”, but
not much else changes.</p>
<p>This is like the old-style HTTPS verification “Yes”/“No” dialog, except that
there is not even an interruption to your workflow. They might put the
“Unverified” status on there, but they’ve gone ahead and clicked “Yes” for you.</p>
<p>It is tempting to think that the “HTTPS” metaphor will map neatly onto Git
commit signatures. It was bad when the web wasn’t using HTTPS, and the next
step in that process was for <a href="https://en.wikipedia.org/wiki/Let%27s_Encrypt">Let’s
Encrypt</a> to come along and for
the browsers to fix their implementations. Getting your certificates properly
set up in the meanwhile and becoming familiar with the tools for properly doing
HTTPS was unambiguously a <em>good</em> thing for an engineer to do. I did, and I’m
quite glad I did so!</p>
<p>However, there is a significant difference: signing and encrypting an HTTPS
request is ephemeral; signing a Git commit is functionally permanent.</p>
<p>This ephemeral nature meant that errors in the early HTTPS landscape were
easily fixable. Earlier I mentioned that there was a time where you might
<em>not</em> want to set up HTTPS on your production web servers, because any small
screw-up would break your site and thereby your business. But if you were
really skilled and you could see the future coming, you could set up
monitoring, avoid these mistakes, and rapidly recover. These mistakes didn’t
need to badly break your site.</p>
<p>We <em>can</em> extend the analogy to HTTPS, but we have to take a detour into one of
the more unpleasant mistakes in HTTPS’s history: <a href="https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning">HTTP Public Key
Pinning</a>, or “HPKP”.
The idea with HPKP was that you could publish a record in an HTTP header where
your site commits<sup id="fnref:2:unsigned-commits-2024-1"></sup> to using certain certificate authorities for a period of
time, where that period of time could be “forever”. Attackers gonna attack,
and <a href="https://scotthelme.co.uk/using-security-features-to-do-bad-things/#usinghpkpforevil">attack they
did</a>.
Even without getting attacked, a site could easily commit “HPKP Suicide” where
they would pin the wrong certificate authority with a long timeline, and their
site was effectively gone for every browser that had ever seen those pins. As
a result, after a few years, HPKP was <a href="https://scotthelme.co.uk/hpkp-is-no-more/">completely removed from all
browsers</a>.</p>
<p>Git commit signing is even worse. With HPKP, you could easily make terrible
mistakes with permanent consequences even though you knew the <em>exact</em> meaning
of the data you were putting into the system at the time you were doing it.
With signed commits, you are saying something permanently, but you don’t really
know <em>what</em> it is that you’re saying.</p>
<hr>
<p><em>Today</em>, what is the benefit of signing a Git commit? GitHub might present it
as “Verified”. It’s worth noting that only <em>GitHub</em> will do this, since they
are the root of trust for this signing scheme. So, by signing commits and
registering your keys with GitHub, you are, at best, helping to lock in GitHub
as a permanent piece of infrastructure that is even harder to dislodge because
they are not only where your code is stored, but also the arbiters of whether
or not it is trustworthy.</p>
<p><em>In the future</em>, what is the possible security benefit? If we all collectively
decide we want Git to be more secure, then we will need to meaningfully treat
signed commits differently from unsigned ones.</p>
<p>There’s a long tail of unsigned commits several billion entries long. And
those are in the permanent record as much as the signed ones are, so future
tooling will have to be able to deal with them. If, as stewards of Git, we
wish to move towards a more secure Git, as the stewards of the web moved
towards a more secure web, we do <em>not</em> have the option that the web did. In
the browser, the meaning of a plain-text HTTP or incorrectly-signed HTTPS site
changed, in order to encourage the site’s operator to <em>change</em> the site to be
HTTPS.</p>
<p>In contrast, the meaning of an unsigned commit cannot change, because there are
zillions of unsigned commits lying around in critical infrastructure and we
need them to remain there. Commits cannot meaningfully be changed to become
signed retroactively. Unlike an online website, they are part of a historical
record, not an operating program. So we cannot establish the difference in
treatment by changing how unsigned commits are treated.</p>
<p>That means that tooling maintainers will need to provide some difference in
behavior that provides some incentive. With HTTPS where the binary choice was
clear: don’t present sites with incorrect, potentially compromised
configurations to users. The question was just <em>how</em> to achieve that. With
Git commits, the difference in treatment of a “trusted” commit is far less
clear.</p>
<p>If you will forgive me a slight straw-man here, one possible naive
interpretation is that a “trusted” signed commit is that it’s OK to run in CI.
Conveniently, it’s not simply “trusted” in a general sense. If you signed it,
it’s trusted to be <em>from you</em>, specifically. Surely it’s fine if we bill the
CI costs for validating the PR that includes that signed commit to your GitHub
account?</p>
<p>Now, someone can piggy-back off a 1-line typo fix that you made on top of an
unsigned commit to some large repo, making you implicitly responsible for
transitively signing all unsigned parent commits, even though you haven’t
looked at any of the code.</p>
<p>Remember, also, that the only central authority that is practically trustable
at this point is your GitHub account. That means that if you are using a
third-party CI system, even if you’re using a third-party Git host, you can
only run “trusted” code if GitHub is online and responding to requests for its
“get me the trusted signing keys for this user” API. This also adds a lot of
value to a GitHub credential breach, strongly motivating attackers to sneakily
attach their own keys to your account so that their commits in unrelated repos
can be “Verified” by you.</p>
<p>Let’s review the pros and cons of turning on commit signing <em>now</em>, before you
know what it is going to be used for:</p>
<table>
<thead>
<tr>
<th align="left">Pro</th>
<th align="left">Con</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">Green “Verified” badge</td>
<td align="left"><strong>Unknown, possibly unlimited future liability</strong> for the consequences of running code in a commit you signed</td>
</tr>
<tr>
<td align="left"><p></p></td>
<td align="left">Further implicitly cementing GitHub as a centralized trust authority in the open source world</td>
</tr>
<tr>
<td align="left"></td>
<td align="left">Introducing unknown reliability problems into infrastructure that relies on commit signatures</td>
</tr>
<tr>
<td align="left"></td>
<td align="left">Temporary breach of your GitHub credentials now lead to potentially permanent consequences if someone can smuggle a new trusted key in there</td>
</tr>
<tr>
<td align="left"></td>
<td align="left">New kinds of ongoing process overhead as commit-signing keys become new permanent load-bearing infrastructure, like “what do I do with expired keys”, “how often should I rotate these”, and so on</td>
</tr>
</tbody>
</table>
<p>I feel like the “Con” column is coming out ahead.</p>
<hr>
<p>That probably seemed like increasingly unhinged hyperbole, and it was.</p>
<p>In reality, the consequences are unlikely to be nearly so dramatic. The status
quo has a very high amount of inertia, and probably the “Verified” badge will
remain the only visible difference, except for a few repo-specific esoteric
workflows, like pushing trust verification into offline or sandboxed build
systems. I do still think that there is <em>some</em> potential for nefariousness
around the “unknown and unlimited” dimension of any future plans that might
rely on verifying signed commits, but any flaws are likely to be subtle attack
chains and not anything flashy and obvious.</p>
<p>But I think that one of the biggest problems in information security is a lack
of <a href="https://owasp.org/www-community/Threat_Modeling">threat modeling</a>. We
encrypt things, we sign things, we institute <a href="https://cryptosmith.com/password-sanity/exp-harmful/">rotation
policies</a> and
<a href="https://neal.fun/password-game/">elaborate</a> useless
<a href="https://xkcd.com/936/">rules</a> for passwords, because we are looking for a
“best practice” that is going to save us from having to think about what our
actual security problems are.</p>
<p>I think the actual harm of signing git commits is to perpetuate an engineering
culture of unquestioningly cargo-culting sophisticated and complex tools like
cryptographic signatures into new contexts where they have no use.</p>
<p>Just from a baseline utilitarian philosophical perspective, for a given action
A, all else being equal, it’s always better <em>not</em> to do A, because taking an
action always has <em>some</em> non-zero opportunity cost even if it is just the time
taken to do it. Epsilon cost and zero benefit is still a net harm. This is
even more true in the context of a complex system. Any action taken in
response to a rule in a system is going to interact with all the other rules in
that system. You have to pay complexity-rent on every new rule. So an
apparently-useless embellishment like signing commits can have potentially
far-reaching consequences in the future.</p>
<p>Git commit signing itself is not particularly consequential. I have probably
spent more time writing this blog post than the sum total of all the time
wasted by all programmers configuring their git clients to add useless
signatures; even the relatively modest readership of this blog will likely
transfer more data reading this post than all those signatures will take to
transmit to the various git clients that will read them. If I just convince
you not to sign your commits, I don’t think I’m coming out ahead in the
<a href="https://en.wikipedia.org/wiki/Felicific_calculus">felicific calculus</a> here.</p>
<p>What I am actually trying to point out here is that it is useful to <em>carefully
consider how to avoid adding junk complexity to your systems</em>. One area where
junk tends to leak in to designs and to cultures particularly easily is in
intimidating subjects like trust and safety, where it is easy to get anxious
and convince ourselves that piling on more <em>stuff</em> is safer than leaving things
simple.</p>
<p>If I can help you avoid adding even a little bit of unnecessary complexity, I
think it will have been well worth the cost of the writing, and the reading.</p>
<h2 id="acknowledgments">Acknowledgments</h2>
<p class="update-note">Thank you to <a href="https://www.patreon.com/creatorglyph">my patrons</a> who are
supporting my writing on this blog. If you like what you’ve read here and
you’d like to read more of it, or you’d like to support my <a href="https://github.com/glyph/">various open-source
endeavors</a>, you can <a href="https://www.patreon.com/join/8655595">support me on Patreon as
well</a>! I am also <a href="mailto:consulting@glyph.im">available for
consulting work</a> if you think your organization
could benefit from expertise on topics such as “What <em>else</em> should I <em>not</em> apply a cryptographic signature to?”.</p>
</article>


<hr>

<footer>
<p>
<a href="/david/" title="Aller à l’accueil"><svg class="icon icon-home">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-home"></use>
</svg> Accueil</a> •
<a href="/david/log/" title="Accès au flux RSS"><svg class="icon icon-rss2">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-rss2"></use>
</svg> Suivre</a> •
<a href="http://larlet.com" title="Go to my English profile" data-instant><svg class="icon icon-user-tie">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-user-tie"></use>
</svg> Pro</a> •
<a href="mailto:david%40larlet.fr" title="Envoyer un courriel"><svg class="icon icon-mail">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-mail"></use>
</svg> Email</a> •
<abbr class="nowrap" title="Hébergeur : Alwaysdata, 62 rue Tiquetonne 75002 Paris, +33184162340"><svg class="icon icon-hammer2">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-hammer2"></use>
</svg> Légal</abbr>
</p>
<template id="theme-selector">
<form>
<fieldset>
<legend><svg class="icon icon-brightness-contrast">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-brightness-contrast"></use>
</svg> Thème</legend>
<label>
<input type="radio" value="auto" name="chosen-color-scheme" checked> Auto
</label>
<label>
<input type="radio" value="dark" name="chosen-color-scheme"> Foncé
</label>
<label>
<input type="radio" value="light" name="chosen-color-scheme"> Clair
</label>
</fieldset>
</form>
</template>
</footer>
<script src="/static/david/js/instantpage-5.1.0.min.js" type="module"></script>
<script>
function loadThemeForm(templateName) {
const themeSelectorTemplate = document.querySelector(templateName)
const form = themeSelectorTemplate.content.firstElementChild
themeSelectorTemplate.replaceWith(form)

form.addEventListener('change', (e) => {
const chosenColorScheme = e.target.value
localStorage.setItem('theme', chosenColorScheme)
toggleTheme(chosenColorScheme)
})

const selectedTheme = localStorage.getItem('theme')
if (selectedTheme && selectedTheme !== 'undefined') {
form.querySelector(`[value="${selectedTheme}"]`).checked = true
}
}

const prefersColorSchemeDark = '(prefers-color-scheme: dark)'
window.addEventListener('load', () => {
let hasDarkRules = false
for (const styleSheet of Array.from(document.styleSheets)) {
let mediaRules = []
for (const cssRule of styleSheet.cssRules) {
if (cssRule.type !== CSSRule.MEDIA_RULE) {
continue
}
// WARNING: Safari does not have/supports `conditionText`.
if (cssRule.conditionText) {
if (cssRule.conditionText !== prefersColorSchemeDark) {
continue
}
} else {
if (cssRule.cssText.startsWith(prefersColorSchemeDark)) {
continue
}
}
mediaRules = mediaRules.concat(Array.from(cssRule.cssRules))
}

// WARNING: do not try to insert a Rule to a styleSheet you are
// currently iterating on, otherwise the browser will be stuck
// in a infinite loop…
for (const mediaRule of mediaRules) {
styleSheet.insertRule(mediaRule.cssText)
hasDarkRules = true
}
}
if (hasDarkRules) {
loadThemeForm('#theme-selector')
}
})
</script>
</body>
</html>

+ 270
- 0
cache/2024/ce5fdc61fd66cdb9ce548fb543eba986/index.md Datei anzeigen

@@ -0,0 +1,270 @@
title: Unsigned Commits
url: https://blog.glyph.im/2024/01/unsigned-commits.html
hash_url: ce5fdc61fd66cdb9ce548fb543eba986
archive_date: 2024-01-25

<p>I am going to tell you why I don’t think you should sign your Git commits, even
though doing so with SSH keys is now easier than ever. But first, to
contextualize my objection, I have a brief hypothetical for you, and then a bit
of history from the evolution of security on the web.</p>
<hr>
<p><img alt="paper reading “Sign Here:” with a pen poised over it" src="https://blog.glyph.im/images/just-sign-here.jpeg"></p>
<hr>
<p>It seems like these days, everybody’s signing all different kinds of papers.</p>
<p>Bank forms, permission slips, power of attorney; it seems like if you want to
securely validate a document, you’ve gotta sign it.</p>
<p>So I have invented a machine that automatically signs every document on your
desk, just in case it needs your signature. Signing is good for security, so
you should probably get one, and turn it on, just in case something needs your
signature on it.</p>
<p>We also want to make sure that verifying your signature is easy, so we will
have them all notarized and duplicates stored permanently and publicly for
future reference.</p>
<p>No? Not interested?</p>
<hr>
<p>Hopefully, that sounded like a silly idea to you.</p>
<p>Most adults in modern civilization have learned that signing your name to a
document has an <em>effect</em>. It is not merely decorative; the words in the
document being signed have some specific meaning and can be enforced against
you.</p>
<p>In some ways the metaphor of “signing” in cryptography is bad. One does not
“sign” things with “keys” in real life. But here, it is spot on: a
cryptographic signature can have an effect.</p>
<p>It should be an <em>input</em> to some software, one that is acted upon. Software
does a thing differently depending on the presence or absence of a signature.
If it doesn’t, the signature probably shouldn’t be there.</p>
<hr>
<p>Consider the most venerable example of encryption and signing that we all deal
with every day: HTTPS. Many years ago, browsers would happily display
unencrypted web pages. The browser would also encrypt the connection, if the
server operator had paid for an expensive certificate and correctly configured
their server. If that operator messed up the encryption, it would pop up a
helpful dialog box that would tell the user “This website did something wrong
that you cannot possibly understand. Would you like to ignore this and keep
working?” with buttons that said “Yes” and “No”.</p>
<p>Of course, these are not the precise words that were written. The words, as
written, said things about “information you exchange” and “security
certificate” and “certifying authorities” but “Yes” and “No” were the words
that most users <em>read</em>. Predictably, most users just clicked “Yes”.</p>
<p>In the usual case, where users ignored these warnings, it meant that no user
ever got meaningful security from HTTPS. It was a component of the web stack
that did nothing but funnel money into the pockets of certificate authorities
and occasionally present annoying interruptions to users.</p>
<p>In the case where the user carefully read and honored these warnings in the
spirit they were intended, adding any sort of transport security to your
website was a potential liability. If you got everything perfectly correct,
nothing happened except the browser would display a picture of a <a href="https://www.wired.com/2016/11/googles-chrome-hackers-flip-webs-security-model/">small green
purse</a>. If
you made any small mistake, it would scare users off and thereby directly harm
your business. You would only want to do it if you were doing something that
put a big enough target on your site that you became unusually interesting to
attackers, or were required to do so by some contractual obligation like credit
card companies.</p>
<p>Keep in mind that the second case here is the <em>best</em> case.</p>
<p>In 2016, the browser makers noticed this problem and started taking some
<a href="https://security.googleblog.com/2016/09/moving-towards-more-secure-web.html">pretty aggressive
steps</a>
towards actually enforcing the security that HTTPS was supposed to provide, by
fixing the user interface to do the right thing. If your site didn’t have
security, it would be shown as “Not Secure”, a subtle warning that would
gradually escalate in intensity as time went on, correctly incentivizing site
operators to adopt transport security certificates. On the user interface
side, certificate errors would be significantly harder to disregard, making it
so that users who didn’t understand what they were seeing would actually be
stopped from doing the dangerous thing.</p>
<p>Nothing fundamental<sup id="fnref:1:unsigned-commits-2024-1"></sup> changed about the technical aspects of the
cryptographic primitives or constructions being used by HTTPS in this time
period, but <em>socially</em>, the meaning of an HTTP server signing and encrypting
its requests changed a lot.</p>
<hr>
<p>Now, let’s consider signing Git commits.</p>
<p>You may have heard that in some abstract sense you “should” be signing your
commits. GitHub puts a little green “verified” badge next to commits that are
signed, which is neat, I guess. They provide “security”. 1Password provides a
<a href="https://developer.1password.com/docs/ssh/git-commit-signing/">nice UI</a> for
setting it up. If you’re not a 1Password user, GitHub itself recommends you
<a href="https://docs.github.com/en/authentication/managing-commit-signature-verification/telling-git-about-your-signing-key#telling-git-about-your-ssh-key">put in just a few lines of
configuration</a>
to do it with either a GPG, SSH, or even an S/MIME key.</p>
<p>But while GitHub’s documentation quite lucidly tells you <em>how</em> to sign your
commits, its explanation of
<a href="https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification">why</a>
is somewhat less clear. Their purse is the word “Verified”; it’s still green.
If you enable “<a href="https://docs.github.com/en/authentication/managing-commit-signature-verification/displaying-verification-statuses-for-all-of-your-commits">vigilant
mode</a>”,
you can make the blank “no verification status” option say “Unverified”, but
not much else changes.</p>
<p>This is like the old-style HTTPS verification “Yes”/“No” dialog, except that
there is not even an interruption to your workflow. They might put the
“Unverified” status on there, but they’ve gone ahead and clicked “Yes” for you.</p>
<p>It is tempting to think that the “HTTPS” metaphor will map neatly onto Git
commit signatures. It was bad when the web wasn’t using HTTPS, and the next
step in that process was for <a href="https://en.wikipedia.org/wiki/Let%27s_Encrypt">Let’s
Encrypt</a> to come along and for
the browsers to fix their implementations. Getting your certificates properly
set up in the meanwhile and becoming familiar with the tools for properly doing
HTTPS was unambiguously a <em>good</em> thing for an engineer to do. I did, and I’m
quite glad I did so!</p>
<p>However, there is a significant difference: signing and encrypting an HTTPS
request is ephemeral; signing a Git commit is functionally permanent.</p>
<p>This ephemeral nature meant that errors in the early HTTPS landscape were
easily fixable. Earlier I mentioned that there was a time where you might
<em>not</em> want to set up HTTPS on your production web servers, because any small
screw-up would break your site and thereby your business. But if you were
really skilled and you could see the future coming, you could set up
monitoring, avoid these mistakes, and rapidly recover. These mistakes didn’t
need to badly break your site.</p>
<p>We <em>can</em> extend the analogy to HTTPS, but we have to take a detour into one of
the more unpleasant mistakes in HTTPS’s history: <a href="https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning">HTTP Public Key
Pinning</a>, or “HPKP”.
The idea with HPKP was that you could publish a record in an HTTP header where
your site commits<sup id="fnref:2:unsigned-commits-2024-1"></sup> to using certain certificate authorities for a period of
time, where that period of time could be “forever”. Attackers gonna attack,
and <a href="https://scotthelme.co.uk/using-security-features-to-do-bad-things/#usinghpkpforevil">attack they
did</a>.
Even without getting attacked, a site could easily commit “HPKP Suicide” where
they would pin the wrong certificate authority with a long timeline, and their
site was effectively gone for every browser that had ever seen those pins. As
a result, after a few years, HPKP was <a href="https://scotthelme.co.uk/hpkp-is-no-more/">completely removed from all
browsers</a>.</p>
<p>Git commit signing is even worse. With HPKP, you could easily make terrible
mistakes with permanent consequences even though you knew the <em>exact</em> meaning
of the data you were putting into the system at the time you were doing it.
With signed commits, you are saying something permanently, but you don’t really
know <em>what</em> it is that you’re saying.</p>
<hr>
<p><em>Today</em>, what is the benefit of signing a Git commit? GitHub might present it
as “Verified”. It’s worth noting that only <em>GitHub</em> will do this, since they
are the root of trust for this signing scheme. So, by signing commits and
registering your keys with GitHub, you are, at best, helping to lock in GitHub
as a permanent piece of infrastructure that is even harder to dislodge because
they are not only where your code is stored, but also the arbiters of whether
or not it is trustworthy.</p>
<p><em>In the future</em>, what is the possible security benefit? If we all collectively
decide we want Git to be more secure, then we will need to meaningfully treat
signed commits differently from unsigned ones.</p>
<p>There’s a long tail of unsigned commits several billion entries long. And
those are in the permanent record as much as the signed ones are, so future
tooling will have to be able to deal with them. If, as stewards of Git, we
wish to move towards a more secure Git, as the stewards of the web moved
towards a more secure web, we do <em>not</em> have the option that the web did. In
the browser, the meaning of a plain-text HTTP or incorrectly-signed HTTPS site
changed, in order to encourage the site’s operator to <em>change</em> the site to be
HTTPS.</p>
<p>In contrast, the meaning of an unsigned commit cannot change, because there are
zillions of unsigned commits lying around in critical infrastructure and we
need them to remain there. Commits cannot meaningfully be changed to become
signed retroactively. Unlike an online website, they are part of a historical
record, not an operating program. So we cannot establish the difference in
treatment by changing how unsigned commits are treated.</p>
<p>That means that tooling maintainers will need to provide some difference in
behavior that provides some incentive. With HTTPS where the binary choice was
clear: don’t present sites with incorrect, potentially compromised
configurations to users. The question was just <em>how</em> to achieve that. With
Git commits, the difference in treatment of a “trusted” commit is far less
clear.</p>
<p>If you will forgive me a slight straw-man here, one possible naive
interpretation is that a “trusted” signed commit is that it’s OK to run in CI.
Conveniently, it’s not simply “trusted” in a general sense. If you signed it,
it’s trusted to be <em>from you</em>, specifically. Surely it’s fine if we bill the
CI costs for validating the PR that includes that signed commit to your GitHub
account?</p>
<p>Now, someone can piggy-back off a 1-line typo fix that you made on top of an
unsigned commit to some large repo, making you implicitly responsible for
transitively signing all unsigned parent commits, even though you haven’t
looked at any of the code.</p>
<p>Remember, also, that the only central authority that is practically trustable
at this point is your GitHub account. That means that if you are using a
third-party CI system, even if you’re using a third-party Git host, you can
only run “trusted” code if GitHub is online and responding to requests for its
“get me the trusted signing keys for this user” API. This also adds a lot of
value to a GitHub credential breach, strongly motivating attackers to sneakily
attach their own keys to your account so that their commits in unrelated repos
can be “Verified” by you.</p>
<p>Let’s review the pros and cons of turning on commit signing <em>now</em>, before you
know what it is going to be used for:</p>
<table>
<thead>
<tr>
<th align="left">Pro</th>
<th align="left">Con</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">Green “Verified” badge</td>
<td align="left"><strong>Unknown, possibly unlimited future liability</strong> for the consequences of running code in a commit you signed</td>
</tr>
<tr>
<td align="left"><p></p></td>
<td align="left">Further implicitly cementing GitHub as a centralized trust authority in the open source world</td>
</tr>
<tr>
<td align="left"></td>
<td align="left">Introducing unknown reliability problems into infrastructure that relies on commit signatures</td>
</tr>
<tr>
<td align="left"></td>
<td align="left">Temporary breach of your GitHub credentials now lead to potentially permanent consequences if someone can smuggle a new trusted key in there</td>
</tr>
<tr>
<td align="left"></td>
<td align="left">New kinds of ongoing process overhead as commit-signing keys become new permanent load-bearing infrastructure, like “what do I do with expired keys”, “how often should I rotate these”, and so on</td>
</tr>
</tbody>
</table>
<p>I feel like the “Con” column is coming out ahead.</p>
<hr>
<p>That probably seemed like increasingly unhinged hyperbole, and it was.</p>
<p>In reality, the consequences are unlikely to be nearly so dramatic. The status
quo has a very high amount of inertia, and probably the “Verified” badge will
remain the only visible difference, except for a few repo-specific esoteric
workflows, like pushing trust verification into offline or sandboxed build
systems. I do still think that there is <em>some</em> potential for nefariousness
around the “unknown and unlimited” dimension of any future plans that might
rely on verifying signed commits, but any flaws are likely to be subtle attack
chains and not anything flashy and obvious.</p>
<p>But I think that one of the biggest problems in information security is a lack
of <a href="https://owasp.org/www-community/Threat_Modeling">threat modeling</a>. We
encrypt things, we sign things, we institute <a href="https://cryptosmith.com/password-sanity/exp-harmful/">rotation
policies</a> and
<a href="https://neal.fun/password-game/">elaborate</a> useless
<a href="https://xkcd.com/936/">rules</a> for passwords, because we are looking for a
“best practice” that is going to save us from having to think about what our
actual security problems are.</p>
<p>I think the actual harm of signing git commits is to perpetuate an engineering
culture of unquestioningly cargo-culting sophisticated and complex tools like
cryptographic signatures into new contexts where they have no use.</p>
<p>Just from a baseline utilitarian philosophical perspective, for a given action
A, all else being equal, it’s always better <em>not</em> to do A, because taking an
action always has <em>some</em> non-zero opportunity cost even if it is just the time
taken to do it. Epsilon cost and zero benefit is still a net harm. This is
even more true in the context of a complex system. Any action taken in
response to a rule in a system is going to interact with all the other rules in
that system. You have to pay complexity-rent on every new rule. So an
apparently-useless embellishment like signing commits can have potentially
far-reaching consequences in the future.</p>
<p>Git commit signing itself is not particularly consequential. I have probably
spent more time writing this blog post than the sum total of all the time
wasted by all programmers configuring their git clients to add useless
signatures; even the relatively modest readership of this blog will likely
transfer more data reading this post than all those signatures will take to
transmit to the various git clients that will read them. If I just convince
you not to sign your commits, I don’t think I’m coming out ahead in the
<a href="https://en.wikipedia.org/wiki/Felicific_calculus">felicific calculus</a> here.</p>
<p>What I am actually trying to point out here is that it is useful to <em>carefully
consider how to avoid adding junk complexity to your systems</em>. One area where
junk tends to leak in to designs and to cultures particularly easily is in
intimidating subjects like trust and safety, where it is easy to get anxious
and convince ourselves that piling on more <em>stuff</em> is safer than leaving things
simple.</p>
<p>If I can help you avoid adding even a little bit of unnecessary complexity, I
think it will have been well worth the cost of the writing, and the reading.</p>
<h2 id="acknowledgments">Acknowledgments</h2>
<p class="update-note">Thank you to <a href="https://www.patreon.com/creatorglyph">my patrons</a> who are
supporting my writing on this blog. If you like what you’ve read here and
you’d like to read more of it, or you’d like to support my <a href="https://github.com/glyph/">various open-source
endeavors</a>, you can <a href="https://www.patreon.com/join/8655595">support me on Patreon as
well</a>! I am also <a href="mailto:consulting@glyph.im">available for
consulting work</a> if you think your organization
could benefit from expertise on topics such as “What <em>else</em> should I <em>not</em> apply a cryptographic signature to?”.</p>

+ 2
- 2
cache/2024/fd6eda56671045e0c1e2d215e07f1a6f/index.html Datei anzeigen

@@ -9,7 +9,7 @@
See: https://codepen.io/tigt/post/meta-viewport-for-2015 -->
<meta name="viewport" content="width=device-width,initial-scale=1">
<!-- Required to make a valid HTML5 document. -->
<title> (archive) — David Larlet</title>
<title>EffVer: Version your code by the effort required to upgrade (archive) — David Larlet</title>
<meta name="description" content="Publication mise en cache pour en conserver une trace.">
<!-- That good ol' feed, subscribe :). -->
<link rel="alternate" type="application/atom+xml" title="Feed" href="/david/log/">
@@ -63,7 +63,7 @@

<article>
<header>
<h1></h1>
<h1>EffVer: Version your code by the effort required to upgrade</h1>
</header>
<nav>
<p class="center">

+ 1
- 1
cache/2024/fd6eda56671045e0c1e2d215e07f1a6f/index.md Datei anzeigen

@@ -1,4 +1,4 @@
title:
title: EffVer: Version your code by the effort required to upgrade
url: https://jacobtomlinson.dev/effver/
hash_url: fd6eda56671045e0c1e2d215e07f1a6f
archive_date: 2024-01-18

+ 326
- 0
cache/2024/ffaf50bf5d5e4cf870a618b518ee5ba7/index.html Datei anzeigen

@@ -0,0 +1,326 @@
<!doctype html><!-- This is a valid HTML5 document. -->
<!-- Screen readers, SEO, extensions and so on. -->
<html lang="fr">
<!-- Has to be within the first 1024 bytes, hence before the `title` element
See: https://www.w3.org/TR/2012/CR-html5-20121217/document-metadata.html#charset -->
<meta charset="utf-8">
<!-- Why no `X-UA-Compatible` meta: https://stackoverflow.com/a/6771584 -->
<!-- The viewport meta is quite crowded and we are responsible for that.
See: https://codepen.io/tigt/post/meta-viewport-for-2015 -->
<meta name="viewport" content="width=device-width,initial-scale=1">
<!-- Required to make a valid HTML5 document. -->
<title>Portable EPUBs (archive) — David Larlet</title>
<meta name="description" content="Publication mise en cache pour en conserver une trace.">
<!-- That good ol' feed, subscribe :). -->
<link rel="alternate" type="application/atom+xml" title="Feed" href="/david/log/">
<!-- Generated from https://realfavicongenerator.net/ such a mess. -->
<link rel="apple-touch-icon" sizes="180x180" href="/static/david/icons2/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/static/david/icons2/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/static/david/icons2/favicon-16x16.png">
<link rel="manifest" href="/static/david/icons2/site.webmanifest">
<link rel="mask-icon" href="/static/david/icons2/safari-pinned-tab.svg" color="#07486c">
<link rel="shortcut icon" href="/static/david/icons2/favicon.ico">
<meta name="msapplication-TileColor" content="#f7f7f7">
<meta name="msapplication-config" content="/static/david/icons2/browserconfig.xml">
<meta name="theme-color" content="#f7f7f7" media="(prefers-color-scheme: light)">
<meta name="theme-color" content="#272727" media="(prefers-color-scheme: dark)">
<!-- Is that even respected? Retrospectively? What a shAItshow…
https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/ -->
<meta name="robots" content="noai, noimageai">
<!-- Documented, feel free to shoot an email. -->
<link rel="stylesheet" href="/static/david/css/style_2021-01-20.css">
<!-- See https://www.zachleat.com/web/comprehensive-webfonts/ for the trade-off. -->
<link rel="preload" href="/static/david/css/fonts/triplicate_t4_poly_regular.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)" crossorigin>
<link rel="preload" href="/static/david/css/fonts/triplicate_t4_poly_bold.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)" crossorigin>
<link rel="preload" href="/static/david/css/fonts/triplicate_t4_poly_italic.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)" crossorigin>
<link rel="preload" href="/static/david/css/fonts/triplicate_t3_regular.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: dark)" crossorigin>
<link rel="preload" href="/static/david/css/fonts/triplicate_t3_bold.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: dark)" crossorigin>
<link rel="preload" href="/static/david/css/fonts/triplicate_t3_italic.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: dark)" crossorigin>
<script>
function toggleTheme(themeName) {
document.documentElement.classList.toggle(
'forced-dark',
themeName === 'dark'
)
document.documentElement.classList.toggle(
'forced-light',
themeName === 'light'
)
}
const selectedTheme = localStorage.getItem('theme')
if (selectedTheme !== 'undefined') {
toggleTheme(selectedTheme)
}
</script>

<meta name="robots" content="noindex, nofollow">
<meta content="origin-when-cross-origin" name="referrer">
<!-- Canonical URL for SEO purposes -->
<link rel="canonical" href="https://willcrichton.net/notes/portable-epubs/">

<body class="remarkdown h1-underline h2-underline h3-underline em-underscore hr-center ul-star pre-tick" data-instant-intensity="viewport-all">


<article>
<header>
<h1>Portable EPUBs</h1>
</header>
<nav>
<p class="center">
<a href="/david/" title="Aller à l’accueil"><svg class="icon icon-home">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-home"></use>
</svg> Accueil</a> •
<a href="https://willcrichton.net/notes/portable-epubs/" title="Lien vers le contenu original">Source originale</a>
<br>
Mis en cache le 2024-01-25
</p>
</nav>
<hr>
<p>Despite decades of advances in document rendering technology, most of the world's documents are stuck in the 1990s due to the limitations of PDF. Yet, modern document formats like HTML have yet to provide a competitive alternative to PDF. This post explores why this is the case, and proposes a way forward based on the EPUB format.</p>
<h2>The Good and Bad of PDF</h2>
<p>PDF is the de facto file format for reading and sharing digital documents like papers, textbooks, and flyers. People use the PDF format for several reasons:</p>
<ul>
<li>PDFs are self-contained. A PDF is a single file that contains all the images, fonts, and other data needed to render it. It's easy to pass around a PDF. A PDF is unlikely to be missing some critical dependency on your computer.</li>
<li>PDFs are rendered consistently. A PDF specifies precisely how it should be rendered, so a PDF author can be confident that a reader will see the same document under any conditions.</li>
<li>PDFs are stable over time. PDFs from decades ago still render the same today. PDFs have a relatively stable standard. PDFs cannot be easily edited.</li>
</ul>
<p>Yet, in the 32 years since the initial release of PDF, a lot has changed. People print out documents less and less. People use phones, tablets, and e-readers to read digital documents. The internet happened; web browsers now provide a platform for rendering rich documents. These changes have laid bare the limitations of PDF:</p>
<ul>
<li>PDFs cannot easily adapt to different screen sizes. Most PDFs are designed to mimic 8.5x11" paper (or worse, 145,161 km2). These PDFs are readable on a computer monitor, but they are less readable on a tablet, and far less readable on a phone.</li>
<li>PDFs cannot be easily understood by programs. A plain PDF is just a scattered sequence of lines and characters. For accessibility, screen readers may not know which order to read through the text. For data extraction, scraping tables out of a PDF is an open area of research.</li>
<li>PDFs cannot easily express interaction. PDFs were primarily designed as static documents that cannot react to user input beyond filling in forms.</li>
</ul>
<p>These pros and cons can be traced back to one key fact: the PDF representation of a document is fundamentally unstructured. A PDF consists of commands like:</p>
<ul>
<li>Move the cursor to the right by 0.5 inches.</li>
<li>Set the current font color to black.</li>
<li>Draw the text "Hello World" at the current position.</li>
</ul>
<p>PDF commands are unstructured because a document's organization is only clear to a person looking at the rendered document, and not clear from the commands themselves. Reflowing, accessibility, data extraction, and interaction all rely on programmatically understanding the structure of a document. Hence, these aspects are not easy to integrate with PDFs.</p>
<p>This raises the question: how can we design digital documents with the benefits of PDFs but without the limitations?</p>
<h2>Can't We Just Fix PDF?</h2>
<p>A simple answer is to improve the PDF format. After all, we already have billions of PDFs — why reinvent the wheel?</p>
<p>The designers of PDF are well aware of its limitations. I carefully hedged each bullet with "easily", because PDF does make it possible to overcome each limitation, at least partially. PDFs can be annotated with their logical structure to create a tagged PDF. Most PDF exporters will not add tags automatically — the simplest option is to use Adobe's subscription-only Acrobat Pro, which provides an "Automatically tag PDF" action. For example, here is a recent paper of mine with added tags:
A LaTeX-generated paper with automatically added tags.</p>
<p>If you squint, you can see that the logical structure closely resembles the HTML document model. The document has sections, headings, paragraphs, and links. Adobe characterizes the logical structure as an accessibility feature, but it has other benefits. You may be surprised to know that Adobe Acrobat allows you to reflow tagged PDFs at different screen sizes. You may be unsurprised to know that reflowing does not always work well. For example:
A section of the paper in its default fixed layout. Note that the second paragraph is wrapped around the code snippet.
The same section of the paper after reflowing to a smaller width. Note that the code is now interleaved with the second paragraph.</p>
<p>In theory, these issues could be fixed. If the world's PDF exporters could be modified to include logical structure. If Adobe's reflowing algorithm could be improved to fix its edge cases. If the reflowing algorithm could be specified, and if Adobe were willing to release it publicly, and if it were implemented in each PDF viewer. And that doesn't even cover interaction! So in practice, I don't think we can just fix the PDF format, at least within a reasonable time frame.</p>
<h2>The Good and Bad of HTML</h2>
<p>In the meantime, we already have a structured document format which can be flexibly and interactively rendered: HTML (and CSS and Javascript, but here just collectively referred to as HTML). The HTML format provides almost exactly the inverse advantages and disadvantages of PDF.</p>
<ul>
<li>HTML can more easily adapt to different screen sizes. Over the last 20 years, web developers and browser vendors have created a wide array of techniques for responsive design.</li>
<li>HTML can be more easily understood by a program. HTML provides both an inherent structure plus additional attributes to support accessibility tools.</li>
<li>HTML can more easily express interaction. People have used HTML to produce amazing interactive documents that would be impossible in PDF. Think: Distill.pub, Explorable Explanations, Bartosz Ciechanowski, and Bret Victor, just to name a few.</li>
</ul>
<p>Again, these advantages are hedged with "more easily". One can easily produce a convoluted or inaccessible HTML document. But on balance, these aspects are more true than not compared to PDF. However, HTML is lacking where PDF shines:</p>
<ul>
<li>HTML is not self-contained. HTML files may contain URL references to external files that may be hosted on a server. One can rarely download an HTML file and have it render correctly without an internet connection.</li>
<li>HTML is not always rendered consistently. HTML's dynamic layout means that an author may not see the same document as a reader. Moreover, HTML layout is not fully specified, so browsers may differ in their implementation.</li>
<li>HTML is not fully stable over time. Browsers try to maintain backwards compatibility (come on and slam!), but the HTML format is still evolving. The HTML standard is a "living standard" due to the rapidly changing needs and feature sets of modern browsers.</li>
</ul>
<p>So I've been thinking: how can we design HTML documents to gain the benefits of PDFs without losing the key strengths of HTML? The rest of this document will present some early prototypes and tentative proposals in this direction.</p>
<h2>Self-Contained HTML with EPUB</h2>
<p>First, how can we make HTML documents self-contained? This is an old problem with many potential solutions. WARC, webarchive, and MHTML are all file formats designed to contain all the resources needed to render a web page. But these formats are more designed for snapshotting an existing website, rather than serving as a single source of truth for a web document. From my research, the most sensible format for this purpose is EPUB.</p>
<p>EPUB is a "distribution and interchange format for digital publications and documents", per the EPUB 3 Overview. Reductively, an EPUB is a ZIP archive of web files: HTML, CSS, JS, and assets like images and fonts. On a technical level, what distinguishes EPUB from archival formats is that EPUB includes well-specified files that describe metadata about a document. On a social level, EPUB appears to be the HTML publication format with the most adoption and momentum in 2024, compared to moribund formats like Mobi.</p>
<p>The EPUB spec has all the gory details, but to give you a rough sense, a sample EPUB might have the following file structure:</p>
<pre><code>sample.epub
├── META-INF
│ └── container.xml
└── EPUB
├── package.opf
├── nav.xhtml
├── chapter1.xhtml
├── chapter2.xhtml
└── img
└── sample.jpg
</code></pre>
<p>An EPUB contains content documents (like chapter1.xhtml and chapter2.xhtml) which contain the core HTML content. Content documents can contain relative links to assets in the EPUB, like img/sample.jpg. The navigation document (nav.xhtml) provides a table of contents, and the package document (package.opf) provides metadata about the document. These files collectively define one "rendition" of the whole document, and the container file (container.xml) points to each rendition contained in the EPUB.</p>
<p>The EPUB format optimizes for machine-readable content and metadata. HTML content is required to be in XML format (hence, XHTML). Document metadata like the title and author is provided in structured form in the package document. The navigation document has a carefully prescribed tag structure so the TOC can be consistently extracted.</p>
<p>Overall, EPUB's structured format makes it a solid candidate for a single-file HTML document container. However, EPUB is not a silver bullet. EPUB is quite permissive in what kinds of content can be put into a content document.</p>
<p>For example, a major issue for self-containment is that EPUB content can embed external assets. A content document can legally include an image or font file whose src is a URL to a hosted server. This is not hypothetical, either; as of the time of writing, Google Doc's EPUB exporter will emit CSS that will @include external Google Fonts files. The problem is that such an EPUB will not render correctly without an internet connection, nor will it render correctly if Google changes the URLs of its font files.</p>
<p>Hence, I will propose a new format which I call a portable EPUB, which is an EPUB with additional requirements and recommendations to improve PDF-like portability. The first requirement is:
Local asset requirement: All assets (like images, scripts, and fonts) embedded in a content document of a portable EPUB must refer to local files included in the EPUB. Hyperlinks to external files are permissible.</p>
<h2>Consistency vs. Flexibility in Rendering</h2>
<p>There is a fundamental tension between consistency and flexibility in document rendering. A PDF is consistent because it is designed to render in one way: one layout, one choice of fonts, one choice of colors, one pagination, and so on. Consistency is desirable because an author can be confident that their document will look good for a reader (or at least, not look bad). Consistency has subtler benefits — because a PDF is chunked into a consistent set of pages, a passage can be cited by referring to the page containing the passage.</p>
<p>On the other hand, flexibility is desirable because people want to read documents under different conditions. Device conditions include screen size (from phone to monitor) and screen capabilities (E-ink vs. LCD). Some readers may prefer larger fonts or higher contrasts for visibility, alternative color schemes for color blindness, or alternative font faces for dyslexia. Sufficiently flexible documents can even permit readers to select a level of detail appropriate for their background (here's an example).</p>
<p>Finding a balance between consistency and flexibility is arguably the most fundamental design challenge in attempting to replace PDF with EPUB. To navigate this trade-off, we first need to talk about EPUB reading systems, or the tools that render an EPUB for human consumption. To get a sense of variation between reading systems, I tried rendering this post as an EPUB (without any styling, just HTML) on four systems: Calibre, Adobe Digital Editions, Apple Books, and Amazon Kindle. This is how the first page looks on each system (omitting Calibre because it looked the same as Adobe Digital Editions):</p>
<ul>
<li>Adobe Digital Editions</li>
<li>Apple Books</li>
<li>Amazon Kindle</li>
</ul>
<p>Calibre and Adobe Digital Editions both render the document in a plain web view, as if you opened the HTML file directly in the browser. Apple Books applies some styling, using the New York font by default and changing link decorations. Amazon Kindle increases the line height and also uses my Kindle's globally-configured default font, Palatino.</p>
<p>As you can see, an EPUB may look quite different on different reading systems. The variation displayed above seems reasonable to me. But how different is too different? For instance, I was recently reading A History of Writing on my Kindle. Here's an example of how a figure in the book renders on the Kindle:
A figure in the EPUB version of A History of Writing on my Kindle</p>
<p>When I read this page, I thought, "wow, this looks like crap." The figure is way too small (although you can long-press the image and zoom), and the position of the figure seems nonsensical. I found a PDF version online, and indeed the PDF's figure has a proper size in the right location:
A figure in the PDF version of A History of Writing on my Mac</p>
<p>This is not a fully fair comparison, but it nonetheless exemplifies an author's reasonable concern today with EPUB: what if it makes my document looks like crap?</p>
<h2>Principles for Consistent EPUB Rendering</h2>
<p>I think the core solution for consistently rendering EPUBs comes down to this:</p>
<ul>
<li>The document format (i.e., portable EPUB) needs to establish a subset of HTML (call it "portable HTML") which could represent most, but not all, documents.</li>
<li>Reading systems need to guarantee that a document within the subset will always look reasonable under all reading conditions.</li>
<li>If a document uses features outside this subset, then the document author is responsible for ensuring the readability of the document.</li>
</ul>
<p>If someone wants to write a document such as this post, then that person need not be a frontend web developer to feel confident that their document will render reasonably. Conversely, if someone wants to stuff the entire Facebook interface into an EPUB, then fine, but it's on them to ensure the document is responsive.</p>
<p>For instance, one simple version of portable HTML could be described by this grammar:</p>
<pre><code>Document ::= &lt;article&gt; Block* &lt;/article&gt;
Block ::= &lt;p&gt; Inline* &lt;/p&gt; | &lt;figure&gt; Block* &lt;/figure&gt;
Inline ::= text | &lt;strong&gt; Inline* &lt;/strong&gt;
</code></pre>
<p>The EPUB spec already defines a comparable subset for navigation documents. I am essentially proposing to extend this idea for content documents, but as a soft constraint rather than a hard constraint. Finding the right subset of HTML will take some experimentation, so I can only gesture toward the broad solution here.
Portable HTML rendering requirement: if a document only uses features in the portable HTML subset, then a portable EPUB reading system must guarantee that the document will render reasonably.
Portable HTML generation principle: when possible, systems that generate portable EPUBs should output portable HTML.</p>
<p>A related challenge is to define when a particular rendering is "good" or "reasonable", so one could evaluate either a document or a reading system on its conformance to spec. For instance, if document content is accidentally rendered in an inaccesible location off-screen, then that would be a bad rendering. A more aggressive definition might say that any rendering which violates accessibility guidelines is a bad rendering. Again, finding the right standard for rendering quality will take some experimentation.</p>
<p>If an author is particularly concerned about providing a single "canonical" rendering of their document, one fallback option is to provide a fixed-layout rendition. The EPUB format permits a rendition to specify that it should be rendered in fixed viewport size and optionally a fixed pagination. A fixed-layout rendition could then manually position all content on the page, similar to a PDF. Of course, this loses the flexibility of a reflowable rendition. But an EPUB could in theory provide multiple renditions, offering users the choice of whichever best suits their reading conditions and aesthetic preferences.
Fixed-layout fallback principle: systems that generate portable EPUBs can consider providing both a reflowable and fixed-layout rendition of a document.</p>
<p>It's possible that the reading system, the document author, and the reader can each express preferences about how a document should render. If these preferences are conflicting, then the renderer should generally prioritize the reader over the author, and the author over the reading system. This is an ideal use case for the "cascading" aspect of CSS:
Cascading styles principle: both documents and reading systems should express stylistic preferences (such as font face, font size, and document width) as CSS styles which can be overriden (e.g., do not use !important). The reading system should load the CSS rules such that the priority order is reading system styles &lt; document styles &lt; reader styles.</p>
<h2>A Lighter EPUB Reading System</h2>
<p>The act of working with PDFs is relatively fluid. I can download a PDF, quickly open it in a PDF reading system like Preview, and keep or discard the PDF as needed. But EPUB reading systems feel comparatively clunky. Loading an EPUB into Apple Books or Calibre will import the EPUB into the application's library, which both copies and potentially decompresses the file. Loading an EPUB on a Kindle requires waiting several minutes for the Send to Kindle service to complete.</p>
<p>Worse, EPUB reading systems often don't give you appropriate control over rendering an EPUB. For example, to emulate the experience of reading a book, most reading systems will chunk an EPUB into pages. A reader cannot scroll the document but rather "turn" the page, meaning textually-adjacent content can be split up between pages. Whether a document is paginated or scrolled should be a reader's choice, but 3/4 reading systems I tested would only permit pagination (Calibre being the exception).</p>
<p>Therefore I decided to build a lighter EPUB reading system, Bene. You're using it right now. This document is an EPUB — you can download it by clicking the button in the top-right corner. The styling and icons are mostly borrowed from pdf.js. Bene is implemented in Tauri, so it can work as both a desktop app and a browser app. Please appreciate this picture of Bene running as a desktop app:
The Bene reading system running as a desktop app. Wow! It works!</p>
<p>Bene is designed to make opening and reading an EPUB feel fast and non-committal. The app is much quicker to open on my Macbook (&lt;1sec) than other desktop apps. It decompresses files on-the-fly so no additional disk space is used. The backend is implemented in Rust and compiled to Wasm for the browser version.</p>
<p>The general design goal of Bene is to embody my ideals for a portable EPUB reader. That is, a utilitarian interface into an EPUB that satisfies my additional requirements for portability. Bene allows you to configure document rendering by changing the font size (try the +/- buttons in the top bar) and the viewer width (if you're on desktop, move your mouse over the right edge of the document, and drag the handle). Long-term, I want Bene to also provide richer document interactions than a standard EPUB reader, which means we must discuss scripting.</p>
<h2>Defensively Scripting EPUBs</h2>
<p>To some people, the idea of code in their documents is unappealing. Last time one of my document-related projects was posted to Hacker News, the top comment was complaining about dynamic documents. The sentiment is understandable — concerns include:</p>
<ul>
<li>Bad code: your document shouldn't crash or glitch due to a failure in a script.</li>
<li>Bad browsers: your document shouldn't fail to render when a browser updates.</li>
<li>Bad actors: a malicious document shouldn't be able to pwn your computer.</li>
<li>Bad interfaces: a script shouldn't cause your document to become unreadable.</li>
</ul>
<p>Yet, document scripting provides many opportunities for improving how we communicate information. For one example, if you haven't yet, try hovering your mouse over any instance of the term portable EPUB (or long press it on a touch screen). You should see a tooltip appear with the term's definition. The goal of these tooltips is to simplify reading a document that contains a lot of specialized notation or terminology. If you forget a definition, you can quickly look it up without having to jump around.</p>
<p>The key design challenge is how to permit useful scripting behaviors while limiting the downsides of scripting. One strategy is as follows:
Structure over scripts principle: documents should prefer structural annotations over scripts where possible. Documents should rely on reading systems to utilize structure where possible.</p>
<p>As an example of this principle, consider how the portable EPUB definition and references are expressed in this document:
Creating a definition
Referencing a definition</p>
<p>The definition uses the <code>&lt;dfn&gt;</code> element wrapped in a custom <code>&lt;dfn-container&gt;</code> element to indicate the scope of the definition. The reference to the definition uses a standard anchor with an addition data-target attribute to emphasize that a definition is being linked. The document itself does not provide a script. The Bene reading system automatically detects these annotations and provides the tooltip interaction.</p>
<h2>Encapsulating Scripts with Web Components</h2>
<p>But what if a document wants to provide an interactive component that isn't natively supported by the reading system? For instance, I have recently been working with The Rust Programming Language, a textbook that explains the different features of Rust. It contains a lot of passages like this one:</p>
<p>This program first binds x to a value of 5. Then it creates a new variable x by repeating let x =, taking the original value and adding 1 so the value of x is then 6. Then, within an inner scope created with the curly brackets, the third let statement also shadows x and creates a new variable, multiplying the previous value by 2 to give x a value of 12. When that scope is over, the inner shadowing ends and x returns to being 6. When we run this program, it will output the following:</p>
<p>A challenge in reading this passage is finding the correspondences between the prose and the code. An interactive code reading component can help you track those correspondences, like this (try mousing-over or clicking-on each sentence):</p>
<p>The interactive code description component is used as follows:</p>
<p>Again, the document content contains no actual script. It contains a custom element <code>&lt;code-description&gt;</code>, and it contains a series of annotations as spans and anchors. The <code>&lt;code-description&gt;</code> element is implemented as a web component.</p>
<p>Web components are a programming model for writing encapsulated interactive fragments of HTML, CSS, and Javascript. Web components are one of many ways to write componentized HTML, such as React, Solid, Svelte, and Angular. I see web components as the most suitable as a framework for portable EPUBs because:</p>
<ul>
<li>Web components are a standardized technology. Its key features like custom elements (for specifying the behavior of novel elements) and shadow trees (for encapsulating a custom element from the rest of the document) are part of the official HTML and DOM specifications. This improves the likelihood that future browsers will maintain backwards compatibility with web components written today.</li>
<li>Web components are designed for tight encapusulation. The shadow tree mechanism ensures that styling applied within a custom component cannot accidentally affect other components on the page.</li>
<li>Web components have a decent ecosystem to leverage. As far as I can tell, web components are primarily used by Google, which has created notable frameworks like Lit.</li>
<li>Web components provide a clear fallback mechanism. If a renderer does not support Javascript, or if a renderer loses the ability to render web components, then an HTML renderer will simply ignore custom tags and render their contents.</li>
</ul>
<p>Thus, I propose one principle and one requirement:
Encapsulated scripts principle: interactive components should be implemented as web components when possible, or otherwise be carefully designed to avoid conflicting with the base document or other components.
Components fallback requirement: interactive components must provide a fallback mechanism for rendering a reasonable substitute if Javascript is disabled.</p>
<h2>Where To Go From Here?</h2>
<p>Every time I have told someone "I want to replace PDF", the statement has been met with extreme skepticism. Hopefully this document has convinced you that HTML-via-EPUB could potentially be a viable and desirable document format for the future.</p>
<p>My short-term goal is to implement a few more documents in the portable EPUB format, such as my PLDI paper. That will challenge both the file format and the reading system to be flexible enough to support each document type. In particular, each document should look good under a range of reading conditions (screen sizes, font sizes and faces, etc.).</p>
<p>My long-term goal is to design a document language that makes it easy to generate portable EPUBs. Writing XHTML by hand is not reasonable. I designed Nota before I was thinking about EPUBs, so its next iteration will be targeted at this new format.</p>
<p>If you have any thoughts about how to make this work or why I'm wrong, let me know by email or Twitter or Mastodon or wherever this gets posted. If you would like to help out, please reach out! This is just a passion project in my free time (for now...), so any programming or document authoring assistance could provide a lot of momentum to the project.</p>
<h2>But What About...</h2>
<p>A brief postscript for a few things I haven't touched on.</p>
<p>...security? You might dislike the idea that document authors can run arbitrary Javascript on your personal computer. But then again, you presumably use both a PDF reader and a web browser on the daily, and those both run Javascript. What I'm proposing is not really any less secure than our current state of affairs. If anything, I'd hope that browsers are more battle-hardened than PDF viewers regarding code execution. Certainly the designers of EPUB reading systems should be careful to not give documents any additional capabilities beyond those already provided by the browser.</p>
<p>...aesthetics? People often intuit that LaTeX-generated PDFs look prettier than HTML documents, or even prettier than PDFs created by other software. This is because Donald Knuth took his job very seriously. In particular, the Knuth-Plass line-breaking algorithm tends to produce better-looking justified text than whatever algorithm is used by browsers.</p>
<p>There's two ways to make progress here. One is for browsers to provide more typography tools. Allegedly, text-wrap: pretty is supposed to help, but in my brief testing it doesn't seem to improve line-break quality. The other way is to pre-calculate line breaks, which would only work for fixed-layout renditions.</p>
<p>...page citations? I think we just have to give up on citing content by pages. Instead, we should mandate a consistent numbering scheme for block elements within a document, and have people cite using that scheme. For example, Bene will auto-number all blocks. If you're on a desktop, try hovering your mouse in the left column next to the top-right of any paragraph.</p>
</article>


<hr>

<footer>
<p>
<a href="/david/" title="Aller à l’accueil"><svg class="icon icon-home">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-home"></use>
</svg> Accueil</a> •
<a href="/david/log/" title="Accès au flux RSS"><svg class="icon icon-rss2">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-rss2"></use>
</svg> Suivre</a> •
<a href="http://larlet.com" title="Go to my English profile" data-instant><svg class="icon icon-user-tie">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-user-tie"></use>
</svg> Pro</a> •
<a href="mailto:david%40larlet.fr" title="Envoyer un courriel"><svg class="icon icon-mail">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-mail"></use>
</svg> Email</a> •
<abbr class="nowrap" title="Hébergeur : Alwaysdata, 62 rue Tiquetonne 75002 Paris, +33184162340"><svg class="icon icon-hammer2">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-hammer2"></use>
</svg> Légal</abbr>
</p>
<template id="theme-selector">
<form>
<fieldset>
<legend><svg class="icon icon-brightness-contrast">
<use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-brightness-contrast"></use>
</svg> Thème</legend>
<label>
<input type="radio" value="auto" name="chosen-color-scheme" checked> Auto
</label>
<label>
<input type="radio" value="dark" name="chosen-color-scheme"> Foncé
</label>
<label>
<input type="radio" value="light" name="chosen-color-scheme"> Clair
</label>
</fieldset>
</form>
</template>
</footer>
<script src="/static/david/js/instantpage-5.1.0.min.js" type="module"></script>
<script>
function loadThemeForm(templateName) {
const themeSelectorTemplate = document.querySelector(templateName)
const form = themeSelectorTemplate.content.firstElementChild
themeSelectorTemplate.replaceWith(form)

form.addEventListener('change', (e) => {
const chosenColorScheme = e.target.value
localStorage.setItem('theme', chosenColorScheme)
toggleTheme(chosenColorScheme)
})

const selectedTheme = localStorage.getItem('theme')
if (selectedTheme && selectedTheme !== 'undefined') {
form.querySelector(`[value="${selectedTheme}"]`).checked = true
}
}

const prefersColorSchemeDark = '(prefers-color-scheme: dark)'
window.addEventListener('load', () => {
let hasDarkRules = false
for (const styleSheet of Array.from(document.styleSheets)) {
let mediaRules = []
for (const cssRule of styleSheet.cssRules) {
if (cssRule.type !== CSSRule.MEDIA_RULE) {
continue
}
// WARNING: Safari does not have/supports `conditionText`.
if (cssRule.conditionText) {
if (cssRule.conditionText !== prefersColorSchemeDark) {
continue
}
} else {
if (cssRule.cssText.startsWith(prefersColorSchemeDark)) {
continue
}
}
mediaRules = mediaRules.concat(Array.from(cssRule.cssRules))
}

// WARNING: do not try to insert a Rule to a styleSheet you are
// currently iterating on, otherwise the browser will be stuck
// in a infinite loop…
for (const mediaRule of mediaRules) {
styleSheet.insertRule(mediaRule.cssText)
hasDarkRules = true
}
}
if (hasDarkRules) {
loadThemeForm('#theme-selector')
}
})
</script>
</body>
</html>

+ 225
- 0
cache/2024/ffaf50bf5d5e4cf870a618b518ee5ba7/index.md Datei anzeigen

@@ -0,0 +1,225 @@
title: Portable EPUBs
url: https://willcrichton.net/notes/portable-epubs/
hash_url: ffaf50bf5d5e4cf870a618b518ee5ba7
archive_date: 2024-01-25

Despite decades of advances in document rendering technology, most of the world's documents are stuck in the 1990s due to the limitations of PDF. Yet, modern document formats like HTML have yet to provide a competitive alternative to PDF. This post explores why this is the case, and proposes a way forward based on the EPUB format.

## The Good and Bad of PDF

PDF is the de facto file format for reading and sharing digital documents like papers, textbooks, and flyers. People use the PDF format for several reasons:

* PDFs are self-contained. A PDF is a single file that contains all the images, fonts, and other data needed to render it. It's easy to pass around a PDF. A PDF is unlikely to be missing some critical dependency on your computer.
* PDFs are rendered consistently. A PDF specifies precisely how it should be rendered, so a PDF author can be confident that a reader will see the same document under any conditions.
* PDFs are stable over time. PDFs from decades ago still render the same today. PDFs have a relatively stable standard. PDFs cannot be easily edited.


Yet, in the 32 years since the initial release of PDF, a lot has changed. People print out documents less and less. People use phones, tablets, and e-readers to read digital documents. The internet happened; web browsers now provide a platform for rendering rich documents. These changes have laid bare the limitations of PDF:

* PDFs cannot easily adapt to different screen sizes. Most PDFs are designed to mimic 8.5x11" paper (or worse, 145,161 km2). These PDFs are readable on a computer monitor, but they are less readable on a tablet, and far less readable on a phone.
* PDFs cannot be easily understood by programs. A plain PDF is just a scattered sequence of lines and characters. For accessibility, screen readers may not know which order to read through the text. For data extraction, scraping tables out of a PDF is an open area of research.
* PDFs cannot easily express interaction. PDFs were primarily designed as static documents that cannot react to user input beyond filling in forms.


These pros and cons can be traced back to one key fact: the PDF representation of a document is fundamentally unstructured. A PDF consists of commands like:

* Move the cursor to the right by 0.5 inches.
* Set the current font color to black.
* Draw the text "Hello World" at the current position.


PDF commands are unstructured because a document's organization is only clear to a person looking at the rendered document, and not clear from the commands themselves. Reflowing, accessibility, data extraction, and interaction all rely on programmatically understanding the structure of a document. Hence, these aspects are not easy to integrate with PDFs.

This raises the question: how can we design digital documents with the benefits of PDFs but without the limitations?

## Can't We Just Fix PDF?

A simple answer is to improve the PDF format. After all, we already have billions of PDFs — why reinvent the wheel?

The designers of PDF are well aware of its limitations. I carefully hedged each bullet with "easily", because PDF does make it possible to overcome each limitation, at least partially. PDFs can be annotated with their logical structure to create a tagged PDF. Most PDF exporters will not add tags automatically — the simplest option is to use Adobe's subscription-only Acrobat Pro, which provides an "Automatically tag PDF" action. For example, here is a recent paper of mine with added tags:
A LaTeX-generated paper with automatically added tags.

If you squint, you can see that the logical structure closely resembles the HTML document model. The document has sections, headings, paragraphs, and links. Adobe characterizes the logical structure as an accessibility feature, but it has other benefits. You may be surprised to know that Adobe Acrobat allows you to reflow tagged PDFs at different screen sizes. You may be unsurprised to know that reflowing does not always work well. For example:
A section of the paper in its default fixed layout. Note that the second paragraph is wrapped around the code snippet.
The same section of the paper after reflowing to a smaller width. Note that the code is now interleaved with the second paragraph.

In theory, these issues could be fixed. If the world's PDF exporters could be modified to include logical structure. If Adobe's reflowing algorithm could be improved to fix its edge cases. If the reflowing algorithm could be specified, and if Adobe were willing to release it publicly, and if it were implemented in each PDF viewer. And that doesn't even cover interaction! So in practice, I don't think we can just fix the PDF format, at least within a reasonable time frame.

## The Good and Bad of HTML

In the meantime, we already have a structured document format which can be flexibly and interactively rendered: HTML (and CSS and Javascript, but here just collectively referred to as HTML). The HTML format provides almost exactly the inverse advantages and disadvantages of PDF.

* HTML can more easily adapt to different screen sizes. Over the last 20 years, web developers and browser vendors have created a wide array of techniques for responsive design.
* HTML can be more easily understood by a program. HTML provides both an inherent structure plus additional attributes to support accessibility tools.
* HTML can more easily express interaction. People have used HTML to produce amazing interactive documents that would be impossible in PDF. Think: Distill.pub, Explorable Explanations, Bartosz Ciechanowski, and Bret Victor, just to name a few.


Again, these advantages are hedged with "more easily". One can easily produce a convoluted or inaccessible HTML document. But on balance, these aspects are more true than not compared to PDF. However, HTML is lacking where PDF shines:

* HTML is not self-contained. HTML files may contain URL references to external files that may be hosted on a server. One can rarely download an HTML file and have it render correctly without an internet connection.
* HTML is not always rendered consistently. HTML's dynamic layout means that an author may not see the same document as a reader. Moreover, HTML layout is not fully specified, so browsers may differ in their implementation.
* HTML is not fully stable over time. Browsers try to maintain backwards compatibility (come on and slam!), but the HTML format is still evolving. The HTML standard is a "living standard" due to the rapidly changing needs and feature sets of modern browsers.


So I've been thinking: how can we design HTML documents to gain the benefits of PDFs without losing the key strengths of HTML? The rest of this document will present some early prototypes and tentative proposals in this direction.

## Self-Contained HTML with EPUB

First, how can we make HTML documents self-contained? This is an old problem with many potential solutions. WARC, webarchive, and MHTML are all file formats designed to contain all the resources needed to render a web page. But these formats are more designed for snapshotting an existing website, rather than serving as a single source of truth for a web document. From my research, the most sensible format for this purpose is EPUB.

EPUB is a "distribution and interchange format for digital publications and documents", per the EPUB 3 Overview. Reductively, an EPUB is a ZIP archive of web files: HTML, CSS, JS, and assets like images and fonts. On a technical level, what distinguishes EPUB from archival formats is that EPUB includes well-specified files that describe metadata about a document. On a social level, EPUB appears to be the HTML publication format with the most adoption and momentum in 2024, compared to moribund formats like Mobi.

The EPUB spec has all the gory details, but to give you a rough sense, a sample EPUB might have the following file structure:

sample.epub
├── META-INF
│ └── container.xml
└── EPUB
├── package.opf
├── nav.xhtml
├── chapter1.xhtml
├── chapter2.xhtml
└── img
└── sample.jpg

An EPUB contains content documents (like chapter1.xhtml and chapter2.xhtml) which contain the core HTML content. Content documents can contain relative links to assets in the EPUB, like img/sample.jpg. The navigation document (nav.xhtml) provides a table of contents, and the package document (package.opf) provides metadata about the document. These files collectively define one "rendition" of the whole document, and the container file (container.xml) points to each rendition contained in the EPUB.

The EPUB format optimizes for machine-readable content and metadata. HTML content is required to be in XML format (hence, XHTML). Document metadata like the title and author is provided in structured form in the package document. The navigation document has a carefully prescribed tag structure so the TOC can be consistently extracted.

Overall, EPUB's structured format makes it a solid candidate for a single-file HTML document container. However, EPUB is not a silver bullet. EPUB is quite permissive in what kinds of content can be put into a content document.

For example, a major issue for self-containment is that EPUB content can embed external assets. A content document can legally include an image or font file whose src is a URL to a hosted server. This is not hypothetical, either; as of the time of writing, Google Doc's EPUB exporter will emit CSS that will @include external Google Fonts files. The problem is that such an EPUB will not render correctly without an internet connection, nor will it render correctly if Google changes the URLs of its font files.

Hence, I will propose a new format which I call a portable EPUB, which is an EPUB with additional requirements and recommendations to improve PDF-like portability. The first requirement is:
Local asset requirement: All assets (like images, scripts, and fonts) embedded in a content document of a portable EPUB must refer to local files included in the EPUB. Hyperlinks to external files are permissible.

## Consistency vs. Flexibility in Rendering

There is a fundamental tension between consistency and flexibility in document rendering. A PDF is consistent because it is designed to render in one way: one layout, one choice of fonts, one choice of colors, one pagination, and so on. Consistency is desirable because an author can be confident that their document will look good for a reader (or at least, not look bad). Consistency has subtler benefits — because a PDF is chunked into a consistent set of pages, a passage can be cited by referring to the page containing the passage.

On the other hand, flexibility is desirable because people want to read documents under different conditions. Device conditions include screen size (from phone to monitor) and screen capabilities (E-ink vs. LCD). Some readers may prefer larger fonts or higher contrasts for visibility, alternative color schemes for color blindness, or alternative font faces for dyslexia. Sufficiently flexible documents can even permit readers to select a level of detail appropriate for their background (here's an example).

Finding a balance between consistency and flexibility is arguably the most fundamental design challenge in attempting to replace PDF with EPUB. To navigate this trade-off, we first need to talk about EPUB reading systems, or the tools that render an EPUB for human consumption. To get a sense of variation between reading systems, I tried rendering this post as an EPUB (without any styling, just HTML) on four systems: Calibre, Adobe Digital Editions, Apple Books, and Amazon Kindle. This is how the first page looks on each system (omitting Calibre because it looked the same as Adobe Digital Editions):

* Adobe Digital Editions
* Apple Books
* Amazon Kindle

Calibre and Adobe Digital Editions both render the document in a plain web view, as if you opened the HTML file directly in the browser. Apple Books applies some styling, using the New York font by default and changing link decorations. Amazon Kindle increases the line height and also uses my Kindle's globally-configured default font, Palatino.

As you can see, an EPUB may look quite different on different reading systems. The variation displayed above seems reasonable to me. But how different is too different? For instance, I was recently reading A History of Writing on my Kindle. Here's an example of how a figure in the book renders on the Kindle:
A figure in the EPUB version of A History of Writing on my Kindle

When I read this page, I thought, "wow, this looks like crap." The figure is way too small (although you can long-press the image and zoom), and the position of the figure seems nonsensical. I found a PDF version online, and indeed the PDF's figure has a proper size in the right location:
A figure in the PDF version of A History of Writing on my Mac

This is not a fully fair comparison, but it nonetheless exemplifies an author's reasonable concern today with EPUB: what if it makes my document looks like crap?

## Principles for Consistent EPUB Rendering

I think the core solution for consistently rendering EPUBs comes down to this:

* The document format (i.e., portable EPUB) needs to establish a subset of HTML (call it "portable HTML") which could represent most, but not all, documents.
* Reading systems need to guarantee that a document within the subset will always look reasonable under all reading conditions.
* If a document uses features outside this subset, then the document author is responsible for ensuring the readability of the document.


If someone wants to write a document such as this post, then that person need not be a frontend web developer to feel confident that their document will render reasonably. Conversely, if someone wants to stuff the entire Facebook interface into an EPUB, then fine, but it's on them to ensure the document is responsive.

For instance, one simple version of portable HTML could be described by this grammar:

Document ::= <article> Block* </article>
Block ::= <p> Inline* </p> | <figure> Block* </figure>
Inline ::= text | <strong> Inline* </strong>

The EPUB spec already defines a comparable subset for navigation documents. I am essentially proposing to extend this idea for content documents, but as a soft constraint rather than a hard constraint. Finding the right subset of HTML will take some experimentation, so I can only gesture toward the broad solution here.
Portable HTML rendering requirement: if a document only uses features in the portable HTML subset, then a portable EPUB reading system must guarantee that the document will render reasonably.
Portable HTML generation principle: when possible, systems that generate portable EPUBs should output portable HTML.

A related challenge is to define when a particular rendering is "good" or "reasonable", so one could evaluate either a document or a reading system on its conformance to spec. For instance, if document content is accidentally rendered in an inaccesible location off-screen, then that would be a bad rendering. A more aggressive definition might say that any rendering which violates accessibility guidelines is a bad rendering. Again, finding the right standard for rendering quality will take some experimentation.

If an author is particularly concerned about providing a single "canonical" rendering of their document, one fallback option is to provide a fixed-layout rendition. The EPUB format permits a rendition to specify that it should be rendered in fixed viewport size and optionally a fixed pagination. A fixed-layout rendition could then manually position all content on the page, similar to a PDF. Of course, this loses the flexibility of a reflowable rendition. But an EPUB could in theory provide multiple renditions, offering users the choice of whichever best suits their reading conditions and aesthetic preferences.
Fixed-layout fallback principle: systems that generate portable EPUBs can consider providing both a reflowable and fixed-layout rendition of a document.

It's possible that the reading system, the document author, and the reader can each express preferences about how a document should render. If these preferences are conflicting, then the renderer should generally prioritize the reader over the author, and the author over the reading system. This is an ideal use case for the "cascading" aspect of CSS:
Cascading styles principle: both documents and reading systems should express stylistic preferences (such as font face, font size, and document width) as CSS styles which can be overriden (e.g., do not use !important). The reading system should load the CSS rules such that the priority order is reading system styles < document styles < reader styles.

## A Lighter EPUB Reading System

The act of working with PDFs is relatively fluid. I can download a PDF, quickly open it in a PDF reading system like Preview, and keep or discard the PDF as needed. But EPUB reading systems feel comparatively clunky. Loading an EPUB into Apple Books or Calibre will import the EPUB into the application's library, which both copies and potentially decompresses the file. Loading an EPUB on a Kindle requires waiting several minutes for the Send to Kindle service to complete.

Worse, EPUB reading systems often don't give you appropriate control over rendering an EPUB. For example, to emulate the experience of reading a book, most reading systems will chunk an EPUB into pages. A reader cannot scroll the document but rather "turn" the page, meaning textually-adjacent content can be split up between pages. Whether a document is paginated or scrolled should be a reader's choice, but 3/4 reading systems I tested would only permit pagination (Calibre being the exception).

Therefore I decided to build a lighter EPUB reading system, Bene. You're using it right now. This document is an EPUB — you can download it by clicking the button in the top-right corner. The styling and icons are mostly borrowed from pdf.js. Bene is implemented in Tauri, so it can work as both a desktop app and a browser app. Please appreciate this picture of Bene running as a desktop app:
The Bene reading system running as a desktop app. Wow! It works!

Bene is designed to make opening and reading an EPUB feel fast and non-committal. The app is much quicker to open on my Macbook (<1sec) than other desktop apps. It decompresses files on-the-fly so no additional disk space is used. The backend is implemented in Rust and compiled to Wasm for the browser version.

The general design goal of Bene is to embody my ideals for a portable EPUB reader. That is, a utilitarian interface into an EPUB that satisfies my additional requirements for portability. Bene allows you to configure document rendering by changing the font size (try the +/- buttons in the top bar) and the viewer width (if you're on desktop, move your mouse over the right edge of the document, and drag the handle). Long-term, I want Bene to also provide richer document interactions than a standard EPUB reader, which means we must discuss scripting.

## Defensively Scripting EPUBs

To some people, the idea of code in their documents is unappealing. Last time one of my document-related projects was posted to Hacker News, the top comment was complaining about dynamic documents. The sentiment is understandable — concerns include:

* Bad code: your document shouldn't crash or glitch due to a failure in a script.
* Bad browsers: your document shouldn't fail to render when a browser updates.
* Bad actors: a malicious document shouldn't be able to pwn your computer.
* Bad interfaces: a script shouldn't cause your document to become unreadable.


Yet, document scripting provides many opportunities for improving how we communicate information. For one example, if you haven't yet, try hovering your mouse over any instance of the term portable EPUB (or long press it on a touch screen). You should see a tooltip appear with the term's definition. The goal of these tooltips is to simplify reading a document that contains a lot of specialized notation or terminology. If you forget a definition, you can quickly look it up without having to jump around.

The key design challenge is how to permit useful scripting behaviors while limiting the downsides of scripting. One strategy is as follows:
Structure over scripts principle: documents should prefer structural annotations over scripts where possible. Documents should rely on reading systems to utilize structure where possible.

As an example of this principle, consider how the portable EPUB definition and references are expressed in this document:
Creating a definition
Referencing a definition

The definition uses the `<dfn>` element wrapped in a custom `<dfn-container>` element to indicate the scope of the definition. The reference to the definition uses a standard anchor with an addition data-target attribute to emphasize that a definition is being linked. The document itself does not provide a script. The Bene reading system automatically detects these annotations and provides the tooltip interaction.

## Encapsulating Scripts with Web Components

But what if a document wants to provide an interactive component that isn't natively supported by the reading system? For instance, I have recently been working with The Rust Programming Language, a textbook that explains the different features of Rust. It contains a lot of passages like this one:

This program first binds x to a value of 5. Then it creates a new variable x by repeating let x =, taking the original value and adding 1 so the value of x is then 6. Then, within an inner scope created with the curly brackets, the third let statement also shadows x and creates a new variable, multiplying the previous value by 2 to give x a value of 12. When that scope is over, the inner shadowing ends and x returns to being 6. When we run this program, it will output the following:

A challenge in reading this passage is finding the correspondences between the prose and the code. An interactive code reading component can help you track those correspondences, like this (try mousing-over or clicking-on each sentence):

The interactive code description component is used as follows:

Again, the document content contains no actual script. It contains a custom element `<code-description>`, and it contains a series of annotations as spans and anchors. The `<code-description>` element is implemented as a web component.

Web components are a programming model for writing encapsulated interactive fragments of HTML, CSS, and Javascript. Web components are one of many ways to write componentized HTML, such as React, Solid, Svelte, and Angular. I see web components as the most suitable as a framework for portable EPUBs because:

* Web components are a standardized technology. Its key features like custom elements (for specifying the behavior of novel elements) and shadow trees (for encapsulating a custom element from the rest of the document) are part of the official HTML and DOM specifications. This improves the likelihood that future browsers will maintain backwards compatibility with web components written today.
* Web components are designed for tight encapusulation. The shadow tree mechanism ensures that styling applied within a custom component cannot accidentally affect other components on the page.
* Web components have a decent ecosystem to leverage. As far as I can tell, web components are primarily used by Google, which has created notable frameworks like Lit.
* Web components provide a clear fallback mechanism. If a renderer does not support Javascript, or if a renderer loses the ability to render web components, then an HTML renderer will simply ignore custom tags and render their contents.


Thus, I propose one principle and one requirement:
Encapsulated scripts principle: interactive components should be implemented as web components when possible, or otherwise be carefully designed to avoid conflicting with the base document or other components.
Components fallback requirement: interactive components must provide a fallback mechanism for rendering a reasonable substitute if Javascript is disabled.

## Where To Go From Here?

Every time I have told someone "I want to replace PDF", the statement has been met with extreme skepticism. Hopefully this document has convinced you that HTML-via-EPUB could potentially be a viable and desirable document format for the future.

My short-term goal is to implement a few more documents in the portable EPUB format, such as my PLDI paper. That will challenge both the file format and the reading system to be flexible enough to support each document type. In particular, each document should look good under a range of reading conditions (screen sizes, font sizes and faces, etc.).

My long-term goal is to design a document language that makes it easy to generate portable EPUBs. Writing XHTML by hand is not reasonable. I designed Nota before I was thinking about EPUBs, so its next iteration will be targeted at this new format.

If you have any thoughts about how to make this work or why I'm wrong, let me know by email or Twitter or Mastodon or wherever this gets posted. If you would like to help out, please reach out! This is just a passion project in my free time (for now...), so any programming or document authoring assistance could provide a lot of momentum to the project.

## But What About...

A brief postscript for a few things I haven't touched on.

...security? You might dislike the idea that document authors can run arbitrary Javascript on your personal computer. But then again, you presumably use both a PDF reader and a web browser on the daily, and those both run Javascript. What I'm proposing is not really any less secure than our current state of affairs. If anything, I'd hope that browsers are more battle-hardened than PDF viewers regarding code execution. Certainly the designers of EPUB reading systems should be careful to not give documents any additional capabilities beyond those already provided by the browser.

...aesthetics? People often intuit that LaTeX-generated PDFs look prettier than HTML documents, or even prettier than PDFs created by other software. This is because Donald Knuth took his job very seriously. In particular, the Knuth-Plass line-breaking algorithm tends to produce better-looking justified text than whatever algorithm is used by browsers.

There's two ways to make progress here. One is for browsers to provide more typography tools. Allegedly, text-wrap: pretty is supposed to help, but in my brief testing it doesn't seem to improve line-break quality. The other way is to pre-calculate line breaks, which would only work for fixed-layout renditions.

...page citations? I think we just have to give up on citing content by pages. Instead, we should mandate a consistent numbering scheme for block elements within a document, and have people cite using that scheme. For example, Bene will auto-number all blocks. If you're on a desktop, try hovering your mouse in the left column next to the top-right of any paragraph.

+ 5
- 1
cache/2024/index.html Datei anzeigen

@@ -72,7 +72,7 @@
<li><a href="/david/cache/2024/877ad04fd329c26c80113e15dec540df/" title="Accès à l’article dans le cache local : The Walk and Talk: Everything We Know">The Walk and Talk: Everything We Know</a> (<a href="https://craigmod.com/ridgeline/176/" title="Accès à l’article original distant : The Walk and Talk: Everything We Know">original</a>)</li>
<li><a href="/david/cache/2024/fd6eda56671045e0c1e2d215e07f1a6f/" title="Accès à l’article dans le cache local : "></a> (<a href="https://jacobtomlinson.dev/effver/" title="Accès à l’article original distant : ">original</a>)</li>
<li><a href="/david/cache/2024/fd6eda56671045e0c1e2d215e07f1a6f/" title="Accès à l’article dans le cache local : EffVer: Version your code by the effort required to upgrade">EffVer: Version your code by the effort required to upgrade</a> (<a href="https://jacobtomlinson.dev/effver/" title="Accès à l’article original distant : EffVer: Version your code by the effort required to upgrade">original</a>)</li>
<li><a href="/david/cache/2024/d236f33cf82727313d17cb23bf36a395/" title="Accès à l’article dans le cache local : Reconsider your partnership with Brave">Reconsider your partnership with Brave</a> (<a href="https://kagifeedback.org/d/2808-reconsider-your-partnership-with-brave/6" title="Accès à l’article original distant : Reconsider your partnership with Brave">original</a>)</li>
@@ -90,6 +90,8 @@
<li><a href="/david/cache/2024/ba977526c7a8cab6935708b2cdba5c0c/" title="Accès à l’article dans le cache local : Aging programmer">Aging programmer</a> (<a href="https://world.hey.com/jorge/aging-programmer-d448bdec" title="Accès à l’article original distant : Aging programmer">original</a>)</li>
<li><a href="/david/cache/2024/ce5fdc61fd66cdb9ce548fb543eba986/" title="Accès à l’article dans le cache local : Unsigned Commits">Unsigned Commits</a> (<a href="https://blog.glyph.im/2024/01/unsigned-commits.html" title="Accès à l’article original distant : Unsigned Commits">original</a>)</li>
<li><a href="/david/cache/2024/5030196507bcf3e06162e9eaed40abbe/" title="Accès à l’article dans le cache local : Blogging and Composting">Blogging and Composting</a> (<a href="https://blog.jim-nielsen.com/2023/blogging-and-compositing/" title="Accès à l’article original distant : Blogging and Composting">original</a>)</li>
<li><a href="/david/cache/2024/7d2d2bce96cbc020423299e7de03d2cf/" title="Accès à l’article dans le cache local : Disillusioned with Deno">Disillusioned with Deno</a> (<a href="https://www.baldurbjarnason.com/2024/disillusioned-with-deno/" title="Accès à l’article original distant : Disillusioned with Deno">original</a>)</li>
@@ -108,6 +110,8 @@
<li><a href="/david/cache/2024/3debc675a055d691b32c7d6904531eb4/" title="Accès à l’article dans le cache local : How Google perfected the web">How Google perfected the web</a> (<a href="https://www.theverge.com/c/23998379/google-search-seo-algorithm-webpage-optimization" title="Accès à l’article original distant : How Google perfected the web">original</a>)</li>
<li><a href="/david/cache/2024/ffaf50bf5d5e4cf870a618b518ee5ba7/" title="Accès à l’article dans le cache local : Portable EPUBs">Portable EPUBs</a> (<a href="https://willcrichton.net/notes/portable-epubs/" title="Accès à l’article original distant : Portable EPUBs">original</a>)</li>
<li><a href="/david/cache/2024/d75afc90a9d3c3b5a56b69446795fbb5/" title="Accès à l’article dans le cache local : plaisir d'ébauche">plaisir d'ébauche</a> (<a href="https://www.la-grange.net/2024/01/06/ebauche" title="Accès à l’article original distant : plaisir d'ébauche">original</a>)</li>
<li><a href="/david/cache/2024/c3272392d462da90874d32841e5caac8/" title="Accès à l’article dans le cache local : Where have all the websites gone?">Where have all the websites gone?</a> (<a href="https://www.fromjason.xyz/p/notebook/where-have-all-the-websites-gone/" title="Accès à l’article original distant : Where have all the websites gone?">original</a>)</li>

Laden…
Abbrechen
Speichern