larlet-fr-david-cache/cache/2022/747b97ed378bc3b334ab6375346a13fa/index.md

title: Coping strategies for the serial project hoarder
url: https://simonwillison.net/2022/Nov/26/productivity/
hash_url: 747b97ed378bc3b334ab6375346a13fa

<p>I gave a talk at DjangoCon US 2022 in San Diego last month about productivity on personal projects, titled “Massively increase your productivity on personal projects with comprehensive documentation and automated tests”.</p>
<p>The alternative title for the talk was <em>Coping strategies for the serial project hoarder</em>.</p>
<p>I’m maintaining a <em>lot</em> of different projects at the moment. Somewhat unintuitively, the way I’m handling this is by scaling down techniques that I’ve seen working for large engineering teams spread out across multiple continents.</p>
<p>The key trick is to ensure that every project has comprehensive documentation and automated tests. This scales my productivity horizontally, by freeing me up from needing to remember all of the details of all of the different projects I’m working on at the same time.</p>
<p>You can watch the talk <a href="https://www.youtube.com/watch?v=GLkRK2rJGB0">on YouTube</a> (25 minutes). Alternatively, I’ve included a detailed annotated version of the slides and notes below.</p>
<p class="resp-container">
  <iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen" frameborder="0" src="https://www.youtube.com/embed/GLkRK2rJGB0">VIDEO</iframe>
</p>

<div class="slide">
<img alt="Title slide: Massively increase your productivity on personal projects with comprehensive documentation and automated tests - Simon Willison, DjangoCon US 2022" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.001.jpeg"><div>
<p>This was the title I originally submitted to the conference. But I realized a better title was probably...</p>
</div>
</div>
<div class="slide">
<img alt="Same title slide, but the title has been replaced" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.003.jpeg"><div>
<p>Coping strategies for the serial project hoarder</p>
</div>
</div>
<div class="slide">
<img alt="A static frame from a video: a monkey sits on some steps stuffing itself with several pastries. In the longer video the monkey is handed more and more pastries and can't resist trying to hold and eat all of them at once, no matter how many it receives." loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.004.jpeg"><div>
<p><a href="https://twitter.com/devisridhar/status/1576170527882121217">This video</a> is a neat representation of my approach to personal projects: I always have a few on the go, but I can never resist the temptation to add even more.</p>
</div>
</div>
<div class="slide">
<img alt="A screenshot of my profile on PyPI - my join date is Oct 26, 2017 and I have 185 pojects listed." loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.005.jpeg"><div>
<p><a href="https://pypi.org/user/simonw/">My PyPI profile</a> (which is only five years old) lists 185 Python packages that I’ve released. Technically I’m actively maintaining all of them, in that if someone reports a bug I’ll push out a fix. Many of them receive new releases at least once a year.</p>
<p>Aside: I took this screenshot using <a href="https://shot-scraper.datasette.io/">shot-scraper</a> with a little bit of extra JavaScript to hide a notification bar at the top of the page:</p>
<div class="highlight highlight-source-shell"><pre>shot-scraper <span class="pl-s"><span class="pl-pds">'</span>https://pypi.org/user/simonw/<span class="pl-pds">'</span></span> \
--javascript <span class="pl-s"><span class="pl-pds">"</span></span>
<span class="pl-s">    document.body.style.paddingTop = 0;</span>
<span class="pl-s">    document.querySelector(</span>
<span class="pl-s">        '#sticky-notifications'</span>
<span class="pl-s">    ).style.display = 'none';</span>
<span class="pl-s">  <span class="pl-pds">"</span></span> --height 1000</pre></div>
</div>
</div>
<div class="slide">
<img alt="A map of the world with the Eventbrite logo overlaid on it. There are pins on San Francisco, Nashville, Mendoza and Madrid." loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.006.jpeg"><div>
<p>How can one individual maintain 185 projects?</p>
<p>Surprisingly, I’m using techniques that I’ve scaled down from working at a company with hundreds of engineers.</p>
<p>I spent seven years at Eventbrite, during which time the engineering team grew to span three different continents. We had major engineering centers in San Francisco, Nashville, Mendoza in Argentina and Madrid in Spain.</p>
<p>Consider timezones: engineers in Madrid and engineers in San Francisco had almost no overlap in their working hours. Good asynchronous communication was essential.</p>
<p>Over time, I noticed that the teams that were most effective at this scale were the teams that had a strong culture of documentation and automated testing.</p>
<p>As I started to work on my own array of smaller personal projects, I found that the same discipline that worked for large teams somehow sped me up, when intuitively I would have expected it to slow me down.</p>
</div>
</div>
<div class="slide">
<img alt="The perfect commit: Implementation + tests + documentation and a link to an issue thread" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.007.jpeg"><div>
<p>I wrote an extended description of this in <a href="https://simonwillison.net/2022/Oct/29/the-perfect-commit/">The Perfect Commit</a>.</p>
<p>I’ve started structuring the majority of my work in terms of what I think of as “the perfect commit”—a commit that combines implementation, tests, documentation and a link to an issue thread.</p>
<p>As software engineers, it’s important to note that our job generally isn’t to write new software: it’s to make changes to existing software.</p>
<p>As such, the commit is our unit of work. It’s worth us paying attention to how we cen make our commits as useful as possible.</p>
</div>
</div>
<div class="slide">
<img alt="Screenshot of a commit on GitHub: the title is Async support for prepare_jinja2_environment, closes #1809" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.008.jpeg"><div>
<p>Here’s <a href="https://github.com/simonw/datasette/commit/ddc999ad1296e8c69cffede3e367dda059b8adad">a recent example</a> from one of my projects, Datasette.</p>
<p>It’s a single commit which bundles together the implementation, some related documentation improvements and the tests that show it works. And it links back to an issue thread from the commit message.</p>
<p>Let’s talk about each component in turn.</p>
</div>
</div>
<div class="slide">
<img alt="Implementation: it should just do one thing (thing here is deliberately vague)" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.009.jpeg"><div>
<p>There’s not much to be said about the implementation: your commit should change something!</p>
<p>It should only change one thing, but what that actually means varies on a case by case basis.</p>
<p>It should be a single change that can be documented, tested and explained independently of other changes.</p>
<p>(Being able to cleanly revert it is a useful property too.)</p>
</div>
</div>
<div class="slide">
<img alt="Tests: prove that the implementation works. Pass if the new implementation is correct, fail otherwise." loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.010.jpeg"><div>
<p>The goals of the tests that accompany a commit are to prove that the new implementation works.</p>
<p>If you apply the implementation the new tests should pass. If you revert it the tests should fail.</p>
<p>I often use <code>git stash</code> to try this out.</p>
<p>If you tell people they need to write tests for <em>every single change</em> they’ll often push back that this is too much of a burden, and will harm their productivity.</p>
</div>
</div>
<div class="slide">
<img alt="Every project should start with a test. assert 1 + 1 == 2 is fine! Adding tests to an existing test suite is SO MUCH less work than starting a new test suite from scratch." loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.011.jpeg"><div>
<p>But I find that the incremental cost of adding a test to an existing test suite keeps getting lower over time.</p>
<p>The hard bit of testing is getting a testing framework setup in the first place—with a test runner, and fixtures, and objects under test and suchlike.</p>
<p>Once that’s in place, adding new tests becomes really easy.</p>
<p>So my personal rule is that every new project starts with a test. It doesn’t really matter what that test does—what matters is that you can run <code>pytest</code> to run the tests, and you have an obvious place to start building more of them.</p>
</div>
</div>
<div class="slide">
<img alt="Cookiecutter repo templates: simonw/python-lib, simonw/click-app, simonw/datasette-plugin" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.012.jpeg"><div>
<p>I maintain three <a href="https://cookiecutter.readthedocs.io/">cookiecutter</a> templates to help with this, for the three kinds of projects I most frequently create:</p>

<p>Each of these templates creates a project with a <code>setup.py</code> file, a README, a test suite and GitHub Actions workflows to run those tests and ship tagged releases to PyPI.</p>
</div>
</div>
<div class="slide">
<img alt="Screenshot of the GitHub page to create a new repsoitory from python-lib-template-repository, which asks for a repository name, a description string and if the new repo should be public or private." loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.013.jpeg">
</div>
<div class="slide">
<img alt="Documentation: Same repository as the code! Document changes that impact external developers. Update the docs in the same commit as the change. Catch missing documentation updates in PR / code review" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.014.jpeg"><div>
<p>This is a hill that I will die on: your documentation must live in the same repository as your code!</p>
<p>You often see projects keep their documentation somewhere else, like in a wiki.</p>
<p>Inevitably it goes out of date. And my experience is that if your documentation is out of date people will lose trust in it, which means they’ll stop reading it and stop contributing to it.</p>
<p>The gold standard of documentation has to be that it’s reliably up to date with the code.</p>
<p>The only way you can do that is if the documentation and code are in the same repository.</p>
<p>This gives you versioned snapshots of the documentation that exactly match the code at that time.</p>
<p>More importantly, it means you can enforce it through code review. You can say in a PR “this is great, but don’t forget to update this paragraph on this page of the documentation to reflect the change you’re making”.</p>
<p>If you do this you can finally get documentation that people learn to trust over time.</p>
</div>
</div>
<div class="slide">
<img alt="Bonus trick: documentation unit tests" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.015.jpeg"><div>
<p>Another trick I like to use is something I call documentation unit tests.</p>
<p>The idea here is to use unit tests to enforce that concepts introspected from your code are at least mentioned in your documentation.</p>
<p>I wrote more about that in <a href="https://simonwillison.net/2018/Jul/28/documentation-unit-tests/">Documentation unit tests</a>.</p>
</div>
</div>
<div class="slide">
<img alt="Screenshot showing pytest running 26 passing tests, each with a name like test_plugin_hook_are_documented[filters_from_request]" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.016.jpeg"><div>
<p>Here’s an example. Datasette has <a href="https://github.com/simonw/datasette/blob/0.63.1/tests/test_docs.py#L41-L53">a test</a> that scans through each of the Datasette plugin hooks and checks that there is a heading for each one in the documentation.</p>
</div>
</div>
<div class="slide">
<img alt="Screenshot of the code linked to above" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.017.jpeg"><div>
<p>The test itself is pretty simple: it uses <code>pytest</code> parametrization to look through every introspected plugin hook name, and for each one checks that it has a matching heading in the documentation.</p>
</div>
</div>
<div class="slide">
<img alt="Everything links to an issue thread" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.018.jpeg"><div>
<p>The final component of my perfect commit is this: every commit must link to an issue thread.</p>
<p>I’ll usually have these open in advance but  sometimes I’ll open an issue thread just so I can close it with a commit a few seconds later!</p>
</div>
</div>
<div class="slide">
<img alt="A screenshot of the issue titled prepare_jinja_enviroment() hook should take datasette argument - it has 11 comments" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.019.jpeg"><div>
<p>Here’s <a href="https://github.com/simonw/datasette/issues/1809">the issue</a> for the commit I showed earlier. It has 11 comments, and every single one of those comments is by me.</p>
<p>I have literally thousands of issues on GitHub that look like this: issue threads that are effectively me talking to myself about the changes that I’m making.</p>
<p>It turns out this a fantastic form of additional documentation.</p>
</div>
</div>
<div class="slide">
<img alt="What goes in an issue?" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.020.jpeg"><div>
<p>What goes in an issue?</p>
<ul>
<li>Background: the reasons for the change. In six months time you’ll want to know why you did this.</li>
<li>State of play before-hand: embed existing code, link to existing docs. I like to start my issues with “I’m going to change this code right here”—that way if I come back the next day I don’t have to repeat that little piece of research.</li>
<li>Links to things! Documentation, inspiration, clues found on StackOverflow. The idea is to capture all of the loose information floating around that topic.</li>
<li>Code snippets illustrating potential designs and false-starts.</li>
<li>Decisions. What did you consider? What did you decide? As programmers we make decisions constantly, all day, about everything. That work doesn’t have to be invisible. Writing them down also avoids having to re-litigate them several months later when you’ve forgotten your original  reasoning.</li>
<li>Screenshots—of everything! Animated screenshots even better. I even take screenshots of things like the AWS console to remind me what I did there.</li>
<li>When you close it: a link to the updated documentation and demo</li>
</ul>
</div>
</div>
<div class="slide">
<img alt="Temporal documentation. It's timestamped and contextual. You don't need to commit to keeping it up-to-date in the future (but you can add more comments if you like)" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.021.jpeg"><div>
<p>The reason I love issues is that they’re a form of documentation that I think of as <em>temporal documentation</em>.</p>
<p>Regular documentation comes with a big commitment: you have to keep it up to date in the future.</p>
<p>Issue comments skip that commitment entirely. They’re displayed with a timestamp, in the context of the work you were doing at the time.</p>
<p>No-one will be upset or confused if you fail to keep them updated to match future changes.</p>
<p>So it’s a commitment free form of documentation, which I for one find incredibly liberating.</p>
</div>
</div>
<div class="slide">
<img alt="Issue driven development" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.022.jpeg"><div>
<p>I think of this approach as <em>issue driven development</em>.</p>
<p>Everything you are doing is issue-first, and from that you drive the rest of the development process.</p>
</div>
</div>
<div class="slide">
<img alt="Don't remember anything: you can go back to a project in six months and pick up right where you left off" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.023.jpeg"><div>
<p>This is how it relates back to maintaining 185 projects at the same time.</p>
<p>With issue driven development you <em>don’t have to remember anything</em> about any of these projects at all.</p>
<p>I’ve had issues where I did a bunch of design work in issue comments, then dropped it, then came back 12 months later and implemented that design—without having to rethink it.</p>
<p>I’ve had projects where I forgot that the project existed entirely! But I’ve found it again, and there’s been an open issue, and I’ve been able to pick up work again.</p>
<p>It’s a way of working where you treat it like every project is going to be maintained by someone else, and it’s the classic cliche here that the somebody else is you in the future.</p>
<p>It horizontally scales you and lets you tackle way more interesting problems.</p>
<p>Programmers always complain when you interrupt them—there’s this idea of “flow state” and that interrupting a programmer for a moment costs them half an hour in getting back up to speed.</p>
<p>This fixes that! It’s much easier to get back to what you are doing if you have an issue thread that records where you’ve got to.</p>
<p>Issue driven development is my key productivity hack for taking on much more ambitious projects in much larger quantities.</p>
</div>
</div>
<div class="slide">
<img alt="Laboratory notebooks - and a picture of a page from one by Leonardo da Vinci" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.024.jpeg"><div>
<p>Another way to think about this is to compare it to laboratory notebooks.</p>
<p>Here’s <a href="https://en.wikipedia.org/wiki/Studies_of_the_Fetus_in_the_Womb">a page</a> from one by Leonardo da Vinci.</p>
<p>Great scientists and great engineers have always kept detailed notes.</p>
<p>We can use GitHub issues as a really quick and easy way to do the same thing!</p>
</div>
</div>

<div class="slide">
<img alt="simonw/public-notes/issues" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.026.jpeg">
</div>
<div class="slide">
<img alt="Tell people what you did! (It's so easy to skip this step)" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.027.jpeg"><div>
<p>The last thing I want to encourage you to do is this: if you do project, tell people what it is you did!</p>
<p>This counts for both personal and work projects. It’s so easy to skip this step.</p>
<p>Once you’ve shipped a feature or built a project, it’s so tempting to skip the step of spending half an hour or more writing about the work you have done.</p>
<p>But you are missing out on <em>so much</em> of the value of your work if you don’t give other people a chance to understand what you did.</p>
<p>I wrote more about this here: <a href="https://simonwillison.net/2022/Nov/6/what-to-blog-about/">What to blog about</a>.</p>
</div>
</div>
<div class="slide">
<img alt="Release notes (with dates)" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.028.jpeg"><div>
<p>For projects with releases, release notes are a really good way to do this.</p>
<p>I like using GitHub releases for this—they’re quick and easy to write, and I have automation setup for my projects such that creating release notes in GitHub triggers a build and release to PyPI.</p>
<p>I’ve done over 1,000 releases in this way. Having them automated is crucial, and having automation makes it really easy to ship releases more often.</p>
<p>Please make sure your release notes have dates on them. I need to know when your change went out, because if it’s only a week old it’s unlikely people will have upgraded to it yet, whereas a change from five years ago is probably safe to depend on.</p>
<p>I wrote more about <a href="https://simonwillison.net/2022/Jan/31/release-notes/">writing better release notes</a> here.</p>
</div>
</div>
<div class="slide">
<img alt="Expand your definition of done to include writing about what you did" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.029.jpeg"><div>
<p>This is a mental trick which works really well for me. “No project of mine is finished until I’ve told people about it in some way” is a really useful habit to form.</p>
</div>
</div>
<div class="slide">
<img alt="Twitter threads (embed images + links + videos)" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.030.jpeg"><div>
<p>Twitter threads are (or were) a great low-effort way to write about a project. Build a quick thread with some links and images, and maybe even a video.</p>
<p>Get a little unit about your project out into the world, and then you can stop thinking about it.</p>
<p>(I’m trying to do this <a href="https://simonwillison.net/2022/Nov/5/mastodon/">on Mastodon now</a> instead.)</p>
</div>
</div>
<div class="slide">
<img alt="Get a blog" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.031.jpeg"><div>
<p>Even better: get a blog! Having your own corner of the internet to write about the work that you are doing is a small investment that will pay off many times over.</p>
<p>(“Nobody blogs anymore” I said in the talk... Phil Gyford disagrees with that meme so much that he launched <a href="https://ooh.directory/blog/2022/welcome/">a new blog directory</a> to show how wrong it is.)</p>
</div>
</div>
<div class="slide">
<img alt="GUILT is the enemy of projects" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.032.jpeg"><div>
<p>The enemy of projects, especially personal projects, is <em>guilt</em>.</p>
<p>The more projects you have, the more guilty you feel about working on any one of them—because you’re not working on the others, and those projects haven’t yet achieved their goals.</p>
<p>You have to overcome guilt if you’re going to work on 185 projects at once!</p>
</div>
</div>
<div class="slide">
<img alt="Avoid side projects with user accounts. If i has user accounts it's not a side-project, it's an unpaid job." loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.033.jpeg"><div>
<p>This is the most important tip: avoid side projects with user accounts.</p>
<p>If you build something that people can sign into, that’s not a side-project, it’s an unpaid job. It’s a very big responsibility, avoid at all costs!</p>
<p>Almost all of my projects right now are open source things that people can run on their own machines, because that’s about as far away from user accounts as I can get.</p>
<p>I still have a responsibility for shipping security updates and things like that, but at least I’m not holding onto other people’s data for them.</p>
</div>
</div>
<div class="slide">
<img alt="If your project is tested and documented, you have nothing to feel guilty about. That's what I tell myself anyway!" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.034.jpeg"><div>
<p>I feel like if your project is tested and documented, <em>you have nothing to feel guilty about</em>.</p>
<p>You have put a thing out into the world, and it has tests to show that it works, and it has documentation that explains what it is.</p>
<p>This means I can step back and say that it’s OK for me to work on other things. That thing there is a unit that makes sense to people.</p>
<p>That’s what I tell myself anyway! It’s OK to have 185 projects provided they all have documentation and they all have tests.</p>
<p>Do that and the guilt just disappears. You can live guilt free!</p>
</div>
</div>
<div class="slide">
<img alt="Thank you - simonwillison.net - twitter.com/simonw / github.com/simonw" loading="lazy" src="https://static.simonwillison.net/static/2022/djangocon-productivity/productivity.035.jpeg"><div>
<p>You can follow me on Mastodon at <a href="https://fedi.simonwillison.net/@simon">@simon@simonwillison.net</a> or on GitHub at <a href="https://github.com/simonw">github.com/simonw</a>. Or subscribe to my blog at <a href="https://simonwillison.net/">simonwillison.net</a>!</p>
<p>From the Q&amp;A:</p>
<ul>
<li>You’ve tweeted about using GitHub Projects. Could you talk about that?
<ul>
<li>
<a href="https://docs.github.com/en/issues/planning-and-tracking-with-projects/learning-about-projects/about-projects">GitHub Projects V2</a> is the perfect TODO list for me, because it lets me bring together issues from different repositories. I use a project called “Everything” on a daily basis (it’s my browser default window)—I add issues to it that I plan to work on, including personal TODO list items as well as issues from my various public and private repositories. It’s kind of like a cross between Trello and Airtable and I absolutely love it.</li>
</ul>
</li>
<li>How did you move notes from the private to the public repo?
<ul>
<li>GitHub doesn’t let you do this. But there’s a trick I use involving a <code>temp</code> repo which I switch between public and private to help transfer notes. More in this TIL.</li>
</ul>
</li>
<li>Question about the perfect commit: do you commit your failing tests?
<ul>
<li>I don’t: I try to keep the commits that land on my <code>main</code> branch always passing. I’ll sometimes write the failing test before the implementation and  then commit them together. For larger projects I’ll work in a branch and then squash-merge the final result into a perfect commit to main later on.</li>
</ul>
</li>
</ul>
</div>