larlet-fr-david-cache/cache/f064de13a0e15573a8051034d081f3c8/index.md

title: API Evolution the Right Way
url: https://emptysqua.re/blog/api-evolution-the-right-way/
hash_url: f064de13a0e15573a8051034d081f3c8

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/dragon.jpg" alt=""/></p>

<p>Imagine you are a creator deity, designing a body for a creature. In your benevelonce, you wish for the creature to evolve over time: first, because it must respond to changes in its environment, and second, because your wisdom grows and you think of better designs for the beast. It shouldn’t remain  in the same body forever!</p>

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/praise-the-creator.jpg" alt=""/></p>

<p>The creature, however, might be relying on features of its present anatomy. You can’t add wings or change its scales without warning. It needs an orderly process to adapt its lifestyle to its new body. How can you, as a responsible designer in charge of this creature’s natural history, gently coax it toward ever greater improvements?</p>

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/creator.jpg" alt=""/></p>

<p>It’s the same for responsible library maintainers. We keep our promises to the people who depend on our code: we release bugfixes and useful new features. We sometimes delete features if that’s beneficial for the library’s future. We continue to innovate, but we don’t break the code of people who use our library. How can we fulfill all those goals at once?</p>

<h1 id="add-useful-features">Add Useful Features</h1>

<p>Your library shouldn’t stay the same for eternity: you should add features that your make your library better for your users.
For example, if you have a Reptile class and it would be useful to have wings for flying, go for it.</p>

<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span>class</span> <span>Reptile</span>:
    <span>@property</span>
    <span>def</span> <span>teeth</span>(<span>self</span>):
        <span>return</span> <span>'sharp fangs'</span>

<span>    <span># If wings are useful, add them!</span>
</span><span>    <span>@property</span>
</span><span>    <span>def</span> <span>wings</span>(<span>self</span>):
</span><span>        <span>return</span> <span>'majestic wings'</span>
</span></code></pre></div>


<p>But beware, features come with risk.
Consider the following feature in the Python standard library, and let us see what went wrong with it.</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span>bool</span>(datetime<span>.</span>time(<span>9</span>, <span>30</span>)) <span>==</span> <span>True</span>
<span>bool</span>(datetime<span>.</span>time(<span>0</span>, <span>0</span>)) <span>==</span> <span>False</span>
</code></pre></div>

<p>This is peculiar: converting any time object to a boolean yields True, except for midnight. (Worse, the rules for timezone-aware times are even stranger.)
I’ve been writing Python for more than a decade but I didn’t discover this rule until last week. What kind of bugs can this odd behavior cause in users’ code?</p>

<p>Consider a calendar application with a function that creates events. If an event has an end time, the function requires it to also have a start time:</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span>def</span> <span>create_event</span>(day,
                 start_time<span>=</span><span>None</span>,
                 end_time<span>=</span><span>None</span>):
    <span>if</span> end_time <span>and</span> <span>not</span> start_time:
        <span>raise</span> <span>ValueError</span>(<span>"Can't pass end_time without start_time"</span>)

<span># The coven meets from midnight until 4am.</span>
create_event(datetime<span>.</span>date<span>.</span>today(),
             datetime<span>.</span>time(<span>0</span>, <span>0</span>),
             datetime<span>.</span>time(<span>4</span>, <span>0</span>))
</code></pre></div>

<p>Unfortunately for witches, an event starting at midnight fails this validation. A careful programmer who knows about the quirk at midnight can write this function correctly, of course:</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span>def</span> <span>create_event</span>(day,
                 start_time<span>=</span><span>None</span>,
                 end_time<span>=</span><span>None</span>):
    <span>if</span> end_time <span>is</span> <span>not</span> <span>None</span> <span>and</span> start_time <span>is</span> <span>None</span>:
        <span>raise</span> <span>ValueError</span>(<span>"Can't pass end_time without start_time"</span>)
</code></pre></div>

<p>But this subtlety is worrisome. If a library creator wanted to make an API that bites users, a “feature” like the boolean conversion of midnight works nicely.</p>

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/bite.jpg" alt=""/></p>

<p>The responsible creator’s goal, however, is to make your library easy to use correctly.</p>

<p>This feature was written by Tim Peters when he first made the datetime module in 2002. Even founding Pythonistas like Tim make mistakes. <a href="https://bugs.python.org/issue13936">The quirk was removed</a>, and all times are True now.</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span># Python 3.5 and later.</span>

<span>bool</span>(datetime<span>.</span>time(<span>9</span>, <span>30</span>)) <span>==</span> <span>True</span>
<span>bool</span>(datetime<span>.</span>time(<span>0</span>, <span>0</span>)) <span>==</span> <span>True</span>
</code></pre></div>

<p>Programmers who didn’t know about the oddity of midnight are saved from obscure bugs, but it makes me nervous to think about any code that actually relies on the weird old behavior and didn’t notice the change. It would have been better if this bad feature were never implemented at all. This leads us to the first promise of any library maintainer:</p>

<div><p>
First Covenant:<br/>Avoid Bad Features
</p></div>

<p>The most painful change to make is when you have to delete a feature. One way to avoid bad features is to add few features in general! Make no public method, class, function, or property without a good reason. Thus:</p>

<div><p>
Second Covenant:<br/>Minimize Features
</p></div>

<p>Features are like children: conceived in a moment of passion, they must be supported for years.
Don’t do anything silly just because you can. Don’t add feathers to a snake!</p>

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/feathers.jpg" alt=""/></p>

<p>But of course, there are plenty of occasions when users need something from your library that it does not yet offer. How do you choose the right feature to give them? Here’s another cautionary tale.</p>

<h1 id="a-cautionary-tale-from-asyncio">A Cautionary Tale From asyncio</h1>

<p>As you may know, when you call a coroutine function, it returns a coroutine object:</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span>async</span> <span>def</span> <span>my_coroutine</span>():
    <span>pass</span>

<span>print</span>(my_coroutine())
</code></pre></div>

<div class="highlight"><pre><code class="language-text" data-lang="text"><span/>&amp;lt;coroutine object my_coroutine at 0x10bfcbac8&amp;gt;
</code></pre></div>


<p>Your code must “await” this object to actually run the coroutine. It’s easy to forget this, so the asyncio developers wanted a “debug mode” that catches this mistake. Whenever a coroutine is destroyed without being awaited, the debug mode prints a warning with a traceback to the line where it was created.</p>

<p>When Yury Selivanov implemented the debug mode, he added as its foundation a “coroutine wrapper” feature. The wrapper is a function that takes in a coroutine and returns anything at all. Yury used it to install the warning logic on each coroutine, but someone else could use it to turn coroutines into the string “hi!”:</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span>import</span> <span>sys</span>

<span>def</span> <span>my_wrapper</span>(coro):
    <span>return</span> <span>'hi!'</span>

sys<span>.</span>set_coroutine_wrapper(my_wrapper)

<span>async</span> <span>def</span> <span>my_coroutine</span>():
    <span>pass</span>

<span>print</span>(my_coroutine())
</code></pre></div>




<p>That is one hell of a customization. It changes the very meaning of “async”. Calling set_coroutine_wrapper once will globally and permanently change all coroutine functions.
It is, as Nathaniel Smith wrote, “a problematic API” which is prone to misuse and had to be removed. The asyncio developers could have avoided the pain of deleting the feature if they’d better shaped it to its purpose. Responsible creators must keep this in mind:</p>

<div><p>
Third Covenant:<br/>Keep Features Narrow
</p></div>

<p>Luckily, Yury had the good judgment to mark this feature provisional, so asyncio users knew not to rely on it. Nathaniel was free to replace set_coroutine_wrapper with a narrower feature that only customized the traceback depth:</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span>import</span> <span>sys</span>

sys<span>.</span>set_coroutine_origin_tracking_depth(<span>2</span>)

<span>async</span> <span>def</span> <span>my_coroutine</span>():
    <span>pass</span>

<span>print</span>(my_coroutine())
</code></pre></div>

<div class="highlight"><pre><code class="language-text" data-lang="text"><span/>&lt;coroutine object my_coroutine at 0x10bfcbac8&gt;

RuntimeWarning:'my_coroutine' was never awaited

Coroutine created at (most recent call last)
  File "script.py", line 8, in &lt;module&gt;
    print(my_coroutine())
</code></pre></div>


<p>This is much better. There’s no more global setting that can change coroutines’ type, so asyncio users need not code as defensively. Deities should all be as farsighted as Yury:</p>

<div><p>
Fourth Covenant:<br/>Mark Experimental Features "Provisional"
</p></div>

<p>If you have merely a hunch that your creature wants horns and a quadruple-forked tongue, introduce the features but mark them “provisional”.</p>

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/horns.jpg" alt=""/></p>

<p>You might discover that the horns are adventitious but the quadruple-forked tongue is useful after all. In the next release of your library you can delete the former and mark the latter official.</p>

<h1 id="deleting-features">Deleting Features</h1>

<p>No matter how wisely we guide our creature’s evolution, there may come a time when it’s best to delete an official feature. For example, you might have created a lizard, and now you choose to delete its legs. Perhaps you want to transform this awkward creature into a sleek and modern python.</p>

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/lizard-to-snake.jpg" alt=""/></p>

<p>There are two main reasons to delete features. First, you might discover a feature was a bad idea, through user feedback or your own growing wisdom. That was the case with the quirky behavior of midnight. Or, the feature might have been well-adapted to your library’s environment at first, but the ecology changes. Perhaps another deity invents mammals. Your creature wants to squeeze into their little burrows and eat the tasty mammal filling, so it has to lose its legs.</p>

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/mammal.jpg" alt=""/></p>

<p>Similarly, the Python standard library deletes features in response to changes in the language itself. Consider asyncio’s Lock. It has been awaitable ever since “await” was added as a keyword:</p>

<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/>lock <span>=</span> asyncio<span>.</span>Lock()

<span>async</span> <span>def</span> <span>critical_section</span>():
<span>    <span>await</span> lock
</span>    <span>try</span>:
        <span>print</span>(<span>'holding lock'</span>)
    <span>finally</span>:
        lock<span>.</span>release()
</code></pre></div>


<p>But now, we can do “async with lock”:</p>

<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/>lock <span>=</span> asyncio<span>.</span>Lock()

<span>async</span> <span>def</span> <span>critical_section</span>():
<span>    <span>async</span> <span>with</span> lock:
</span>        <span>print</span>(<span>'holding lock'</span>)
</code></pre></div>


<p>The new style is much better! It’s short, and less prone to mistakes in a big function with other try-except blocks. Since “there should be one and preferably only one obvious way to do it” <a href="https://bugs.python.org/issue32253">the old syntax is deprecated in Python 3.7</a> and it will be banned soon.</p>

<p>It’s inevitable that ecological change will have this effect on your code too, so learn to delete features gently. Before you do so, consider the cost or benefit of deleting it. Responsible maintainers are reluctant to make their users change a large amount of their code, or change their logic. (Remember how painful it was when Python 3 removed the “u” string prefix, before it was added back.) If the code changes are mechanical, however, like a simple search and replace, or if the feature is dangerous, it may be worth deleting.</p>

<div><p>
Whether to Delete a Feature
</p></div>

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/scale.jpg" alt=""/></p>

<table class="pro-con">
<thead>
<tr>
<th>Con</th>
<th>Pro</th>
</tr>
</thead>

<tbody>
<tr>
<td>Code must change</td>
<td>Change is mechanical</td>
</tr>

<tr>
<td>Logic must change</td>
<td>Feature is dangerous</td>
</tr>
</tbody>
</table>

<p>In the case of our hungry lizard, we decide to delete its legs so it slither into a mouse’s hole and eat it. How do we go about this? We could just delete the <code>walk</code> method, changing code from this:</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span>class</span> <span>Reptile</span>:
    <span>def</span> <span>walk</span>(<span>self</span>):
        <span>print</span>(<span>'step step step'</span>)
</code></pre></div>

<p>To this:</p>

<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span>class</span> <span>Reptile</span>:
    <span>def</span> <span>slither</span>(<span>self</span>):
        <span>print</span>(<span>'slide slide slide'</span>)
</code></pre></div>


<p>That’s not a good idea, the creature is used to walking! Or, in terms of a library, your users have code that relies on the existing method. When they upgrade to the latest version of your library, their code will break.</p>

<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span># User's code. Oops!</span>
Reptile<span>.</span>walk()
</code></pre></div>


<p>Therefore responsible creators make this promise:</p>

<div><p>
Fifth Covenant:<br/>Delete Features Gently
</p></div>

<p>There’s a few steps involved in deleting a feature gently. Starting with a lizard that walks with its legs, you first add the new method, “slither”. Next, deprecate the old method.</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span>import</span> <span>warnings</span>

<span>class</span> <span>Reptile</span>:
    <span>def</span> <span>walk</span>(<span>self</span>):
        warnings<span>.</span>warn(
            <span>"walk is deprecated, use slither"</span>,
            <span>DeprecationWarning</span>, stacklevel<span>=2</span>)
        <span>print</span>(<span>'step step step'</span>)

    <span>def</span> <span>slither</span>(<span>self</span>):
        <span>print</span>(<span>'slide slide slide'</span>)
</code></pre></div>

<p>The Python warnings module is quite powerful. By default it prints warnings to stderr, only once per code location, but you can silence warnings or turn them into exceptions, among other options.</p>

<p>As soon as you add this warning to your library, PyCharm and other IDEs render the deprecated method with a strikethrough. Users know right away that the method is due for deletion.</p>

<pre><code>Reptile().<span>walk()</span></code></pre>

<p>What happens when they run their code with the upgraded library?</p>

<div class="highlight"><pre><code class="language-text" data-lang="text"><span/>&gt; python3 script.py

DeprecationWarning: walk is deprecated, use slither
  script.py:14: Reptile().walk()

step step step
</code></pre></div>


<p>By default, they see a warning on stderr, but the script succeeds and prints “step step step”. The warning’s traceback shows what line of the user’s code must be fixed. (That’s what the “stacklevel” argument does: it shows the call site that users need to change, not the line where the warning is generated.) Notice that the error message is instructive, it describes what a library user must do to migrate to the new version.</p>

<p>Your users will want to test their code and prove they call no deprecated library methods. Warnings alone won’t make unittests fail, but exceptions will. Python has a command-line option to turn deprecation warnings into exceptions:</p>

<div class="highlight"><pre><code class="language-text" data-lang="text"><span/>&gt; python3 -Werror::DeprecationWarning script.py

Traceback (most recent call last):
  File "script.py", line 14, in &lt;module&gt;
    Reptile().walk()
  File "script.py", line 8, in walk
    DeprecationWarning, stacklevel=2)
DeprecationWarning: walk is deprecated, use slither
</code></pre></div>


<p>Now, “step step step” is not printed, because the script terminates with an error.</p>

<p>So once you’ve released a version of your library that warns about the deprecated “walk” method, you can delete it safely in the next release. Right?</p>

<p>Consider what your library’s users might have in their projects’ requirements:</p>

<pre><code># User's requirements.txt has a dependency on the reptile package.
reptile
</code></pre>

<p>The next time they deploy their code, they’ll install the latest version of your library. If they haven’t yet handled all deprecations then their code will break, because it still depends on “walk”. You need to be gentler than this. There are three more promises you must keep to your users: to maintain a changelog, choose a version scheme, and write an upgrade guide.</p>

<div><p>
Sixth Covenant:<br/>Maintain a Changelog
</p></div>

<p>Your library must have a change log; its main purpose is to announce when a feature that your users rely on is deprecated or deleted.</p>

<div>
<p>Changes in Version 1.1</p>

<p><strong>New features</strong></p>

<ul><li>New function Reptile.slither()</li></ul>

<p><strong>Deprecations</strong></p>

<ul><li>Reptile.walk() is deprecated and will be removed in version 2.0, use slither()</li></ul></div>

<p>Responsible creators use version numbers to express how a library has changed, so users can make informed decisions about upgrading. A “version scheme” is a language for communicating the pace of change.</p>

<div><p>
Seventh Covenant:<br/>Choose a Version Scheme
</p></div>

<p>There are two schemes in widespread use, <a href="https://semver.org">semantic versioning</a> and time-based versioning. I recommend semantic versioning for nearly any library. The Python flavor thereof is defined in <a href="https://www.python.org/dev/peps/pep-0440/">PEP 440</a>, and tools like “pip” understand semantic version numbers.</p>

<p>If you choose semantic versioning for your library, you can delete its legs gently with version numbers like:</p>

<blockquote>
<p>1.0: First “stable” release, with walk()<br/>
1.1: Add slither(), deprecate walk()<br/>
2.0: Delete walk()</p>
</blockquote>

<p>Your users should depend on a range of your library’s versions like so:</p>

<pre><code># User's requirements.txt.
reptile&gt;=1,&lt;2
</code></pre>

<p>This allows them to upgrade automatically within a major release, receiving bugfixes and potentially raising some deprecation warnings, but not upgrading to the <strong>next</strong> major release and risking a change that breaks their code.</p>

<p>If you follow time-based version your releases might be numbered thus:</p>

<blockquote>
<p>2017.06.0: A release in June 2017<br/>
2018.11.0: Add slither(), deprecate walk()<br/>
2019.04.0: Delete walk()</p>
</blockquote>

<p>And users can depend on your library like:</p>

<pre><code># User's requirements.txt for time-based version.
reptile==2018.11.*
</code></pre>

<p>This is terrific, but how do your users know your versioning scheme and how to test their code for deprecations? You have to advise them how to upgrade.</p>

<div><p>
Eighth Covenant:<br/>Write an Upgrade Guide
</p></div>

<p>Here’s how a responsible library creator might guide users:</p>

<div>
<p>Upgrading to 2.0</p>

<p><strong>Migrate from Deprecated APIs</strong></p>

<p>See the <span>changelog</span> for deprecated features.</p>

<p><strong>Enable Deprecation Warnings</strong></p>

<p>Upgrade to 1.1 and test your code with:</p>

<code>python -Werror::DeprecationWarning</code>


<p>Now it's safe to upgrade.</p></div>

<p>You must teach users how to handle deprecation warnings by showing them the command line options. Not all Python programmers know this—I certainly have to look up the syntax each time. And take note, you must <strong>release</strong> a version that prints warnings from each deprecated API, so users can test with that version before upgrading again. In this example, version 1.1 is the bridge release. It allows your users to rewrite their code incrementally, fixing each deprecation warning separately until they have entirely migrated to the latest API. They can test changes to their code, and changes in your library, independently from each other, and isolate the cause of bugs.</p>

<p>If you chose semantic versioning, this transitional period lasts until the next major release, from 1.x to 2.0, or from 2.x to 3.0, and so on. The gentle way to delete a creature’s legs is to give it at least one version in which to adjust its lifestyle. Don’t remove the legs all at once!</p>

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/skink.jpg" alt=""/></p>

<p>Version numbers, deprecation warnings, the changelog, and the upgrade guide work together to gently evolve your library without breaking the covenant with your users. The <a href="https://twistedmatrix.com/documents/current/core/development/policy/compatibility-policy.html">Twisted project’s Compatibility Policy</a> explains this beautifully:</p>

<blockquote>
<p>“The First One’s Always Free”</p>

<p>Any application which runs without warnings may be upgraded one minor version of Twisted.</p>

<p>In other words, any application which runs its tests without triggering any warnings from Twisted should be able to have its Twisted version upgraded at least once with no ill effects except the possible production of new warnings.</p>
</blockquote>

<p>Now, we creator deities have gained the wisdom and power to add features by adding methods, and to delete them gently. We can also add features by adding parameters, but this brings a new level of difficulty. Are you ready?</p>

<h1 id="adding-parameters">Adding Parameters</h1>

<p>Imagine that you just gave your snake-like creature a pair of wings. Now you must allow it the choice whether to move by slithering or flying. Currently its “move” function takes one parameter:</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span># Your library code.</span>
<span>def</span> <span>move</span>(direction):
    <span>print</span>(f<span>'slither </span><span>{direction}</span><span>'</span>)

<span># A user's application.</span>
move(<span>'north'</span>)
</code></pre></div>

<p>You want to add a “mode” parameter, but this breaks your users’ code if they upgrade, because they pass only one argument:</p>

<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span># Your library code.</span>
<span>def</span> <span>move</span>(direction, mode):
    <span>assert</span> mode <span>in</span> (<span>'slither'</span>, <span>'fly'</span>)
    <span>print</span>(f<span>'</span><span>{mode}</span><span> </span><span>{direction}</span><span>'</span>)

<span># A user's application. Error!</span>
<span>move(<span>'north'</span>)
</span></code></pre></div>


<p>A truly wise creator promises not to break users’ code this way.</p>

<div><p>
Ninth Covenant:<br/>Add Parameters Compatibly
</p></div>

<p>To keep this covenant, add each new parameter with a default value that preserves the original behavior.</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span># Your library code.</span>
<span>def</span> <span>move</span>(direction, mode<span>=</span><span>'slither'</span>):
    <span>assert</span> mode <span>in</span> (<span>'slither'</span>, <span>'fly'</span>)
    <span>print</span>(f<span>'</span><span>{mode}</span><span> </span><span>{direction}</span><span>'</span>)

<span># A user's application.</span>
move(<span>'north'</span>)
</code></pre></div>

<p>Over time, parameters are the natural history of your function’s evolution. They’re listed oldest first, each with a default value. Library users can pass keyword arguments to opt in to specific new behaviors, and accept the defaults for all others.</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span># Your library code.</span>
<span>def</span> <span>move</span>(direction,
         mode<span>=</span><span>'slither'</span>,
         turbo<span>=</span><span>False</span>,
         extra_sinuous<span>=</span><span>False</span>,
         hail_lyft<span>=</span><span>False</span>):
    <span># ...</span>

<span># A user's application.</span>
move(<span>'north'</span>, extra_sinuous<span>=</span><span>True</span>)
</code></pre></div>

<p>There is a danger, however, that a user might write code like this:</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span># A user's application, poorly-written.</span>
move(<span>'north'</span>, <span>'slither'</span>, <span>False</span>, <span>True</span>)
</code></pre></div>

<p>What happens if, in the next major version of your library, you get rid of one of the parameters, like “turbo”?</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span># Your library code, next major version. "turbo" is deleted.</span>
<span>def</span> <span>move</span>(direction,
         mode<span>=</span><span>'slither'</span>,
         extra_sinuous<span>=</span><span>False</span>,
         hail_lyft<span>=</span><span>False</span>):
    <span># ...</span>


<span># A user's application, poorly-written.</span>
move(<span>'north'</span>, <span>'slither'</span>, <span>False</span>, <span>True</span>)
</code></pre></div>

<p>The user’s code still compiles, and this is a bad thing. The code stopped moving extra-sinuously and started hailing a Lyft, which was not the intention. I trust that you can predict what I’ll say next: deleting a parameter requires several steps. First, of course, deprecate the “turbo” parameter. I like a technique like this one that detects whether any user’s code is relying on this parameter:</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span># Your library code.</span>
_turbo_default <span>=</span> <span>object</span>()

<span>def</span> <span>move</span>(direction,
         mode<span>=</span><span>'slither'</span>,
         turbo<span>=</span>_turbo_default,
         extra_sinuous<span>=</span><span>False</span>,
         hail_lyft<span>=</span><span>False</span>):
    <span>if</span> turbo <span>is</span> <span>not</span> _turbo_default:
        warnings<span>.</span>warn(
            <span>"'turbo' is deprecated"</span>,
            <span>DeprecationWarning</span>,
            stacklevel<span>=2</span>)
    <span>else</span>:
        <span># The old default.</span>
        turbo <span>=</span> <span>False</span>
</code></pre></div>

<p>But your users might not notice the warning. Warnings are not very loud: they can be suppressed, or lost in log files. Users might heedlessly upgrade to the next major version of your library, the version that deletes “turbo”. Their code will run without error and silently do the wrong thing! As the Zen of Python says, “Errors should never pass silently.” Indeed, reptiles hear poorly, so you must correct them very loudly when they make mistakes.</p>

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/loudly.jpg" alt=""/></p>

<p>The best way to protect your users is with Python 3’s star syntax, which requires callers to pass keyword arguments.</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span># Your library code.</span>
<span># All arguments after "*" must be passed by keyword.</span>
<span>def</span> <span>move</span>(direction,
         <span>*</span>,
         mode<span>=</span><span>'slither'</span>,
         turbo<span>=</span><span>False</span>,
         extra_sinuous<span>=</span><span>False</span>,
         hail_lyft<span>=</span><span>False</span>):
    <span># ...</span>

<span># A user's application, poorly-written.</span>
<span># Error! Can't use positional args, keyword args required.</span>
move(<span>'north'</span>, <span>'slither'</span>, <span>False</span>, <span>True</span>)
</code></pre></div>

<p>With the star in place, this is the only syntax allowed:</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span># A user's application.</span>
move(<span>'north'</span>, extra_sinuous<span>=</span><span>True</span>)
</code></pre></div>

<p>Now when you delete “turbo”, you can be certain any user code that relies on it will fail loudly. If your library also supports Python 2, there’s no shame in that, you can simulate the star syntax thus (<a href="http://www.informit.com/articles/article.aspx?p=2314818">credit to Brett Slatkin</a>):</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span># Your library code, Python 2 compatible.</span>
<span>def</span> <span>move</span>(direction, <span>**</span>kwargs):
    mode <span>=</span> kwargs<span>.</span>pop(<span>'mode'</span>, <span>'slither'</span>)
    sinuous <span>=</span> kwargs<span>.</span>pop(<span>'extra_sinuous'</span>, <span>False</span>)
    lyft <span>=</span> kwargs<span>.</span>pop(<span>'hail_lyft'</span>, <span>False</span>)

    <span>if</span> kwargs:
        <span>raise</span> <span>TypeError</span>(<span>'Unexpected kwargs: </span><span>%r</span><span>'</span>
                        <span>%</span> kwargs)

    <span># ...</span>
</code></pre></div>

<p>Requiring keyword arguments is a wise choice, but it requires foresight. If you allow an argument to be passed positionally, you cannot convert it to keyword-only in a later release. So, add the star now. You can observe in the asyncio API that it uses the star pervasively in constructors, methods, and functions. Even though “Lock” only takes one optional parameter so far, the asyncio developers added the star right away. This is providential.</p>
<div class="highlight"><pre><code class="language-py3" data-lang="py3"><span/><span># In asyncio.</span>
<span>class</span> <span>Lock</span>:
    <span>def</span> <span>__init__</span>(<span>self</span>, <span>*</span>, loop<span>=</span><span>None</span>):
        <span># ...</span>
</code></pre></div>

<p>Now we’ve gained the wisdom to change methods and parameters while keeping our covenant with users. The time has come to try the most challenging kind of evolution: changing behavior without changing either methods or parameters.</p>

<h1 id="changing-behavior">Changing Behavior</h1>

<p>Let’s say your creature is a rattlesnake, and you want to teach it a new behavior.</p>

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/rattlesnake.jpg" alt=""/></p>

<p>Sidewinding! The creature’s body will appear the same, but its behavior will change. How can we prepare it for this step of its evolution?</p>

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/sidewinding.jpg" alt=""/></p>

<p>A responsible creator can learn from the following example in the Python standard library, when behavior changed without a new function or parameters. Once upon a time, the os.stat function was introduced to get file statistics, like the creation time. At first, times were always integers.</p>
<div class="highlight"><pre><code class="language-pycon" data-lang="pycon"><span/><span>&gt;&gt;&gt; </span>os<span>.</span>stat(<span>'file.txt'</span>)<span>.</span>st_ctime
<span>1540817862</span>
</code></pre></div>

<p>One day, the core developers decided to use floats for os.stat times, to give sub-second precision. But they worried that existing user code wasn’t ready for the change.
They created a setting in Python 2.3, “stat_float_times”, that was false by default. A user could set it to True to opt in to floating-point timestamps.</p>
<div class="highlight"><pre><code class="language-pycon" data-lang="pycon"><span/><span>&gt;&gt;&gt; </span><span># Python 2.3.</span>
<span>&gt;&gt;&gt; </span>os<span>.</span>stat_float_times(<span>True</span>)
<span>&gt;&gt;&gt; </span>os<span>.</span>stat(<span>'file.txt'</span>)<span>.</span>st_ctime
<span>1540817862.598021</span>
</code></pre></div>

<p>Starting in Python 2.5, float times became the default, so any new code written for 2.5 and later could ignore the setting and expect floats. Of course, you could set it to False to keep the old behavior, or set it to True to ensure the new behavior in all Python versions, and prepare your code for the day when stat_float_times is deleted.</p>

<p>Ages passed. In Python 3.1 the setting was deprecated to prepare people for the distant future, and finally, after its decades-long journey, <a href="https://bugs.python.org/issue31827">the setting was removed</a>. Float times are now the only option. It’s a long road, but responsible deities are patient because we know this gradual process has a good chance of saving users from unexpected behavior changes.</p>

<div><p>
Tenth Covenant:<br/>Change Behavior Gradually
</p></div>

<p>Here are the steps:</p>

<ul>
<li>Add a flag to opt in to the new behavior, default False, warn if it’s False<br/></li>
<li>Change default to True, deprecate flag entirely<br/></li>
<li>Remove the flag</li>
</ul>

<p>If you follow semantic versioning, the versions might be like so:</p>



<table class="versions-table" rules="rows">
<thead>
<tr>
<th>Library version</th><th>Library API</th><th>User code</th>
</tr>
</thead>
<tbody>
<tr>
</tr><tr><td>1.0</td><td>No flag</td><td>Expect old behavior</td>
</tr><tr><td>1.1</td><td>Add flag, default False,<br/>
warn if it's False </td><td>Set flag True,<br/>
handle new behavior</td></tr>
<tr><td>2.0</td><td>Change default to True,<br/>
deprecate flag entirely</td><td>Handle new behavior</td></tr>
<tr><td>3.0</td><td>Remove flag</td><td>Handle new behavior</td></tr>
</tbody></table>

<p>You need <strong>two</strong> major releases to complete the maneuver. If you had gone straight from “Add flag, default False, warn if it’s False” to “Remove flag” without the intervening release, your users’ code would be unable to upgrade. User code written correctly for 1.1, which sets the flag to True and handles the new behavior, must be able to upgrade to the next release with no ill effect except new warnings, but if the flag were deleted in the next release that code would break. A responsible deity never violates the Twisted policy: “The First One’s Always Free.”</p>

<h1 id="the-responsible-creator">The Responsible Creator</h1>

<p><img src="https://emptysqua.re/blog/api-evolution-the-right-way/demeter.jpg" alt=""/></p>

<p>Our ten covenants belong loosely in three categories:</p>

<h3 id="evolve-cautiously">Evolve Cautiously</h3>

<ol>
<li>Avoid Bad Features</li>
<li>Minimize Features</li>
<li>Keep Features Narrow</li>
<li>Mark Experimental Features “Provisional”</li>
<li>Delete Features Gently</li>
</ol>

<h3 id="record-history-rigorously">Record History Rigorously</h3>

<ol>
<li>Maintain a Changelog</li>
<li>Choose a Version Scheme</li>
<li>Write an Upgrade Guide</li>
</ol>

<h3 id="change-slowly-and-loudly">Change Slowly and Loudly</h3>

<ol>
<li>Add Parameters Compatibly</li>
<li>Change Behavior Gradually</li>
</ol>

<p>If you keep these covenants with your creature, you’ll be a responsible creator deity. Your creature’s body can evolve over time, forever improving and adapting to changes in its environment, but without sudden changes the creature isn’t prepared for. If you maintain a library, keep these promises to your users, and you can innovate your library without breaking the code of the people who rely on you.</p>

<hr/>

<p>Illustrations:</p>

<ul>
<li><a href="https://www.gutenberg.org/files/42224/42224-h/42224-h.htm">The World’s Progress, The Delphian Society, 1913</a></li>
<li><a href="https://publicdomainreview.org/product-att/artist/charles-owen/">Essay Towards a Natural History of Serpents, Charles Owen, 1742</a></li>
<li><a href="https://archive.org/details/onbatrachiarepti00cope/page/n3">On the batrachia and reptilia of Costa Rica: With notes on the herpetology and ichthyology of Nicaragua and Peru, Edward Drinker Cope, 1875</a></li>
<li><a href="https://www.flickr.com/photos/internetarchivebookimages/20556001490">Natural History, Richard Lydekker et. al., 1897</a></li>
<li><a href="https://www.oldbookillustrations.com/illustrations/stationery/">Mes Prisons, Silvio Pellico, 1843</a></li>
<li><a href="https://www.alamy.com/mediacomp/ImageDetails.aspx?ref=D7Y61W">Tierfotoagentur / m.blue-shadow</a></li>
<li><a href="https://www.vintag.es/2013/06/riding-alligator-c-1930s.html">From Los Angeles Public Library, 1930</a></li>
</ul>
</div>