title: Mercurial’s Journey to and Reflections on Python 3
Mercurial 5.2 was released on November 5, 2019. It is the first version of Mercurial that supports Python 3. This milestone comes nearly 11 years after Python 3.0 was first released on December 3, 2008.
Speaking as a maintainer of Mercurial and an avid user of Python, I feel like the experience of making Mercurial work with Python 3 is worth sharing because there are a number of lessons to be learned.
This post is logically divided into two sections: a mostly factual recount of Mercurial's Python 3 porting effort and a more opinionated commentary of the transition to Python 3 and the Python language ecosystem as a whole. Those who don't care about the mechanics of porting a large Python project to Python 3 may want to skip the next section or two.
Let's start with a brief history lesson of Mercurial's support for Python 3 as told by its own commit history.
The Mercurial version control tool was first released in April 2005
(the same month that Git was initially released). Version 1.0 came out
in March 2008. The first reference to Python 3 I found in the code base
was in September 2008.
Then not much happens for a while until
June 2010, when
someone authors a bunch of changes to make the Python C extensions
start to recognize Python 3. Then things were again quiet for a while
until January 2013,
when a handful of changes landed to remove 2 argument
raise. There were
a handful of commits in 2014 but nothing worth calling out.
Mercurial's meaningful journey to Python 3 started in 2015. In code, the work started in April 2015, with effort to make Mercurial's test harness run with Python 3. Part of this was a decision that Python 3.5 (to be released several months later in September 2015) would be the minimum Python 3 version that Mercurial would support.
Once the Mercurial Project decided it wanted to port to Python 3 (as opposed to another language), one of the earliest decisions was how to perform that port. Mercurial's code base was too large to attempt a flag day conversion where there would be a Python 2 version and a Python 3 version and one day everyone would switch from Python 2 to 3. Mercurial needed a way to run the same code (or as much of the same code) on both Python 2 and 3. We would maintain a single code base and users would gradually switch from running with Python 2 to Python 3.
In May 2015,
Mercurial dropped support for Python 2.4 and 2.5. Dropping support for
these older Python versions was critical, as it was effectively impossible to
write Python code that ran on this wide gamut of versions because of
incompatibilities in syntax and language features. For example, you needed
Python 2.6 to get
from __future__ import print_function.
The project's late start at a Python 3 port can be significantly attributed
to Python 2.4 and 2.5 compatibility holding us back.
The main goal with Mercurial's early porting work was just getting the code base
to a point where
import mercurial would work. There were a myriad of places
where Mercurial used syntax that was invalid on Python 3 and Python 3
couldn't even parse the source code, let alone compile it to bytecode and
This effort began in earnest in
with global source code rewrites like using modern octal syntax,
modern exception catching syntax (
except Exception as e instead of
except Exception, e),
print() instead of
from __future__ import absolute_import.
In the early days of the port, our first goal was to get all source code
parsing as valid Python 3. The next step was to get all the modules
cleanly. This entailed fixing code that ran at
import time to work on
Python 3. Our thinking was that we would need the code base to be
clean on Python 3 before seriously thinking about run-time behavior. In reality,
we quickly ported a lot of modules to
import cleanly and then moved on
to higher-level porting, leaving a long-tail of modules with
This initial porting effort played out over months. There weren't many people working on it in the early days: a few people would basically hack on Python 3 as a form of itch scratching and most of the project's energy was focused on improving the existing Python 2 based product. You can get a rough idea of the timeline and participation in the early porting effort through the history of test-check-py3-compat.t. We see the test being added in December 2015, By June 2016, most of the code base was ported to our modern import convention and we were ready to move on to more meaningful porting.
One of the biggest early hurdles in our porting effort was how to overcome
the string literals type mismatch between Python 2 and 3. In Python 2, a
'' string literal is a sequence of bytes. In Python 3, a
'' string literal
is a sequence of Unicode code points. These are fundamentally different types.
And in Mercurial's code base, most of our string types are binary by design:
use of a Unicode based
str for representing data is flat out wrong for our use
case. We knew that Mercurial would need to eventually switch many string
b'' to preserve type compatibility. But doing so would
In the early days of Mercurial's Python 3 port in 2015, Mercurial's project maintainer (Matt Mackall) set a ground rule that the Python 3 port shouldn't overly disrupt others: he wanted the Python 3 port to more or less happen in the background and not require every developer to be aware of Python 3's low-level behavior in order to get work done on the existing Python 2 code base. This may seem like a questionable decision (and I probably disagreed with him to some extent at the time because I was doing Python 3 porting work and the decision constrained this work). But it was the correct decision. Matt knew that it would be years before the Python 3 port was either necessary or resulted in a meaningful return on investment (the value proposition of Python 3 has always been weak to Mercurial because Python 3 doesn't demonstrate a compelling advantage over Python 2 for our use case). What Matt was trying to do was minimize the externalized costs that a Python 3 port would inflict on the project. He correctly recognized that maintaining the existing product and supporting existing users was more important than a long-term bet in its infancy.
This ground rule meant that a mass insertion of
b'' prefixes everywhere
was not desirable, as that would require developers to think about whether
a type was a
str, a distinction they didn't have to worry about
on Python 2 because we practically never used the Unicode-based string type in
In addition, there were some other practical issues with doing a bulk
prefix insertion. One was that the added
b characters would cause a lot of lines
to grow beyond our length limits and we'd have to reformat code. That would
require manual intervention and would significantly slow down porting. And
a sub-issue of adding all the
b prefixes and reformatting code is that it would
break annotate/blame more than was tolerable. The latter issue was addressed
by teaching Mercurial's annotate/blame feature to skip revisions. The project
now has a convention of annotating commit messages with
# skip-blame <reason>
so structural only changes can easily be ignored when performing an
A stop-gap solution to the
b'' everywhere issue came in
July 2016, when I
introduced a custom Python module importer that rewrote source code as part
import when running on Python 3. (I have
about this hack.) What this did was transparently add
b'' prefixes to all
un-prefixed string literals as well as modify how a few common functions were
called so that we wouldn't need to modify source code so things would run natively
on Python 3. The source transformer allowed us to have the benefits of progressing
in our Python 3 port without having to rewrite tens of thousands of lines of
source code. The solution was hacky. But it enabled us to make significant
progress on the Python 3 port without externalizing a lot of cost onto others.
I thought the source transformer would be relatively short-lived and would be removed shortly after the project inevitably decided to go all in on Python 3. To my surprise, others built additional transforms over the years and the source transformer persisted all the way until October 2019, when I removed it just before the first non-alpha Python 3 compatible version of Mercurial was released.
A common problem Mercurial faced with making the code base dual Python 2/3 native
was dealing with standard library differences. Most of the problems stemmed
from changes between Python 2.7 and 3.5+. But there are changes within the
versions of Python 3 that we had to wallpaper over as well. In
April 2016, the
mercurial.pycompat module was introduced to export aliases or wrappers around
standard library functionality to abstract the differences between Python
versions. This file grew over time
and eventually became
Mercurial's version of six. To be honest, I'm
not sure if we should have used
six from the beginning.
six probably would
have saved some work. But we had to eventually write a lot of shims for
bytes and would have needed to invent a
pycompat layer in some form anyway. So I'm not sure
six would have saved
enough effort to justify the baggage of integrating a 3rd party package into
Mercurial. (When Mercurial accepts a 3rd party package, downstream packagers
like Debian get all hot and bothered and end up making questionable patches
to our source code. So we prefer to minimize the surface area for
problems by minimizing dependencies on 3rd party packages.)
Once we had a source transforming module importer and the
compatibility shim, we started to focus in earnest on making core
functionality actually work on Python 3. We established a convention of
annotating changesets needed for Python 3 with
py3, so a
commit message search
yields a lot of the history. (But it isn't a full history since not every Python 3
oriented change used this convention). We see from that history that after
the source importer landed, a lot of porting effort was spent on things
very early in the
hg process lifetime. This included handling environment
variables, loading config files, and argument parsing. We introduced a
test to track the progress of
hg commands working in Python 3. The very early
history of that file shows the various error messages changing, as underlying
early process functionality was slowly ported to work on Python 3. By
December 2016, we
hg version working on Python 3!
hg command dispatch ported to Python 3 at the end of 2016,
2017 represented an inflection point in the Python 3 porting effort. With the
early process functionality working, different people could pick up different
commands and code paths and start making code work with Python 3. By
March 2017, basic
repository opening and
hg files worked. Shortly thereafter,
hg init started working as well.
And hg status and
hg commit did as well.
Within a few months, enough of Mercurial's functionality was working with Python 3 that we started to track which tests passed on Python 3. The evolution of this file shows a reasonable history of the porting velocity.
In May 2017, we dropped support for Python 2.6. This significantly reduced the complexity of supporting Python 3, as there was tons of functionality in Python 2.7 that made it easier to target both Python 2 and 3 and now our hands were untied to utilize it.
In November 2017, I landed a test harness feature to report exceptions seen during test runs. I later refined the output so the most frequent failures were reported more prominently. This feature greatly enabled our ability to target the most common exceptions, allowing us to write patches to fix the most prevalent issues on Python 3 and uncover previously unknown failures.
By the end of 2017, we had most of the structural pieces in place to complete the port. Essentially all that was required at that point was time and labor. We didn't have a formal mechanism in place to target porting efforts. Instead, people would pick up a component or test that they wanted to hack on and then make incremental changes towards making that work. All the while, we didn't have a strict policy on not regressing Python 3 and regressions in Python 3 porting progress were semi-frequent. Although we did tend to correct regressions quickly. And over time, developers saw a flurry of Python 3 patches and slowly grew awareness of how to accommodate Python 3, and the number of Python 3 regressions became less frequent.
As useful as the source-transforming module importer was, it incurred some
additional burden for the porting effort. The source transformer effectively
converted all un-prefixed string literals (
'') to bytes literals (
to preserve string type behavior with Python 2. But various aspects of Python
3 didn't like the existence of
bytes. Various standard library functionality
now wanted unicode
str and didn't accept
bytes, even though the Python
2 implementation used the equivalent of
bytes. So our
grew pretty large to accommodate calling into various standard library
functionality. Another side-effect which we didn't initially anticipate
**kwargs calling convention. Python allows you to use
with a dict with string keys to turn those keys into named arguments
in a function call. But Python 3 requires these
dict keys to be
str and outright rejects
bytes keys, even if the
is ASCII safe and has the same underlying byte representation of the
string data as the
str instance would. So we had to invent support
functions that would convert
dict keys from
**kwargs and another to convert a
**kwargs dict from
str keys to
bytes keys so we could use
'' syntax to access keys
in our source code! Also on the string type front, we had to sprinkle
the codebase with raw string literals (
r'') to force the use of
str irregardless of which Python version you were running on (our
source transformer only changed unprefixed string literals, so existing
r'' strings would be preserved as
Blind transformation of all string literals to
bytes was less than ideal
and it did impose some unwanted side-effects. But, again, most strings
in Mercurial are bytes by design, so we thought it would be easier to
byteify all strings then selectively undo that where native strings
were actually warranted (like keys in most
dicts) than to take the
up-front cost to examine every string and make an intelligent determination
as to what type it should be. I go back and forth as to whether this was the
correct call. But when you factor in that the source transforming
module importer unblocked Python 3 porting at a time in the project's
history when there was so much focus on improving the core product and it
did so without externalizing many costs onto the people doing the critical
core product work, I think it was the right call.
By mid 2019, the number of test failures in Python 3 had been whittled down to a reasonable, less daunting number. It felt like victory was in grasp and inevitable. But a few significant issues lingered.
One remaining question was around addressing differences between Python 3 versions. At the time, Python 3.5, 3.6, and 3.7 were released and 3.8 was scheduled for release by the end of the year. We had a surprising number of issues with differences in Python 3 versions. Many of us were running Python 3.7, so it had the fewest failures. We had to spend extra effort to get Python 3.5 and 3.6 working as well as 3.7. Same for 3.8.
Another task we deferred until the second half of 2019 was standing up robust CI for Python 3. We had some coverage, but it was minimal. Wanting a distraction from PyOxidizer for a bit and wanting to overhaul Mercurial's CI system (which is officially built on Buildbot), I cobbled together a serverless CI system built on top of AWS DynamoDB and S3 for storage, Lambda functions and CloudWatch events for all business logic, and EC2 spot instances for job execution. This CI system executed Python 3.5, 3.6, 3.7, and 3.8 variants of our test harness on Linux and Python 3.7 on Windows. This gave developers insight into version-specific failures. More importantly, it also gave insight into Windows failures, which was previously not well tested. It was discovered that Python 3 on Windows was lagging significantly behind POSIX.
By the time of the Mercurial developer meetup in October 2019, nearly all tests were passing on POSIX platforms and we were confident that we could declare Python 3 support as at least beta quality for the Mercurial 5.2 release, planned for early November.
One of our blockers for ripping off the alpha label on Python 3 support
was removing our source-transforming module importer. It had performance
implications and it wasn't something we wanted to ship because it felt
too hacky. A blocker for this was we wanted to automatically format
our source tree with black
because if we removed the source transformer, we'd have to rewrite
a lot of source code to apply changes the transformer was performing,
which would necessitate wrapping a lot of lines, which would involve a lot
of manual effort. We wanted to blacken our code base first so that
mass rewriting source code wouldn't involve a lot of tedious reformatting
black would handle that for us automatically. And rewriting the
source tree with
black was blocked on a specific feature landing in
black! (We did not agree with
black's behavior of
unwrapping comma-delimited lists of items if they could fit on a single
line. So one of our core contributors wrote a patch to
changed its behavior so a trailing
, in a list of items will force
items to be formatted on multiple lines. I personally find the multiple line
formatting much easier to read. And the behavior is arguably better for
code review and annotation, which is line based.) Once this feature
black, we reformatted our source tree and started ripping
out the source transformations, starting by inserting
everywhere. By late October, the source transformer was no more and
we were ready to release beta quality support for Python 3 (at least
on UNIX-like platforms).
Having described a mostly factual overview of Mercurial's port to Python 3, it is now time to shift gears to the speculative and opinionated parts of this post. I want to underscore that the opinions reflected here are my own and do not reflect the overall Mercurial Project or even a consensus within it.
Mercurial's port to Python 3 is still ongoing. While we've shipped Python 3 support and the test harness is clean on Python 3, I view shipping as only a milestone - arguably the most important one - in a longer journey. There's still a lot of work to do.
It is now 2020 and Python 2 support is now officially dead from the perspective of the Python language maintainers. Linux distributions are starting to rip out Python 2. Packages are dropping Python 2 support in new versions. The world is moving to Python 3 only. But Mercurial still officially supports Python 2. And it is still yet to be determined how long we will retain support for Python 2 in the code base. We've only had one release supporting Python 3. Our users still need to port their extensions (implemented in Python). Our users still need to start widely using Mercurial with Python 3. Even our own developers need to switch to Python 3 (old habits are hard to break).
I anticipate a long tail of random bugs in Mercurial on Python 3. While the tests may pass, our code coverage is not 100%. And even if it were, Python is a dynamic language and there are tons of invariants that aren't caught at compile time and can only be discovered at run time. These invariants cannot all be detected by tests, no matter how good your test coverage is. This is a feature/limitation of dynamic languages. Our users will likely be finding a long tail of miscellaneous bugs on Python 3 for years.
At present, our code base is littered with tons of random hacks to bridge the gap between Python 2 and 3. Once Python 2 support is dropped, we'll need to remove these hacks and make the source tree Python 3 native, with minimal shims to wallpaper over differences in Python 3 versions. Removing this Python version bridge code will likely require hundreds of commits and will be a non-trivial effort. It's likely to be deemed a low priority (it is glorified busy work after all), and code for the express purpose of supporting Python 2 will likely linger for years.
We are also still shoring up our packaging and distribution story on Python 3. This is easier on some platforms than others. I created PyOxidizer partially because of the poor experience I had with Python application packaging and distribution through the Mercurial Project. The Mercurial Project has already signed off on using PyOxidizer for distributing Mercurial in the future. So look for an oxidized Mercurial distribution in the near future! (You could argue PyOxidizer is an epic yak shave to better support Mercurial. But that's for another post.)
Then there's Windows support. A Python 3 powered Mercurial on Windows still has a handful of known issues. It may require a few more releases before we consider Python 3 on Windows to be stable.
Because we're still on a code base that must support Python 2, our
adoption of Python 3 features is very limited. The only Python 3
feature that Mercurial developers seem to almost universally get excited
about is type annotations. We already have some people playing around
pytype using comment-based annotations and
pytype has already
caught a few bugs. We're eager to go all in on type annotations and
uncover lots of dynamic typing bugs and poorly implemented APIs.
Beyond type annotations, I can't name any feature that people are screaming
to adopt and which makes a lot of sense for Mercurial. There's a long
tail of minor features I'm sure will get utilized. But none of the
marquee features that define major language releases seem that interesting
to us. Time will tell.
Having described Mercurial's ongoing journey to Python 3, I now want to focus more on Python itself. Again, the opinions here are my own and don't reflect those of the Mercurial Project.
Succinctly, my experience porting Mercurial and other projects to Python 3 has significantly soured my perceptions of Python. As much as I have historically loved Python - from the language to the welcoming community - I am still struggling to understand how Python could manage to inflict so much hardship on the community by choosing the transition plan that they did. I believe Python's choices represent a terrific example of what not to do when managing a large project or ecosystem. Maintainers of other largely-deployed systems would benefit from taking the time to understand and reflect on Python's missteps.
Python 3.0 was released on December 3, 2008. And it took the better part of a decade for the community to embrace it. This should be universally recognized as a failure. While hindsight is 20/20, many of the issues with Python 3 were obvious at the time and could have been mitigated had the language maintainers been more accommodating - and dare I say empathetic - to its users.
Initially, Python 3 had a rather cavalier attitude towards backwards and forwards compatibility. In the early years of Python 3, the attitude of Python's maintainers was Python 3 is a new, better language: you should target it explicitly. There were some tools and methods to ease the transition. But nothing super polished, especially in the early years. Adoption of Python 3 in the overall community was slow. Python developers in the wild justifiably complained that the value proposition of Python 3 was too weak to justify porting effort. Not helping was that the early advice for targeting Python 3 was to rewrite the source code to become Python 3 native. This is in contrast with using the same source to run on both Python 2 and 3. For library and application maintainers, this potentially meant maintaining separate versions of your code or forcing end-users to make a giant leap, which would realistically orphan users on an old version, fragmenting your user base. Neither of those were great alternatives, so you can understand why many projects didn't bite.
For many projects of non-trivial size, flag day transitions from Python 2 to 3 were simply not viable: the pathway to Python 3 was to make code dual Python 2/3 compatible and gradually switch over the runtime to Python 3. But initial versions of Python 3 made this effectively impossible! Let me give a few specific examples.
In Python 2, a string literal
'' is effectively an array of bytes. In
Python 3, it is a series of Unicode code points - a fundamentally different
type! In Python 2, you could write
b'' to be explicit that a string literal
was bytes or you could write
u'' to indicate a Unicode literal, mimicking
Python 3's behavior. In Python 3, you could write
b'' to create a
instance. But for whatever reason, Python 3 initially removed the
meaning there wasn't as easy way to explicitly denote the type of each
string literal so that it was consistent between Python 2 and 3! Python 3.3
(released September 2012) restored
u'' support, making it more viable to
write Python source code that worked on both Python 2 and 3. For nearly 4
years, Python 3 took away the consistent syntax for denoting bytes/Unicode
Another feature was
% formatting of strings. Python 2 allowed use of the
% formatting operator on both its string types. But Python 3 initially
removed the implementation of
bytes. Why, I have no clue. It
is perfectly reasonable to splice byte sequences into a buffer via use of
a formatting string. But the Python language maintainers insisted otherwise.
And it wasn't until the community complained about its absence loudly enough
that this feature was
restored in Python 3.5,
which was released in September 2015. Fun fact: the lack of this feature was
once considered a blocker for Mercurial moving to Python 3 because
bytes almost universally, which meant that nearly every use
% would have to be changed to something else. And to this day, Python
bytes still doesn't have a
format() method, so the alternative was
effectively string concatenation, which is a massive step backwards from the
The initial approach of Python 3 mirrors a folly that many developers and projects make: attempting a rewrite instead of performing incremental evolution. For established projects, large scale rewrites often go poorly. And Python 3 is no exception. Yes, from a code level, CPython (and likely other Python implementations) were incremental changes over Python 2 using the same code base. But from a language and standard library level, the differences in Python 3 were significant enough that I - and even Python's core maintainers - considered it a new language, and therefore a rewrite. When your random project attempts a rewrite and fails, the blast radius of that is often contained to that project. Maybe you don't publish a new release as soon as you otherwise would. But when you are powering an ecosystem, the ripple effects from a failed rewrite percolate throughout that ecosystem and last for years and have many second order effects. We see this with Python 3, where poor choices made in the late 2000s are inflicting significant hardship still in 2020.
From the initial restrained adoption of Python 3, it is obvious that the
Python ecosystem overwhelmingly rejected the initial boil the oceans approach
of Python 3. Python's maintainers eventually got the message and started
restoring features like
% formatting back into the
language to placate the community. All the while Python 3 had been accumulating
new features and the cumulative sum of those features was compelling enough
to win over users.
For many projects (including Mercurial), Python 3.4/3.5 was the first viable porting target for Python 3. Python 3.5 was released in September 2015, almost 7 years after Python 3.0 was released in December 2008. Seven. Years. An ecosystem that falters for that long is generally not healthy. What may have saved Python from total collapse here is that Python 2 was still going strong and people were generally happy with it. I really do think Python dodged a bullet here, because there was a massive window where the language could have hemorrhaged a critical amount of its user base and been relegated to an afterthought. One could draw an analogy to Perl, which lost out to PHP, Python, and Ruby, and whose fall from grace aligned with a lengthy transition from Perl 5 to 6.
If you look back at the early history of Python 3, I think you are forced to conclude that Python effectively kneecapped itself for 5-7 years through questionable implementation choices that prevented users from incurring incremental transitions between the major language versions. 2008 to 2013-2015 should be known as the lost years of Python because so much opportunity and energy was squandered. Yes, Python is still healthy today and Python 3 is (finally) being adopted at scale. But had earlier versions of Python 3 been more empathetic towards Python 2 users porting to it, Python and Python 3 in 2020 would be even stronger than it is. The community was artificially hindered for years. And we won't know until 2023-2025 what things could have looked like in 2020 had the Python core language team spent more time paving a smoother road between the major language versions.
To be clear, I do think Python 3 is generally a better language than Python 2. It has fewer warts, more compelling features, and better performance (except for startup time, which is still slower than Python 2). I am ecstatic the community is finally rallying around Python 3! For my Python coding, it has reached the point where I curse under my breath when I need to support Python 2 or even older versions of Python 3, like 3.5 or 3.6: I just wish the world would move on and adopt the future already!
But I would be remiss if I failed to mention some of my gripes with Python 3 beyond the transition shenanigans.
Perhaps my least favorite feature of Python 3 is its insistence that the
world is Unicode. In Python 2, the default string type was backed by
bytes. In Python 3, the default string type is backed by Unicode code
points. As part of that transition, large parts of the standard library
now operate in the Unicode space instead of the domain of bytes. I understand
why Python does this: they want strings to be Unicode and don't want
users to have to spend that much energy thinking about when to use
bytes. This approach is admirable and somewhat defensible
because it takes a stand on a solution that is arguably good enough for
most users. However, the approach of assuming the world is Unicode is
flat out wrong and has significant implications for systems level
applications (like version control tools).
There are a myriad of places in Python's standard library where Python
insists on using the Unicode-backed
str type and rejects
example, various networking modules refuse to accept
bytes for hostnames
or URLs. HTTP libraries won't accept
bytes for HTTP header names or values.
Functions that are proxies to POSIX-defined functions won't accept
even though the POSIX function it calls into is using
char * and isn't
Unicode aware. Then there's filename handling, where Python assumes the
existence of a global encoding for filenames and uses this encoding to convert
bytes. And it does this despite POSIX filesystem paths
being a bag of bytes where the only rules are that
\0 terminates the
/ is special.
In cases like Python refusing to accept
bytes for things like HTTP
header names (which will just be spit out over the wire as bytes), Python's
pendulum has swung too far towards Unicode only. In my opinion, Python needs
to be more accommodating and allow
bytes when it makes sense. I hope the
pendulum knocks some sense into people when it swings back towards a more
reasonable solution that better acknowledges the realities of the world we
For areas like filename handling, the world is more complicated. Python
is effectively an abstraction layer over the operating system APIs exposing
this functionality. And there is often an impedance mismatch between operating
systems. For example, POSIX (Linux) tends to use
char * for everything
and doesn't care about encoding and Windows tends to use 16 bit character
types where the encoding is... a can of worms.
The reality here is that it is impossible to abstract over differences between operating system behavior without compromises that can result in data loss, outright wrong behavior, or loss of functionality. But Python 3 attempts to do it anyway, making Python 3 unsuitable (or at least highly undesirable) for certain systems level applications that rely on it (like a version control tool).
In fairness to Python, it isn't the only programming language that gets
this wrong. The only language I've seen properly implement higher-order
abstractions on top of operating system facilities is Rust, whose approach can
be generalized as use Python 3's solution of normalizing to Unicode/UTF-8 by
default, but expose escape hatches which allow access to the raw underlying
types and APIs used by the operating system for the advanced consumers who
require it. For example, Rust's
Path type which represents a filesystem path
to the raw OsStr value
used by the operating system, not a normalization of it to bytes or Unicode,
which may be lossy. This allows consumers to e.g. create and retrieve
OS-native filesystem paths without data loss. This functionality is critical
in some domains. Python 3's awareness/insistence that the world is
Unicode (which it isn't universally) reduces Python's applicability in these
Speaking of Rust, at the Mercurial developer meetup in October 2019, we were discussing the use of Rust in Mercurial and one of the core maintainers blurted out something along the lines of if Rust were at its current state 5 years ago, Mercurial would have likely ported from Python 2 to Rust instead of Python 3. As crazy as it initially sounded, I think I agree with that assessment. With the benefit of hindsight, having been a key player in the Python 3 porting effort, seeing all the complications and headaches Python 3 is introducing, and having learned Rust and witnessed its benefits for performance, control, and correctness firsthand, porting to Rust would likely have been the correct move for the project at that point in time. 2020 is not 2014, however, and I'm not sure if I would opt for a rewrite in Rust today. (Most rewrites are follies after all.) But I know one thing: I certainly wouldn't implement a new version control tool in Python 3 and I would probably choose Rust as an implementation language for most new projects in the systems level space or with an expected shelf life of 10+ years. (I really should blog about how awesome Rust is.)
Back to the topic of Python itself, I'm really soured on Python at this point in time. The effort required to port to Python 3 was staggering. For Mercurial, Python 3 introduces a ton of problems and doesn't really solve many. We effectively sludged through mud for several years only to wind up in a state that feels strictly worse than where we started. I'm sure it will be strictly better in a few years. But at that point, we're talking about a 5+ year transition. To call the Python 3 transition disruptive and distracting for the project would be an understatement. As a project maintainer, it's natural to ask what we could have accomplished if we weren't forced to carry out this sideshow.
I can't shake the feeling that a lot of the pain afflicted by the Python 3
transition could have been avoided had Python's language leadership made
a different set of decisions and more highly prioritized the transition
experience. (Like not initially removing features like
and not introducing gratuitous backwards compatibility breaks, like with
items()/iteritems(). I would have also liked to see a feature like
from __future__ - maybe
from __past__ - that would make it easier for
Python 3 code to target semantics in earlier versions in order to provide
a more turnkey on-ramp onto new versions.) I simultaneously see Python 3
losing its position as a justifiable tool in some domains (like systems
level tooling) due to ongoing design decisions and poor implementation (like
startup overhead problems). (In contrast, I see Rust excelling where Python
is faltering and find Rust code surprisingly expressive to write and maintain
given how low-level it is and therefore feel that Rust is a compelling
alternative to Python in a surprisingly large number of domains.)
Look, I know it is easy for me to armchair quarterback and critique with the benefit of hindsight/ignorance. I'm sure there is a lot of nuance here. I'm sure there was disagreement within the Python community over a lot of these issues. Maintaining a large and successful programming language and community like Python's is hard and you aren't going to please all the people all the time. And speaking as a maintainer, I have mad respect for the people leading such a large community. But niceties aside, everyone knows the Python 3 transition was rough and could have gone better. It should not have taken 11 years to get to where we are today.
I'd like to encourage the Python Project to conduct a thorough postmortem on the transition to Python 3. Identify what went well, what could have gone better, and what should be done next time such a large language change is wanted. Speaking as a Python user, a maintainer of a Python project, and as someone in industry who is now skeptical about use of Python at work due to risks of potentially company crippling high-effort migrations in the future, a postmortem would help restore my confidence that Python's maintainers learned from the various missteps on the road to Python 3 and these potentially ecosystem crippling mistakes won't be made again.
Python had a wildly successful past few decades. And it can continue to thrive for several more. But the Python 3 migration was painful for all involved. And as much as we need to move on and leave Python 2 behind us, there are some important lessons to be learned. I hope the Python community takes the opportunity to reflect and am confident it will grow stronger by taking the time to do so.