Talk @shapescience

My python journey 2/4

I wrote my first line of Python in 2011, during a summer Internship at DxO Labs, a digital imaging startup. We wanted to quantify colored lens shading. The leading engineer had heard a lot of good about python’s booming scientific computing ecosystem… and I got started!

Integrating everything with Python

Python is a programming language that lets you work quickly and integrate systems more effectively.

That is how python is marketed on its homepage. This promise is ambitious and… fulfilled: application backends, scientific computing, interop with libraries, build CLI applications, deep learning… With python you can endlessly build and glue together stuff. Python is rarely the best at anything, but you can’t beat it without a sweat.

Integration and knowledge breadth also happens to be my key value in engineering teams, so python fits great!

Ironically, the fun thing with python is that it’s often called a “scripting language”. Despite this (usually derogatory…) qualification, writing scripts with it is often painful :upside_down_face:. What works well is indeed binding performant packages and libraries.[^python-scripts]

Writing Python

Python encourages an straightforward idiomatic style. I like that.[^abstractions]

Python promotes interactivity: while I don’t use an IDE, I love how easy it is to fire a REPL with ipython for quick sketching, or use notebooks to write-up experiments. For data analysis and visualisations I haven’t reached with python the zen of R’s tidyverse, but that bar is high! I value how easy it is to read, debug, even monkey-patch dependencies to clarify how they work.

Python is doing great: it’s more popular every year, everyone writes binding for it, and along javascript it became the default “accessible” programming languages. The package ecosystem is lively. In recent years, many quality of life features were added to the language and standard library: fstrings, pathlib, functools, and even the walrus operator (:=).

What I find annoying with Python

Along those features, type hints. While it follows important trends in language design (strongly typed boundaries and type inference within), it also tells something about python’s weaknesses. Relying on tests to avoid errors gets annoying, especially since so much could be checked by a compiler and expressive types. Should we really complain about that? How much should python change to address those issues? Do we really want a more complex python at the cost of ease of onboarding new users? Are we just going to solve this with linters and a pythonic TypeScript?

Packaging and distribution is difficult: I don’t like having to spend time explaining how packages work, the subtleties of relative imports, what python distributions like anaconda can provide (or not), setting up environments, dealing with bootstrapping python/pip, why upgrade to python3, which of the competiting project dependency manager is best… Today I yearn for simplicity: statically linked binaries.

Python looks so simple that many people never read the docs, never read any open-source code, or looked for best practices beyond a few tutorials and one-off stackoverflow answers. Or write C with python :sweat_smile:. While being welcoming is amazing, today I want tooling, like formatters and linters, enabled by default out of the box.

There are a few other things I don’t like about Python:

  • How exactly should I launch async tasks? I almost wrote a post about my troubles with asyncio. Or maybe I should use twisted? gevent, tornado? Even understanding what each tool provides takes effort. I guess I’m being a little unfair here, as I also feel language users are better served by small standard libraries and competitive package ecosystems… And anyway we can’t expect async to be resolved immediately! :wink:
  • Error handling is verbose, errors are annoying to print.
  • Python could be more elegant with destructuring, pipes… But that’s not the goal.
  • This post really touches good points about missing features from the language and the stdlib.

I also wish Google would stop ranking rushed outdated blog posts and the python2 docs above the official python3 documentation… (Is someone listening?)

Should I learn a new language?

Despite those complains, python is great. I only have strong opinions because I use it a lot!

In the end I want to get things done, so I don’t look forward to yet another big learning curve (hi rust :sweat_smile:). But time will tell, some day my curiosity might make me go beyond a new language’s manual!

[^python-scripts]: Calling subprocess functions is complicated, os.system is supposed to be deprecated. Dealing with paths is great with pathlib but no one seems to know about it. And anyway you often need to also use the mess that is shutils. Bundling strings together for the CLI used to be a mess until fstrings not so long ago… In the end scripts are only easy if a neat package wraps the application you interact with. At least you get something cross-platform without extra effort.

[^abstractions]: I don’t like abstractions. They are often just superfluous indirections that make the author feel smarter: “look at my metaprogramming skills!”, “here is big class hierarchy I architectured for you”.
Yes, I get that in large/complex projects, OOO adventures are a useful, but even then complexity should be fought teeth and nails. The good questions are API design, side-effects, and implementation, not how to abstract it all from day one.