Odd, I don't see any mention of subprocess.run, the workhorse of python scripting.
Quick rundown for the unfamiliar:
Give it a command as a list of strings (e.g., subprocess.run(["echo", "foo"]).)
It takes a bunch of flags, but the most useful (but not immediately obvious) ones are:
check=True: Raise an error if the command fails
capture_output=True: Captures stdout/stderr on the CompletedProcess
text=True: Automatically convert the stdout/stderr bytes to strings
By default, subprocess.run will print the stdout/stderr to the script's output (like bash, basically), so I only bother with capture_output if I need information in the output for a later step.
Also `asyncio.subprocess`, which lets you manage multiple concurrently running commands. Very handy if you need to orchestrate several commands together.
By default (1) captures stdout and stderr of all processes and (2) create tty for processs stdout.
Those are really bad defaults. The tty on stdout means many programs run in "interactive" rather then "batch" mode: programs which use pager get output truncated, auto-colors may get enabled and emit ESC controls into output streams (or not, depending on user's distro... fun!). And captured stderr means warnings and progress messages just disappear.
For example, this hangs forever without any output, at least if executed from interactive terminal:
from sh import man
print(man("tty"))
Compare to "subprocess" which does the right thing and returns manpage as a string:
Can you fix "sh"? sure, you need to bake in option to disable tty. But you've got to do it in _every_ script, or you'll see failure sooner or later. So it's much easier, not to mention safer, to simply use "subprocess". And as a bonus, one less dependency!
(Fun fact: back when "sh" first appeared, everyone was using "git log" as an example of why tty was bad (it was silently truncating data). They fixed it.. by disabling tty only for "git" command. So my example uses "man" :) )
`sh` is nice but it requires a dependency. No dependencies is nicer IMHO. uv makes this way easier but for low dependency systems, or unknown environments stdlib is king.
I love how this import trick shows how hackable Python is - and it’s this very hackability that has led to so many of the advances we see in AI. Arguably without operator overloads we’d be 5 or more years behind.
I think the point is that for most things, you don't need to call any external tools. Python's standard library comes already with lots of features, and there are many packages you can install.
The Python stdlib does not get enough credit. People complain about things like how its http client is dated and slow, but it’s pretty amazing that it’s just right there if you need it, no external dependencies needed. And it’s sitting right next to difflib, graphlib, pathlib, struct, glob, tkinter, and dozens of others. Sure, every one of these is limited individually, but those limitations are stable and well understood!
Currently developing a tkinter app, and loving, that I don't have to install an additional GUI framework. This will be much easier to package than with GTK or QT or Pyside or something. I am sure people have figured out all of that, but my project has so minimal dependencies, and yet offers a full GUI.
tkinter is the best kept secret in the Python std lib.
Pair it with numpy and matplotlib (two external dependencies that personally I consider part of Python itself), and you’ve got 80% of an interactive scientific simulation environment.
I strongly agree! Sure, the stuff in the Python stdlib might not be the best available - but it's always there. I've been in environments where I couldn't install requests but urllib2 meant that I could still get the job done. I think that every language should have a batteries-included stdlib.
How do you handle packages? I want scripts to a be a single file with a shebang, not a repo with a requirements.txt that I need to run in a venv. To me, this is the biggest blocker to using Python for any non-trivial scripting (which is precisely the kind where I wouldn't want to use bash), but I'd like to know how others deal with it.
C# scripts let you reference packages in a comment at the top of the file, for example:
As someone who writes a lot of python, I love uv, but isn't on nearly every system like python is, which is one of the arguments for using python here in the first place
The catch is that there could be a version mismatch between the version of Python installed on the end user's computer and the version on which the script was developed. This problem can be solved with uv, and there aren't really Python-native ways available.
What can you do with bash that isn't in the stdlib of python?
Generally the only nontrivial scripting I ever do is associated with a larger project, so I often already have a pyproject.toml and a UV environment, and I just add the dependencies to the dev group.
Well, that's kind of what I mean. For scripts in a python project, you can freely use whatever packages you need. But for one-off scripts, if you need bs4 or something, you're screwed. Either your script now has external dependencies or it requires special tooling.
It just feels strange that C# of all languages is now a better scripting tool than Python, at least out of the box. I did notice uv has exactly the feature I'm looking for, though it's obviously third-party:
Is everyone just using uv now instead of pip, perhaps? Or is just another alongside pipenv, conda, poetry, etc.? (Python's not my main these days, so I'm out of the loop.)
I don't understand. To return to GP's point, what can you do in bash that you can't do in Python? Said in another way, what does bash offer that you would need to tackle with a dependency in Python? My understanding is that there is no such thing, and accordingly, you can still end up with something that is better than bash if you just use Python and call out to other tools with subprocess.
There's bash. Then you need better loops/conditionals and more usable structures. That's when one should think of using a scripting language instead. I think the parent goes too far after that and what he's talking about is not something bash can do (well).
That said, a lot of very complicated things are actually written in bash. Distrobox I think is for example.
>That said, a lot of very complicated things are actually written in bash. Distrobox I think is for example.
They're only complicated BECAUSE they're written in bash. If they were written in Python they would be much less complicated, more modular, able to use many existing industrial strength well tested python modules instead of rolling their own ad-hoc ones, and much easier to maintain.
Yea that's a common theme of excuses for both Rust and Nix. Wrong though, because most anyone who can use a computer at all can learn the basics of Vim.
Seeing that flake.nix badge of complexity lets me know a project will be a nightmare to set up and will break every other week. It's usually right next to the Cargo.toml badge with 400 dependencies underneath.
To be honest I don't know what to say, you can use nix in many ways, and you don't even require to know the language.
The easiest entry-point is to just use it like a package manager, you install nix (which is just a command...) and then you have available the whole set of packages which are searchable from here: https://search.nixos.org/packages
nix-shell is just to download&add programs temporary to your PATH.
I don't feel that this is harder than something like "sudo apt install -y xxxxx" but for sure more robust and portable, and doesn't require sudo.
If at some point you want to learn the language in order to create configurations or packaging software, it may require to check a lot more documentation and examples, but for this I think it's pretty straightforward and is not harder than any other package manager like aptitude, homebrew or pacman.
I’m not sure what you mean by “a nightmare to set up”. You install Nix on your current OS with the determinate.systems installer, and you enter `nix run github:johndoe/project-containing-a-flake-dot-nix-file` to try out the project and have the full reproducible build taken care of by Nix.
Sure, installing packages the proper way requires a little bit more setup (Home Manager, most likely, and understanding where is the list of packages and which command to build to switch configuration), but as trivial as other complex tasks most of us hackers are capable of doing (like using `jq` or Vim).
Nix with Flakes never randomly break, I still have projects from 3 or 4 years ago that I can still run `nix build` and getting it running. Yes, if you try to update the `flake.lock` this may introduce breakages, but this is expected if you're pining `nixos-unstable` instead of a stable branch.
Whoa! This is a revelation. I already loved Nix and used nix-shell extensively, but this is the missing piece: fully reproducible Python scripts without compromise.
Or install direnv and put your dependencies into a shell.nix, setup the bash hook according the manual, create the .envrc with the content "use nix". Then type "direnv allow".
Then you can do e.g. use other persons python scripts without modifying their shebang.
Isn't the fact that bash doesn't have a wonderful ecosystem of reusable modules a much more enormous insurmountable problem than the fact that you have to install python modules? You're really missing the forest for the trees.
Not only does bash not have a module system like python, or a vast ecosystem of modules like python, but also that it's much too weak and brittle a language to implement most of those modules that Python has, and can't even call native code libraries directly.
Even with just its standard built in "batteries included" libraries and no extension modules or native code modules, Python is still much more powerful than bash and easier to code and maintain.
If you're complaining about having to install Python modules, you're usually already doing something that's impossible or incredibly difficult to do in bash anyway.
Even something as simple and essential as fetching files via http or parsing json. Bash has to call out to other programs to do that, but you have to install those programs too, and while Python can certainly call out to curl or wget or jq, it doesn't have to, since it has all that and more built in.
There really is no comparison, because bash loses along so many dimensions at once compared to Python.
once the script is non-trivial, 'install' it using pipx, in editable mode when you work on the script and as normal pipx installed cli utility otherwise.
Honestly, I just use a requirements.txt file. It's not as convenient as having a single file, it's true, but that is far outweighed by how much easier Python is to use than bash. So I'm giving up a 3/10 convenience but gaining a 10/10 convenience, which I think is worth it.
I think it is the fact that python packaging has been problematic for some time. setuptools/easy_install, pip/pipx, poetry, conda, uv all promise they will be the thing that "fixes" it
Eh. Python packaging is just fine. Set up a venv, install with pip, done. I think that the difficulty of installing Python packages is wildly overblown on this site.
Ok tbh bash uses the system package manager to install command-line utilities, and using the system package manager for python packages HAS been tried, and the parallel use of system packaging and independent package managers is part of why python package distribution is such a mess.
Why using independent package managers alongside the system one? I think the introduction of non system packagers is what brought us the whole mess we are in. Most system packagers allows for custom repositories.
Because the system package repository doesn't package everything & isn't always up to date. And if you introduce other repos to fix this, then you have an ecosystem with multiple instances of the same package distributed at different versions in different repositories, and God help you if you need multiple versions of one package, or if you need a different version of Python entirely. Nix could do it, but not anything else.
No—system python is for the system's scripts, and user projects should have their dependencies sandboxed in a virtual environment. That's the only model that really works.
If a script is simple - I use posix sh + awk, sed, etc.
But if a script I write needs to use arrays, sets, hashtable or processes many files - I use Nim[0]. It's a compiled systems-programming language that feels like a scripting language:
- Nim is easy to write and reads almost like a pseudocode.
- Nim is very portable language, runs almost anywhere C can run (both compiler and programs).
- `nim r script.nim` to compile and run (cached on subsequent runs) or use a shebang `#!/bin/env -S nim r`
- Nim programs are fast to compile (use debug mode and tcc compiler for almost instant compile times)
- Nim scripts run very fast <10ms (something that was very annoying to me with bash and Python)
- good chances you don't need external dependencies, because stdlib is batteries included and full of goodies.
- if you need external deps - just statically link them and distribute a cross-compiled binary (use zigcc[1] for easy Nim cross-compilation).
I might need to try it out. However, I haven't really found a use case yet where the speed of Python has been a major factor in my day job. It's usually fast enough and is a lot easier to optimize than many languages.
I actually sped up a script the other day that had been written in bash by 200x by moving it over to Python and rewriting the regexes so they could run on whole files all at once instead of line by line. Performance problems are most often from poorly written code in my experience, not a slow language.
> If a script is simple - I use posix sh + awk, sed, etc.
> But if a script I write needs to use arrays, sets, hashtable or processes many files
One option that I sometimes use at work (in addition to writing some Python CLIs) that is a pretty nice next step on this spectrum is Bash-with-all-the-fixins'.
I use a Nix-based dependency resolver for shell scripts called resholve¹ to parse scripts so that it can identify all of their dependencies (including Bash itself), then produce a "compiled" script as a Nix build where all of the references to external programs are replaced with pinned Nix store paths.
Then I have a fixed (and recent) version of GNU Bash regardless of platform, so I'm free to use Bash features that are newer or nicer than POSIX sh. My favorite such features are `mapfile` and `lastpipe` for writing in a more functional style, plus of course maps ("associative arrays").
I don't have to worry about portability problems with common utilities, because my scripts will bring along the expected implementations of coreutils, `find`, `grep`, etc. I'm free to use "modern" alternatives to classic utilities (like `rg` instead of GNU grep or `fd` instead of GNU findutils) if they offer better performance or more readable syntax. Polished interactivity is easy to build by just embedding a copy of `fzf`. And while I don't love Bash for working with structured data, it becomes a lot less painful when I can just pull in `jq`.
It's obviously got some disadvantages versus an option like Python, but the amount-of-code to functionality ratio is also much more favorable than Python's.
I typically use this for scripting build and development tasks in projects that already use Nix (via Devenv— I have some unpublished changes to the scripts module that add resholve and ShellCheck integration) to manage their environments, so there's no special packaging/distribution/setup burden.
IME it makes for vastly more readable and maintainable shell scripts than painstakingly limiting oneself to pure POSIX sh, so it can serve quite well as a middle ground option between barebones sh and a "real" programming language as your little script gradually grows in complexity.
> if you need external deps - just statically link them and distribute a cross-compiled binary (use zigcc[1] or easy Nim cross-compilation).
The deployment story sounds very slick and hard to beat! This is something I might want to try for scripts I want to be easy to distribute on macOS systems that don't have Nix installed.
At work my machine has probably ten or more installations of Python hidden in various tools. I'm certainly not alone. So we could say "on average Python is installed on every machine". /s
/*
Oh, you're looking at my CSS. Here be dragons.
Let's bundle in a rant while you're here: I'd like to use variable width fonts
instead of fixed size fonts. I enjoy 300 (i.e. light) and 600 (i.e. semibold)
over regular and bold for Crimson Pro only. But Chrome doesn't like to show
anything but 400 and 700 with variable width fonts, so I'll have to trick Chrome
by declaring my 300 and 600 fonts as if they were 400 and 700 respectively.
... or well, I could use font-variation-settings, but that's a global override
which means I have to specify it EVERYWHERE and that's just too much effort for
poor me.
*/
So they suggest to write scripts in Python rather than shell because Python is stable, probably installed on the target machine, has a big standard library, and is more readable. Many people do so.
That's the bright side of Python. They should mention the dark side, or Why _not_ to use Python for scripting.
First of all, the promise of easy portability breaks as soon as the script has dependencies. Try to install some Python program on a server where you're not root and a minimal python3 is installed.
The stability isn't very good in my experience either. I've often seen programs not compatible with recent releases of Python, either explicitly in the README or implicitly at runtime. Unmaintained Python code breaks.
Unfortunately, there is no silver bullet. Posix shell or bash may be better for simple scripts; Perl or Python if you know you won't require dependencies or if you have a good control on where to install the script; languages that compile to static executables are not really "scripting", but may be a better choice for long(term usage. These past years, I tend to keep away from Python as much as I can.
> First of all, the promise of easy portability breaks as soon as the script has dependencies.
And bash has a good dependency story? At least with python you can bundle your script with a requirements.txt file and it is doable for the target machine to get up and running.
I strongly agree with this.
At $WORK I usually work on projects comprising many small bits in various languages - some PowerShell here, some JS there, along with a "build process" that helps minify each and combine them into the final product.
After switching from shell scripts to Just and having to deal with a ton of issues on the way (how does quoting work in each system? How does argument passing? Environment variables?) I simply wrote a simple script with Python, UV shebang and PEP723 dependencies. Typer takes care of the command line parsing, and each "build target" is a simple, composable, readable python function that takes arguments and can call other ones if it needs to. Can't be simpler than that and the LLMs love it too.
I like the message the article is trying to convey, Python is good alternative to complicated shell scripts in my opinion.
I do wonder, let's say the scripting file is using lots of libraries, do you have to include some kind of requirements.txt file with it aswell when you want to share it with other people?
In Ruby, there is inline bundler which makes sharing a single Ruby script very portable.
I think the issue with dependencies is a bit overstated. There's uv for that. And shell scripts have dependencies as well! You're not guaranteed to have bash, jq, and curl on a minimal install.
That being said, for real portability of programs that have slightly outgrown the script moniker I really like Janet.
Funny this shows up on HN now! Jean Niklas wrote his thesis about RRB trees, which is an evolution of the persistent vectors of clojure fame. I have spent many hours reading his thesis lately, because
I just spent two months porting his c-rrb (https://github.com/hypirion/c-rrb) to c#, which was a fun endeavour. I wish I would have read hus thesis better since I spent two weeks debugging issues that arise from me not enforcing the leftwise dense invariant with regards to the tail.
I've never liked shell scripting. Last year, I switched my build system of a Rust project over to Python (Cargo is actually quite limited as a build system). For a newer project, I'm using Rust itself with the XTask pattern. I'm not sure if I prefer the Python or Rust approach yet.
I have been converting a lot of my makefiles to pyinvoke and fabric and it makes things so much easier to manage than bash or make. Don't know why I held on for so long.
Pretty much anything longer then a throwaway one liner I write in python.
Would be cool if python had a pipe operator though.
The back ticks in ruby is pretty ergonomic too. Wish python had a simpler way to run commands. Kind of tedious to look up subprocess run arguments and also break things up into arrays.
You can always set shell=True and pass in an entire command line as a string, but… don’t do that. It seems really nice until the first time you get the shell escaping wrong, and then it’s something you tend never to do again.
What is the difference in behavior? They both look like they would delete the user's home directory. I assume the latter would try to delete a directory literally named with a tilde instead?
The latter passes each item in the list into the child processes’s argv, as-is, without the shell parsing them. That means this would delete a single item named “~/ some file”, spaces and all, instead of three items named “~/“, “some”, and “file”.
Edit: I’m typing this on my phone, so brevity won over explicitness. The latter probably wouldn’t expand ~. Imagine a file named “/home/me/ some file” for a better example.
nushell (https://www.nushell.sh/) is essentially perfect for this use case! i use it rather than posix(y) shells for most projects where id normally reach for python.
I wrote another comment here about a strategy for writing portable Bash scripts without compromising on features and freely using arbitrary external commands. I wanted to give an example from the article of how I'd likely write one of his examples of a "somewhat unreadable shell script" in this style.
His ugly sh example:
morning_greetings=('hi' 'hello' 'good morning')
energetic_morning_greetings=()
for s in "${morning_greetings[@]}"; do
energetic_morning_greetings+=( "${s^^}!" )
done
and his more readable Python equivalent:
morning_greetings = ['hi', 'hello', 'good morning']
energetic_morning_greetings = \
[s.upper() + '!' for s in morning_greetings]
And I'd write the shell version in Bash something like this:
Does it still involve more syntax? Yeah. Printing arrays in Bash always involves some. But it's clearer at a glance what it does, and it doesn't involve mutating variables.
The piece this example is too simple to show (since it's focused only on data operations and not interacting with the filesystem or running external programs) is how much shorter the Bash usually ends up being than the Python equivalent.
Because the instant I want a dependency, I’m screwed. Half of what I want isn’t on CPAN, and the other half is ten years out of date and requires a compiler toolchain, fifty undocumented build dependencies at nonstandard paths, and five obscure/package-specific environment variables set in order to install. Python? Python either has it in the stdlib or has wheels available for pretty much everything.
Talking super simple stuff here, too: database drivers, markdown formatters, structured data parsers.
Odd, I don't see any mention of subprocess.run, the workhorse of python scripting.
Quick rundown for the unfamiliar:
Give it a command as a list of strings (e.g., subprocess.run(["echo", "foo"]).)
It takes a bunch of flags, but the most useful (but not immediately obvious) ones are:
By default, subprocess.run will print the stdout/stderr to the script's output (like bash, basically), so I only bother with capture_output if I need information in the output for a later step.Also `asyncio.subprocess`, which lets you manage multiple concurrently running commands. Very handy if you need to orchestrate several commands together.
I'm mostly using this nowadays >>> subprocess.getoutput('ls')
One thing I can recommend that makes scripting in python with external commands a lot easier is the `sh` module:
https://pypi.org/project/sh/
Basically you can just `from sh import [command]` and then have an installed binary command available as function
Please don't use "sh" python library!
By default (1) captures stdout and stderr of all processes and (2) create tty for processs stdout.
Those are really bad defaults. The tty on stdout means many programs run in "interactive" rather then "batch" mode: programs which use pager get output truncated, auto-colors may get enabled and emit ESC controls into output streams (or not, depending on user's distro... fun!). And captured stderr means warnings and progress messages just disappear.
For example, this hangs forever without any output, at least if executed from interactive terminal:
Compare to "subprocess" which does the right thing and returns manpage as a string: Can you fix "sh"? sure, you need to bake in option to disable tty. But you've got to do it in _every_ script, or you'll see failure sooner or later. So it's much easier, not to mention safer, to simply use "subprocess". And as a bonus, one less dependency!(Fun fact: back when "sh" first appeared, everyone was using "git log" as an example of why tty was bad (it was silently truncating data). They fixed it.. by disabling tty only for "git" command. So my example uses "man" :) )
> They fixed it.. by disabling tty only for "git" command.
Wow... yes sounds like a library to avoid!
I've used Plumbum for this for some projects at work, and really like it for this.
https://plumbum.readthedocs.io/en/latest/local_commands.html...
It also does argument parsing and validation, so it's generally pretty useful for writing little CLI tools that invoke other CLI tools.
https://plumbum.readthedocs.io/en/latest/cli.html
And for the opposite, where you keep your main pipeline in shell but want to use python for some parts of it, there is pypyp.
https://pypi.org/project/pypyp/
It takes cares of the input and output boilerplate so you can focus on the actual code that you wanted python for.
So cool! I made pawk [1] to get some of the same features, but yours is better! Congrats!
[1] https://github.com/jean-philippe-martin/pawk
`sh` is nice but it requires a dependency. No dependencies is nicer IMHO. uv makes this way easier but for low dependency systems, or unknown environments stdlib is king.
I love how this import trick shows how hackable Python is - and it’s this very hackability that has led to so many of the advances we see in AI. Arguably without operator overloads we’d be 5 or more years behind.
https://github.com/amoffat/sh/blob/2a90b1f87a877e5e09da32fd4...
uv for using sh as a dependency in scripts, managed inline, has changed it from “eh, I’ll just use subprocess” to “why not” for me.
https://docs.astral.sh/uv/guides/scripts/#using-different-py...
I love uv, the why not works great if it’s your machine but in places without your machine uv is just another step for a customer.
Does this give live output (meaning before completion) of processes run?
I think the point is that for most things, you don't need to call any external tools. Python's standard library comes already with lots of features, and there are many packages you can install.
The Python stdlib does not get enough credit. People complain about things like how its http client is dated and slow, but it’s pretty amazing that it’s just right there if you need it, no external dependencies needed. And it’s sitting right next to difflib, graphlib, pathlib, struct, glob, tkinter, and dozens of others. Sure, every one of these is limited individually, but those limitations are stable and well understood!
Currently developing a tkinter app, and loving, that I don't have to install an additional GUI framework. This will be much easier to package than with GTK or QT or Pyside or something. I am sure people have figured out all of that, but my project has so minimal dependencies, and yet offers a full GUI.
tkinter is the best kept secret in the Python std lib.
Pair it with numpy and matplotlib (two external dependencies that personally I consider part of Python itself), and you’ve got 80% of an interactive scientific simulation environment.
In fact, I am planning to use matplotlib soon, to visualize progress in learning, since my app is a language learning app.
I strongly agree! Sure, the stuff in the Python stdlib might not be the best available - but it's always there. I've been in environments where I couldn't install requests but urllib2 meant that I could still get the job done. I think that every language should have a batteries-included stdlib.
Absolutely agree, but it's funny that you mentioned graphlib. It has a single algorithm (topological sort)!
The sqlite, tkinter, and shelve modules are the ones I find most impressive.
How do you handle packages? I want scripts to a be a single file with a shebang, not a repo with a requirements.txt that I need to run in a venv. To me, this is the biggest blocker to using Python for any non-trivial scripting (which is precisely the kind where I wouldn't want to use bash), but I'd like to know how others deal with it.
C# scripts let you reference packages in a comment at the top of the file, for example:
https://devblogs.microsoft.com/dotnet/announcing-dotnet-run-...
You can specify requirements at the top of file and uv can run the script after automatically installing the dependencies.
https://avilpage.com/2025/04/learn-python-uv-in-100-seconds....
This works really well in my experience, but it does mean you need to have a working internet connection the first time you run the script.
Then: It manages a disposable hidden virtual environment automatically, via a very fast symlink-based caching mechanism.You can also add a shebang line so you can execute it directly:
Then:If not having a working internet connection is a concern, I would package your script into a shiv zipapp - https://shiv.readthedocs.io/en/latest/
I wish env -S was more portable. It's a newer feature of the coreutils env implementation and isn't supported elsewhere afaik.
You can use so-called "exec magic" instead of `env -S`. Here is an explanation with Python examples: https://dbohdan.com/scripts-with-dependencies#exec-magic (disclosure: my site). In short:
On systems that can't run uv, like NetBSD and OpenBSD, switch to pipx:FreeBSD 6.0 added 'env -S'. They have adopted a few different GNU inspired options recently, which I am happy about.
As someone who writes a lot of python, I love uv, but isn't on nearly every system like python is, which is one of the arguments for using python here in the first place
The catch is that there could be a version mismatch between the version of Python installed on the end user's computer and the version on which the script was developed. This problem can be solved with uv, and there aren't really Python-native ways available.
Sure but the sub question here was about packages.
If you are installing packages, then starting with installing uv should be fine.
https://peps.python.org/pep-0723/, e.g.:
What can you do with bash that isn't in the stdlib of python?
Generally the only nontrivial scripting I ever do is associated with a larger project, so I often already have a pyproject.toml and a UV environment, and I just add the dependencies to the dev group.
Well, that's kind of what I mean. For scripts in a python project, you can freely use whatever packages you need. But for one-off scripts, if you need bs4 or something, you're screwed. Either your script now has external dependencies or it requires special tooling.
It just feels strange that C# of all languages is now a better scripting tool than Python, at least out of the box. I did notice uv has exactly the feature I'm looking for, though it's obviously third-party:
https://docs.astral.sh/uv/guides/scripts/#declaring-script-d...
Is everyone just using uv now instead of pip, perhaps? Or is just another alongside pipenv, conda, poetry, etc.? (Python's not my main these days, so I'm out of the loop.)
I don't understand. To return to GP's point, what can you do in bash that you can't do in Python? Said in another way, what does bash offer that you would need to tackle with a dependency in Python? My understanding is that there is no such thing, and accordingly, you can still end up with something that is better than bash if you just use Python and call out to other tools with subprocess.
There's bash. Then you need better loops/conditionals and more usable structures. That's when one should think of using a scripting language instead. I think the parent goes too far after that and what he's talking about is not something bash can do (well).
That said, a lot of very complicated things are actually written in bash. Distrobox I think is for example.
>That said, a lot of very complicated things are actually written in bash. Distrobox I think is for example.
They're only complicated BECAUSE they're written in bash. If they were written in Python they would be much less complicated, more modular, able to use many existing industrial strength well tested python modules instead of rolling their own ad-hoc ones, and much easier to maintain.
UV is taking over really fast, it seems to be much more popular any other option.
I suspect conda still has some market share too but I've never needed it.
It's amazing what happens when something just works.
Nix allows you to do this with any language and required dependency: https://wiki.nixos.org/wiki/Nix-shell_shebang
Now you have to set up Nix first and deal with that nightmare of a learning curve. Just to auto-install some dependencies for a script.
Might as well rewrite all your scripts in Rust too while you're layering on unholy amounts of complexity.
It’s like Vim, you learn it once, and you keep using it forever once you’re used to it.
I’m so thankful to see a flake.nix file in every single cool project on code forges.
Yea that's a common theme of excuses for both Rust and Nix. Wrong though, because most anyone who can use a computer at all can learn the basics of Vim.
Seeing that flake.nix badge of complexity lets me know a project will be a nightmare to set up and will break every other week. It's usually right next to the Cargo.toml badge with 400 dependencies underneath.
To be honest I don't know what to say, you can use nix in many ways, and you don't even require to know the language.
The easiest entry-point is to just use it like a package manager, you install nix (which is just a command...) and then you have available the whole set of packages which are searchable from here: https://search.nixos.org/packages
nix-shell is just to download&add programs temporary to your PATH.
I don't feel that this is harder than something like "sudo apt install -y xxxxx" but for sure more robust and portable, and doesn't require sudo.
If at some point you want to learn the language in order to create configurations or packaging software, it may require to check a lot more documentation and examples, but for this I think it's pretty straightforward and is not harder than any other package manager like aptitude, homebrew or pacman.
I’m not sure what you mean by “a nightmare to set up”. You install Nix on your current OS with the determinate.systems installer, and you enter `nix run github:johndoe/project-containing-a-flake-dot-nix-file` to try out the project and have the full reproducible build taken care of by Nix.
Sure, installing packages the proper way requires a little bit more setup (Home Manager, most likely, and understanding where is the list of packages and which command to build to switch configuration), but as trivial as other complex tasks most of us hackers are capable of doing (like using `jq` or Vim).
Nix with Flakes never randomly break, I still have projects from 3 or 4 years ago that I can still run `nix build` and getting it running. Yes, if you try to update the `flake.lock` this may introduce breakages, but this is expected if you're pining `nixos-unstable` instead of a stable branch.
Whoa! This is a revelation. I already loved Nix and used nix-shell extensively, but this is the missing piece: fully reproducible Python scripts without compromise.
Or install direnv and put your dependencies into a shell.nix, setup the bash hook according the manual, create the .envrc with the content "use nix". Then type "direnv allow".
Then you can do e.g. use other persons python scripts without modifying their shebang.
Isn't the fact that bash doesn't have a wonderful ecosystem of reusable modules a much more enormous insurmountable problem than the fact that you have to install python modules? You're really missing the forest for the trees.
Not only does bash not have a module system like python, or a vast ecosystem of modules like python, but also that it's much too weak and brittle a language to implement most of those modules that Python has, and can't even call native code libraries directly.
Even with just its standard built in "batteries included" libraries and no extension modules or native code modules, Python is still much more powerful than bash and easier to code and maintain.
If you're complaining about having to install Python modules, you're usually already doing something that's impossible or incredibly difficult to do in bash anyway.
Even something as simple and essential as fetching files via http or parsing json. Bash has to call out to other programs to do that, but you have to install those programs too, and while Python can certainly call out to curl or wget or jq, it doesn't have to, since it has all that and more built in.
There really is no comparison, because bash loses along so many dimensions at once compared to Python.
I don't have much need of this personally, but I was playing around with an example from earlier in the thread and ended up with this:
which is a pretty nice experience assuming you already have `uv` in the target environment.My principles is that I do not. If PyPI packages are needed, rewrite it in Rust (or Go or D or whatever allows me to use statically-linked libraries).
Python packages are fine for servers but not for CLI tools.
Isn't pipx addressing exactly that?
once the script is non-trivial, 'install' it using pipx, in editable mode when you work on the script and as normal pipx installed cli utility otherwise.
the venv part is then completely under the hood.
What is wrong with writing a short wrapper in bash to activate a conda env before running the script? Too unsexy?
what can you do with bash that you cannot do with the python standard library and shelling out?
Honestly, I just use a requirements.txt file. It's not as convenient as having a single file, it's true, but that is far outweighed by how much easier Python is to use than bash. So I'm giving up a 3/10 convenience but gaining a 10/10 convenience, which I think is worth it.
> How do you handle packages?
The same way you handle them with bash?
Install them?
What are we talking about here?
I think it is the fact that python packaging has been problematic for some time. setuptools/easy_install, pip/pipx, poetry, conda, uv all promise they will be the thing that "fixes" it
Eh. Python packaging is just fine. Set up a venv, install with pip, done. I think that the difficulty of installing Python packages is wildly overblown on this site.
It's not overblown.
There's 10 different package managers and none of them seem to have any common conventions.
To even run a script you need to setup a venv and enter it by sourcing scripts.
Try to move a venv, it breaks.
Try to understand how/what/where it installs things its a black box if you are not "in" the ecosystem.
Node seems easy in comparison. I hate how python makes it so hard to use.
not “AI Bubble” overblown but close
What does Bash use for its packaging?
Use that.
Ok tbh bash uses the system package manager to install command-line utilities, and using the system package manager for python packages HAS been tried, and the parallel use of system packaging and independent package managers is part of why python package distribution is such a mess.
Why using independent package managers alongside the system one? I think the introduction of non system packagers is what brought us the whole mess we are in. Most system packagers allows for custom repositories.
Because the system package repository doesn't package everything & isn't always up to date. And if you introduce other repos to fix this, then you have an ecosystem with multiple instances of the same package distributed at different versions in different repositories, and God help you if you need multiple versions of one package, or if you need a different version of Python entirely. Nix could do it, but not anything else.
No—system python is for the system's scripts, and user projects should have their dependencies sandboxed in a virtual environment. That's the only model that really works.
If a script is simple - I use posix sh + awk, sed, etc.
But if a script I write needs to use arrays, sets, hashtable or processes many files - I use Nim[0]. It's a compiled systems-programming language that feels like a scripting language:
- Nim is easy to write and reads almost like a pseudocode.
- Nim is very portable language, runs almost anywhere C can run (both compiler and programs).
- `nim r script.nim` to compile and run (cached on subsequent runs) or use a shebang `#!/bin/env -S nim r`
- Nim programs are fast to compile (use debug mode and tcc compiler for almost instant compile times)
- Nim scripts run very fast <10ms (something that was very annoying to me with bash and Python)
- good chances you don't need external dependencies, because stdlib is batteries included and full of goodies.
- if you need external deps - just statically link them and distribute a cross-compiled binary (use zigcc[1] for easy Nim cross-compilation).
[0] - https://nim-lang.org
[1] - https://github.com/enthus1ast/zigcc
I might need to try it out. However, I haven't really found a use case yet where the speed of Python has been a major factor in my day job. It's usually fast enough and is a lot easier to optimize than many languages.
I actually sped up a script the other day that had been written in bash by 200x by moving it over to Python and rewriting the regexes so they could run on whole files all at once instead of line by line. Performance problems are most often from poorly written code in my experience, not a slow language.
> If a script is simple - I use posix sh + awk, sed, etc.
> But if a script I write needs to use arrays, sets, hashtable or processes many files
One option that I sometimes use at work (in addition to writing some Python CLIs) that is a pretty nice next step on this spectrum is Bash-with-all-the-fixins'.
I use a Nix-based dependency resolver for shell scripts called resholve¹ to parse scripts so that it can identify all of their dependencies (including Bash itself), then produce a "compiled" script as a Nix build where all of the references to external programs are replaced with pinned Nix store paths.
Then I have a fixed (and recent) version of GNU Bash regardless of platform, so I'm free to use Bash features that are newer or nicer than POSIX sh. My favorite such features are `mapfile` and `lastpipe` for writing in a more functional style, plus of course maps ("associative arrays").
I don't have to worry about portability problems with common utilities, because my scripts will bring along the expected implementations of coreutils, `find`, `grep`, etc. I'm free to use "modern" alternatives to classic utilities (like `rg` instead of GNU grep or `fd` instead of GNU findutils) if they offer better performance or more readable syntax. Polished interactivity is easy to build by just embedding a copy of `fzf`. And while I don't love Bash for working with structured data, it becomes a lot less painful when I can just pull in `jq`.
It's obviously got some disadvantages versus an option like Python, but the amount-of-code to functionality ratio is also much more favorable than Python's.
I typically use this for scripting build and development tasks in projects that already use Nix (via Devenv— I have some unpublished changes to the scripts module that add resholve and ShellCheck integration) to manage their environments, so there's no special packaging/distribution/setup burden.
IME it makes for vastly more readable and maintainable shell scripts than painstakingly limiting oneself to pure POSIX sh, so it can serve quite well as a middle ground option between barebones sh and a "real" programming language as your little script gradually grows in complexity.
> if you need external deps - just statically link them and distribute a cross-compiled binary (use zigcc[1] or easy Nim cross-compilation).
The deployment story sounds very slick and hard to beat! This is something I might want to try for scripts I want to be easy to distribute on macOS systems that don't have Nix installed.
--
1: https://github.com/abathur/resholve
> Python is installed on pretty much every machine
> Python 3 is installed on basically every machine out there.
> Python will work the same on all the machines you run your script on
No, no, and no.
More like, "no, but installation is easy", "yes, if the machine is safe", "depends on what you do"
At work my machine has probably ten or more installations of Python hidden in various tools. I'm certainly not alone. So we could say "on average Python is installed on every machine". /s
This from the site:
So they suggest to write scripts in Python rather than shell because Python is stable, probably installed on the target machine, has a big standard library, and is more readable. Many people do so.
That's the bright side of Python. They should mention the dark side, or Why _not_ to use Python for scripting.
First of all, the promise of easy portability breaks as soon as the script has dependencies. Try to install some Python program on a server where you're not root and a minimal python3 is installed.
The stability isn't very good in my experience either. I've often seen programs not compatible with recent releases of Python, either explicitly in the README or implicitly at runtime. Unmaintained Python code breaks.
Unfortunately, there is no silver bullet. Posix shell or bash may be better for simple scripts; Perl or Python if you know you won't require dependencies or if you have a good control on where to install the script; languages that compile to static executables are not really "scripting", but may be a better choice for long(term usage. These past years, I tend to keep away from Python as much as I can.
> First of all, the promise of easy portability breaks as soon as the script has dependencies.
And bash has a good dependency story? At least with python you can bundle your script with a requirements.txt file and it is doable for the target machine to get up and running.
Python relies on env variables to find where to load modules. Updating those variables is what venv does essentially.
I strongly agree with this. At $WORK I usually work on projects comprising many small bits in various languages - some PowerShell here, some JS there, along with a "build process" that helps minify each and combine them into the final product. After switching from shell scripts to Just and having to deal with a ton of issues on the way (how does quoting work in each system? How does argument passing? Environment variables?) I simply wrote a simple script with Python, UV shebang and PEP723 dependencies. Typer takes care of the command line parsing, and each "build target" is a simple, composable, readable python function that takes arguments and can call other ones if it needs to. Can't be simpler than that and the LLMs love it too.
I like the message the article is trying to convey, Python is good alternative to complicated shell scripts in my opinion.
I do wonder, let's say the scripting file is using lots of libraries, do you have to include some kind of requirements.txt file with it aswell when you want to share it with other people?
In Ruby, there is inline bundler which makes sharing a single Ruby script very portable.
https://bundler.io/guides/bundler_in_a_single_file_ruby_scri...
uv does that these days:
One would be well-advised to provide not only a requirements.txt file, but also a lock file, if the use case is important enough.
I think the issue with dependencies is a bit overstated. There's uv for that. And shell scripts have dependencies as well! You're not guaranteed to have bash, jq, and curl on a minimal install.
That being said, for real portability of programs that have slightly outgrown the script moniker I really like Janet.
https://janet.guide/scripting/
It can compile your script to a static binary for distribution.
Xonsh is perfect for this: https://xon.sh
Funny this shows up on HN now! Jean Niklas wrote his thesis about RRB trees, which is an evolution of the persistent vectors of clojure fame. I have spent many hours reading his thesis lately, because I just spent two months porting his c-rrb (https://github.com/hypirion/c-rrb) to c#, which was a fun endeavour. I wish I would have read hus thesis better since I spent two weeks debugging issues that arise from me not enforcing the leftwise dense invariant with regards to the tail.
I've never liked shell scripting. Last year, I switched my build system of a Rust project over to Python (Cargo is actually quite limited as a build system). For a newer project, I'm using Rust itself with the XTask pattern. I'm not sure if I prefer the Python or Rust approach yet.
I quite liked SCons back when I wrote C++!
I have been converting a lot of my makefiles to pyinvoke and fabric and it makes things so much easier to manage than bash or make. Don't know why I held on for so long.
Pretty much anything longer then a throwaway one liner I write in python.
Would be cool if python had a pipe operator though.
The back ticks in ruby is pretty ergonomic too. Wish python had a simpler way to run commands. Kind of tedious to look up subprocess run arguments and also break things up into arrays.
You can always set shell=True and pass in an entire command line as a string, but… don’t do that. It seems really nice until the first time you get the shell escaping wrong, and then it’s something you tend never to do again.
For example,
and have significant different behavior.What is the difference in behavior? They both look like they would delete the user's home directory. I assume the latter would try to delete a directory literally named with a tilde instead?
The latter passes each item in the list into the child processes’s argv, as-is, without the shell parsing them. That means this would delete a single item named “~/ some file”, spaces and all, instead of three items named “~/“, “some”, and “file”.
Edit: I’m typing this on my phone, so brevity won over explicitness. The latter probably wouldn’t expand ~. Imagine a file named “/home/me/ some file” for a better example.
cytoolz.pipe (https://toolz.readthedocs.io/en/latest/api.html#toolz.functo...), cytoolz's curried namespace (https://toolz.readthedocs.io/en/latest/curry.html#the-currie...) and method chaining can pretty much fully replace a pipe operator. The only problem is that you don't really want to write anonymous functions in Python with its needlessly verbose syntax, and long pipes are kind of limited without that.
nushell (https://www.nushell.sh/) is essentially perfect for this use case! i use it rather than posix(y) shells for most projects where id normally reach for python.
> By the way, don’t add a comma after the elements in the list, ...
Python:
You don’t even need parentheses here.
This expression is a tuple all by itself.Python for scripting honestly is Xonsh :)
Dupe: https://news.ycombinator.com/item?id=46176113
this discussion took off, however...
I wrote another comment here about a strategy for writing portable Bash scripts without compromising on features and freely using arbitrary external commands. I wanted to give an example from the article of how I'd likely write one of his examples of a "somewhat unreadable shell script" in this style.
His ugly sh example:
and his more readable Python equivalent: And I'd write the shell version in Bash something like this: Does it still involve more syntax? Yeah. Printing arrays in Bash always involves some. But it's clearer at a glance what it does, and it doesn't involve mutating variables.The piece this example is too simple to show (since it's focused only on data operations and not interacting with the filesystem or running external programs) is how much shorter the Bash usually ends up being than the Python equivalent.
I like using Go for “scripting”. To each their own I suppose.
Is this comment some kind of subliminal message?
Weren't we already?
Same, use Javascript for Scripting. https://github.com/gutenye/script.js
Why not use Perl?
Because the instant I want a dependency, I’m screwed. Half of what I want isn’t on CPAN, and the other half is ten years out of date and requires a compiler toolchain, fifty undocumented build dependencies at nonstandard paths, and five obscure/package-specific environment variables set in order to install. Python? Python either has it in the stdlib or has wheels available for pretty much everything.
Talking super simple stuff here, too: database drivers, markdown formatters, structured data parsers.
Because it is write-only.
Because of its culture, and what it does to your brain.
why not use perl?
[dead]