POSIX compliance for the most part. Which is meant to give you a consistent API you can write software against that will work on any UNIX system. At least that is the theory behind it. It basically acts as a lowest common denominator for UNIX systems. Want to write a script that will work on all systems - bash or a similar shell will almost certainly be present.

A new system can change this if they wanted to. But then they won’t be able to take advantage of all the existing bash scripts out there that everyone is writing. Which adds a lot of friction to adoption.

So in a way we are locked into a bash like shell now. Unless you can get a lot of distros to change this shell it is very hard to change it for a single one. And changing something so fundamental across lots of distros will upset a lot of people that are used to bash.

And shells overall are preferred to general purpose programming languages as their main focus is executing other commands and piping input/output from one to another or to/from files. Something most programming languages make a bit awkward as that is not their focus.

There are a bunch of non-POSIX shells, like fish, nushell, ion etc but these will likely not become the default anytime soon due to the inertia that bash has. But luckjly you can install what ever shell or language you want in addition to bash and switch to that for your interactive shell or even scripting if you want to. They just won’t become the defaults for a long time - if ever.

Bash isn’t the only shell. Most systems can use zsh or fish or tcsh or whatever shell you prefer, if you like that better than bash. You’re gonna have to run a shell if you want to use the terminal, though, you understand that?

Shell scripts are very good for specific tasks. Don’t use them for tasks that are unsuitable. Use python or go or node or c if those are better for your needs. Use the right tool for the job. But also, learn to understand why the industry has been using shell scripts for decades.

Everyone talks about the fact that Bash is what it is because it is first and foremost an interactive shell, but nowadays some design decisions are just inexcusable in my opinion, like the awful syntax of common programming constructs, the if in particular, that would only benefit from following how every other language works even if they aren’t meant as shells.
Some also argue against the non-modularity with the fact that you should use it for only quick and easy stuff, but that’s just an excuse, if the language runtime that comes preinstalled in your system had modern features and sane syntax you would stick to that and save yourself from installing Python/Ruby if they’re not needed; and it is clear that there is a need for modularity, otherwise plugin managers wouldn’t exist, many swear by downloading the scripts directly and sourcing them in the name of “KISS”, but that is just silly when there is a good system set in place that makes it actually easier to manage it all.
Then there’s the issue of the holy pipelining, that has more or less been overcome by some languages already, this example in Rust shows that it can be easy, so there’s no reason why a terse scripting language couldn’t achieve the same.

In the end I don’t know what’s holding the landscape back, I noticed Xonsh that looks very interesting, but I never tried it, I wonder if it is POSIX compliant and if that aspect even is so fundamental to the success of a shell

Let’s see here. We’re talking about a thirty-plus-year-old language that was deliberately written as a superset of sh from 1979. Its main purpose is to glue together other command-line programs on 'NIXish systems to automate complex procedures for sysadmins. Using it as a general-purpose programming language, while not impossible, is kinda dumb if you have any other options. Within its original niche, it ain’t broke, so there’s no need to fix it.

It’s a contemporary of Tcl and Perl, not of Python and Java (and sh is from the era where microcomputers mostly used BASIC dialects). Unlike Perl, it didn’t bolt on object orientation or other more modern features afterwards—and really, Perl was intended as just the kind of “shell script killer” you seem to be thinking of in your introduction. However, it never completely displaced the shell built-in scripting languages even in the days before it fell out of favour itself. Might be that there’s a reason for that.

Bash adheres to the POSIX standard for a shell, which limits what it can have in terms of UX. If you really want something with a different UX you can try something like fish. Just be warned that if you do, you’ll quickly learn how wrong your bullet point about bash not being portable is.

why are you stuck with bash? just write a shebang and then your interpreter in the first line of your script.

i can use bash, python, lua and al kinds of stuff…

i don’t understand the question. if you mean, why does my shell only accept bash syntax, if i set my shell to bash, idk. use another shell?

‘sh’ is kind of the smallest common thing that’s available everywhere. so when you got to script something that needs to run somewhere not under your control, you use ‘sh’. and that’s kind of it works. you’ll find something, that’s been around for some time, otherwise it won’t have spread everywhere. and now you can’t replace it in newer products, because there is so much stuff using it.

if that isn’t one of your problems, go with my first suggestions and just use python or something like that as your scripting language.

I wouldn’t use the word “stuck”, Bash is a result of over 50 years of technological advancement and experience. It works on servers, remotely, is lean and powerful. It’s not that user friendly, because it doesn’t need to be.

ZSH and Fish are available on *nix systems for a little bit more sugar. I personally don’t use raw Bash because I just love what ZSH can do, but I love writing Bash scripts.

Check out xonsh.

What features does bash have that make it so suitable for shells?

You mean, except being automatically available in basically every Linux distribution without having to install any additional software?

Bash is fine. Now powershell, that’s utter garbage.