Shades of command-line-ness and JSON streams

Not all command-line programs are created equal

I think there might be some value in distinguishing different kinds of command-line programs. I’ve had the experience in the past when I say: “there’s no command line app for that”; to which others reply:”what about X”: to which I reply: “that’s not a real command-line program.”

Some picking-apart of the concept of command-line may be in order.

Now, useful definitions tend to exist for a practical activity rather than just being an intellectual exercise. So before jumping into defining things, we should think about what people actually do with command line programs.

Command-line programs can be useful because:

  • They can let you use them with only a keyboard: this can be efficient.
  • They can work without a graphic display, which is particularly useful when talking to remote machines.
  • They can provide a consistent interface that follows standards, which allows you to do things with the app that you might not be able to otherwise. Think about things like tmux, screen, and command-line redirection
  • They can be controlled simply with arguments from the command line and input data.
  • They produce text output.
  • They produce output that other tools can easily understand (e.g. grep, xargs)

Now, not all “command-line applications” do all of these things. And many programs that aren’t at all command-line programs have some of these properties. Web pages are an example: HTML can in many ways be better than text for repurposeing after the fact (e.g. with greasemonkey). Also you have emacs or vim gui interfaces on linux. In many ways these are command line programs: controlled exclusively by the keyboard and with good interoperability.

At the other extreme, there are things like mutt and other curses applications: non-extensible, impossible to script, and producing output that you can’t even capture.

To my mind, the following hierarchy of “command-liney-ness” is meaningful:

  • Applications that run in a terminal (sometimes called TUIs).
  • Applications that you execute from the command line (without pressing any keys).
  • Applications that can produce pure text output that is possible to parse
  • Applications that produce friendly machine readable output that is guaranteed not to change.

Programs capable of producing JSON output (or YAML, or XML) represent the extreme case.

  • mutt would be the first class
  • w3m might be in the second class (you can open a specific url with w3m, but the display is through a curses app).
  • nmcli might be in th third class. Have a go at parsing the output for `nmcli device wifi list`.
  • journalctl is an example of application that has embraced machine readability.

Now, we shouldn’t necessarily view one class of program as better than another: curses apps do lots of useful things that purer command line apps can’t. Many applications straddle classes, git has many interactive options. However, one should acknowledge that things like mutt, or ncdu, or midnight commander often have more in common with GUI applications than they do command line tools.

The birth of JSON-stream interface

Amongst the class of tools that produce machine readable output, I think that we are starting to see a range of command line applications that produce JSON output. We are also seeing a collection of tools designed to process this JSON output and do useful things with it from the command line, analogous to the suite of command line tools for text (grep, sed, awk, head, tail, and friends).

Tools in this class include:

Relatively speaking we are still at the early stages of the emergence of de facto standards. It’s not entirely clear what the correct combination of do-what-I-mean specificness versus do-everything-I-need generality is a good fit for these tools.

It is also difficult to claim that producing JSON output doesn’t add complexity and verbosity to output. The growth of intelligent pagers and display tools might make a difference, | jq is often good enough to render JSON readable. Perhaps we’ll even grow a JSON aware shell one day.

Rosy as the future of JSON may look, is is no coincidence that the computing landscape is littered with unadopted standards that provide more functionality than the standards that won out. The bash command-line is just such a lowest common interface. Even minimal levels of complexity / problems with interoperability seem capable of killing adoption of a protocol. Ad-hoc boilerplate seems preferable to being made to understand other people’s florid intellectually-self-satisfying ontologies, and so it may be the case with this technology.