With about half a century of life, the Unix shell is pervasive and entrenched in our computing infrastructure—with recent virtualization and containerization trends only propelling its use. A fresh surge of academic research highlights show potential for tackling long-standing open problems that are central to the shell and enable further progress. A recent panel discussion at HotOS ’21 concluded that improvements and research on the shell can be impactful and identified several such research directions. Maybe it’s time for your research to be applied to the shell too?
The shell is one of the most critical pieces of our computing infrastructure. It is one of the first interfaces one encounters when faced with the task of configuring and setting up a system. As shell scripts can directly call and manipulate any program available in a system, they are used pervasively for orchestration and deployment of modern container workloads, for succinct and concise data processing, and for other automation and configuration tasks. And with the Windows Subsystem for Linux, the Unix shell can be found across other platforms and environments. The shell is everywhere.
Despite its ubiquity, the shell has been largely ignored by the academic research community—until recently, that is. A recent buzz of academic activity across a variety of related disciplines is tackling 40-year-old challenges, such as the formalization of POSIX semantics, provably correct shell script parallelization, and resource-aware distribution of shell pipelines. The increased community interest also led to the organization of a panel on the future of the shell that took place at HotOS XVIII and was attended by more than 100 researchers from various communities.
Three challenges have stymied research on the shell, but recent innovations enable new approaches. For a longer presentation of shell research challenges, enablers, and potential future work you can read our HotOS paper and a detailed report on the HotOS panel. Overall, the shell and its ecosystem are understudied; considering how pervasive the shell is, improving things is particularly impactful. Could your research apply to the shell?
Challenges obstruct shell research
Despite its ubiquity, research and improvements on the shell have been rather slow. In our view, there are three fundamental characteristics of the shell that have contributed to this, adding to the inertia and slowing down research progress.
First, the shell is abstruse—the semantics of the language itself are a serious challenge. Consider the following program:
true
a=$? b=$(exit 1) b=$? >$(echo /dev/null; exit 2)
echo $? $a $b
Taken from recent discussion on the Austin Group mailing list, different shells behave differently here! The POSIX spec leaves a great deal unspecified, often due to historical reasons. Different shells interpret even simple programs quite differently. Improvements to the shell—whether to the specification or to tooling—must grapple with abstruse semantics and tight backwards compatibility requirements.
Second, the shell is polyglot. The shell language composes arbitrary commands, each written in any number of languages. Consider the following (simplified) excerpt from a real grading script, applying the grade for a group assignment to each submitter:
for s in $(cat tograde.txt | sed 's/ (LATE)//g'); do
grade="$s/grade.txt"
subs=$(echo $s |
sed 's/.*Z-\(.*\)/\1/' |
sed 's/-/ /g' |
tr '[:upper:]' '[:lower:]')
pct=$(echo "scale=1; 100 * ${score}" | bc)
for s in $subs; do
echo $s,$pct
done
done
This ‘simple’ program uses four ‘languages’: the shell itself, sed, tr, and bc. Tooling that only works for the shell language would fall flat here.
Finally, the shell is dynamic. The shell’s execution depends on dynamic elements unknowable ahead of time—particularly the state of the file system and the environment. Consider an emblematic installation routine:
wget https://foo.bar/baz.tgz
tar xzf baz.tgz
cd baz
./configure
make && make test
sudo make install
eval $(baz env)
Ahead-of-time analysis is hopeless here. How can we know what’s in the tarball? Unless the configure script is hand-written, it’s absurdly long and full of redundant checks (are you still running HP-UX?). What do the build and test do? What will be installed, and where? When we update our environment to know about baz, what will our PATH be? Apparently, all we can do is try it and hope for the best.
Recent work opens new ways forward
Recent academic activity (by us, among others) has started addressing these three challenges, hinting at a rediscovered potential for exciting research on and around the shell.
Smoosh is a recent effort to tame and formalize the abstruse semantics of the POSIX shell. Smoosh enables precise reasoning and higher assurance when developing solutions and analyses for the shell. Smoosh begat libdash, a linkable POSIX shell parser and unparser that others can build on.
PaSh, an academic project that is now under the Linux Foundation umbrella, and POSH proposed annotation languages as a means for providing partial high-level command specifications for the shell’s polyglot ecosystem. These specifications act as an intermediate abstraction that can facilitate the development of analyses and transformations irrespective of the underlying command source language.
We recently identified just-in-time (JIT) interposition as an effective approach for managing the shell’s dynamism. By interposing on the shell at runtime, we can apply analyses and transformations over shell scripts knowing critical, up-to-the-minute information about the script’s components and the broader environment (environment variables, the state of the file system). Interspersing analysis and evaluation opens up a wealth of research opportunities such as speculative execution and incremental computation.
In addition to enabling future work, these recent papers show how it is possible to focus on a well-defined but constrained shell fragment. It’s tractable to develop effective solutions for these specific fragments instead of general but ineffective ones. Smoosh’s semantics form a foundation for delineating those fragments and precisely defining them. PaSh and POSH show how massive performance gains through parallelization and distribution can be achieved for a dataflow subset of the shell. JIT interposition allows lifting specialized solutions for shell fragments to the complete shell by tightly integrating with the shell and switching back and forth between standard execution and optimized execution when in a particular fragment.
Future directions
Believing that the time was ripe for research on the shell and its ecosystem, we organized a panel discussion titled “The Future of the Shell: Unix and Beyond” at HotOS XVIII—a venue for provocative, forward-looking research. By bringing together over 100 researchers and engineers from disparate communities—e.g., systems, languages, HCI, security—the event explored and highlighted impactful research directions around the shell and its ecosystem.
The most striking theme of our panel was that the shell is such pervasive, critical infrastructure that research on it can be incredibly impactful; a corollary was that the shell has been neglected for so long that there is much to do. The discussions covered several interesting topics and future research directions in four loose themes: usability and human factors, expressiveness, performance, and legacy and compatibility. Within these themes, the discussions were a rich vein of research ideas concerning user interface issues, the terminal interface and its affordances, program synthesis, ecosystem contributions, and primitives for supporting cloud computing and distribution in the shell.
A primary concern with usability and pedagogy was the shell’s sociotechnical system. How can we simultaneously give users the shell’s concise power while also making it easy for novices to learn the shell and its world? How can we support shell users in their varieties of domains? The shell is incredibly successful as an interactive tool; how can other tools learn from that success?
The shell’s expressiveness was well appreciated in some domains, but derided in others. Researchers have made several attempts to design a shell for managing distributed computation, but work in practice remains challenging. What would a shell-ish system of separate, composable utilities look like? Another common woe was the shell’s dynamic, global state. How can we help write deterministic or idempotent or debuggable shell scripts? What can we do to make scoped set-up and tear-down easier? As we expand the shell’s capabilities, what belongs in the shell, and what belongs in the ecosystem (tools, filesystems, operating system interfaces, etc.)?
The shell’s performance was an important theme, though a more challenging ground for research ideas: the shell composes other commands, so it’s hard to find “one weird trick”. Existing work in this space, like PaSh (for data parallelism) and POSH (for distribution), uses annotations to reason about how to recompose individual commands. What would a more general notion of annotation look like? What about other areas of performance improvement, like incrementality?
Finally, legacy and compatibility was a common theme throughout all discussions. Tools for the shell and solutions to its problems can have impact only if they actually work on the vast amount of shell scripts that exist out there.
After the completion of the panel, we (us three and Stephen Kell) wrote a report on the “The Future of the Shell” Panel at HotOS 2021 that summarizes the challenges and possible research directions around the shell and its ecosystem, collected during and after the HotOS21 Panel on the future of the shell. Its goal is to create a snapshot of what all these researchers from various disciplines—connected to the shell to varying degrees—think about its future, and to serve as a reference for future research on the shell and its ecosystem.
The last few years have seen a new wave of interest in ops in general and in the shell in particular. Since the shell is ubiquitous, improvements to it (and its ecosystem) are particularly impactful. The shell is used for many things, and it touches many areas of computer science: programming languages, systems, HCI. With better foundations for working with the shell, we can apply decades of improvements, addressing issues with the shell that have resisted attempts at improvement for forty years. Work on the shell is important and impactful; could your work touch the shell in some way?
Bios: Michael Greenberg is an assistant professor at Stevens Institute of Technology; his research focuses on directly applying formalism to practical problems. Konstantinos Kallas is a PhD student at the University of Pennsylvania working in the intersection of programming languages, distributed systems, and software verification. Nikos Vasilakis is a research scientist at MIT CSAIL interested in programming languages, parallel and distributed systems, and computer security.
Editors: Todd Millstein and Tianyin Xu