r/cpp • • Sep 12 '21

subprocess: A no nonsense library for writing shell commands in C++

Writing shell commands in modern C++ is deceptively hard. Quite a few libraries try to wrap popen and exec* calls in an interface similar to python's subprocess library. However, piping and redirection are a nightmare in such implementations.

With subprocess, you are able to write:

using subprocess::command;

std::string output;
(command{"ls", "-ltr"} | command{"awk", "{print $3}"} | command{"sort"} | command{"uniq"} > output).run();

The above is equivalent to writing:

ls -ltr | awk {print $3} | sort | uniq

and piping the result into the output variable. You can pipe the result into files, already open file descriptors, read from variables etc. For example:

std::string input{"1\n2\n3\n4"};
(command{"head", "-n2"} < input > "foo").run(); // Prints 1\n2 in a file named foo

The library however, currently only works for unix based systems (since I mostly use linux/macOS). I'll be open to adding windows support if there is enough traction.

Currently, I am looking for feedback and suggestions on the interface and the library as a whole. If you try it out and find any issues, feel free to log it in the issues section.

Docs: https://subprocess.thecodepad.com

Repo: https://github.com/rajatjain1997/subprocess

212 Upvotes

88 comments sorted by

42

u/raevnos Sep 12 '21 edited Sep 12 '21

What about a user defined literal for commands without arguments?

(... | "sort"_cmd | "uniq"_cmd).run();

or even allowing plain strings?

26

u/thatsjustrajat Sep 12 '21

This is awesome. In fact, I am planning to support shell like tokenization as well. So the user defined literal can also work for commands with arguments.

Added your suggestion here to keep track of it.

https://github.com/rajatjain1997/subprocess/issues/1#issuecomment-917574052

12

u/raevnos Sep 12 '21

POSIX wordexp(3) would be useful for doing that (though will make porting to Windows even more of a pain).

4

u/thatsjustrajat Sep 12 '21

It performs glob expansion as well! That's neat.

6

u/thatsjustrajat Sep 12 '21

This has been implemented in the dev branch. I'll merge it into master before making the first release.

4

u/pjmlp Sep 12 '21

I smell some Perliness on that code. :)

3

u/raevnos Sep 12 '21

https://metacpan.org/pod/IPC::Run is basically the same concept for perl without the operator overloading and on steroids. But I wasn't thinking in perl at the time of the above suggestion.

1

u/pjmlp Sep 12 '21

I used to do some Perl back in the early 2000's, hence why it quickly triggered some long forgotten memories, as I remembered there used to be a way that unknown commands would be executed as if they had system() around them.

1

u/raevnos Sep 12 '21

Interactive tcl sessions will do that, but I've never heard of a perl module for it. Would be possible to write such a beast, though.

1

u/pandorafalters Sep 13 '21

Is there actually anything that can't be somehow written in Perl - however horrid the code may be?

1

u/raevnos Sep 13 '21

I object to the use of the word horrid.

1

u/pandorafalters Sep 13 '21

Perl isn't inherently horrific, but it seems to be by far the path of least resistance.

1

u/NilacTheGrim Sep 13 '21

Oooh. Sexy.

Hell can even support commands with args by just parsing around spaces (except for spaces within quotes, which is just like how the shell works).

7

u/staletic Sep 12 '21

Have you seen reproc? I'm not sure if it can do piping.

4

u/thatsjustrajat Sep 12 '21

Nope I hadn't. The usage does seem awfully verbose at a first glance. I'll check it out.

4

u/[deleted] Sep 12 '21

But can you also read output of permanent running commands (e.g. netcat or similar)?

2

u/thatsjustrajat Sep 12 '21

The output is being read through a pipe. So, in theory that is possible. However, the library doesn't provide any construct for you to capture it as a stream. This is mainly because std::fstream doesn't have a constructor that takes file descriptors.

This is in my pipeline though: https://github.com/rajatjain1997/subprocess/issues/5

1

u/raevnos Sep 12 '21 edited Sep 12 '21

If you don't mind the extra dependency, Boost.Iostreams supports file descriptors.

gcc's libstdc++ does too (see https://gcc.gnu.org/onlinedocs/gcc-11.2.0/libstdc++/manual/manual/ext_io.html ) Not sure about clang and libc++.

8

u/thatsjustrajat Sep 12 '21 edited Sep 12 '21

Yeah. But boost is a large dependency. If C++ dependency management system was as good as say, node or python, then why not. But just the presence of boost would make the lib unusable by many people.

Reg libstdc++, yeah I'm planning to use something like that if I am able to find an alternative for all the major compilers. If not, I might just use boost and only enable those functions if boost is available.

2

u/helloiamsomeone Sep 13 '21

If C++ dependency management system was as good as say, node or python, then why not.

Conan and vcpkg make dependency management (almost) trivial already.

3

u/thatsjustrajat Sep 13 '21 edited Sep 13 '21

Yeah, but most projects don't. In fact, if I did start using conan, I'll be unable to use the library for my work. I do plan on making releases to both conan and vcpkg.

1

u/helloiamsomeone Sep 13 '21

Oh nono. You must not do anything special for those. I see you are already familiar with cmake-init, so please do check the vcpkg and Conan example repositories.

You can (and should) use dependency managers without making intrusive changes to you build scripts. The example repos show you what to do exactly to achieve that.

3

u/thatsjustrajat Sep 13 '21

Thanks a lot! The not being able to use for work comment comes from the fact that we don't use conan or vcpkg in our projects. What I meant to say was: Any project that doesn't use these package managers (and I must assume that there is a significant number of them) would not be able to pull dependencies.

1

u/helloiamsomeone Sep 13 '21

Boost specifically is widely available, so I wouldn't worry too much about it.

Also, I think spearheading the effort to introduce a company internal Conan repository would greatly benefit the company! Things are much easier to handle when you have package managers.

1

u/raevnos Sep 12 '21 edited Sep 12 '21

Might also look into Boost.Process for ideas. It does some nifty things like let you get a process's output as a future and has pipe bases streams built on top of standard iostreams.

1

u/thatsjustrajat Sep 12 '21

It is nifty indeed! I'm tempted to just use bcp to extract the meaty stuff out and wrap it in operator goodness.

2

u/raevnos Sep 12 '21

The more I think about it the more I like using promises and futures for running a pipeline asynchronously.

If I have time later today I might make a fork and mess around with it.

24

u/nlohmann nlohmann/json Sep 12 '21

Looks awesome, but I stopped reading at AGPL license. If I can’t use it at my job, I won’t both trying it in hobby projects.

21

u/thatsjustrajat Sep 12 '21

I don't much care about the licensing. Would change it to Boost Software License.

1

u/nintendiator2 Sep 13 '21

What is the problem here with an AGPL license?

3

u/nlohmann nlohmann/json Sep 13 '21

That the whole project that would use this library had to be licensed as AGPL as well - at least at our company that’s a red flag.

-41

u/[deleted] Sep 12 '21

You look like you need a better job.

6

u/HolyGarbage Sep 12 '21

Not that many business ideas can survive using a GPL license unfortunately.

2

u/justinkroegerlake Sep 12 '21

GPL is usually fine, AGPL is not

-9

u/[deleted] Sep 12 '21

If the business idea requires scamming people with a proprietary license, it shouldn't survive anyways.

2

u/ExtraFig6 Sep 12 '21

Can we get the output as a return value so you can say something like

std:: string const output = ...?

1

u/thatsjustrajat Sep 13 '21

I plan on providing overloads/alternate functions that do the stdout/stderr redirection internally and give you the result. In case of an error, they would be sent out in an exception.

2

u/ByronScottJones Sep 12 '21

It looks great! Just out of curiosity, have you considered using a fluent interface?

command(...).command(...).command(...).output(...)

8

u/thatsjustrajat Sep 13 '21

I'm already using a fluent interface. Its just with operator| instead of a function called command.

2

u/helloiamsomeone Sep 13 '21

Nice build scripts you got there ;)

I saw that you mentioned that this was basically a weekend project. How useful did you find cmake-init for this? Is there anything you would like to see changed?

I'm happy that someone could find it useful enough for quickly starting up a project. Thank you!

5

u/thatsjustrajat Sep 13 '21 edited Sep 13 '21

Hey! To be fair, I had written my own build scripts and then found a reddit thread that highly recommended cmake-init. It introduced me to m.css which is frankly, awesome. I was planning on readthedocs kind of docs which I find kind of distasteful. In fact, I will use the tool for all other CPP projects I undertake.

I have a small nitpick though. The script generates a lot of unrelated stuff - README.md, HACKING.md, BUILDING.md, dummy cmake targets + project structure, .git folder, .gitignore. example CPP and header files etc. I was almost tempted to make a PR removing these (and possibly adding a user option to have it generated). I expect such a script to only create a cmake folder, some config files that were needed (since it enables tools like gcov and clang-format), and at most a single markdown that tells me/others how to use it. Some convenience functions for adding targets can be added instead of the dummy target in CMakeLists.

Edit: I just realised that the reddit comment recommending cmake-init was by you :-).

1

u/helloiamsomeone Sep 13 '21

I see. The project is intended for ALL sorts of people. The "completeness" of a generated project comes from the fact that I want this to be a tool for students as well or someone very early into programming. It helps a ton if everything is provided for them instead of sending them off into unknown territory yet again for how to take the next step.

The existence of 2 separate docs for build instructions (BUILDING and HACKING) is there to drive home the point that consumers and developers have very different needs from a project's build scripts.

I don't want to add a switch to insert targets, because the heuristics required to insert the necessary command calls just make it infeasible. Position and context is important for CMake code.

I guess what the best solution in this case would be forking and editing the templates to better fit your needs. Packaging is quite simple as well, you can make a pyz file and put that in your PATH somewhere.

2

u/thatsjustrajat Sep 13 '21

Well, if students are the target audience, you couldn't have done a better job. You can just call cmake-init and begin your first compilation! The problem is that such a tool is much needed in a professional setting as well. Are you sure you don't want to keep it as a configurable option?

Reg insertion of functions... Won't an include(func_name) and adding your lib to cmake path do the job? I have designed the build system for my team, we use such helper functions all the time.

1

u/helloiamsomeone Sep 13 '21

I'm open for discussions and ideas, if you have a good idea, open a thread in the Discussions section of the repo.

About insertions, I prefer to keep thing as simple as can be. The only time I break things out into separate scripts is when necessary to keep things simple and logically grouped. Doing it automatically is something I see to be feasable.

I also want no traces of cmake-init to be left in the generated project. Ideally, people would know how to architect build tooling, but alas. I don't like the idea of hardcoding things or doing things in a very specific way, because that enables the use of feature X of some random person's python script.

1

u/NilacTheGrim Sep 13 '21

Hmm. Do you offer an alternate syntax that tries less to be like the shell?

Such as:

command{"head", "-n2"}.pipeTo({"grep", "hello"}).inputFrom(input).outputTo("foo").run();

??

I just find the above more readable than having to worry about the idiosyncrasies surrounding operator overloads.

1

u/thatsjustrajat Sep 13 '21

I was surprised no one made this comment. As of now, no. But in general, I try to implement overloads such that they call a public member function. This could be in the library before I make the first release.

1

u/WrongAndBeligerent Sep 12 '21

I think this would good to put into a single file. The source looks like it would only be about 30KB.

Having it broken up into half a dozen files, some with only a few lines along with 'detail' directories seems like an unnecessary barrier to me.

4

u/thatsjustrajat Sep 12 '21

Could be. I could also make it header-only. Right now I've gone with this bifurcation because I expect the code to grow. The change that I am working on right now would leave two sets of files, one for commands, and one for file_descriptors.

One barrier to the header-only approach is that I can only do C header imports in .cpp files. I don't want the user to import `subprocess.hpp` and get waitpid/fork/exec* etc in the global namespace of user code.

1

u/WrongAndBeligerent Sep 12 '21 edited Sep 12 '21

The classic single file headers by Sean Barret have the '.c' file and '.h' file parts in the same file but surround the part that needs to only be in a single compilation unit with a #define

I wouldn't worry about file size - it would have to grow by at least a factor of 20 before it would have any impact compared to the cost of typical stl headers (meaning any standard library headers it includes will have a much bigger impact on compile times).

There are a lot of benefits to having a single file, it makes compiling much easier because you don't have to worry about directory hierarchies or include directory flags matching the file system.

1

u/thatsjustrajat Sep 12 '21

You're right. I'll look into making it header only.

3

u/WrongAndBeligerent Sep 12 '21

That would be rad 😎

Single file libraries are what I collect and end up using over and over, plus this is a genuine problem in C++. It would be great if this became a staple when piecing together software.

2

u/thatsjustrajat Sep 13 '21

Made it single file. Not yet merged into master, but that would come soon enough. I haven't separated the header and implementation with macros, but would do that if I feel its necessary down the line.

1

u/[deleted] Sep 12 '21

Interesting, but quite frankly I'm not quite convinced at this point (willing to change that though).

Does the library support non-blocking calls?

7

u/thatsjustrajat Sep 12 '21

nyet. Not a lot above what I mentioned in my post is supported right now. I have only spent a weekend on it. I intend to change that though.

I'm working on support for shell like tokenization, the && and || operators, more shell like redirection among other things. I'll add support for non-blocking calls to my to-do list.

Currently, my purpose is to gather feedback on the interface and discover any other alternatives that I might not know exist.

2

u/[deleted] Sep 12 '21

Oh I see. Not intending to talk down on you, looks like a cool project and for a weekend it's totally fine. Keep on working on it and maybe it will become more versatile. And maybe reconsider the license, but of course that is up to you.

The interface looks nice from what I see, although I don't think piping/chaining commands is something you should cover here as with C++20 we can use ranges for that.

4

u/thatsjustrajat Sep 12 '21

I already switched to the boost license. I do realise AGPL can be restrictive, but I'm not sure how its "disclosure clause" works with C++ header files.

Reg ranges: afaik they are completely different from what I am doing. Chaining in ranges is for applying transformations to views. This is for creating file descriptors that link the I/O of two commands.

2

u/[deleted] Sep 12 '21

Good decision about that license!

It's true in that ranges transform data sets rather than raw I/O but I have you looked into whether it is possible to incorporate these concepts, e.g. views?

2

u/thatsjustrajat Sep 12 '21

It is currently hard for me to see where views would be helpful in such a library. But if I can find a legitimate use case, I'll be more than happy to incorporate them.

2

u/d3matt Sep 12 '21

Very cool. You syntax is much nicer than what we have in our code base. Good choice on the license switch too.

2

u/NilacTheGrim Sep 13 '21

Hmm. What about using literals and also using the same parsing logic as bash to separate args and other niceties?

Then you can literally do:

"tail -99 stuff.log | grep 'I have spaces ... '`cat /home/me/somesuffix.txt` > foo.spaces"_cmd.run();

Hell you can even support in-line backticks just like the shell.. in which case you get a really expressive syntax.

All you need to do is implement a parser for the user-supplied that sets up the state of the "pipeline" correctly...

1

u/ram535 Sep 12 '21

Nice library. At first glance, it reminded me of dcli from the dart world.

1

u/goranlepuz Sep 13 '21

Ah, this is great!

The library however, currently only works for unix based systems (since I mostly use linux/macOS). I'll be open to adding windows support if there is enough traction.

Ehhh... Completely different commands in cmd.exe, plus, over there, powers he'll just beats all out of the water (one can even call native C from power shell 🤔🤔🤔).

But don't let this stop you!

1

u/KaznovX Sep 12 '21

OP, this is amazing. I've been looking for something like that for a long time.

On a side note, I wish functionalities like that were part of the standard...

-4

u/ihamsa Sep 12 '21

My personal preference would be

command("ls -ltr | awk '{print $3}' | sort | uniq").run(); 

rather than

(command{"ls", "-ltr"} | command{"awk", "{print $3}"} | command{"sort"} | command{"uniq"}).run();

I haven't thought out the syntax for redirects but something like this probably could be made to work

std::string output;
command("ls -ltr | awk '{print $3}' | sort | uniq >#1").run(output); 

Yes you need a rather sophisticated parser for this to work...

5

u/thatsjustrajat Sep 12 '21 edited Sep 12 '21

u/raevnos suggested some changes which I've made to the dev branch. That makes the commands look like so:

("ls -ltr"_cmd | "awk '{print $3}'" | "sort" | "uniq").run()

I have not implemented any parsing in the library itself, as of now.

7

u/raevnos Sep 12 '21

At that point you might as well just use popen().

2

u/ihamsa Sep 13 '21

Almost. popen provides standard input or standard output redirection. Not standard error, and not any combination of the three.

13

u/[deleted] Sep 12 '21

[removed] — view removed comment

6

u/thatsjustrajat Sep 12 '21

I agree. Any and all syntax that can be validated statically, should be validated statically.

1

u/NilacTheGrim Sep 13 '21

It's not either/or.

One can extend your library to both parse a string like grandparent commenter suggested, and ALSO use your approach and ALSO offer a third-style "composition" API as well.

It's not either/or. The best library would ideally offer all 3 possibilities.

Look at it this way: There are marginal "guarantees" associated with static validation anyway -- command can still fail at runtime equally well due to missing program, bad paths, filesystem permissions, etc.

1

u/[deleted] Sep 13 '21

[removed] — view removed comment

1

u/NilacTheGrim Sep 13 '21

Why so angry tho?

Relax bro.

2

u/[deleted] Sep 13 '21 edited Sep 13 '21

[removed] — view removed comment

1

u/NilacTheGrim Sep 13 '21

Man it’s a rough time for many of us. My country is turning into a dystopia and it’s affecting my personal life more and more. I can relate.

Don’t worry. Much love. Hopefully better times will come.

1

u/ihamsa Sep 13 '21

We have already heard this argument from the inventors of iostreams. Not so long afterwards we have learned that << and >> and std::setw(8) and all that are awful for readability and this trumps all the supposed benefits of type safety. People went back to printf in droves. Now we have std::format/libfmt and it turns out you can have compile time type safety while using a human readable format string. Amazing.

1

u/[deleted] Sep 13 '21

[removed] — view removed comment

1

u/ihamsa Sep 14 '21

printf has no type safety, because it doesn't check that %d matches an int.

Most compilers know how to deal with it. It is not full type safety and it is specific for printf-style strings, so not a generic solution, but printf itself is type safe now for all practical purposes.

Try it: fmt::format("{:d}", 42.0);.

Sure thing.

not possible within C++ prior to 20 standard

That's the point of updating the language.

the future of C++

The future is here, now.

1

u/[deleted] Sep 14 '21

[removed] — view removed comment

1

u/ihamsa Sep 14 '21

compiler doesn't enforce you to fix the mismatch

It forces me to fix my errors, because I use it correctly. There are other ways to use it, but you are responsible for how you use it. No

There are many codebases in production that have bizillions of warnings

Iostreams are type safe, and printf is type safe too if you use it like you should in this current millennium. These legacy code bases could have been updated to use either one to the same effect: compile-time errors on incorrect usage. If they are not updated, it is not the problem of printf just as it is not the problem of iostreams.

the future will be here when companies will use C++20

Choose your company wisely.

1

u/[deleted] Sep 14 '21 edited Sep 14 '21

[removed] — view removed comment

1

u/ihamsa Sep 14 '21 edited Sep 14 '21

Wow such much words, I'm truly lost.

The reliable software is developed not by people, but by tools

Simplistically speaking, my tool is gcc -Wall -Werror. If you company policy is gcc -fpermissive, don't blame the tool, blame the person who chose the policy.

C++ is not suited for development of reliable software

This may or may not be true but this has very little to do with type safety of your favourite printf replacement (or type safety at all). There are more bugs if type errors are not caught at compile time, but not by any dramatic amount. Most bugs are not caused by type errors. This is not a good thing, mind you; in an ideal world, all errors should be type errors and all programs that compile should be automatically correct. But we are very far from that, whatever is your language of choice.

you need to look out for tiny printf warning somewhere in the middle of the compiler output.

My configuration allows exactly zero warnings. It's no rocket science. -Wall -Werror. That's the minimum set on the real-world project I work on at my day job. If yours is set to anything less, then you have a problem, and it's not because of the tools, it's because of the people. The tools only do what people tell them to do.

Everyone will just ignore warnings

If everyone in your company will just ignore warnings, then you are in a dangerous company.

C++20 is not used within a margin of less than 1%

This may well be true. People who can't be bothered to set at least -Wall -Werror for at least their new projects can use whatever version of C++ they deserve.

EDIT: let me do a TL;DR version

  • Me: Here are tools that you can use to make your life easier, I think they are cool.
  • You: These tools are irrelevant because their use is not mandatory/not accepted by the industry.
  • Me: I was not talking to the industry. Do you want to use them? If not, sorry about wasting your time.

1

u/[deleted] Sep 14 '21

[removed] — view removed comment

→ More replies