5

I want to put together an IPython notebook with some shell commands and their input. In the bash prompt I can use "here-document" syntax:

bash-3.2$ mysql -u root <<END_IPUT > use mydb; > show tables; > END_INPUT

How do I get the same effect in IPython, and specifically in a jupyter notebook? I know how to execute shell commands as IPython as "line magics" or "cell magics", e.g.:

In [7]: !! ls -tF Out[7]: ['Demo-notebook.ipynb', 'createdb.sql', ...

I've looked at IPython as a system shell, which shows how to enable some syntactic niceties. After the following, I can run system commands without prepending ! or !! 

# Turn everything in $PATH into an alias; # then enable calling aliases without ! or % %rehashx %autocall 2

But none of this helps with providing input to these commands inline: The here-document syntax is invalid in IPython, and results in a python SyntaxError. So how do I do it?

Improve this question
6
  • Have you tried the bash cell magic? Start a cell with the line %%bash, and put bash code in the rest of it. Commented May 17, 2016 at 17:07
  • 1
    Good suggestion, thanks! I'd already found %%sx, which does the same I think. But the documentation for %bash led me to %%script, which is exactly what I was looking for. In the meantime I also found an extension specifically for sql, which solved my immediate problem even better. But if you want to write an answer expanding on your suggestion, I'll accept it. Commented May 17, 2016 at 21:28
  • @Thomas, before you delete my tag for a third time, look at how many people follow each one. If ipython-notebook is such a problem, please take up the process to have the tags merged. Commented May 22, 2016 at 23:17
  • I am trying to merge the tags, but I can't do that until the new tag has 80% as many uses as the old. I am periodically sweeping new questions over to the new tag to boost its numbers. Thanks. Commented May 23, 2016 at 10:21
  • I see. Well, please stop deleting my tags. Having both should be good enough for you. (And just add jupyter-notebook to other questions.) And making the tags themselves synonyms, of course. Commented May 23, 2016 at 10:25

2 Answers 2

Reset to default
8

The bash cell magic allows you to enter multiple lines of bash. Just start any cell with %%bash:

%%bash echo "This is bash" ls

If you're using a heredoc to pipe text into another program, you can also use the script cell magic:

%%script bc 2+3

There may also be other cell magics which address your problem more specifically!

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

there is few small differences between ! and %%bash when running a command. 1. %%bash doesn't display ASCII colors, while color appear fine using!. 2. %%bash collect the output and flush them on the screen once while ! displays the output one by one as it executes.
6

With some more research, and with the help of @ThomasK's tip (see accepted answer), I found a couple of ways to do this.

  • One solution is the cell magic %sx. It executes the cell contents as a bash script, and captures and returns the output as a list of lines. Sometimes handy, sometimes not. 

    In[1]: %%sx echo Hello, world cat -n <<DATA this and that DATA Out[1]: ['Hello, world', ' 1\tthis', ' 2\tand that']

  • Change that to %%bash, as @ThomasK suggested, and the output is printed out, not returned. To capture it for further processing, ipython provides the --out parameter. But then I need to print it myself-- it won't show up by default.

    In[1]: %%bash --out var echo Hello, world echo " Again" In[2]: var Out[2]: 'Hello, world\n Again\n'

  • %%bash is actually shorthand for %%script bash, billed as working "like the #! line in a script" (see the help with %%script?). Which it does. So here's how to dispense with the here-document syntax and have any program read the cell contents as input. %%script also accepts the --out parameter.

     %%script --out var mysql -u root -p XYZ USE somedb; SELECT * FROM users WHERE passwd IS NULL\G

This is nice enough, but I ended up not using it because my use case is mysql, and eventually I discovered that there's a great third-party ipython-sql extension that I can use instead:

%load_ext sql %sql mysql+pymysql://user:passwd@localhost/somedb

This presupposes installing the extension, with pip install ipython-sql. For mysql, I also needed pip install pymysql. After the above setup, I just talk to the database like this

In[1]: %%sql SELECT * FROM users WHERE passwd IS NULL Out[1]: 1 rows affected. ...

The returned table is actually drawn as a notebook table with borders, which is kinda nice. And it is made available as a Pandas dataframe (a smart list), which I can grab from _.

Improve this answer

1 Comment

Could I do: %%bash /usr/bin/env ipython in order to run an exported notebook.py at the command line?

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.