7

I would like to know how to exchange data between Mathematica and a C/C++ with pipes. In the Mathematica tutorial it says that "when you open a file or a pipe, Mathematica creates a 'stream object' that specifies the open stream associated with the file or pipe".

I know how to create files in both C and Mathematica and I can make each program read and write to them. What I still don't know how to do is how to send output from C through a pipe to another program, much less how to even do this from Mathematica.

Here is a function in which Mathematica that writes a matrix to a binary file as well as reading a file written in that format.

writeDoubleMatrix[obj_, fileName_] := Module[{file}, file = OpenWrite[fileName, BinaryFormat -> True]; BinaryWrite[file, Length@obj, "Integer32"]; BinaryWrite[file, Length@obj[[1]], "Integer32"]; BinaryWrite[file, Flatten[obj], "Real64"]; Close[file] ] readDoubleMatrix[fileName_] := Module[{file, obj, m, n}, file = OpenRead[fileName, BinaryFormat -> True]; m = BinaryRead[file, "Integer32"]; n = BinaryRead[file, "Integer32"]; obj = BinaryReadList[file, "Real64", m*n]; Close[file]; Partition[obj, n] ]

The first function will write 2 integers to a file (the size of the matrix) and the data of the matrix. I'm not doing any error checking here and thus I'm assuming that the data to be written is specifically in the form {{r11, r12, ..., r1n}, ...., {rm1, rm2, ..., rmn}}. The second function will be able to read the binary file and return the matrix.

Next comes my C program. This program will read the data stored in the file MathematicaData.bin, multiply this matrix by 2 and write data to another file.

// genData.c #include <stdlib.h> #include <stdio.h> int main(int argc, char** argv){ int m, n, i; double* matrix; FILE* fin; FILE* fout; // Reading input file fin = fopen(argv[1], "rb"); fread(&m, sizeof(int), 1, fin); fread(&n, sizeof(int), 1, fin); matrix = (double*)malloc(m*n*sizeof(double)); fread(matrix, sizeof(double), m*n, fin); fclose(fin); //Modifying data for (i = 0; i < m*n; ++i) matrix[i] = 2*matrix[i]; // Writing output file fout = fopen(argv[2], "wb"); fwrite(&m, sizeof(int), 1, fout); fwrite(&n, sizeof(int), 1, fout); fwrite(matrix, sizeof(double), m*n, fout); fclose(fout); // De-allocate memory used for matrix. free(matrix); return 0; }

This program does not have any error checking. You need to be careful how you use it otherwise the program may not detect the files or even allocate the amount of memory that you want. In any case, we can compile the program with your compiler of your choice.

gcc -o genData genData.c

And now we can try to use these functions to communicate between the two languages from Mathematica.

matrix = {{1, 2, 3, 4}, {5, 6, 7, 8}, {9, 10, 11, 12}}; writeDoubleMatrix[matrix, "MathematicaData.bin"]; Run["./genData MathematicaData.bin CData.bin"]; readDoubleMatrix["CData.bin"]

If everything went well the output you should get is

{{2., 4., 6., 8.}, {10., 12., 14., 16.}, {18., 20., 22., 24.}}

Yes, this is a very time consuming way of multiplying a matrix by 2 but this is just a simple example to show how to exchange data from Mathematica to C and from C to Mathematica. What I do not like is the fact that everything is stored to a file first and then it is read in the other program. Can someone show me how to exchange data without writing files. I have a feeling that pipes is what I need but I have no idea how to read or write them from neither language. It would be helpful if you could modify this program to adapt it to pipes.

UPDATE:

I found out how to make the C program "pipeable".

//genDataPipe.c #include <stdlib.h> #include <stdio.h> int main(int argc, char** argv){ int m, n, i; double* matrix; // Reading input file fread(&m, sizeof(int), 1, stdin); fread(&n, sizeof(int), 1, stdin); matrix = (double*)malloc(m*n*sizeof(double)); fread(matrix, sizeof(double), m*n, stdin); //Modifying data for (i = 0; i < m*n; ++i) matrix[i] = 2*matrix[i]; // Writing output file fwrite(&m, sizeof(int), 1, stdout); fwrite(&n, sizeof(int), 1, stdout); fwrite(matrix, sizeof(double), m*n, stdout); // Deallocate memory used for matrix. free(matrix); return 0; }

This means that we have to use the program like this:

./genDataPipe < fileIn.bin > fileOut.bin

I have been searching in the documentation on the Mathematica side but all I have figured out that I can open a file by piping an external command. Take OpenWrite for instance:

On computer systems that support pipes, OpenWrite["!command"] runs the external program specified by command, and opens a pipe to send input to it.

This means that I can give my binary input directly to the c program. The problem is that I can't find a way of redirecting the output of the program. What I have come up with is, write a data to a file and make a wrapper to run the external command and read the contents of the output of the external command. Here we assume the existence of writeDoubleMatrix from before.

getDataPipe[fileName_] := Module[{file, obj, m, n}, file = OpenRead["!./genDataPipe < " <> fileName, BinaryFormat -> True]; m = BinaryRead[file, "Integer32"]; n = BinaryRead[file, "Integer32"]; obj = BinaryReadList[file, "Real64", m*n]; Close[file]; Partition[obj, n] ] matrix = {{1, 2, 3}, {4, 5, 6}}; writeDoubleMatrix[matrix, "MData.bin"]; output = getDataPipe["MData.bin"]

Which results in output having the following contents {{2., 4., 6.}, {8., 10., 12.}}.

The only goal that remains now is to find out how to eliminate the need of writeDoubleMatrix and pass the data directly without having to write a file.

Improve this question
5
  • See linux.die.net/man/1/mkfifo. Commented Jun 24, 2011 at 1:06
  • Is there a reason you're not using MathLink? Commented Jun 24, 2011 at 1:22
  • @Joshua, I could probably find out how to use it, but I really want the two programs to be independent of each other. I don't want to mess so much with the data types in Mathematica. All I want is something self contained. If it is not too many lines of code, could you adapt MathLink to the example I wrote? Commented Jun 24, 2011 at 2:17
  • It's odd, I can't find a function where can specify both the standard input and output of the program you'll be running. I think your options here are using at least one file (in the general sense; a named pipe would be best) or using a MathLink wrapper that would serve as a bridge between Mathematica and the C program. Commented Jun 24, 2011 at 8:37
  • @Artefacto, I also find it very strange. I had forgot that I once used pipes in python. It made me wonder why Mathematica didn't implement something like that or if they did, why it is not documented. I mean, what is the purpose of sending input through a pipe to an external program which you cannot get the output from? This forces you to make the program write to a file doesn't it? Commented Jun 24, 2011 at 8:58

4 Answers 4

Reset to default
5

The first step would be to create a named pipe, (as mentioned by Artefacto's comment above). This is called a fifo, First In First Out. A pipe makes the output from one file into the input of the other. I'd like to note that these methods are Linux only before I start.

Basically, pipes work like this:

mkfifo mypipe or in C: system ("mkfifo mypipe");

Next step is to write the output to the pipe, because everything in Linux is a file, you can just use the standard i/o operations for that, or even redirect standard input to the pipe. You already have the code for that in both Mathematica and C. So after writing the output of the file to your pipe, the Mathematica version can then read input from the pipe, multiply it, and display it in stdout or wherever you please. You really shouldn't have a problem with this method, seeing as the pipe is emptied after the reading and can be removed easily after you're done. If you want to remove the pipe afterwards, just run system ("rm myfifo");.

If you really don't want any accessory file, even though it's not that bad, try making another file that actually outputs in standard output. Then make one that let's Matematica read from the standard input. Now, to pipe through:

./cprogram | ./mprogram

This means that the output of your C program should be the input of your Mathematica program. To my knowledge, this still creates a pipe, but it will automatically be deleted by the system when it's done, and the average end user probably won't see it.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

Is it possible to do something like this under Windows?
@Johnathon, Seems like I got the piping fixed in my C program. Now I need to learn how to redirect input and output in Mathematica
@Alexey, does this answer your question? I don't have a windows machine so I can't try it out. BTW, now I just need to find out how to redirect the input and output in mathematica.
@jmlopez If I understand correctly, @Johnathon tells us that it is possible under Linux to create and use from Mathematica a permanent pipe with reusable both reading from and writing to ability. As I know, command redirection operators under Windows allow to redirect input/output only one time and then the program must be finished. My question was: is it possible to create and use such permanent pipe under Windows?
@Alexey As for Windows, there's no standard way to create a named pipe. In fact, Windows "named pipes" cannot be created by any utility and are used as sockets. Perhaps you should create a new file under %TEMP% and use ‘tmpname‘ and ‘tmpfile‘ to make a temporary file and just redirect I/O there. To sum it up: there is no way to make a named pipe in Windows.
|
4

The closest function that Mathematica has to reading and writing simultaneously from an external program is RunThrough. According to the documentation, though, it does not operate like you want: passing info through Write and Read operations. Instead, it writes the second parameter to a temporary file, executes the command specified in the first parameter passing the temporary file to it, and captures the output of the command. This is clearly not what you're asking for, but it seems to be the closest of the built-in commands. Personally, I'd look into using MathLink, but since you don't seem to want to do that, Johnathon's answer is on the right track.

To expand on his answer a bit, the documentation implies that Mathematica cannot read and write simultaneously to the same file/pipe. This suggests to me, a round about solution. First, you need 2 pipes: one for writing and one for reading, and these can be made via Run, e.g.

Run["mkfifo in"] Run["mkfifo out"].

Second, open the pipes for i/o

instrm = OpenRead["in"] outstrn = OpenWrite["out"].

Last, run your external program by either passing the names of the pipes to it, if it is set up to handle command line params, 

Run["prog in out"],

or via redirection

Run["prog <in >out"].
Improve this answer

4 Comments

this is very similar what I had written in the example. I guess we can just conclude that it is not possible to redirect input and output of subprocesses in Mathematica. It would be good if they looked at how python deals with pipes.
No, the docs explicitly say it is not capable of redirecting both.
@rcollyer here on mma 9.0.0 for MacOSX I cannot do OpenRead["in"] and OpenWrite["out"] hangs my kernel permanently. See also this question, maybe you can answer that one. I think it must be possible to set a Method for OpenRead and still succeed, but so far no luck for me :).
@JacobAkkerboom I have not tried this with v9, any incarnation of it. I'll play with and see.
3

MathLink may be the best choice for you. Mathematica and the C program will be separate processes and communicate using pipes or shared memory. You'll have to look at the MathLink examples in your Mathematica install. There's a tool to generate glue code from a template file called mprep, and you'll need to link in the MathLink library, but it's dead easy after that to call a C function with an array of data.

MLPutReal64Array - send array of data to Mathematica

MLGetReal64Array - read array from Mathematica

I'd take one of the examples and start working from there. You'll probably want a Manual ReturnType and Real64List for the ArgumentType.

Improve this answer

4 Comments

Joshua - I checked out some of the examples but this however seems very restrictive. In the sense that they only support certain types and if they do support other things it is not so clear how to do it. I rather write my output in my own format that I will know how to read in C and then find a way to store it back in Mathematica. In the end I think I'll end up doing what MathLink does with less efficiency because I'm not sharing memory.
@jmlopez I'm not sure what you mean by restrictive. In fact, mathlink allows your C program to exchange arbitrary Mathematica expressions (i.e. anything) with the Mathematica kernel. It really is the correct tool for communication between Mathematica and C.
@Mark, What I'm worried about is lists of lists of lists. And these lists may not even have the same type of objects. Anyway, I guess I'll try to write my above example using MathLink and post the result here once I find out.
MathLink can send general expressions to C. It's a little more work because you have to ask what comes next and read the right thing, but it's perfectly general.
3

It's a bit awkward, but you can use JLink:

Needs["JLink`"]; InstallJava[]; LoadJavaClass["java.lang.Runtime"]; runtime = java`lang`Runtime`getRuntime[]; process = runtime@exec["my program"]; input = process@getInputStream[]; output = process@getOutputStream[];

Now you can write to a pipe that's stdin on the other end:

output@write[ToCharacterCode["hi!"]];

And read from the launched program's stdout:

nextByte = input@read[];
Improve this answer

1 Comment

I'm sorry but it is not so clear how I could pipe the binary input from Mathematica to the process and then pipe the output back to Mathematica. I guess I'll check it out later.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.