2

I've written a small UDP server in PHP which listens on a socket, accepts a command, and performs an action. It's really quite basic. I can connect to it manually like this:

% nc -u host port

(where nc = Ncat: Version 7.50 ( https://nmap.org/ncat ))

As I enter commands, I see the resulting response. It works exactly the way I want it to work from the command line.

However, if I simply "cat" a file in like this:

cat FILE| nc -u host port

or send it data like this:

echo "command1\ncommand2\n" | nc -u host port

... then my PHP app reads everything including the end of line characters all at once. I only want to read up to the end of the line.

Sure, I could wrap around the contents of the file, and send each line to nc:

for x in `cat <file>`; do echo $x | nc -u host port done

... but that's a complete waste. I want 1 connection to nc, not many.

The EOL characters make it into the string because when I print the output in the PHP app, I see: command1 command2 ... but it's all in one string.

Why is the interactive mode behaving differently than the non-interactive mode?

I've been experimenting all afternoon, and I can't seem to make this work.

I'm sure there's an explanation though.

Thanks for any information you can provide.

PS: The basics of the PHP code is:

$sock = socket_create(AF_INET, SOCK_DGRAM, SOL_UDP); socket_bind($sock, '0.0.0.0', 2000); for (;;) { socket_recvfrom($sock, $cmd, 1024, 0, $ip, $port); echo "command is $cmd"; $reply = process($cmd); echo "reply is $reply"; socket_sendto($sock, $reply, strlen($reply), 0, $ip, $port); }

While entering text with nc interactively, you can hit CTRL-D to separate packets. If I do the same thing in a shell script:

printf 'command1\0014commandd2\0014'| nc -u host port

... the commands all appear as one command.

If I reduce the packet size in the PHP code to say, 5, then send data that is 20 bytes long, the data is truncated, and not separated.

I've also ready online this could be a buffering issue. I tried using: 

stdbuf -oL -eL cat FILE | nc -u host port

... but that surprisingly didn't make a difference either.

Finally, I have discovered that if I do:

for x in command1 command2; do echo $x sleep 1 done | nc -u host port

... that everything goes as planned. The server receives the first command, then the second command.

What isn't clear really is why the sleep 1 makes a difference. Take it out, and it fails. The above is certainly better than sending each echo to nc.

Improve this question
2
  • You're slurping in up to 1024 bytes at a time which will probably be several lines of input. You might have to re-engineer your parser to work one character at a time. Commented Oct 4, 2018 at 20:32
  • I believe that's the maximum packet size, but if I can send small short commands via the command line directly to nc, so I should be able to script them as well. All the UDP examples online, be it for PHP, C, etc. all do the same thing - reading a maximum packet size. If there's a better way, I'm hoping someone can suggest what it is. Commented Oct 4, 2018 at 23:26

2 Answers 2

Reset to default
1

l0b0 solved your problem, but to answer some of your other points:

Sure, I could wrap around the contents of the file, and send each line to nc: for x in $(cat <file>); do echo $x | nc -u host port; done but that's a complete waste. I want 1 connection to nc, not many.

1) that doesn't actually send lines from the file, it sends anything separated by whitespace; the whitespace can be newline but also tab or space (given you haven't altered IFS). Only if your idea of a 'command' is restricted to a single word are these the same, but in general most Unix(y) commands are multiple words. Those words are also processed for shell pathname expansion (often called 'globbing') so if they contain any of ?*[..] the value(s) used for $x may be different than what was in the file.

2) No connections are 'wasted' because UDP is connectionless and no connections even exist. Maybe you mean you don't want executions of the nc program (locally).

Why is the interactive mode behaving differently than the non-interactive mode?

Input on a terminal in Unix is normally handled a line at a time; when the program (here nc) does a read it waits while you type as many characters as you like -- and optionally edit them -- until you hit return (or equivalent), then the line of input is delivered to the program (which for nc sends it as a packet). (Unless the program's buffer is too small; then only the part that fits is delivered, and any residue remains pending for the next read.) Optionally this can be turned off and as little as each character (or group of characters produced by one keystroke such as a function key) can be delivered separately, which is used by programs like vi -- and, although it's less obvious, bash. Officially these are called 'canonical' and 'non-canonical' modes, but historically the non-canonical mode was described as 'raw' and the canonical mode as 'cooked'.

Reading from any other type of file -- including a pipe -- ignores line-breaks (i.e. newline characters) and reads as much as fits in the buffer, which in your case includes multiple lines.

While entering text with nc interactively, you can hit CTRL-D to separate packets. If I do the same thing in a shell script: printf 'command1\0014commandd2\0014'| nc -u host port ... the commands all appear as one command.

3) control-D on a terminal is special -- or more exactly the character configured as the eof setting in the terminal driver, which is usually control-D but can be changed, is special. When you type this character while a program is reading, the read is terminated without waiting for a return and also without including the control-D in the data. The character code that corresponds to control-D is not special when it occurs in a file.

4) and the character you've generated isn't control-D anyway. Control-D is \004 in the octal notation used for many versions of echo, and \x04 in some other contexts. \001 is control-A, also called STX, and practically unused in Unix (except in the program screen which uses it to control switching among multiple virtual-terminal windows).

I've also ready online this could be a buffering issue. I tried using: stdbuf -oL -eL cat FILE | nc -u host port ... but that surprisingly didn't make a difference either.

5) stdbuf affects (only) the C-library 'stdio' routines, and cat probably doesn't use those because it is designed to work on both text and non-text 'binary' data. On my most convenient test system (CentOS6) strace confirms GNU-coreutils-8.4 cat isn't affected by stdbuf, but I can't rule out that other implementations could be. Using programs designed to handle text in lines like sed '' file; grep '' file; awk 1 file I confirm stdbuf -oL does make a difference.

But it may not make the difference you want. There will be a race condition between the writer program adding lines to the pipe and nc reading (and sending) them, and if the writer is faster you will still have multiple lines clumped into one outgoing packet.

Finally, I have discovered that if I do: for x in command1 command2; do echo $x; sleep 1; done | nc -u host port that everything goes as planned. The server receives the first command, then the second command.

What isn't clear really is why the sleep 1 makes a difference. ...

6) This avoids the race condition. Each time the shell writes one line to the pipe nc will definitely have time to read (and send) it before the shell's next write.

Improve this answer
3
  • Thanks for taking the time to address my questions. I see that among the utilities that you mention, grep even has a "--line-buffered" option. Therefore, I would expect that if I did: grep "^" --line-buffered <file> | ncat -u <host> <port> (without even using stdbuf) that I would then see each line from file sent individually to ncat (or at least a clump lf lines). The server only sees 1 packet with all the data. Is that because of this race condition in 6? Commented Oct 8, 2018 at 20:17
  • Obviously if I had my own "client" talk to the server, then I could control talking to the server one line at a time. However, more for research, I'm interested to know whether there's a way to do this from the command line using nc that would not be affected by this race (other than the sleep that I used). (without the change that I implemented for buffering in the code) Commented Oct 8, 2018 at 20:18
  • GNU grep (only) --line-buffered does the same thing as stdbuf -oL grep and yes is subject to the race in 5. I'm not sure I understand your second comment, but the only way nc with pipe input will read lines separately is if the pipe-writer does writes that are sufficiently separated in time; unless the writer is naturally slow (maybe because it's doing something time-consuming like SETI) you need some kind of forced delay in the writer, of which sleep is the simplest. Commented Oct 10, 2018 at 22:33
0

Your code is most of the way there, all you need to do now is to treat $cmd as a queue of newline-terminated strings. In naive Python-ish pseudo-code because I haven't written PHP for a long time:

input = '' try: first_command, remaining_input = input.split('\n', 1) execute(first_command) input = remaining_input except ValueError: # No newline yet; we have to wait for more input input = socket.receive()

This way you can handle both streams of input and single, interactive commands, which will probably not be the way your application will be used.

Improve this answer
1
  • Thanks for your suggestion. I adjusted the code accordingly, and it does work. However, in the end, it looks like the problem wasn't really on the server end. The problem was on the client end, and misunderstanding the buffering that goes on with pipes. Definitely a useful lesson indeed. Commented Oct 8, 2018 at 20:08

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.