2

I am new to posting questions on this site so forgive me if this isn't in the best place or if I do something wrong.

I have two Pi's and I am trying to capture images in a loop (giving me a video), with the Raspberry Pi camera and send each through a socket to the other Pi and display them with pygame. I will also need low latency.

I have tried many different methods and think I have gotten fairly close. The code I have right now is posted at the bottom. There are however, a lot of other things I have tried. For example, I tried using the PiCamera module to capture an image and then make it a PIL image right away (instead of capturing an image with pygame). The code below seems to be fairly close but I can only seem to get the delay to about 5 seconds.

I also tried using UDP but could not get time image.tostring small enough to send through the socket.

I realize there are other ways to do this, like mjpg-streamer but I wanted something a bit less complicated that I could understand.

Any help would be appreciated!

server:

import socket import pygame as game import sys from PIL import Image port=9012 address="192.168.1.10" #server's ip size=width, height= 640, 480 scale=width, height=40, 10 timer = 0 previousImage = "" image = "" message=[] #establish socket connection s=socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind((address, port)) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) #initialize pygame game.init() screen=game.display.set_mode(size) game.display.set_caption('Socket Raspi Camera Viewer') #clock = game.time.Clock() #listen on the port s.listen(1) #main loop while 1: for event in game.event.get(): if event.type == game.QUIT: sys.exit() #accept socket connection connection, addr= s.accept() #Recieve data from the server: data=connection.recv(1024000) print(sys.getsizeof(data)) #store the previous recieved image incase the client fails to recive all of the data for the new image previousImage=image #We use a try clause to the program will not abort if there is an error: try: #We turn the data we revieved into a 120x90 PIL image: #image = Image.fromstring("RGB",(120,90),data) #We resize the image to 640x480: pil_image = Image.fromstring("RGBA", size, data) pil_image= pil_image.resize(size) new_data=pil_image.tostring() image=game.image.fromstring(new_data, size, "RGBA") except: #If we failed to recieve a new image we display the last image we revieved: print("falied to recieve image") image = previousImage #Set the var output to our image: #output = image #We use PyGame to blit the output image to the screen: screen.blit(image,(0,0)) #We update the screen: game.display.flip()`

client:

import pygame as game import pygame.camera import sys import time from PIL import Image port=9012 address="192.168.1.10" #server's ip size=width, height= 640, 480 scale=width, height= 40, 10 timer=0 game.init() game.camera.init() screen=game.display.set_mode(size) camera=game.camera.Camera("/dev/video0", size ) camera.start() time.sleep(1) while True: if timer < 1: #connect to the server s=socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((address, port)) #get image image=camera.get_image() #convert to pill image to scale string_image=game.image.tostring(image, "RGBA",False) pil_image=Image.fromstring("RGBA",size,string_image) pil_image =pil_image.resize(scale) buffer=pil_image.tostring() #print size of buffer print(sys.getsizeof(buffer)) s.sendall(buffer) #time.sleep(0.01) s.close() timer=30
#

Stream without resizing: client-

while True: image=camera.getimage() buffer=pygame.image.tostring(image) socket.send(buffer)

Server:

while True: message=[] while True: string=socket.recieve(1024 * 1024) if string == '': break else: message.append(string) full_string="".join(message) image=game.fromstring(full_string) game.blit(image, (0,0)) game.display.update()
Improve this question

1 Answer 1

Reset to default
1

I must confess that I know nothing about image processing from the Pi camera and I've only played around a little bit with Python. Yet some things look suspicious about your code.

You seem to be creating a socket connection, sending one image (still frame) down the pipe, closing the socket, then repeat. You don't want to be doing it this way. Too much overhead and I think you will quickly run out of socket numbers. On the server side, the part with the listen and accept: you only want to do this ONCE. i.e. Do your listen and wait for the client side to connect, then server accepts, THEN you want to stay in your loop sending and receiving image frames. Only when you are all done do you want to close the socket. The way you are doing it (I think) is creating tremendous overhead in the comms portion of your system. You may want to invent a little protocol for yourself so that after processing a received image frame, the server sends (for example) "OK" back to the client to say "last frame good, ready for next". The client waits for "OK" before sending another frame.

Likewise, with the image processing, again please forgive me if I'm talking out of turn because I don't know anything about this specifically, but I think you want to get the data from the camera and transmit it immediately, not be messing around resizing, converting to/from strings, etc, on the client side. Let the server do the needed transformations because he is the guy who is going to display it. In my opinion, as soon as you camera.get_image you should have the data for one image sitting in your buffer "image". SEND IT. As long as you know the length of your data, sockets should have no problem sending it as binary data. You don't need to convert to string (unless that is the only way you can find out the length). All those conversions and resizes are expensive and that has to be killing your performance. Just get the data from the camera and do as little as possible and then send it. Let the server side monkey around with it.

Here again, DON't CLOSE YOUR SOCKET. Keep using it!

That's my two cents. Good luck.

Improve this answer
7
  • I have considered that but couldn't seem to get it to work that way. It does make sense. This version of my code has the resizing other versions did not need it. The client resizes it so it can be sent and received through the socket with out having to add a whole new while loop, in the server, for receiving the huge image string. Are you saying send the image without converting it to a string? I was under the impression that was impossible. I will however keep working on sending the data with one socket connection. Thanks for the help. Commented May 21, 2017 at 23:04
  • The "image" should just be a series of bytes. As long as you know the length, you should be able to send it through a socket. The only reason I could see that would mandate converting it to a "String" would be if your communications channel could not support binary data (socket should be able to handle it) or, for example, storing in a database which could not handle binary data. Now I understand that you were trying to reduce the volume of data with your resize. That may be a reasonable thing to try. Any idea how big (bytes) is a single image before and after a resize? Commented May 23, 2017 at 3:47
  • sending an image like this: image=camera.capture() and then socket.send(image) did not work. It says "must be string or buffer, not pygame surface". Using sys.getsizeof(buffer) the full size image is about 921621 and the resized image is 1621 (these are the string versions of the image). Doing sys.getsizeof(image) gives about 28, but after resizing i am getting 36. I must be doing something wrong. Also, sys.getsizeof() may be the wrong function, i don't know what else to use. The main problem i am having with resizing is when blit ing the image on the server side it looks weird. Commented May 23, 2017 at 17:14
  • Ok, given what you said, I'd have to agree that converting to string is necessary to get a plain ol' "series of bytes" that I'm talking about. Can you imagine how much of your image you must be losing going from 920K to 1621 bytes? Something seems wrong there... But, even at 920K, on 100Mb LAN, seems like we could probably send the buffer in (let's say) .5 seconds max, allowing for TCP/IP stack to bust it down into packets, add a lot of protocol and re-assemble on the other end. I'd concentrate on sending the UN-resized buffer first, make sure you rcv same # bytes on server side, then refine. Commented May 24, 2017 at 7:48
  • Thank you so much for your help, i really appreciate it. I edited the original question to contain a very simple streaming method that pretty much works (not every detail is there but you should get the idea). As you can see it takes an inner loop to receive the image, then the outside loop continues and blits the image to the screen. The version above does not have a socket connection for every image but WITH them i am able to get the latency to about 5 seconds. I have somewhat got it working without them but the latency more than doubles to about 12 seconds. Both codes are fairly similar. Commented May 24, 2017 at 16:19

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.