9
$\begingroup$

Mathematica's CurrentScreenImage is very buggy on multiple monitor setups, so Wolfram tech support has proposed to use .NET or Java frameworks to implement my own.

This is what I have so far:

Needs["JLink`"] ReinstallJava[] LoadJavaClass["java.awt.Toolkit"]; robot = JavaNew["java.awt.Robot"] img = robot@createScreenCapture[JavaNew[ "java.awt.Rectangle",Toolkit`getDefaultToolkit[]@getScreenSize[]]]

But, img is a Java.awt.image.BufferedImage. How do I convert img to a Mathematica's Image? In particular, I'm concerned about memory leaks, so how do I do this in such a way, that I can grab a screen shot once per minute and not consume all memory after a day?

UPDATE

A had a concern of memory leaks. In Ben's answer below, I appear to have hit a java memory leak. Here's what's happening, if I do the appropriate preamble

Needs["JLink`"] ReinstallJava[] LoadJavaClass["java.awt.Toolkit"]; LoadJavaClass["javax.imageio.ImageIO"]; LoadJavaClass["java.io.ByteArrayOutputStream"]; robot = JavaNew["java.awt.Robot"]

Define a getScreen command as follows:

getScreen := Module[{}, tmpImgBuffer = JavaNew["java.io.ByteArrayOutputStream"]; ImageIO`write[ robot@createScreenCapture[ JavaNew[ "java.awt.Rectangle", Toolkit`getDefaultToolkit[]@getScreenSize[]]], "png", tmpImgBuffer]; ImportByteArray[ByteArray@Mod[tmpImgBuffer@toByteArray[], 2^8], "PNG"] ] Table[getScreen // ImageDimensions // Total, 1000] // Total

Then, getScreen consumes all Java memory very consistently after 30 calls. I have screen size of 2560,1440.

Improve this question
$\endgroup$
3

2 Answers 2

Reset to default
6
$\begingroup$

Courtesy of this post, one solution is to use java to get the PNG data then import it to Mathematica as Image.

For that you need to import the following class:

LoadJavaClass["javax.imageio.ImageIO"]; LoadJavaClass["java.io.ByteArrayOutputStream"];

Assuming:

LoadJavaClass["java.awt.Toolkit"]; robot = JavaNew["java.awt.Robot"]; img = robot@createScreenCapture[JavaNew[ "java.awt.Rectangle",Toolkit`getDefaultToolkit[]@getScreenSize[]]]

Write the image data in our variable:

temp = JavaNew["java.io.ByteArrayOutputStream"]; ImageIO`write[img, "png", temp];

Since Mathematica read the numbers as Integer8 instead of UnsignedInteger8, I used Mod to convert:

ImportByteArray[ByteArray@Mod[temp@toByteArray[], 2^8], "PNG"]

Update 1

As for @Alexey_Popkov's comment, I decided to extend this answer to get screenshots like CurrentScreenImage.

First I thought with help of AbsoluteOptions[EvaluationNotebook[], WindowSize] and AbsoluteOptions[EvaluationNotebook[], WindowMargins] it should be easy, but I realize their unit is not in pixel!

Exploring functions with Names, introduced me to RobotTools context which I think is similar to "java.awt.Robot" because of some common methods. It has a helpful function named GetWindowRectangle which works like CurrentValue[EvaluationNotebook[], WindowMargins] but doesn't include Automatic if you maximize. Since they also have different units than pixels, I used Rescale to interpolate to the actual pixel (but in some cases, it could <10 pixels off)

Requirements:

Needs["JLink`"]; LoadJavaClass /@ {"java.awt.Toolkit", "java.awt.Robot", "java.awt.Rectangle", "java.io.ByteArrayOutputStream", "javax.imageio.ImageIO"}; Needs["RobotTools`"]; 
ClearAll[cCurrentNotebookImage]; cCurrentNotebookImage[] := Block[{xMin, yMin, xMax, yMax, xLimit, yLimit, screenSizeX, screenSizeY, temp, robot}, {xLimit, yLimit} = System`Convert`TeXDump`getWindowSizeFromScreenRectangle[]; {{xMin, yMin}, {xMax, yMax}} = Floor@RobotTools`GetWindowRectangle[]; JavaBlock[ (*screen size in their unit*) screenSizeX = Toolkit`getDefaultToolkit[]@getScreenSize[]@width; screenSizeY = Toolkit`getDefaultToolkit[]@getScreenSize[]@height; (* clip extra pixels if windows is out of range *) {xMin, xMax} = Clip[{xMin, xMax}, {0, xLimit}]; {yMin, yMax} = Clip[{yMin, yMax}, {0, yLimit}]; (*interpolate*) {{xMin, xMax}, {yMin, yMax}} = Floor@{Rescale[{xMin, xMax}, {0, xLimit}, {0, screenSizeX}], Rescale[{yMin, yMax}, {38`, yLimit}, {0, screenSizeY - 13}]}; robot = JavaNew["java.awt.Robot"]; temp = JavaNew["java.io.ByteArrayOutputStream"]; img = robot@createScreenCapture[ JavaNew["java.awt.Rectangle", xMin, yMin, xMax - xMin, yMax - yMin]]; ImageIO`write[img, "png", temp]; ImportByteArray[ByteArray@Mod[temp@toByteArray[], 2^8], "PNG"] ] ]

If we consider MaxMemoryUsed is accurate on JLink calls, compared to CurrentNotebookImage using RepeatedTiming on version 13.1, it was ~30% slower but uses ~6 times less memory! (if you change the format from PNG to JPG, performance doesn't change but it significantly uses less memory, around 31 times less than CurrentNotebookImage!)

The remaining issues are:

  1. On maximized windows, black pixels appear on top + taskbar at the bottom,
  2. If you move windows out of range, the title bar may be clipped

Otherwise in not-maximized in-range windows, I didn't find any issue.

Advance User Note:

  • 38 came from Mathematica offset, in maximize mode it shows 32.25 and not-maximized but sticks to the top, it shows 38.25.
  • 13 was found in experiments (works fine in windows), feel free to change it

Update 2 3

Thanks to @Alexey_Popkov, this new version is ~40% faster than CurrentNotebookImage, more accurate than the previous update and does not use undocumented functions.

ClearAll[cCurrentNotebookImage]; cCurrentNotebookImage[] := Block[{xMin, yMin, xMax, yMax, screenSize, screenLimit, temp, robot, windowMargins, width, height, windowWidth, windowHeight, ratio, windowSize, displayInformation = First@SystemInformation["Devices", "ConnectedDisplays"]}, (* screen size in the unit *) screenLimit = ("Region" /. displayInformation)[[All, 2]]; (* ratio for each axis *) ratio = ("Resolution" /. First[SystemInformation["Devices", "ScreenInformation"]])/72.; (* screen size in pixel - without taskbar *) screenSize = Round[screenLimit*ratio]; (* current windows margin/size *) windowMargins = AbsoluteCurrentValue[WindowMargins]; windowSize = AbsoluteCurrentValue[WindowSize] + {0, 35}; If[NumberQ[windowMargins], (* maximized *) {xMin, yMin} = {0, 0}; {width, height} = screenSize; ,(* not maximized *) {xMin, yMin} = {windowMargins[[1, 1]], windowMargins[[2, 2]]}*ratio // Round; {width, height} = Round[windowSize*ratio]; (* make x/y in range *) xMin = Clip[xMin + 8, {0, First@screenSize}]; yMin = Clip[yMin + 2, {0, Last@screenSize}]; (* make width/height in range - clip the part gets out of view *) width = Clip[width, {0, First@screenSize - xMin}]; height = Clip[height, {0, Last@screenSize - yMin}]; ]; JavaBlock[ robot = JavaNew["java.awt.Robot"]; temp = JavaNew["java.io.ByteArrayOutputStream"]; img = robot@createScreenCapture[ JavaNew["java.awt.Rectangle", xMin, yMin, width, height]]; ImageIO`write[img, "png", temp]; ImportByteArray[ByteArray@Mod[temp@toByteArray[], 2^8], "PNG"] ] ]

As a side note, I notice Mathematica have RobotTools`CaptureScreenshot context but I couldn't execute their functions (all of them are in the Private context).

32.5 of 35 is for Mathematica's WindowSize not giving the full view range (it's -32.5 from "Region"), I added a little extra to make it snap (same goes for 8 and 2).

Update 4

Here is the NetLink version (faster than Java):

Needs["NETLink`"]; LoadNETType /@ {"System.Drawing.Graphics", "System.IO.MemoryStream", "System.Drawing.CopyPixelOperation", "System.Drawing.Imaging.ImageFormat"}; Module[{temp, g, memory}, NETBlock[ temp = NETNew["System.Drawing.Bitmap", 2560, 1080]; g = Graphics`FromImage[temp]; g@CopyFromScreen[0, 0, 0, 0, temp@Size, CopyPixelOperation`SourceCopy]; memory = NETNew["System.IO.MemoryStream"]; temp@Save[memory, ImageFormat`Png]; ImportByteArray[ByteArray@memory@ToArray[], "PNG"] ] ]
Improve this answer
$\endgroup$
15
  • 1
    $\begingroup$ Thanks for the update. Instead of Rescale, you could use the conversion coefficient OptionValue[First@SystemInformation["Devices", "ConnectedDisplays"], "Scale"] (see here) to get pixels from printer's points returned by AbsoluteCurrentValue[WindowMargins] and AbsoluteCurrentValue[WindowSize]. But for some reason direct use of these values gives slightly shifted screenshot (your code also gives a screenshot, which is slightly extended at the bottom of the window). $\endgroup$ Commented Aug 17, 2022 at 2:36
  • 1
    $\begingroup$ Also, the whole screen size can be obtained as OptionValue[First@SystemInformation["Devices", "ConnectedDisplays"], "PixelDimensions"]. $\endgroup$ Commented Aug 17, 2022 at 2:38
  • 1
    $\begingroup$ MaxMemoryUsed doesn't include the memory used by Java. It is better to control the memory usage from MemoryAvailable[] or SystemInformation["Machine", "MemoryAvailable"]. $\endgroup$ Commented Aug 17, 2022 at 2:41
  • 1
    $\begingroup$ The conversion from points to pixels is discussed in the "Possible Issues" section on the Docs page for SystemInformation. $\endgroup$ Commented Aug 17, 2022 at 2:45
  • 1
    $\begingroup$ I think that we shouldn't apply Round before multiplying by xyRatios: the values returned by AbsoluteCurrentValue are real by design, hence only after converting them to pixels we expect to get integers. $\endgroup$ Commented Aug 17, 2022 at 8:11
1
$\begingroup$

Here's a way to take a fullscreen screenshot using the PIL ImageGrab module, that is already installed in many Python environments:

fullscreenShot[] := ExternalEvaluate["Python", "import PIL.ImageGrab PIL.ImageGrab.grab()"]

Here is a function to take a screenshot of a specified screen rectangle using the same module:

screenShot = ExternalFunction["Python", "import PIL.ImageGrab def screenshot(left_x, top_y, right_x, bottom_y): return PIL.ImageGrab.grab(bbox=(left_x, top_y, right_x, bottom_y))" ]

out

Here is a pure Python function to take a screenshot of the current active window on Windows(1),(2):

activeWindowScreenShot[] := ExternalEvaluate["Python", "import ctypes from ctypes import wintypes import PIL.ImageGrab user32 = ctypes.windll.user32 handle = user32.GetForegroundWindow() rect = wintypes.RECT() user32.GetWindowRect(handle, ctypes.pointer(rect)) PIL.ImageGrab.grab(bbox=(rect.left,rect.top,rect.right,rect.bottom))"]

And here is a function to take a screenshot of the current Notebook window:

Clear[currentNotebookImage0] currentNotebookImage0[margins_ : 6] := Module[{screenInformation = First[SystemInformation["Devices", "ScreenInformation"]], windowMargins = AbsoluteCurrentValue[WindowMargins], resolution, screenArea, width, height, left, right, bottom, top}, {resolution, screenArea} = {"Resolution", "ScreenArea"} /. screenInformation; {width, height} = Round[screenArea[[All, 2]]*resolution/72.]; {{left, right}, {bottom, top}} = Round[If[MatrixQ[windowMargins], windowMargins + {{margins, margins}, {margins, margins}}, {{0, 0}, {0, 0}}]* resolution/72.]; {{left, right}, {top, bottom}} = {Clip[{left, width - right}, {0, width}], Clip[{top, height - bottom}, {0, height}]}; ExternalEvaluate["Python", <|"Command" -> "import PIL.ImageGrab def screenshot(left_x, top_y, right_x, bottom_y): return PIL.ImageGrab.grab(bbox=(left_x, top_y, right_x, bottom_y))", "Arguments" -> {left, top, right, bottom}|>]]

The default value 6 for the parameter margins is taken from what returns AbsoluteCurrentValue[WindowMargins] when the Notebook window is maximized on my system (Mathematica 13.1.0 on Windows 10 x64):

AbsoluteCurrentValue[WindowMargins] 
-6

With Mathematica 12.3.1 on the same machine I get the value -8.

The value -6 means that when the Notebook window is maximized, Mathematica thinks that the window is actually cropped by 6 printer's points from all sides. Translating into normal language, this means that the window is always enlarged by 6 printer's points on all sides as compared to its maximized size. The latter is equivalent in the meaning to the ImageMargins option of Graphics in Mathematica. Hence the margins parameter name.

These considerations are confirmed by comparing the values of the window width obtained from AbsoluteCurrentValue[WindowSize] and AbsoluteCurrentValue[WindowMargins] (output is for Mathematica 13.1.0):

windowMargins = AbsoluteCurrentValue[WindowMargins]; screenArea = "ScreenArea" /. First[SystemInformation["Devices", "ScreenInformation"]]; windowSize = AbsoluteCurrentValue[WindowSize]; screenArea[[All, 2]] - Total /@ windowMargins - windowSize 
{12., 44.25}

(for Mathematica 12.3.1 the output is {16, 59}).

We see that the window width reported by AbsoluteCurrentValue[WindowSize] is 6*2 = 12 printer's points smaller than the value obtain from AbsoluteCurrentValue[WindowMargins]. Hence we can calculate the value for the parameter margins from AbsoluteCurrentValue[WindowSize] as follows:

margins = If[MatrixQ[windowMargins], (screenArea[[1, 2]] - Total[windowMargins[[1]]] - windowSize[[1]])/2, -windowMargins] 
6.

With this definition, currentNotebookImage0 can be rewritten as follows:

Clear[currentNotebookImage] currentNotebookImage[] := Module[{screenInformation = First[SystemInformation["Devices", "ScreenInformation"]], windowMargins = AbsoluteCurrentValue[WindowMargins], margins, left, right, bottom, top, windowSize = AbsoluteCurrentValue[WindowSize], resolution, screenArea, width, height}, {resolution, screenArea} = {"Resolution", "ScreenArea"} /. screenInformation; margins = If[MatrixQ[windowMargins], (screenArea[[1, 2]] - Total[windowMargins[[1]]] - windowSize[[1]])/2, -windowMargins]; {width, height} = Round[screenArea[[All, 2]]*resolution/72.]; {{left, right}, {bottom, top}} = Round[(windowMargins + margins*{{1, 1}, {1, 1}})*resolution/72.]; {{left, right}, {top, bottom}} = {Clip[{left, width - right}, {0, width}], Clip[{top, height - bottom}, {0, height}]}; ExternalEvaluate["Python", <|"Command" -> "import PIL.ImageGrab def screenshot(left_x, top_y, right_x, bottom_y): return PIL.ImageGrab.grab(bbox=(left_x, top_y, right_x, bottom_y))", "Arguments" -> {left, top, right, bottom}|>]]

The main advantage of this approach is that it does not require "magic" numbers and works perfectly well both with version 13.1.0 and 12.3.1:

currentNotebookImage[]

output

It is easy to rewrite the great cCurrentNotebookImage[] function by Ben Izd using the new method for determining the window rectangle coordinates:

Needs["JLink`"]; LoadJavaClass /@ {"java.awt.Robot", "java.awt.Rectangle", "java.io.ByteArrayOutputStream", "javax.imageio.ImageIO"}; Clear[cCurrentNotebookImage] cCurrentNotebookImage[] := Module[{screenInformation = First[SystemInformation["Devices", "ScreenInformation"]], windowMargins = AbsoluteCurrentValue[WindowMargins], margins, left, right, bottom, top, windowSize = AbsoluteCurrentValue[WindowSize], resolution, screenArea, width, height}, {resolution, screenArea} = {"Resolution", "ScreenArea"} /. screenInformation; margins = If[MatrixQ[windowMargins], (screenArea[[1, 2]] - Total[windowMargins[[1]]] - windowSize[[1]])/2, -windowMargins]; {width, height} = Round[screenArea[[All, 2]]*resolution/72.]; {{left, right}, {bottom, top}} = Round[(windowMargins + margins*{{1, 1}, {1, 1}})*resolution/72.]; {{left, right}, {top, bottom}} = {Clip[{left, width - right}, {0, width}], Clip[{top, height - bottom}, {0, height}]}; JavaBlock[robot = JavaNew["java.awt.Robot"]; temp = JavaNew["java.io.ByteArrayOutputStream"]; img = robot@ createScreenCapture[ JavaNew["java.awt.Rectangle", left, top, right - left, bottom - top]]; ImageIO`write[img, "png", temp]; ImportByteArray[ByteArray@Mod[temp@toByteArray[], 2^8], "PNG"]]]

There is an old but still working GUIScreenShot[] from "GUIKit`". To take a fullscreen screenshot, just evaluate:

Needs["GUIKit`"] GUIScreenShot[]

To take a screenshot of the currently active Notebook, use the following function:

gCurrentNotebookImage[] := Module[{screenInformation = First[SystemInformation["Devices", "ScreenInformation"]], windowMargins = AbsoluteCurrentValue[WindowMargins], margins, left, right, bottom, top, windowSize = AbsoluteCurrentValue[WindowSize], resolution, screenArea, width, height}, {resolution, screenArea} = {"Resolution", "ScreenArea"} /. screenInformation; margins = If[MatrixQ[windowMargins], (screenArea[[1, 2]] - Total[windowMargins[[1]]] - windowSize[[1]])/2, -windowMargins]; {width, height} = Round[screenArea[[All, 2]]*resolution/72.]; {{left, right}, {bottom, top}} = Round[(windowMargins + margins*{{1, 1}, {1, 1}})*resolution/72.]; {{left, right}, {top, bottom}} = {Clip[{left, width - right}, {0, width}], Clip[{top, height - bottom}, {0, height}]}; GUIKit`GUIScreenShot[{{left, right}, {top, bottom}}]]
Improve this answer
$\endgroup$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.