4

I have the following code:

puppeteer.launch().then(async browser => { for (let id of ids) { try { const page = await browser.newPage(); //const url = 'chrome://crash'; await page.goto(url + id) await page.waitFor(5000); await page.screenshot({ path: path.join(__dirname, "../public/images/screenshots/" + id + ".png"), clip: { x: 10, y: 70, width: 780, height: 470} }); } catch (error) { console.log('Exception', id, error.message); page.close(); } }; browser.close(); }); 

It generally works OK but I'm having problems with a particular page (with a URL that unfortunately I can't share).

This page tries to load a GB of data and causes Chrome to crash, so I guess it's causing Chromium to crash too.

The error I see from this page is: Exception 6766 Navigation Timeout Exceeded: 30000ms exceeded. This is fine, but it doesn't seem to stop at this point - it causes my whole server to hang, I guess because it's trying to use too much memory at the OS level.

How can I stop my server from hanging and handle this gracefully? Are there flags I can provide to Chromium to limit the memory used and give up gracefully? I'm also not sure my error handling as a whole is correct, so any tips would be appreciated.

1
  • 2
    I think it's probably best to separate the actual browser from your infrastructure so it crashing doesn't bring down your entire stack. I'm the creator of browserless (browserless.io) and it's objective is to do just that. There's open-sourced docker builds available here as well: github.com/joelgriffith/browserless Commented Jan 18, 2018 at 18:12

2 Answers 2

9

a few ideas:

First, this may not be related to your initial problem but it will save some memory. You're opening many tabs (pages) without closing them, that can use a lot of memory if your list of Ids is long.

Try to iterate over the IDs by loading a new url instead of creating new pages.

Also you can extend the timeout to check if more time will eventually do the trick.

There are chromium flags that you can pass on when you launch puppeteer. Here is the complete Chromium switch list

You could try with: --unlimited-storage or --force-gpu-mem-available-mb.

Also this may come handy to debug: --full-memory-crash-report

You could search for all the other memory related flags there.

Here is how you can pass on the flags to puppeteer, and the other suggestions that I mentioned:

puppeteer.launch({args: ['--unlimited-storage', '--full-memory-crash-report']}).then(async browser => { const page = await browser.newPage(); for (let id of ids) { try { //const url = 'chrome://crash'; // Timeout will be 2 min and will wait till network is idle before taking the screenshot await page.goto(url + id, {timeout: 120000, waitUntil: 'networkidle0'}); await page.screenshot(); } catch (error) { console.log('Exception', id, error.message); // If you catch an error you should throw it and handle it on the parent function // or set up an event listener so you can now what's the state of your app throw new Err (error); // Since we are only opening one page and browsing from here // no need to close when you encounter an error // just goto the next iteration // page.close(); } }; browser.close(); }); 
Sign up to request clarification or add additional context in comments.

Comments

9

Try passing the {args: ['--disable-dev-shm-usage']} launch option the launch() command--this is supposed to improve Puppeteer's ability to render large/complex pages.

2 Comments

Thanks! I'm not actually running inside a docker container though - just on Ubuntu. So this is probably irrelevant?
@Richard Yes true, this will probably only help if you're running Docker.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.