Best practices

You will find on this page a collection of best practices that we discussed internally over time but lacks a centralized reference page.

1. Difference between step type: change, write, and keyDown.
2. The internal workings of the waitForNavigation attribute
3. The difference between waitFor and waitForNavigation
4. Why use waitForNavigation Instead of waitFor
5. Why Isn’t There a Generic Way to Determine if a Page Has Successfully Loaded?

Difference between step type: change, write, and keyDown.

Summarizing below the difference between the three different steps:

Main differences between change and write:

• The change steps work with input, textarea, andselect fields. While you can also use a write step for aninput ortextarea field, you cannot use it for aselect field to pick up values.

• The change step is faster to fill values, while the write step has a 200ms delay between each character typed which can slow down the screenshot process if there are a lot of forms to fill or a lot of text to type.

• The write step should be the preferable method when typing into a field that relies on keyboardEvent to trigger. A typical use case would be a search box with autosuggestion/autocomplete functionality.

The internal workings of the waitForNavigation attribute

The waitForNavigation attribute ensures that a page fully loads before executing the next step whenever there is a URL change, such as when clicking a hyperlink or a login button.

In Puppeteer, using waitForNavigation: 5000 translates to the following:

// Wait for the page to load completely using the 'networkidle2' algorithm
await page.waitForNavigation({waitUntil: 'networkidle2'})
// Then, wait an additional 5 seconds
await new Promise((resolve) => setTimeout(resolve, 5000));

In most cases, using only page.waitForNavigation({ waitUntil: 'networkidle2' }) is sufficient to ensure that a page has loaded successfully.

However, in some situations, the page might appear to be loaded before all elements are fully rendered. To address this, the second line, new Promise((resolve) => setTimeout(resolve, 5000)), adds a precautionary 5-second delay. This additional wait time helps prevent issues where page.waitForNavigation({ waitUntil: 'networkidle2' }) completes too early.

This approach is based on observations documented here, where adding a small delay proved beneficial in specific scenarios.

Difference Between waitFor and waitForNavigation

waitFor introduces a fixed delay before continuing, while waitForNavigation waits based on page load completion using the networkidle2 algorithm, plus an additional delay.

The delay value in waitForNavigation should typically be small (e.g., 500 to 5000 ms) because the primary waiting time is already handled by await page.waitForNavigation({ waitUntil: 'networkidle2' }). In many cases, a short delay like waitForNavigation:500 is enough.

To clarify, consider the following values:

waitFor:5000
waitForNavigation:2000

• Total waiting time for waitFor = 5000 ms (fixed)
• Total waiting time for waitForNavigation = page load time based on networkidle2 + 2000 ms

This distinction helps optimize loading times by combining flexible navigation waiting with minimal additional delay.

Why Use waitForNavigation Instead of waitFor

Consider the following delay values for waitFor and waitForNavigation:

waitFor: 5000 ms
waitForNavigation: 2000 ms

When Using waitFor:

• If the page takes 10,000 ms to load, waitFor will stop waiting after 5000 ms, which causes subsequent steps to fail or behave inaccurately since the page hasn’t fully loaded.
• If the page loads quickly, say in 2000 ms, waitFor still waits for the full 5000 ms, resulting in an unnecessary delay of 3000 ms.

When Using waitForNavigation:

• If the page takes 10,000 ms to load, waitForNavigation waits until it fully loads, resulting in a total wait time of 12,000 ms (10,000 ms for loading + 2000 ms delay).
• If the page loads in 1000 ms, waitForNavigation only waits for 3000 ms (1000 ms for loading + 2000 ms delay), saving time compared to waitFor.
• If the page takes 40,000 ms to load, waitForNavigation will wait the entire time until it loads, totaling 42,000 ms (40,000 ms for loading + 2000 ms delay).

Using waitForNavigation provides greater accuracy, executes more efficiently, and therefore reduces AWS Lambda infrastructure costs over time by avoiding unnecessary delays and handling varying load times more gracefully.

Using waitFor should be discouraged. Here's a great article explaining why.

Why Isn’t There a Generic Way to Determine if a Page Has Successfully Loaded?

There’s no universal method to determine when a page has fully loaded. Puppeteer offers several options, each suited for different scenarios:

page.waitForNavigation({ waitUntil: 'domcontentloaded' })
page.waitForNavigation({ waitUntil: 'load' })
page.waitForNavigation({ waitUntil: 'networkidle0' })
page.waitForNavigation({ waitUntil: 'networkidle2' })
page.waitForNetworkIdle({ idleTime: 250 })
page.waitForSelector("form.register", { visible: true })

Each of these methods waits for a different condition:

• domcontentloaded waits until the HTML is parsed and the DOM is ready.
• load waits until all resources like images and subframes are loaded.
• networkidle0 and networkidle2 wait for network activity to reduce to 0 or 2 connections, respectively.
• waitForNetworkIdle allows setting a custom idle time for finer control.
• waitForSelector waits for a specific element to appear on the page, which can be useful for pages with specific content.

At some point, we may consider introducing a similar feature, but this would mean that the application itself would need to select the appropriate waiting algorithm. This approach could potentially be confusing for users or customers, who may not be familiar with the nuances of each method.

For now, a combination of waitForNetworkIdle followed by a brief additional delay seems to be the most reliable and generic way to ensure pages load fully before moving to the next step.