WordPress Plugin i18n, Webpack, and Composer

By: Brad Jorsch

A whole lot of work has been happening within the Jetpack plugin these days. We’ve UIs in-built React, with the JavaScript bundles being created by Webpack. We’ve Composer packages used for code sharing, more and more in order we glance into creating standalone plugins like Jetpack Backup and Jetpack Search. And we would like all the pieces to be translated for individuals who communicate languages aside from English.

A number of months again we began getting experiences that some translations in Jetpack had gone lacking. As we seemed into it, we ultimately discovered no fewer than six totally different ways in which translation was damaged!

1. JavaScript bundles weren’t being scanned on account of dangerous file naming.
2. Webpack’s optimizations have been breaking the i18n operate calls so WordPress.org’s translation infrastructure couldn’t discover them.
3. Lazy-loaded Webpack bundles weren’t lazy-loading translation knowledge.
4. Shared React part translations didn’t work in plugins aside from Jetpack itself.
5. Bundled Composer packages weren’t being scanned.
6. Composer bundle translations didn’t work in plugins aside from Jetpack itself.

It took us a couple of months, however we’ve now fastened all of them. This put up will describe how we did it.

Background: How plugins get translated

The really useful method to have your plugin translated is to let WordPress.org extract the translatable strings out of your plugin, then construct language packs for you based mostly on the work of volunteer translators.

In your code, each PHP and JavaScript, you move translatable strings to features similar to __(), _x(), and so forth. When that code is uploaded to WordPress.org SVN, it will get scanned for these calls. The strings for every name are collected and handed into the GlotPress set up at translate.wordpress.org, the place volunteers translate them into varied languages. The translations are later collected into language packs, which might be downloaded and put in into WordPress so individuals can expertise your plugin in their very own language.

(Apart: The extraction is a part of the WP-CLI instrument: wp i18n make-pot. They use the --slug and --ignore-domain choices. Era of JavaScript translation recordsdata is completed in a fashion much like wp i18n make-json, however skipping any JS recordsdata in a src/ listing)

The extracted strings are all related to a “textual content area” matching your plugin’s slug. If the area parameter handed to __() and so forth doesn’t match, your translations received’t be discovered at runtime.

All this makes some assumptions about your code, a few of which turned out to not be true for the way in which we have been doing issues in Jetpack.

Downside 1: JavaScript bundle naming

After we dropped assist for Web Explorer 11 in our JavaScript, Babel stopped transpiling trendy syntax similar to template strings into an ES5 kind that IE11 may perceive. This then broke after we deployed to WordPress.com, as that surroundings mechanically applies it personal minifier to JavaScript and CSS whereas serving it and their minifier doesn’t perceive template strings both. WordPress.com doesn’t apply its minifier if the recordsdata are named like “bundle.min.js”, so we renamed our recordsdata like that.

However that bumped into one of many WordPress.org translation infrastructure’s assumptions: they assume any “bundle.min.js” might be ignored as a result of it can have a corresponding “bundle.js” subsequent to it. 😬

We didn’t need to embody a number of hundred Okay of non-minified JS in Jetpack although. Jetpack already has an undeserved fame for being “bloated”, and the additional recordsdata wouldn’t assist even when the non-minified JS isn’t used.

Answer: The URL handed to wp_register_script() can embody a question half, and WordPress.com’s minifier may also be bypassed by together with minify=false within the question half.

Then we took it a step additional. The registration of a Webpack bundle normally entails a good bit of boilerplate because you additionally must learn the knowledge produced by @wordpress/dependency-extraction-webpack-plugin and infrequently register some CSS too, one thing like

$relative_to = __FILE__; // Or one thing.
$belongings = require dirname( $relative_to ) . '/construct/bundle.asset.php';
wp_register_script(
    'deal with',
    plugin_url( 'construct/bundle.js', $relative_to ), // TODO: Add "?minify=false".
    $belongings['dependencies'],
    $belongings['version']
);
wp_set_script_translations( 'deal with', 'textdomain' );
wp_register_style(
    'deal with',
    plugin_url( is_rtl() ? 'construct/bundle.rtl.css' : 'construct/bundle.css', $relative_to ),
    array( /* Another dependencies? */ ),
    $belongings['version']
);

So we added a technique in our automattic/jetpack-assets Composer bundle to deal with all that in a better manner. And we will have it carry out some easy checks, like requiring {that a} textdomain be given if the dependencies embody wp-i18n.

Property::register_script( 'deal with', 'construct/bundle.js', __FILE__, array( 'textdomain' => 'area' ) );

Downside 2: Webpack optimization breaking i18n operate calls

The extraction of translatable strings from JavaScript is determined by seeing the decision to a operate or methodology named __(), _x(), _n(), or _nx(), with the assorted parameters being handed as literal strings. These calls could also be preceded by a “translator remark” which can also be extracted.

In its default configuration, Webpack in manufacturing mode is prone to rename these features to single-character names and to throw away these translator feedback. And even when it’s working now, a modified configuration or a brand new code sample would possibly break it sooner or later (as occurred to us after we up to date to Webpack 5). 😬

Answer, half 1: Step one was to determine the mandatory configuration to protect the i18n operate calls and the translator feedback.

• Set Webpack’s .optimization.concatenateModules false, because the concatenation typically winds up renaming the strategies.
• As an alternative of counting on Webpack’s default configuration for Terser, provide (by way of .optimization.minimizer) an occasion of terser-webpack-plugin configured to protect the calls and feedback.
  • .terserOptions.mangle.reserved set to order the 4 strategies.
  • .terserOptions.format.feedback set to a callback that identifies translator feedback.
  • .extractComments set to a callback that identifies the license feedback Terser preserves by default, which can now be extracted to a separate file as an alternative to scale back the scale of the bundle.
• We additionally included Calypso’s @automattic/babel-plugin-preserve-i18n plugin to additional assist protect the i18n methodology names.

Answer, half 2: To deal with the “even when it’s working now, it’d break later” drawback, and to assist determine coding patterns that may break the i18n methodology calls even with the above configuration, we created @automattic/i18n-check-webpack-plugin. This plugin extracts the strings from the unique sources and the output bundle to check them and see if something appears to have gone lacking, so if one thing breaks it’ll make the construct fail as an alternative of getting to attend for somebody to note the damaged i18n and report it.

The documentation for the examine plugin consists of some identified problematic code patterns and fixes for them.

Downside 3: Lazy-loaded Webpack bundles

For “entry” bundles, Webpack expects your HTML to incorporate any extra recordsdata (e.g. CSS extracted by mini-css-extract-plugin) your self. In a WordPress plugin that is pretty simple to do from PHP (and we made it even simpler for ourselves utilizing automattic/jetpack-assets as described above), and that features loading of the suitable translation knowledge into @wordpress/i18n.

However if you happen to use code like import( /* webpackChunkName: "async" */ './one thing' ), Webpack will create a “lazy-loaded” bundle that isn’t loaded till that import() name is executed. In Jetpack we’ve considered one of these within the Prompt Search module. For such lazy-loaded bundles the Webpack runtime is aware of the best way to load the extracted CSS, however it is aware of nothing about WordPress translation knowledge. 😬

After we seemed round we noticed that Calypso had a reasonably difficult resolution of their code, a generic hook added within the Webpack runtime and particular code to load knowledge when that hook fired. Woo had tried to adapt that however gave up in favor of tricking WordPress into loading the lazy bundle’s translations non-lazily. Neither resolution appealed.

Answer: We created @automattic/i18n-loader-webpack-plugin to show Webpack the best way to load the WordPress translation knowledge. It’s designed to work in live performance with @wordpress/dependency-extraction-webpack-plugin and automattic/jetpack-assets: when i18n-loader encounters a bundle that may lazy-load different bundles that use @wordpress/i18n, it can register a dependency on a “@wordpress/jp-i18n-state” module by way of the previous that’s supplied by the latter. The state knowledge lets the Webpack runtime contained in the bundle know the best way to find the interpretation knowledge, which it can then obtain and register with @wordpress/i18n throughout the lazy-loading course of.

Downside 4: Textual content domains in shared React parts

As a part of the “Jetpack RNA” mission, we’ve begun creating React parts that may be shared by a number of plugins, like our personal Jetpack Backup and Jetpack Search plugins.

However keep in mind how the __() name must specify a website, which is meant to be a relentless string (not a variable) and should match the plugin’s slug? 😬

Answer: We created @automattic/babel-plugin-replace-textdomain, a easy Babel plugin to rewrite the domains because the parts are being bundled.

Downside 5: Bundled Composer packages being skipped

WordPress core doesn’t actually use Composer; they’ve a composer.json, however simply to drag in PHPUnit and some different improvement instruments. The place WordPress core wants libraries at runtime, they copy them in statically. Plugins both do the identical or embody Composer’s vendor/ listing within the code checked into WordPress.org SVN.

The WordPress.org translation infrastructure assumes that something in vendor/ both has no translations or has its personal translation mechanism completely, as an alternative of intending to make use of WordPress’s. (Though since __() and such can be outlined by WordPress relatively than the plugin, I’m undecided how that’s meant to work.) In our case, we do really need these packages’ strings included within the plugin’s language pack. 😬

Answer: Composer permits for customized installer plugins, which may set up packages into totally different areas based mostly on the “sort” subject within the bundle’s composer.json. We created automattic/jetpack-composer-plugin that installs “jetpack-library” packages into jetpack_vendor/, and set the forms of the related packages to “jetpack-library”.

Downside 6: Textual content domains in Composer packages

As with the shared React parts, the bundled Composer packages should be utilizing the plugin’s textual content area as a result of that’s the place the translations are going to be. And this time we’re not compiling them right into a bundle, so a compile-time replacer wouldn’t work. 😬

Answer: The answer right here is available in a number of components.

1. We’ve automattic/jetpack-composer-plugin write an “i18n-map.php” file into jetpack_vendor/, accumulating the WordPress plugin’s slug (set in its composer.json) and every bundle’s textdomain and model (from their composer.jsons).
2. The WordPress plugin passes that file to automattic/jetpack-assets, which determines the mapping from every bundle’s area to an applicable plugin’s.
3. Property hooks into __() and such to strive the plugin’s area if no translation was discovered for the bundle’s. It additionally hooks into the script translation file loader to level to the script translation recordsdata included with the plugin’s language pack as an alternative of the nonexistent packs for the packages’ textual content domains. And eventually it consists of the mapping within the state knowledge for @automattic/i18n-loader-webpack-plugin so that may load the right file for any lazy-loaded bundles.

We additionally made certain that our monorepo’s CI checks would catch frequent instances the place builders would possibly wind up with flawed textual content domains, utilizing present linting guidelines from @wordpress/eslint-plugin and wp-coding-standards/wpcs and customized checks to confirm these guidelines’ configurations are in sync with one another and with composer.json.

If WordPress Core have been to tackle this drawback, I believe they may do it a bit higher:

1. Swap the interpretation infrastructure from being based mostly on plugins and themes (which all use the plugin or theme slug because the textual content area) to being based mostly on the textual content domains straight. For instance, as an alternative of https://api.wordpress.org/translations/plugins/1.0/ and https://api.wordpress.org/translations/themes/1.0/ simply have one endpoint that takes the area.
2. Let code declare to WordPress which domains it wants past the defaults of “plugin slug” and “theme slug”. That is so WordPress can obtain these further domains, I don’t assume WordPress cares past that.
3. Allow us to register initiatives that aren’t plugins or themes (e.g. our packages) on translate.wordpress.org, so the packages might be translated and WordPress can fetch these translations prefer it does plugins and themes.

That manner the plugin solely must declare the packages’ textual content domains, and the translators would solely should translate every bundle’s strings as soon as as an alternative of doing so for each plugin utilizing the bundle.

Abstract and conclusion

We created a number of items to make all the pieces work:

We additionally took benefit of @wordpress/dependency-extraction-webpack-plugin and Calypso’s @automattic/babel-plugin-preserve-i18n, in addition to linter guidelines from @wordpress/eslint-plugin and wp-coding-standards/wpcs (and a fork of phpcs that we’ve been making an attempt to upstream to allow us to have per-directory configs) to assist builders in our monorepo hold textual content domains straight.

Total this was fairly a bit of labor, however Jetpack’s i18n is now higher than ever earlier than. And we hope that this put up describing the issues we discovered and our options would possibly assist different plugin builders enhance their i18n as properly.