PEP Weblib: using the JavaScript API
Structure
- C++
/cpp/pep/weblib/Weblib.cppclass - Compiled to
pepWeblib.wasmWebAssembly file withpepWeblib.mjsJavaScript module.mjsloads.wasmand exposes functions and contains Emscripten runtime (supporting functionality, e.g. spawning threads, JS interop, etc.)
/weblib/pep-repo-client-lib/pep.mtsTypeScript wrapper exposingPepclass- Also has supporting functionality (e.g. fetching config, opening popup, etc.)
utils.mtsstream utilities etc.
Customer perspective
- (Future work:) We publish package with
.wasm+ TypeScript wrapper & utils (compiled to JS) - Customer installs with NPM
- Hosts package files (may bundle with Webpack) or uses CDN + SubResource Integrity
- Imports package files in their script
- Points script to
ClientConfig.json - Script connects to PEP servers via WebSockets
Hosting details
- Required HTTP headers (for
SharedArrayBuffer):Cross-Origin-Embedder-Policy: require-corpCross-Origin-Opener-Policy: same-origin- Don't use
<meta http-equiv=... />
- Pages served over HTTPS need a Weblib built with
-DPEP_WEBLIB_USE_WSS=ON- Browsers block plain
ws://fromhttps://pages - The PEP servers must then be reachable via websockify with a publicly trusted certificate, see
/docker/websockify/
- Browsers block plain
- Compression recommended
- On-the-fly GZip fast:
- Release: 9.6 MiB → 3.3 MiB
- Debug: 68 + 304 MiB → 11 + 58 MiB
- Precompressed recommended for high traffic (e.g. Nginx
gzip_static)
- On-the-fly GZip fast:
- Sample Nginx & Webpack setup with compression in
/weblib/pep-sample-client/- Used for development & demo
Content-Security-Policy
- When using CSP, WASM compilation requires
script-src 'wasm-unsafe-eval'; - Emscripten uses
eval/Functionto generate bindings for performance reasons- Requires
script-src 'unsafe-eval'; - We could disable, with performance cost
- Requires
Initialization
import Pep from 'pep-repo-client';
const pep = await Pep.create({
clientConfig: new URL('dist/ClientConfig.json', location.href),
authLandingPage: new URL('pepWasmTest-auth-landing', location.href),
});
// Log in via popup
await pep.authenticate();
Authentication
pep.authenticate()
Opens Authserver dialog in popup, redirecting to authLandingPage afterwards.
This sends OAuth code or error back via BroadcastChannel in chan URL parameter.
ClientKeys.json persisted via IndexedDB.
Enumerate
const entriesStream = pep.list({
columnGroups: (await pep.listColumns()).map(cg => cg.name),
subjectGroups: (await pep.listSubjectGroups()).map(sg => sg.name),
});
for await (using entry of entriesStream) {
console.log(`${entry.subjectLocalPseudonym} ${entry.column}: ${entry.fileSize} B`)
}
Retrieve
import {byteStreamToString} from 'pep-repo-client/utils';
const entries = await Array.fromAsync(pep.list(...));
const dataStream = pep.retrieve(entries);
for await (using data of dataStream) {
console.log(`${data.entry.subjectLocalPseudonym} ${data.entry.column}:`,
await byteStreamToString(data.content));
}
Content should be read immediately: RxCpp will not stop sending.
Future work: Pause between file batches until reader catches up (pausing between pages impossible with RxCpp).
Memory management
using obj = ...;
Deletes backing C++ object on scope end.
Unsupported in Safari: instead, use try-finally with obj.delete(), or backport with TypeScript/Babel.
Stream loops should ideally look like this:
import {deleteObjectsAsync} from 'pep-repo-client/utils';
try {
for await (using obj of stream.values({preventCancel: true})) {
...
}
} finally {
await deleteObjectsAsync(stream);
}
Unfortunately, remaining objects otherwise leak when loop exits prematurely (e.g. exception).
Saving large file: Chromium
import {binaryToString} from 'pep-repo-client/utils';
const fileExtensionBin = entry.partialMetadataView().get('fileExtension');
const suggestedName = 'file' + (fileExtensionBin ? binaryToString(fileExtensionBin) : '');
const file = await showSaveFilePicker({suggestedName});
await data.content.pipeTo(await file.createWritable());
Saving large file: Firefox & Safari
// Origin-Private File System (virtual)
const rootDir = await navigator.storage.getDirectory();
const tmpDir = await rootDir.getDirectoryHandle('tmp-retrieve', {create: true});
const tmpFile = await tmpDir.getFileHandle(`file-${crypto.randomUUID()}`, {create: true});
await data.content.pipeTo(await tmpFile.createWritable());
const blob = await tmpFile.getFile();
const blobUrl = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = blobUrl;
a.download = suggestedName;
a.click();
// Unclear if a delay is required
setTimeout(() => {
URL.revokeObjectURL(blobUrl);
// Might want to make sure by deleting tmp-retrieve/ on page load
void tmpDir.removeEntry(tmpFile.name);
}, 1_000);
Future work: Compare with (more complex) Service Worker implementation from SURFfilesender.
Exceptions
Weblib functions may throw C++ exceptions represented in JS by WebAssembly.Exception.
Pass to pep.handleWasmException(ex):
- Obtains error message
- Converts to regular JS
Error - Deletes backing C++ exception object
Or use wrapper: pep.runHandleWasmException(() => pep.listColumns()).
Future work: add extra wrapping layer that does this automatically?
More future work
- Data Administrator functionality
- More cell operations:
- File upload
- Retrieve encrypted cell metadata
- ⇒ Should API be split up in explicit
getTicket(with modes) →getKeys→ ... after all?
- Downloading folders (via ZIP/TAR if needed)
- Structure metadata
- Publishing package (and semver)
Debugging
Chromium: Supports DWARF debug info from pepWeblib.wasm.debug.wasm (separated for performance) via extension.
Firefox & Safari: Support source map from pepWeblib.wasm.map. No variable names.