Repository navigation
Repeated package.json reads in ts-loader ts api wrapper compared to tsc --build #42670
Description
I'm noticing extreme performance problems comparing a tsc --build to webpack with ts-loader of the same project. Sysinternals Process Monitor is showing about 11k file reads from a 1 minute tsc --build and nearly 10x ReadFile events through ts-loader, with matchingly awful CPU and memory work before an ultimately successful build hours later. I am pursuing this as a ts-loader optimization problem, but I'm not sure if the ts api gives enough control to succeed.
TypeScript/src/compiler/moduleNameResolver.ts
Lines 963 to 966 in d36df0d
Perhaps it's exactly what it needs to do, but this /*considerPackageJson*/ true here is causing repeated reads/parsing of package.json files from dependencies in a row. I can override the injected readFile() and possibly avoid the read with a cache, but I don't think we can avoid the json parse because that is done internal to typescript.
Common dependencies with many internal .d.ts files are costing 6000+ read/parse of their package.json files in a run, with a long tail down to single reads.
Here is the call to compiler.resolveModuleName()
Here is the injected readFile that is called back to read the contents of many package.json again and again, with each relative path import of a .d.ts file within its library:
Once the packageJsonInfo is extracted, I suspect additional waste but one step at a time.
Activity
JasonKleban commentedon Feb 11, 2021
Sheetal Nandi (@sheetalkamat) Ryan Cavanaugh (@RyanCavanaugh) I created a self-contained repro for demonstrating this behavior here: https://github.com/JasonKleban/practices-2021
JasonKleban commentedon Feb 18, 2021
When comparing the output of traceResolutions: true between the two build methods of the linked sample repo, tsc has 898 lines of output while the webpack & ts=loader version generates 6201 lines. Here is the first significant divergence and it may contain a clue?
and this is the next difference suggesting that tsc is getting a cache hit whereas ts-loader way is not attempting that (maybe, but perhaps the diff proximity is misleading here):
sheetalkamat commentedon Apr 9, 2021
This is a multi repo issue and i suggest we fix this with:
readJsonas well asgetPackageJsonInfoand handle those things. Need to see when and which kind of invalidation should happen in watch and tsserver mode.Also
1might not be needed with2and3sheetalkamat commentedon Apr 20, 2021
JasonKleban i have PR #43700 and TypeStrong/ts-loader#1287 which i verified with your sample repro.
Can you check if the builds from these two help with your scenario and report. Thanks