Skip to content

Pre-Compiler Plugin Proposal #38736

Description

Search Terms

Extension, Plugin, Vue, Custom extensions

I found some issues but no proposal :/

Suggestion

Hi! I'm an author of the fork-ts-checker-webpack-plugin. Last few months I was busy with rewriting this plugin to pay off the technological debt.

One of the features that this plugin provides is support for the Vue.js Single File Components. I was able to implement it but in order to work with the Watch program and SolutionBuilder I had to do some workarounds. It's because there is an assertion in the TypeScript source code that files cannot have a custom extension.

The main issue with this approach is that you have to implement these workarounds in every TypeScript's API client and they can behave differently (like appendTsSuffixTo in the ts-loader). I know that there are already Language Service Plugins, but they don't work if you use different API, like Watch.

The proposal

A new type of TypeScript plugins that can pre-compile custom source code to the format that TypeScript understands. The simplified interface would be:

////////////////////////////////////////////////
// update existing implementation and typings //

/**
 * Keep it for the backward compatibility
 * @deprecated
 */
interface PluginCreateInfo {
  project: Project;
  languageService: LanguageService;
  languageServiceHost: LanguageServiceHost;
  serverHost: ServerHost;
  config: any;
}
// the new name for this type
type LanguageServicePluginCreateInfo = PluginCreateInfo;

/**
 * Keep it for the backward compatibility
 * @deprecated
 */
interface PluginModule {
  type?: 'language-service-plugin'; // not required due to backward compatibility
  getExternalFiles?(project: Project): string[];
  onConfigurationChanged?(config: any): void;
  create(createInfo: PluginCreateInfo): LanguageService;
}
// the new name for this type
type LanguageServicePluginModule = PluginModule & {
  type: 'language-service-plugin'; // required in the new type
}

/////////////////////////////
// add new type of plugins //

interface PreCompilerPluginModuleCreateInfo {
  moduleResolutionHost: ModuleResolutionHost;
  sourceMapHost: SourceMapHost;
  config: any;
}
interface PreCompilerPluginModule {
  type: 'pre-compiler-plugin'; // to distinguish between LanguageServicePlugin and PreCompilerPlugin
  onConfigurationChanged?(config: any): void;
  create(createInfo: PreCompilerPluginModuleCreateInfo): PreCompilerPlugin;
}
interface PreCompilerPlugin {
  extensions: string[]; // list of supported extensions, for example ['.vue'] or ['.mdx']
  createPreCompiledSourceFile(
    fileName: string,
    sourceText: string,
    languageVersion: ScriptTarget,
    setParentNodes?: boolean
  ): PreCompiledSourceFile;
}
// we need to define an extended version of the SourceFile to support additional sourceMap
interface PreCompiledSourceFile extends SourceFile {
  preSourceMapText: string; // we need to provide source map to calculate diagnostics positions and source maps
}

// we need to define a function to create a PreCompiledSourceFile 
// (as we can't infer ScriptKind so it's not an optional parameter)
function createPreCompiledSourceFile(
  fileName: string,
  sourceText: string,
  languageVersion: ScriptTarget,
  scriptKind: ScriptKind,
  setParentNodes?: boolean,
  sourceMapText?: string
): PreCompiledSourceFile

// this will help with source map generation
interface SourceTextNavigator {
  getLineAndCharacterOfPosition(position: number): LineAndCharacter;
  getPositionOfLineAndCharacter(line: number, character: number): number;
  getLineEndOfPosition(position: number): number;
  getLineStarts(): number[];
}
interface SourceMapHost {
  createSourceMapGenerator(fileName: string): SourceMapGenerator;
  createSourceTextNavigator(text: string): SourceTextNavigator;
}


////////////////////////////////////////////////
// update existing implementation and typings //

type AnyPluginModule = PluginModule | PreCompilerPluginModule;

interface PluginModuleWithName {
  name: string;
  module: AnyPluginModule;
}
type PluginModuleFactory = (mod: {
  typescript: typeof ts;
}) => AnyPluginModule

This architecture would allow adding new plugin types in the future (so we could add ModuleResolutionPlugin to support Yarn PnP in the future)

The maintenance cost of this feature should be low because the exposed API is pretty simple.

There is a point in the TypeScript Design Goals

  1. Provide an end-to-end build pipeline. Instead, make the system extensible so that external tools can use the compiler for more complex build workflows.

If you think that this feature is incompatible with this point, we could limit it to emitting only .d.ts files.

I could try to implement it, but first I need to know if it's something that you would like to add to the TypeScript and if this API is a good direction :)

Use Cases

I imagine that community would create PreCompilerPlugins for a lot of use cases. For example:

  • vue-typescript-plugin - support for the Single File Components
  • mdx-typescript-plugin - support for embedded TypeScript in the MDX files
  • graphql-typescript-pluging - support for generation of GraphQL types / clients in the runtime (so we don't have to use generators anymore)
  • css-modules-typescript-plugin - support for typed css modules (instead of declare module '*.css';)

Examples

tsconfig.json

{
  "compilerOptions": {
    "target": "ES6",
    "module": "CommonJS",
    "lib": ["ES6"],
    "moduleResolution": "Node",
    "resolveJsonModule": true,
    "esModuleInterop": true,
    "baseUrl": "./src",
    "outDir": "./lib",
    "plugins": [{ "name": "vue-typescript-plugin" }]
  },
  "include": ["./src"],
  "exclude": ["node_modules"]
}

node_modules/vue-typescript-plugin/index.ts

It's a pseudo-code based on the fork-ts-checker-webpack-plugin implementation. It adds support for the .vue files.

import compiler from 'vue-template-compiler';

function init(modules: { typescript: typeof import("typescript/lib/typescript") }) {
  const ts = modules.typescript;

  function create({ sourceMapHost }: ts.PreCompilerPluginModuleCreateInfo): ts.PreCompilerPlugin {
    function getScriptKindByLang(lang: string | undefined) {
      switch (lang) {
        case 'ts':
          return ts.ScriptKind.TS;
        case 'tsx':
          return ts.ScriptKind.TSX;
        case 'jsx':
          return ts.ScriptKind.JSX;
        case 'json':
          return ts.ScriptKind.JSON;
        case 'js':
        default:
          return ts.ScriptKind.JS;
      }
    }

    function createNoScriptSourceFile(
      fileName: string,
      sourceText: string,
      languageVersion: ts.ScriptTarget,
      setParentNodes: boolean | undefined
    ): ts.PreCompilerSourceFile {
      const compiledText = 'export default {};';

      // generate source map
      const sourceMap = sourceMapHost.createSourceMapGenerator(fileName);
      const sourceIndex = sourceMap.addSource(fileName);
      sourceMap.setSourceContent(sourceIndex, sourceText);

      const sourceTextNavigator = sourceMapHost.createSourceTextNavigator(sourceText);

      const sourceStart = sourceTextNavigator.getLineAndCharacterOfPosition(0);
      const sourceEnd = sourceTextNavigator.getLineAndCharacterOfPosition(sourceText.length);

      sourceMap.addMapping(0, 0, sourceIndex, sourceStart.line, sourceStart.character);
      sourceMap.addMapping(0, compiledText.length, sourceIndex, sourceEnd.line, sourceEnd.character);

      // create source file
      return ts.createPreCompiledSourceFile(
        fileName,
        'export default {};',
        languageVersion,
        ts.ScriptKind.JS,
        setParentNodes
      );
    }

    function createSrcScriptSourceFile(
      fileName: string,
      sourceText: string,
      languageVersion: ts.ScriptTarget,
      setParentNodes: boolean | undefined,
      sourceStartPosition: number,
      sourceEndPosition: number,
      scriptTagSrc: string,
      scriptTagLang: string | undefined,
    ): ts.PreCompilerSourceFile {
      // import path cannot be end with '.ts[x]'
      const compiledText = `export * from "${scriptTagSrc.replace(/\.tsx?$/i, '')}";`;

      // generate source map
      const sourceMap = sourceMapHost.createSourceMapGenerator(fileName);
      const sourceIndex = sourceMap.addSource(fileName);
      sourceMap.setSourceContent(sourceIndex, sourceText);

      const sourceTextNavigator = sourceMapHost.createSourceTextNavigator(sourceText);
      const sourceStart = sourceTextNavigator.getLineAndCharacterOfPosition(sourceStartPosition);
      const sourceEnd = sourceTextNavigator.getLineAndCharacterOfPosition(sourceEndPosition);

      sourceMap.addMapping(0, 0, sourceIndex, sourceStart.line, sourceStart.character);
      sourceMap.addMapping(0, compiledText.length, sourceIndex, sourceEnd.line, sourceEnd.character);

      // create source file
      return ts.createPreCompiledSourceFile(
        fileName,
        compiledText,
        languageVersion,
        getScriptKindByLang(scriptTagLang),
        setParentNodes,
        sourceMap.toString()
      );
    }

    function createInlineScriptSourceFile(
      fileName: string,
      sourceText: string,
      languageVersion: ts.ScriptTarget,
      setParentNodes: boolean | undefined,
      sourceStartPosition: number,
      scriptTagContent: string,
      scriptTagLang: string | undefined
    ): ts.PreCompilerSourceFile {
      const compiledText = sourceTagContent;

      // generate source map
      const sourceMap = sourceMapHost.createSourceMapGenerator(fileName);
      const sourceIndex = sourceMap.addSource(fileName);
      sourceMap.setSourceContent(sourceIndex, sourceText);

      const sourceTextNavigator = sourceMapHost.createSourceTextNavigator(sourceText);
      const compiledTextNavigator = sourceMapHost.createSourceTextNavigator(compiledText);

      const compiledLineStarts = compiledTextNavigator.getLineStarts();
      compiledLineStarts.forEach((compiledLineStart) => {
        // map line by line
        const sourceStart = sourceTextNavigator.getLineAndCharacterOfPosition(sourceStartPosition + compiledLineStart);
        const sourceEnd = sourceTextNavigator.getLineAndCharacterOfPosition(sourceTextNavigator.getLineEndOfPosition(sourceStartPosition + compiledLineStart));
        const compiledStart = compiledTextNavigator.getLineAndCharacterOfPosition(compiledLineStart);
        const compiledEnd = compiledTextNavigator.getLineAndCharacterOfPosition(compiledTextNavigator.getLineEndOfPosition(compiledLineStart));

        sourceMap.addMapping(compiledStart.line, compiledStart.character, sourceIndex, sourceStart.line, sourceStart.character);
        sourceMap.addMapping(compiledEnd.line, compiledEnd.character, sourceIndex, sourceEnd.line, sourceEnd.character);
      });

      // create source file
      return ts.createPreCompiledSourceFile(
        fileName,
        compiledText,
        languageVersion,
        getScriptKindByLang(scriptTagLang),
        setParentNodes,
        sourceMap.toString()
      );
    }


    return {
      extensions: ['.vue'],
      createPreCompiledSourceFile(fileName, sourceText, languageVersion, setParentNodes) {
        const { script } = compiler.parseComponent(sourceText, { pad: 'space' });
    
        if (!script) {
          // No <script> block
          return createNoScriptSourceFile(fileName, sourceText, languageVersion, setParentNodes);
        } else if (script.attrs.src) {
          // <script src="file.ts" /> block
          return createSrcScriptSourceFile(
            fileName, 
            sourceText, 
            languageVersion, 
            setParentNodes, 
            script.start, 
            script.end, 
            script.attrs.src, 
            script.attrs.lang
          );
        } else {
          // <script lang="ts"></script> block
          return createInlineScriptSourceFile(
            fileName, 
            sourceText, 
            languageVersion, 
            setParentNodes, 
            script.start,
            sourceText.slice(script.start, script.end), 
            script.attrs.lang
          );
        }
      }
    }
  }

  return {
    type: 'pre-compiler-plugin',
    create
  };
}

export = init;

Checklist

My suggestion meets these guidelines:

  • This wouldn't be a breaking change in existing TypeScript/JavaScript code
  • This wouldn't change the runtime behavior of existing JavaScript code
  • This could be implemented without emitting different JS based on the types of the expressions
  • This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, etc.)
  • This feature would agree with the rest of TypeScript's Design Goals.

Activity

MartinJohns commented on May 22, 2020

@MartinJohns
Contributor

Related: #16607

johnnyreilly commented on May 23, 2020

@johnnyreilly

cc Maël Nison (@arcanis) and Lenz Weber-Tronic (@phryneas) - figure this may be interesting to both of you 🥰

orta commented on Jul 22, 2020

@orta
Contributor

(This was to a now deleted comment, so I've edited it up a bit )

WRT this issue I wouldn't expect anything quickly in the short term, what plugin support looks like in TS is still a pretty constant internal debate with strong opinions on many sides of what support looks like. This is why it's shown up a few times on roadmapping but tended to never get past experimental internal demos.

That means we're also real cautious of talking about it because we still don't know what it'll look like, and how we can be sure it doesn't affect perf at the kind of scale that we'd expect to see people use plugins at.

piotr-oles commented on Aug 29, 2020

@piotr-oles
Author

Orta Therox (@orta)
Could you elaborate on what are the performance issues that you are afraid of?

If we assume that transformation to typescript format is a pure function, then we can easily cache it. In this situation, the overhead could be noticeable on the initial compilation, but not in the watch mode (as we rebuild one or a few files). Currently, users use some workarounds to make this work which probably is much slower than we could achieve with plugins. If TypeScript would provide plugins API, keeping the good performance of plugins would be a responsibility of plugin's maintainers - not the TypeScript team. In this context, I don't see performance blockers to move forward but I guess there are other factors that I didn't take into account 😄

piotr-oles commented on Oct 16, 2020

@piotr-oles
Author

Orta Therox (@orta) Do you have any update regarding this feature request? :)

orta commented on Oct 30, 2020

@orta
Contributor

Nope, no updates, other than the move to a node factory API #35282 was long considered a blocker around this

piotr-oles commented on Oct 31, 2020

@piotr-oles
Author

Thanks for the update. I suppose you want to use AST as a format that a plugin should produce. That makes sense - it will be faster. I was also thinking about that but I was afraid you will not want to make AST factories API public.

Is there anything I could do to move this forward? I could implement it if you are busy with other features :)

orta commented on Nov 3, 2020

@orta
Contributor

Afraid not, it is still undecided by the team if and what the extent of plugin support looks like in the TypeScript compiler - plugins systems are not normal for compilers.

piotr-oles commented on Nov 4, 2020

@piotr-oles
Author

Ok, that's a bummer 😞

plugins systems are not normal for compilers.

That's an interesting claim 🤔 https://babeljs.io/docs/en/plugins - babel is made of plugins. https://eslint.org/docs/developer-guide/working-with-plugins - eslint, which performs static analysis, also uses a plugins system. So I would say that at least in the JavaScript community, it's normal.

armano2 commented on Jan 18, 2021

@armano2

plugins systems are not normal for compilers.

your claim is not true as there is a lot of compilers that support plugins, most notable one is gcc

there is more, but this short list should be enough


Orta Therox (@orta) do you have any updates about compile time plugins? this feature was requested few years ago

orta commented on Jan 23, 2021

@orta
Contributor

No updates

dummdidumm commented on Jul 10, 2021

@dummdidumm

This would be very handy for Vue, Angular, Svelte. Right now all of these have implemented a TypeScript language service and a TypeScript plugin, all going different routes and patching different internal things because there's no blessed/official way to support this. The proposal seems good as it would provide a very low level and flexible API which would enable tricking TS into thinking file X is TS and providing a mechanism for mapping code positions back and forth. This in combination with the existing options of decorating the TS features would work out great I think.

kf6kjg commented on Jul 21, 2022

@kf6kjg

Additional use cases, analogous to the the use case already given for graphql in that the current solution is to use code generators, which is both annoying and prone to human error - aka I updated the spec, but forgot to re-run the codegen, so everything looks to be running correctly...

  • Automatically providing the types from an OpenAPI spec file.
  • Automatically providing the types from an AsyncAPI spec file.

And ditto for anything else that defines interfaces in non-TS files that could be converted to TS type specifications.

Bonus points if the plugins are able to find and override imports. That would allow the plugin to be added to files in a way that doesn't corrupt the global namespace. Something like the "node:" prefix that node uses:

import * as MyNamespace from "tsplugin:mytypeplugin";
import { MyType } from "tsplugin:mytypeplugin";

function myFunc(p as MyType): MyNamespace.MyOtherType {
  //
}

sashafirsov commented on Mar 7, 2024

@sashafirsov

Another use cases which would use such pre-compiler plugin:

  • Declaratice Custom Elements to read HTML and convert to typescript with custom elements definitions.
  • HTML embedded script code to treat embedded typescript

I am working on the DCE and have to replicate whole chain of IDE and type checking support by converting into TS plus other IDE-specific metadata. Hooking pre-compilation plugin into custom extension and conversion into TS during TSC and in language service would serve the purpose best.

OskarLebuda commented on Mar 22, 2024

@OskarLebuda

Orta Therox (@orta) do you have any informations about this proposal?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    In DiscussionNot yet reached consensusSuggestionAn idea for TypeScript

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions