Skip to content

Latest commit

 

History

History

tokenizer

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

@jscpd/tokenizer

tokenizer is package from @jscpd used for convert programming code to list of tokens

Installation

npm install @jscpd/tokenizer --save

Usage

import {IOptions,  ITokensMap} from '@jscpd/core';
import {Tokenizer} from '@jscpd/tokenizer';

const tokenizer = new Tokenizer();
const options: IOptions = {};

const maps: ITokensMap[] = tokenizer.generateMaps('source_id', 'let a = "11"', 'javascript', options);

ga tracker

License

MIT © Andrey Kucherenko