TfidfTransformer
Transform a count matrix to a normalized tf or tf-idf representation.
Tf means term-frequency while tf-idf means term-frequency times inverse document-frequency. This is a common term weighting scheme in information retrieval, that has also found good use in document classification.
The goal of using tf-idf instead of the raw frequencies of occurrence of a token in a given document is to scale down the impact of tokens that occur very frequently in a given corpus and that are hence empirically less informative than features that occur in a small fraction of the training corpus.
The formula that is used to compute the tf-idf for a term t of a document d in a document set is tf-idf(t, d) = tf(t, d) * idf(t), and the idf is computed as idf(t) = log [ n / df(t) ] + 1 (if smooth\_idf=False), where n is the total number of documents in the document set and df(t) is the document frequency of t; the document frequency is the number of documents in the document set that contain the term t. The effect of adding “1” to the idf in the equation above is that terms with zero idf, i.e., terms that occur in all documents in a training set, will not be entirely ignored. (Note that the idf formula above differs from the standard textbook notation that defines the idf as idf(t) = log [ n / (df(t) + 1) ]).
If smooth\_idf=True (the default), the constant “1” is added to the numerator and denominator of the idf as if an extra document was seen containing every term in the collection exactly once, which prevents zero divisions: idf(t) = log [ (1 + n) / (1 + df(t)) ] + 1.
Furthermore, the formulas used to compute tf and idf depend on parameter settings that correspond to the SMART notation used in IR as follows:
Tf is “n” (natural) by default, “l” (logarithmic) when sublinear\_tf=True. Idf is “t” when use_idf is given, “n” (none) otherwise. Normalization is “c” (cosine) when norm='l2', “n” (none) when norm=None.
Read more in the User Guide.
Python Reference (opens in a new tab)
Constructors
constructor()
Signature
new TfidfTransformer(opts?: object): TfidfTransformer;Parameters
| Name | Type | Description |
|---|---|---|
opts? | object | - |
opts.norm? | "l1" | "l2" | Each output row will have unit norm, either: Default Value 'l2' |
opts.smooth_idf? | boolean | Smooth idf weights by adding one to document frequencies, as if an extra document was seen containing every term in the collection exactly once. Prevents zero divisions. Default Value true |
opts.sublinear_tf? | boolean | Apply sublinear tf scaling, i.e. replace tf with 1 + log(tf). Default Value false |
opts.use_idf? | boolean | Enable inverse-document-frequency reweighting. If false, idf(t) = 1. Default Value true |
Returns
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:35 (opens in a new tab)
Methods
dispose()
Disposes of the underlying Python resources.
Once dispose() is called, the instance is no longer usable.
Signature
dispose(): Promise<void>;Returns
Promise<void>
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:125 (opens in a new tab)
fit()
Learn the idf vector (global term weights).
Signature
fit(opts: object): Promise<any>;Parameters
| Name | Type | Description |
|---|---|---|
opts | object | - |
opts.X? | any | A matrix of term/token counts. |
opts.y? | any | This parameter is not needed to compute tf-idf. |
Returns
Promise<any>
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:142 (opens in a new tab)
fit_transform()
Fit to data, then transform it.
Fits transformer to X and y with optional parameters fit\_params and returns a transformed version of X.
Signature
fit_transform(opts: object): Promise<any[]>;Parameters
| Name | Type | Description |
|---|---|---|
opts | object | - |
opts.X? | ArrayLike[] | Input samples. |
opts.fit_params? | any | Additional fit parameters. |
opts.y? | ArrayLike | Target values (undefined for unsupervised transformations). |
Returns
Promise<any[]>
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:184 (opens in a new tab)
get_feature_names_out()
Get output feature names for transformation.
Signature
get_feature_names_out(opts: object): Promise<any>;Parameters
| Name | Type | Description |
|---|---|---|
opts | object | - |
opts.input_features? | any | Input features. |
Returns
Promise<any>
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:235 (opens in a new tab)
get_metadata_routing()
Get metadata routing of this object.
Please check User Guide on how the routing mechanism works.
Signature
get_metadata_routing(opts: object): Promise<any>;Parameters
| Name | Type | Description |
|---|---|---|
opts | object | - |
opts.routing? | any | A MetadataRequest encapsulating routing information. |
Returns
Promise<any>
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:275 (opens in a new tab)
init()
Initializes the underlying Python resources.
This instance is not usable until the Promise returned by init() resolves.
Signature
init(py: PythonBridge): Promise<void>;Parameters
| Name | Type |
|---|---|
py | PythonBridge |
Returns
Promise<void>
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:81 (opens in a new tab)
set_output()
Set output container.
See Introducing the set_output API for an example on how to use the API.
Signature
set_output(opts: object): Promise<any>;Parameters
| Name | Type | Description |
|---|---|---|
opts | object | - |
opts.transform? | "default" | "pandas" | Configure output of transform and fit\_transform. |
Returns
Promise<any>
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:314 (opens in a new tab)
set_transform_request()
Request metadata passed to the transform method.
Note that this method is only relevant if enable\_metadata\_routing=True (see sklearn.set\_config). Please see User Guide on how the routing mechanism works.
The options for each parameter are:
Signature
set_transform_request(opts: object): Promise<any>;Parameters
| Name | Type | Description |
|---|---|---|
opts | object | - |
opts.copy? | string | boolean | Metadata routing for copy parameter in transform. |
Returns
Promise<any>
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:353 (opens in a new tab)
transform()
Transform a count matrix to a tf or tf-idf representation.
Signature
transform(opts: object): Promise<any[]>;Parameters
| Name | Type | Description |
|---|---|---|
opts | object | - |
opts.X? | any | A matrix of term/token counts. |
opts.copy? | boolean | Whether to copy X and operate on the copy or perform in-place operations. Default Value true |
Returns
Promise<any[]>
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:390 (opens in a new tab)
Properties
_isDisposed
boolean=false
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:33 (opens in a new tab)
_isInitialized
boolean=false
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:32 (opens in a new tab)
_py
PythonBridge
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:31 (opens in a new tab)
id
string
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:28 (opens in a new tab)
opts
any
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:29 (opens in a new tab)
Accessors
feature_names_in_
Names of features seen during fit. Defined only when X has feature names that are all strings.
Signature
feature_names_in_(): Promise<ArrayLike>;Returns
Promise<ArrayLike>
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:459 (opens in a new tab)
n_features_in_
Number of features seen during fit.
Signature
n_features_in_(): Promise<number>;Returns
Promise<number>
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:432 (opens in a new tab)
py
Signature
py(): PythonBridge;Returns
PythonBridge
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:68 (opens in a new tab)
Signature
py(pythonBridge: PythonBridge): void;Parameters
| Name | Type |
|---|---|
pythonBridge | PythonBridge |
Returns
void
Defined in: generated/feature_extraction/text/TfidfTransformer.ts:72 (opens in a new tab)