DocumentationClassesTfidfTransformer

Class: TfidfTransformer

Transform a count matrix to a normalized tf or tf-idf representation.

Tf means term-frequency while tf-idf means term-frequency times inverse document-frequency. This is a common term weighting scheme in information retrieval, that has also found good use in document classification.

The goal of using tf-idf instead of the raw frequencies of occurrence of a token in a given document is to scale down the impact of tokens that occur very frequently in a given corpus and that are hence empirically less informative than features that occur in a small fraction of the training corpus.

The formula that is used to compute the tf-idf for a term t of a document d in a document set is tf-idf(t, d) = tf(t, d) * idf(t), and the idf is computed as idf(t) = log [ n / df(t) ] + 1 (if smooth_idf=False), where n is the total number of documents in the document set and df(t) is the document frequency of t; the document frequency is the number of documents in the document set that contain the term t. The effect of adding “1” to the idf in the equation above is that terms with zero idf, i.e., terms that occur in all documents in a training set, will not be entirely ignored. (Note that the idf formula above differs from the standard textbook notation that defines the idf as idf(t) = log [ n / (df(t) + 1) ]).

If smooth_idf=True (the default), the constant “1” is added to the numerator and denominator of the idf as if an extra document was seen containing every term in the collection exactly once, which prevents zero divisions: idf(t) = log [ (1 + n) / (1 + df(t)) ] + 1.

Furthermore, the formulas used to compute tf and idf depend on parameter settings that correspond to the SMART notation used in IR as follows:

Tf is “n” (natural) by default, “l” (logarithmic) when sublinear_tf=True. Idf is “t” when use_idf is given, “n” (none) otherwise. Normalization is “c” (cosine) when norm='l2', “n” (none) when norm=None.

Read more in the User Guide.

Python Reference

Constructors

new TfidfTransformer()

new TfidfTransformer(opts?): TfidfTransformer

Parameters

ParameterTypeDescription
opts?object-
opts.norm?"l1" | "l2"Each output row will have unit norm, either:
opts.smooth_idf?booleanSmooth idf weights by adding one to document frequencies, as if an extra document was seen containing every term in the collection exactly once. Prevents zero divisions.
opts.sublinear_tf?booleanApply sublinear tf scaling, i.e. replace tf with 1 + log(tf).
opts.use_idf?booleanEnable inverse-document-frequency reweighting. If false, idf(t) = 1.

Returns TfidfTransformer

Defined in generated/feature_extraction/text/TfidfTransformer.ts:35

Properties

PropertyTypeDefault valueDefined in
_isDisposedbooleanfalsegenerated/feature_extraction/text/TfidfTransformer.ts:33
_isInitializedbooleanfalsegenerated/feature_extraction/text/TfidfTransformer.ts:32
_pyPythonBridgeundefinedgenerated/feature_extraction/text/TfidfTransformer.ts:31
idstringundefinedgenerated/feature_extraction/text/TfidfTransformer.ts:28
optsanyundefinedgenerated/feature_extraction/text/TfidfTransformer.ts:29

Accessors

feature_names_in_

Get Signature

get feature_names_in_(): Promise<ArrayLike>

Names of features seen during fit. Defined only when X has feature names that are all strings.

Returns Promise<ArrayLike>

Defined in generated/feature_extraction/text/TfidfTransformer.ts:469


idf_

Get Signature

get idf_(): Promise<any[]>

The inverse document frequency (IDF) vector; only defined if use_idf is true.

Returns Promise<any[]>

Defined in generated/feature_extraction/text/TfidfTransformer.ts:417


n_features_in_

Get Signature

get n_features_in_(): Promise<number>

Number of features seen during fit.

Returns Promise<number>

Defined in generated/feature_extraction/text/TfidfTransformer.ts:442


py

Get Signature

get py(): PythonBridge

Returns PythonBridge

Set Signature

set py(pythonBridge): void

Parameters

ParameterType
pythonBridgePythonBridge

Returns void

Defined in generated/feature_extraction/text/TfidfTransformer.ts:68

Methods

dispose()

dispose(): Promise<void>

Disposes of the underlying Python resources.

Once dispose() is called, the instance is no longer usable.

Returns Promise<void>

Defined in generated/feature_extraction/text/TfidfTransformer.ts:122


fit()

fit(opts): Promise<any>

Learn the idf vector (global term weights).

Parameters

ParameterTypeDescription
optsobject-
opts.X?any[]A matrix of term/token counts.
opts.y?anyThis parameter is not needed to compute tf-idf.

Returns Promise<any>

Defined in generated/feature_extraction/text/TfidfTransformer.ts:139


fit_transform()

fit_transform(opts): Promise<any[]>

Fit to data, then transform it.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters

ParameterTypeDescription
optsobject-
opts.fit_params?anyAdditional fit parameters.
opts.X?ArrayLike[]Input samples.
opts.y?ArrayLikeTarget values (undefined for unsupervised transformations).

Returns Promise<any[]>

Defined in generated/feature_extraction/text/TfidfTransformer.ts:180


get_feature_names_out()

get_feature_names_out(opts): Promise<any>

Get output feature names for transformation.

Parameters

ParameterTypeDescription
optsobject-
opts.input_features?anyInput features.

Returns Promise<any>

Defined in generated/feature_extraction/text/TfidfTransformer.ts:226


get_metadata_routing()

get_metadata_routing(opts): Promise<any>

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Parameters

ParameterTypeDescription
optsobject-
opts.routing?anyA MetadataRequest encapsulating routing information.

Returns Promise<any>

Defined in generated/feature_extraction/text/TfidfTransformer.ts:264


init()

init(py): Promise<void>

Initializes the underlying Python resources.

This instance is not usable until the Promise returned by init() resolves.

Parameters

ParameterType
pyPythonBridge

Returns Promise<void>

Defined in generated/feature_extraction/text/TfidfTransformer.ts:81


set_output()

set_output(opts): Promise<any>

Set output container.

See Introducing the set_output API for an example on how to use the API.

Parameters

ParameterTypeDescription
optsobject-
opts.transform?"default" | "pandas" | "polars"Configure output of transform and fit_transform.

Returns Promise<any>

Defined in generated/feature_extraction/text/TfidfTransformer.ts:302


set_transform_request()

set_transform_request(opts): Promise<any>

Request metadata passed to the transform method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

Parameters

ParameterTypeDescription
optsobject-
opts.copy?string | booleanMetadata routing for copy parameter in transform.

Returns Promise<any>

Defined in generated/feature_extraction/text/TfidfTransformer.ts:340


transform()

transform(opts): Promise<any[]>

Transform a count matrix to a tf or tf-idf representation.

Parameters

ParameterTypeDescription
optsobject-
opts.copy?booleanWhether to copy X and operate on the copy or perform in-place operations. copy=False will only be effective with CSR sparse matrix.
opts.X?anyA matrix of term/token counts.

Returns Promise<any[]>

Defined in generated/feature_extraction/text/TfidfTransformer.ts:376