Class: TfidfTransformer
Transform a count matrix to a normalized tf or tf-idf representation.
Tf means term-frequency while tf-idf means term-frequency times inverse document-frequency. This is a common term weighting scheme in information retrieval, that has also found good use in document classification.
The goal of using tf-idf instead of the raw frequencies of occurrence of a token in a given document is to scale down the impact of tokens that occur very frequently in a given corpus and that are hence empirically less informative than features that occur in a small fraction of the training corpus.
The formula that is used to compute the tf-idf for a term t of a document d in a document set is tf-idf(t, d) = tf(t, d) * idf(t), and the idf is computed as idf(t) = log [ n / df(t) ] + 1 (if smooth_idf=False
), where n is the total number of documents in the document set and df(t) is the document frequency of t; the document frequency is the number of documents in the document set that contain the term t. The effect of adding “1” to the idf in the equation above is that terms with zero idf, i.e., terms that occur in all documents in a training set, will not be entirely ignored. (Note that the idf formula above differs from the standard textbook notation that defines the idf as idf(t) = log [ n / (df(t) + 1) ]).
If smooth_idf=True
(the default), the constant “1” is added to the numerator and denominator of the idf as if an extra document was seen containing every term in the collection exactly once, which prevents zero divisions: idf(t) = log [ (1 + n) / (1 + df(t)) ] + 1.
Furthermore, the formulas used to compute tf and idf depend on parameter settings that correspond to the SMART notation used in IR as follows:
Tf is “n” (natural) by default, “l” (logarithmic) when sublinear_tf=True
. Idf is “t” when use_idf is given, “n” (none) otherwise. Normalization is “c” (cosine) when norm='l2'
, “n” (none) when norm=None
.
Read more in the User Guide.
Constructors
new TfidfTransformer()
new TfidfTransformer(
opts
?):TfidfTransformer
Parameters
Parameter | Type | Description |
---|---|---|
opts ? | object | - |
opts.norm ? | "l1" | "l2" | Each output row will have unit norm, either: |
opts.smooth_idf ? | boolean | Smooth idf weights by adding one to document frequencies, as if an extra document was seen containing every term in the collection exactly once. Prevents zero divisions. |
opts.sublinear_tf ? | boolean | Apply sublinear tf scaling, i.e. replace tf with 1 + log(tf). |
opts.use_idf ? | boolean | Enable inverse-document-frequency reweighting. If false , idf(t) = 1. |
Returns TfidfTransformer
Defined in generated/feature_extraction/text/TfidfTransformer.ts:35
Properties
Property | Type | Default value | Defined in |
---|---|---|---|
_isDisposed | boolean | false | generated/feature_extraction/text/TfidfTransformer.ts:33 |
_isInitialized | boolean | false | generated/feature_extraction/text/TfidfTransformer.ts:32 |
_py | PythonBridge | undefined | generated/feature_extraction/text/TfidfTransformer.ts:31 |
id | string | undefined | generated/feature_extraction/text/TfidfTransformer.ts:28 |
opts | any | undefined | generated/feature_extraction/text/TfidfTransformer.ts:29 |
Accessors
feature_names_in_
Get Signature
get feature_names_in_():
Promise
<ArrayLike
>
Names of features seen during fit. Defined only when X
has feature names that are all strings.
Returns Promise
<ArrayLike
>
Defined in generated/feature_extraction/text/TfidfTransformer.ts:469
idf_
Get Signature
get idf_():
Promise
<any
[]>
The inverse document frequency (IDF) vector; only defined if use_idf
is true
.
Returns Promise
<any
[]>
Defined in generated/feature_extraction/text/TfidfTransformer.ts:417
n_features_in_
Get Signature
get n_features_in_():
Promise
<number
>
Number of features seen during fit.
Returns Promise
<number
>
Defined in generated/feature_extraction/text/TfidfTransformer.ts:442
py
Get Signature
get py():
PythonBridge
Returns PythonBridge
Set Signature
set py(
pythonBridge
):void
Parameters
Parameter | Type |
---|---|
pythonBridge | PythonBridge |
Returns void
Defined in generated/feature_extraction/text/TfidfTransformer.ts:68
Methods
dispose()
dispose():
Promise
<void
>
Disposes of the underlying Python resources.
Once dispose()
is called, the instance is no longer usable.
Returns Promise
<void
>
Defined in generated/feature_extraction/text/TfidfTransformer.ts:122
fit()
fit(
opts
):Promise
<any
>
Learn the idf vector (global term weights).
Parameters
Parameter | Type | Description |
---|---|---|
opts | object | - |
opts.X ? | any [] | A matrix of term/token counts. |
opts.y ? | any | This parameter is not needed to compute tf-idf. |
Returns Promise
<any
>
Defined in generated/feature_extraction/text/TfidfTransformer.ts:139
fit_transform()
fit_transform(
opts
):Promise
<any
[]>
Fit to data, then transform it.
Fits transformer to X
and y
with optional parameters fit_params
and returns a transformed version of X
.
Parameters
Parameter | Type | Description |
---|---|---|
opts | object | - |
opts.fit_params ? | any | Additional fit parameters. |
opts.X ? | ArrayLike [] | Input samples. |
opts.y ? | ArrayLike | Target values (undefined for unsupervised transformations). |
Returns Promise
<any
[]>
Defined in generated/feature_extraction/text/TfidfTransformer.ts:180
get_feature_names_out()
get_feature_names_out(
opts
):Promise
<any
>
Get output feature names for transformation.
Parameters
Parameter | Type | Description |
---|---|---|
opts | object | - |
opts.input_features ? | any | Input features. |
Returns Promise
<any
>
Defined in generated/feature_extraction/text/TfidfTransformer.ts:226
get_metadata_routing()
get_metadata_routing(
opts
):Promise
<any
>
Get metadata routing of this object.
Please check User Guide on how the routing mechanism works.
Parameters
Parameter | Type | Description |
---|---|---|
opts | object | - |
opts.routing ? | any | A MetadataRequest encapsulating routing information. |
Returns Promise
<any
>
Defined in generated/feature_extraction/text/TfidfTransformer.ts:264
init()
init(
py
):Promise
<void
>
Initializes the underlying Python resources.
This instance is not usable until the Promise
returned by init()
resolves.
Parameters
Parameter | Type |
---|---|
py | PythonBridge |
Returns Promise
<void
>
Defined in generated/feature_extraction/text/TfidfTransformer.ts:81
set_output()
set_output(
opts
):Promise
<any
>
Set output container.
See Introducing the set_output API for an example on how to use the API.
Parameters
Parameter | Type | Description |
---|---|---|
opts | object | - |
opts.transform ? | "default" | "pandas" | "polars" | Configure output of transform and fit_transform . |
Returns Promise
<any
>
Defined in generated/feature_extraction/text/TfidfTransformer.ts:302
set_transform_request()
set_transform_request(
opts
):Promise
<any
>
Request metadata passed to the transform
method.
Note that this method is only relevant if enable_metadata_routing=True
(see sklearn.set_config
). Please see User Guide on how the routing mechanism works.
The options for each parameter are:
Parameters
Parameter | Type | Description |
---|---|---|
opts | object | - |
opts.copy ? | string | boolean | Metadata routing for copy parameter in transform . |
Returns Promise
<any
>
Defined in generated/feature_extraction/text/TfidfTransformer.ts:340
transform()
transform(
opts
):Promise
<any
[]>
Transform a count matrix to a tf or tf-idf representation.
Parameters
Parameter | Type | Description |
---|---|---|
opts | object | - |
opts.copy ? | boolean | Whether to copy X and operate on the copy or perform in-place operations. copy=False will only be effective with CSR sparse matrix. |
opts.X ? | any | A matrix of term/token counts. |
Returns Promise
<any
[]>
Defined in generated/feature_extraction/text/TfidfTransformer.ts:376