diff --git a/docs/api_reference.md b/docs/api_reference.md index cc5fa610..dcfa0d5c 100644 --- a/docs/api_reference.md +++ b/docs/api_reference.md @@ -109,103 +109,103 @@ Parameters are listed with their defaults as defined in the constructors. | `file_object` | `None` | An open binary file object. | | `file_name` | `None` | Override the file name. | -### `FileInput` +### `DetokenizeRequest` -`skyflow.vault.detect` — wrapper for a file passed to `DeidentifyFileRequest`. Provide one of: +`skyflow.vault.tokens` — passed to `vault().detokenize()`. | Parameter | Default | Description | |-----------|---------|-------------| -| `file` | `None` | An open binary file (`BufferedReader`). | -| `file_path` | `None` | Path to a file. | +| `data` | _(required)_ | List of `{token, redaction_type}` dicts to detokenize. See [`RedactionType`](#redactiontype). | +| `continue_on_error` | `False` | Continue despite per-token errors. | -### `DeidentifyTextRequest` +### `TokenizeRequest` -`skyflow.vault.detect` — passed to `detect().deidentify_text()`. +`skyflow.vault.tokens` — passed to `vault().tokenize()`. | Parameter | Default | Description | |-----------|---------|-------------| -| `text` | _(required)_ | Text to de-identify. | -| `entities` | `None` | Entity types to detect. See `DetectEntities`. | -| `allow_regex_list` | `None` | Regex patterns to always treat as detectable. | -| `restrict_regex_list` | `None` | Regex patterns to exclude from detection. | -| `token_format` | `None` | `TokenFormat` controlling token types per entity. | -| `transformations` | `None` | `Transformations` (e.g. date shifting). | +| `values` | _(required)_ | List of `{value, column_group}` dicts to tokenize. | -### `DeidentifyFileRequest` +### `DeleteRequest` -`skyflow.vault.detect` — passed to `detect().deidentify_file()`. +`skyflow.vault.data` — passed to `vault().delete()`. | Parameter | Default | Description | |-----------|---------|-------------| -| `file` | `None` | A `FileInput`. | -| `entities` | `None` | Entity types to detect. | -| `allow_regex_list` | `None` | Regex patterns to always treat as detectable. | -| `restrict_regex_list` | `None` | Regex patterns to exclude. | -| `token_format` | `None` | `TokenFormat` per entity. | -| `transformations` | `None` | `Transformations` (not supported for Documents/Images/PDFs). | -| `output_processed_image` | `None` | Include the processed image in output. | -| `output_ocr_text` | `None` | Include OCR text in the response. | -| `masking_method` | `None` | See [`MaskingMethod`](#maskingmethod). | -| `pixel_density` | `None` | Pixel density for PDF processing. | -| `max_resolution` | `None` | Max resolution for PDF processing. | -| `output_processed_audio` | `None` | Include processed audio. | -| `output_transcription` | `None` | See [`DetectOutputTranscriptions`](#detectoutputtranscriptions). | -| `bleep` | `None` | Audio bleep config. See [`Bleep`](#bleep). | -| `output_directory` | `None` | Directory to write the processed file. | -| `wait_time` | `None` | Max seconds to wait (≤ 64). | +| `table` | _(required)_ | Target table name. | +| `ids` | _(required)_ | List of Skyflow IDs to delete. | -### `DetokenizeRequest` +### `QueryRequest` -`skyflow.vault.tokens` — passed to `vault().detokenize()`. +`skyflow.vault.data` — passed to `vault().query()`. | Parameter | Default | Description | |-----------|---------|-------------| -| `data` | _(required)_ | List of `{token, redaction_type}` dicts to detokenize. See [`RedactionType`](#redactiontype). | -| `continue_on_error` | `False` | Continue despite per-token errors. | +| `query` | _(required)_ | The SQL query string to execute. | -### `TokenizeRequest` +### `DeidentifyStringRequest` -`skyflow.vault.tokens` — passed to `vault().tokenize()`. +`skyflow.vault.detect` — passed to `detect().deidentify_string()`. Set `configuration_id` or `configuration`. | Parameter | Default | Description | |-----------|---------|-------------| -| `values` | _(required)_ | List of `{value, column_group}` dicts to tokenize. | +| `text` | _(required)_ | Text to de-identify. Must be non-empty. | +| `configuration_id` | `None` | ID of an existing Detect configuration. | +| `configuration` | `None` | Inline configuration. See [`DetectConfiguration`](#detectconfiguration). | -### `DeleteRequest` +### `DeidentifyFileRequest` -`skyflow.vault.data` — passed to `vault().delete()`. +`skyflow.vault.detect` — passed to `detect().deidentify_file()`. Supply the file with `file_path`, `file`, or `data_source` and `value`. A local `file` or `file_path` can't be combined with `value`. | Parameter | Default | Description | |-----------|---------|-------------| -| `table` | _(required)_ | Target table name. | -| `ids` | _(required)_ | List of Skyflow IDs to delete. | +| `file_path` | `None` | Path to a local file. The SDK base64-encodes it. | +| `file` | `None` | A local file opened in binary mode. The SDK base64-encodes it. If you also set `file_path`, the SDK reads `file_path`. | +| `data_source` | `None` | Source of `value`. See [`DataSourceType`](#datasourcetype). Required when you don't pass `file` or `file_path`. | +| `value` | `None` | Base64 content, Skyflow ID, or presigned URL, matching `data_source`. | +| `data_format` | `None` | Format of the input file. See [`FileDataFormat`](#filedataformat). Inferred from the file extension for a local file when not set. | +| `configuration_id` | `None` | ID of an existing Detect configuration. | +| `configuration` | `None` | Inline configuration. See [`DetectConfiguration`](#detectconfiguration). | +| `poll_options` | `None` | Poll the run until it finishes or the polling limits are reached. See [`PollOptions`](#polloptions). Without it, the call returns after submitting the run. | +| `output_directory` | `None` | Directory for processed outputs of a local file. Requires `poll_options`. | -### `QueryRequest` +### `GetRunRequest` -`skyflow.vault.data` — passed to `vault().query()`. +`skyflow.vault.detect` — passed to `detect().get_run()`. | Parameter | Default | Description | |-----------|---------|-------------| -| `query` | _(required)_ | The SQL query string to execute. | +| `run_id` | _(required)_ | The `run_id` returned by `deidentify_file`. Must be non-empty. | + +### `ReidentifyStringRequest` + +`skyflow.vault.detect` — passed to `detect().reidentify_string()`. + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `text` | _(required)_ | De-identified text to re-identify. Must be non-empty. | +| `redaction_level` | `None` | List of [`RedactionLevel`](#redactionlevel) objects that set how each entity type displays. | -### `ReidentifyTextRequest` +### `ReidentifyFileRequest` -`skyflow.vault.detect` — passed to `detect().reidentify_text()`. +`skyflow.vault.detect` — passed to `detect().reidentify_file()`. | Parameter | Default | Description | |-----------|---------|-------------| -| `text` | _(required)_ | The redacted/de-identified text to re-identify. | -| `redacted_entities` | `None` | Entity types to keep redacted. See `DetectEntities`. | -| `masked_entities` | `None` | Entity types to mask. | -| `plain_text_entities` | `None` | Entity types to reveal as plain text. | +| `data_source` | `None` | Source of `value`. See [`DataSourceType`](#datasourcetype). Required. | +| `value` | `None` | Base64 content, Skyflow ID, or presigned URL, matching `data_source`. Required. | +| `data_format` | `None` | Format of the input file: `txt`, `csv`, `json`, `xlsx`, `docx`, `pptx`, `xml`, or `jsonl`. See [`FileDataFormat`](#filedataformat). | +| `redaction_level` | `None` | List of [`RedactionLevel`](#redactionlevel) objects that set how each entity type displays. | -### `GetDetectRunRequest` +### `CheckGuardrailsRequest` -`skyflow.vault.detect` — passed to `detect().get_detect_run()`. +`skyflow.vault.detect` — passed to `detect().check_guardrails()`. | Parameter | Default | Description | |-----------|---------|-------------| -| `run_id` | _(required)_ | The `run_id` returned by a prior `deidentify_file` call. | +| `text` | _(required)_ | Text to check. Must be non-empty. | +| `check_toxicity` | `None` | Set to `True` to check the text for toxicity. | +| `deny_topics` | `None` | List of topics to deny. | ### `InvokeConnectionRequest` @@ -225,7 +225,7 @@ Parameters are listed with their defaults as defined in the constructors. Every vault, token, connection, and Detect operation returns a typed response object. Each attribute below lists its type and meaning. Types use `| None` to mark attributes that may be absent. -> **The `errors` attribute** is common to most responses. It is `list[dict] | None` and is populated only on partial failure (for example when `continue_on_error=True`); it is `None` when there are no errors. Each error dict contains `request_index`, `request_id`, `error`, and `http_code`. The per-class tables below describe only the operation-specific attributes and refer back to this note for `errors`. +> **The `errors` attribute** is common to the vault, token, and connection responses. It is `list[dict] | None` and is populated only on partial failure (for example when `continue_on_error=True`); it is `None` when there are no errors. Each error dict contains `request_index`, `request_id`, `error`, and `http_code`. The per-class tables below describe only the operation-specific attributes and refer back to this note for `errors`. Detect responses don't have an `errors` attribute; a failed Detect call raises `SkyflowError`. ```python response = skyflow_client.vault('').insert(insert_request) @@ -315,47 +315,63 @@ print(response.errors) # None unless there was a partial failure | `metadata` | `dict` | Response metadata (for example `request_id`). Defaults to `{}`. | | `errors` | `list[dict] \| None` | See the note above. | -### `DeidentifyTextResponse` +### `DeidentifyStringResponse` -`skyflow.vault.detect` — returned by `detect().deidentify_text()`. +`skyflow.vault.detect` — returned by `detect().deidentify_string()`. | Attribute | Type | Description | |-----------|------|-------------| | `processed_text` | `str` | The de-identified text. | -| `entities` | `list[EntityInfo]` | Detected entities. See [`EntityInfo`](#entityinfo). | -| `word_count` | `int` | Word count of the input text. | -| `char_count` | `int` | Character count of the input text. | -| `errors` | `list \| None` | See the note above. | +| `entities` | `list[DetectedEntity]` | Detected entities. Defaults to `[]`. See [`DetectedEntity`](#detectedentity). | +| `metrics` | `Metrics \| None` | Processing metrics. See [`Metrics`](#metrics). | -### `ReidentifyTextResponse` +### `DeidentifyFileResponse` -`skyflow.vault.detect` — returned by `detect().reidentify_text()`. +`skyflow.vault.detect` — returned by `detect().deidentify_file()`. Without `poll_options`, only `run_id` is set, `status` is `None`, and `output` is `[]`. + +| Attribute | Type | Description | +|-----------|------|-------------| +| `run_id` | `str \| None` | Run identifier. Pass it to `get_run`. | +| `status` | `DetectRunStatus \| None` | Status of the run. See [`DetectRunStatus`](#detectrunstatus). | +| `output_type` | `DataSourceType \| None` | Form of the outputs. See [`DataSourceType`](#datasourcetype). | +| `output` | `list[FileOutput]` | Processed outputs. See [`FileOutput`](#fileoutput). | +| `metrics` | `Metrics \| None` | Processing metrics. See [`Metrics`](#metrics). | +| `message` | `str \| None` | Message returned for the run. | + +### `GetRunResponse` + +`skyflow.vault.detect` — returned by `detect().get_run()`. Has the same attributes as [`DeidentifyFileResponse`](#deidentifyfileresponse). A status the SDK doesn't recognize maps to `DetectRunStatus.UNKNOWN`. + +### `ReidentifyStringResponse` + +`skyflow.vault.detect` — returned by `detect().reidentify_string()`. | Attribute | Type | Description | |-----------|------|-------------| | `processed_text` | `str` | The re-identified text. | -| `errors` | `list \| None` | See the note above. | +| `metrics` | `Metrics \| None` | Processing metrics. See [`Metrics`](#metrics). | -### `DeidentifyFileResponse` +### `ReidentifyFileResponse` + +`skyflow.vault.detect` — returned by `detect().reidentify_file()`. + +| Attribute | Type | Description | +|-----------|------|-------------| +| `status` | `DetectRunStatus` | Status of the operation. See [`DetectRunStatus`](#detectrunstatus). | +| `output_type` | `DataSourceType \| None` | Form of the outputs. See [`DataSourceType`](#datasourcetype). | +| `output` | `list[FileOutput]` | Re-identified outputs. See [`FileOutput`](#fileoutput). | +| `metrics` | `Metrics \| None` | Processing metrics. See [`Metrics`](#metrics). | -`skyflow.vault.detect` — returned by `detect().deidentify_file()` and `detect().get_detect_run()`. All non-error attributes are optional (default `None`) and are populated based on the file type and processing status. If processing exceeds `wait_time`, only `run_id` and `status` are set; poll with `get_detect_run`. +### `CheckGuardrailsResponse` + +`skyflow.vault.detect` — returned by `detect().check_guardrails()`. | Attribute | Type | Description | |-----------|------|-------------| -| `file_base64` | `str \| None` | The processed file as a base64 string. | -| `file` | `File \| None` | The processed file wrapper. See [`File`](#file). | -| `type` | `str \| None` | MIME type of the processed file. | -| `extension` | `str \| None` | File extension of the processed file. | -| `word_count` | `int \| None` | Word count (text-bearing files). | -| `char_count` | `int \| None` | Character count (text-bearing files). | -| `size_in_kb` | `float \| None` | Size of the processed file in KB. | -| `duration_in_seconds` | `float \| None` | Duration in seconds (audio files). | -| `page_count` | `int \| None` | Page count (PDF/document files). | -| `slide_count` | `int \| None` | Slide count (presentation files). | -| `entities` | `list[EntityInfo]` | Detected entities. Defaults to `[]`. See [`EntityInfo`](#entityinfo). | -| `run_id` | `str \| None` | Run identifier; pass to `get_detect_run` to poll for results. | -| `status` | `str \| None` | Processing status of the run. | -| `errors` | `list \| None` | See the note above. | +| `text` | `str` | The text that was checked. | +| `validation` | `GuardrailsValidation` | Whether the text passed. See [`GuardrailsValidation`](#guardrailsvalidation). | +| `toxic` | `bool \| None` | Whether the text is toxic. | +| `denied_topic` | `bool \| None` | Whether the text includes a denied topic. | --- @@ -384,7 +400,7 @@ Vault hostnames per environment (used internally; exported for reference). ### `RedactionType` -How retrieved data is displayed. Values: `PLAIN_TEXT`, `MASKED`, `DEFAULT`, `REDACTED`. See [Redaction Types](../README.md#redaction-types). +How retrieved or re-identified data is displayed. Used by `GetRequest`, `DetokenizeRequest`, and [`RedactionLevel`](#redactionlevel). Values: `PLAIN_TEXT`, `MASKED`, `DEFAULT`, `REDACTED`. See [Redaction Types](../skyvault/README.md#redaction-types). ### `TokenMode` @@ -396,10 +412,6 @@ Bring-your-own-token mode for `InsertRequest`/`UpdateRequest`. | `ENABLE` | Accept caller-supplied tokens. | | `ENABLE_STRICT` | Accept caller-supplied tokens with strict validation. | -### `TokenType` - -Token format for Detect. Values: `VAULT_TOKEN` (`vault_token`), `ENTITY_UNIQUE_COUNTER` (`entity_unq_counter`), `ENTITY_ONLY` (`entity_only`). - ### `ContentType` Content type for connection requests. Values: `JSON`, `PLAINTEXT`, `XML`, `URLENCODED`, `FORMDATA`, `HTML`. @@ -410,67 +422,243 @@ HTTP method for connections. Values: `GET`, `POST`, `PUT`, `DELETE`, `NONE`. > Note: `PATCH` is **not** a member of this enum. -### `MaskingMethod` +### `EntityType` + +Entity types for Detect, used in [`Entity`](#entity) and [`RedactionLevel`](#redactionlevel). Examples: `NAME`, `EMAIL_ADDRESS`, `SSN`, `CREDIT_CARD`, `DOB`, `PHONE_NUMBER`, `ALL`. + +### `DeidentificationType` + +How Detect de-identifies an entity. Values: `VAULT_TOKEN`, `ENTITY_UNIQUE_COUNTER`, `ENTITY_ONLY`, `MASK`, `UNKNOWN`. `VAULT_TOKEN` supports only `txt`, `json`, `csv`, `xml`, `jsonl`, `docx`, `xlsx`, and `pptx` files. + +### `ReturnEntitiesType` + +Which detected entities a de-identify response includes. + +| Member | Meaning | +|--------|---------| +| `EXCLUDE_SENSITIVE_DATA` | Each entity's token, type, location, and scores, without the original value. | +| `ALL` | Each entity's metadata and the original value. | +| `NONE` | No entities. | + +### `DataSourceType` + +Source of a file `value`, and form of file outputs. Values: `BASE64`, `SKYFLOW_ID`, `PRESIGNED_URL`. + +### `FileDataFormat` + +File formats for Detect. Values: `BMP`, `CSV`, `DCM`, `DOC`, `DOCX`, `GIF`, `JPEG`, `JPG`, `JSON`, `JSONL`, `MP3`, `PDF`, `PNG`, `PPT`, `PPTX`, `TIF`, `TIFF`, `TXT`, `WAV`, `XLS`, `XLSX`, `XML`, `ZIP`. Each value is the lowercase extension, for example `FileDataFormat.PDF.value` is `pdf`. + +### `DetectRunStatus` + +Status of a Detect file run. Values: `QUEUED`, `IN_PROGRESS`, `SUCCESS`, `FAILED`, `UNKNOWN`. Polling stops at `SUCCESS`, `FAILED`, or `UNKNOWN`. -Image masking method for Detect file de-identification. Values: `BLACKBOX` (`blackbox`), `BLUR` (`blur`). +### `ObjectEntityType` -### `DetectOutputTranscriptions` +Object types for Detect to find. Values: `FACE`, `LICENSE_PLATE`, `LOGO`, `SIGNATURE`, `ALL`, `NONE`. -Audio transcription output type for Detect. Values: `DIARIZED_TRANSCRIPTION`, `MEDICAL_DIARIZED_TRANSCRIPTION`, `MEDICAL_TRANSCRIPTION`, `TRANSCRIPTION`, `PLAINTEXT_TRANSCRIPTION`. +### `ObjectDeidentificationType` -### `DetectEntities` +How Detect handles a detected object. Values: `REDACT`, `UNREDACT`, `UNKNOWN`. -Entity types Detect can identify (e.g. `SSN`, `CREDIT_CARD`, `NAME`, `DOB`). Import from `skyflow.utils.enums`. +### `OutputTranscriptionType` + +Audio transcription to include in the output. Values: `TRANSCRIPTION`, `MEDICAL_TRANSCRIPTION`, `DIARIZED_TRANSCRIPTION`, `MEDICAL_DIARIZED_TRANSCRIPTION`, `NONE`. + +### `PdfProcessingMode` + +How Detect processes a PDF. Values: `TEXT_LAYER` (process the PDF's text layer), `OCR` (process each page as an image, for scanned or image-based PDFs), `NONE`. + +### `MaskingMethod` + +How Detect masks entities in an image. Values: `BLUR`, `BLACKBOX`, `NONE`. + +### `GuardrailsValidation` + +Result of a guardrails check. Values: `PASSED`, `FAILED`, `UNKNOWN`. --- ## Detect helper classes -Importable from `skyflow.vault.detect`. +Importable from `skyflow.vault.detect`. Every parameter defaults to `None`. -### `EntityInfo` +### `DetectConfiguration` -A detected entity, returned inside `DeidentifyTextResponse.entities`. +An inline Detect configuration, passed as `configuration` to `deidentify_string` or `deidentify_file`. The SDK adds the vault ID from your vault configuration. -| Attribute | Type | Description | +| Parameter | Type | Description | |-----------|------|-------------| -| `token` | `str` | The token replacing the entity. | -| `value` | `str` | The original entity value. | -| `text_index` | `TextIndex` | Position in the input text. | -| `processed_index` | `TextIndex` | Position in the processed text. | -| `entity` | `str` | Entity type. | -| `scores` | `Dict[str, float]` | Confidence scores. | +| `name` | `str` | Name of the configuration. | +| `description` | `str` | Description of the configuration. | +| `detect` | `Detect` | Entity detection and de-identification settings. See [`Detect`](#detect). | +| `file_mapping` | `list[FileMapping]` | Source and destination locations for files. See [`FileMapping`](#filemapping). | +| `media` | `Media` | Options for audio, documents, and images. See [`Media`](#media). | -### `TextIndex` +### `Detect` -| Attribute | Type | Description | +| Parameter | Type | Description | +|-----------|------|-------------| +| `entities` | `list[Entity]` | Entity types to detect and how to de-identify each. See [`Entity`](#entity). | +| `object_entities` | `list[ObjectEntity]` | Object types to detect and how to handle each. See [`ObjectEntity`](#objectentity). | +| `restrict` | `list[str]` | Patterns for entities to replace with `[RESTRICTED]`. A pattern must match the entity's full value. | +| `skip` | `list[str]` | Patterns for entities to leave in plain text. A pattern can match the full value or a substring. `skip` takes precedence over `restrict`. | +| `custom_types` | `list[CustomType]` | Custom entity types. See [`CustomType`](#customtype). | +| `return_entities` | `ReturnEntitiesType` | Which detected entities the response includes. See [`ReturnEntitiesType`](#returnentitiestype). | + +### `Entity` + +| Parameter | Type | Description | |-----------|------|-------------| -| `start` | `int` | Start offset. | -| `end` | `int` | End offset. | +| `entity_type` | `EntityType` | Entity type to detect. See [`EntityType`](#entitytype). | +| `deidentification_type` | `DeidentificationType` | How to de-identify the entity. See [`DeidentificationType`](#deidentificationtype). | +| `destination` | `str` | Vault column that stores detected values, in `table.column` format (for example, `persons.name`). | +| `transformation` | `Transformation` | Transformation to apply. See [`Transformation`](#transformation). | + +### `Transformation` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `shift_dates` | `ShiftDates` | Shift dates within a range of days. See [`ShiftDates`](#shiftdates). | + +### `ShiftDates` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `min_days` | `int` | Minimum number of days to shift dates. | +| `max_days` | `int` | Maximum number of days to shift dates. | + +### `ObjectEntity` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `entity_type` | `ObjectEntityType` | Object to detect. See [`ObjectEntityType`](#objectentitytype). | +| `deidentification_type` | `ObjectDeidentificationType` | How to handle the object. See [`ObjectDeidentificationType`](#objectdeidentificationtype). | + +### `CustomType` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `label` | `str` | Label of the custom type, used as its token prefix. | +| `match` | `list[str]` | Regular expressions. A detected entity that a pattern matches in full is replaced with `[