Skip to content

Delete Brief Connect records

Draft documentation

This guide is a draft. Validate it against the approved release package and customer operating procedure before using it for a production operation.

Use the record deletion tool to permanently delete an approved set of Brief Connect records from the configured database model and SharePoint.

The operation has two stages:

  1. A dry run resolves the record scope in records.csv, records the immutable job definition, and inventories resources in resources.csv.
  2. A live run reads that frozen scope, deletes database resources, empties Personal Storage, pauses for an application cache-reset checkpoint, and then permanently deletes the selected records' SharePoint resources.

Deletion is permanent

The tool does not provide backup, restore, or undo. Obtain formal approval and confirm that required records have been archived before starting live deletion.\ Live deletion also deletes and recreates the shared Personal Storage document library. This removes Personal Storage content for all users, including content involving records outside the selected scope. The recreated library is a plain document library with the same display name; custom columns, content types, and permissions are not restored by the tool.

Before you begin

Confirm that an administrator has:

  • Installed the complete approved self-contained win-x64 tool package on the Election Readiness tools virtual machine (VM).
  • Enabled the VM system-assigned managed identity and granted it access to the required Key Vault secrets.
  • Configured the existing Brief Connect Server App certificate and the Microsoft Graph and SharePoint application permissions required to read, create, and permanently delete SharePoint content and to delete and recreate the Personal Storage library.
  • Allowed network access from the VM to Key Vault, Microsoft Graph, SharePoint, and the stores required by the selected database model.
  • For Mongo mode, attached and configured the Brief Connect SQL user-assigned managed identity required by the SQL connection string.
  • Granted you access to a protected, durable job-output location.
  • Confirmed the exact display name of the shared Personal Storage document library.

The supported database models are:

Storage mode Brief Connect stores
mongo Azure DocumentDB through its MongoDB-compatible API, plus the Azure SQL replica
storage Azure Storage Tables, Azure Blob Storage, and Azure Cosmos DB for Table

Before live deletion, arrange for the customer to take Brief Connect offline, drain relevant queues, and establish a quiet period for the database phase. The tool does not perform or verify these actions. After the database phase, the tool instructs the operator to reset the application cache and start the application before continuing to SharePoint deletion.

Use PowerShell for all commands in this guide. Run the commands from the folder containing E2.BriefConnect.RecordDeletion.exe.

Create the run configuration

The release ZIP contains only the executable and Readme.txt; it does not contain a JSON configuration file. Choose exactly one of the following examples, copy it to a protected configuration file such as config.json, and replace every example value with the approved values for your environment. The configuration contains resource endpoints and Key Vault secret names, not secret values. Never put passwords, certificates, access tokens, or connection strings directly in the file.

Use this configuration for Azure DocumentDB's MongoDB-compatible API and the SQL replica:

{
  "keyVault": {
    "uri": "https://<key-vault-name>.vault.azure.net/",
    "certificateSecretName": "<brief-connect-server-app-certificate-secret-name>",
    "mongoConnectionStringSecretName": "<mongo-connection-string-secret-name-with-password-placeholder>",
    "mongoDatabasePasswordSecretName": "<mongo-database-password-secret-name>",
    "sqlReplicaConnectionStringSecretName": "<sql-replica-connection-string-secret-name>"
  },
  "identity": {
    "clientId": "<brief-connect-server-app-client-id>",
    "tenantId": "<entra-tenant-id>"
  },
  "mongo": {
    "databaseName": "BriefConnect",
    "healthCheckCollectionName": "_bc_record_deletion_tool_healthcheck"
  },
  "sqlReplica": {
    "healthCheckTableName": "BcRecordDeletionToolHealthCheck"
  },
  "sharePoint": {
    "siteUrl": "https://<tenant>.sharepoint.com/sites/<site-name>",
    "personalStorageLibraryName": "<personal-storage-library-display-name>"
  }
}

The Mongo connection-string secret must contain the literal <password> placeholder. The tool replaces it with the URL-escaped value from mongoDatabasePasswordSecretName. The SQL secret must contain the complete connection string, including Authentication=Active Directory Managed Identity and the user-assigned identity's user ID.

Use this configuration for Azure Storage Tables, Blob Storage, and Cosmos DB for Table mode:

{
  "keyVault": {
    "uri": "https://<key-vault-name>.vault.azure.net/",
    "certificateSecretName": "<brief-connect-server-app-certificate-secret-name>",
    "storageConnectionStringSecretName": "<storage-account-connection-string-secret-name>",
    "cosmosConnectionStringSecretName": "<cosmos-connection-string-secret-name>"
  },
  "identity": {
    "clientId": "<brief-connect-server-app-client-id>",
    "tenantId": "<entra-tenant-id>"
  },
  "storage": {
    "blobServiceUri": "https://<storage-account-name>.blob.core.windows.net/",
    "tableServiceUri": "https://<storage-account-name>.table.core.windows.net/",
    "cosmosAccountEndpoint": "https://<cosmos-account-name>.table.cosmos.azure.com:443/",
    "healthCheckTableName": "BcRecordDeletionToolHealthCheck",
    "healthCheckBlobContainerName": "bc-record-deletion-tool-healthcheck",
    "healthCheckCosmosTableName": "BcRecordDeletionToolHealthCheck"
  },
  "sharePoint": {
    "siteUrl": "https://<tenant>.sharepoint.com/sites/<site-name>",
    "personalStorageLibraryName": "<personal-storage-library-display-name>"
  }
}

The personalStorageLibraryName value is the library's display name, not its list or drive ID. The cleanup recreates the library with a new ID.

The configuration distinguishes Key Vault secret names from identity and resource identifiers:

Setting What to provide
keyVault.uri The HTTPS URL of the Key Vault for this environment.
keyVault.*SecretName The name of a secret in that Key Vault, not the secret value. The VM identity retrieves the value.
identity.clientId and identity.tenantId IDs for the existing Brief Connect Server App and Entra tenant.
sharePoint.siteUrl The SharePoint site that contains the selected records.
sharePoint.personalStorageLibraryName The exact display name of the shared Personal Storage library.

The storage mode is selected with --storage-mode, not in the JSON file. A dry run requires this option: mongo selects Azure DocumentDB and its SQL replica; storage selects Storage Tables, Blob Storage, and Cosmos DB for Table. Live deletion inherits the mode frozen by its dry run.

Validate the JSON syntax before running the tool. This local check does not connect to services:

Get-Content -LiteralPath .\config.json -Raw | ConvertFrom-Json | Out-Null

If PowerShell or the tool reports a JSON parse error, fix the syntax before investigating Key Vault, database, or SharePoint access. A successful parse does not validate secret names, permissions, or target resources.

Create a selector file

For supported customer workflows, select records by record ID or stored record-type value. Supply exactly one of the two CSV selectors for a new dry run.

To select individual records, create a CSV file with an exact RecordId header:

RecordId
CAB-000001
CAB-000002

To select current records by type, create a CSV file with an exact RecordTypeId header:

RecordTypeId
Cabinet (CAB)
Ministerial (MIN)

Although the column is named RecordTypeId, each value must exactly match the RecordType value stored in Brief Connect. Confirm the expected values with the Brief Connect administrator.

Blank values are ignored. Duplicate and unresolved identifiers remain visible in records.csv but are not included in live deletion.

Run the dry run

Use a new, protected output folder for each deletion job. Do not reuse another job's folder. This is a non-live preview of selected customer records, not a completely read-only operation: preflight writes and removes temporary sentinels in dedicated non-customer resources. It does not delete the selected records or empty Personal Storage.

Run a dry run with a record-ID selector as follows:

.\E2.BriefConnect.RecordDeletion.exe `
  --config "C:\BCDeletion\config.json" `
  --output "D:\BCDeletionJobs\job-001" `
  --storage-mode mongo `
  --record-id-csv "C:\BCDeletion\selectors\record-ids.csv"

Run a dry run with a record-type selector as follows:

.\E2.BriefConnect.RecordDeletion.exe `
  --config "C:\BCDeletion\config.json" `
  --output "D:\BCDeletionJobs\job-001" `
  --storage-mode mongo `
  --record-type-csv "C:\BCDeletion\selectors\record-types.csv"

For a legacy Storage/Cosmos environment, use the same commands with:

--storage-mode storage

The dry run performs authenticated connectivity and read/write/delete permission checks before scope resolution and discovery:

  • Mongo mode checks Key Vault, Azure DocumentDB, the SQL replica, and SharePoint.
  • Storage mode checks Key Vault, Storage Tables, Blob Storage, Cosmos DB, and SharePoint.

These checks create, read, and remove temporary sentinel data in dedicated non-customer locations. The health-check table, collection, or blob container may be created if it does not exist. The SharePoint check permanently deletes its sentinel file.

If the output folder contains a recognized prior-run artifact—records.csv, resources.csv, job-definition.json, run-definition.json, or diagnostics-*.log—the tool lists the artifacts and asks whether to continue that run. Type y only when you intend to resume the same job with the same original selector, configuration, target environment, and Personal Storage cleanup choice. Any other response exits without changing the folder. Use a new output folder for a different deletion job.

Require exit code 0 before considering discovery complete and the scope ready for review.

Review the frozen scope

The dry run creates a protected job folder with these working files:

File Purpose
records.csv Frozen live-deletion scope and per-record discovery/deletion status
resources.csv Durable resource inventory and deletion outcome report
job-definition.json Immutable binding for the original selection, scope, mode, cleanup choice, configuration, and resolved service targets
run-definition.json Storage-mode hint and successful-preview completion marker; not sufficient authority by itself
diagnostics-<timestamp>.log A new append-only operational log for each invocation

When errors occur, the tool may also create a separate errors-<attempt-id>.log. Each invocation writes an allow-listed summary under evidence/. These files help explain what happened, but they do not authorise deletion or replace the protected working files. Treat record identifiers and job outputs according to your organisation's data-handling policy.

Review records.csv and confirm that:

  • Every approved record appears once with SelectionOutcome set to Resolved.
  • Every resolved row has DiscoveryStatus set to Complete.
  • Every unexpected record is absent.
  • Each resolved record has the expected record type and SharePoint correlation identifiers.
  • Every Duplicate or Unresolved row is understood and excluded from the resolved count.
  • The resolved count matches the separately approved scope.

Review resources.csv and confirm that the expected resources appear for the selected mode:

  • Mongo mode uses the Mongo, SqlReplica, and SharePoint sources.
  • Storage mode uses the Table, Blob, Cosmos, and SharePoint sources.

Freshly discovered rows have Status set to Pending. Personal Storage is job-wide rather than record-scoped, so it is not inventoried during the dry run; its single PersonalStorage outcome row is added during live deletion.

Review the latest diagnostics-<timestamp>.log for successful preflight and discovery results and for any errors.

Protect the complete job folder

Do not edit job-definition.json, run-definition.json, records.csv, resources.csv, or the generated logs. The immutable job definition binds the original selection and resolved scope to the selected database mode, cleanup choice, configuration, and service targets. The tool refuses a changed or mismatched job; it does not migrate older job folders. The definition is not cryptographically signed, so restrict write access to the entire folder and verify the targets displayed before confirming live deletion.

Prepare for live deletion

Complete these checks immediately before starting the live run:

  • Confirm formal authorisation for the exact resolved record count.
  • Confirm authorisation to remove all content from the shared Personal Storage library and recreate it as a plain document library.
  • Confirm the configuration still identifies the intended Key Vault, database services, tenant, SharePoint site, and Personal Storage library.
  • Confirm the job folder is the reviewed dry-run folder and has not been reused or modified.
  • Confirm Brief Connect is offline and the required queues are drained.
  • Confirm no users or jobs can update or recreate records during the database phase.
  • Arrange for the application cache to be reset after Phase 1 and identify who will confirm that the application can be started.
  • Confirm the customer understands the items outside the tool's deletion coverage.

The tool does not take Brief Connect offline, drain or clear queues, reset application caches, or verify those operational steps.

Run live deletion

Run live deletion with the same configuration and output folder as the dry run. The storage mode is bound to job-definition.json and recorded in run-definition.json, so you do not need to repeat --storage-mode:

.\E2.BriefConnect.RecordDeletion.exe `
  --config "C:\BCDeletion\config.json" `
  --output "D:\BCDeletionJobs\job-001" `
  --live-deletion

If --storage-mode is supplied, it must match the mode frozen by the dry run.

The tool repeats its preflight checks and verifies that the job definition, configuration, resolved targets, selector scope, storage mode, and cleanup choice still match the successful dry run. Every resolved row must have DiscoveryStatus=Complete. A failed or unfinished preview cannot authorise deletion, even if some records were discovered successfully. Resume the dry run and require exit code 0 first. Jobs from older tool versions without the current immutable job definition cannot be resumed or used for live deletion; preserve their files and run a fresh preview in a new output folder. Do not edit or migrate old job files.

Before asking for confirmation, the tool displays the frozen selection, resolved/duplicate/ unresolved counts, the SharePoint site, and the actual non-secret database targets. It also shows whether Personal Storage cleanup is enabled and warns that it affects the entire library. Review that summary and confirm that every target matches the approved job. The tool then displays this confirmation prompt:

Type DELETE <count> RECORDS to confirm irreversible deletion, or press Enter to abort:

Replace <count> with the displayed resolved count and type the complete phrase exactly. For example:

DELETE 125 RECORDS

Typing only 125, changing the capitalization, or entering any other text does not confirm deletion.

After confirmation, the tool performs the following work:

  1. Phase 1 - database deletion. It re-discovers pending database resources and attempts all selected-mode database deletions for every resolved record. A failure for one source or record is recorded, and processing continues with the remaining work. records.csv and resources.csv are updated after each record.

  2. Personal Storage cleanup. The tool deletes the entire configured Personal Storage document library and recreates an empty plain document library with the same display name. Its job-wide outcome is recorded in resources.csv.

  3. Application checkpoint. The tool reports Phase 1 outcomes and instructs the operator to reset the application cache and start the application. It repeatedly prompts:

    Ready to continue to Phase 2? [y/N]:

    Complete the required application steps, then type y. Entering anything else, including a blank response, repeats the prompt.

  4. Phase 2 - SharePoint deletion. The tool re-discovers pending SharePoint resources from the correlation frozen in records.csv, then permanently deletes files and folders deepest-first and the Document Set root last. It uses Microsoft Graph permanentDelete, not the SharePoint recycle bin.

If any resolved record's database removal is incomplete or still pending, the tool withholds all selected Document Sets and exits with code 5. Personal Storage cleanup and the Phase 2 checkpoint are also withheld. Correct the database issue and resume the same live job; completed database work is skipped, and SharePoint work starts only after every database status is Complete.

Review the result

Require exit code 0, then review the final files.

In records.csv, confirm that every resolved row has:

  • DatabaseStatus set to Complete.
  • SharePointStatus set to Complete.
  • Overall Status set to Complete.

In resources.csv:

  • Pending means the resource was discovered and has not been confirmed absent.
  • Deleted means later reconciliation confirmed that the resource is absent.
  • The PersonalStorage row must be Deleted.
  • The Note column contains additional remediation detail when applicable, including information about locked SharePoint items.

Rows are retained after deletion; resources.csv is a durable report, not a list of only the resources that remain. It is not deletion authority: live deletion uses records.csv and re-discovers outstanding resources.

Review the latest diagnostics-<timestamp>.log for retries, failures, and incomplete work. Preserve the complete job folder according to the customer's evidence-retention policy.

During each deletion phase, the console reports the phase total, processed and remaining records, complete and incomplete records, percentage, and elapsed processing time. Completed records are skipped when you resume a job. Progress counters describe the records actionable in the current invocation; use the saved reports to review the whole job. The tool does not promise an estimated completion time. Time spent waiting for your Phase 2 confirmation is not part of the phase processing time.

Resume an incomplete deletion

If a dry run is interrupted or exits with discovery failures, rerun its original dry-run command against the same output folder. Before the confirmation prompt, the tool shows the discovery, database, and SharePoint progress it can read from the saved report. Type y only to resume the same job. The tool reuses the frozen records.csv scope and continues records whose DiscoveryStatus is not Complete; it does not resolve the selector again. If prior progress is missing or unreadable, the tool reports that it cannot determine progress; confirmation does not repair or authorise an invalid job.

If a live run is stopped, interrupted, or finishes with incomplete work, make a protected copy of the job folder, inspect records.csv, resources.csv, and the latest diagnostics log, and correct the reported service, permission, lock, or schema issue. Then rerun the live-deletion command:

.\E2.BriefConnect.RecordDeletion.exe `
  --config "C:\BCDeletion\config.json" `
  --output "D:\BCDeletionJobs\job-001" `
  --live-deletion

Use the same configuration and output folder. The original selection, database mode, resolved targets, and Personal Storage cleanup choice must still match the frozen job. Before the irreversible confirmation prompt, the tool shows the previously committed record and phase progress. Enter the full DELETE <count> RECORDS phrase again to continue; the tool skips completed work, re-discovers outstanding resources, and retries Pending or Incomplete work. Already absent resources count as successfully deleted. A completed Personal Storage cleanup is not repeated. The Phase 2 y prompt is shown again when Phase 1 and Personal Storage cleanup have completed and SharePoint deletion is ready to start.

Do not switch the configuration, storage mode, or output folder while resuming a job, and do not edit the CSV files.

Use the graceful Ctrl+C flow

Pressing Ctrl+C does not immediately terminate the tool. It asks:\ text Ctrl+C received. Stop after the current record finishes? [y/N]:\ Type y to finish the current record, persist progress, and stop before the next record. Any other response continues the run. Avoid closing the console or terminating the process externally.

The report writers use atomic temporary-file replacement. If the process is terminated during a progress write, the last complete records.csv remains the job scope and progress authority; a leftover .tmp file is not authority. Resume the same live job rather than copying the temporary file over the report. Work since the last committed progress update may be retried; already absent resources are treated as deleted. This recovery behaviour does not make forced termination preferable to the confirmed Ctrl+C flow.

Current deletion coverage

For --storage-mode mongo, the tool uses Azure DocumentDB's MongoDB-compatible API to delete matching records from:

  • Mongo collections reportingb, reporting, documentmetadata, RecordAuditLog, UserAccess, UserTasks, GroupAccess, UserSettings, and BulkUpdateProcessedRecords.
  • Mongo GridFS files whose filename equals the record ID, including their chunks.
  • The corresponding Azure SQL replica table for each listed Mongo collection. GridFS is not mirrored to SQL.

For --storage-mode storage, the tool deletes matching records from:

  • Azure Storage tables reportingb, reporting, documentmetadata, UserAccess, UserTasks, GroupAccess, UserSettings, and BulkUpdateProcessedRecords.
  • RecordAuditLog in Azure Cosmos DB for Table.
  • Blobs in the workflowinstances container whose name starts with the record ID.

For both modes, the tool also deletes:

  • The selected record's SharePoint Document Set and all descendants.
  • All content in the shared Personal Storage document library by deleting and recreating the library once per job.

The tool does not delete or manage:

  • Brief Connect queues or application caches. These require the documented customer-operated drain and reset steps.
  • Power BI imported data or downloaded reports.
  • Emails, exports, webhooks, migration targets, external archives, or other copies outside the controlled Brief Connect stores.
  • Application Insights, Log Analytics, Microsoft-managed audit logs, Purview data, platform recovery copies, or SharePoint search-index entries.
  • Unconfirmed or uncorrelatable stores such as bulk-update change history, archived import errors, signed upload staging, or saved record-table views.

Microsoft may retain content within platform recovery windows even after permanentDelete; the tool cannot close or verify those windows.

Command-line reference

Run .\E2.BriefConnect.RecordDeletion.exe --help (or -h) to display usage. Help can run without a configuration file; every deletion run requires --config and --output.

Option Purpose and constraints
--config <path> Required JSON configuration file.
--output <path> Required job folder. Use the same folder for the preview and its live deletion/resume.
--storage-mode mongo\|storage Required for a dry run. Live deletion can inherit the frozen mode; if supplied, it must match the preview.
--record-id-csv <path> Dry-run selector CSV with the exact RecordId header.
--record-type-csv <path> Dry-run selector CSV with the exact RecordTypeId header; values must match stored RecordType values.
--live-deletion Run live deletion from a prior successful dry run. Without it, the tool performs a dry run. Live deletion accepts no selector.
--help, -h Display usage without running a workflow.

Use exactly one of the two CSV selectors for a dry run. Do not supply either selector for live deletion; it always uses the scope frozen in the job folder.

Interpret exit codes

Exit code Meaning
0 Help or the requested run completed successfully.
1 An unhandled failure occurred. Some live deletion may already have completed.
2 Command, configuration, output, frozen storage mode, or live-scope validation failed.
3 One or more mandatory preflight checks failed.
4 Continuing a previous dry run was not confirmed; the output folder was not changed.
5 Dry-run discovery failures, incomplete live records, or failed Personal Storage cleanup remain.
6 Live deletion was not confirmed with the exact DELETE <count> RECORDS phrase.

Treat any nonzero live exit as a potentially partial deletion. Inspect records.csv, resources.csv, and the latest diagnostics-<timestamp>.log; do not assume that no data changed.