Skip to content

Delete Brief Connect records

Draft documentation

This guide is a draft. Validate it against the approved release package and customer operating procedure before using it for a production operation.

Use the record deletion tool to permanently delete an approved set of Brief Connect records from the currently supported database, SQL replica, and SharePoint stores.

The operation has two runs:

  1. A dry run resolves and freezes the record scope in records.csv and inventories resources in resources.csv.
  2. A live run reads the frozen scope, deletes database resources, pauses for an operator checkpoint, and then permanently deletes SharePoint resources.

Deletion is permanent

The tool does not provide backup, restore, or undo. Obtain formal approval and confirm that required records have been archived before starting live deletion. Protect the dry-run output because records.csv controls what the live run deletes.

Before you begin

Confirm that an administrator has:

  • Installed the complete approved tool package on the Election Readiness tools virtual machine (VM).
  • Enabled the VM system-assigned managed identity and granted it access to the required Key Vault secrets.
  • Attached and configured the Brief Connect SQL user-assigned managed identity required by the SQL connection string.
  • Configured the Brief Connect Server App certificate and Microsoft Graph permissions required to discover and permanently delete SharePoint items.
  • Allowed network access to Key Vault, Azure DocumentDB, Azure SQL, Microsoft Graph, and SharePoint.
  • Granted you access to a protected, durable output location.

Before live deletion, arrange for the customer to take Brief Connect offline, drain relevant queues, and establish a quiet period. The tool does not perform or verify these actions.

Use PowerShell for all commands in this guide. Run the commands from the folder containing E2.BriefConnect.RecordDeletion.exe.

Create the run configuration

Create a configuration file that contains resource identifiers and Key Vault secret names only. Do not include passwords, certificates, access tokens, or connection strings.

The following example shows the supported Mongo mode:

{
  "keyVault": {
    "uri": "https://<vault>.vault.azure.net/",
    "certificateSecretName": "<server-app-certificate-secret>",
    "mongoConnectionStringSecretName": "<mongo-connection-secret>",
    "mongoDatabasePasswordSecretName": "<mongo-password-secret>",
    "sqlReplicaConnectionStringSecretName": "<sql-connection-secret>"
  },
  "identity": {
    "clientId": "<server-app-client-id>",
    "tenantId": "<tenant-id>"
  },
  "mongo": {
    "databaseName": "BriefConnect",
    "healthCheckCollectionName": "_bc_record_deletion_tool_healthcheck"
  },
  "sqlReplica": {
    "healthCheckTableName": "BcRecordDeletionToolHealthCheck"
  },
  "sharePoint": {
    "siteUrl": "https://<tenant>.sharepoint.com/sites/<site>"
  }
}

The Mongo connection-string secret must contain the literal <password> placeholder. The tool replaces it with the URL-escaped value from mongoDatabasePasswordSecretName. The SQL secret must contain the complete connection string, including its managed-identity authentication settings.

The following example shows the supported Mongo mode:

{
  "keyVault": {
    "uri": "https://<key-vault-name>.vault.azure.net/",
    "certificateSecretName": "<brief-connect-server-app-certificate-secret-name>",
    "storageConnectionStringSecretName": "<storage-account-connection-string-secret-name>",
    "cosmosConnectionStringSecretName": "<cosmos-connection-string-secret-name>"
  },
  "identity": {
    "clientId": "<brief-connect-server-app-client-id>",
    "tenantId": "<entra-tenant-id>"
  },
  "storage": {
    "blobServiceUri": "https://<storage-account-name>.blob.core.windows.net/",
    "tableServiceUri": "https://<storage-account-name>.table.core.windows.net/",
    "cosmosAccountEndpoint": "https://<cosmos-account-name>.documents.azure.com:443/",
    "cosmosDatabaseName": "BriefConnect"
  },
  "sharePoint": {
    "siteUrl": "https://<tenant>.sharepoint.com/sites/<site>"
  }
}

Create a selector file

Select records either by record ID or by stored record-type value. Use only one selector for a dry run.

To select individual records, create a CSV file with a RecordId header:

RecordId
CAB-000001
CAB-000002

To select all records with specified record types, create a CSV file with a RecordTypeId header:

RecordTypeId
Cabinet (CAB)
Ministerial (MIN)

Although the column is named RecordTypeId, each value must exactly match the RecordType value stored in Brief Connect. Confirm the expected values with your Brief Connect administrator.

Run the dry run

Create a new output folder for each deletion job. Do not reuse a folder from another dry run.

Run a dry run with a record-ID selector as follows:

.\E2.BriefConnect.RecordDeletion.exe `
  --config "C:\BCDeletion\config.json" `
  --output "D:\BCDeletionJobs\job-001" `
  --storage-mode mongo `
  --record-id-csv "C:\BCDeletion\selectors\record-ids.csv"

Run a dry run with a record-type selector as follows:

.\E2.BriefConnect.RecordDeletion.exe `
  --config "C:\BCDeletion\config.json" `
  --output "D:\BCDeletionJobs\job-001" `
  --storage-mode mongo `
  --record-type-csv "C:\BCDeletion\selectors\record-types.csv"

The dry run performs authenticated connectivity checks before discovery. These checks create and remove temporary sentinel data in Mongo, SQL, and SharePoint. SQL preflight can create the configured health-check table if it does not exist. The dry run does not delete selected customer records.

Require exit code 0 before considering the scope ready for review.

Review the frozen scope

The dry run writes three files to the job folder:

File Purpose
records.csv Frozen live-deletion scope and per-record phase status.
resources.csv Resources discovered for each resolved record.
diagnostics.log Append-only operational messages and exception details.

Review records.csv and confirm that:

  • Every approved record appears once with SelectionOutcome set to Resolved.
  • Every unexpected record is absent.
  • Each resolved record has the expected record type and SharePoint identifiers.
  • Every Duplicate or Unresolved row is understood and excluded from live deletion.
  • The resolved count matches the separately approved scope.

Review resources.csv and confirm that expected Mongo collections, SQL replica tables, GridFS files, SharePoint Document Sets, folders, and files appear. The current inventory covers only the stores listed in Current deletion coverage.

Review diagnostics.log for successful Key Vault, Mongo, SQL, and SharePoint preflight results and for any discovery errors.

Protect records.csv

Live deletion trusts a structurally valid records.csv; the current version does not cryptographically verify its contents or bind it to the configuration and environment. Do not edit the file. Restrict write access and preserve a protected copy of the complete job folder before live deletion.

Prepare for live deletion

Complete these checks immediately before starting the live run:

  • Confirm formal authorisation for the exact resolved record count.
  • Confirm the configuration still identifies the intended Key Vault, database, SQL replica, tenant, and SharePoint site.
  • Confirm the job folder is the reviewed dry-run folder and has not been reused or modified.
  • Confirm Brief Connect is offline and the required queues are drained.
  • Confirm no users or jobs can update or recreate records during the database phase.
  • Confirm the customer understands that the tool does not clear Personal Storage or caches.

Run live deletion

Run live deletion with the same configuration, storage mode, and output folder as the dry run. Do not pass a selector.

.\E2.BriefConnect.RecordDeletion.exe `
  --config "C:\BCDeletion\config.json" `
  --output "D:\BCDeletionJobs\job-001" `
  --storage-mode mongo `
  --live-deletion

The tool repeats its preflight checks and displays this confirmation prompt:

Type <count> to confirm irreversible deletion of <count> resolved record(s), or press Enter to abort:

Type the resolved record count as digits only. For example, type 125 for 125 resolved records. The current version does not accept DELETE 125 RECORDS.

The tool then processes all resolved records through the database phase. For each record, it attempts Mongo and SQL deletion and updates records.csv.

When the database phase ends, the tool displays its outcome counts and waits at this prompt:

Press Enter to continue to Phase 2 (SharePoint deletion)...

Review the database outcomes before pressing Enter. If any record is incomplete, follow your approved operating procedure before deciding whether to continue. The tool permits SharePoint deletion even when database records are incomplete.

Press Enter to permanently delete SharePoint resources. The tool deletes child files and folders before permanently deleting the Document Set root. It does not place these items in the SharePoint recycle bin.

Review the result

Require exit code 0 and review the final files.

In records.csv, confirm that every resolved row has:

  • DatabaseStatus set to Complete.
  • SharePointStatus set to Complete.
  • Overall Status set to Complete.

Review diagnostics.log for failures and preserve it with records.csv according to the customer's evidence-retention policy.

The tool refreshes resources.csv after each deletion phase. The final file lists resources that remain discoverable; it is not an append-only record of everything deleted.

Resume an incomplete deletion

If a run stops or finishes with incomplete records, do not run another dry run in the same folder. A dry run overwrites the progress-bearing CSV files.

Make a protected copy of the job folder, inspect records.csv and diagnostics.log, and correct the reported connectivity, permission, schema, or service issue. Then rerun the same live-deletion command:

.\E2.BriefConnect.RecordDeletion.exe `
  --config "C:\BCDeletion\config.json" `
  --output "D:\BCDeletionJobs\job-001" `
  --storage-mode mongo `
  --live-deletion

Enter the confirmation count again. The tool skips phases already marked Complete and retries phases marked Pending or Incomplete. Individual deletion operations tolerate resources that are already absent.

Avoid forced termination

The current version does not provide a graceful Ctrl+C handler and rewrites records.csv in place. Forced termination can leave the file incomplete or corrupt. If termination occurs, preserve the folder and verify that records.csv is complete and readable before attempting recovery.

Current deletion coverage

The current Mongo-mode implementation deletes matching data from:

  • reportingb
  • reporting
  • documentmetadata
  • RecordAuditLog
  • UserAccess
  • UserTasks
  • GroupAccess
  • UserSettings
  • BulkUpdateProcessedRecords
  • GridFS files whose filename equals the record ID
  • Corresponding Azure SQL replica tables
  • The SharePoint Document Set and its descendants
  • Personal Storage content

The tool does not currently delete or manage:

  • Brief Connect caches or queues
  • Power BI imported data or downloaded reports
  • Emails, exports, webhooks, external archives, or other external copies
  • Microsoft-managed audit logs, recovery copies, or SharePoint search-index entries

Interpret exit codes

Exit code Meaning
0 Help or the requested run completed successfully.
1 An unhandled failure occurred. Some live deletion may already have completed.
2 Command, configuration, output, or live-scope validation failed.
3 One or more preflight checks failed.
4 The selected storage implementation is unavailable.
5 Dry-run discovery failures or incomplete live records remain.
6 Live deletion was not confirmed with the exact resolved count.

Treat any nonzero live exit as a potentially partial deletion. Inspect records.csv and diagnostics.log; do not assume that no data changed.