Every compliance software demo looks the same: clean dashboards, a control library, a promise of exam readiness. The differences that matter only surface when you ask questions the demo script was not built for. These seven questions separate platforms that run a compliance program from platforms that store one.
Key Takeaways:
- The questions that expose weak platforms are about lineage, evidence, and change response — not features or dashboards
- A strong answer shows you the system doing the thing live; a weak answer describes a workflow your team would still perform manually
- Every question here maps to something an examiner will eventually ask you, so a vendor who cannot answer it is transferring the problem to your team
- Run the same questions as pass/fail tests in a structured pilot before signing
1. Can You Trace Every Control Back to the Exact Source Requirement?
Why it matters: Examiners increasingly ask not just "what do you do?" but "why do you do it?" A control with no traceable source is unjustifiable; a requirement with no linked control is a gap. Lineage is the difference between a compliance execution platform and a task tracker with a compliance skin.
A strong answer: The vendor clicks from a control to the specific obligation it satisfies, then to the exact provision — say, 31 CFR § 1020.210 — with the source text visible. The chain works in both directions.
A weak answer: "You can add a reference field to each control." A free-text citation nobody validates is documentation, not lineage.
2. When a Policy or Regulation Changes, How Do You Identify Every Affected Workflow?
Why it matters: Regulatory change is constant, and the dangerous failure is silent: a rule changes, and a workflow built on the old version keeps running unchanged. If impact analysis depends on a person reading the redline and remembering what depends on it, impact analysis does not exist.
A strong answer: The system versions sources, diffs changes, and lists every obligation, control, and work item downstream of the changed text — then routes updates to the owners of the affected work.
A weak answer: "We send regulatory update alerts." An alert feed tells you something changed; it does not tell you what breaks in your program because of it.
3. How Much Evidence Is Collected Manually Before an Exam?
Why it matters: This is the single best proxy for whether the platform runs the work or merely records intentions. If evidence is gathered in a pre-exam sprint, the platform was not in the path of the work.
A strong answer: "None — evidence is captured when each work item completes, so an exam response is assembled from records that already exist." The vendor should show a completed work item with its timestamped artifact attached.
A weak answer: "We have an evidence request module." A tool for chasing colleagues faster is an admission that evidence lives outside the system.
4. Can You Prove the Control Operated Every Required Period?
Why it matters: Examiners sample across the full review period, and gaps are what samples find. Proving operating effectiveness requires an execution record for every period — not the best three examples.
A strong answer: A per-control history view: every scheduled occurrence, its completion status, the performer, the date, and the artifact. Missed periods are visible and escalated when they happen, not discovered at exam time.
A weak answer: "Owners update the control status quarterly." A status field says someone claimed the control is fine; it is not a record that the work occurred.
5. How Do You Identify Which Vendors or Fintech Partners Are Missing Evidence?
Why it matters: Third-party oversight fails at the portfolio level, not the relationship level. The interagency guidance in OCC Bulletin 2023-17 expects ongoing monitoring across the relationship lifecycle, which means someone must see — at a glance — which of forty vendors is behind on which requirement.
A strong answer: A cross-portfolio view showing every third party against a common set of requirements, with overdue evidence flagged automatically.
A weak answer: "Each vendor has a folder." Folders answer "where do documents go?" — not "who is out of compliance right now?"
6. How Quickly Can You Respond to an Examiner Asking for Six Months of Support?
Why it matters: This is the moment the platform exists for. Ask the vendor to simulate it live: pick a control, request six months of operating evidence, and time the response.
A strong answer: Minutes. The system produces the source citation, the obligation, the control, six completion records with owners and dates, and the artifacts — as one package.
A weak answer: Any answer that begins with "your team would export..." If assembling the response is your team's job, you are buying storage, not readiness.
7. Does the Platform Create the Work, or Does Your Team Still Translate Documents into Tasks?
Why it matters: The most expensive step in compliance operations is translation: a human reads a regulation, a policy, or a review finding, and manually builds trackers, tasks, and calendar reminders from it. If the platform does not perform that translation, you have automated the filing cabinet and kept the manual labor.
A strong answer: The system extracts obligations from source documents and generates the recurring work — owners, cadences, evidence requirements — with your team reviewing and approving rather than authoring from scratch.
A weak answer: "You can create custom workflows." Configurability is fine; it is also the vendor telling you the translation work remains yours.
How to Run These Questions in a Pilot
Demos reward polish; pilots reward truth. Structure a short pilot around these seven questions as pass/fail tests: load one real policy and one real regulation, let the platform extract obligations, run two or three controls through actual cycles with your staff, then simulate an exam request and time the response. Score each question strong/weak in writing before the commercial conversation starts. Our guide to running a 30-day compliance platform pilot lays out the full protocol week by week.
How Canarie Answers These Questions
We publish this list knowing buyers will ask us the same seven questions, because they map directly to how Canarie is built: obligations extracted and versioned from sources, lineage from every control to the exact requirement, recurring work with owners and cadences, evidence captured at completion, and exam responses assembled from existing records. The honest way to evaluate any vendor — including us — is to make them show it live with your documents.
Put these seven questions to a real test →
Frequently Asked Questions
What is the most important question to ask a compliance software vendor?
Question six — how quickly can you respond to an examiner asking for six months of support — because it tests everything else at once. A fast, complete answer requires lineage, scheduled execution, and automatic evidence capture to already be working. If a vendor can pass that test live, the other capabilities are almost certainly present; if they cannot, no feature list compensates.
How do I evaluate compliance software without a long procurement process?
Run a time-boxed pilot with real materials instead of a feature-matrix comparison. Load one actual policy and one applicable regulation, let the system generate obligations and work, run two or three control cycles with your own staff, and end by simulating an examiner request. Thirty days is enough to score every question in this article as strong or weak.
Should compliance software requirements come from IT or from the compliance team?
The compliance team should own the functional requirements because they own the exam outcome; IT should own security, integration, and data requirements. Where evaluations go wrong is when the requirements document lists features (dashboards, reporting, workflow builder) rather than outcomes (prove any control operated for any period, identify every workflow affected by a rule change). Outcome-based requirements are what these seven questions encode.
What's the difference between a compliance workflow tool and a compliance execution platform?
A workflow tool executes tasks someone manually defines; a compliance execution platform derives the tasks from regulatory sources and maintains the chain of proof. The practical test is question seven: if your team must read documents and author the workflows themselves, the tool manages work but knows nothing about the requirements behind it, which means lineage, change impact analysis, and examiner-ready responses all remain manual.