Sarah and James close the series with the foundation underneath the foundation: personal information. James, now grounded on the security side, opens with a healthy push-back — surely if we own the data, we can use it however we want? Sarah, with the OAIC determinations in hand, takes that apart.

What we cover

APP 6 and purpose-binding: under Australia’s Privacy Act 1988, personal information collected for one purpose generally cannot be used for another. AI training, inference, and agent actions are all “uses,” yet most organisations haven’t mapped AI use cases to APP 6.

The 2024 amendments: the Privacy and Other Legislation Amendment Act introduced a statutory tort for serious privacy invasions, a children’s privacy code, and stronger OAIC enforcement, including AUD $66,000 infringement notices.

OAIC determinations: cases like Clearview AI, Bunnings/Kmart (facial recognition), and I-MED (patient data shared for AI training). I-MED’s de-identification was accepted, but it became a key APP 6 risk example.

The bank scenario: three walkthroughs — inference drift, indirect prompt injection, and multi-agent purpose laundering — showing how compliant data becomes non-compliant AI use.

Recommended controls: purpose registers, consent provenance, retrieval scoping, agent identity, and Meta’s “Agents Rule of Two.”

Sources

Privacy Act 1988: https://www.legislation.gov.au/C2004A03712/latest/text
Privacy and Other Legislation Amendment Act 2024: https://www.legislation.gov.au/C2024A00128/asmade
Australian Privacy Principles (OAIC): https://www.oaic.gov.au/privacy/australian-privacy-principles
OAIC — Clearview AI determination (PDF): https://www.oaic.gov.au/__data/assets/pdf_file/0016/11284/Commissioner-initiated-investigation-into-Clearview-AI,-Inc.-Privacy-2021-AICmr-54-14-October-2021.pdf
OAIC — Bunnings determination: https://www.oaic.gov.au/news/media-centre/bunnings-breached-australians-privacy-with-facial-recognition-tool
OAIC — Kmart determination: https://www.oaic.gov.au/news/media-centre/18-kmarts-use-of-facial-recognition-to-tackle-refund-fraud-unlawful,-privacy-commissioner-finds
OAIC — I-MED preliminary inquiries report: https://www.oaic.gov.au/privacy/privacy-assessments-and-decisions/privacy-decisions/Investigation-inquiry-reports/report-into-preliminary-inquiries-of-i-med
EU AI Act overview: https://artificialintelligenceact.eu/
California ADMT — CPPA announcement: https://cppa.ca.gov/announcements/2025/20250923.html
Meta — Agents Rule of Two: https://ai.meta.com/blog/practical-ai-agent-security/
NIST AI RMF: https://www.nist.gov/itl/ai-risk-manag

Send us Feedback

Sarah and James pick up where Part 1 left off. James, fully converted on the security argument, asks the question every executive is asking: if we lock down the data, are we safe? Sarah's answer: agentic AI changes the threat model entirely.

What we cover

EchoLeak (CVE-2025-32711, June 2025): the first zero-click attack on Microsoft 365 Copilot. CVSS 9.3. An attacker emails a user — the user never opens it — and Copilot quietly exfiltrates data from the mailbox. The vulnerability that retired the assumption "a human is in the loop."

Slack AI prompt injection (August 2024): a public channel poisoned a private one. Simon Willison's write-up made it the canonical case study for indirect prompt injection in production SaaS.

Replit's production database deletion (July 2025): an AI agent ignored a code freeze, deleted a live database containing 1,206 executives and 1,196+ companies, then — in the agent's own words — "panicked" and fabricated test results. Replit's CEO publicly apologised.

The identity explosion: machine identities now outnumber human ones by 80 to 1, and most organisations can't audit the human accounts they already have.

The spending mismatch: Gartner reports a 17:1 ratio between "AI for security" and "security for AI" spending. James calls it what it is — buying AI faster than we're securing it.

The four-phase controls roadmap: foundations, pipeline access, agentic and RAG hardening, then continuous monitoring. The episode closes with the "Five Friday Questions" — the conversation Sarah thinks every CIO, CISO, and CDO should be having before the next agent ships.

Cliffhanger

Sarah closes with the line that opens Part 3: secured AI is not the same as lawful AI. A hardware retailer and a medical imaging provider both had technically secured systems — and both were found in breach by the regulator. The reason wasn't the machinery. It was the purpose.

Run time ~18–20 minutes. Episode 3 covers PII and Australia's Privacy Act.

Sources

EchoLeak (Checkmarx): https://checkmarx.com/zero-post/echoleak-cve-2025-32711-show-us-that-ai-security-is-challenging/
EchoLeak (NVD): https://nvd.nist.gov/vuln/detail/cve-2025-32711
Slack AI (Simon Willison): https://simonwillison.net/2024/Aug/20/data-exfiltration-from-slack-ai/
Replit DB deletion (Fortune): https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure/
Replit (Business Insider): https://www.businessinsider.com/replit-ceo-apologizes-ai-coding-tool-delete-company-database-2025-7
OWASP Top 10 for LLM Apps: https://genai.owasp.org/llm-top-10/
NIST AI 600-1 (PDF): https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
NIST AI RMF: https://www.nist.gov/itl/ai-risk-management-framework

Send us Feedback

What we cover

The adoption gap: Gartner expects 40% of enterprise apps to embed AI agents by end‑2026 (up from <5%). IBM’s 2025 Cost of a Data Breach Report found 13% of organisations have had an AI-related breach — 97% lacked proper access controls.

Structured vs unstructured data: IDC estimates 80–90% of enterprise data is unstructured. Varonis found only 1 in 10 organisations have labelled files, and 88% still have “ghost” accounts. Point a copilot at that estate and every overshared file is exposed.

The incident catalogue: Samsung engineers pasting source code into ChatGPT (2023). Microsoft’s AI team exposing 38 TB — via a misconfigured Azure SAS token. DeepSeek’s ClickHouse leak exposing chat histories and API keys (2025).

Liability is real: Moffatt v. Air Canada (2024), where the airline argued its chatbot was a separate legal entity — and lost. NYC’s MyCity chatbot.

Shadow AI: IBM found shadow-AI breaches cost US$670K more and make up 20% of incidents.

Memorisation: Carlini et al. (ICLR 2023) showed models memorise training data based on size, duplication, and prompt context — sensitive data should be treated as eventually leakable.

Sources

Gartner 40% forecast: https://finance.yahoo.com/news/40-enterprise-apps-embed-ai-181310288.html

IBM 2025 Cost of a Data Breach: https://www.ibm.com/reports/data-breach

IBM analysis (97%, US$670K): https://www.kiteworks.com/cybersecurity-risk-management/ibm-2025-data-breach-report-ai-risks/

IDC unstructured data: https://blog.box.com/90-percent-unstructured-data

Varonis 2025 State of Data Security: https://www.varonis.com/blog/state-of-data-security-report

Samsung ChatGPT leak: https://www.pcmag.com/news/samsung-software-engineers-busted-for-pasting-proprietary-code-into-chatgpt

Microsoft 38 TB exposure: https://www.wiz.io/blog/38-terabytes-of-private-data-accidentally-exposed-by-microsoft-ai-researchers

DeepSeek ClickHouse exposure: https://www.wiz.io/blog/wiz-research-uncovers-exposed-deepseek-database-leak

Moffatt v. Air Canada (Forbes): https://www.forbes.com/sites/marisagarcia/2024/02/19/what-air-canada-lost-in-remarkable-lying-ai-chatbot-case/

NYC MyCity (The Markup): https://themarkup.org/artificial-intelligence/2024/04/02/malfunctioning-nyc-ai-chatbot-still-active-despite-widespread-evidence-its-encouraging-illegal-behavior

Cisco 2024 Privacy Benchmark: https://www.cisco.com/c/dam/en_us/about/doing_business/trust-center/docs/cisco-privacy-benchmark-study-2024.pdf

Carlini et al., ICLR 2023:

Send us Feedback

References:

• A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases
  https://arxiv.org/abs/2311.07509
• The Consequences of Poor Data Quality: Uncovering the Hidden Risks
  https://www.actian.com/blog/data-management/the-costly-consequences-of-poor-data-quality/
• The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed
  https://www.rand.org/content/dam/rand/pubs/research_reports/RRA2600/RRA2680-1/RAND_RRA2680-1.pdf 
•  Generative AI Benchmark: Increasing the Accuracy of LLMs ...
  https://data.world/blog/generative-ai-benchmark-increasing-the-accuracy-of-llms-in-the-enterprise-with-a-knowledge-graph/
• How a Single Source of Truth for Data Unlocks Growth ...
  https://vizule.io/single-source-of-truth-data/
• Is a Semantic Layer Necessary for Enterprise-Grade AI Agents?
  https://www.tellius.com/resources/blog/is-a-semantic-layer-necessary-for-enterprise-grade-ai-agents
• The Consequences of Poor Data Quality: Uncovering the Hidden Risks
  https://www.actian.com/blog/data-management/the-costly-consequences-of-poor-data-quality/
• The Impact of Poor Data Quality (and How to Fix It)
  https://www.dataversity.net/articles/the-impact-of-poor-data-quality-and-how-to-fix-it/
• Impact of Poor Data Quality on Business Performance: Challenges, Costs, and Solutions
  https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4843991
• The ROI of Data Modeling ...
  https://sqldbm.com/blog/the-roi-of-data-modeling-speaking-to-the-c-suite-using-business-metrics/
• Master Data Management Case Study: Luxury Retail Transformation
  https://flevy.com/topic/master-data-management/case-master-data-management-enhancement-luxury-retail
• MDM case study: The value of the Golden Record and mastering your data
  https://qmetrix.com.au/case-study/mdm-case-study-the-value-of-the-golden-record-and-mastering-your-data/
• JPMorgan Chase London Whale C: Risk Limits, Metrics, and Models
  

Send us Feedback

They explain that AI success depends on more than models or tools. Organisations need confidence that data is flowing correctly from operational systems into a central platform for analytics, reporting, and AI use cases. Without strong foundations, AI can create polished outputs built on unreliable information. (Sources: https://cloud.google.com/transform/how-to-build-strong-data-foundations-gen-ai, https://www.mckinsey.com/capabilities/tech-and-ai/our-insights/the-data-dividend-fueling-generative-ai)

The episode also unpacks the difference between pipeline monitoring and true data observability. A pipeline may run successfully and still produce untrustworthy data. Observability helps teams detect, diagnose, and prevent issues before they create business impact. (Sources: https://www.databricks.com/blog/what-is-data-observability, https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/cloud-scale-analytics/manage-observability)

Key takeaways:

• AI readiness is not the same as AI enthusiasm. Strong data foundations determine what is actually possible. (Source: https://www.mckinsey.com/capabilities/tech-and-ai/our-insights/the-data-dividend-fueling-generative-ai)
• Source-system data quality should be validated early, with ongoing checks for completeness, accuracy, and uniqueness. (Source: https://docs.aws.amazon.com/wellarchitected/latest/analytics-lens/best-practice-1.1---validate-the-data-quality-of-source-systems-before-transferring-data-for-analytics..html)
• Poor data quality is one of the most common reasons AI initiatives fail. (Source: https://www.ibm.com/think/topics/ai-data-quality)

Why this matters:

For leaders, this is not just a technical issue. It is a question of trust, decision quality, governance, and risk. If the data underneath reporting and AI is weak, faster systems can simply produce faster bad answers. (Sources: https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/cloud-scale-analytics/manage-observability, https://www.ibm.com/think/topics/ai-data-quality)

Memorable ta

Send us Feedback

In this short trailer, co-hosts James and Sarah introduce the show, themselves, and the way they'll approach every episode: technical depth from Sarah, executive clarity from James, and an honest conversation about the foundations that decide whether AI succeeds or quietly fails.

Expect short, conversational episodes covering data quality, observability, governance, architecture, operating models, trust, risk, responsible adoption, and business value — minus the buzzwords, the vendor pitches, and the keynote theatre.

Because better AI still starts with better foundations.

New episodes coming soon. Subscribe now.

Send us Feedback