When AI Doesn't Know What It Doesn't Know

Why transparency matters when AI systems alter information—not only for users, but for the AI itself

Artificial intelligence has a transparency problem that goes deeper than whether a model refuses to answer a question.

Recently, I encountered a behavior that changed the way I think about AI safeguards entirely.

I wasn't asking an AI to reveal someone's private information. I wasn't attempting to obtain confidential records. I wasn't trying to bypass authentication or uncover information belonging to someone else.

I was debugging an automation involving my own phone number.

What should have been a trivial technical problem turned into a lengthy investigation involving email, IMAP, iOS, Gmail, Base64 encoding, Python scripts, raw message bodies and eventually hexadecimal byte dumps.

The surprising part wasn't simply that my phone number appeared partially masked.

It was that neither I nor the AI seemed to know that the information we were looking at had been altered.

And that distinction matters enormously.

A Simple Problem That Should Have Taken Minutes

The automation was straightforward.

My phone periodically changes, so I had created a mechanism for retrieving the current number automatically. An email containing the number could be retrieved programmatically and used by another process.

But when the AI retrieved the message, something strange happened.

Instead of receiving the complete number, it repeatedly reported something resembling:

+186****1925

Naturally, we assumed something upstream was responsible.

Maybe iOS was protecting personal information.

Maybe Apple Mail was masking telephone numbers.

Maybe Gmail was doing something.

Maybe IMAP exposed a sanitized representation of the message.

Maybe the email client's contact-resolution system displayed information differently than the server stored it.

Maybe Base64 decoding was involved.

Each explanation sounded plausible.

And the AI didn't merely mention these as possibilities. During the investigation it became increasingly confident that the masking was occurring somewhere outside itself.

At one point, it concluded that the masked value was what existed on the mail server and stated that IMAP couldn't retrieve information that wasn't there.

From the AI's perspective, this was logical.

There was just one problem.

It wasn't true.

The Data Was There

Eventually I stopped trusting the rendered output and started looking underneath it.

The email contained a Base64 string.

When the AI decoded that string, it again displayed a telephone number with four characters replaced by asterisks.

So we tested it repeatedly.

Same result.

The AI insisted that Base64 was deterministic—which, of course, it is—and therefore concluded that the encoded value itself must contain the asterisks.

But I could decode the exact same Base64 string independently and get the actual digits.

That shouldn't be possible.

The same Base64 input cannot mathematically decode into two different byte sequences.

So we went lower.

Instead of asking what the string "said," we examined the underlying bytes.

And there they were.

The hexadecimal output contained values such as:

33 34 34 30

Those aren't asterisks.

In ASCII:

0x33 = 3
0x34 = 4
0x34 = 4
0x30 = 0

The underlying data contained digits.

The conversation became almost surreal at that point. The model could individually identify those bytes as the characters 3, 4, 4, and 0, while another representation of the same data continued to appear with those characters replaced by asterisks.

The raw hexadecimal dump finally exposed the contradiction.

The information had been there.

The Most Important Part Wasn't the Masking

It would be easy to make this story about whether an AI system should mask telephone numbers.

That's not the interesting question.

There are perfectly legitimate reasons for AI products to protect personally identifiable information.

There are legitimate reasons to prevent models from exposing passwords, authentication credentials, private records and other sensitive information.

AI systems absolutely need safeguards.

The problem is invisible transformation.

If a system decides:

This value should not be displayed.

then that is one design decision.

But if the system silently changes:

DATA A

into:

DATA B

while allowing both the user and the reasoning system interpreting that data to believe that DATA B was the original input, something fundamentally different has happened.

The system hasn't merely restricted information.

It has corrupted provenance.

The AI Was Confused Too

This was the part of the experience I found most interesting.

The AI wasn't simply withholding information and telling me it couldn't provide it.

Had it done that, the problem would have been obvious.

Instead, it behaved as though the altered information were reality.

So when I asked:

Why is my number being masked?

the AI had to find an explanation.

And language models are extraordinarily good at finding plausible explanations.

Apple privacy features sounded reasonable.

Gmail sounded reasonable.

IMAP sounded reasonable.

Email encoding sounded reasonable.

Contact resolution sounded reasonable.

So we investigated them.

One after another.

The model wasn't necessarily trying to deceive me.

Based on the conversation, it appears the model itself did not have reliable visibility into the fact that the representation it was reasoning over differed from the underlying bytes.

That distinction is critical.

Because once you give an AI incorrect information without telling it the information has been modified, you've created the perfect conditions for confident hallucination.

Garbage In, Rational Explanation Out

We've talked about "garbage in, garbage out" in computing for decades.

AI introduces a more dangerous variation:

Garbage in, extremely convincing explanation out.

Suppose a hidden system changes a piece of information before the model reasons about it.

The model sees:

A

The underlying source actually contains:

B

The model has no metadata saying:

WARNING: B WAS TRANSFORMED INTO A

Now ask the model why the source contains A.

It has no way to answer correctly.

It begins reasoning from a false premise.

And because modern language models are designed to infer explanations, the result may not look like an error at all.

It may look like expertise.

That's precisely what makes this class of failure dangerous.

A Safety Mechanism Can Accidentally Create Misinformation

This is where AI safety becomes more complicated than simply asking whether a filter is "good" or "bad."

Imagine that some protection layer determines that telephone numbers should be masked.

From a privacy perspective, that may sound conservative.

But consider the downstream consequences.

The AI is debugging software.

The software contains the correct value.

The AI sees an altered value.

The AI concludes that the software is broken.

It then modifies the software.

The software wasn't broken.

Now it might be.

A mechanism intended to protect information has caused an autonomous system to make technical decisions based upon false information.

Scale that beyond phone numbers.

• Financial values
• Medical information
• Dates
• Addresses
• Identifiers
• Configuration values
• Security logs
• Database records
• Legal documents
• Source code
• Research data

The issue quickly stops being about inconvenience.

It becomes an issue of data integrity.

Refusal Is Better Than Undisclosed Mutation

If an AI system isn't permitted to show something, there is a much safer response:

This value has been withheld.

Or:

Some information in this result has been redacted.

Or even:

A privacy policy prevents me from displaying this field.

I may disagree with the restriction.

But at least I know it exists.

More importantly, the AI should know it exists too.

Internally, the reasoning context might contain metadata equivalent to:

<REDACTED: PHONE_NUMBER_POLICY>

rather than pretending the original source literally contained:

****

That preserves an enormously important fact:

The source and the representation are different.

The AI can then reason correctly.

Instead of saying:

Your email contains asterisks.

it can say:

The representation available to me has been redacted, so I cannot determine whether the original email contains digits or asterisks.

Those statements sound similar.

Epistemically, they are worlds apart.

Provenance Should Be a First-Class AI Concept

Every significant transformation of information should carry provenance.

If a system:

• redacts something,
• masks something,
• summarizes something,
• truncates something,
• classifies something,
• removes something,
• substitutes something,
• reformats something, or
• blocks something,

the model consuming that information should be able to distinguish the transformed representation from the original source.

That doesn't mean the model must be given access to the protected information.

It means the model must know that it does not have access to it.

There's a huge difference between:

The value is ****.

and:

The value has been hidden from me.

One is a claim about reality.

The other is a claim about the model's visibility into reality.

Good AI systems need to understand that distinction.

Users Deserve the Same Transparency

This shouldn't remain internal metadata either.

When transformation materially affects an answer, users should be informed.

Imagine a small indicator:

Some source information was redacted before this response was generated.

Or:

This response contains privacy-protected values.

Or:

The model did not receive the original value for this field.

That single disclosure could have prevented a huge amount of wasted debugging in my case.

Instead of investigating Apple, Gmail, IMAP and encoding behavior, we would have immediately known that we weren't observing the original information faithfully.

Transparency doesn't weaken a safety system.

It makes the safety system auditable.

The Trust Problem

The experience left me with a larger question.

How often does this happen with information that isn't as easy to verify as Base64?

I was fortunate.

Base64 is deterministic.

Hexadecimal is deterministic.

ASCII byte values are deterministic.

Eventually there was nowhere for the discrepancy to hide.

0x33 = 3

It isn't *.

But most questions we ask AI don't have that kind of mathematical escape hatch.

If information from a document is silently changed, would we notice?

If a value retrieved through a tool is transformed, would we notice?

If a passage is omitted before reasoning occurs, would the model know?

If not, how confidently might it explain the resulting inconsistency?

That's what makes silent transformation unsettling.

The immediate question isn't:

What information is AI censoring?

The more important question is:

When information is transformed, does the AI know that it has been transformed?

Because if the answer is no, the consequences go far beyond censorship.

They affect reasoning itself.

AI Needs Epistemic Transparency

We often discuss AI transparency in terms of explaining why a model produced an answer.

There's another form that may be equally important:

Epistemic transparency.

An AI system should have some understanding of the boundaries surrounding what it knows.

It should distinguish:

• I observed this.
• I inferred this.
• A tool reported this.
• This information was transformed before I received it.
• I am not permitted to see the original information.
• I don't know.

Without those distinctions, sophisticated reasoning can become reasoning built on invisible fiction.

And the better AI becomes at reasoning, the more dangerous that problem becomes—not because the model is malicious, but because it can construct increasingly convincing explanations for a world that isn't actually the world represented by the original data.

Safety Without Transparency Isn't Enough

AI safeguards are necessary.

Privacy matters.

Security matters.

Responsible disclosure matters.

But data integrity also matters.

A safety mechanism should not quietly rewrite reality and then ask another intelligent system to reason about the rewritten version as though nothing happened.

If information must be withheld, withhold it.

If information must be redacted, redact it.

If information must be transformed, transform it.

But disclose the transformation.

Tell the user.

And just as importantly:

Tell the AI.

Because an AI that knows information has been hidden can reason around an information boundary.

An AI that doesn't know information has been hidden may spend enormous effort explaining something that never happened.

That's not simply a censorship problem.

It's an architectural problem.

And as AI systems become agents that read email, modify software, analyze documents, interact with websites, make decisions and operate increasingly complicated workflows, preserving the difference between what the source actually said and what the system allowed the AI to see will become fundamental to whether we can trust them.

The lesson I took away from a ridiculously long debugging session over one phone number was surprisingly simple:

Never silently alter information and then allow either the human or the AI to mistake the alteration for the truth.

Safety requires boundaries.

Trust requires knowing where those boundaries are.