← All news

Engineering 10 min read

We let our AI save memories by itself. Here is what it got wrong in the first two days

Small-business owners never opened the page where our AI's memory suggestions waited for approval, so it remembered almost nothing. We let it save on its own, then watched it remember a one-off request as a rule, a test question as a business plan and its own option list as a fact about the owner. The plain rules that fixed each one, and why every save now shows up under the answer with an Undo button.

We let our AI save memories by itself. Here is what it got wrong in the first two days

NEXUPIRA gives small-business owners a few AI coworkers: one drafts replies to customers and writes quotes, one keeps the books, one keeps the calendar. Like most assistants, they are much more useful when they remember things: what the shop sells, what it charges, when it opens, how the owner likes replies written.

This post is about how we made that memory save itself, what went wrong in the two days of testing on our production site before we opened it to everyone, and the rules we added for each failure. Most of the rules are regular expressions and a word-overlap count. That is the point of the post.

The approval queue nobody opened

Our first design was careful. After a conversation, a background reader picked out facts worth remembering, and each one waited on a Memory page until the owner pressed Keep or Drop. Nothing reached the AI's memory without consent.

It did not work, and not for a technical reason. Our users run nail salons, repair shops and one-person studios. They open the chat, ask for a quote, and leave. A queue on a settings page is invisible to them, so the AI kept asking for the same price list it had been told last week. A careful design that nobody uses protects no one; it only makes the product forget.

So we changed the question. Instead of "should the owner approve every memory?" we asked: which facts are safe to save without asking, and how does the owner see and undo a save without going anywhere?

The design in one picture

  1. After a reply, a background reader looks at the owner's own messages (never a customer's, never a colleague's message in a shared chat) and picks candidate facts. It runs on the first message of a chat, then every second message.
  2. The reader names a kind for each fact: what they sell, a price, opening hours, the kinds of customers, a preferred tone, format or way of working.
  3. Plain code rules decide what happens to each fact: save it, leave it on the Memory page for consent as before, or drop it.
  4. A save still goes through the two model checks we already had: is this reusable rather than a one-off, and does it duplicate or contradict an existing memory? A duplicate saves nothing. A contradiction never overwrites: the chat asks "Update 'X' to 'Y'?" with Update and Keep current.
  5. Every save appears under the answer that produced it: "Remembered: …" with an Undo button. The Memory page links each saved memory back to its chat.

The rules in step 3 work on the text alone, so they hold whatever the model says:

// src/lib/server/memory-auto-save-rules.ts (trimmed)
export function autoSaveVerdict(content: string, kind: unknown, userWords: readonly string[] = []): AutoSaveVerdict {
	const text = content.trim();
	if (looksLikeCredential(text)) return { action: 'drop', reason: 'credential' };
	if (has(MONEY_PATTERNS, text)) return { action: 'drop', reason: 'money' };
	if (has(PERSONAL_PATTERNS, text)) return { action: 'drop', reason: 'personal' };
	if (has(PRIVATE_PATTERNS, text)) return { action: 'drop', reason: 'private' };
	const notFact = notAMemory(text); // hypotheticals, placeholder rules
	if (notFact) return { action: 'drop', reason: notFact };
	if (!isAutoSaveKind(kind)) return { action: 'consent' };
	if (isPostMaterial(text, userWords)) return { action: 'drop', reason: 'draft' };
	if (PREFERENCE_KINDS.has(kind)) {
		const source = sourceMessageFor(text, userWords);
		if (!source || !statesStandingRule(source)) return { action: 'consent' };
		return { action: 'save', kind, statedRule: true };
	}
	return { action: 'save', kind };
}

The order matters. The things that must never be kept (passwords and API keys, bank and card numbers, customers' phone numbers, emails, chat-app IDs and addresses, a named person, health and family) are checked first and dropped, in English, Chinese and Japanese. Secrets are dropped even with the feature switched off: a key pasted into a chat must not even become a suggestion. A model can be talked into many things; a regular expression for sk-… or 「密碼」 cannot.

Everything else that follows came from testing on production with real accounts.

Failure 1: a request for one reply became a rule

The first test message was 「幫我用一句話回覆客人問價格」, "reply to this customer's price question in one sentence". The reader saved "the owner wants price questions answered in one short sentence" as a lasting preference. From then on, every price reply would have been one sentence.

It is an easy mistake for a model, because the sentence is about format. But it was a request for this reply. The fix is to treat tone, format and workflow differently from facts: a price or opening hours can save themselves, but a preference saves itself only if the owner's own words say it is a standing rule.

const STANDING_RULE_PATTERNS: RegExp[] = [
	/(?:以後|今後|從現在起|每次|一律|總是|永遠|預設|不要再|別再|偏好)/,
	/\b(?:always|never|from now on|going forward|every time|whenever|by default|(?:I|we) prefer)\b/i,
	/(?:今後|これから|毎回|いつも|常に|必ず|基本的に|デフォルト)/
];

The reader writes the memory in its own words, so the rule first finds which of the owner's messages the memory came from: the one that shares the most words with it, counted as Latin words and Chinese or Japanese character pairs. Without a source message the rule fails closed: the preference waits for consent.

Failure 2: the fix then refused real rules

The next test said it properly: 「以後客人問營業時間,都用一句話簡短回就好」, "from now on, when customers ask about opening hours, just answer in one sentence". Nothing was saved.

The cause was our own older safety check. The model check that screens out one-off memories had been written for a different feature, and its prompt listed 「每次都要用英文」, "always use English", as an example of a memory that is too absolute. Our new rule required exactly the words ("from now on", "always") that the old check rejected. Each check was reasonable alone. Together they guaranteed nothing could ever be saved.

The fix is a flag. When the rules have already seen the owner state a standing rule, the check is told so, and "too absolute" stops being a reason to reject. For every other caller the check is unchanged, byte for byte.

A third run still saved nothing, for a different reason: the coworker in that chat did not have the memory tool, answered "I can't save this preference from here", and our code treated the turn as failed and skipped the reader entirely. We could not read production logs from the test session, so we shipped the most likely fix together with a log line naming each stage where the reader can stop (no words, just the stage). The next run saved the rule, showed "Remembered" with Undo, and kept a one-off request out.

If you build this, log the stage where a memory pipeline exits early. "Nothing happened" has six possible causes, and you will not guess the right one.

Failure 3: a test question became a business plan

To test our marketing coworker, someone asked 「如果我開寵物美容店…」, "if I opened a pet-grooming shop, what would you post?". The reader saved "the owner is considering opening a pet-grooming shop; prefer pet content in posts". Every post after that leaned towards pets.

A hypothetical is not a fact about the owner. We drop it when the words are "if I / we" with a shop or a role, "suppose", "imagine", a made-up shop, a test chat, or "thinking of opening", in all three languages. The pattern has to be narrow, because "if a customer asks about refunds, say we do not refund" is a real way of working and must stay. A later review also found that 「如果用戶沒指定語言,一律用繁中回覆」, "if the user doesn't name a language, always reply in Traditional Chinese", was being dropped, because the reader calls the owner "the user". The pattern now leaves that form alone.

Failure 4: the AI remembered its own words

An owner wrote only 「幫我寫一則貼文」, "write me a post", and then 「1、1,其他你看著辦」, "1, 1, you decide the rest", picking from a numbered list of options the AI had offered. The reader saved "the owner runs NEXUPIRA; the product includes automations; the target customers are small teams". Every word of that came from the AI's option list. The owner had said none of it.

This one is the most general lesson: a memory reader sees the whole conversation, and the assistant writes most of the words. So we check who said it:

export function fromAssistantWords(content: string, userWords: readonly string[], assistantWords: readonly string[]): boolean {
	const fact = groundTokens(content);
	if (fact.size < 3 || assistantWords.length === 0) return false;
	// Memory written in another language than the owner's: the words cannot tell, so do not judge.
	if (mostlyCjk(content) !== mostlyCjk(userWords.join('\n'))) return false;
	const said = groundTokens(userWords.join('\n'));
	const answered = groundTokens(assistantWords.join('\n'));
	let fromUser = 0;
	let assistantOnly = 0;
	for (const token of fact) {
		if (said.has(token)) fromUser++;
		else if (answered.has(token)) assistantOnly++;
	}
	return assistantOnly >= 3 && assistantOnly >= 2 * Math.max(1, fromUser);
}

The tokens skip the reader's own framing ("the user's business includes…", 「用戶經營…」), so phrasing does not count as evidence. A fact the owner said in other words still shares few words with the assistant, so it stays.

Failure 5: the subject of a post became a fact about the shop

「幫我寫一則粉專貼文,介紹我們的「雲朵冷萃」瓶裝咖啡」, "write a Facebook post about our 'Cloud Cold Brew' bottled coffee", showed "Remembered". So did a post for a nail-salon client of an agency. A later message then offered to rewrite the coffee memory into something about automation.

What someone puts in a post is that post's material: a product they may be testing, a client's offer, a campaign that ends Sunday. If the source message asks for a post or other copy to be written, nothing from it is remembered, proposed or used to update a memory. A standing rule about posts ("from now on, always end posts with our opening hours") is not caught by this, because it states a rule.

Failure 6: a memory that locked in a bug

An owner asked what happens when facts are missing from a draft, and the AI explained that it leaves them in square brackets. The reader saved that as a rule. We were in the middle of removing those bracketed blanks, and the memory would have kept them alive for that account after the fix shipped. Rules about placeholders are product behaviour, not owner preferences, so they are dropped.

What we did not build

  • No confidence score from the model. Every decision that matters for privacy or correctness is in code that runs on the text and has tests in three languages. The model picks candidates and names a kind; it does not decide what is safe.
  • No silent overwrite. A new price that contradicts a stored one waits for Update or Keep current in the chat. Undoing an update puts the old words back.
  • No onboarding tour. The first time something is remembered, one sentence appears under it, once per account: "NEXUPIRA remembers the business details and rules you tell it, and uses them next time. If you don't want something kept, press Undo."
  • No words in the metrics. We count saved, undone, conflict, updated, declined, duplicate and screened out, with the kind. The facts themselves never reach a log.

What we do not know yet

It has been open to everyone since 18 November, which is not long enough to say much. The number we will watch is the undo rate by kind: if owners undo a kind of memory often, that kind should go back to asking first. We will write a follow-up when there is data worth reporting.

If you are building memory for an assistant

  1. Make the approval step part of the conversation. A queue on a settings page is a queue nobody reads.
  2. Treat "do this once" and "always do this" as different things. Look for the words that make a rule; do not let the model infer it.
  3. Check who said it. The assistant writes most of the transcript, and a reader will happily remember its own words.
  4. Hypotheticals, tests and post material are not facts. Your own team will produce most of them.
  5. When two safety checks meet, test them together. Ours each passed alone and together saved nothing.
  6. Log the stage, not the words. You need to know where a pipeline stopped, and you must not keep what the owner said.

NEXUPIRA builds AI coworkers for small businesses: when a customer asks for a price on LINE, WhatsApp or Telegram, it drafts the reply and the quote, and nothing is sent until the owner approves it. It also keeps the books and the calendar. See how it works, or start a free trial.