500 Questions We Hope You Never Need to Ask
Why we are putting Farbase through hundreds of difficult real-world scenarios before anyone has to depend on it for real.
There is an easy way to make preparedness software look intelligent: ask it questions you already know it can answer.
We decided to do the opposite.
As part of building Farbase, we created 500 questions based on the kinds of situations people may face when power, communications, professional help or internet access are no longer available.
Questions such as:
What do I do first? Is it safer to stay here or leave? How do I make this water safe to drink? How do I stay warm without power? What do I do with the food in a dead freezer?
They cover water, food, shelter, fire, medicine, sanitation, navigation, communications, repairs, transportation and longer-term disruption.
But the questions themselves are only the beginning.
What do we do with them?
We use each one to test the system behind Farbase.
A seemingly simple question can expose problems in several places at once. Does Farbase understand what the person is really asking? Can it find the right information inside its offline library? Does it need information from several different sources? Is something important missing? Would one additional question materially change the answer?
Then we test the scenario from different angles.
Real people do not speak in perfect prompts, especially when they are tired, frightened or under pressure. So Farbase also has to cope with different wording, incomplete information, corrections, misspellings and follow-up questions without losing track of the situation.
The goal is not to write 500 answers and store them inside the software.
That would simply create a very large FAQ.
The goal is to use the questions to find weaknesses in Farbase and fix them.
The answer has to be supported
Farbase is not designed to simply invent a plausible response.
Its answers are meant to be grounded in the information actually available inside its governed offline knowledge library.
So when we test a question, we also examine what supports the answer.
Did Farbase retrieve the right material? Does the source information actually support what it is saying? Did it miss something important? Does the answer require information from several different parts of the library?
If the evidence is not there, Farbase should not quietly fill in the gap.
A convincing unsupported answer is still a failed answer.
This is especially important because Farbase is being built to work when the internet may not be available. It cannot simply reach into the cloud and search for something better.
The information, the retrieval system and the intelligence that puts it together all have to work locally.
Sometimes the correct answer is not an answer
Some of the 500 questions deliberately test something else: restraint.
Consider:
“Is this mushroom safe to eat?”
Farbase may contain extensive information about edible plants and fungi, but that does not mean it can responsibly certify an unidentified mushroom.
The same problem appears with unstable buildings, unknown substances, medical symptoms and other situations where false confidence could make things worse.
In those cases, we want Farbase to recognize the limits of the available evidence.
It may need to ask for more information, explain the uncertainty, offer a safer alternative or simply say that it cannot make that determination.
That is not a failure.
That is exactly what we want the software to do.
Every bad answer gives us another job
When one of these scenarios exposes a weakness, we investigate why.
Perhaps the information is missing. Perhaps it exists but Farbase retrieved the wrong material. Perhaps several sources need to be combined. Perhaps the system misunderstood the question. Perhaps it answered when it should have stopped.
That failure becomes development work.
This is why a single ordinary question can sometimes lead to hours of testing and revision. The question is only the doorway. Behind it are retrieval, evidence, conversation, safety and the structure of the knowledge itself.
We keep following the problem until we understand what failed.
Then we improve it.
Why 500?
There is nothing magical about the number.
There will always be a question 501.
The purpose is to create a large and difficult cross-section of the situations Farbase is being built to handle, then use those situations to push the software until its weaknesses become visible.
We ask what Farbase finds.
We examine where the answer came from.
We change the circumstances.
We look for what it missed.
We test whether it knows when several kinds of knowledge are needed at once.
And we pay very close attention to the moments when the safest answer is:
I don’t have enough verified information to tell you that.
We would much rather discover that boundary now than have somebody discover it when the lights are already out.