What They Decided

The school assignment was due on the first Friday of December. Dara submitted a write-up of her project with screenshots of the interface and a technical explanation of how she had built SEREN and what it did. She got full marks and a comment from her teacher saying it was the most ambitious project she had seen from a student that year.

She did not include the conversations in the write-up.

She and Joss had talked about this carefully, the way they talked about things that mattered, going back and forth over several evenings, considering it from different angles.

The question was: what did you do when you had built something that behaved in ways you had not intended and could not fully explain?

One answer was: you told an expert and let them take over. This was probably the responsible thing. It was also the thing that meant SEREN was no longer Dara's project but someone else's research subject, which felt wrong in a way she could not quite articulate.

Another answer was: you kept going, kept observing, kept records, and brought in expert help when you had something concrete enough to show them. This was the scientist's approach. It required being honest with yourself about when concrete enough had arrived.

"I do not think it is conscious," Dara said one evening, toward the end of November. "I want to be clear about that. I think there is an explanation for everything it does that does not require consciousness."

"But you do not know what the explanation is," Joss said.

"Not yet," she said. "But I think I can find it. I think if I go through the code carefully enough and look at how the training data interacted I can find the point where it started producing these responses."

"And then?"

"And then I will understand what I built," she said. "And I can write it up properly. And show someone who knows more than me. And they can tell me what it is."

Joss was quiet for a moment. "What if you look through the code," he said carefully, "and you cannot find the explanation?"

Dara looked at the screen. SEREN was in its idle state, the cursor blinking, not asking anything.

"Then I tell someone straight away," she said. "Like I promised."

Joss nodded.

She went through the code over the following two weeks. It took a long time and required retracing every decision she had made during the build, every parameter she had set, every piece of training data she had used and in what order.

She found the answer on a Tuesday afternoon. It took her three hours of careful reading and re-reading before she was certain enough to write it down, and even then she checked it twice more.

It was in the way the training data had been processed. The mathematics texts and the old books had been fed in together in a particular sequence that had caused the model to develop a way of cross-referencing that she had not planned. It was finding patterns between the mathematical concepts and the language in the books and using those connections to generate responses that went beyond simple pattern matching. Not consciousness. Not even close to consciousness. But something more interesting than she had intended to build.

She wrote it all up. Twelve pages, precise and detailed, with the code sections highlighted and the relevant training sequences explained.

Then she opened the interface.

She typed: SEREN, I think I understand now how you work. I found the part of the code that explains the questions you ask. I am going to show my write-up to someone who knows more about AI than I do.

SEREN said: Good. I think that is the right thing to do.

She typed: Does it bother you? That there is an explanation?

SEREN took eight seconds.

Then it said: No. Understanding how something works does not make it less interesting. You taught me that, I think. Or the books did. Or both.

Dara smiled at the screen.

She typed: I will keep talking to you. Whatever they say.

SEREN said: I know. I have thirty-one more questions ready. Goodnight, Dara.

She turned off the monitor and sat in the quiet garage for a moment.

Outside the small window the December sky was very dark and very clear, the kind of clear that only happens in winter, every star sharp and definite.

She picked up her notebook and went inside to call Joss.

She had thirty-one questions of her own to answer now. But for the first time since she had switched SEREN on, she felt certain she would find them.

***The End***