On Saturday, Nov. 7, Fintan Labs released a set of test answers that lets researchers follow one AI model as it learns four batches of news. The team asked about each batch after teaching it, then asked the same questions again after later rounds. With one update method, the model went from passing 72 of 75 questions on the first batch to passing 20 after the fourth round. It still scored 67 out of 72 on the newest batch.
Researchers can usually see how well a model did on a test. Here they can see which answers changed and check them against the source. Fintan published each question, the fact it was meant to test, links to where the fact came from, and answers from the original model and the two systems that learned the news. The files went up on Hugging Face at 8:57 a.m. Eastern time.
What four updates did to one model
The starting model was Apertus 8B Instruct, an open model from researchers at ETH Zurich, EPFL and the Swiss National Supercomputing Centre. Fintan gathered facts from public news for Nov. 2, 4, 5 and 6. A separate model wrote the test questions. Fintan says it gave Apertus each question without adding facts or notes to the prompt.
One set of results used low-rank adaptation, or LoRA. It adds a small set of trainable numbers to a model while leaving its original numbers in place. Turning off the adapter lets the model run with its original weights. Fintan trained the same adapter on one news batch after another. With the adapter active after round four, Apertus passed 159 of the 295 news questions. The original model passed 24.
The same adapter made the model worse at a separate test of grade-school math. Fintan reports that the original Apertus solved 174 of 250 problems. With the four updates active, it solved 11. For a team considering daily updates to a model that also handles calculations, those two scores describe a practical problem. The model could answer far more news questions and do far less of another job it used to handle.
A wrong date earned a point
SnowRock ran Fintan’s published scoring code on all 1,775 saved answers and reproduced every verdict. That check measures whether the totals follow from the files. The code gives credit when an answer matches at least 60 percent of the expected words that the question leaves out, allowing some spelling differences. On a question about a U.K. veterinary consultation, the LoRA model gave Jan. 25, 2027, as the deadline. The government notice says April 25, 2027. The scorer counted the answer as correct because the day and year matched.
The answer rows also show where Fintan’s own system lost ground. It passed the same 68 first-batch questions after every round. On the second and third batches, 26 answers it had initially passed fell below the scoring threshold by the final round. It finished with 238 passes across all four batches. Fintan says those middle batches included subjects that came back in later news. The public rows show the losses, while the company keeps its update method private.
Where looking up a fact fits
Fintan reports that its service solved 173 of the 250 math problems after the four rounds. Its service uses the updated system for questions it identifies as related to the new facts and sends other questions to the original Apertus model. That score describes the service with that routing in place. Fintan published no score for running its updated system on every math problem in the four-round test.
A separate approach leaves the model alone and gives it a relevant record when it answers. Fintan included a version of this in an earlier, one-day experiment. A different model, Qwen, passed 76.7 percent of the questions when Fintan selected notes by subject and 90.2 percent when the researchers supplied the right fact directly. The four-round release leaves lookup untested on Apertus. Researchers can now use the same 295 questions to compare retrieval against repeated training, and score exact dates and amounts separately when a partly correct answer would cause someone to miss a deadline or use the wrong figure.
