GoTo’s Indonesian AI Model Shows Why Each Language Needs Its Own Score

GoTo, the Indonesian company behind Gojek, announced a new AI language model in Bali on October 19. Built on Google’s Gemma 3, it is intended to help developers build services in Indonesian and several regional languages. Its published results offer a more useful story than a single claim of national AI progress: the gains vary substantially by language and task. GoTo’s announcement

GoTo’s evaluation shows its sharpest language-composite gain over Google’s same-size Gemma 3 in Batak Toba. Indonesian and Balinese two-turn conversation scores fall slightly. GoTo’s published results

For an Indonesian developer choosing a system for customers, those differences affect what to test first. A service that translates local-language messages has a different job from one that must answer a customer’s follow-up question correctly. Combining their results into one ranking makes a product decision harder.

Where the reported gains appear

The release, developed with AI Singapore, supports English, Indonesian, Javanese, Sundanese, Batak Toba and Balinese. Its model card names six languages and describes additional training for Indonesian language and culture. The downloadable checkpoint has 27 billion parameters, the numerical settings learned during training. GoTo’s model card

On GoTo’s own published evaluation, the results against Google’s Gemma 3 27B instruction model are:

These publisher-reported composites combine different available tasks by language. Compare models within the same language and task; the totals do not measure the percentage of customer questions answered correctly. Leaderboard data

Batak Toba translation from Indonesian rises from 23.26 to 54.75. Indonesian two-turn conversation falls from 54.21 to 53.75; Balinese falls from 43.79 to 42.23. The small declines need uncertainty estimates and further testing before supporting a broader conclusion. Leaderboard task results

Local knowledge takes local contributors

Hans Patuwo, GoTo’s group chief executive, framed the project as an Indonesian capability: “Indonesia makes great use of global AI, but we must build our own too.” The company says it worked with the University of North Sumatra on present-day Batak language material and involved interns from seven universities in collecting data, training and testing. GoTo’s announcement

Those contributions help explain what a local-model effort involves. A published foundation model supplies a starting architecture and learned knowledge. People still have to choose appropriate local material, prepare examples and decide what counts as a natural answer. A translated English test can miss forms of address, cultural references or changes of language that matter to the intended users.

The model card describes linguists and native speakers filtering, localizing and translating instruction-following tests. One test checks whether responses satisfy explicit constraints and appear in the requested language. Another uses an older GPT-4 checkpoint to judge two-turn conversations against an older GPT-3.5 baseline. That method is useful to disclose, but it does not establish a comparison with the best commercial models available in October 2026. Evaluation method

Independent research offers broader ways to assess local capability. IndoMMLU, created by Fajri Koto, Nurul Aisyah, Haonan Li and Timothy Baldwin, contains 14,981 questions across 64 tasks and education levels. Its material includes school and entrance-exam questions, Indonesian language, and nine local languages and cultures. Published in 2023, it makes regional knowledge visible in a way an English-only test would not. IndoMMLU research

Professional teachers helped assemble IndoMMLU, and 46% of its questions concern proficiency in Indonesian and knowledge of nine local languages and cultures. That published subject mix lets a developer judge how closely an exam result matches the knowledge its application needs. Benchmark composition

An exam benchmark still leaves many product questions unanswered. Multiple-choice knowledge does not measure whether a service can recover from an unclear request, preserve a customer’s meaning when switching languages or follow a company’s policy over several messages. Developers need examples of those interactions in addition to established academic tests.

A useful review would retain the actual answers, not just the grades. Native speakers could then explain whether an apparent error came from misunderstood instructions, an unnatural expression or a factual mistake. Those findings lead to different changes in an application, even when they produce the same numerical penalty.

The SEA-HELM research team, led by Yosephine Susanto and colleagues, organized its 2025 evaluation around language tasks, instruction-following, regional linguistics, culture and safety. That published suite covers Filipino, Indonesian, Tamil, Thai and Vietnamese. GoTo’s regional-language evaluation therefore needs to be understood through its own documented adaptations, rather than assuming the original suite covered every language named in this release. SEA-HELM research

Downloading weights leaves a hosting decision

The model is available under Google’s Gemma terms. Those terms allow use, modification and distribution subject to conditions, including use restrictions and requirements when passing the model to others. The release gives developers access to weights, but calling it an unrestricted Apache-licensed model would misstate its license. Gemma terms

Hosting also has a measurable starting cost. At two bytes per parameter, 27 billion parameters require about 54 gigabytes for weights alone. Working memory, stored conversation state and other software add to that calculation. Developers deciding between their own hardware and a hosted service need a complete serving estimate, including concurrent users and message length.

Control over deployment can be valuable to a team that needs a fixed model version or a particular operating environment. It also assigns that team responsibility for updates, monitoring and the cost of keeping capacity available. Download access does not settle those choices.

A commercial trial should put response time and serving cost beside language quality. Testers can hold message length and simultaneous users constant, then compare how many responses meet their quality threshold within the required time. The published language ranking cannot supply that operating estimate.

Reporting task results separately would also let a prospective user weight them for a particular service. A translation product and a conversation product can select different models from the same underlying evidence without either team misreading the overall ranking.

The most useful next evidence would pair GoTo’s published scores with independent tests by native speakers on clearly defined tasks, including the smaller languages. Its leaderboard already gives those testers a place to start: Batak Toba translation, follow-up conversations and the cases where aggregate gains conceal a different task result.