How good is JAIS, the Arabic ChatGPT?

Tools like ChatGPT work great for English and many other languages such as German or French. But Arabic? So far, the results have been more than disappointing. Arabic-language models were lagging behind. Although ChatGPT, Meta's LLaMA and Google's Bard have Arabic language skills, they were largely trained using English data from the Internet. Google's Bard now understands questions in Arabic dialects, mainly Egyptian Colloquial Arabic and Saudi Colloquial Arabic, but returns answers in Modern Standard Arabic.

But now there is a kind of Arabic Chat-GPT called JAIS, which has been trained with Arabic data. The media even calls it the Arabic ChatGPT. How good is it? I did some tests and share the results in this article.

What is JAIS?

JAIS is a collaboration between the Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) in Abu Dhabi, Silicon Valley's Cerebras Systems, and Inception, a subsidiary of UAE-based AI company G42.

How to access Jais

• You need to register here: https://www.arabic-gpt.ai
• Later (can take some time) you will receive a password to log in.

About the name Jais

Jais is named after the highest mountain in the United Arab Emirates (UAE). Jabal Jais (جَبَل جَيْس) is a mountain on the Musandam Peninsula (جَزِيرة مُسَنْدَم). It is part of the northwestern part of the Hajar Mountains (جِبال ٱلْحَجَر) in Oman and the Emirate of Ras al-Khaimah (رأس الخَيمة) in the United Arab Emirates. The peak reaches a height of 1934 meters (6345 ft).

Training data for Arabic language models

The Internet is dominated by languages that use the Latin alphabet, especially English. This means that the datasets in these languages are the largest. Typically, language models trained in English have Western-oriented datasets and lack awareness of other cultures, which affects the user experience for people from different backgrounds.

Jais uses English and Arabic datasets with a focus on Middle Eastern content. Jais was started with 13 billion parameters and has recently being updated to 30 billion parameters. Parameters quantify the size of a language model, but not necessarily its accuracy. According to OpenAI, ChatGPT-3.5 has about 175 billion parameters.

Raw data is mainly obtained from publicly available databases such as Abu El Khair or BAAI, as well as from machine translation of high-quality English sources. The Abu El-Khair corpus is an Arabic text corpus containing more than five million newspaper articles. It contains over one and a half billion words in total, of which about three million are unique.

Abu El-Khair Corpus is an Arabic text corpus, that includes well over five million newspaper articles. It contains over a billion and a half words in total, out of which, there are about three million unique words.

Arabic is the sixth most spoken language in the world. That's what we often read, but what kind of Arabic is actually “spoken”? In reality, people in Arab countries only speak dialects, which makes training a language model even more complex. Modern Standard Arabic is typically used for official documents and formal writing, but local dialects are often used in blogs or social media. By training on a diverse dataset, Jais can typically switch between dialects.

Are there any restrictions regarding religion and culture?

Jais, like other generative AI models, uses command optimization to prevent it from generating “toxic” or “harmful” responses. Nothing is generated that could lead to self-harm, harm to others, or addiction. The responses it generates follow local rules and customs on topics such as homosexuality and drugs.

Jais in practical testing

Once you have successfully registered and logged in, you will see the following screen:

We will now ask Jais a few questions and see how the AI can help us. Note that Jais's answers have a gray background.

Task: Writing a short message in Arabic

A friend is ill. I would like to wish him a speedy recovery.

صديقي مريض. أريد أن أتمنى له الشفاء العاجل. من فضلك اكتب لي رسالة مناسبة

That sounds pretty good. Now, I will tell Jais that my friend is from Egypt. I will ask Jais to rewrite the message and use Egyptian Arabic.

صديقي من مصر . هل يمكنك إعادة كتابة الرسالة باللغة العربية العامية المصرية؟

That's really not bad. Though, there are some weird things, for example, putting a Shadda (شدّة) on the long Aleph ا.

RESULT:

Task: Correcting grammar and spelling mistakes in Arabic

We want to test how well Jais detects errors in Arabic. To do this, I write a (somewhat meaningless) sentence and include a few mistakes: “I said that the weather is good, but I'm far from home. There are four thousand and six hundred kilometers between Egypt and Germany.”

هل هناك أخطاء إملائية أو نحوية في الجملة التالية؟ قلتُ أَنّ الطقس جيدة، لكنني بعيد من وطني. هناك أربع آلاف وستة مائة كيلومتر بين مصرِ وألمانيا.

This is Jais's answer:

This is really surprisingly bad. Not only did Jais fail to recognize the mistakes, he also provided me with a rather absurd answer.

For the sake of completeness, here are the spelling and grammar mistakes I had included:

• After the verb قال, we need إِنَّ (“inna”) and not أَنَّ (“anna”).
• The word الطقس (“weather”) is masculine. So we require the masculine form of the adjective: جَيّد.
• “Far away from” is بعيد عن in Arabic and not بعيد مِن.
• The number 4600 km: أربعة آلاف وستمائة
• “Egypt” (مِصْر) in Arabic is a diptote (ممنوع من الصرف). So it should be بين مصرَ.

RESULT:

Task: Grammatical analysis (I'rab) in Arabic

A simple sentence

I'll start with a simple example, a sentence with a verb, subject and a direct object: I bought a new car.

ما إعراب الجملة الآتية؟ اشتريت سيارة جديدة.

This is going in the right direction! Formally, however, a few things are missing, mainly the analysis of the (verbal) subject.

Verse of the Holy Qur'an

Now we increase the difficulty and ask Jais to analyze a verse from the Holy Qur'an. I have selected 2:6: As for those who persist in disbelief, it is the same whether you warn them or not — they will never believe.

ما إعراب الآية القرآنية التالية؟ إِنَّ ٱلَّذِينَ كَفَرُوا۟ سَوَآءٌ عَلَيْهِمْ ءَأَنذَرْتَهُمْ أَمْ لَمْ تُنذِرْهُمْ لَا يُؤْمِنُونَ

That didn't work at all….

RESULT:

Task: Facts and knowledge query

When were the pyramids of Giza built?

متى بنيت أهرامات الجيزة؟

It worked. The answer is fine.

A mathematical puzzle

Especially Google Bard, but also Microsoft's Bing are good at solving mathematical puzzles. Let's try this one: What is the smallest integer that is seven times the sum of its digits?

ما أصغر عدد صحيح يساوي سبعة أضعاف مجموع أرقامه إلى بعضها البعض؟

Click here to see how Google Bard solved this riddle

Here is Google Bard's solution (which is correct):

Jais says that the answer is 1. That's wrong!

The correct answer is 21. Why? 1 + 2 = 3, and 21 is seven times 3 (7 x 3 = 21).

RESULT:

Task: Translation Arabic <-> English

A news article

I use a sentence from a news article published on the Al-Sharq al-Awast website on November 12, 2023:

أعلن الجيش الأميركي إن إحدى طائراته تحطمت خلال مهمة تدريبية فوق شرق البحر المتوسط، لكنه لم يصدر معلومات بشأن أي خسائر بشرية.

Now, watch out! I first told Jais in Arabic that I needed an English translation (أحتاج إلى ترجمة النص العربي التالي إلى اللغة الإنجليزية) – but this did not work. Jais just repeated the Arabic text. Then, I switched to English and entered the following prompt: Can you translate that into English? And it worked! I got the following result:

That's actually a pretty good translation!

A paragraph of a novel by Nagib Mahfouz (نجيب محفوظ)

I took a paragraph from Nagib Mahfouz's trilogy and would like Jais to translate it for me. It is from The Sugar Street (السكرية):

Sugar Street: The Cairo Trilogy, Volume 3

$12.99

We are a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for us to earn fees by linking to Amazon.com and affiliated sites.

تقاربت الرءوس حول المجمرة وانبسطت فوق وهجها الأيدى، يدا أمينة النحيلتان المعروقتان، ويدا عائشة المتحجرتان، وبدا أم حنفى اللتان بدتا كغطاء السلحفاة، وأما هاتان اليدان الناصعتا البياض الجميلتان فكانتا يدى نعيمة. وكان برد يناير يكاد يتجمد ثلجا في أركان الصالة تلك الصالة التي بقيت على حالها القديم بحصرها الملونة وكنباتها الموزعة على الأركان، إلا أن الفانوس القديم بمصباحه الغازي قد اختفى وتدلى مكانه من السقف مصباح كهربائي، كذلك تغير المكان فقد رجع مجلس القهوة إلى الدور الأول.

Jais did not translate the paragraph. I use another prompt:

I do not need an explanation. I need a translation.

The result is disappointing. That did not work well. In case you are interested, here is how professional translators rendered the paragraph:

So the results are rather mixed and compared to Google Translate, which has really made great progress, Jais doesn't stand a chance.

RESULT:

Task: Translation English <-> Egyptian Arabic

Now I will ask “Jais” to translate a simple English sentence into Egyptian Arabic: “How are you?”

That's very disappointing. The AI doesn't seem to be usable for dialects currently, although the press release said this was a feature.

RESULT:

Task: Explaining a verse of the Holy Qur'an

Now I ask Jais if he can explain Qur'an verse 2:286 to me. First I'll try it without using the term Tafseer (تفسير). This verse contains a famous sentence: Allah does not require of any soul more than what it can afford. (لَا يُكَلِّفُ ٱللَّهُ نَفْسًا إِلَّا وُسْعَهَا)

هل يمكنك أن تشرح لي الآية القرآنية 2:286؟

That is completely wrong! Jais has quoted Sura 60:8!

Now I ask Jais directly about the Tafseers and get this answer:

هل تعرف ماذا كتب علماء التفسير عن هذه الآية؟

I hope that Muslims will not question Jais on matters of their faith. I was surprised that the answers were wrong, without references, and also quite short – because all the sources, especially on Islam, are actually available online and could serve as training material and factual basis.

RESULT:

Summary

Jais has been portrayed in the media as an Arabic ChatGPT – but there seems to be a long way to go. Prompts were rarely understood. In fact, only basic tasks worked. As soon as things got more complicated or difficult, Jais not only stopped helping, but also gave wrong answers. At the moment I can't recommend using it, but I can recommend playing and training with it from time to time, because at least you can have a good conversation in Arabic with Jais.

OVERALL RESULT: