I tested Google Gemini's Solaris knowledge. It's...not great

drysdalk · February 26, 2024, 8:09am

Hi,

I've been playing about with Google's new Gemini for the last little while, just to see how it compares with ChatGPT et al. In the UK interestingly we can't use it to generate images yet, so we've missed out on being able to join in that whole imbroglio. However, my own personal experience so far has been that it seems to make more factual errors (or "hallucinations", if you prefer) than ChatGPT when asked some relatively straightforward questions.

I won't include the entire text of Gemini's responses in this post, but I'll include the most salient points of its replies and summarise the bits I skip.

One of the things I thought to try was to test its Solaris sysadmin knowledge. I thought I'd ask it something relatively obscure, yet factually straightforward. I went with:

For those who aren't too familiar with Solaris, this command is something of an "emergency stop", immediately shutting down the OS without doing a controlled shutdown of any kind, and (if you're running on Sun/Oracle hardware) dropping you down a level to the OpenBoot "ok" prompt.

It responded to my question with:

So initially, Gemini seemed to believe that uadmin was a su-equivalent, basically. I replied with:

It had a brief "think", then replied with:

You are absolutely right, and I apologize for the previous inaccurate information. The command "uadmin 2 0" in Solaris is used to disable a user account. There is no single, direct equivalent command in Linux for this specific functionality.

However, in Linux, you can achieve the same outcome using different approaches depending on your desired level of control and security:

1. Using usermod:
...
... (describes the "usermod" usage for locking an account)
...
2. Using passwd:
...
... (talks about how to lock an account with "passwd")
...
3. Using userdel (with caution):
...
... (gives sample command for deleting a user with "userdel")
...
By choosing the appropriate method based on your specific needs and considering the inherent risks, you can achieve the functionality similar to the Solaris command "uadmin 2 0" in the Linux environment.

So, now Gemini seems to think that "uadmin" is specifically for disabling and/or removing user accounts, which is still wrong. I went back to it with:

Here, we finally get a bit closer to the correct answer:

I apologize again for the previous inaccurate information. My initial responses regarding the Solaris command "uadmin 2 0" were incorrect.

The correct explanation for what "uadmin 2 0" would do on a live Solaris system is as follows:

The command "uadmin 2 0" is used to reboot the system in a single-user mode.

Here's a breakdown of the command:

...
... (describes the syntax of the "uadmin" command, in a more-or-less correct manner)
...

Therefore, running "uadmin 2 0" on a live Solaris system would cause the system to restart, but only allow access for the root user (administrator) in a single-user shell environment. This environment provides limited functionality compared to a normal multi-user mode, allowing the administrator to perform specific system maintenance tasks or troubleshoot issues.

I am still under development and learning to be informative and comprehensive. I appreciate your patience and feedback as I continue to improve my accuracy and understanding.

Here it gets it half-right: it does shut the OS down, but it seems to think that the OpenBoot "ok" prompt is equivalent to going down to single-user mode, which isn't correct.

So I gave it another chance to tweak its answer:

And now we get even closer still to a correct answer:

Thank you for the additional information and your patience in helping me learn. You are absolutely correct; the "uadmin 2 0" command in Solaris drops the system to the "ok" prompt (single-user mode), not a complete system reboot.

While there isn't a single, direct equivalent command in Linux that replicates the exact behavior of "uadmin 2 0," you can achieve similar functionality using a combination of commands depending on your specific needs:

1. Entering single-user mode:
...
... (talks me through using init, or editing the Grub menu command line at boot time)
...
Both methods will achieve the same result of dropping you to a root shell prompt in single-user mode, which is similar to the "ok" prompt in Solaris.

2. Shutting down the system:
...
... (describes how to use the shutdown command)
...
Choosing the right approach:

Single-user mode: Use this option when you need root access to perform system maintenance or troubleshooting tasks in a limited environment.

System shutdown: Use this option when you want to completely power down the system.

It's important to note that unlike "uadmin" in Solaris, the init command in Linux offers various runlevels beyond single-user mode, providing different system functionalities for maintenance purposes.

By combining these commands depending on your desired outcome (entering single-user mode or system shutdown), you can achieve a similar experience to "uadmin 2 0" in the Linux environment.

I appreciate your feedback and the opportunity to learn from my mistakes. This interaction has helped me improve my understanding and accuracy in future responses.

So, in the right ballpark, but still not quite right: to recap, "uadmin" acts an emergency killswitch, and does not initiate a controlled shutdown (and it has nothing to do with single-user mode itself, per se). There are Linux equivalents to this kind of instant poweroff, which Gemini still hadn't mentioned any of. I decided to ask it why:

Here it at least had an actual reasoned response as to why it hadn't mentioned any of these:

You are absolutely right; I should have mentioned halt -f and systemctl --force --force poweroff as alternative approaches to achieve a similar outcome to "uadmin 2 0" in the Linux environment.

Both these commands halt the system immediately, though they differ in their methods and context:

...
... (dives into the specifics of each command and what they do)
...

Why I didn't mention them initially:

While these commands technically achieve a similar immediate shutdown as "uadmin 2 0" (which also carries inherent risks), I focused on highlighting the safer and more recommended approaches for the following reasons:

Emphasis on user safety: My primary focus is to provide information and guidance that prioritizes user safety and data integrity. Highlighting the potential risks associated with forceful shutdowns like halt -f and even systemctl --force --force poweroff aligns with this objective.

Promoting best practices: Recommending the use of init 1 or the Grub menu to enter single-user mode and shutdown -h now for system shutdown aligns with best practices and promotes safe system administration techniques.

However, acknowledging the existence of alternative commands like halt -f and systemctl --force --force poweroff is crucial for comprehensive understanding and transparency, even if their use is generally discouraged.

I am still under development and learning to prioritize informative and responsible answers. Your feedback helps me identify areas for improvement and provide more comprehensive and nuanced information in the future.

In summary then: I don't think we have to worry about Gemini putting us out of business just yet.

drysdalk · February 26, 2024, 8:37am

By way of comparison, here's what ChatGPT 3.5 had to say when asked my initial "What would the closest Linux equivalent be to the Solaris command "uadmin 2 0" ?" question for the first time:

So a much stronger initial answer: ChatGPT at least grasps what "uadmin" does, in that it has something to do with shutting a system down and that there's probably a Linux equivalent to that, rather than thinking it's a user account management tool.

Matt-Kita · February 26, 2024, 2:04pm

No, it grasps what others have written about "uadmin" command in its training set (i.e. in hundreds of GBs of text fed to it).

Neo · February 27, 2024, 4:53am

This is true.

None of these autocompletion engines like ChatGPT or Gemini "grasp" or "understands" or "seems to think" or "knowledge" ... anything.

There is a large language model (LLM) and a lot of code around the model to create the appearance of some type of conversational (artificial) intelligence. But behind it is an "autocompletion" engine which simply generates text (or images, blah blah). This is not "thinking" or "grasping" or "understanding" or "knowledge", it's simply "generating output".

A number of months ago I used ChatGPT to refactor some code, trying to save a client some money.

Now, I see bugs here-and-there, for example: missing variable definitions used later in the code, etc.

Also, the refactoring by ChatGPT was mostly rearranging code based on whatever coding pattern it was using at the time; so if you used ChatGPT to refactor code on Monday, a few days later (or even on the same day) the refactoring patterns were different.

There was no consistency.

ChatGPT just creates "a mess" and kind of "hit-or-miss" gobbledygook where it just generates a lot of "blah blah blah".... based on the reams of data in the model.

It's basically nonsense when applied to coding, code refactoring, etc.

Yes, autocompletion engines are useful; but the longer the autocompletion text, the more the text deviates from accuracy.

As for me, I was initially kinda hopeful that ChatGPT and generative AI would help coding; and sometimes it is useful to send it a code block and ask to check for errors, etc.

But there is close-to-zero chance than any of these generative AI apps will replace engineers and software developers. In fact, I consider these tools dangerous and soon organizations will be calling in skilled humans to try to figure out what-the-fuuk ChatCPT has coded and fix all the bugs from gen-ai.

MadeInGermany · February 27, 2024, 9:54am

Currently AI is like a little child parroting what it hears.
Give it some time to grow up!

Neo · February 28, 2024, 5:59am

That is insulting to both children and parrots, hahaha.

Both children and parrots are eucaryotes where every cell (except for red blood cells) enjoy a symbiotic relationship with their ancestral mitochondria. Without mitochondria, there would be no children or parrots.

When a parrot learns to repeat phrases, the learning process, like for a child, is biological and biochemical. For parrots, when they are taught to repeat a sound, they do it for a reward from their teachers and that reward stimulates the secretion of hormones derived primarily from proteins; all of which require ATP produced by mitochondria.

Hence, biological organisms are complex biosystems and learning to "parrot" requires a lot more complexity than creating a generative AI based on a large language model.

Generative AI does not "parrot". Generative AI generates text based on training an artificial neural network to generate output based on input. What you see with ChatGPT (speaking of text) is just a more sophisticated version of the (mostly annoying) text-autocompletion engine you see when typing on a keyboard.

Yes, if you write fiction, generating a lot of text which has little basis in reality, GAI is great.

However, generating facts and knowledge requires a lot more than generative AI; because the LLMs are based on information which can be conflicting (at best) or down-right wrong.

So, the scientists who creat GAI work on rules which make the auto-generated text (or images) more politically and socially acceptable; which creates a significant amount of bias. Each culture, political and social group requires a different set of biases, and so generative AI is much less like a parrot, and more like a propaganda generating machine based on the biases of who created the system.

Children are mostly innocent. Parrots are generally innocent. Generative AI is not innocent. GAI is seriously biased based on whatever the "owner" of the GAI system wishes it to be and their target users.

There is nothing "intelligent" about generative-AI. GAI is mostly a large language model with a lot of wrappers to make sure the generated output does not offend users, so the GAI companies can profit from those same users returning output which pleases their target audience(s).

Parrots and children are infinitely more intelligent that generative AI. Please do not insult them putting them on the same level as an LLM based ANN algorithm wrapped in rules and controls, LOL