General, General Science, Security, Privacy, Technical, Uncategorized

AI: A Check on the Hype and Hysterics

It has been interesting to watch the events of the past year or so — from the first release of ChatGPT in November 2022, to now. As a researcher with one foot in the Machine Learning world and the other in the Cyber Security world, my interest piqued as both a practitioner and an adversary. And I was not surprised to see the same concerns rise once again to the forefront: Concerns that are considered cliche in movies, but not in reality it seems.

It must be said that the slow march of progress in Machine Learning and AI research did not stop, and that it was prevalent long before November 2022. What OpenAI did in 2022 was simply reignite the fading spotlight on a topic that was nearly relegated to spam filters, movie and product recommendation engines, and the like: Some call it the AI Spring.

Along with a renewed interest in the area comes an increased profile in the public domain. This is a double-edged sword with both benefits and criticism.

It benefits researchers because new funding streams open up (and so much of research is predicated on the timely acquisition of funds — see the timeline for the Covid19 vaccines for a great example), it also means that institutions see value in the research that individuals have always been doing. This opens up career, networking, and collaboration opportunities which further promotes development.

Though the benefits are great, there are always drawbacks. With an increased profile, this area of research is seeing a significant increase in criticism and scrutiny. This is not a bad thing. As new technologies are developed, there is a need to assess them and their impact in the current legal frameworks, or in society in general — especially technology with far-reaching implications such as AI.

But it is important to not succumb to hysterics, or to take our cues from Hollywood, or follow the path that led many a crypto-bro to their financial doom.

A sane and informed assessment is warranted. Perhaps I can lend a hand in that regard.

Over the past year, I have been asked by numerous people what I thought about the concept of “AI taking over the world” or the advent of Artificial General Intelligence (AGI) or simply what is currently going on. In answering these questions, I thought I would write them down so as to provide a basis for any further, more verbose, response one may have. So here it is.

To answer the most commonly asked question (in some form), “Is AI going to take over the world?“, the answer is most definitely “Not anytime soon“.

But I suspect one may want more of an answer than that. Some reasoning perhaps. Maybe some figures? Or a chart? Alas, let me enumerate the reasons:

Enormity of Scale: Data

The scale of data required to train these models is enormous! Specifically, most AI models (usually a deep neural network) require a “labeled” training set to train the model for use. Labeled data is data that also has the proper classification in it so that the model knows if its right or not. But data doesn’t come labeled by default. Someone has to put in the effort to properly label each item. Further, access to very large datasets of labeled data is difficult to come by and its creation usually requires human assistance. This includes the latest major development: generative algorithms.

Granted, there are some automated ways to label data, but to properly fine-tune your models, humans need to check — at least a subset of the data. There is research into reducing the amount of training data required to achieve some level of accuracy, but the current state of the art requires a significant amount of data to train models. The current rule is “more data and larger models allows for more complex connections and relationships to be expressed”.

So from a pure data access and scalability perspective, the storage of all this data is a limitation — and a model is only as good as the data it is trained on (not to mention any training feedback or fine tuning which we will talk about further down).

Enormity of Scale: Processing

The processing power to continually train new models is enormous! Microsoft is pumping money into OpenAI to train its models. It costs a lot of money to train and power these models (Processing). Various sources place the training of a single ChatGPT model at $4 million. With the growth of better models, training sets, and data sources, this could balloon.

Currently, it is estimated that ChatGPT costs $700,000 per day to run. Microsoft invested $10 billion into OpenAI — which is keeping things afloat, but this places pressure on the organization to seek ways to monetize to cover the long term costs of training and operating the system.

For a true AGI to exist, it must be able to build and use many of its own trained models based on a dataset it created. This is unlikely due to the significant cost (computationally but ultimately financially) of training models — amongst other limitations (see: Enormity of Scale: Data, and Fragility: Training Feedback Autonomy/Human Intervention).

Enormity of Scale: Storage

This is pursuant of the scale of data. Models and their training sets can take a LOT of space. This needs to be stored if a model is to be retrained with more data in the future. Sure data storage is modular, and you can trivially add more, but you have to actually do this — and there is a cost associated with it.

The more models you train, the more datasets you need, and the more storage you require. This isn’t a major limitation like some of the other items in this article, but it is a constant cost that adds up over time, and shouldn’t be ignored.

Fragility: Applicability

Guess what? Models do not transfer well. The model that detects cancer better than 99% of doctors won’t do well detecting faces in an image. This makes sense from a practical sense, considering that models are trained on a dataset and optimized for the outcome of that training set. This means that the cancer detection model was given a large set of, let’s say, labeled chest X-rays to train on (some with cancer, some without), and the face detection model is given a set of labeled images (some with faces, some with no faces in them). These model’s “entire world” is their training set, and a major assumption is that the training set properly and accurately reflects the “real world” — so if you train a model on faces then give the model a bunch of chest X-rays, the model will try and find faces in the X-rays, not cancer. Not transferable.

There are some tricks to make the end user think that an “AI” is generalized such as swapping models based on application, etc., but now one has to train a set of models for each application — and then one runs into the scalability issues identified above. One can make the claim that they could make an ‘AI’ that can create other models, but this ignores the fact that the creation, vetting, de-duplicating, labeling, assessment for bias (and subsequent removal of bias) of datasets — and the extraction of pertinent features contained within dataset for use within a machine learning/AI model requires a significant amount of Human interaction — so even though it can “hot-swap” models, the datasets would need to already exist in pristine formats, ready-to-go, for it to create a new model trained on it. This is unlikely — especially since there is an entire area of academic research that is dedicated to the creation, assessment, labeling, de-duplicating, and assessment of bias of datasets.

This leads into the next issue — training the models in the first place.

Fragility: Training Feedback Autonomy & Human Intervention

ChatGPT and other such models require scripts, filters, heuristics, and human feedback to fine-tune their training. They are not “hands off”, and constant fine-tuning by humans is required to improve model accuracy and applicability. The current state of the art is very much a “Wizard of Oz” scenario, with data scientists behind the curtain, pruning datasets, filtering out data that is undesirable or biased, and generally “tweaking the dials”. Of course a model can simply ingest all information available, but the resulting model has a high likelihood of ending up like one of Microsoft’s earlier AI chat-bot attempts. “Garbage-in, garbage-out.” Suffice it to say, this is an open area of research with challenges and limitations.

Fragility: Access

These large models require access to resources (Computation, storage, etc.) as stated above — which are physical locations. As such, A “runaway” AI has the very real threat of simply getting unplugged — either remotely or by someone walking into the room (or data-center) and doing so physically.

Imagine a scenario where a simple, well-written software solution is created to solve a temporary problem — Let’s say to provide a simple tunnel to servers, or access to an AI’s compute cluster (Giving it the horsepower to do its computations). The solution is simple, stand-alone, and temporary, so it is put on a small machine the developer just-happens to have around. It works great and everybody’s happy. The machine hums along for years. Eventually it unwittingly becomes a critical part of the organization: The only way to access the compute cluster. One day, someone asks, “Does anyone know who owns this machine?”, someone responds, “Unplug it. Someone will show up.” And all-of-a-sudden, all access to the system is lost. Someone walks in and unplugs a random machine sitting on a desk (or under it), and it causes a crisis. This is how fragile large systems can be. It is relatively easy to walk into a room and unplug the control server of a cluster — should the need arise or in an emergency.

This story is not hypothetical. This happened to Twitter (Or ‘X’ if you prefer) many years ago. It was a Mac Mini. The Mini was controlling access to their servers. It caused a huge crisis in the organization, and the story has come to be known as the “Load Bearing Mac Mini”.

Also, situations like this are not as uncommon as one would think:

Fragility: Exploitation

My favourite. This is my area of research. Large models are vulnerable to a plethora of attacks. Some people may have already seen (or heard of) a few simple attacks against Large Language Models (LLMs) — the prompt injection, for example. These are certain prompts that make the model do things it was not supposed to do, like circumvent the user controls or to breach the terms of use.

There are also more advanced attacks that can expose the original training data — verbatim — of the model which is a significant privacy concern, and the larger the model, the more susceptible it is to such an attack. This is a particularly applicable attack given that the general emphasis for models is “more data and larger models allows for more complex connections and relationships to be expressed” — as stated before.

This all leads to a thriving adversarial community of professionals and enthusiasts who attack these models with varying degrees of success and impacts. Though the above attacks may not seem like ways to “stop a rogue AI” (should one arise), a potent injection, or tracing the source of a piece of training data and exploiting it, or exposing the underlying mechanisms of the models for further abuse, is an active area of research, and it is producing significant results.

Final Thoughts

This non-exhaustive list of limitations shows that many of the currently claimed issues with AI are not probable in the near future. This isn’t to say that in some distant future these concerns are not pertinent, but considering today’s state of the art, it is definitely farther down the road than many would claim. My goal for this article is not to attack those who make such claims but to provide an objective lens for a highly-technical and interesting field of research.

I will close with a direct quote from OpenAI on ChatGPT’s limitations:

  • ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows.
  • ChatGPT is sensitive to tweaks to the input phrasing or attempting the same prompt multiple times. For example, given one phrasing of a question, the model can claim to not know the answer, but given a slight rephrase, can answer correctly.
  • The model is often excessively verbose and overuses certain phrases, such as restating that it’s a language model trained by OpenAI. These issues arise from biases in the training data (trainers prefer longer answers that look more comprehensive) and well-known over-optimization issues.
  • Ideally, the model would ask clarifying questions when the user provided an ambiguous query. Instead, our current models usually guess what the user intended.
  • While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behavior. We’re using the Moderation API to warn or block certain types of unsafe content, but we expect it to have some false negatives and positives for now. We’re eager to collect user feedback to aid our ongoing work to improve this system.

Hopefully this was useful in sifting through the plethora of articles with fantastical claims ranging from simple statements to claims of world-ending doom. Ignore the hype and keep a level head.

Some Interesting Reading

Standard