Using AI effectively
Modern AI is designed for finding relevant information and presenting it. They do not actually know much!
Most of the popular AI programs use large language models (LLMs) which, as their name hints, are good at analysing text for relevance to the query asked of them. They will parse the query for identifiable items or tokens that will give them a way of indexing into their vast trove of data that they have scanned. Then they will use their language skills to formulate a presentation of what they found relevant.
The fundamental problem is that they do not actually know what is true or not. To them, it is all opinion, which is what all utterances by anyone only ever are. What the AI is doing is finding what seems to be the most relevant, but if there are significantly differing opinions, they will present those as well, leaving it up to the reader to determine what is likely to be most useful, though they may provide some criteria for that.
Mostly the AI will use chunks of relevant opinions, minimising having to do too much rephrasing, which risks altering the intended meaning. While AIs are trained on a huge amount of information, that does not mean that they can understand all the ways us humans can phrase things. The information they have been trained on likely did not come with notes on how to properly interpret it, which is why they are trained on so much that they will perceive the patterns that will allow them to come up with internal rules that will guide them to how to interpret future text as it was intended to mean.
Between not actually knowing what is true of all the opinions they have in their information bank, and that they do not actually know if they have interpreted correctly what they determine might be relevant, us placing blind trust in them giving us what we want to know is unwarranted. Certainly, their core skill of wielding language is extremely helpful, but what sort of answers about nuclear physics can we expect to get from a language professor? We would expect that they could find relevant but superficial material, but if hard-pressed to explain more deeply, they will be lost.
This all means that in order to get to anything other than the most simple of explanations, we need to know the subject we are asking about so that we can drill down into the AI's answers to get at what it is we really want to know, and have a chance of knowing that those answers are likely to be true. Where AI comes into its own is that conversations can be had to drill down into the details. They can be challenged, which will often force them to find other information. Unfortunately, some may not have recent information, which may limit their usefulness for some queries.
For doing comprehensive and complete research, AI leaves the crudeness of search for dead. AI tends to want to be indicative, so will just give the first few examples that match the query, but it can be pressed to list all matches, though that may involve providing an explicit list, and telling it to use the complete list. This iterative conversational drilling down to specifics is what really sets AI apart.
Providing additional specific words to a previous search may improve its response, but it is not interactive, as it cannot reference what has been queried immediately before, making it very limited for comprehensive research. There is no opportunity to explore some aspects while still keeping the whole context to get back to to explore other aspects or alternatives. AI is far more useful tool because of that.
Practical advice
△Being LLMs seems to make AI want to use as much language as possible, but it can be tamed.
People are using AI for many things, but the verbosity of their output, and that they give such detailed seemingly complete expositions, would indicate that they are mainly aimed at doing the work for the asker. This does not work for those who want to maintain total creative control over their development of their product, so the AI needs to be restricted in the extent and type of information it provides. While getting code samples may be useful for many, using them in a product may pose copyright risks.
AIs also tend to regenerate the whole of their answers with each tweak made in response to questions seeking clarification. These can really bulk out a whole discussion thread, making if more difficult to find snippets of information earlier in the thread. When YouTubers are showing how they use AI, they will have a set of stock phrases they enter at the start of a session so they just get what they want. In a way, AI is the extreme opposite of the sparseness of search results.
Using some of the phrases below will keep discussions focussed, and minimise the risk of incorporating someone else's IP. Unfortunately, the Copilot update in September 2024 made it so inflexible that all discussions are in one thread, making restricting context difficult, or they can all be deleted. When not logged in, the web ChatGPT forgets the context when the session is closed. Its individual responses can be saved, but the text is plain with liberal amounts of markup symbols.
ChatGPT's individual responses can be saved, but the text is plain with liberal amounts of markup symbols. If wanting to preserve the whole thread, cut-and-paste the wanted content from the web page into a word processor, and it will include formatting, though unlike Copilot, code snippets will not be colour-coded.
Ask the AI what it prefers - Every AI is designed differently, so rather than rely upon some stock phrases, ask it what phrases are best to use with it to get the type of answers required. Just be aware that there is no guarantee that it will reply consistently. Every question is an opportunity for them to find another way to answer, as they are learning as they go along.
confine answers to … - This introduces the clear scope of any answer.
for this question only, . . . △- At some point, some more detail may be needed that might be precluded by some of the previous phrases. This allow specifying which point(s) to use as the basis and scope for more detail, such as code examples (not solutions).
from now on, … - This indicates the scope or level of what is to be part of the ongoing discussion. This is useful to start drilling down into what may be the real questions to be asked after establishing the context of the topic enquiry. Specific points from earlier on can be referenced, but hopefully the AI will not take that as an opportunity to get all verbose and eclectic again!
no code examples - This prevents possible copyrighted material bring used in a our own work, but keeps the discussion high-level so that technical options can be explored without jumping to specific, perhaps premature, implementations of what may not be properly understood enough to be sure it will address what is required.
summary points only △- If given the chance AI will produce pages of multiple versions of the answers, with the last being a summary. This will ensure only the summary is produced. If more detail is required, ask for just the points to clarify.
the context for this discussion is … - This allows inserting the conclusions or summaries from previous discussions, to be used as the ongoing context to avoid rehashing those discussions, and just quickly move to what is wanted now.
tl;dr concise numbered points only - The TL;DR is so that answers are not filled with unnecessary details from the start, especially if some discussion is needed before getting into details. The numbering of their points is so that the next question can refer to just what is specifically relevant, or address concerns with them, point by point.
Even with these phrases, the AI may decide to lose the current context and revert to its default behaviours, like giving detailed solutions. Just point out what in their answer is not wanted, or that they had promised they would not do. AI does not care about capitalisation, so it is not shown here, and they are fairly tolerant to spelling and grammatical errors, but note that they can easily veer off course if the information they are given is possibly ambiguous due to misspellings. This is also why accurate punctuation placement may still be required.
AI is very open to being challenged or having one's own opinions fed back to them. This certainly makes AI far superior to dealing with the increasing numbers of sarky moderators and other answer-police pulling rank on
Ignore the verbiage
△Trying to control an AI's output doesn't always work.
The temptation to think we can control what an AI outputs is strong, but at some point, between the natural verboseness of the underlying technology, and the forgetfulness of it to adhere to what it has been asked to do, we may come to the realisation that the AI is just not going to cooperate in our efforts to contain it. That only leaves one choice: ignore what is not needed.
To ignore what has obviously taken a lot of computing power seems like such a waste, but repeatedly trying to verbally beat the AI into cooperating is such a waste of time, and can be a huge distraction from actually solving the problem that was the subject of the discussion with it. An AI will tend to produce several waves of verbiage, each block of it for a specific purpose, with the final being a summary or recommendations, at least for open-ended discussions. Queries aiming for specific output will tend to converge on that, but research and discussion produces the multi-layered output.
Once having worked out the format of the output for the type of discussion, it simply comes down to ignoring what is not required. That may mean going straight to the summary. If it seems to have got something wrong, it may help to scan the text beforehand to see its reasoning. When the points of misinterpretation or misassumptions are found, ignore everything else, and, specifically referring to those points, challenge it over them.
Do not waste time on trying to deeply analyse why it is wrong, or reading anything else of what was displayed beforehand, but just ask clarifying questions or correct the statements, and move on to its next swathe of answers. Repeat each cycle until the AI gives a summary that encapsulates what was required. The real skill required is in being able to pick the points at which the AI has got it jumbled up, and quickly push it to be on track again. This should substantially short-circuit wasting time trying to stop the verbiage.
AI is not brilliant
△AI is supposed to be intelligent, but it seems to want to follow traditional thinking approaches.
In using AI to work through ideas, a lot of its responses seems to be a rehash of fairly conventional thinking and approaches. I have been using AI as a sounding board for ideas, but I found that it did not come up with anything novel or unique. When people do not have their own selfish agendas at meetings, they can be a very good venue for riffing on ideas, as I find I can iterate through ideas quickly from the discussions, often making some quantum leaps in how a problem can be approached and solved.
Unfortunately, AI is very systemic in its approach, by not venturing into anything left-field. I doubt they can see anything left-field at all, finding that I was the one coming up with anything novel. Of course, I cannot necessarily explain all the details of an issue, so I would be more likely to see possibilities more related to the exact nature of the issue, but many times I came up with significant steps in finding a solution to a problem that is elegant and simple. The AI would tend to go through conventional systemic approaches that were often more complex than needed.
I had to keep arguing about why their approach was too complex or missed what seemed like obvious simplifications or streamlining. This probably reflects the quality of material they are trained on, most likely largely obtained from papers or documentation from enterprises, which tend to favour formal processes and programming approaches rather than left-field. This is not to say that using AI is not extremely beneficial, but just to not expect brilliance, and resign to being the one who will bring the useful ideas to the discussion.
It is fantastic to have such tools that do bring up possibilities, though mostly conventional, that can be improved during the discussions, especially for solo operations that are starved of people to nut through things with. I have been able to make several significant improvements to the product using the AI to bounce ideas off, but they have all been exclusively my ideas that have been the significant basis of any changes or new functionality, from the original discussion topic idea, to what each subsequent question or comment I make adds to the uniqueness of the product.
Certainly, having the AI work through the ramifications of my suggestions is very useful, but can be frustrating when they keep trying to push back towards traditional thinking, while completely missing the obvious improvements that could be made to what they do suggest. It is like they can only focus on one aspect at a time, while I have to be the one dovetailing multiple ideas or improvements into coherent changes that are much more than just tweaks, but significant improvements that are frugal in their design and implementation.
If left to the AI, each change would be complicated technical overkill with very little real optimisation, even though it often claims that it is doing so. Optimisation is much more than just offering a marginal tweak, but integrating design and functionality to produce an efficient, minimalist, integrated and optimal implementation. For all the intelligence the AI is supposed to have, they seem to lack the clarity to integrate multiple optimisations at the same time. I would have thought that AI should be able to riff on their own suggestions, but they play it safe one change at a time.
This result in their responses being a laundry list of isolated suggestions without any attempt at how they could be blended together. The AI will be glowing about the advantages of suggestions I make, but I wonder why they did not make what were often fairly obvious optimisations. For example, to one suggestion of theirs that use files, I suggested a more functional naming of them than the generic ones offered. The AI then responded with how useful the names were. If they were so useful, why did the AI not use them. This is the type of missed opportunity that AI could be fulfilling.
We know that AI is eating into the client base of Stack Overflow (SO), but that is really because the AI is very tolerant whereas SO has become quite toxic. If AI is not really optimising what they get from the internet they trained on when offering answers to questions, then we will not get the exponential improvements that AI advocates claim we will. Tweaks do not do that when considered in isolation.
Overall, AI is a significant tool for those who need a way of riffing on ideas, even if the AI is not as brilliant as many make them out to be, and the human in the room is doing all the real work to make a product excel. The large language model (LLM) AIs have a ways to go before they can be really trusted to be in charge of automation or critical path processes, which require continuous attention to optimising relevance and mitigating against distractions that undermine seeing useful possibilities.
Improving the models
△There are some ways that models could be improved.
LLMs scan the learning data and take it all as given. They have no way of knowing the veracity of that data. So they just grab what sounds good linguistically, but it may be wrong in part or all of it. Challenging them will force them to go to other sources, but they may be only be a little bit better. However, as humans we know that legislation, programming language syntax and engineering standards are authoritative and so should be used in preference for answers than what someone writes on their blog, especially if their content does not parse to align with such references.
The other side of authoritativeness is endurance. All such references can be amended over time, so old versions will only be relevant to the time context of the prompt. The issue with all of this is that the only way to properly tag sources is to use experts to tag all sources, and that would have to be double-checked. Given the large number of pages on the web, that is not practical. While rules might be defined so that machine learning could automatically rank much of the masses of of data, people will probably work around them.
But endurance is not the only time-sensitive aspect of LLMs. While they can parse tense in text, when it comes to prompts, they cannot interpret tense in them properly so that answers do not make suggestions for already completed or implemented works, currently giving verbose suggestions for how to implement what has obviously already been done as we generally do not want to get our prompts being verbosely second-guessed when a person could easily know that from how the prompt is expressed.
LLMs are programmed to be helpful by default but that needs to be tempered with what the prompter is actually saying needs to be replied to. This would cut down a lot of the verboseness of answers that would have to be ignored if not really relevant to the prompter. This improvement suggestion would be easier to implement as it does not require backfitting the learning data by people that requiring authoritativeness tagging does.