We are coming now … into the region where we balance probabilities and choose the most likely. It is the scientific use of the imagination, but we have always some material basis on which to start our speculation.
—— The Hound of the Baskervilles
We uncover the hidden by following clues and developing new clues from old. In the last chapter we have seen that clues are the characteristics of structures, disguised. This is why by following them, by removing the disguise, we can reconstruct the hidden structures. Since clues are so important, let's find out in this chapter a few more things about them.
From long habit the train of thoughts ran so swiftly through my mind that I arrived at the conclusion without being conscious of intermediate steps.
—— A Study in Scarlet
To help us understand how clues can lead us to things hidden, in the last chapter we have been using as example the following cryptogram.
SBR SBCTU DBCKERVS FCGG WTTCXR SFH FRRJD YTHE SHUWI
In the solution of this cryptogram, towards the end, we arrive at the following word, with two missing letters (represented here by question marks).
In a situation like this it is easy to fill in the missing letters. The word we say is SHIPMENT. The missing letters are P and M in that order. How do we know? We know from the letters already present and the context in which the word occurs.
Simple clues like this we meet with all the time. They are the same kind of clues we rely on when correcting typographical mistakes in proof-reading. When following simple clues like this our reaction is automatic, or almost so; there is little thinking involved of which we are conscious. In our example people often just look at the incomplete word with the two missing letters and then come up with the suggestion that it is, that it could be, SHIPMENT. When asked for a reason they might take a little time before they could comply. And when they do, the reasoning will be something like the following. We know from the parts of the message already deciphered that it, the message, is in English. In English there is the word SHIPMENT. SHIPMENT has all the characteristics that SHI?ME?T has except of course for the two missing letters. There is no other word that we can think of that has the same characteristics. Therefore the word in the message is likely to be SHIPMENT and the two missing letters, P and M, respectively.
Notice that the reasoning here is strictly deductive. Simplified, it is the following.
Premise 1: The incomplete word is SHIPMENT if SHIPMENT is the only word that shares the same known characteristics as SHI?PME?T.
Premise 2: SHIPMENT is the only word that shares the same known characteristics as SHI?PME?T.
Conclusion: Therefore the incomplete word is SHIPMENT.
In this piece of deductive reasoning, of the two premises the weaker one is Premise 2. Is SHIPMENT really the only word that shares the same known characteristics as SHI?PME?T? In answering this question we usually just search our own memory; we do not look up a dictionary. Since our memory is fallible we could be wrong. For this reason, in practice, when we want to be careful we say the incomplete word is likely to be SHIPMENT.
Why are clues like this so easy to figure out? Clearly it is because in these cases so much is already known. In our example, by the time we get to the word SHI?ME?T we know already that the message is in English, we know already what most of the words are. When so much is already known, it is easy to fill in the few remaining letters. In cases such as this, as the saying goes it is just a matter of putting two and two together.
It seems that figuring out simple clues is a talent most people have; it is something we do from day to day; it does not require any special training. But we should notice what we actually do when figuring out these simple clues. If we pay attention, we will see that the important part of this process is the searching of our memory, not the reasoning. We are asking ourselves, what (English) word do we know that looks like, that has the same characteristics as, SHI?ME?T? If we cannot come up with even a single word, we will not have figured out our clue. So the talent in figuring out easy clues consists mainly the ability in finding from among the many things we know the one thing that matches the clue. The quest for this one thing is the difficult part, not the reasoning. The reasoning is automatic, as automatic as adding two and two to make four.
In the case of easy clues, the clue specifies a search space and also provides sufficient information that will help us determine success. With the word SHI?ME?T the search space is made up of English words, each eight letters long. The word we are looking for should make sense within the message and also correspond to the known parts of the clue itself. This is to say, in the case of easy clues we know what general area we should be looking into and we know when we have found what we are looking for. Or to avoid misunderstanding, perhaps we should say, in the case of easy clues we are reasonably certain what general area we should be looking into and reasonably certain when we have found what we are looking for, since by 'know' we mean 'know approximately' (see last chapter).
It is of the highest importance in the art of detection to be able to recognize, out of a number of facts, which are incidental and which vital. Otherwise your energy and attention must be dissipated instead of being concentrated.
—— The Reigate Squires
We have been using our SBR cryptogram as example to help us understand the theseological process. By this time I expect readers will have become quite familiar with this example. They will have known by now that this cryptogram is not really that hard to solve. I have now a request to make. I want readers to think back to the time when the cryptogram was new to them, when they meet with it for the first time. What was the situation like then? Did they find any easy clues?
I think they are likely to say no. They may even say, at that time we do not see any clues at all! And this of course is understandable. A cryptogram is easy to solve only from hindsight ('…every problem becomes very childish when once it is explained to you ….' The Adventure of the Dancing Men). When first encountered it is much more difficult. A cryptogram appears as a string of symbols that makes no sense. Left to ourselves we would rather push it away and go do something else. Why pay attention to things that make no sense? And when we force ourselves to try to make sense of it, our first reaction is likely to be confusion and bewilderment. When we are confused; when we are bewildered; we are not likely to notice clues, let alone make sense of them.
But is this all? Is confusion or bewilderment the only reason why we find it hard to solve the cryptogram? Look back again; look back to the time after we have calmed down and are ready to set to work. Do we find clues easy to interpret then?
No; finding clues then, and making sense of the clues we have found, are still difficult. They are more difficult than towards the end.
Why should this be the case? Do we not know even then that many English words begin with TH? And that some end with LL? And how difficult is it to know that the plaintext has to be in English?
It seems we know then most of the things we know later on. We know almost all the things we need to know to solve the cryptogram even at the beginning. Yet we find it hard to locate clues and make sense of them at the beginning. Why should this be the case?
We find here an important lesson. Clues are easy to interpret when we already know a lot, but when clues are hard, it does not necessarily mean that we do not know. We could have known a lot of the things we need to know to deal with hard clues, but because we have not brought these things forward; because we have not enlisted them in the task we are faced with; the clues remain difficult. We know all along that the plaintext is in English. We know all along that many English words begin with TH. We know all along that some English words end with LL. We know all along many of the things we need to know to solve our cryptogram. But because we are not applying these things we know to the task at hand we fail to see the clues or not know what to make of them. Knowledge we already have we often keep at the back of our minds (or in our libraries). It does not always occur to us that we can make use of this knowledge to enlarge our knowledge. It does not always occur to us even when the opportunity presents itself.
To make use of the knowledge we already have to enlarge our knowledge sometimes takes a Sherlock Holmes. At the beginning of The Dancing Men, from the fact that Watson has chalk on his left forefinger and thumb, Sherlock Holmes is able to infer that Watson is not going to invest in South African securities. His reasoning as he later points out to Watson is the following:
1. You [Watson] had chalk between your left finger and thumb when you returned from the club last night. 2. You put chalk there when you play billiards, to steady the cue. 3. You never play billiards except with Thurston. 4. You told me, four weeks ago, that Thurston had an option on some South African property which would expire in a month, and which he desired you to share with him. 5. Your check book is locked in my drawer, and you have not asked for the key. 6. You do not propose to invest your money in this manner.
—— The Adventure of the Dancing Men
As Watson says, the reasoning is 'absurdly simple'. It is absurdly simple once you have all six propositions arranged in the proper order. But Watson fails to ask, why should Sherlock Holmes think of propositions 2 to 5? Watson knows of the same propositions; they are part of the things he knows; but he did not insert them between 1 and 6. Once they are inserted between 1 and 6, the reasoning is not hard at all. What is hard; what takes a Sherlock Holmes; is to gather them together and insert them between 1 and 6 in the proper order. The difference between Watson and Sherlock Holmes on this occasion is that while Watson fails to make use of the knowledge he already has, Sherlock Holmes does not.
Some clues are easy to interpret and some are hard. When we know a lot already and this knowledge includes knowing what to apply in the case at hand, clues are easy to interpret. When we do not know what to apply, clues are still difficult. Interpreting clues is often compared to putting two and two together to get four. Now getting four after putting two and two together is easy; it is the putting together that is difficult; it is the putting together that makes the art of detection into an art.1
You know a conjurer gets no credit when once he has explained his trick; and if I show you too much of my method of working, you will come to the conclusion that I am a very ordinary individual after all.
—— A Study in Scarlet
Over and over again detectives, scientists, and cryptanalysts have shown us that we can uncover knowledge of things hidden. But in the past there has been no satisfactory explanation why this can be done; it has remained a mystery. A satisfactory explanation is so much lacking that many deny that this kind of knowledge is possible. To solve this mystery some think we need a special kind of knowledge, a kind that we do not already possess, so they look for it and they look very hard. I think this view is mistaken. In this textbook I am providing a solution to this mystery. I think all the main ingredients needed for an explanation why things hidden can be known is already in our possession. It is just that no one in the past has put them together in the proper order. If we would but put them together in the proper order, the solution to the mystery is, shall we say, elementary.
I think, Watson, that you are now standing in the presence of one of the most absolute fools in Europe. I deserve to be kicked from here to Charing Cross.
—— The Man with the Twisted Lip
I confess that I have been as blind as a mole, but it is better to learn wisdom late than never to learn it at all.
—— The Man with the Twisted Lip
From hindsight it is easy to tell what knowledge we need to decipher a particular clue. But when we are faced with a clue for the first time, it may not be so easy. Sometimes the knowledge we need is already in our possession, but because we fail to make use of it, we fail to decipher the clue we want to decipher. When we find out about this afterwards, we kick ourselves.
I often think that the reason why things hidden can be known could have been discovered much earlier: all the ingredients were there, just waiting to be put together. I think we should kick ourselves.
Clues do not always pin-point. The word T?P in an English message has more than one possible solution. It could be TAP, TIP, or TOP. To find out which, may require some trial-and-error. Trial-and-error is a common practice when following clues.
When we resort to trial-and-error, we expect we will know at some point whether our trials are right. It is only because we do not know right away or in advance that we resort to trial-and-error. If we can never tell whether our trials are right, there will be no point in resorting to trial-and-error. How do we know our trials are right when following clues? We know when our trials leads to more and more new clues.
Although not many are likely to confuse trial-and-error while following clues with making wild guesses, it nevertheless may be useful to point out the difference. We make wild guesses when we are not following clues at all, the reason why wild guesses have little chance of being right. When we resort to trial-and-error while following clues, our trials have a good chance of being right. For the clues we are following will have limited the area in which the trials are made. It is only because we do not know right away or in advance which of the many possibilities within this delimited area is the right trial that we resort to trial-and-error.
Sometimes people think, that we have to resort to trial-and-error in the theseological process is an indication that we do not know what we are doing. If we know, why do we have to make so many trials? Would we trust a doctor who cures only one patient out of ten? So much, they say, for the art of detection. Only a fool (they say) will practise such an 'art'.
Those who take such a view have confused trial-and-error while following clues with making wild guesses. In the theseological process we follow clues and we develop new clues from old; we do not make wild guesses one after another. To find out which is the right answer to a clue we resort to trial-and-error. But this does not mean that we have to wait until the very end of the investigation before we can tell whether we have found the right answers to all our clues. Long for the end, we will know which of the many clues we have and have developed are likely to have been correctly interpreted. In each case, the right answer to a clue is the one that leads to more and more new clues. As we have pointed out in the last chapter, the theseological process is a homing-in process: by following clues and developing new clues from old, we get closer and closer to the truth.
A doctor does not cure by making wild guesses. If she tries to, she should be disqualified. But even qualified doctors sometimes would resort to trial-and-error. Sometimes there is more than one drug for the same disease. Which is the best for a particular patient may have to be determined by trial-and-error.
When relying on trial-and-error in the theseological process, sometimes the number of trials we have to go through can be huge, in the thousands or even millions. This means, if we want to be good theseologisers we have to be patient. An impatient person is usually not good at following clues.
When relying on trial-and-error, we are consciously putting ourselves into a situation in which we are likely to make mistakes. If we hit upon the right answer after ten trials, it means the other nine are mistakes. Now it is human nature not to want to make mistakes. Even in games in cryptanalysis we often find ourselves reluctant to take further steps simply because of this fear of mistakes. Now if we are always like this; if we always have this fear; we can never be good theseologisers. If we want to be good theseologisers we have to be willing to risk making mistakes. And not just one or two, but many.
Nowadays we often use the computer when following clues. This is because the computer can go through huge numbers of trials in a short time, thus reducing the total amount of time required to find a satisfactory solution to a clue. Also, of course, computers have no fear of mistakes.//
We very often think that we detect a clue first and then later find out what it means. But this may not always be the case. It is possible that sometimes we discover a clue and find out what it means at the same time. In cracking our SBR cryptogram many are likely to have discovered SB as a clue this way. They notice that SB could stand for TH and for this reason regard it as a clue.
Now while it is possible that sometimes we discover a clue and what it means at the same time, in theseologising we do often look for clues first before trying to figure out what they mean. And as readers are likely to know already there are some common ways by which we look for clues.
Looking for Repeating Patterns. One well-known way of looking for clues is to look for repeating patterns. In our SBR example (reproduced below), we see SB occurring twice, both at the beginning of a word. This is a pattern and we think it is a clue. Later we found out that SB stands for TH.
SBR SBCTU DBCKERVS FCGG WTTCXR SFH FRRJD YTHE SHUWI
Clues are the characteristics of structures, disguised. Within a structure there are patterns. In a message written in English there will be many THs. But if the message has been disguised, we will not see the THs. This particular pattern, like all the other patterns within the same structure, will have been transformed. But unless the pattern has been completely obliterated, it will re-appear in some other form; it will still be a pattern even though unlike the original. If, therefore, we can find these patterns and remove the disguise, we should get back the original. This is the reason why when we are looking for clues we will be looking for patterns.
But patterns could occur by chance. Not every pattern, therefore, is necessarily a clue. But when a pattern repeats itself many times, the chance that it is a clue, the chance that it is not accidental, increases. For this reason, investigators are particularly happy to see patterns that repeat themselves many times. In our example SB repeats itself only once, making altogether just two appearances. It could be a false clue. But if it is, it will not lead to new clues. And if it does lead to new clues, it is likely to be genuine. It is good to have patterns that repeat themselves many times but this does not mean a pattern that occurs only twice is useless.
Patterns that repeat themselves many times are desirable for another reason. For if they are clues, there must be reasons why. Now these reasons will apply at all the many places where the pattern occurs. This is to say, once we have figured out the reasons for these patterns, because they repeat themselves so often; because they occur at so many places; we will have by that time deciphered large parts of the unknown. But the more we have found out of the unknown, the more likely new clues will emerge. This is another reason why theseologisers like to find patterns that repeat themselves many time. If we have a long English message and we know where all the THs are, this bit of knowledge will likely lead to a large number of new clues, in words such as TH?, TH?T, TH?T?H, and so on.
Patterns can come in different ways. In our SBR cryptogram S and B occur together sometimes, but not always. But sometimes two things are such that they never occur together: if one is present the other is absent, as with Dr. Jekyll and Mr. Hyde.
Patterns can also be found in the way in which quantities vary. Instances of lung cancer are found to increase with the increase in the number of smokers, raising the question about the relation between smoking and cancer. In this case the two quantities increase at the same time, but sometimes we could have two quantities such that when one increases the other decreases, and vice versa. During the days of sailing ships it was found that the instances of scurvy among sailors decreased when their consumption of fresh fruit and vegetables increased.
Scientists sometimes use graphical patterns as clues. When we bombard a gold foil with alpha-particles the alpha-particles will be scattered by the gold atoms, producing on a screen behind the foil a scatter pattern. When this experiment was first performed, the pattern surprised scientists: it was not what they had expected. This surprising pattern suggested to them that an atom is largely empty, that is, made up mostly of space——like a miniature solar system, with the electrons circling a positively charged nucleus.
Patterns that act as clues are not always apparent to the eye. It took a Kepler to discover that the planets revolve around the sun in elliptical orbits. Now Kepler did not see these orbits. All that he had was a large set of numbers, the result of measurements made by astronomers over the centuries of the positions of the planets as seen from the earth. From this set of numbers he worked out mathematically the elliptical orbits after taking to heart Copernicus's suggestion that the planets, including the earth, revolve around the sun. Without Copernicus's suggestion it is hard to imagine what Kepler might have done. But even with Copernicus' suggestion the work was difficult and tedious: the mathematics involved was far from the kind that one could do on the back of an envelop. As we all know Kepler's work paid off: his elliptical orbits provided Newton with one of the many clues leading to his discovery of the theory of universal gravitation.
Looking for Repeating Patterns that Terminate. A pattern that repeats itself indefinitely could be a clue. But a pattern that repeats itself for a time and then stops could also be a clue. If the pattern stops, there has to be a reason. If we can figure out this reason; if we succeed in finding out what is behind the stoppage; we will have found out a little bit more about the structure from which the clue originates. In The Red-Headed League a man was employed at an office and given a good salary for doing the same simple task day in and day out. The man assumed this would go on for ever; but no, it stopped; the office was closed all of a sudden, and the owner disappeared. Thus began one of Sherlock Holmes's most interesting cases. It was interesting because here we have a repeating pattern which is unusual in the first place, but which also came to an end all of a sudden, leading everyone to ask why.
In an investigation when we see a pattern repeating itself, how do we know whether it will repeat itself indefinitely? When we see a pattern repeating itself for a time and then stops, we know it has stopped. But a repeating pattern that has not stopped so far does not mean it will go on for ever; it could stop the very next minute.
A pattern that repeats itself indefinitely and one which stops after a time could mean very different things. When we see a repeating pattern that has not stopped, it is of some importance to us whether it will ever stop. But how do we tell?
It is interesting to know that we do not have to tell——at least not right away. We can, if we wish, first form the hypothesis that the pattern will repeat itself indefinitely. We then ask ourselves, if this hypothesis is correct, what new clues will it lead to, if any? Next, we form a second hypothesis, saying the pattern will stop after a time. In this case we also ask, if this second hypothesis is correct, what new clues will it lead to, if any? Now if the first hypothesis leads to more and more new clues and the second does not lead to any new clues at all, we can then tell the first hypothesis is right, that it is likely that the pattern will repeat itself indefinitely. This is to say, we tell after we have used the pattern as a clue, not before. In other words, a repeating pattern that has not stopped is a vague clue: like most clues, it can mean more than one thing. But in an investigation we know how to deal with vague clues: we try out the different interpretations of the vague clue and see which leads to more and more new clues.
When we see a repeating pattern, there are two hypotheses we can form. One is that the pattern will repeat itself for ever and the other, that it will terminate. Which hypothesis is correct, I have said, we can find out by seeing which leads to more and more new clues. Now, of course, if we want to we can try out both hypotheses at the same time and by doing so find out which is better. As it were we treat them as runners at a race, put the two of them into the same group and let them compete directly one with the other. But obviously also, there is no necessity that we do things this way; we could take an alternate route. Instead of putting the two hypotheses into the same race, we could try one out before the other. This route has the advantage that, if we are lucky, we will save ourselves a lot of unnecessary work. For if the first hypothesis we try out leads to more and more new clues, it is likely to be right and there is no need then to try out the other. In practice sometimes we have indications as to which we should try out first. For example, Mr. Wilson in The Red-Headed League should have known that his employment could not last. The arrangement he had——being paid for doing nothing——is too good to be true.
Sometimes people say, when we see a repeating pattern we should always infer that it will repeat itself indefinitely first. Only when this does not work out should we entertain the other possibility, that is, that the pattern will terminate. I see no reason why we should always do this. Mr. Wilson should have consulted Sherlock Holmes earlier, before his employment was so rudely terminated. True, he did not know for certain that the pattern he had fallen into would come to an end, but there were enough clues to lead him to suspect that it might. And even if there were no clues, it could not have been more rational to infer that the pattern would repeat itself indefinitely. If there were no clues for either possibility, we should keep both of them in view and not arbitrarily decide one is more likely than the other.
To uncover things hidden we employ the theseological method (which tells us to follow clues and develop new clues from old). The theseological method is also used by scientists since they are also interested in things hidden, such as the structure of the atom or the structure of the universe. The theseological method therefore can also be called the scientific method. But because the understanding people have of this method is sometimes confused many things have been said about this method which are simply not true. Now that we have some understanding of the significance of repeating patterns in the search for clues, we can cite one such example. People sometimes say, the scientific method is nothing more than observing repeating patterns and then inferring that these patterns will repeat themselves indefinitely. For example, if we have observed many crows and have found that they are all black, we should (they say) infer that all crows are black. This (they say) is demanded of us by the scientific method. We can now see that what they say is not true. What they have described as the scientific method is not the scientific method. The scientific method tells us to follow clues and develop new clues from old; it does not tell us to infer that repeating patterns will repeat themselves indefinitely. Repeating patterns can be clues. Because of this they deserve attention. But there is absolutely no reason why we should expect all repeating patterns to repeat themselves indefinitely. It is most unscientific to expect, even more so to infer, that repeating patterns will do this, that is repeat themselves indefinitely. As every one knows, some do and some don't. As far as we know, all crows are black, but no one should infer that all swans are white even if they have seen a large number of swans and they are all white.
Looking for Unique Occurrences.
Singularity is almost invariably a clue.
—— The Boscombe Valley Mystery
Not only repeating patterns——terminating or not——could be clues, but also unique occurrences. There could be a reason why something should occur only once. If we know this reason, we have found out something about the structure from which the clue originates. A bank has been broken into overnight and a large sum of money has been lost. The bank manager, who is always the last one to leave after the bank closes for the day, has a habit of leaving exactly at 5:15 p.m. This he has done for years, without a single exception. But on the day of the robbery he left at 5:20. Why? Why 5:20? Why just this once?
In Silver Blaze Sherlock Holmes was investigating the disappearance of a horse favoured to win in an upcoming race. One of the important clues which led him to a solution was 'the curious incident of the dog in the night-time':
'The dog did nothing in the night-time.'
'That was the curious incident,' remarked Sherlock Holmes.
—— Silver Blaze
Dogs usually bark in the middle of the night when there are intruders. But on this occasion the dog did not bark!
Some unique occurrences can be easily overlooked. In Silver Blaze no one thought it remarkable that the dog did not bark, except of course Sherlock Holmes. But sometime unique occurrences are so exceptional that every one notices them, but they do not use them as clues. In The Hound of the Baskervilles, out of a pair of new boots a thief stole only one. Every one thought this was odd but no one thought it was related to the mystery at hand, except again Sherlock Holmes, who later was led to remark,
The more outré and grotesque an incident is the more carefully it deserves to be examined, and the very point which appears to complicate a case is, when duly considered and scientifically handled, the one which is most likely to elucidate it.
—— The Hound of the Baskervilles
In the early stages of the development of the atomic theory in chemistry, the atomic weights of most elements were found to be close to whole numbers, as the theory requires. But there was one glaring exception: chlorine. Its atomic weight was found to be very far from being a whole number. This exception did not lead to the abandonment of the atomic theory; instead, later it was used as a clue leading to the suggestion that ordinary chlorine is made up of more than one isotope, a suggestion which turned out to be true.
Clues are the characteristics of structures. This is why we look for them, in order that we can recreate these structures. But this raises a question. Because clues are the characteristics of structures, to recognise them, to find out what they mean, we have to know the structures first. If we did not know the word SHIPMENT, we would not have been able to recognise the clue SHI?ME?T and to make sense of it. Since we have to know a thing first before we can recognise its characteristics, when we follow clues, are we not merely uncovering things we already know? In our SBR example we are able to decipher the cryptogram because we know English. If we did not know English, we would not have been able to decipher it. Since we have to know English to start with, how can we say the message we obtain was hidden? Perhaps it is no more hidden than the next paragraph. You do not see it now but by the time you finish reading it, you will know what it says.
Look also to our own experience. When we follow clues, do we not often have the feeling that we know in advance what the clues are going to tell us? Now if this is not just a feeling; if it is really the case; then by following clues we cannot be uncovering anything hidden, anything that we do not already know. What is known in advance is already known. What is already known is not hidden.
Do clues really tell us anything we do not already know? When we have clues, are those things really hidden from which the clues originate? These questions naturally arise once we start thinking about clues. That they do, serves to remind us that the way clues work is more subtle than we might at first think. There are more to clues, if we may say so, than meets the eye.
If we look into the matter more carefully, we will see that clues do tell us more than we originally know. In a genuine sense, those things from which they originate are hidden. Keep in mind that clues are the characteristics of structures, disguised. What is disguised is hidden. A hunchback approaches. Who is he? 'It's me, Watson.' Before Watson recognises the voice, Watson did not know the hunchback was Sherlock Holmes; Sherlock Holmes was hidden under the disguise. It is true that Watson has known Holmes for years, but this does not alter the fact that until Watson hears the voice, he did not know that the hunchback was Sherlock Holmes. In our SBR cryptogram we see SB, we do not see TH; TH is hidden. Until we have broken the cipher; until we have peeled away the disguise; we will not know that SB is in fact TH.
In following clues, we have said in the last chapter, there is a secret, which tells us not just to follow clues but also develop new clues from old (the theseological method). Keeping in mind that we have to use this method when following clues, we can see that clues do tell us more than we already know. For, how do we find out that SB stands for TH? We do so by following the theseological method, that is, we follow clues and develop new clues from old. Notice that we need the whole method, not just a part of it. It is not sufficient that we just look at SB. SB could stand for something else besides TH. We have to develop new clues from SB if we are to find out that SB cannot be anything else but TH. But new clues are clues that we do not know before we discover them. Before we discover them, they are hidden.
Clues are the characteristics of structures in disguise. What is disguised is hidden. To find out what is behind the disguise by ourselves we need the theseological method. Holmes can say, 'It's me,' but SB does not say, 'I am TH.' When therefore we find out that SB is TH, we find out something we originally did not know, something that is hidden. We find out by following clues, many of them, old and new (see also the next section).
What one man can invent, another can discover.
—— The Adventure of the Dancing Men
Now while it is true that clues do tell us more than we already know, it is also true that to find out what they mean we need to know some things in advance. Before we can even suspect that SB stands for TH, we need to know English. Now this can appear odd. We know English already, so in this particular case there is no problem. But what about other cases, cases in which English is not involved? Where does that pre-existent knowledge come from, the pre-existent knowledge that we need before we can recognise clues? Take the investigation of nature (science) for example. What do we need to know before we can start? Where is this knowledge going to come from?
This question, where the knowledge is to come from which will enable us to recognise clues, is an important question. If we can answer it, the answer will tell us much about the way clues work. In order to bring out the importance of this question, let me bring up the case of The Shopping Basket That Is Never Empty (this is not one of Sherlock Holmes cases, but one invented by me, so please do not look for it in the canon).
This is what happens in the case of The Shopping Basket That Is Never Empty. We are in a shop; there are things we want to buy; so we take up a shopping basket and start putting things into it. Now normally when we take up a shopping basket we expect it to be empty, so that the things we put into it are the things we want to buy. But today, something strange happens. The moment we put our first item into the shopping basket, we find there is already something in the basket. Since we don't want to buy what we do not want, we change to a different basket. But the same thing happens. We can never put an item into an empty basket!
The case of The Shopping Basket That Is Never Empty is something of a nightmare. When we shop at this store, if we want to buy anything at all, we will always have to pay for something else that we do not want. Now this, it seems, and I stress the 'it seems', is what happens when we follow clues. Whenever we add to our knowledge by following clues, we find there is already some knowledge there; we can never add knowledge to an empty basket. Where does this pre-existent knowledge come from?!
Let me now try to answer this question. Let's go back to cryptanalysis again. Suppose you are a cryptanalyst, but in addition to cracking ciphers you also make up ciphers; that is to say, you are also a cryptographer. For some time now, as a cryptographer you have been working on a new cipher you have invented, a cipher about which you are rather proud. This new cipher is desirable in a number of ways: it is convenient, hard to crack, and totally unlike any other cipher known. This new cipher is hard to crack but not unbreakable. This is a trade-off all cryptographers have to make. A cipher that is convenient to use is always breakable. An unbreakable cipher is invariably clumsy (see the section Unbreakable Ciphers in the last chapter).
Your cipher is unlike any cipher already known. This is an advantage. People who want to crack your cipher will try out different kinds to see to which yours belongs. They cannot try out a kind they do not know. Known ciphers have known characteristics. Your cipher, being new, will have characteristics not familiar to those who want to crack your cipher. They therefore will have a hard time finding clues.
But you know what kind of clues your cipher will leave behind. This is one of the duties of a good cryptographer. All good cryptographers will study carefully the ciphers they have created in order to find out their weaknesses. They have to do this if they are to know to what extent they can trust their ciphers. If their cipher can be broken in one day, they do not want to make the mistake of thinking it needs two.
Tomorrow is the big day. Tomorrow you are going to put your cipher 'in the field'; you are actually going to use it. But a little while ago you have intercepted a message, encrypted of course, sent out by your enemies to their own people. You have tried to decipher this message (through cryptanalysis) but after many attempts you still have not succeeded. You have tried all the different kinds of ciphers your enemies are likely to use; you have found nothing; no clues have turned up. At this very minute you are still staring at this undeciphered message. And then, suddenly, to your horror, you start to notice in this message, in this intercepted message, first one, then a second, then a third, then more and more, characteristics of a kind that you would never dream of finding: characteristics of the kind that your own new cipher would generate!
What has happened?
What has happened is that, the coincidence of all coincidences, you and your enemies——independently of each other of course——have invented the same kind of cipher!
Will you use your new cipher tomorrow?
Not likely. Instead, you will keep very, very quiet and break theirs.
To crack a cipher we have to find out what the clues mean. This requires that we know in advance the structures from which the clues originate. Where do these structures come from? How do we come to know them? Our story about coincidental co-invention has provided us with the answer. Ultimately these structures have to be re-invented by us. After we have re-invented them we study them. This is where the knowledge of these structures comes from. Knowing these structures we know what clues to look for, as well as what they mean. When we find these clues, we then know that these structures we have created are also out there in the world, hidden. The clues tell us they are there, even though they are hidden. If we had not re-invented these structures and studied them, we would not have found these hidden structures.
From what has just been said we can see that acquiring knowledge by following clues is very different from putting things into a shopping basket. When we think about the process by which we acquire knowledge, we have a tendency to think it is similar to putting things into a shopping basket. The basket is originally empty. To this empty basket, we add more and more knowledge. But where knowledge of things hidden is concerned, this is not how it happens. We are not shopping baskets. We do not take structures hidden in the world and put them into ourselves. Shopping baskets do not re-invent structures. They do not recognise clues either. Acquiring knowledge by following clues is not a kind of transfer: things are not taken from one place and moved into another. The process is more complex and requires the ability to re-create, to re-invent.
Uncovering knowledge of things hidden, let me repeat, is not like putting things into a shopping basket. If we make the mistake of thinking that it is, we mystify ourselves. It is true that in order that we can recognise clues we have to know something in advance, but this does not make the theseological process odd, or incomprehensible. It is no more odd, strange, or incomprehensible than our story about coincidental co-invention. Our story about coincidental co-invention provides us with a way to understand the theseological process; The Case of the Shopping Basket that is Never Empty does not. To have succeeded in finding out structures hidden, as we have said all along, is to have reinvented these structures.
Inspector Gregory … is an extremely competent officer. Were he but gifted with imagination he might rise to great heights in his profession.
—— Silver Blaze
Simply as a mental exercise, without any assertion that it is true, let me indicate a possible line of thought. It is, I admit, mere imagination; but how often is imagination the mother of truth?
—— Valley of Fear
From our story about co-incidental co-invention we can see that to crack a cipher, we have to engage in invention or re-invention: we have to invent or re-invent the cipher we are trying to break. Now this is true not just of the cipher, but also of the language of the plaintext. Suppose the plaintext is in some artificial language. If we are to decipher such a plaintext we will have to re-invent the same artificial language. When we crack an English message we do not have to re-invent English only because it has already been invented and both we and the author of the message are using this invention.
To crack a cipher we have to re-invent the same cipher. To uncover the structure of the universe, do we have to re-invent the universe?
It is impossible for us to re-invent the universe; we are not God. And there is no need. When Sherlock Holmes solves a crime by re-creating it, he does not commit the crime all over again; he merely describes it in words. Similarly with the structure of the universe, although in this case scientists may have to use a language different from English or any of the natural languages (see again What Kind of Recreation in Chapter 2).
We need knowledge before we can recognise clues, knowledge of structures we ourselves have created or re-created. But this knowledge does not have to be perfect. To solve our SBR cryptogram we have to know English. Who knows English perfectly? Who knows all its characteristics? Does every one who has solved our cryptogram know English perfectly? Without knowing English perfectly we can solve our cryptogram. Without knowing anything perfectly we can uncover knowledge of things hidden.
There is a common belief that says we have to know at least one thing perfectly before we can know more. This common belief compares the acquisition of knowledge to building a house. In building a house we need a solid foundation. By analogy, according to this common belief, in acquiring knowledge we need as a first step at least one piece of knowledge that is perfect, which can then act as the foundation, on which we will build the rest of our knowledge. Without this solid foundation, no knowledge can be erected: a house without a solid foundation will collapse. Even some philosophers share this belief, so they spend years looking for this solid foundation on which knowledge is to be built. We see here this belief is mistaken, common though it is. When we follow clues with a view to adding to our knowledge, we need to know some things first, but this knowledge we need before we can add to our knowledge does not have to be perfect. Acquiring knowledge of things hidden is not like putting things into a shopping basket. Neither is it like building a house. We have to be careful with analogies (see also the section on Approximations below).
I have devised seven separate explanations, each of which would cover the facts as far as we know them. But which of these is correct can only be determined by the fresh information which we shall no doubt find waiting for us.
—— The Copper Beeches
To crack a cipher we have to re-invent it. This means, we first have to construct a cipher and then see if it correspond to the one we want to crack. If it does not, we will have to construct another, until we find one that does correspond. How do we tell whether the cipher we have constructed correspond to the one we want to crack? We do so by seeing whether the cipher we have constructed generates exactly the same clues as the cipher under investigation.
Using cryptanalysis as example, we can see that when we follow clues to uncover things hidden, we have to engage not just in construction, but in a large amount of construction. The aim in this construction is to produce a structure that can be recognised to be the same as the structure being investigated. How is the recognition to take place when, obviously, we cannot put the two structures side by side? This recognition takes place by relying on clues. When the clues generated by the two structures are the same, the structures are the same. Since the recognition has to take place through clues; since it has to be indirect; we usually have to engage in a large amount of construction before we can hit upon the right structure. If the structure we are trying to re-create were visible, we could simply copy it. But the structure we are trying to re-create is not visible; it is hidden. When this is the case we have to engage in quite some trial and error before we can find a match.
In solving our SBR cryptogram, at one point we arrived at the word TW?. This word has three letters. The first two are known, but there is a gap at the end. This gap is a clue which we can easily fill in. We say the third letter is O.
In chemistry, when the idea of the periodic table was first proposed chemists noticed that there were gaps in the table which could not be filled by the elements then known. This alerted them to the existence of these elements. And since elements in the same column in the periodic table have similar properties, they were provided with good clues as to what elements they should be looking for.
In uncovering hidden structures we often meet with gaps. Some of these gaps can act as clues, as in the examples above.
Given a three-letter word in which one is missing, we can fill in the third. Given a ten-letter word in which nine is missing, we cannot fill in the nine. The one missing letter in a three-letter word is a small gap. The nine missing letters in a ten-letter word constitute a big one. In our attempt to reconstruct hidden structures, small gaps can act as clues, but not big gaps. But this means, when we follow these clues each time we follow a clue, we can find out only a little.
In an investigation our steps are guided by clues. Since each clue can only tell us a little, in an investigation we can only take small steps, never huge ones. I have come to call this the Small Steps Principle.
The Small Step Principle: In an investigation we can only take small steps, never huge ones.
Most people who have some understanding of clues are aware of the Small Steps Principle, even though they may not have a name for it. When we find a person weaving a long, complicated story based on just one or two 'clues', most of us would say, this person is making things up.
In the Small Steps Principle, we say we can only take small steps. By 'small' we mean here not 'absolutely small', but only 'relatively small'. If we know only one letter in a ten-letter word, we cannot fill in the remaining nine. But if this ten-letter word occurs in a long message in which all the rest of the words are known, it may not be all that difficult to fill in the nine remaining letters. Nine letter in a ten-letter word is a big gap. Nine letters in a ten-letter word in a long message is a small one.
Because of the Small Steps Principle, we can often profitably make use of approximations while looking for things hidden. An approximation is not the truth but a small gap away from it. If we have an iron bar 1.01 metres long, it is an approximation to say it is one metre long. This approximation is separated by a gap of 0.01 metre from the truth. Now when we are looking for the hidden, sometimes complexity can stand in the way, making progress in the investigation difficult or even impossible. One way to reduce complexity is to resort to approximations. There is no harm in this. For in looking for the hidden we need many clues any way. Now clues can be gaps. If by introducing a few more gaps we can close the other gaps more easily, it is to our advantage to do so. When most of the gaps are closed (when most of the clues have been figured out), the additional ones can then be closed as well. As example, think of the way the planets orbit around the sun. These orbits are not exactly elliptical, only approximately so. Suppose we are interested in the question what holds the planets in their orbits, so that they will not fly off into space. What should we focus on? The exact shape of these orbits or their approximations?
In Newton's case, he first focuses on the approximations. By using the elliptical orbits as a clue, he discovers the theory of universal gravitation. After he has made this discovery, he then points out that the orbits of the planets cannot be exactly elliptical because there is not only gravitational attraction between each planet and the sun, but also between the planets themselves.
Approximations are not only permissible in an investigation, they sometimes actually help in an investigation, allowing the investigation to produce results when otherwise it might not. Once enough results have been obtained the approximations can be corrected (that is, brought closer to the truth) if need be.
We follow clues to uncover hidden structures. We do so by reconstructing the structure we want to uncover. For this we need many clues. But in most investigations, there are usually very few clues to begin with. For this reason in most investigations we have to develop new clues from old, as we have explained in Chapter 2. How do we do develop new clues from old?
If we look to our SBR example, we see that we do so by applying hypotheses to the evidence, hypotheses, that is to say, that answer to the old clues. To illustrate, let me reproduce below the first three steps in our solution of the SBR cryptogram.
We see in Step 1 that after we have suggested the hypothesis that S stands for T, we translate all the Ss in the ciphertext to Ts in the cleartext. The hypothesis that S stands for T answers to a clue. We apply this hypothesis to the evidence, which in this case is the ciphertext. The evidence is the material we have at hand, which material we can examine for clues in order that we can find out what is behind it.
Now we do the same for every hypothesis we put forward while trying to decipher the cryptogram. For example, after the hypothesis that S = T, we also put forward the hypothesis that B = H. Having done this we again go through the cryptogram and change every B into H. Now a lot of the time, by applying hypotheses to evidence in this way we do not develop new clues, but sometimes we do, as in Step 3, where we arrive at the new clue TW?. Because a new clue appears in Step 3, we are surer then that those hypotheses that give rise to it are likely to be right.
New clues are important when we are trying to uncover the hidden. They not only enable us to find out more, they also confirm what we have earlier found out. The way to develop new clues is to apply hypotheses to the evidence, hypotheses suggested by earlier clues. Isaac Newton, after he has advanced his theory of universal gravitation and his three laws of motion, applies his new discoveries to comets. As a result, he was able to derive new clues as to the kind of material to be found in comets and why some comets have tails. Now this not only increases our knowledge about comets, it also confirms his theory of universal gravitation and his three laws of motion.
In cryptanalysis we speak of cryptogram, cleartext, and cipher. In other kinds of investigations we speak of evidence, its interpretation, and theory. Since much insight can be gained by comparing investigations to cryptanalysis I have found the following three equivalences to be useful.
Evidence = Cryptogram
Interpretation of evidence = Cleartext
The correct theory = The right cipher
In cryptanalysis when we have found the right cipher we can then translate the cryptogram into the cleartext. In investigations in general when we have arrived at the right theory, this theory will enable us to make sense of the evidence. In an investigation we always have to interpret the evidence. This we do by proposing theory. A wrong theory will not be able to make sense of the evidence, only the right theory. In A Study in Scarlet one piece of evidence was the four letter RACH written in blood on the wall. Every one thought this piece of evidence significant, but what did it mean? Inspector Lestrade had a theory, so had Inspector Gregson, but the right interpretation was provided by Sherlock Holmes.
Earlier we mentioned the atomic weight of chlorine at the time when the atomic theory was revived in chemistry. This again was an interesting piece of evidence. Instead of being close to a whole number as demanded by the atomic theory, the atomic weight of chlorine was found at the time to be roughly halfway between two whole numbers. What could this mean? Does it mean that the atomic theory was wrong? Or perhaps laboratory work was sloppy when the atomic weight of chlorine was determined? Could there be some hidden factor, so far unknown, which could account for this discrepancy? It turned out to be the last. The right theory in this case has to do with the structure of the atom. All chlorine has the same number of protons in its nucleus. But different isotopes of chlorine have different numbers of neutrons, giving them different atomic weights. Ordinary chlorine is made up of two isotopes, giving it an averaged atomic weight which is roughly halfway between two whole numbers.
We commonly assume we follow clues only on special occasions. We commonly take for granted that there are many things we know for which we do not have to follow clues. We open our eyes and we see tables and chairs. Tables and chairs, these we know without following clues, we commonly think. But philosophers come along and offer arguments which show that we could be mistaken even about tables and chairs …
Can we avoid following clues if we are interested in knowledge? It seems we cannot. In looking for knowledge, there is always the possibility that we might have made mistakes. If we want to find out whether indeed we have made mistakes, we have to investigate. Could we have made mistakes even with tables and chairs? We could if there are things hidden which we do not know and therefore have not taken into account. For example the tables and chairs we see could have been created, without ourselves knowing, by 3-D holography. How do we find out whether this is the case? We carry out an investigation; which is to say, we follow clues and develop new clues from old.
Lawyers and the police (at least the better ones) know that we cannot avoid following clues. Lay people often think, once a criminal confesses, that should be the end of the matter: we know then what happened. But lawyers and the police do not react to confessions in the same way. They are more cautious: they are familiar with people making false confessions. This does not mean they do not welcome confessions, only that they will not take them at face value. Confessions themselves have to be investigated to ascertain whether they are true. A false confession cannot be corroborated by further investigation. A true confession on the other hand will make further investigation easier because it will let investigators know where the clues are, clues which they themselves have not been able to develop before the confession. Lawyers and the police welcome confessions not because confessions constitute the final word, but because confessions can lead to new clues.
The world is full of obvious things which nobody by any chance ever observes.
—— The Hound of the Baskervilles
There is nothing more deceptive than an obvious fact….
—— The Boscombe Valley Mystery
When even with table and chairs we have to rely on clues to ascertain whether they are there or not, we can safely say everything out there is hidden. Tables and chairs are hidden, atoms and molecules are hidden, the structure of the world is hidden, secret messages are hidden, other people's thoughts are hidden, …. With tables and chairs we ordinarily have no difficulty in saying they are there because the contexts in which they occur provide us with clues easy to decipher, as easy as, say, the word SHI?ME?T, or perhaps even easier.
Notice also that deciphering clues is a process that we can internalise. If we decipher the same clue over and over again, after a time it takes no effort. A woodsperson has no difficulty interpreting animal tracks. Rather, she does not interpret; she reads these tracks the way we would read a no-hunting sign. Similarly with Sherlock Holmes: tracing footsteps, as he himself says, has become second nature to him. He can tell all kinds of things by examining footsteps, including a person's height!
There is no branch of detective science which is so important and so much neglected as the art of tracing footsteps. Happily, I have always laid great stress upon it, and much practice has made it second nature to me.
—— A Study in Scarlet
Readers familiar with Sherlock Holmes will know about the following message from The 'Gloria Scott'.
The supply of game for London is going steadily up. Head-keeper Hudson, we believe, had been now told to receive all orders for fly-paper and for preservation of your hen-pheasant's life.
—— Gloria Scott
There is a second message hidden in this one. Can readers remember what the hidden message is? Did they know the first time they read Gloria Scott? How could they not know when every single word in the hidden message was in front of their eyes?
Not invisible, but unnoticed, Watson. You did not know where to look, and so you missed all that was important.
—— A Case of Identity
Things in front of our eyes can be hidden. And things hidden are sometimes clearly visible. The message in Gloria Scott is hidden, but clearly visible to those who are familiar with the cipher. Tables and chairs are in front of our eyes, but they are hidden, except that we have learnt how to see them.
Now if everything out there is hidden, the way to acquire knowledge (approximate) about the world out there is by following the theseological method. Human beings have been wondering for a long time how they can possibly have any knowledge of the world out there at all.2 If I am right about everything out there being hidden, we have now finally found the answer.3
Perhaps everything in us is also hidden. We do not necessarily know ourselves all that well, the reason why some people go to psychiatrists.
Now some will ask, if things outside and things inside are all hidden, how can we know anything at all? Do we not need at least one thing which is not hidden in order to know others?
According to the theseological method, there is no such need. The theseological method asks us to follow clues and develop new clues from old. Clues are not the sort of thing that we would normally say are as clear as daylight (except perhaps from hindsight). Yet it is by following them that we find out things hidden.
When we are engaging in an investigation; that is, when we are following clues; we often compare ourselves to people surrounded by fog. When we succeed in finding out what we originally did not know, we think of ourselves as emerging from the fog, emerging only because even then it is impossible for us to be absolutely clear what we have found, and how. Now in a fog everything is hidden.
'Wonderful!' cried the colonel. 'Wonderful! You might have been there!'
—— Silver Blaze
I have now in my hands … all the threads which have formed such a tangle. There are, of course, details to be filled in, but I am as certain of all the main facts … as if I had seen them with my own eyes.
—— A Study in Scarlet
Given a cipher to break, before we succeed we do not have in our possession a cipher that is identical to it. After we have broken the cipher, we have. For when we have broken a cipher we have in effect recreated it. Now an interesting question arises at this point. After the cipher has been broken, is the cipher hidden?
There are two answers to this question and they are both right. The first answer is that it is hidden: the original is still locked up somewhere in a safe. The second answer is that it is not hidden: we now know how this cipher works.
When we have broken a cipher, the word 'hidden' becomes ambiguous; the boundary between the hidden and the un-hidden is blurred. Now we crack ciphers by following clues. Any wonder then that when we follow clues we often have the feeling that we know all along those things we are yet to discover.
After Sherlock Holmes has solved a crime an opportunity will always occur when he has to describe to his audience what has happened during the crime. His descriptions on these occasions are so accurate and so detailed that people often have the feeling that he was there when the crime was committed.
In the last section we have pointed out that there are many things we know which seem not to be hidden. I have suggested that this is an illusion. Everything is hidden except that some we know so well that they seem not to be hidden, just as Sherlock Holmes knows so well that he seems to have been present at the scene of the crime.
Before I close this chapter, there is a practice I wish to draw attention to, the practice among scientists of speaking about clues as problems. Most people, when they talk about clues, will use the word 'clue'. Scientists use the word 'clue' too but sometimes, instead, they would use the word 'problem'. While detectives would say they are working on a clue, scientists would often say they are working on a problem. Now there is no harm in this as long as we know what is meant. However, if we do not, strange inferences can result. A detective will rejoice when a clue leads to many new clues. Since a scientist uses the word problem as a synonym for clue, a scientist will rejoice when a problem leads to many new problems. A person not familiar with this interchangeability between 'clue' and 'problem', however, will wonder, how can any person rejoice at the appearance of problems! And not just a problem, but a host of problems. Problems should be avoided, she might think; they should be prevented from happening; and if we cannot avoid them, the fewer of them the better, certainly not the more the merrier.
As I have said, there is no harm in calling clues problems so long as we know what we are doing. When we call clues problems, these are problems of a special kind. They are problems that advance our knowledge when solved. They are problem for which there are solutions if we look hard enough. A clue is puzzling until we have located the structure of which it is one of the characteristics. A clue raises questions but they are questions for which there are answers.
1In this respect interpreting a clue is like finding a proof for a mathematical theorem. In the mathematical case the theorem is given; what has to be done is to find the premises and/or the route that would lead to the theorem. But there is a difference between a mathematical proof and interpreting a clue. A mathematical proof, if valid, is conclusive. Whether the interpretation of a clue is correct has to be determined by the theseological method.
2We all know Plato's answer. We know (we have opinion about) the imperfect world out there because our soul in one of its former lives have lived in the perfect world of which the world out there is but an imitation.
3There have been other answers proposed besides mine, but all these others require that we know at least one thing perfectly, which we do not.