Programming a chatterbot - understanding language [closed]

Natural language processing is incredibly difficult, because human languages are so ambiguous, and require a lot of general background context to make sense of what is being said. Consider the following sentences:

1. "Monkeys like bananas when they wake up"
2. "Monkeys like bananas when they are ripe"
3. "I saw the lab technician with the microscope."

In (1) and (2) does the "they" refer to the monkeys or the bananas? Well, as a human being it's clear to me that in (1) the "they" refers to the monkeys, whereas in (2) it refers to the bananas. But how exactly did I know that? The information isn't inherent in the grammar of the sentence - it's only available because as a human being I have access to a large database of background knowledge in my brain, which tells me that bananas don't ever "wake up", and that monkeys aren't "ripe". Therefore, my brain provides the contextual information to parse the sentence properly.

In (3) we see a similar problem: Does (3) mean that I saw a lab technician who was using a microscope, or does it mean I actually used a microscope to see a lab technician? Grammatically speaking, it could mean either. But to a human being there is no ambiguity. It must mean the former, because the latter is absurd. But the only reason that I even realize that the latter is absurd is because I know what a microscope is, I know that a lab technician is a person, I know that you wouldn't use a microscope to see a person, and I know that lab technicians often work with microscopes. So my brain synthesizes all of that background knowledge, allowing me to interpret the sentence correctly.

Now imagine trying to parse sentences like these with a computer. You'd need a vast database (known as an ontology) of real-world facts and relationships to provide context, and a pretty sophisticated parser. There have been some attempts to do this in the past, but it's still a very theoretical, very research-oriented endeavor.

So far there are at least 50 years of research in natural language understanding. You aren't going to solve it all by yourself, and choosing a programming language is about one-trillionth of the effort. Look at the effort that went into building IBM's Watson. They didn't even attempt to solve the general problem of language understanding. They only had to understand clues well enough to formulate a probable answer. Automatic translation systems are another good reference. After hundreds of man-years of effort we have useful automatic translators, but nothing good.

Here's a helpful link: http://ai-programming.com/bot_tutorial.htm

That's where I learned a lot of techniques I'm using in my own chatterbot called Quanda Sparks. Although organizing, analyzing, and creating common language algorithmically is a complex subject, and so far no bot has passed the Turing Test (as far as I know), there are a few ways to mimic human speech.

Template Responses: This is a way to make it seem as if a robot understands by having it talk of what the human inputted. Let's say I type in "I like tacos and enchiladas." The bot responds with "What is so great about tacos and enchiladas?" How did it arrive with this response? When it sees "I like", it takes whatever follows and makes a template, in this case "tacos and enchiladas." Then it can construct a sentence with "What is so great about " + template. The robot can fool its audience without ever even know what a taco is.

Generic Responses: Give the bot lots of responses that could fit in almost any scenario. Some examples include "okay", "That's cool", "I hear you", etc. That way, if the bot has nothing to say, it still has a chance at keeping up the act.

Thesaurus: Let's say you have plenty of responses looking for the word "awesome", but the user enters in "amazing". Normally, the bot would be stumped and would have to resort to one of its generic responses; however, if you replace all instances of the word "amazing" with "awesome", it will still be able to find a good response.

Foil cheaters' plans: Users will try all they can to fool the bot into revealing its weaknesses in speech. Some things they will try: leaving the input field blank, entering gibberish, repeating the same thing twice, mimiking the bot, and putting words out of place, such as "you are a bot who". Make sure you give good responses under these circumstances. In the last example, you could check to make sure that there is a word after "who" before telling the bot the user asked a question.

Speak in context: This is the hardest part and the main reason bots fail the Turing Test. Chatterbots can only pretend to understand by giving responses that make sense. If you ask a chatterbot to remember the number 490, talk for a while, and then ask the bot what the number was, most bots will have no clue what you are talking about. Most good bots can look at least one or two lines back. Make sure you also make your bot context-sensitive otherwise.

Do not repeat responses: When you talk to a person, does he usually say "Oh, that's cool ... oh, that's cool ... oh, that's cool" like that three times in a row? No. Even if he is not really interested, he will change up the words like this: "oh, that's cool ... awesome ... that's good." Make sure your bot also waits a while before repeating itself.

Learn from mistakes: It's hard to make the robot learn by itself, but you can teach it more efficiently like this: have people converse with it. Have it record the conversation and any input it does not have a response for. Then, collect this data and you it to build more responses.

I hope these ideas can help you get started. As for what technologies you would need for this, I am programming mine in Java and storing it's knowledge base in a text document. You really don't need anything past that. I was originally going to integrate in an SQL database, but soon found it wasn't worth it. You don't need any fancy APIs. Just use your head and problem-solving ability. If you want to be able to program conversation, you need to understand how conversation works. Study real conversations and look for patterns. When you find one, integrate it.