Book analysis with
AI techniques

Q: The presenter uses the site Project Gutenberg to obtain the text for Alice in Wonderland. Why is this site used?

A: Copyright law might prohibit the use of many modern books and songs. Project Gutenberg hosts texts that are out of copyright, often due to their age. Note that texts available on the site may not be suitable for all audiences.

Q: The presenter refers to repl.it as a Python IDE. What is an IDE, and why do we use one?

A: IDE stands for Integrated Development Environment. Unless you are coding in Windows Notepad, you are probably using an IDE. IDEs gather the tools a programmer needs to type out code, test the program with the push of a button, and more easily identify errors. Code text is usually colour-coded to help with readability, documentation is readily available and syntax errors are underlined automatically, similarly to how spelling or grammar errors are identified in a modern word processor. Advanced IDEs also provide drag-and-drop tools to create graphical user interfaces.

Q: When obtaining Alice in Wonderland, the presenter selects the Plain Text format. Why might other formats like HTML, EPUB or Kindle be unsuitable for text analysis?

A: HTML and other formats include information that is not part of the text itself. This information is used for attractive presentation, such as images, text formatting and page setup. As the name suggests, Plain Text does not have this extra information.

Q: The goal in this part is to remove all punctuation so that only words remain. Could this hinder our efforts to analyse the text for sentiment?

A: Punctuation can certainly change the meaning of a string of words. (eg. "Give to charity. Please no presents." vs. "Give to charity? Please, no. Presents!") The sentiment analysis in this learning sequence is less advanced than other approaches that might be in existence. It will rely on the volume and strength of certain words associated with other words.

Q. The presenter demarcates the string of punctuation characters using triple apostrophes at the start and end. eg, '''#%^&*'".,_-=()''' Why was this necessary?

A: Python needs something distinctive to make it clear where a text string starts and ends. Normally this is done with a quote (") or a single apostrophe (') at each end, but the string of punctuation characters already includes both these.

Q: The code for removing punctuation is placed inside a function. What are some advantages of this approach?

A: Functions allow a programmer to organise things by separating some code from the main program, so that section of code can be called (run) whenever required. Parameters then allow the main program to supply different values to the function. In the video, a parameter st is added so that any book text can be supplied to the remove_punctuation(st) function.

Q: The presenter names the function remove_punctuation(…). Are there rules about naming functions and variables in Python?

A: The naming of functions and variables is very important for code readability, so that other programmers can understand your code. One useful approach is to name functions with verbs, since they are performing an action. (Note: Using underscores in names is a Python convention rather than a rule. In other languages, like JavaScript, the convention is to use 'camel case', eg. removePunctuation(…))

Q: When writing the create_word_list(…) function, most of the work is done by a built-in string function called split(). What exactly does split() do? (Hint: you can always look up the official Python documentation, or a Python cheat sheet.)

A: Python's built-in split(…) function breaks up a string, resulting in a list that contains each separate part. If no argument is given, as in the case of creating a list of words, the string is split wherever a space occurs. A different character can be provided as an argument, such as a full stop ('.') to break up the string by sentences.

Q. During the video, the presenter decides to improve the remove_punctuation(…) function. What change is made?

A: An additional parameter called exception is added. This allows one punctuation character to be designated to not be removed along with all the others. Inside the function, the exception character is immediately removed from the punctuation string, so that it does not get removed from the text itself.

Q: In the improved remove_punctuation(…) function, what does it mean that the new parameter is written as exception=''

A: When parameters are written with an = sign, this means there is a default value. It allows someone to call the function without having to supply an argument for that parameter. In this case, exception has a default value of an empty string ''. With this value for the parameter, the punctuation string will remain intact and the removal of punctuation will work as normal.

Q: Besides cleaner main programs, what other advantages might come from this modular approach of placing groups of functions into separate files?

A: A modular approach to coding makes it easier for different programmers to work on a project. Programmer A can use a function from a file written by Programmer B without needing to see the code inside it, as long as there is documentation explaining how to use the function. Programmer B can make changes to the internal code in a function without necessarily disturbing Programmer A.

Q: In the video, the presenter uses the import statement to connect the functions from the new tokenization.py file into the main program. Have you used import to access other modules in the past?

A: A common module imported into Python programs is the random module, which provides functions for generating pseudo-random numbers like rolling a die. Students may also be familiar with the turtle module, used to issue Logo commands to an onscreen turtle.

Q: What does polarity measure?

A: Polarity is a measure of how “positive” or “negative” a block of text seems to be. Sentiment Analysis modules like TextBlob draw on large lists of words compiled from research into linguistics.

Q: What is the numerical range of the polarity value?

A: The polarity value can range from -1 (very negative) to 1 (very positive). A value of 0 suggests a block of text that is not weighted in either direction.

Q: What does subjectivity measure?

A: Subjectivity can be thought of as a measure of how emotive the language in a block of text seems to be. A high value means that the text contains a high number of adjectives and nouns that are associated with strong feelings from the writer. A low value means that the text can be thought of as more “clinical” or objective, containing fewer emotive words.

Q: What is the numerical range of the subjectivity value?

A: The subjectivity value can range from 0 (completely objective) to 1 (highly subjective).

Q: Do you think this way of analysing text sentiment is foolproof?

A: Despite its basis in linguistics research, this is still a crude means of determining sentiment when compared to a mature human’s analysis. This demonstrates the complexity of human communication. (Try inputting a sarcastic sentence, for example.) Developments in Machine Learning may result in more reliable sentiment analysis results.

Q: Why is the dictionary data structure suitable for storing each unique word in the book with its frequency?

A: The dictionary data structure consists of a set of pairs – keys and values. Our data is also a set of pairs. The key is the unique word from the book, and the value is the number of times that word appears in the book.

Q. Couldn’t two lists be used instead of a dictionary?

A: It would be feasible to use two parallel lists. One list would contain all the unique words and the other list would contain all the frequencies. This may not be considered an ideal solution because it relies on the lists always being kept in sync. For example, removing an element from one list means the same element must be removed from the other list. This leaves room for a programmer to forget, and then you have a bug.

Q: The function create_frequency_dictionary(words) has a single parameter words. What is expected to be given here?

A: The words parameter is expected to be a complete list of all words from the book, ie. the full book split up into its words. We already created a function create_word_list(…) to do this job.

Q: The function create_frequency_dictionary(words) has two sections of code inside, each with a loop. The first section builds a list of unique words from the complete list of the book’s words. What does the second section do?

A: The second section builds the dictionary by storing each unique word alongside the number of times it appears in the book.

Q: What is the purpose of the sorted(…) function used in this video?

A: The sorted(…) function is a very powerful function built into Python (see this article from the Python documentation for more). In our case, we are using it to sort the unique words from the dictionary containing the unique words and their frequencies. We sort the words from most to least common according to their frequencies. The result is not another dictionary, but a simple list called ranked_list.

Q: What is the difference between the two new functions remove_common(words) and remove_stop_words(words) we created in this video.

A: The remove_common(words) function looks for the occurrences of 1000 well-known words within the full list of words in the book, removing any that it sees. The remove_stop_words(words) function does the same thing, but it only filters out a smaller selection of words like “if”, “and”, “the”.

Q: Given a character’s name, exactly how does our new function try to guess whether that character is a hero or a villain?

A: The function measures the positivity of each paragraph in which the character appears by name. Each positivity value is used to increase or decrease an overall score. Finally, a verdict of “hero” or “villain” is pronounced based on the final score.

Q: Do you think this is an effective way to determine whether a character is a hero or villain?

A: Clearly, the function does not always get it right. It is looking for positive or negative words associated with the character by being in the same paragraph.