This chapter is devoted to the data analysis and working with files. We will learn about exceptions, how to handle errors so taht your programs don't crash when they encounter unexpected situations.
File handling will significantly improve your programs. They will be able to save their data, enter it and store it and then reuse it again, when they decide to run your program again.
Learning to handle exceptions whill help your programs deal with situations when files are missing or don't exist and other problems that can cause your programs to crash.
Reading from a file is useful not only in data analysis, but when you want to modify, edit stored information in a file. You can write programs that could read the contents of a text file and edit the file with formatting that allows a browser to display it.
When you want to work with the data from a text file, first thing to do is to read the file into your computer's memory. You can read the entire contents of a file, or you can work through the file one line at a time.
Here's a program that opens the file pi_digits.txt, reads it, and prints the contents of the file to the screen:
#file_reader.py
with open('pi_digits.txt') as file_object: # Python assigns the file to file_object
contents = file_object.read()
print(contents)open() - open the file to access it. open() function needs one argument: the name of the file you want to open. Python looks for this file in the directory where the program that's currently being executed is stored.
with - keyword closes the file once access to it is no longer needed. Notice how we call open() in this program but not close(). You could open and close the file by calling open() and close(), but if a bug in your program prevents the close() method from being executed, the file may never close. Improperly closed files can cause data to be lost or corrupted, also if you call close() too early, you may try to work with a closed file that you don't have access to anymore, which will also lead to errors. That is why we use with and trust Python to close the file after the with block finished execution.
read() - once we have a file object that represents the text file we have indicated (pi_digits.txt), we use read() method to read teh entire contents of the file and store it as one long string in contents. When we print the value of contents, we get the entire text file back.
The output would look like that:
3.1415926535 8979323846 2643383279 The only difference between this output and the original file is the extra blank line at the end of the output. The blank line appears because read() method returns an empty string when it reaches the end of the file; this empty string shows up as a blank line.
If you want to remove the extra blank line, you can use rstrip() in the call to print():
with open('pi_digits.txt') as file_object:
contents = file_object.read()
print(contents.rstrip())When you pass a simple filename like pi_digits.txt to the open() function, Python looks in the diretory where the file that's currently being executed (that is, your .py program file) is stored.
Sometimes, depending on how you organize your work, the file you want to open won't be in the same directory as your program file.
You could use a relative file path to open a file:
with open('text_files/filename.txt) as file_object
you can also tell Python exactly where the file is on your computer regardless of where the program that's being executed is stored. This is called absolute file path. Because absolute paths are usually longer than relative paths, it is helpful to assing them to a variable and then pass that variable to open():
file_path = "/home/user/other_files/text_files/filename.txt"
with open(file_path) as file_object:Using absolute paths, you can read files from any location on your system.
If you try to use backslashes in a file path, you’ll get an error because the backslash is used to escape characters in strings. For example, in the path "C:\path\to\file.txt", the sequence \t is interpreted as a tab. If you need to use backslashes, you can escape each one in the path, like this: "C:\path\to\file.txt".
You can use a for loop on the file object to examine each line from a file one at a time:
filename = 'Chapter10/pi_digits.txt'
with open(filename) as file_object:
for line in file_object:
print(line.rstrip()) # user rstrip() to get rid of extra lines generated by print functionWhen you use with, the file object returned by open() is only available inside the with block that contains it. If you want to retain access to a file's contents outside the with block, you can store the file's lines in a list inside the block and then work with that list. You can process parts of the file immediately and postpone some processing for later in the program.
filename = 'Chapter10/pi_digits.txt'
with open(filename) as file_object:
lines = file_object.readlines() # readlines() take each line from the file and store it in a list.
for line in lines: # print each line from lines.
print(line.rstrip())After you've read a file into memory, you can do whatever you want with that data, so let's brifly explore the digits of pi.
First, we will build a single string containing all the digits in the file with no witespace in it:
filename = 'Chapter10\pi_digits.txt'
with open(filename) as file_object:
lines = file_object.readlines()
pi_string = '' # create a varible to hold the digits of pi
for line in lines: # create a loop to add each line of digits to pi_string
pi_string += line.rstrip() # and remove character from each line
print(pi_string) # print the string
print(len(pi_string))When Python reads from a text file, it interprets all text in the file as a string. If you read in a number and want to work with that value in a numerical context, you’ll have to convert it to an integer using the int() function or convert it to a float using the float() function.
The code that we have written in our examples so far would work just as well on much larger files.
filename = 'Chapter10\pi_million_digits.txt'
with open(filename) as file_object:
lines = file_object.readlines()
pi_string = '' # create a varible to hold the digits of pi
for line in lines: # create a loop to add each line of digits to pi_string
pi_string += line.rstrip() # and remove character from each line
print(f"{pi_string[:52]}...") # print the string
print(len(pi_string))Python has no inherent limit to how much data you can work with; you can work with as much data as your system’s memory can handle.
I've always been curious to know if my birthday appears anywhere in the digits of pi. Let's write a program that will find out if someones birthday appears anywhere in the first million digits of pi.
filename = 'Chapter10\pi_million_digits.txt'
with open(filename) as file_object:
lines = file_object.readlines()
pi_string = '' # create a varible to hold the digits of pi
for line in lines: # create a loop to add each line of digits to pi_string
pi_string += line.rstrip() # and remove character from each line
birthday = input("Enter your birthday, in the form mmddyy: ")
if birthday in pi_string:
print("Your birthday appears in the first million digits of pi!")
else:
print("Your birthday does not appear in the first million digits of pi.")One of the simplest ways to save data is to write it to a file. When you write text to a file, the output will still be available after you close the terminal containing your program's output. You can examine output after a program finishes running, and you can share the output files with others as well.
You can also write programs that read text back into memory and work with it again later.
To write text to a file, you need to call open() with a second argument telling Python that you want to write the file.
Example:
filename ='Chapter10\programming.txt' # file adress
with open(filename, 'w') as file_object: # we use second argument 'w' to open the file in WRITE MODE.
file_object.write("I love programming") # use write() method on the file object to write a string to the file.You can also open a file in read mode -> ('r'), write mode -> ('w'), append mode -> ('a'), or a mode that allows you to read and write to the file -> ('r+')
When you omit the mode argument, Python opens the file in read-only mode by default. Python can only write strings to a text file. If you want to store numerical data in a text file, you'll have to convert the data to string format first using the str() function.
The write() function doesn't add any newlines to the text you write. So if you write more than one line without including newline characters, your file may not look the way you want it to:
filename ='Chapter10\programming.txt' # file adress
with open(filename, 'w') as file_object:
file_object.write("I love programming.\n") # just using write method will not add the string into a new line
file_object.write('I love creating new games.\n') # use \n at the end to add new linesYou can also use spaces, tab characters, and blank lines to format your output, just as you've been doing with terminal-based output.
If you want to add content to a file instead of writing over existing content, you can open the file in append mode. When you open a file in append mode, Python doesn't erase the contents of the file before returning the file object. Any lines you write to the file will be added at the end of the file. If the file doesn't exist yet, Python will create an empty file for you.
filename = 'Chapter10\programming.txt'
with open(filename, 'a') as file_object: # 'a' argument to open the file for appending rather writing over the existing file.
file_object.write("\nI also love finding meaning in large datasets.\n") # write two new lines, which are added to programming.txt
file_object.write("I love creatig apps that can run in a browser.\n")Python uses special objects called exceptions to manage errors that arise during a program's execution. Whenever an error occurs that makes Python unsure what to do next, it creates an exception object. If you will not code that will handle the exception condition when it's met, then the program will halt and show a traceback, which includes a report of the exception that was raised.
Exceptions are handled with try-except blocks.
print(5/0) # Python will throw a ZeroDivisionError error as it can't divide int by zerooutput will looklike:
Traceback (most recent call last):
File "division_calculator.py", line 1, in
print(5/0) u ZeroDivisionError: division by zero
Instead of running into this error we could write:
try:
print(5/0) # Python will throw an error as it can't divide int by zero
except ZeroDivisionError:
print("You can't divide by zero!") # print this when error occursIf more code followed the try-except block, the program would continue running because we told Python how to handle the error.
Handling errors correctly is especially important when the program has more work to do after the error occurs. This happens often in programs that prompt users for input. If the program responds to invalid input appropriately, it can prompt for more valid input instead of crashing.
print("Give me two numbers, and I'll divide them.")
print("Enter 'q' to quit.")
while True:
first_number = input("\nFirst number: ")
if first_number == 'q': # 1
break
second_number = input("Second number: ") # 2
if second_number == 'q':
break
answer = int(first_number) / int(second_number)
print(answer)This program prompts the user to input a first_number (#1) and, if the user does not enter q to quit, a second_number (#2). We then divide these two numbers to get an answer (#3). This causes following output:
Traceback (most recent call last): File "division_calculator.py", line 9, in answer = int(first_number) / int(second_number) ZeroDivisionError: division by zero
We can make this program more error resistant by wrapping the line that might produce errors in a try-except block. The error occurs on the line that performs the division, so that's where we'll put the try-except block.
print("Give me two numbers, and I'll divide them.")
print("Enter 'q' to quit.")
while True:
first_number = input("\nFirst number: ")
if first_number == 'q':
break
second_number = input("Second number: ")
if second_number == 'q':
break
try: # 1
answer = int(first_number) / int(second_number)
except ZeroDivisionError: # 2
print("You can't divide by 0!")
else: # 2
print(answer)#1 - we ask Python to try to complete the division operation in a try block, which includes the code that might cause an error. #2 - The except block tells Python how to respond when a ZeroDivisionError arises. If the try block doesn't succeed because of a division by zero error, we print a friendly message telling the user how to avoid this kind of error. #3 - If the division operation is successful, we use the else block to print the result.
try-except-else block works like this: Python attempts to run the code in the try block. The only code that should go in a try block is code that might cause an exception error. The except block tells Python what to do in case a certain exception arises when it tries to run the code in the try block.
By anticipating likely sources of errors, you can write robust programs that continue to run even when they encounter invalid data and missing resources. Your code will be resistant to innocent user mistakes and malicious attacks.
One common issue when working with files is handling missing files. The file you're looking for might be in a different location, the filename may be misspelled, or the file may not exist at all. All these situations can be handled with a try-except block.
Example:
file_name = "Chapter10/alice.txt"
with open(file_name, encoding='utf-8') as f:
contents = f.read()Python can't read the from a missing file, so it raises an exception:
Traceback (most recent call last):
File "alice.py", line 3, in
with open(filename, encoding='utf-8') as f: FileNotFoundError: [Errno 2] No such file or directory: 'alice.txt'
Because error occurs when open() function is at play we will use try block to handle it:
file_name = "Chapter10/alice.txt"
try:
with open(file_name, encoding='utf-8') as f:
contents = f.read()
except FileNotFoundError:
print(f"Sorry, the file {file_name} does not exist.")You can analyze text files containing entire books. Many classic works of literature are available as simple text files because they are in the public domain.
To count the number of words in Alice in Wonderland book we'll use split() method on the entire text. Then we'll count the items in the list to get a rough idea of the number of words in the text:
file_name = "Chapter10/alice.txt"
try:
with open(file_name, encoding='utf-8') as f:
contents = f.read()
except FileNotFoundError:
print(f"Sorry, the file {file_name} does not exist.")
else:
# Count the approximate number of words in the file.
words = contents.split()
num_words = len(words)
print(f"The file {file_name} has about {num_words} words.")Let's add more files to analyze.
def count_words(file_name):
"""Count the approximate number of words in a file"""
try:
with open(file_name, encoding='utf-8') as file:
contents = file.read()
except FileNotFoundError:
print(f"Sorry, the file {file_name} does not exist.")
else:
words = contents.split()
num_words = len(words)
print(f"The file {file_name} has about {num_words} words.")
file_names = ['Chapter10/alice.txt', 'Chapter10/guest_book.txt', 'siddhartha.txt']
for file_name in file_names:
count_words(file_name)Output:
The file Chapter10/alice.txt has about 30360 words.
The file Chapter10/guest_book.txt has about 42 words.
Sorry, the file siddhartha.txt does not exist.
As you can see the try-except block in this example provides significant advantages. We prevent our users from seeing a traceback message, and we let the program to continue analyzing the text files it was able to find. I we didn't use try-except block then our program would raise FileNotFoundError and stop running.
Python has pass statement that tells it to do nothing in a block:
def count_words(file_name):
"""Count the approximate number of words in a file"""
try:
with open(file_name, encoding='utf-8') as file:
contents = file.read()
except FileNotFoundError:
pass # using pass statement to do nothing
else:
words = contents.split()
num_words = len(words)
print(f"The file {file_name} has about {num_words} words.")
file_names = ['Chapter10/alice.txt', 'Chapter10/guest_book.txt', 'siddhartha.txt']
for file_name in file_names:
count_words(file_name)The pass statement also acts as a placeholder. It's a reminder that you're choosing to do nothing at a specific point in your program's execution and that you might want to do something there later. For example, in this program we might decide to write any missing filenames to a file called missing_files.txt . Our users woudn't see this file, but we'd be able to read the file and deal with any missing texts.
Many of your programs will ask users to input certain kinds of information. You might allow users to store preferences in a game or provide data for a visualization. Whatever the focus of your program is, you'll store the information users provide in a data structures such as lists and dictionaries.
When the user closes a program, you'll almost always want to save the information they entered. A simple way to do this involves storing your data using the json module.
The json module allows you to dump simple Python data structures into a file and load the data from that file the next time the program runs. You can also use json to share data between different Python programs. Even better, the JSON data format is not specific to Python, so you can share data you store in the JSON format with people who work in many other programming languages.
The JSON (JavaScript Object Notation) format was originally developed for JavaScript. However, it has since become a common format used by many languages, including Python
Let's write a short program that stores a set of numbers and another program that reads these numbers back into memory.
import json
numbers = [1, 3, 5, 7, 11, 13]
filename = 'Chapter10/numbers.json' # we choose a filename to store the numbers.
with open(filename, 'w') as f: # We open the file in write mode, which allows json to write data to the file.
json.dump(numbers, f) # we use json.dump() function to store the list numbers in the file numbers.jsonNow we'll write a program thta uses json.load() to read the list back into memory:
import json
filename = 'Chapter10/numbers.json' # file address
with open(filename) as file: # open the file in read mode
numbers = json.load(file) # json.load() load the information stored in numbers.json and assign in to the variable names
[print(numbers)]Saving data with json is useful when you're working with user-generated data, because if you don't store your user's information somehow, you'll lose it when the program stops running.
Example where we prompt the user for their name the first time they run a program and then store their name when they run the program again:
#remember_me.py
import json
username = input("What is your name?\n") # prompt for a username to store
filename = "Chapter10/username.json"
with open(filename, 'w') as file:
json.dump(username, file) # use json.dump(), pass username and a file as an object, to sore it in a file
print(f"We'll remember you when you come back, {username}!") # print message, informing the user that we've stored their informationOther program that greets a user whose name has already been stored:
# greet_user.py
import json
file_name = "Chapter10/username.json"
with open(file_name) as file:
username = json.load(file) # user json.load() to read info in username.json
print(f"Welcome back, {username}!") # welcome userLet's combine both programs in to one with the help of try-except-else block:
import json
# Load teh username, if it has been stored previously.
# Otherwise, prompt for the username and store it.
filename = "Chapter10/username.json"
try:
with open(filename) as file: # open usernam.json, if the file exist read it into memory
username = json.load(file)
except FileNotFoundError: # If file doesn't exist we prompt the user to enter their username
username = input("What is your name?\n")
with open(filename, 'w') as file: # we use json.dump() to store the username and print a greeting
json.dump(username, file)
print(f"We'll remember you when you come back, {username}!")
else:
print(f"Welcome back, {username}!") # welcome back if username existsOften you'll come to a point where your code will work, but you'll recognize that you could improve the code by breaking it up into a series of functions that have specific jobs. This process is called refactoring. Refactoring makes your code cleaner, easier to understand, and easier to extend.
Here's a refactored version of the program above:
import json
# Load the username, if it has been stored previously.
# Otherwise, prompt for the username and store it.
def get_stored_username():
"""Get stored username if available."""
filename = "Chapter10/username.json"
try:
with open(filename) as file: # open usernam.json, if the file exist read it into memory
username = json.load(file)
except FileNotFoundError: # If file doesn't exist we prompt the user to enter their username
return None
else:
return username
def get_new_username():
"""Prompt for a new username."""
username = input("What is your name?\n")
filename = 'Chapter10/username.json'
with open(filename, 'w') as file:
json.dump(username, file)
return username
def greet_user():
"""Greet the user by name."""
username = get_stored_username()
if username:
print(f"Welcome back, {username}!") # welcome back if username exists
else:
username = get_new_username()
print(f"We'll remember you when you come back, {username}!")
greet_user()