How to Repeatedly Append to a String in Python

To illustrate, imagine you are reading through a long log file looking for lines that begin with "ERROR:" You want to store all those lines in a string and write them out to a new file.

Your first inclination might be to do that like this:

with open('log.txt') as f:
    log = f.readlines()

new_log = ''
for line in log:
    if line[:6] == 'ERROR:':
        new_log += line
        
with open('newlog.txt','w') as f:
    f.write(new_log)

But as Python strings are immutable, that method requires creating new memory allocation each time you modify the string.

Instead, you should do the following:

1. Open and read the log file into a list (e.g., log):

   with open('log.txt') as f:
       log = f.readlines()

2. Create a new empty list (e.g., new_log):

   new_log = []

3. Loop through the log list looking for matches and append each matching line to the new_log list:

   for line in log:
       if line[:6] == 'ERROR:':
           new_log.append(line)

4. Join new_log on an empty string and write it to the new log file:

   with open('newlog.txt','w') as f:
       f.write(''.join(new_log))

If you're only making a few changes to the string, you won't notice much difference, but if you're making thousands, this method is much more efficient.