Vfrdtyky

I have several blocks of text that look like this:

  steps:

  - class: pipe.steps.extract.Extract

    conf:

      unzip_patterns:

      - .*EstimatesDaily_RealEstate_Q.*_{FD_YYYYMMDD}.*

    id: extract

  - class: pipe.steps.validate.Validate

    conf:

      schema_def:

        fields:

I want to replace this block of text with this:

  global:

    global:

      schema_def:

        fields:

The catch here is that the text crosses several lines in each text file. Maybe there is an easy workaround for this, not sure. More troublesome, is that is don't always have '- .*EstimatesDaily_RealEstate_Q.*_{FD_YYYYMMDD}.*'. Sometimes the text is '- .*EstimatesDaily_RealEstate_Y.*_{FD_YYYYMMDD}.*' or it could be '- .*EstimatesDaily_RealEstate_EAP_Nav.*_{FD_YYYYMMDD}.*' One thng that is always the same in each block is that it starts with this ' steps:' and ends with this ' fields:'.

My sample code looks like this:

import glob

import re

path = 'C:\Users\ryans\OneDrive\Desktop\output\*.yaml'

regex = re.compile("steps:.*fields:", re.DOTALL)

print(regex)

replace = """global:

global:

  schema_def:

    fields:"""

for fname in glob.glob(path):

    #print(str(fname))

    with open(fname, 'r+') as f:

        text = re.sub(regex, replace, '')

        f.seek(0)

        f.write(text)

        f.truncate()

Of course, my example isn't simple.

Since you're doing a general replacement of things between strings, I'd say this calls for a regular expression [EDIT: Sorry, I see you've since replaced your string "replace" statements with regexp code]. So if your file is "myfile.txt", try this:

>>> import re

>>> f = open('myfile.txt', 'r')

>>> content = f.read()

>>> f.close()

>>> replacement = ' global:n   global:n     schema_def:n       fields:'

>>> print re.sub(r"(ssteps:)(.*?)(sfields:)", replacement, content, flags=re.DOTALL)

The output here should be the original contents of "myfile.txt" with all of the substitutions.

Instead of editing files directly, the usual convention in Python is to just copy what you need from a file, change it, and write everything back to a new file. It's less error prone this way, and should be fine unless you're dealing with an astronomically huge amount of content. So you could replace the last line I have here with something like this:

>>> newcontent = re.sub(r"(ssteps:)(.*?)(sfields:)", replacement, content, flags=re.DOTALL)

>>> f = open('newfile.txt', 'w')

>>> f.write(newcontent)

>>> f.close()

Regex is the best answer here probably. Will make this simple. Your mileage will vary with my example regex. Make it as tight as you need to make sure you only replace what you need to and dont get false positives.

import re



#re.DOTALL means it matches across newlines!    

regex = re.compile("steps:.*?fields:", flags=re.DOTALL, count=1) 



replace = """global:

global:

  schema_def:

    fields:"""



def do_replace(fname):

    with open(fname, 'r') as f:

        in = f.read()

    with open(fname, 'w') as f:

        f.write(re.sub(regex, replace, in))



for fname in glob.glob(path):

    print(str(fname))

    do_replace(fname)