2

I have a drupal website, which exports its configuration into yaml files. For its webform component, we have some 200 webforms that blind-carbon-copy submissions to an internal email address, to track

We want to remove that email from all of our webforms, and the easiest way to do that is to remove it from the yaml configuration files, and re-import them.

So, my current workflow is to open one of the files, find the line that has email_webdev:, and then delete it and the next 42 lines, save the file, open the next, wash , repeat, for some 200 files.

I'm looking for a one-liner or script that will automatically remove an ordered set of identical 42 lines from all of the files in which they appear.

Technical explanation

In our config directory, there are some 200 files which I want to remove an identical set of lines from:

$ ls webform.webform.* webform.webform.incoming_student_housing_applica.yml webform.webform.info_for_students.yml webform.webform.info_request_for_viewbook.yml webform.webform.inquire_about_a_project.yml

Each of the webform configuration yaml is a few hundred lines of configuration data, indented, with basic formatting. Heres an example excerpt, with leading line numbers:

.... 170 test: 171 roles: { } 172 users: { } 173 permissions: { } 174 configuration: 175 roles: { } 176 users: { } 177 permissions: { } 178 handlers: 179 email_webdev: 180 id: email 181 label: 'Webdev Email' 182 handler_id: webdev_email 183 status: true 184 conditions: { } 185 weight: 0 186 settings: ... 214 parameters: { } 215 variants: { } 216 uuid: 6073470f-bb3b-40ad-8440-a7cb5f3be4d2

The 42-line stanza, from lines 179-214 in the excerpt above, is what I want to remove from some 200+ files in this directory. So what I'm doing manually, is removing lines 179-214 in vim, and saving. The result looks like this:

... 170 test: 171 roles: { } 172 users: { } 173 permissions: { } 174 configuration: 175 roles: { } 176 users: { } 177 permissions: { } 178 handlers: 179 variants: { } 180 uuid: 6073470f-bb3b-40ad-8440-a7cb5f3be4d2

It's 42 lines long. Those 42 lines will be identical between files, and in the exact same order, but the stanza starts at different places in the file. For instance, in one file it might begin at line 1068, in another, 872.

$ grep -n email_webdev * webform.webform.404.yml:183: email_webdev: webform.webform.accommodations_letter_request_fo.yml:219: email_webdev: webform.webform.agency_survey.yml:219: email_webdev: ...

All I've been able to figure out so is use the grep switch -A to find the lines after the pattern match:

$ grep -A42 email_webdev * webform.webform.volunteer_sign_up.yml: email_webdev: webform.webform.volunteer_sign_up.yml- id: email webform.webform.volunteer_sign_up.yml- label: 'Webdev Email' webform.webform.volunteer_sign_up.yml- handler_id: webdev_email webform.webform.volunteer_sign_up.yml- status: true webform.webform.volunteer_sign_up.yml- conditions: { } webform.webform.volunteer_sign_up.yml- weight: 0 webform.webform.volunteer_sign_up.yml- settings: ...

So, those are the lines that I want to remove from that file (and all the files in the directory), But, I can't just have it remove the line id: email, because that appears in other email respones, nor can I just pattern match any other line, like weight: 0, which appears in almost every other element, too, and conditions: { }. They can only the the lines in this stanza, which begins with email_webdev:, and continues on through the next 42 lines, identically, in every file.

Is there an easier way to do this?

Improve this question
2
  • May you provide a minimal, reproducible example? I am a bit confused. Commented Jun 25, 2020 at 13:57
  • @Quasímodo I apologize for being unclear. I've tried to better summarize the background and my ask at the top, and then expounded on technical details in the body. Let me know if this is helpful or not. Commented Jun 25, 2020 at 14:33

2 Answers 2

Reset to default
4

Please try this without the mv first to check if the new .out files created are exactly as you want. Re-enable the mv operation to overwrite the original files.

for file in webform.webform.*; do awk '/^ email_webdev:$/{n=42}{if(!n){print}else{n--}}' "$file" > "$file.out" mv -- "$file.out" "$file" done

The awk script detects a line exactly like this:

 email_webdev:

I.e., two spaces, email_webdev:, end-of-line character.

Once that line is detected, it sets n=42, thus stopping printing the lines, and then only starts printing again after 42 lines have been counted.

Improve this answer
1

You can use the ex line editor to do this.

for f in webform.webform.*; do printf '%s\n' /email_webdev:/ .,+41d x | ex -s "$f" end

Working:

  • Whenever we see relative addressing, we should reach for the ed/ex line editors, unless the file sizes are huge, because they support a wide variety of line addressing modes.
  • printf is emitting ex editor code, which ex reads on its stdin and applies on the input file.
  • First we make the email_webdev as the current line by searching for it: /email_webdev:/.
  • Then, starting from the current line, we grab the 42 chunk of lines .,.+41d.
  • The deed is done, so now we save and quit: x

PS: The 179-214 range is not 42 lines.

Improve this answer
2
  • +1 This is actually a very cool solution. Would you please explain what ex is doing there? I'm sure few users are familiar with it. Commented Jun 25, 2020 at 21:40
  • Thank you for the kind words. I have given a brief working method. Commented Jun 25, 2020 at 22:49

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.