sed not working on large files. Any other options available?

TimBurke

6/5/24, 2:51 PM

Trying to insert a line feed on large files in UNIX when a text string matches. Any file around 1GB or less, it works. Anything over that size, it does not complete the replace. It appears to do nothing.

I'm using the following command:

sed -i 's/"sourceSystemCode": "xyz"}{"active": true/"sourceSystemCode": "xyz"}\n{"active": true/g' filename.txt

I have even tried:

sed 's/"sourceSystemCode": "xyz"}{"active": true/"sourceSystemCode": "xyz"}\n{"active": true/g' filename.txt > newfile.txt

Any other suggestions to add the line feed using any other command or syntax with sed is greatly appreciated.

1 + 1

unix

text

sed

Cyrus

6/8/24, 11:31 PM

[There](https://stackoverflow.com/help/formatting) you will find information on how to format your question with Markdown.

Score:1

Server

mfinni

6/5/24, 4:39 PM

I see an answer on SO here https://stackoverflow.com/questions/48388833/sed-not-working-on-large-file-looking-for-other-options

Perl will let you change the input separator ($/) from newline to another character. You could take advantage of this to get some convenient chunking.

perl -pe'BEGIN{$/="}"}s/^({"seq_id")/\n$1/' my_giant_json.json That sets the input separator to be "}". Then it looks for chunks that start with {"seq_id" and prefixes them with a newline.

Note that it puts an unnecessary empty line at the beginning. You could complicate the program to eliminate that or just delete it manually after.

If you search SO for the first sentence of your question, you'll find a lot of other ideas too.

+ 2

TimBurke

6/5/24, 5:39 PM

I saw that but I don't understand how it works. That example talks about adding it to the begining of a string. I'm looking to replace it in the middle of a string. How would I go about modifying that command to do so.

djdomi

6/5/24, 6:24 PM

just as a trick did you try to ask chatgpt about? ;)

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: sed not working on large files. Any other options available?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.