Questions tagged as ['text-processing']

Questions about modifying, creating or parsing plain text files. This is a very broad tag, if your question is about using a specific tool, please tag it with the tool's name as well (e.g. sed, awk, perl, etc.)
Score: 0
Bruno Henrique Peixoto avatar
Strip ANSI sequences from string through bash-python code
in flag

I try to use bash to strip ANSI color escape sequences from a string without success. I tried already some regex-based code.


Blue='\033[0;34m'         # Blue
Clear='\033[0m'           # Text Reset

removeColors (){
    local uncolored_string=''
    local import_row='import re; \n'
    local regex_='(\x9B|\x1B\[)[0-?]*[ -\/]*[@-~]'
    local func_def_row='def escape_ansi(line): \n' ...
Score: 4
S.Khajeh avatar
(LibreOffice) Replace every character in a selection with *
sl flag

I want to shadow portions of a .odt document with stars for every character in the selection area. I guess it should be possible using the find and replace tool in LibreOffice. Just need to know how to tell it to find every character, some thing like a wildcard star character.

Score: 1
Bruno Henrique Peixoto avatar
Bash string split by delimiter constrained by number of characters
in flag

I want to split a long text separated by spaces through bash, but I fail. The command below splits into characters, but not delimiters.

echo "The quick fox jumped over the lazy dog" | fold -w 10
echo "The quick fox jumped over the lazy dog" | sed -e 's/.\{9\}/&\n/g'

It would be nice to have it for some user bash interaction.

Input syntax

format_text 10 "The quick fox jumped over the lazy dog"
Score: 0
How can I capitalise letters after a certain character
eg flag

How can I capitalize letters before a certain character? (^)

I am trying to do the reverse of the above link.

I want to capitalise everything after a certain character or word.

It can be using awk, sed or bash



 foobar ^ foobar


 foobar ^ FOOBAR


Score: 4
Jatin Kashyap avatar
Using sed with a variable inside double quote
kr flag

I have a file input.xml which contains a line: <exciton lambda="1" fix="hole"/>

in that line I want to replace 1 with 2 but I can't use just 1 as key word since there are other instances of "1" in the same file so I have to use the keyword lambda="1"

I am trying to use sed command as below:

sed -i "s/lambda="1"/lambda="$value"/g" input.xml

But it's not working, can anybody please help.

Score: 0
Tom avatar
Using awk/sed/perl i want to remove the self closing tag and its content and add an empty space
jp flag

I have the following string:

<Sub4Topic hi="1d9bhb" cbdshfghfgs(C)(2)(d)"><step stepid=" ajhdjgjgjfhgjhgrhghr" stepid="du" />

Here I want the step tag and its content to be removed. The output should be:

<Sub4Topic hi="1d9bhb" cbdshfghfgs(C)(2)(d)">
Score: 0
JayDin avatar
Analyse DMARC reports
cn flag

I receive Google's DMARC reports daily as a zip file containing an XML file. I manually download each zip file ~/Documents/DMARC.

I want to aggregate the information from all the saved DMARC reports and produce a human-readable readable report (e.g. table in LibreOffice Calc spreadsheet). I am on Ubuntu 20.04 standard desktop, so I would rather not have memory-hungry or CPU-hungry services runnin ...

Score: 0
Unable to see and edit the character that is typed when you press the Enter key in simple text editors like KWrite
in flag

I'm a newbie to the KDE Plasma interface. I find myself unable to see and edit the character that is typed when you press the Enter key in simple text editors like KWrite.

Score: 0
Andreas avatar
Filter blocks of log-output from a log stream
gb flag

Here's my task:

I've got a source stream of live log output from a messaging process. Lots of output is irrelevant to me but there are sections i want to collect and evaluate separately. Those blocks start with "---BEGIN Request---" at the end of a separate line which begins with date / time, hostname and process[pid]: . And accordingly a block ends with "---END Request---" at the end of another  ...

Score: 3
Chander PRabh Jain avatar
Replace a text within single quotes with another string using sed(just a first occurrence of regex)
ph flag

I have a file with the following content:

$ cat file.txt
CREATE COLLATION public.collation123 (provider = libc, locale = 'fr-FR');  
CREATE COLLATION public.collation_1 (provider = libc, locale = 'bas');  
CREATE COLLATION public.collation_test (provider = libc, locale = 'fr-FR');   

I want to replace the first occurrence of locale = 'fr-FR' with locale = 'fr_FR.utf8'.

After replacing, the file should ...

Score: 0
How can I pad numbers with leading zeroes using the seq command?
vn flag

I had to generate a text file for a given range of numbers. As usual I had to use seq range_start range_end > text_file.txt to put it in a text file.

I gave the seq range where the starting point was 01700 and the ending point was 01800. I know it's useless to have 0 before the number but in my case it was important. The file generated had lines like 1700, 1701 ... 1800. But there was no 0 before the  ...

Score: 4
Annabella avatar
How do I make a Markdown table of two columns out of a list whose items alternate to each column?
cn flag

I have a long list of common abbreviations for words in periodical titles. In the list, the full word is followed by its abbreviation. For example:

  • Administration
  • Admin.
  • Applied
  • Appl.
  • Administrative
  • Administ.
  • Approximate
  • Approx.

I want to turn the list into a Markdown table, like this:

Word Abbreviation
Administration Admin.
Applied Appl.

The problem is that doing this by hand would  ...

Score: 1
Joe Jobs avatar
How to count the number of consecutive identical lines
in flag

I have a file that looks like this:


I need a script to show me this:

2000 - 2
2001 - 4
2002 - 2

I prefer using sed or awk

Score: 2
Tim R avatar
Filtering content of command with grep
kn flag

I am trying to filter out a the output of ssh-keyscan. The goal of this is to filter the output so I can use it in my python code to identify hosts connected to my VPN. Normally I would use grep to filter, one of my greps is filtering properly, but the other is not. The first grep is working to get just the ed25519 ID, but not sure why I am getting the SSH-2.0... lines also. The command I ran along wit ...

Score: 1
jf4i2d avatar
Apply sed operations only to lines beginning with a particular string
bd flag

I have the following file format

Received from +11231231234 at 2021-10-10T19:56:50-07:00:
This is a message that contains words like from, at, etc.

Sent to +11231231234 at 2021-10-11T06:50:57+00:00:
This is another message that contains words like to, at, etc.

I want to clean up the "Received" and "Sent" lines, the following sed commands achieves this

cat file |  sed 's/from//g' | sed 's/to/     ...
Score: 0
Pauling0304 avatar
How to append to the end of a line before the first pattern match?
in flag

I have a file containing some random data like:

number1 number2 number3

I want to add a \ to the end of the line containing number3, but I cannot match a pattern on that line as that number keeps changing. So I try matching "&END" in sed and then append a line before it, but that does not work for me because it creates a new line, like

number1 number2 number3



My desired out ...

Score: 0
Mona Jalal avatar
awk -- duplicating a column and also modifying its text
gb flag
awk {'printf ("%5s\t%s\n", $0, $NF)'} test_VN.txt

works for duplicating a column in a text file however, how can I add _VN before the .jpeg extension of the duplicated column?

For example, I want to have this output:

21_48.jpeg  21_48_VN.jpeg
24_48.jpeg  24_48_VN.jpeg
25_48.jpeg  25_48_VN.jpeg

I want to read the first column from test_VN.txt which only has one column and save it to test_VN_2.txt with tw ...

Score: 6
james simmons avatar
How can I count each type of character (and total them) in a text file?
gs flag

I was just wondering if anyone could tell me how to count the occurrences of each different character in a text file and also a total of all the occurrences of everything added together at the end.

I'm just trying to learn the process for my own knowledge.

Score: 0
How to copy partial contents (specifically few sections denoted by square brackets) from one file to another with shell script?
et flag

I have one project's .git/config file in which I have following contents:

    name = <FullName>
    email = <EmailID>
    username = <UserName>
    editor = nvim
    whitespace = fix,-indent-with-non-tab,trailing-space,cr-at-eol
    pager = delta
    browser = google-chrome
    a = add --all
    ai = add -i
    ap = apply
    as ...
Score: 1
Extracting specific data from different tables and merging it
in flag

I would like to extract specific rows from specific columns from different tables, based on the column header. However, my knowledge seems to be too limited to solve it myself (both in command-line and python).

I have a directory with a growing number of .tsv files (OTU tables). These files all have the same layout:

  • Row 1 contains headers.
  • Rows 2 - x contain data. The number of columns can differ between ...
Score: 14
Justin Washek avatar
Delete a file if multiple conditions are met
ma flag

What I need

I have an existing script that pulls port information for domains and stores it into a text file called portscan.txt. Example:

portscan.txt file:

I want to delete the information only if certain conditions are met. These conditions include:

  • The file with the domains should have 2 or less lines
  • The ports should only be 80 or 443 (i.e., I don't want t ...
Score: 0
KesiScarlet avatar
How to cut intervals of lines of a file and put them into multiple files?
af flag

I have a fasta file containing ids and sequences like this:


I want to recursively cut the lines containing the ids and the sequence ...

Score: 0
cybermizz avatar
Purely parse the column from the bash command output
ng flag

We need to extract the column from the command output. I have tried the methods using awk, and cut commands. Whereas we have the spaces in the 2nd column values due to which delimiter for space or other character does not parse the 2nd column output in a correct way. Do we have another method to get purely the second column of the output as mentioned above?

# cat info.txt
Vmid      Name             ...
Score: 0
How to use bash/sed to extract XML attribute value
in flag

I have the following grep command piped into sed to find an element name attribute and store the sed result into a name variable.

 name=$(grep -E "<element.*name=.*/>" "$F" | sed -e "s/.*<element.*name=\(.*\)\/>.*?/\1/")

Sample Data -

<element name="Barium"/>

Desired Output -


Actual Output -

<element name="Barium"/>

I'm a little confused on how to get the sed comma ...

Score: 2
ben sunny avatar
display the filenames with 4 or more characters using ls
sd flag

How to display the filenames with 4 or more characters using the ls command

Score: 0
Pizza avatar
CSV tools for terminal?
in flag

I'm using Ubuntu 18.04 and above, both desktop and server, and looking for terminal tools to query / write to large CSV file efficiently, and also easy to interact (few simple commands or commands that resemble SQL query language).

What would you recommend?

Score: 0
thinh2k1310 avatar
Save name and MAC ADRESS of all visible bluetooth device to array
us flag

I want to save name and mac adress of all bluetooth device to two arrays after run this command bt-device -l: 1

The result should like this :

 MAC_ADDRESS=["Mac_address1" , "Mac_address2"]

Please help me! I'm using ubuntu 20.04 , working with bash script.

Score: -2
Introducing blank lines between matched sections
jp flag

I have the following bash function that uses sed to extract sections occurring between ## Mode: org and ## # End of org, where # is the comment character. Finally I remove the comment character and any spaces.

This is my input

cat /home/flora/docs/recnotes.txt
   ## Mode: org
   #  Assigns shell positional parameters or changes the values of shell
   #  options.  The -- option assigns the positiona ...
Score: 0
Anders Begtorp avatar
Sorting textfile with rows of [word number] numerically
ci flag

I have a .txt file that consists of:

Aa 6
Bb 3
Aa Aa 2
Bb Bb 10

I need to sort it so the result is:

Bb Bb 10
Aa 6
Bb 3
Aa Aa 2

This only gives the alphabetical order:c

cat .txt | sort -n

Can you help here? Kind regards Anders

Score: 0
Janchi Garcia avatar
How to auto edit the first line content in text file using shell script
cn flag

Here is the file name: SUMAAI011.A01

Here is the file content and would like to rename the first line 01UMAGL011 to 01UMAKBO11