Remove Specific Lines From a Hosts File?

[hello]

I'd like to know how to automate this:

I use a hosts file from here to block ads, but, every time I update it, I need to manually edit it.
Why: To allow certain hosts.

My question: How can I

1. Find all instances of, say "googleanalytics", and
2. Remove any line that mentions it

Example lines from file located here:
(Note: They might be in non consecutive rows)

0.0.0.0 googleanalytics.com
0.0.0.0 ssl.googleanalytics.com
0.0.0.0 click.googleanalytics.com
0.0.0.0 alt2.googleanalytics.com
0.0.0.0 alt4.googleanalytics.com
0.0.0.0 sandbox.googleanalytics.com

Note 2: I use Gas Mask to download the hosts file, this app automatically downloads the most recent file if it finds a new one.

[Sleepy]

If you want to delete lines from a file that contain a string stored in a variable named DeleteString, this should probably work.

The "grep -v" returns only the lines that are not in the specified string.

I haven't tested if the string can handle blanks, but it looks like you don't require blank handling.

It's too bad that the KM action above doesn't allow you to save to the same file, because it lets you save to the same variable. Seems inconsistent.

P.S. I didn't test this, because I do this sort of thing very often. I have the philosophy that answers don't have to be tested if they are probably right, or at least a very good lead to a solution.

[ccstone]

@hello

My question: How can I

1. Find all instances of, say "googleanalytics", and
2. Remove any line that mentions it

That's pretty straightforward.

1. Use Keyboard Maestro's search/replace to change the given file directly.

2. Use Keyboard Maestro to read the file, process the contents, and write it back to the file – by whatever means works for you.

3. Script it with the Shell, AppleScript, JXA, etcetera.

@Sleepy's idea is fine, although the >> operator appends to the given file.

Personally I'd use sed, because it's made precisely for such things.

srcFilePath=~/'test_directory/Source_Directory/Hosts.txt'
# cat "$srcFilePath"
sed -ni '' '/googleanalytics/d;p;' "$srcFilePath"

This script operates directly on the given file and deletes any line it finds containing the string googleanalytics – so make sure to have a backup of your data file.

Sed takes some getting used to, but it's a very mature and powerful Unix tool.

-Chris

[ComplexPoint]

And if you wanted a longer exclusion list (several hosts), you could list them in a multi-line KM variable:

and then write JavaScript for Automation in the pattern of:

lines(hostList)
    .filter(s => !exclusions.some(e => s.includes(e)))
    .join("\n");

to filter out lines which contained any of the excluded strings.

for example:

Filtered host list.kmmacros (3.1 KB)

Expand disclosure triangle to view JS Source

(() => {
    "use strict";

    const instanceID = ObjC.unwrap(
        $.NSProcessInfo.processInfo.environment
        .objectForKey("KMINSTANCE")
    );

    const
        kme = Application("Keyboard Maestro Engine"),
        kmvar = k => kme.getvariable(k, {
            instance: instanceID
        });

    const lines = s =>
        0 < s.length ? (
            s.split(/[\r\n]+/u)
        ) : [];

    const
        hostList = kmvar("localHostList"),
        exclusions = lines(
            kmvar("localExclusions")
        ).map(x => x.trim());


    return lines(hostList)
        .filter(s => !exclusions.some(e => s.includes(e)))
        .join("\n");
})();

[hello]

Thanks everyone, great answers!

And if you wanted a longer exclusion list (several hosts), you could list them in a multi-line KM variable

Excellent, great to know there's a way to have several exclusions!

[hello]

Thanks, looks like this worked:

1. Opened a Terminal
2. Used a wget to download the hosts file
3. Ran the sed script
   just added sudo to allow overwriting hosts file

[drdrang]

If you go with sed and want additional exclusions, you can use the -E option and add other terms to search for, with the terms separated by vertical bars. For example:

sed -Ei '' '/googleanalytics|adservice|another/d' hosts

This becomes unwieldy after more than a few terms, and then you'll probably prefer @ComplexPoint's solution.

[ccstone]

@drdrang

This becomes unwieldy after more than a few terms

That easily solved:

hostsFile=~/'test_directory/Source_Directory/Hosts.txt'

sed -En '
   /adservice/d
   /another/d
   /googleanalytics/d
   /ssl/d
   p
' "$hostsFile"

You can offload all the commands to a separate file if desired.

[Sleepy]

KM also has a cool way to read commands (for sed or for most other shell commands) from an additional text box in the KM macro, similar to this, which I think is very "readable":

Note that I did not test this macro. The correct syntax depends upon the application (in this case, sed) and so this is just an example in principle of how to add multi-line parameter data to a Shell app in an easy to read manner.

[ccstone]

@Sleepy

Note that I did not test this macro

Hey @Sleepy,

I can't get that working...

Do you have an example that does work, so I can see it in action?

-Chris

[Sleepy]

The simplest working example to illustrate the idea is this:

This illustrates how the STDIN for a shell command is fed into the script.

In the case of cat, the "-" that I used is optional.

In the case of sed, the man page says that to feed the subcommands you have to use the "-f" parameter, and you probably specify the STDIN using "-" as I did.

The reason I didn't test is that I didn't want to create the files and directories that your example required.

I would like to test it, but I have a visitor coming to try to get my furnace working. It's pretty cold right now in Canada and I need to get my furnace working before I can spend much time here today.

[ccstone]

@Sleepy

In the case of sed, the man page says that to feed the subcommands you have to use the "-f" parameter, and you probably specify the STDIN using "-" as I did.

Sed expects a file reference and chokes if it doesn't find one – nor does it recognize “-” as an operator meaning stdin.

I was able to get stdin to work but had to use process substitution to do it.

The stock shell on Mojave doesn't support process substitution. If I remember correctly Bash 4+ is required. (I have installed GNU Bash 5.1.8 on my system.)

Execute a Shell Script.kmactions (922 B)

If you discover an easier way please do let me know.

-Chris

[Sleepy]

@ccstone

Sed expects a file reference and chokes if it doesn't find one – nor does it recognize “-” as an operator meaning stdin.

That's odd because the man page for sed says:

The commands are read from the standard input if command_file is “-”.

To me that means it should work. I must be dreaming because I thought I've done it before.

[ccstone]

@Sleepy

The commands are read from the standard input if command_file is “-”.

Not on Mojave or in the GNU sed man page...

What version are you working with?

[Sleepy]

I'm using Monterey. But this isn't a new feature of Monterey. I've used it before. Here's a screenshot showing the man page in Monterey for sed. At the bottom of the image is the description of the -f parameter which says what I quoted earlier.

[ccstone]

@Sleepy

I'm using Monterey. But this isn't a new feature of Monterey.

That will be post Mojave. As you can see the stock Mojave version of sed is ancient, and Apple didn't even bother to give it a version number – just a date.

-f command_file
       Append the editing commands found in the file command_file to the list of commands.  The editing commands should each be listed on a separate
       line.

BSD     May 10, 2005     BSD

I'm pleased to hear that Apple has employed a more modern version of sed in later versions of macOS.

Look at the bottom of the man page and see if there's a version number. I'm guessing they're using a later version of BSD sed, but I'd like to know.

-Chris

[Sleepy]

@ccstone

Look at the bottom of the man page and see if there's a version number. I'm guessing they're using a later version of BSD sed, but I'd like to know.

The only version number I see is the macOS version number, 12.0.

Although I see these lines, which might be different from your man pages...

STANDARDS
     The sed utility is expected to be a superset of the IEEE Std 1003.2
     (“POSIX.2”) specification.
AUTHORS
     Diomidis D. Spinellis <dds@FreeBSD.org>

[ccstone]

Shame on them... Is there a date?

[Sleepy]

Beside the macOS version number is the date "June 10, 2020 " which I presume is when the man page was formatted. I don't imagine that's related to when the command was produced, but I guess it could be.