How to trim URL to root domain?

Questions & Suggestions
,
33

No. You didn't tell me to. :wink:
You just said:

But no problem. I'll try that now. . .

OK, it seems to have installed OK:

iMac-27-JMU:~ jimunderwood$ cpanm Domain::PublicSuffix
--> Working on Domain::PublicSuffix
Fetching http://www.cpan.org/authors/id/N/NM/NMELNICK/Domain-PublicSuffix-0.14.1.tar.gz ... OK
Configuring Domain-PublicSuffix-0.14 ... OK
==> Found dependencies: Class::Accessor::Fast, Net::IDN::Encode
--> Working on Class::Accessor::Fast
Fetching http://www.cpan.org/authors/id/K/KA/KASEI/Class-Accessor-0.34.tar.gz ... OK
Configuring Class-Accessor-0.34 ... OK
Building and testing Class-Accessor-0.34 ... OK
Successfully installed Class-Accessor-0.34
--> Working on Net::IDN::Encode
Fetching http://www.cpan.org/authors/id/C/CF/CFAERBER/Net-IDN-Encode-2.400.tar.gz ... OK
==> Found dependencies: Module::Build
--> Working on Module::Build
Fetching http://www.cpan.org/authors/id/L/LE/LEONT/Module-Build-0.4224.tar.gz ... OK
Configuring Module-Build-0.4224 ... OK
Building and testing Module-Build-0.4224 ... OK
Successfully installed Module-Build-0.4224
Configuring Net-IDN-Encode-2.400 ... OK
==> Found dependencies: Test::NoWarnings
--> Working on Test::NoWarnings
Fetching http://www.cpan.org/authors/id/A/AD/ADAMK/Test-NoWarnings-1.04.tar.gz ... OK
Configuring Test-NoWarnings-1.04 ... OK
Building and testing Test-NoWarnings-1.04 ... OK
Successfully installed Test-NoWarnings-1.04
Building and testing Net-IDN-Encode-2.400 ... OK
Successfully installed Net-IDN-Encode-2.400
Building and testing Domain-PublicSuffix-0.14 ... OK
Successfully installed Domain-PublicSuffix-0.14
5 distributions installed
iMac-27-JMU:~ jimunderwood$

Now I'll try my KM macro.

Failed again with same error.

Can't locate Domain/PublicSuffix.pm in @INC (you may need to install the Domain::PublicSuffix module) (@INC contains: /Library/Perl/5.18/darwin-thread-multi-2level /Library/Perl/5.18 /Network/Library/Perl/5.18/darwin-thread-multi-2level /Network/Library/Perl/5.18 /Library/Perl/Updates/5.18.2 /System/Library/Perl/5.18/darwin-thread-multi-2level /System/Library/Perl/5.18 /System/Library/Perl/Extras/5.18/darwin-thread-multi-2level /System/Library/Perl/Extras/5.18 .) at /var/folders/hb/6xgg0y8j4g530m81rd1f9mpc0000gn/T/Keyboard-Maestro-Script-3801BC5F-2E88-424A-B738-F5981E9E5658 line 7.
BEGIN failed--compilation aborted at /var/folders/hb/6xgg0y8j4g530m81rd1f9mpc0000gn/T/Keyboard-Maestro-Script-3801BC5F-2E88-424A-B738-F5981E9E5658 line 7.

But the BBEdit version worked:

================================================================================
Sep 8, 2017, 8:10:29 PM
/Users/Shared/Dropbox/SW/DEV/Projects/[KM] Extract Domain Name/Get-Domain.pl
--------------------------------------------------------------------------------
db.de

Hmmm, Terminal reports a different Perl ver than the above km error.
From Terminal:

iMac-27-JMU:~ jimunderwood$ perl -v

This is perl 5, version 26, subversion 0 (v5.26.0) built for darwin-thread-multi-2level

My KM Macro is one simple Action:

I think I see the problem. In your script you have:
use 5.010;

yet I have Ver 5.26.

Should I change the use 5.010;, or just remove it?

34

Well, when I use use 5.26.0;, I get:

Perl v5.26.0 required--this is only v5.18.2, stopped at /var/folders/hb/6xgg0y8j4g530m81rd1f9mpc0000gn/T/Keyboard-Maestro-Script-B863094A-49FB-49F9-9DBE-217ED2E186B9 line 3.
BEGIN failed--compilation aborted at /var/folders/hb/6xgg0y8j4g530m81rd1f9mpc0000gn/T/Keyboard-Maestro-Script-B863094A-49FB-49F9-9DBE-217ED2E186B9 line 3.

I must have the wrong path or something in the KM Shell Script.

35

Leave it.

Please read my last post. I think we have posted at the same time.

If KM reports a perl version 5.18 then your ENV_PATH variable (in KM) does not include the Homebrew install path.

In that case change the shebang line as said here.

36

The use 5.010 only tells the minimum required version. And both, 5.26 and 5.18, are higher, so it doesn’t matter. But make sure KM is using the 5.26 version (the one installed via Homebrew.) Otherwise you are likely to get additional problems with the modules path.

37

OK, many thanks, Tom. That worked!!! :+1:

#!/usr/local/bin/perl

use 5.010;
use strict;
use warnings;
. . .

Thanks for putting up with me. I feel really bad being such an idiot user. :wink:

Having said that, I don't think this solution is a good solution for the general user domain, especially if they, like me, have very limited experience with Shell Scripting. OTOH, maybe it is just me.

Unless, there is a way to package this entire thing into a simple install . . .

I'm worn out on this topic, but if I have time later I may look at using JavaScript and/or cURL to get the data and perform the extraction.

But for now, it is well past my martini-time. :smile:

Thanks again, my friend.

38

No, I didn't intend this solution for the OP of this thread.

Yes, there is. Basically you can distribute any module with your script. (You just have to add 2 lines in the script with the location of the module.) I've done this with LB actions that include a perl script and need modules. With LB this is easy, since you can just include all resources inside the action bundle.

With KM it's not so easy because a macro file cannot include resources und you would have to tell the user where to put exactly the other files. See this thread :wink:

Cheers

1 Like
39

Wow. Tom. You guys really are cool! Thanks.

40

@JMichaelTX @Tom

I actually made use of document.domain JS, and [^.]+.[^.]+$ to trim server names to domain name. The JS part actually saved me my additional step of copying URL to clipboard.

I actually designed a macro that shows age in format like 2 years, 3 months, and 4 days old. It's inflating the actual age by 4-5 days for old domains. I think this is due to Keyboard Maestro rounds off values especially in the line that says zzzAgeDaysTemp MOD 30.5

Get Domain Age .kmmacros (7.1 KB)

41

Yes the modulo seems to accept only integers, so MOD 30.5 is the same as MOD 30.

42

So this should work (calculating in seconds):

Get Domain Age (Calc with Seconds).kmmacros (8.2 KB)

Used values:

1 year = 365.25 days = 31557600 seconds
1 month = 30.4375 days = 2629800 seconds
1 day = 86400 seconds

43

Hi @JMichaelTX and @AkshayHallur,

since I already started with the Perl script yesterday, I couldn't resist to complete it :wink:

Advantages over the other solution:

  • No wonky regex used. The domain root is determined with the help of the Current Effective TLD List. So, the script has no problems with fancy stuff like www.cs.tut.fi or kartolo.sby.datautama.net.id.
  • The script is auto-choosing the correct Whois server for the domain.
  • Since different Whois servers are using different output formats, server-specific parsers are used to extract the Domain Creation Date.
  • Not really important, but the age is calculated properly, not based on an average month length.

Output example:

Domain Date and Age (Perl Variant).kmmacros (4.5 KB)

The content of the script:

#!/usr/bin/env perl

use 5.010;
use strict;
use warnings;

use Domain::PublicSuffix;
use Net::Domain::ExpireDate;
use Date::Calc qw(Today Delta_Days N_Delta_YMD);

# Getting the input from KM
my $url_input = qx/osascript -e 'tell application "Keyboard Maestro Engine" to getvariable "ddaURL"'/;
chomp($url_input);

# Getting domain (root)
my $suffix = Domain::PublicSuffix->new();
my $domain = $suffix->get_root_domain($url_input);

# Get creation and expiry date of domain
(my $creation_str, my $expiry_str) = domain_dates($domain, '%Y-%m-%d');

# Date calculations
my $gmt = 1;
(my $year_creation, my $month_creation, my $day_creation) = split('-', $creation_str);
(my $year_now, my $month_now, my $day_now) = Today([$gmt]);

my $delta_days_total = Delta_Days($year_creation, $month_creation, $day_creation, $year_now, $month_now, $day_now);

(my $delta_years, my $delta_months, my $delta_days) = N_Delta_YMD($year_creation, $month_creation, $day_creation, $year_now,  $month_now, $day_now);

# Output
my $year_label = $delta_years == 1 ? ' year, ' : ' years, ';
my $month_label = $delta_months == 1 ? ' month, ' : ' months, ';
my $day_label = $delta_days == 1 ? ' day' : ' days';

say 'Domain:        ' . $domain;
say 'Creation date: ' . $creation_str;
say 'Expiry date:   ' . $expiry_str;
say 'Age:           ' . $delta_years . $year_label . $delta_months . $month_label . $delta_days . $day_label;
say 'Age in days:   ' . $delta_days_total;

__END__

=begin comment

For best results:
Download the most recent TLD names list from 
http://mxr.mozilla.org/mozilla-central/source/netwerk/dns/effective_tld_names.dat?raw=1
and save it @ /usr/local/effective_tld_names.dat

=end comment

Update (2017-09-11):

Output: Correct singular forms now ("1 month", not "1 months", etc.)

1 Like
44

Looks good Tom. Could you please upload your final macro?

Thanks.

45

Done.

1 Like
46

Hey @JMichaelTX, let me know if it works for you.

If not, we can continue the install-a-perl-module odyssey :grinning:

47

Oh heck, the odyssey is such fun, let's continue it! :wink:

I get this error:

Can't locate Domain/PublicSuffix.pm in @INC (you may need to install the Domain::PublicSuffix module) (@INC contains: /Library/Perl/5.18/darwin-thread-multi-2level /Library/Perl/5.18 /Network/Library/Perl/5.18/darwin-thread-multi-2level /Network/Library/Perl/5.18 /Library/Perl/Updates/5.18.2 /System/Library/Perl/5.18/darwin-thread-multi-2level /System/Library/Perl/5.18 /System/Library/Perl/Extras/5.18/darwin-thread-multi-2level /System/Library/Perl/Extras/5.18 .) at /var/folders/_2/7b0tgl916vg3ft82hgl011vm0000gn/T/Keyboard-Maestro-Script-66D1CE2A-91C9-43AA-AD6C-3FB89C73DDBF line 7.
BEGIN failed--compilation aborted at /var/folders/_2/7b0tgl916vg3ft82hgl011vm0000gn/T/Keyboard-Maestro-Script-66D1CE2A-91C9-43AA-AD6C-3FB89C73DDBF line 7.

1 Like
48

LOL, OK :slight_smile:

First:

Have you installed (with cpanm) the three modules listed in the script (after the “use”)?

Domain::PublicSuffix
Net::Domain::ExpireDate
Date::Calc

If yes, did you get any install errors from cpanm? (“Install failed” or something similar)

49

No. Sorry, I'm on a different Mac now, and I forgot I needed to do all that stuff.
May I suggest that you add the following to your macro:

  1. Instructions on installing all that stuff, including perl.
  2. At the top of your macro, test for all the modules you need, and report an error if you don't find all of them.

So, if you don't mind, could you please list all of the Terminal install commands for me? Can I run them together, or just one at a time?

Also, what about the path? Why is my path different?

Thanks.

50

Maybe, someday. But, as you have seen, there is more than one way to install perl :wink:

This information can be found in the Perl script, and perl itself is already testing for the presence of the required modules. (See the error message you have posted above.)

IIRC you succeeded to install perl 5.26 and cpanm on your other Mac two days ago. So I suggest to first read through the posts from two days ago.

To sum it up:

  1. Install perl via Homebrew. Follow the instructions you see in the Terminal during/after the install. (Speaking namely of this one.)

  2. Install cpanm as you have done two days ago.

  3. Create the ENV_PERL5LIB variable in KM with the paths as shown. It is also extremely helpful that you set up your ENV_PATH variable so that it includes the Homebrew install path. These are for example my paths:

    ENV_PATH: /Users/tom/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin

    ENV_PERL5LIB: /Users/tom/perl5/lib/perl5:/usr/local/Cellar/perl/5.26.0/lib/perl5/site_perl/5.26.0

    Replace tom with your account name, of course.

  4. Install the modules as required by the script (cpanm <module name>)

Nope. Do as described above and keep an eye on messages in the Terminal.

Also, I would install the modules one by one, because sometimes cpanm throws an error. (Most of the times a dependency problem.)

This has to do how you have configured it during/after the install. (locallib or not, the mentioned HB message).

51

BTW, since you are on a different Mac now, do you have the ~/.bash_profile and/or the ~/.bashrc on this computer?

52

What is the best way to check for these and/or display them?