Does anyone have experience with Easy Data Transform software:
https://www.easydatatransform.com/
I came across this today - it seems like a wonderful way to do all sorts of data transformations which we often discuss as part of KM macros. As it can be executed from the command line it can be integrated with KM fairly easily.
Has anyone tried it?
Well, it's a collection of 52 functions (marketed as 'transforms'), all of them either already available, (or readily written in) JavaScript etc, which might yield a bit more flexibility,
but mileage varies – not everyone wants to mess with script actions – and I wish them well.
In the context of bulk transformations over row and field data (csv and excel files etc) recent builds of Excel (beta channel only at the time of writing, I think) have introduced a new LAMBDA function, which makes it possible to define any function over Excel array types.
There are examples of Excel LAMBDA syntax on Rosetta Code,
and an overview here:
[LAMBDA: The ultimate Excel worksheet function - Microsoft Research]( LAMBDA: The ultimate Excel worksheet function - Microsoft Research )
Just got an email - you can get a discount on this software through Summerfest 2021 - see here; SummerFest 2021
I’ve no connection to either this software nor the festival.
Interesting - do you know if the Lambda functionality will be supported on the Mac? Often Microsoft delays in adding feature parity for the Mac vs PC.
Yes, it's part of the Beta distribution of Excel on Mac already. I used it to draft the Rosetta Code examples.
Looks useful, but at $100 is way too expensive, IMO.
Even with the 25% discount, $75 is much to expensive.
Unless the things it does just happens to meet some specific needs you have, and will save you lots of time, or will work for you instead of hiring custom support.
Of course, YMMV.
Can Javascript easily/efficiently execute "transformations" or functions spanning the entirety of a large CSV file?
Simple Answer: Yes.
I have processed very large text files with JXA using a combination of Regex and JavaScript code. Of course, this requires knowledge of both RegEx and JavaScript.
I think @ComplexPoint can best speak to this question. He has given us many examples of it.
The short answer is yes, though strictly speaking, it depends on what you mean by "JavaScript".
The ECMAScript standards which define the JS language itself don't cover file access, simply because JS has evolved as a language embedded in contexts like browsers, from which the file system has to be protected.
Particular embeddings of JS however, (JXA, Node.js) do provide libraries which define their own file system functions.
Using the JXA Automation library on macOS for example, we can write things like:
// readFile :: FilePath -> IO String
const readFile = fp => {
// The contents of a text file at the
// filepath fp.
const
e = $(),
ns = $.NSString
.stringWithContentsOfFileEncodingError(
$(fp).stringByStandardizingPath,
$.NSUTF8StringEncoding,
e
);
return ObjC.unwrap(
ns.isNil() ? (
e.localizedDescription
) : ns
);
};
and
// writeFile :: FilePath -> String -> IO ()
const writeFile = fp => s =>
$.NSString.alloc.initWithUTF8String(s)
.writeToFileAtomicallyEncodingError(
$(fp)
.stringByStandardizingPath, false,
$.NSUTF8StringEncoding, null
);
For really huge CSV files (several gigabytes), I've found buffered processing through the Python CSV library rather fast and painless, and that too is accessible from within the work-flow of a KM macro.
And more generally, as @JMichaelTX says, all of the data transforms in that package can be written as JavaScript functions.
(It's possible that it may actually be implemented in JavaScript – I notice that for fuller flexibility it allows you to add your own JS code in a 'JavaScript Transform', so perhaps that is the idiom, under the hood, of the other functions/transforms too).
Thank you - excellent example
I admire your knowledge and skill in Javascript immensely - I can see why an app like Easy Data Transform seems mostly pointless for a professional dev. For someone like me who is mostly interested in my own knowledge domain (medicine) I have the mental bandwidth to retain the basics of Applescript/JXA but clearly it would take me way more time to be proficient writing the sort of example you probably did in under a minute. No point for me to code stuff that I can do quickly/easily with a GUI - then use KM or very basic JXA to finish the automation connections.
you think so ?
Those functions, and other general reusable abstractions, can, incidentally be found on Github at:
prelude-jxa: Generic functions for macOS and iOS scripting in Javascript
with AppleScript equivalents, in most cases, at:
prelude-applescript: Generic functions for macOS scripting with Applescript
( i.e. no need to write them – just copy and paste. I've accumulated them slowly over the course of writing scripts for my own work, which is not in computing
I completely agree with that. In fact, I think one of the big use cases for KM is for people just like you who don't have the interest or time to write complex scripts.
So, as I said above, if a commercial app/package will save you lots of time, and it is available for a price that is reasonable to you, then that might be your best approach.
Agreed - and the developer seems very interested in working to add new features - he is considering adding Applescript/JXA support to call the "Transformations" (a set of functions defined in the GUI) which would make it even easier to coordinate with KM without having to deal with the command line.
On a separate but somewhat related note, I continue to learn Retool when working with online data or when I want to write a web app - this is a "low code" web app builder with extremely strong support for Javascript just about anywhere you want to tweak its operation. So far I am finding it to be a really well-done piece of software that can really save time but not sacrifice the flexibility of writing a full web app.
Hi. I am the developer of Easy Data Transform. I noticed the link from here in my weblogs and just wanted to clarify a few things:
Hi all
I'm in a similar position to @rkaplan. I saw Easy Data Transform for the first time this week as part of the Summerfest promotion. I completely understand its functionality can be replicated in AS / JXA, but for someone like me who has a small set of specific requirements, it got me what I wanted much more quickly than "First, learn Javascript." 
(That reminds me of the recipe for rabbit stew in Mrs Beeton's Victorian cookbook: "First catch your rabbit.")
Stacking multiple transforms is very reminiscent of building KM actions, and because you can take a look at the "output" of each transform, it's relatively easy to see what's working and what's not.
Not for everyone I realise, particularly given the price, but it will be the right thing for some.
I've no affiliation with the developer, and I'm not on commission ;-))
Kevin
Hello Everyone,
First a disclaimer: I am not associated at all with the developer of Easy Data Transform.
I bought the software in its first days when it was available as part of a bundle.
I have to say that it is the app that I have been useing the most of the bundle and definitely the one that I recommend to colleagues who need to do any data crunching.
As @rkaplan my specialty is medicine, and even if could potentially implement javascript routines to possibly get to the result offered by EDT, it is definitely going to take me way more time than an experimented programmer.
With EDT you have a graphic view of your transforms, you can easily insert other transforms in the connecting nodes, stem out, check the effect of the transform, have alerts for data incongruence, add notes and modify your transforms to adapt to data source changes.
I had initially some difficulties with the transforms and the developer has been very quick to respond with advice and updating the software to incorporate suggestion on functions and usability from the users.
It is great to have developers so dedicated in continuously improving their software and supporting the users.
I think that this kind of software addresses a specific need and it does it elegantly and very well.
I am using it even for simpler conversions from the clipboard, for instance when receiving messed up lists of e-mails to clear out and sort.
At the end, it all depends on how much you value your time. Yes, it is fun and you always learn something in writing up your code, but it takes time and errors, while with an existing software it can be very efficiently done (and easily reused).
Same as KM, I likely underuse it, but it makes it easy to whip up automations that let me gain already a lot of time and efficiency.
Again I am not affiliated and receiving no benefit from Easy Data Transform, just loving the software.
GLS
Just a quick follow up to this post, from earlier in the year (I hope that is ok). Easy Data Transform, and some other great software, is again available at a discount at: https://www.artisanalsoftwarefestival.com/. We have also made a lot of improvements to the software.