Download data from a Custom HTML Prompt?

[Scotty2Hotty]

Hello,

I've been struggling with this all day, and I've admitted defeat and saved the data to a KM Variable in the meantime until I can solve this problem.

In short - I can't find a way to download data from a Custom HTML prompt that doesn't involve using a KM variable as an intermediary. I've made a table that presents data to my from my database, and I'd rather not use a KM variable incase I need to save so much data that it might overpopulate the KM environment.

My most involved attempt revolved around trying to create a blob file from the CSV data to download, either with a URL or URI. In KM it presents an error from Mac OS's dialogue that it can't find an application like so:

"There is no application set to open the URL blob:null/26580a6e-86c7-439c-86b7-21637dfa55fb."

I'm generating this code from an 'execute shell script' that populates the HTML with data from a CSV file. Below is the javascript from the HTML that I've been trying to use to create a download:

		// Define the downloadBlobAsFile function
		const downloadBlobAsFile = (function closure_shell() {
		    const a = document.createElement(\"a\");
		    return function downloadBlobAsFile(blob, filename) {
		        const object_URL = URL.createObjectURL(blob);
		        a.href = object_URL;
		        a.download = filename;
		        a.click();
		        URL.revokeObjectURL(object_URL);
		    };
		})();
        
        // Function to save the spreadhseet data as a CSV
		function saveChanges() {
		    var table = document.getElementById('csvTable');
		    var rows = table.getElementsByTagName('tr');
		    var csvContent = [];
		    for (var i = 1; i < rows.length; i++) {
		        if (rows[i].style.display !== 'none') {
		            var cells = rows[i].getElementsByTagName('td');
		            var rowData = [];
		            for (var j = 1; j < cells.length; j++) {
		                rowData.push(cells[j].textContent.replace(/,/g, ''));
		            }
		            while (rowData.length > 0 && rowData[rowData.length - 1] === '') {
		                rowData.pop();
		            }
		            csvContent.push(rowData.join(','));
		        }
		    }
		
		    var csvString = csvContent.join('\n');
		    
		    // Create a Blob with the CSV content
		    var blob = new Blob([csvString], {type: \"text/csv;charset=utf-8\"});
		    
		    // Use the downloadBlobAsFile function to trigger the download
		    downloadBlobAsFile(blob, \"IncomingCSV.csv\");
		    
		    // Show instructions to the user
		    var instructions = document.createElement(\"p\");
		    instructions.textContent = \"The CSV file download has been initiated. \" +
		                               \"Please save it to this location:\n\" +
		                               \"CSV/IncomingCSV.csv\";
		    instructions.style.textAlign = \"center\";
		    instructions.style.marginTop = \"20px\";
		    document.body.appendChild(instructions);
		
		    alert('CSV file download has started. Please check your downloads and move the file to the correct location.');
		}

This actually works if I run this on Safari, but I want to keep it to a HTML prompt if possible.

I thought about using window.KeyboardMaestro.ProcessAppleScript to create the file, but the escaping required to program this in javascript within html within shell was enough to make a grown man cry. I couldn't figure it out.

Any help would be greatly appreciated!

[Scotty2Hotty]

I came up with something that is useful enough for me in the interim to finding out a more graceful solution that doesn't involve saving the data to a KM variable.

It is, however, almost guaranteed to make Peter Lewis violently ill if he finds out.

I've used:

window.KeyboardMaestro.Log( 'Message' )

in the HTML prompt to save the CSV data from the prompt to the engine log of Keyboard Maestro. I begin the macro by deleting the engine log using a standard KM "Delete File" action. In the code for the HTML window, I use the log command above as a stand-in for saving the data to a file like so:

window.KeyboardMaestro.Log('CSV_CONTENT_START\n' + csvString + '\nCSV_CONTENT_END')

this saves the CSV data as text on separate rows to the engine log, with the unique markers starting and ending the data so it can be easily identified and retrieved. I've then set up another macro to hold a Javascript that extracts this data from the engine log file and formats it as a CSV file. Below is the code, edit the file paths as necessary:

// Extract HTML CSV Data From Log

// Ensure we have access to the necessary Objective-C bridging.
ObjC.import('Foundation')

function kmVariable(name, value = null) {
    const km = Application('Keyboard Maestro Engine')
    if (value === null) {
        return km.getvariable(name)
    } else {
        km.setvariable(name, { to: value })
    }
}

function log(message) {
    if (kmVariable("CSVFromEngineLogLogWrite") === "true") {
        kmVariable("CSVFromEngineLogLog", kmVariable("CSVFromEngineLogLog") + "\n" + message);
    }
}

function writeCSV(filePath, content) {
    try {
        const nsString = $.NSString.alloc.initWithUTF8String(content);
        const data = nsString.dataUsingEncoding($.NSUTF8StringEncoding);
        const success = data.writeToFileAtomically(filePath, true);
        log(`Write operation result for ${filePath}: ${success}`);
        return success;
    } catch (error) {
        log(`Error writing CSV file ${filePath}: ${error.message}`);
        throw error;
    }
}

function fileExists(filePath) {
    return $.NSFileManager.defaultManager.fileExistsAtPath(filePath);
}

function deleteFile(filePath) {
    try {
        const fileManager = $.NSFileManager.defaultManager;
        const error = $();
        const success = fileManager.removeItemAtPathError(filePath, error);
        if (success) {
            log(`Deleted file: ${filePath}`);
        } else {
            log(`Error deleting file ${filePath}: ${ObjC.unwrap(error.value)}`);
        }
        return success;
    } catch (error) {
        log(`Error deleting file ${filePath}: ${error.message}`);
        return false;
    }
}

function processCSVFromLog() {
    const logPath = "[Replace with full file path]/Logs/Keyboard Maestro/Engine.log";
    const csvPath = "[Replace with full file path]/ExampleCSV.csv";

    try {
        log("Starting CSV processing");
        
        // Read the log file as plain text
        const fileManager = $.NSFileManager.defaultManager;
        const data = fileManager.contentsAtPath(logPath);
        const nsString = $.NSString.alloc.initWithDataEncoding(data, $.NSUTF8StringEncoding);
        const logContent = ObjC.unwrap(nsString);
        
        log("Engine log read successfully");

        const startMarker = 'CSV_CONTENT_START\n';
        const endMarker = '\nCSV_CONTENT_END';
        const startIndex = logContent.lastIndexOf(startMarker);
        const endIndex = logContent.lastIndexOf(endMarker);

        if (startIndex === -1 || endIndex === -1) {
            throw new Error("CSV content markers not found in the log");
        }

        const csvContent = logContent.substring(startIndex + startMarker.length, endIndex);
        log("CSV content extracted from log");

        // Process the CSV content to maintain row and column structure
        const rows = csvContent.trim().split('\n');
        const processedCsvContent = rows.join('\n');

        // Only delete the existing file after we've successfully extracted the content
        if (fileExists(csvPath)) {
            if (!deleteFile(csvPath)) {
                throw new Error("Failed to delete existing CSV file");
            }
        }

        if (writeCSV(csvPath, processedCsvContent)) {
            log("CSV content successfully written to file");
            kmVariable("CSVFromEngineLogCompletionState", "true");
        } else {
            throw new Error("Failed to write CSV content to file");
        }

    } catch (error) {
        log(`Error processing CSV: ${error.message}`);
        kmVariable("CSVFromEngineLogCompletionState", "false");
    }
}

// Run the main function
processCSVFromLog();

This Javascript writes to these KM variables so you can what's going on / configure further actions:

CSVFromEngineLogLog - logs for this javascript.
CSVFromEngineLogLogWrite - set to true or false prior the script running depending if you want logs written to the above KM variable.
CSVFromEngineLogCompletionState - returns true or false depending if the script was successful or not.

[Nige_S]

@Scotty2Hotty

and I'd rather not use a KM variable incase I need to save so much data that it might overpopulate the KM environment.

How much data are you talking about? For giggles I've just read in, "Filter" reversed, and written out a 110MB log file and KM didn't even blink -- took 7 seconds on my old, spinning rust, iMac.

If you make sure to use a local variable then the data won't be in "the KM environment" after the macro completes anyway.

[DanThomas]

@Scotty2Hotty

In short - I can't find a way to download data from a Custom HTML prompt that doesn't involve using a KM variable as an intermediary.

Here's how I handle the situation:

If the data can be passed as HTMLResult, then I return it there. Otherwise, I write a macro that takes the data in TriggerValue, and "Trigger" the macro from the custom HTML prompt.

Yes, I suppose technically the data is stored in a KM variable at some point for a short amount of time, but it goes away really quickly. And KM can handle huge strings, so it's just not an issue.

[Scotty2Hotty]

I'm returning to this because I'm interfacing with the custom HTML prompt again and running into the same issues.

How much data are you talking about? For giggles I've just read in, "Filter" reversed, and written out a 110MB log file and KM didn't even blink -- took 7 seconds on my old, spinning rust, iMac.

If you make sure to use a local variable then the data won't be in "the KM environment" after the macro completes anyway.

I'm not super sure what you're saying here. Did you try to stress-test how much a global KM variable could store or how much the engine log could store? I'm assuming the former.

Personally, when I was getting used to KM, I was saving everything to global KM variables, from logs to CSV data to massive paragraphs of text. That's obviously a terrible way to go about things (I now try to use LOCAL and INSTANCE variables as much as possible). Still, even as a novice user, at least a couple of times, I hit my head against the issue of the KM environment getting 'full' and becoming unable to write anything else to variables because the existing variables contained too much data. You could see this error appearing in the engine log. Something to the effect of "too large for environment". KM slowed to a crawl, and variable reading/writing wasn't functioning correctly. I then spent a few hours spring-cleaning my variable list, reconfiguring macros to have variables cleared after use, and then learning about LOCAL/INSTANCE variables (a good learning experience for best practices).

I don't have this exact error message to hand anymore, but I remember at the time googling for answers and finding a couple of threads from other users with the same issue. The storage for KM variables is relatively small, and you can't expand it. Therefore, if there's ever a path to saving a particularly huge string of data that involves other methods, I'm inclined to go for them.

Anyhow, if I, as a beginner, hit this ceiling, it must happen for a few others, too. It seems intuitive to me now that you should priotirise using solutions other than KM variables if you are handling large data strings. An immediate example that comes to mind, because it's what I'm doing right now, is importing JS libraries

Here's how I handle the situation:

If the data can be passed as HTMLResult, then I return it there. Otherwise, I write a macro that takes the data in TriggerValue, and "Trigger" the macro from the custom HTML prompt.

Yes, I suppose technically the data is stored in a KM variable at some point for a short amount of time, but it goes away really quickly. And KM can handle huge strings, so it's just not an issue.

Interesting. Honestly, I was a bit bewildered by the %TriggerValue% token until recently and never tried using it. I'm guessing %HTMLResult% is similar - If I trigger a "return result" in the HTML prompt it is picked up by this token?

I will try that, but it's still a bit limiting that KM variables and tokens are our only means for loading/outputting data from HTML prompts (as useful as they are! The JSON tokens are incredible). I'm trying to move away from using shell scripts within KM to create HTML files with data already injected into them at the point of their creation, and I still feel like a madman using the engine log to output data.

I appreciate your suggestions, but I still think the custom HTML prompts should be able to natively read/write to the disk with support for reading/writing files directly. They're already one of the most powerful parts of KM, but feel a bit hamstrung by this limitation.

[DanThomas]

How much data are we talking about, coming from an HTML Prompt?

[Scotty2Hotty]

Potentially, several hundred lines of CSV data with multiple columns.

I know this is a non-starter for most data you might want to pull from a HTML prompt. My point is that using KM variables as a vehicle to load in / pull out data from the prompt is limiting.

Inputting data is actually much worse. I initially started using shell scripts to generate the HTML files from within KM with the data pre-loaded because I couldn't pull in more than a couple dozen CSV lines into a table without the data being cut off. Doing this means you have to escape all of the Javascript in your HTML, which is a super tedious thing to do to compensate. You can't use Local or Instance variables in this environment, either.

I'm currently trying to build a map using the leaflet libraries to present as a Custom HTML prompt. You cannot reference a local .js file to do this. I am definitely not going to try loading the 147,000+ characters in as a KM global variable, so I'm having to hardcode the libraries into the script. This all seems mad to me as workarounds for this limitation (i.e. no native file read/writing in the prompt)

Please do feel free to school me if I'm missing something obvious here.

[Nige_S]

@Scotty2Hotty

I'm not super sure what you're saying here

That was, IIRC, reading a 110MB file into a local variable, reversing it, and writing it back out again. So KM has no problems handling at least 110MB of data in a variable. You'll have trouble passing that much into a shell script because of the shell's limit on environment variables, but that's another matter altogether.

@Scotty2Hotty

but I still think the custom HTML prompts should be able to natively read/write to the disk with support for reading/writing files directly.

They probably can -- subject to certain limitations. See The File System API with Origin Private File System | WebKit. But it sounds like you want to write data directly to your general user-space, and WebKit simply doesn't allow that (for very good reason!).

@Scotty2Hotty

I am definitely not going to try loading the 147,000+ characters in as a KM global variable

Again -- if I can push 110MB though, 150,000 characters (or 150KB) should be a walk in the park...

[DanThomas]

I see that @Nige_S beat me to the size issue, So you need to get it out of your mind that KM variables are going to limit you.

Yes, if you put the data into global variables, KM would decide not to write it to the environments of shell scripts. But if you need it in a shell script, you can write it to disk. Although I wouldn't personally keep that amount of data in a global variable.

@Scotty2Hotty

Inputting data is actually much worse. I initially started using shell scripts to generate the HTML files from within KM with the data pre-loaded because I couldn't pull in more than a couple dozen CSV lines into a table without the data being cut off. Doing this means you have to escape all of the Javascript in your HTML, which is a super tedious thing to do to compensate. You can't use Local or Instance variables in this environment, either.

I'm not sure what you're saying here, so let me just say that you can indeed have the JavaScript in a separate .js file, and the CSS in a separate .css file. They just need to be in the same folder as the HTML file, and possibly in subfolders although I haven't tried that.

[Scotty2Hotty]

That was, IIRC, reading a 110MB file into a local variable, reversing it, and writing it back out again. So KM has no problems handling at least 110MB of data in a variable. You'll have trouble passing that much into a shell script because of the shell's limit on environment variables, but that's another matter altogether.

So I'll ask again -- how much data are you talking about?

Are global variables and local variables subject to different restrictions here? I was running into the issues I described with global variables. At that point in time I was using global KM variables for everything including logs, CSV data. There would be nothing exceeding large there. Honestly, I doubt it consumed more than a few megabytes's worth of text.

If these are global-variable-only issues, then that makes sense to me, but doesn't really help me given you cannot set/get local variables in a HTML javascript environment, which is the context to all of this.

Further to this, I believe I found one of the aforementioned threads where someone was running into this:

Why am I getting KM Engine Logs about Local Variables?

@peternlewis, why am I getting KM engine log entries for Local Variables being "excluded from environment to ensure the environment is not too large"? Since Local Variables do NOT persist beyond execution of a macro, why would they be in the environment? I'm keeping a close eye on my KME Log looking for suspicious entries. Do I need to take any action to prevent these logs? 0097 2018-08-06 18:05:54 Variable “KMVAR_Local__LogLines” excluded from environment to ensure the environment is not t…

@peternlewis replies as follows:

This happens when you run a script (AppleScript, Shell Script, etc).

As you know, the variables are placed into environment variables, with names that start with KMVAR.

However then total space for the environment variables is limited (by an unspecified limit, sigh), and if the total size of the environment variables exceeds this limit, then the script will not run at all (the attempt to start the script will fail). So Keyboard Maestro takes care to ensure that the total size of the environment variables it includes total up to less than around 100k. If the total size of all your variables (which includes non-password variables, local variables and instance variables) exceeds 100k, then the largest of them will be omitted.

The notification in the log is so if that happens to be a variable you really want to access, and it is not there, you have a change of figuring out what is going on.

You can always access larger variables via AppleScript to the Keyboard Maestro Engine as normal.

So no, you don't have to take any action to prevent these log entries and unless you need to use those specific variables in your scripts, you don't need to do anything.

I trust that is clear.

I actually didn't realise until reading this again now that that user did the same thing to get around the issue as what I'm doing (utilising the engine log for parsing large data strings). Unless something's changed, this is still the main method to getting around using KM variables all these years on. Perhaps it's time for a better solution?

Otherwise, I don't know what to tell you. Those were the issues I was running into it. I'm surprised you were able to load 110MB into one after having re-read that thread. KM was fully functional after you did that and when you continued to use KM for other things I'm guessing? No engine log errors?

if you know how to use local/instance variables in javascript in HTML, without the size restriction, I'm all ears.

They probably can -- subject to certain limitations.

Great! I'll take a look.

But it sounds like you want to write data directly to your general user-space, and WebKit simply doesn't allow that (for very good reason!).

Zero issues here as this is for personal use only.

I'm not sure what you're saying here, so let me just say that you can indeed have the JavaScript in a separate .js file, and the CSS in a separate .css file. They just need to be in the same folder as the HTML file, and possibly in subfolders although I haven't tried that.

I will have a go at this.

I see that @Nige_S beat me to the size issue, So you need to get it out of your mind that KM variables are going to limit you.

Again, I don't know what to say. These were issues I was having as a result of using variables. Clearly other people have had issues too.

I respect that you both don't percieve any issues here, and don't see the need for native file read/writing with the prompt action, but I disagree and still advocate that this would be a worthwhile inclusion.

[DanThomas]

@Scotty2Hotty

If these are global-variable-only issues, then that makes sense to me, but doesn't really help me given you cannot set/get local variables in a HTML javascript environment, which is the context to all of this.

You can get and set variables in a Custom HTMl Prompt, using

window.KeyboardMaestro.GetVariable()
window.KeyboardMaestro.SetVariable()

You can trigger a KM macro using

window.KeyboardMaestro.Trigger( macro, value )

For "Value", you can pass really really long strings if you want. The macro being triggered could then write the data to a disk file, for instance.

Honestly, regarding the issues you have encountered, there's nothing we can do without examples. But I suspect that the problems you ran into can be overcome, and if you want to provide real examples, you might be surprised at the answers you get.

Until then, I feel like we're just wasting bytes talking about it.

[Scotty2Hotty]

You can get and set variables in a Custom HTMl Prompt, using

I understood this only works for global KM variables and not local/instance variables. I might be mistaken.

You can trigger a KM macro using...
... For "Value", you can pass really really long strings if you want. The macro being triggered could then write the data to a disk file, for instance.

I will have a play with setting up specific macros intended to capture specific data, thank you.

Honestly, regarding the issues you have encountered, there's nothing we can do without examples. But I suspect that the problems you ran into can be overcome, and if you want to provide real examples, you might be surprised at the answers you get.

Until then, I feel like we're just wasting bytes talking about it.

Yes, I won't be able re-create those exact conditions, but I appreciate everyone's input here on this.

[DanThomas]

@Scotty2Hotty

You can get and set variables in a Custom HTMl Prompt, using

I understood this only works for global KM variables and not local/instance variables. I might be mistaken.

If your prompt is running synchronously, I'm pretty sure you can set local and instance variables, but I'd test that to make sure. You can read local and instance variables, but only in the state they were when the prompt was launched.

When you trigger a macro from the prompt, the macro can actually give a result back to the prompt, but you have to use an "Execute JavaScript in Custom HTML Prompt" action to do it, which is slightly annoying because that action doesn't allow variables. But even though that's annoying, it's pretty easy to work around. If you need more info, just ask.

[Nige_S]

@Scotty2Hotty

@peternlewis replies as follows:

And his very first line is:

This happens when you run a script (AppleScript, Shell Script, etc).

...which is what I said above. You're talking KM variables in general and using them in the Custom HTML Prompt in particular, so none of that applies.

@Scotty2Hotty

KM was fully functional after you did that and when you continued to use KM for other things I'm guessing?

Instead of asking me -- try it yourself! It would take 30 seconds to write a test macro.

The best way to learn about KM's features is to poke at them

@Scotty2Hotty

Zero issues here as this is for personal use only.

You misunderstand. WebKit allowing JS to read/write willy-nilly would be an huge security flaw, so Apple (and every other sane web engine developer) doesn't allow it. So the Custom HTML Prompt, which is based on WebKit, can't do it. So, regardless of whether

@Scotty2Hotty

this would be a worthwhile inclusion.

...it will not (as things currently stand in the web engine world) happen.

@Scotty2Hotty

I understood this only works for global KM variables and not local/instance variables. I might be mistaken.

Again, it's easy enough to find out. I'm no coder, so ripped the business part from the action's Wiki page:

Custom HTML Test.kmmacros (2.1 KB):

The gotcha is that you must not use underscores in your KM variable names (which I do as a matter of habit...).

So there seems to be nothing stopping you from using "Read File", local variables, and the Custom HTML Prompt to do what you need.

@Scotty2Hotty

Yes, I won't be able re-create those exact conditions, but I appreciate everyone's input here on this.

That's often the way, for many reasons. But you should be able to make a cut-down macro that shows any problems you are having, so post that if you're still stuck. There's a bunch of people here (not me!) who have done some very clever things with the HTML Prompt.

[Airy]

@Nige_S

I'm no coder,

That's very hard to believe since your knowledge, diagnostic abilities and communication skills are broad and deep.