PalOMoney Blog

Wouldn’t you know, after so many years, the Yahoo Finance API has stopped working. All the various scripts from us and other developers no longer work with the message:

“It has come to our attention that this service is being used in violation of the Yahoo Terms of Service. As such, the service is being discontinued. For all future markets and equities data research, please refer to finance.yahoo.com.”

Of course, we don’t know what the terms of service are or were, but the buzz on the web is that Verizon, the new owner of Yahoo, is no longer interested in this sort of free service. This makes perfect sense.

But, no closing quotes for Friday, November 3, 2017. I can’t go back to entering prices by hand. I miss seeing the bubble expand every week, and hand entering prices would spoil the thrill of it all.

Since quotes are half of PalOMoney (the other half is Direct Connect for statements), it seems only fitting that the PalOMoney developers should spring into action, or at least awake from their slumber. Accordingly, research is ongoing into what free and paid services could be used instead of Yahoo Finance.

One service that has been talked up a bit is the free Alpha Vantage quote service. This service requires a (free) API key that you can immediately obtain from their web site. This is a .co not .com site, which might mean something. At any rate, anyone can instantly get this API key that works for all sorts of useful security and currency price time series. And the quotes are supposedly real-time (not delayed).

The PalOMoney and MoneyPal applications have been updated to use the Alpha Vantage Quote Source included in version 1.0.8. Find directions on the wiki. All existing users should be able to download the new version at no charge.

Requiring an update because a product no longer functions is a very undesirable characteristic. Because of this, it seems prudent to implement additional quote sources as well. There are many sources for real time and delayed quotes, some free, others paid. We would like to solicit suggestions for additional sources to add to PalOMoney. If you have any suggestions, please post them in the forum, or use the contact form to request a module. We will try to implement the easiest and most requested suggestions.

There was no blog post for PalOMoney Versions 1.0.6 and 1.0.7, since they were small updates to the system.

1.0.7 OFX 103 Supported

Chase was no longer working, returning a 15500 authorization error since last fall 2015. The message returned said to open the web page and look for a message in the secure messaging center. But there was no message there. Chase customer service was not helpful. Fortunately, others using Pocketsense and other scripts figured out that Chase was requiring OFX 103 instead of 102. OFX 103 is very similar, but it adds authentication goodies. Chase uses and requires a so-called ‘CLIENTUID’ value in the sign-on. This globally unique (no two the same) value identifies the software and computer to the server. PalOMoney has been updated to support a specified OFX version for each provider (server), plus generate and save a UUID for each installation. After testing the software and logging into Chase with OFX 103, there was a confirmation message in the secure message center, and, after clicking the link, the downloads worked.

Apparently, there is a problem with multiple accounts at an institution using the same UUID. Accordingly, the UUID needs to be a property of the account, and not the money file, or the provider. Before this version is released, a mechanism for generating unique UUIDs for each account at an institution will be implemented. Also, the control for specifying the OFX version will be changed to a radio button or so because the software supports only OFX 102 and 103. This is a work in progress.

1.0.6 Problems Fixed

This is a bug fix release downloaded by parties who requested bug fixes. The problems are discussed in the forum. Will document this more fully later.

However, that never happened. Perhaps one day, it will.

After a few months and plenty of distractions, PalOMoney 1.0.5 has been released for testing. This version adds “direct connect” bank, credit card, and investment statement downloads and decent integration with Money. It also includes a patch utility app to fix defective Money software, as discussed in the wiki article Money Sunset Import Bug Remediation. As a public service, the patch utility is available to anyone as a free standalone download.

By design, the ofx statement downloads are driven by Money file settings that would have been used for the MS Money online services. Put another way, setting up a Money file for direct connect downloading involves setting it up as if Money were to do the downloading itself, instead of PalOMoney. This might just enable original Money 2008 (not Sunset) versions to resume downloads, but that will require more experimentation.

The statement download settings are much more complicated than the Money download substitute software currently available. The dialogs that control the settings are a user-interface nightmare, with dozens of controls, with a few modes thrown in. This kind of user interface design is not suitable for a non-technical user. No wonder the Money interface used so-called “branding servers” and databases to automatically configure their arcane settings! Because of this, the main settings dialog has an import and export capability that can be extended to automatically download settings from a web site or on-disk database, so that an easier user interface can be implemented. But it is still nice to have the “advanced” interface, with access to all parameters, for those who like to do it themselves.

The new PalOMoney Statement view window provides an interesting view into your finances. After download all your statements, the window shows the unreconciled entries (or statement items). Sorting this in inverse date order, newest on top, you can see what has just been charged to your accounts, without waiting for the monthly statement, which we don’t read anyway. This can make for some interesting conversations regarding where the money is going. The statement items disappear after you reconcile them in Money, so the list is only of the transactions of which you are not fully aware, and haven’t fully reconciled with your budget. I wonder if this new view will change any spending practices.

We still regret this business about fixing broken Money software. Because the Money source code is unavailable, a tremendous amount of effort is required to derive information using a debugger, which displays only the raw assembly language, if you’re lucky. It would be really great if some reverse engineer, professional or not, could take over this part of the project. Feel free to volunteer!

 

PalOMoney OFX Statement download has a Money import method that writes directly to the Money queue and notifies Money, bypassing that dreadful Money shell import handler that launches when you “open” an .ofx file, and requires you to click Ok for each download if Money is not launched. When Money is launched, an equally obnoxious offender is the Money “Import a file” dialog, which is a bit more pleasant on the eye but ultimately just as annoying. Who wants to wait an indeterminate amount of time, only to click Ok 16 times in a row? Something had to be done.

SayOk Auto Closer

The SayOk Auto Closer utility is a separate app that sits in your system tray and watches for a certain window title, in this case “Import a file.” When it notices a window with that title, it looks for a likely “Ok” button, and activates it, thereby closing the window. SayOk opens a notification balloon whenever it closes a window, and logs the closure in its display pane, and in a log file in your Documents folder. The app can be extended to look for other annoying windows.

See the wiki SayOk Auto-Closer page for up-to-date information.

Microsoft Money Sunset users who experience crashes importing OFX files can download and use the new free experimental utility PatchOMoney from PalOMoney.com. The software comes with no warranty, and the user assumes all risks. The user must agree to back up their Money installation before using the software.

To download the software, start at the wiki page https://www.palomoney.com/wiki/index.php/Money_Sunset_Import_Bug_Remediation. Read the article, then follow the link to the  Money Sunset Patch Utility link. The download link is at the end of that article. Unzip the file, and follow the instructions. Good luck.

If this patch does not work for you, please post a message to the PalOMoney support forum.

Anyway, we can now go back to releasing PalOMoney 1.0.5, which was postponed prior to resolution of the OFX import crash issue.

Introduction

PalOMoney version 1.0.5 release candidate build 893 was being tested with Money Sunset on a 64 bit Windows 7 machine. While testing the Statement view Update Now function, it downloaded about a dozen statements, each time calling the (annoying) Money Shell Handler to queue them for processing in Money. After 12 Ok clicks, the downloaded and log files were examined and found to be Ok. However, when Money Sunset was launched, it crashed after a few seconds.

PalOMoney was used to repair the Money file outside of Money. Money was launched again, and suffered another crash. There was no way to prevent a crash. However, the Money file worked fine on another machine. After renaming the Money file, and opening an older file, the crash still occurred. The Money Sunset environment itself had become unstable.

This was not an auspicious beginning for PalOMoney version 1.0.5 release candidate build 893, and delayed the release of 1.0.5 pending a resolution of this issue. Ordinary users cannot tolerate this kind of performance.

This topic also has a wiki version, which will be kept up to date.

Research

It wasn’t really a problem with PalOMoney. There is a known problem importing OFX files into Money. Referring to Microsoft’s Raymond Chen’s Microsoft Money crashes during import of account – MSDN Blogs and Windows 8 64 Bit Version Compatibility with Sunset Money/. The 4 byte patch in the articles was applied, but Money still crashed. More research turns up MS Money Sunset+Windows 8, Crash when adding new Payee, which has a similar problem.

In order for Microsoft Money Sunset to be viable for ongoing use, it will require patches to fix bugs possibly introduced when downgrading Money 2008 to produce Sunset. This task has been (reluctantly) added to the PalOMoney project, but it adds the task of reverse engineering to the project.

Reverse Engineering to Correct Bugs

In the words of commenter to Chen’s blog entry above, “Reverse-engineering is a violation of the Microsoft Money EULA…,” and PalOMoney.com does not want to run afoul of that issue. Another commenter wrote that “There are certain court rulings about warranty of merchantability that override the EULA in situations like this one.” Although this comment is not a duly prepared legal memorandum that can be relied upon safely, PalOMoney.com has done the necessary examination and identified a bug and has developed workarounds and remediation.

Any readers with insight into this issue are encouraged to add it to the wiki topic or discussion, blog comments, or discuss it in the forums.

OFX Crash Bug Cause

The fault occurs when doing a string compare on uninitialized memory.

Apparently, the bug is caused several functions up the call stack by failing to initialize in all circumstances what seems to be an in-memory hash table. The initialization occurs in two phases. First the table is initialized with zeros to indicate no entries. Then an entry is made into the table. Each phase tests a pointer in a large object supplied by the calling function, and either skips or performs the initialization. When the system crashes, the first test skips the initialization. A call to the hash table find function then calls string compare on an entry in the uninitialized hash table, which causes the fault.

The crash occurs with OFX downloads from certain accounts at certain financial institutions. However, not all downloads of those accounts cause a crash. The cause of this is being investigated and will be reported here as answers are discovered.

Money Sunset crashes when processing the “Import Files” queue. This queue is not part of the Money file, so it will cause a crash when opening any Money file on that machine.

The “Payee Change” crash mentioned in the above blogs has not been investigated. It may be caused by the same problem.

OFX Crash Bug Workarounds

One workaround is to not import transactions from problem accounts at problem institutions, or to import them using Money’s File Import, and NOT by double clicking them and NOT using the Shell Open method to process downloads via PalOMoney, both of which add the downloads to the Import Files queue. However, this does not help very much when a user’s Money Sunset installation is unstable. So the first thing to do is to fix the installation, below.

The Money file is left with open objects by Money crashing. This is fixed either by opening it with Money (which repairs with a “Working…” progress bar, or by running the the PalOMoney File | Repair | Repair function.

The “Import Files” queue is saved in the registry under [HKEY_CURRENT_USER\Software\Microsoft\Money\17.0] “Import Files” key. The MULTI_SZ value contains the names of temporary files copied by the Money shell handler. The key is deleted (not there) when the queue is empty.

You can just delete the key, but PalOMoney adventuresome explorers are encouraged to rename the entry instead of deleting it.

• Open RegEdit. You may not need administrative privilege to access HKCU. Try it. If you do, run RegEdit as administrator.
• Locate the “Import Files” key, and press F2 to edit the key name.
• Add a space and “00”, “01”, etc. to the key name (e.g., “Import Files 02”).

This leaves the files in your TEMP folder (e.g., \users\UserName\AppDate\Local\Temp), which are names with a “~of” plus a random number then .tmp (e.g, ~of896.tmp). You can ignore them for now, and use them for testing the remediation, below.

PalOMoney is being enhanced to include these functions. Also, PalOMoney will offer a direct queue import method, which write to the registry key directly, bypassing the annoying Money Shell Handler with its “Launch Money Now?” nagging.

Finally, the investigation into why some downloads crash and others don’t continues, and may result in PalOMoney being able to “filter” a download to add required or remove problematic OFX entries to prevent the crash in the first place. However, this would not fix the general problem of crashing when importing or changing payee names, etc. mentioned in the blog posts above.

Ofx Crash Bug Remediation

The Money Sunset software can be patched to eliminate the first test that skips the hash table initialization by replacing the test and jump with NOPs (we regret not having a more elegant solution). The exact dll location has to be calculated, and will be published along with a simple .exe file to apply the patches Money Sunset Patch Utility here when known.

The PalOMoney application cannot perform this function, because patching a program file requires elevated administrator privileges, and it is undesirable to run PalOMoney or any ordinary application with elevated privileges.

PalOMoney is working on a patch utility, which will be made available on this site to anyone without charge. The utility will require administrator elevation to run. Whether the Money Sunset software is under Windows file protection is unknown, but Chen’s article leads one to conclude that the software can be patched, and not automatically revert to an earlier version.

This was supposed to be a couple of days’ project, setting up a new web site to handle the PalOMoney project. Well, how time flies! Time is up, and the site is still not complete. It’s going to have to go live, anyway, and our apologies to users who will have to suffer with it.

For those of you who are not familiar with web site construction, it is possible to subscribe to a hosting service and implement a web site with just a few clicks. Oh, happy life! However, this site is not a hosted service. Instead, it is on its own server in the cloud, which means that everything is possible, and everything can go wrong. Having dedicated servers is necessary because one of the purposes of the site is to provide substitute quote feeds to MS Money users. But no one here has ever done web site or public server administration before!

This site consists of a WordPress blog, a MediaWiki wiki, a store for download control, a Phorum forum, and a Bugzilla issue (or bug) tracker for support. These portals are powered by popular, well respected software that nearly every web user has encountered on many other web sites.

Prior to this moment, the most critical web site issue was integrating the various portal logons into a unified site logon that would work with all the portals. It was the top priority project for the web site.

Now there an even worse problem, which has triggered this post. It is the new Bugzilla issue support portal, intended for use as customer support portal. The idea is to combine support tickets, bug reports, and enhancement requests plus road map reports into one web application, and Bugzilla does most of that beautifully. However, there is a fly in the ointment: Bugzilla requires site login using the user’s valid email address as the user name. This email address is viable to anyone who visits the portal. This is horrible if you care about spam not to mention privacy. Here is the text from the “create a new account” page:

“To create a PalOMoney Support account, all you need to do is to enter a legitimate email address. … PRIVACY NOTICE: PalOMoney Support is an open issue tracking system. Activity on most issues, including email addresses, will be visible to the public. We recommend using a secondary account or free web email service (such as Gmail, Yahoo, Hotmail, or similar) to avoid receiving spam at your primary email address.”

This is a ridiculous requirement. We refuse to compromise any user’s privacy and email to the merciless spamming web bots, secondary email account or not. This makes the portal unusable. We will have to go live using the forum for support, and lock everyone (especially spammers) out of the support portal until this is resolved.

Prior to this, we tried Redmine, which is too much a project manager. We now have to evaluate Trac or Mantis. But it looks like this project has to be pushed into the future, because it is impacting the site’s core purpose, which is, lest we forget, to develop the alternatives to MS Money.

Anyway, although it has been great fun playing system administrator on the site, it is now time to recruit a web master who can take this over, and make things right.