Adventures in Linux porting #1

With 9.3.1 freshly released, we made a promise that 9.3.2 will finally bring Linux support!

Now this isn’t the first time we promised Linux support, but each time we tried, it turned out to be a very time consuming task: fixing hundreds of small bugs, most of them related to paths, while we would rather work on more pressing issues, like finishing the language and library for one platform first. But we did learn a lot from the path related issues and we will introduce a platform agnostic path class in the library, one that won’t allow you to use the wrong path format on the wrong operating system.

Anyway, each time we attempted Linux, we had to postpone it due to deadlines for the next release, but each time progress was made, so this final attempt should be relatively quick and easy and give great results. In theory! Let’s see how it goes!

While we need to finish this ASAP, it is still not possible to allocate too much time during the week for it because of other tasks. This means weekend work! My task for this weekend was to fix up as much as possible and prepare a status update for the project under Linux. On Friday night, I dusted off the old virtual machine and made it install all the updates.

On Saturday at 9 AM, I was up and ready to go. The first step was grabbing the compiler sources and seeing if it compiles. There was just one compilation error, because Linux file-system is case sensitive and Windows is not. We have the convention of threating Windows as case sensitive too in order to not have any problems, but a single file naming error snuck through. After fixing this, all the project specific compilation errors were gone, but one of the third party libraries we use had an older version installed on our Linux and it was not compatible. This library is available in a third party repository, not the main Ubuntu ones, but I’ve always had problems installing it from there: the installation works, but causes crashes when invoking the library. So I like to build it by hand. I’ve downloaded the latest sources and started the compilation.

18 minutes later it was done. Great! Mental note: now that Linux is a full-time platform, it might be time to no longer use VMs. Maybe use native install. Dual boot or new PC. After this, the compiler was compiled and ready for testing.

The first test I did was running the compiler from the command line without any parameters. And the first problem manifested: the compiler supports a lot of build methods, so you need to choose one. You need to give it as a command line parameter. The compiler also lists all the available configurations, so you can choose one. But it was complaining that you did not provide one and exited before showing you the list. The correct order is to show you the list, then complain that you did not choose one from the list. We use the command line compiler all the time, but not manually, so we never noticed that if you give it no parameters, it is not that usable.

I fixed this issue and also fine tunned GCC detection. On Saturday I was convinced the detection was made better. On Sunday I’m thinking the change I did was neutral. Anyway, the compiler was up and running, listing build methods and taking parameters correctly, so it was time to compile “Hello world”. The compiler exited with a short message telling me that compilation finished correctly in a very short period of time.

Naturally, I did not believe it! The amount of time passed was a clear give-away, especially on a VM, plus it running on the first attempt would be a minor miracle. I did not check the folder where the native binary is to be put, but first I checked the temporary build folder and found something nasty:

Blasted path bugs! This screenshot also reveals that the compiler is trying to pull in the Windows libraries instead of the Posix ones. And the builder was still thinking it is on Windows. I fixed these problems and everything was working as expected: all auxiliary bugs looked like they were fixed and compilation was failing as expected because some standard library functions were implemented with Windows API and they need to be ported over to Linux.

The compiler looked like it was successfully ported and I arrived to the phase where the library needs to be ported. Porting it requires examining compilation error messages and jumping around a lot from one source to the other, the perfect job for our resident Z2 IDE, ZIDE. I compiled ZIDE and it ran fine, only I did notice that it too was trying to pull in Windows sources:

I fixed this issue again and there needs to be in the future some mechanic added to fix this problem better. I also went though the code and removed a lot of Windows bias. For starters, I commented out the offending Windows API calls and tried to compile. ZIDE hung and I had to kill it.

After a lot of research, I found the culprit: under Windows, execution return codes are 32 bit, with negative values meaning failure and positive success. Under Linux, return codes are 8 bit, 0 is success and positive values are failure. So exactly the opposite. Every time the compiler exited with -1 32 bit, it became a positive number and ZIDE was not expecting that. I wrote a quick platform independent return code mapper and made ZIDE much more robust, so it no longer hung, but reported error gracefully when something unexpected was received.

So ZIDE is considered ported too. The next step was to look at what the compiler was feeding to the backend. As expected, it was not good, uncompilable, but there were no major issue. Some required files are put in the right places under Windows for backend support, but under Linux, such an interfacing profile was not created yet. I created it and will be included in all future builds from now on.

After this was fixed, it compiled correctly. Unfortunately, “hello world” was not doing anything. If you remember, I wrote above that I commented out the offending windows code and this included console support. This was fixed last time we attempted Linux, but it broke. The Z2 compiler has a great dependency analyzer, so all the Windows specific parts of the library can exist in peace as long as you don’t call them. “Hello world” was calling just on platform specific bit, the one that outputs to the console. And beyond that, the runtime environment we created need to report bugs and that uses the console too. There was no code to pull the console in though the dependency analyzer, so I fixed that.

Z2 inherited the “extern” import mechanism from C. It turns out that that system is not good enough for our needs and will be removed next version. Instead I came up on the spot with a new “bind” mechanism, much more powerful, and hacked in support for it into the compiler in 30 minutes. Now, this is not how we do things. Everything is properly designed, sometimes even feeling like the design work never stops and there is never enough time to code. So starting Monday, this bind feature will be properly discussed and designed. There are still a lot of unknowns, but I think it is better than “extern”.

Anyway, with the “ieee.posix” package made to use the new “bind” feature and a few more fixes, finally it was time to see it all put together:

Too good to be true! So I checked the binary file. It was named “.exe”, something I quickly fixed. And it was almost 1 MiB large. Turns out the builds were made with debug information and static linking. Good to know that these features work. But without debug information and using the Linux standard .so that you inevitable get even in a small hello world like program (nothing Z specific here), the executable was around 17 KiB and working properly.

These results are so good (I did further testing to make sure that all major bugs were squashed) that I don’t want to ruin my mood by running the test suite. Next time… I will probably say “1 out of 206 tests have passed successfully”.

So there is a lot of work still. “Hello world” by coincidence only uses a single Linux specific function, so porting it over was easy. The Z2 standard library tries to be as native as possible, meaning most functions are implemented in pure Z, no platform specific bindings, but for some things, like console output, file system work, getting the time, etc. this can’t be avoided. All those functions need to be ported over to Linux. Not just ported, but using the new “bind” feature. Which first needs to be properly designed.

This port is only a 32 bit one. A 64 bit Linux must be installed and tested toughly. And then there is CLANG support too which we need to add, not just GCC.

But this was a good start and a weekend well spent!

And who knows? Maybe someday there will be a Mac/iOS port? No promises!


Z2 PT 9.3.1 Available for Download

Z2 Compiler version PT 9.3.1 is now available for download under Windows!. Here are some links:
PT 9.3.1
Prepachakged GCC TDM for Windows 32/64 bit
Previous versions
Main GitHub page for standard library

This version brings minor improvements and fixes, but also fixes a massive backend compilation performance/resulting binary size issue. The fix is not that pretty but it works. Unfortunately, it is only supported for modern backends, so for older ones an even uglier fix must be found. This issue will continue to improve as versions get released.

But the main thing that this version has is better documentation. Last version we intruded the basic documentation architecture, but that was the first time we attempted anything like this so it was far from perfect. A new 2.0 documentation format has been introduced, much more readable and powerful. Additionally, more classes have been documented. This is an ongoing process and we’ll make an announcement when all of the API has been documented.

When some of the most common container algorithms were added to the standard library, the focus was on the design of the API and correctness, not on performance. So a lot of the common operations like insertion and deletion were created to set the tone of the library, but they are not particularly fast. This version we experimented with a standardized set of optimizations that will be rolled into the object model to help make the API as fast as humanly possible. The experiment show that performance has been multiplied by a double digit number in the worst case scenarios, meaning at least ten 10 faster. This applies for any subclass that can be inserted into containers. We also experimented with a particularly aggressive form of optimization that can only be applied to more special classes, where the performance gain can be more than doubled.

We left out the aggressive optimization out of 9.3.1 and only included the normal one in some very small parts as a test. If no issues are found, 9.3.2 will have both optimizations included everywhere.

So what is next for 9.3.2? Linux version of course! Except for optimization described above and smaller fixes we can include, 9.3.2 is on feature freeze and all the work will go into finally finalizing Linux support!

Here is the change-log for 9.3.1:

Z2 PT 9.3.1
– literal constants keep track of base 16 in C++ backend output
– better C++ backend else if handling
– const, mem and return nodes cleanup
– refactored literal CArrays

– new AsciiParser class

– new doc format
– option to regenerate all docs
– empty doc entries in the DB are marked

– massive backend compilation time fix for large CArray literals
– nameless constructors fixed for core types
– calling nested CArray method bug fixed


In the heat of the release announcement, I’ve completely forgot to mention the new automated build scripts. Creating a build is not that easy or fast: every time we need to go though roughly 14 steps and it can take up to 2 hours. Every build we get faster and faster at it, but you still need to do a fresh compile of 3 binaries, package sources, licenses and other files, run the basic test suite, the full test suite and do some visual QA, since we have a GUI too (our IDE) and GUIs are hard to test without somebody using their eyes.

So we created an automated script that removes some of those steps and cuts down time by 20-30 minutes. It is not that smart yet, but eventually it will do the whole process from checking out the source code to running tests, leaving just the visual QA as a manual step.

PT 9.3.1 is the first release created by the script, so if it is worse than usual or missing something, blame the scripts! All versions from now on will use the script and it will be improved each version.