Adventures in Linux porting #1

With 9.3.1 freshly released, we made a promise that 9.3.2 will finally bring Linux support!

Now this isn’t the first time we promised Linux support, but each time we tried, it turned out to be a very time consuming task: fixing hundreds of small bugs, most of them related to paths, while we would rather work on more pressing issues, like finishing the language and library for one platform first. But we did learn a lot from the path related issues and we will introduce a platform agnostic path class in the library, one that won’t allow you to use the wrong path format on the wrong operating system.

Anyway, each time we attempted Linux, we had to postpone it due to deadlines for the next release, but each time progress was made, so this final attempt should be relatively quick and easy and give great results. In theory! Let’s see how it goes!

While we need to finish this ASAP, it is still not possible to allocate too much time during the week for it because of other tasks. This means weekend work! My task for this weekend was to fix up as much as possible and prepare a status update for the project under Linux. On Friday night, I dusted off the old virtual machine and made it install all the updates.

On Saturday at 9 AM, I was up and ready to go. The first step was grabbing the compiler sources and seeing if it compiles. There was just one compilation error, because Linux file-system is case sensitive and Windows is not. We have the convention of threating Windows as case sensitive too in order to not have any problems, but a single file naming error snuck through. After fixing this, all the project specific compilation errors were gone, but one of the third party libraries we use had an older version installed on our Linux and it was not compatible. This library is available in a third party repository, not the main Ubuntu ones, but I’ve always had problems installing it from there: the installation works, but causes crashes when invoking the library. So I like to build it by hand. I’ve downloaded the latest sources and started the compilation.

18 minutes later it was done. Great! Mental note: now that Linux is a full-time platform, it might be time to no longer use VMs. Maybe use native install. Dual boot or new PC. After this, the compiler was compiled and ready for testing.

The first test I did was running the compiler from the command line without any parameters. And the first problem manifested: the compiler supports a lot of build methods, so you need to choose one. You need to give it as a command line parameter. The compiler also lists all the available configurations, so you can choose one. But it was complaining that you did not provide one and exited before showing you the list. The correct order is to show you the list, then complain that you did not choose one from the list. We use the command line compiler all the time, but not manually, so we never noticed that if you give it no parameters, it is not that usable.

I fixed this issue and also fine tunned GCC detection. On Saturday I was convinced the detection was made better. On Sunday I’m thinking the change I did was neutral. Anyway, the compiler was up and running, listing build methods and taking parameters correctly, so it was time to compile “Hello world”. The compiler exited with a short message telling me that compilation finished correctly in a very short period of time.

Naturally, I did not believe it! The amount of time passed was a clear give-away, especially on a VM, plus it running on the first attempt would be a minor miracle. I did not check the folder where the native binary is to be put, but first I checked the temporary build folder and found something nasty:

Blasted path bugs! This screenshot also reveals that the compiler is trying to pull in the Windows libraries instead of the Posix ones. And the builder was still thinking it is on Windows. I fixed these problems and everything was working as expected: all auxiliary bugs looked like they were fixed and compilation was failing as expected because some standard library functions were implemented with Windows API and they need to be ported over to Linux.

The compiler looked like it was successfully ported and I arrived to the phase where the library needs to be ported. Porting it requires examining compilation error messages and jumping around a lot from one source to the other, the perfect job for our resident Z2 IDE, ZIDE. I compiled ZIDE and it ran fine, only I did notice that it too was trying to pull in Windows sources:

I fixed this issue again and there needs to be in the future some mechanic added to fix this problem better. I also went though the code and removed a lot of Windows bias. For starters, I commented out the offending Windows API calls and tried to compile. ZIDE hung and I had to kill it.

After a lot of research, I found the culprit: under Windows, execution return codes are 32 bit, with negative values meaning failure and positive success. Under Linux, return codes are 8 bit, 0 is success and positive values are failure. So exactly the opposite. Every time the compiler exited with -1 32 bit, it became a positive number and ZIDE was not expecting that. I wrote a quick platform independent return code mapper and made ZIDE much more robust, so it no longer hung, but reported error gracefully when something unexpected was received.

So ZIDE is considered ported too. The next step was to look at what the compiler was feeding to the backend. As expected, it was not good, uncompilable, but there were no major issue. Some required files are put in the right places under Windows for backend support, but under Linux, such an interfacing profile was not created yet. I created it and will be included in all future builds from now on.

After this was fixed, it compiled correctly. Unfortunately, “hello world” was not doing anything. If you remember, I wrote above that I commented out the offending windows code and this included console support. This was fixed last time we attempted Linux, but it broke. The Z2 compiler has a great dependency analyzer, so all the Windows specific parts of the library can exist in peace as long as you don’t call them. “Hello world” was calling just on platform specific bit, the one that outputs to the console. And beyond that, the runtime environment we created need to report bugs and that uses the console too. There was no code to pull the console in though the dependency analyzer, so I fixed that.

Z2 inherited the “extern” import mechanism from C. It turns out that that system is not good enough for our needs and will be removed next version. Instead I came up on the spot with a new “bind” mechanism, much more powerful, and hacked in support for it into the compiler in 30 minutes. Now, this is not how we do things. Everything is properly designed, sometimes even feeling like the design work never stops and there is never enough time to code. So starting Monday, this bind feature will be properly discussed and designed. There are still a lot of unknowns, but I think it is better than “extern”.

Anyway, with the “ieee.posix” package made to use the new “bind” feature and a few more fixes, finally it was time to see it all put together:

Too good to be true! So I checked the binary file. It was named “.exe”, something I quickly fixed. And it was almost 1 MiB large. Turns out the builds were made with debug information and static linking. Good to know that these features work. But without debug information and using the Linux standard .so that you inevitable get even in a small hello world like program (nothing Z specific here), the executable was around 17 KiB and working properly.

These results are so good (I did further testing to make sure that all major bugs were squashed) that I don’t want to ruin my mood by running the test suite. Next time… I will probably say “1 out of 206 tests have passed successfully”.

So there is a lot of work still. “Hello world” by coincidence only uses a single Linux specific function, so porting it over was easy. The Z2 standard library tries to be as native as possible, meaning most functions are implemented in pure Z, no platform specific bindings, but for some things, like console output, file system work, getting the time, etc. this can’t be avoided. All those functions need to be ported over to Linux. Not just ported, but using the new “bind” feature. Which first needs to be properly designed.

This port is only a 32 bit one. A 64 bit Linux must be installed and tested toughly. And then there is CLANG support too which we need to add, not just GCC.

But this was a good start and a weekend well spent!

And who knows? Maybe someday there will be a Mac/iOS port? No promises!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s