Saturday, October 11, 2014

GNOME Bugzilla attachment status

I have been asked to help in migrating GNOME's Bugzilla instance to a latest stable version. Currently it is a version 3.4 with lots of modifications (or extensions, if you will). Quite an oldie - it is 2.5 year old. And it (obviously) does not get any bugfixes or security updates anymore.

If I remember correctly, I was asked because I admitted to Chris that I know a wee bit of Perl and I did not dislike it. So when Kat (a member of GNOME Foundation Board) asked him if he knows anybody who could help with it, he directed her to me.

(Did I mention that I, Chris and Kat used to work in the same company, at Openismus, for a short while? "lovely how our company links still work" said André, GNOME Bugmaster and also a colleague from the company. Heh heh.)

I am no Bugzilla expert, but after reading a diff between stock 3.4 and GNOME Bugzilla I had an impression that 3.4 is not really extensible in clean way - lots and lots of modifications are sprinkled all over the place. In default templates, in Bugzilla core, in CGI scripts.

So I was asked to help in porting the attachment status to 4.4. From user's perspective this extension only adds three things:
  • Another column to attachments table in bug view (example) showing status of patch. Or nothing, if attachment is not a patch. 
  • A combobox in attachment editing page (example). Click "edit details" at the top of the page to see it.
  • An option to fields combobox in advanced search page (example, see "custom search" section).
Development was painfully slow, mostly because it was done in my oh so copious free time. But, after 5 months or so, it is finally there, ready for testing. Just see the example links above. The code is on GNOME git now. Note that this extension is probably not compatible with an old one, so third-party tools might not work with it. I also managed to keep all the code inside the extension, that is - no changes in core. It is unfortunately not very pretty.

Any future work on this extension? Probably making third-party tools (namely git-bz) work with it. And maybe adding an attachment specific section to advanced search if GNOME bug wranglers want it.

Monday, July 23, 2012

Gmmproc progress

I have recently pushed over 30 commits to glibmm gmmproc-refactor branch. This concludes gmmproc's first "milestone" - to generate glibmm sources, build a library, build the tests and examples and finally succesfully run the tests. But by saying "glibmm" I mean only GLib wrappers - Gio wrappers are next on the list.

First milestone?


Yeah, right, as if there are any. That "first milestone" may look modest, but I actually think that it is like 75% of work done. GIR parsing, type conversions system, parsing templates, replacing m4 code with perl and then tying them and squeezing them with foot into some semblance of a program - all of those had to be written to have something working.

I haven't yet made any benchmarks of gmmproc rewrite, but right now it seems to be faster than the old one. Still old one can be faster if we run parallel build (make -jX) - there is one of main differences between currently used gmmproc and its rewrite. The former processes and writes one set of files (like from widget.hg and widget.ccg to widget.h, widget.cc and widget_p.h) on every run, while the former processes all files in single run. That forces me to actually do parallel processing inside my program instead of reusing make's powers. Fortunately, I tried to write most of my code in a way that parallelizing it would be easy if done in OpenMP style.

Talking about speed - there is one area I would like to speed up in general - GIR parsing. It looks like the slowest part of whole gmmproc, but I didn't yet tried to profile it. I was thinking about using typelib instead, but from what I heard, there is no detailed C type information there (the "c:type" attributes from gir). This is still to be confirmed as it is not on top of my nowhere written list. Otherwise, using typelib would be, I imagine, faster. Provided that C-to-perl interface to GIRepository is written first - whooo, yak shaving!

Some plans


Still, I feel no hurry in rewriting as I rather see it to be used in next version of glibmm/gtkmm (I assume that those will be glibmm-4 and gtkmm-4). Thus I am not writing a strict drop-in replacement - I already have made (or I plan to make) some changes in generated API (like exception-to-GError conversion in vfuncs wrappers) and in template API (I would like to get rid of _CUSTOM_CTOR_CAST and similar in favour of _CLASS_FOO options).

There is still much to be done beside threading the application:
1. generating Doxygen documentation based on docs in GIR,
2. code documentation,
3. gmmproc macros documentation,
4. tests (there are some, but still not enough),
5. reports about unwrapped classes and functions,
6. and probably more.

I am happy to get this far and to actually finally see something more or less working.

Small disclaimer


The code is ugly mess. It needs to be reorganized, reshuffled in several places. WrapParser class is huge, even with moving some functions into shared modules. 'use' clauses have to be reviewed. 'constant' module for section traits was probably a crappy idea. Sometimes class/function naming is bogus (identity conversions, imbued types, tokens store?). Code style is very inconsistent (I started reading Modern Perl somewhere in the middle).
Oh, and commit messages are useless - they are probably going to be squashed at some point into single enormous commit for merging into master (and I am going to get only +1 commit of contribution on glibmm on Ohloh, damn!).

Wednesday, June 6, 2012

SyncEvolution

These are bit dated news (well, only two weeks or so), but the fork/exec rewrite of DBus server was merged into master branch. The main aim of rewrite was to allow several concurrent synchronisations to be done (provided that the sync sessions don't conflict with each other). It wasn't possible before, because libsynthesis (the main library used for sync) does not have asynchronous API. There were two solutions: either use threads or spawn children processes doing syncs. The latter solution was chosen, because Patrick Ohly did not want threads to avoid complications with some possibly thread-unaware libraries. (Of course waiting for libsynthesis to get asynchronous API wasn't an option.)

The rewrite involved splitting the code in one file of syncevo-dbus-server into several files and then detangling all tightly coupled classes - I guess that the result is quite nice compared to what it used to be. In the meantime new C++ DBus layer based on GLib GDBus was added. Most of this work was done by Chris Kühl - I was mostly helping in fixing issues in new DBus layer and then working full-time on it and the code using it. As a final step I ported command line test from C++ to Python, so our work could be proved to work.

This was a really long task, very often I felt that I was losing grasp on the code I was working on and probably sometimes I had actually lost it. Now I feel happy to see it finally merged into master.

Tuesday, February 14, 2012

gmmproc refactor

What?

gmmproc is a script used by most of C++ bindings of GNOME stack (called also mm-modules). It is responsible for generating actual C++ headers and sources from templates. That is so, because wrappers for most C API are straightforward, so writing them manually would be just a tedious job. Instead of that we just provide C API information, specify what names should C++ wrappers have and how to get a C instance from C++ one and vice versa and all happens automagically. Even documentation from C sources is taken and translated to use C++ names.

1 or 2 years or so ago.

Some time ago I was thinking about refactoring gmmproc. I was not the only one, because there were several notions that a rewrite could be welcome. I even made a topic on gtkmm-list about it and created a page at live.gnome.org. Main points would be using GIR as a source of API information and documentation, and getting rid of m4 as it makes it hard to add new features. I had some grand plans like creating two backends - one for defs (old API information) and GIR (new API information), creating conversion files on-the-fly, some configuration file a'la Doxygen, blah blah blah. Eventually I started some coding in Python, drawed some diagrams in my sketchbook, switched to C++ and then Perl and suddenly development slowed to molasses, because I already got bored. Apparently writing API structures is really uninteresting. Especially when it goes really slow and no end of it is in sight.

Now and ongoing.

I got back to it recently with a firm decision that I will finish it. I got over API structures by generating them basing on GIR definition I had to deduce from some python code in gobject-introspection (yeah, I wrote my own definition, because there seems to be none of such thing upstream - documentation is pitifully out of date). Also, most of the parser is also generated, but most important things are hand-written. Then I switched to refactoring a WrapParser and Output, which are 10 years old Perl code, added some needed infrastructure replacing m4, converted _CLASS_GOBJECT from m4/defs to Perl/GIR, added a scanner finding C <-> C++ type equivalents. My first aim is to have any compilable code being generated. For that still much work left to be done - mostly understanding and translating rest of m4 code to Perl (booooring!) and writing C <-> C++ conversion code (the one saying how to get `GtkWidget*' from `const Gtk::Widget&'; slightly less boring, but still).

Future plans are... no no no, no future plans for now. I did that before and got baffled by amount of work that needs to be done. Slowly all needed features will be introduced.

All of the code sits in gmmproc-refactor branch of glibmm. Additions here are mostly a bunch of kilo-line commits happenning once for two-three weeks, because I work on it offline at home in my free time. In the end all of it is going to be probably squashed into single commit before merging it to master.

I am usually pushing changes to repository when my single patch grows to at least 500kb. Now it has only 200 kb, so you have to wait. :)

Monday, December 5, 2011

SOAP server example

Recently at Openismus I was asked to write a SOAP server, which would run on an OpenWRT router set up by Dave. The server uses libsoup, so writing it was really straightforward, but fun. Of course there were some glitches, like trying to use SoupAddress to set up where server should listen - it ended up in segfaults somewhere inside libsoup. I decided to not use it in the end, but Jens gave me a hint about a function of SoupAddres resolving it first. Segfaults are clearly a sign of bug somewhere in libsoup. Primarily SOAP server had to be someting bigger but for now ended up as an example. There are some things lacking for sure. Like actual compliance to SOAP standard (probably), documentation or subclassing the SoupServer into SoapServer and providing some nice interface hiding all ugly details. But at least it has autotools based build system, client application and some basic tests. So it maybe doesn't look that bad. The code (LGPL 3+) is on Openismus Playground.

Friday, October 28, 2011

Key overrides in Maliit

After finishing my work on Syncevolution's build system I was asked to get in touch with Maliit. My new task was to write some examples showing how to use dynamic key overrides in both application and plugin side, simplify a bit the use of them in application side and implement key overrides handling for QtQuick plugins.

Examples

Key overrides are useful because they allow application author to make action key (this is what we usually call "Enter") of virtual keyboard have different caption (e.g."Send" in an instant messaging application) instead of standard bent arrow icon. Imagine having an application which logs you to some online service - all it could have is two fields, one for login and another one for password. Depending which of this fields has focus the action key caption could be "Next" or "Login" respectively. In fact, I wrote a sort of mockup showing it - it is called twofields:

Twofields example application

There are also examples of C++ and QtQuick plugins reacting to changes of action key caption. You can find them in examples directory in Maliit framework repository.

Simple overriding

Overriding a key from application side is done in three steps:
  1. Create Maliit::AttributeExtension.
  2. Call setAttribute method of Maliit:AttributeExtension.
  3. Bind attribute extension to some input widget.

Previously the third step was a bit messy - you had to subclass an input widget and override its inputMethodQuery() method to return an id of attribute extension:

class MyLineEdit : public QLineEdit {
public:
    MyLineEdit()
        : QLineEdit(),
          extension (new Maliit:AttributeExtension)
    {
        extension->setAttribute("/keys/actionKey/label", "Wheee");
    }

protected:
    QVariant inputMethodQuery(Qt::InputMethodQuery query) const {
        typedef Maliit::InputMethodQueryExtensions MaliitQuery;

        MaliitQuery maliit_query(static_cast<MaliitQuery>(query));

        switch (maliit_query) {
        case Maliit::InputMethodAttributeExtensionIdQuery:
            return QVariant(extension->id());
        default:
            return QLineEdit::inputMethodQuery(query);
        }
    }

private:
    QScopedPointer extension;
};

Now, forget about the above code. The one below will do:

Maliit:AttributeExtension* extension(new Maliit:AttributeExtension);
QLineEdit* line_edit(new QLineEdit());
extension->setAttribute("/keys/actionKey/label", "Wheee");
line_edit->setProperty(Maliit::InputMethodQuery::attributeExtensionId,
                       QVariant(extension->id()));

All names of properties are in maliit/namespace.h.

Key overrides in QtQuick plugins

QtQuick plugins now also can have overriden action key. It is a matter of some property bindings and setting a default icon or label. Below is very minimal QML code that could represent an action key. Obviously, some styling and sizing should be done also, but were stripped for clarity.

Rectangle {
    id: action_key
    property string caption: MInputMethodQuick.actionKeyOverride.label

    MouseArea {
        id: mouse_area
        anchors.fill: parent
        onReleased: { MInputMethodQuick.activateActionKey() }
    }

    Text {
        anchors.centerIn: parent
        text: caption
    }

    Component.onCompleted: {
        MInputMethodQuick.actionKeyOverride.setDefaultLabel("Enter")
    }
}

Note that only actionKey overrides are supported in QtQuick plugins. There is a plugin that already uses it - meego-keyboard-quick. For those who want to know how to write Maliit plugins in QtQuick, Michael Hasselmann wrote a short tutorial about it.

Friday, October 7, 2011

Syncevolution build system work

In August Openismus asked me to work on Syncevolution's build system. The most important part of this work was to convert it from recursive Automake to non-recursive one with some features from previous build system still being available. Those were namely version number generation and, if possible, avoiding of manual listing of files to distribute. An added value of this conversion would be faster parallel build. Also additional objective was to make the build system less confusing for newcomers, but I doubt if I succeeded in that one.

Before I started the work the build system of Syncevolution consisted of two
sort-of-configure.in files (configure-pre.in and configure-post.in), Makefile-gen.am in src/, gen-autotools.sh, configure-sub.in files in every backend directory and a bunch of regular Makefile.am files. How this worked? autogen.sh was calling gen-autotools.sh creating proper configure.in by sandwiching contents of all configure-sub.in files between configure-pre.in and configure-post.in and substituting the version in AC_INIT with the one it computed. Also it was doing some find-and-sed magic on Makefile-gen.am to generate Makefile.am with list of found backend directories for SUBDIRS variable. After finishing this steps, it called autotools. In different order than autoreconf is doing it, but that was minor issue. Pretty messy, eh? Another issue was toplevel Makefile.am and src/Makefile-gen.am being a mess - lots nested ifs mixed with some custom rules and variables here and there.

While I liked the idea of injecting contents of configure-sub.in into final configure.in I didn't quite like how it was done. I wanted autogen.sh to call just autoreconf with some flags. I wanted no sort-of-configure.in files. I wanted no Makefile-gen.am. I wanted no script doing sed on neither configure.ac nor Makefile.am.

For version number generation I just stole an idea from autoconf (it uses m4_esyscmd in AC_INIT), so now it calls gen-git-version.sh.

I put code injecting contents of configure-sub.in files inside a m4 macro (which in fact calls a script) so merging of sort-of-configure.in files into configure.ac was possible.

Doing magic on Makefile-gen.am was apparently not needed after conversion to non-recursive Automake, because SUBDIRS are rather not used there. Instead, the backends.am with 'include <backend_name>/<backend_name>.am' lines is generated by yet another script.

Above steps clearly don't help the readability of build system. Maybe at some point such tricks could be removed.

Automake's documentation says that Automake itself should have enough support for generating a non-recursive build system. But still there were some hurdles to clear.

  1. Automake has a useful feature of installing/distributing the directory structure without need of specifying foodir and foo_DATA variables for every subdirectory - it is a nobase_ prefix. Apparently it is not that very useful in non-recursive Automake. Why? I wrote it in detail in feature request I reported to Automake.
  2. At some point 'make -j4 distcheck' failed. After some digging I noticed that libtool was trying to relink a backend against a library that should be installed at that point but it seemingly wasn't. That looked like a race condition because of incomplete dependencies. There are already some reports/feature requests for ability to specify install-time dependencies. In general Automake generates install-am rules as follows:

    install-am: all-am
     @$(MAKE) $(AM_MAKEFLAGS) install-exec-am install-data-am
    

    so install-exec-am and install-data-am can be executed in parallel. install-exec-am rule installs all libraries, scripts and programs that should reside directly in one of standard directories ($(libdir), $(bindir), $(libexecdir) and so on), while install-data-am installs the rest. So a library is installed during install-exec-am and backend during install-data-am. And seemingly thread doing the latter rule installs (and relinks) the backend before the library is installed.

    My temporary hack was just to override install-am rule to do these steps sequentially:

    install-am: all-am
      @$(MAKE) $(AM_MAKEFLAGS) install-exec-am
      @$(MAKE) $(AM_MAKEFLAGS) install-data-am
    

    This is ugly because it invades Automake's namespace.
  3. The task of distributing files, that is - creating a tarball is a special task of Automake. The list of files to distribute is independent from conditionals. Unfortunately Automake's documentation is not clear about it. The misunderstanding of this issue often leads to a situation like the one below:

    in configure.ac:

    # need rst2html for HTML version of README
    AC_ARG_WITH(rst2html,
                ...,
                [AC_PATH_PROG(RST2HTML, rst2html, "no")])
    AM_CONDITIONAL([COND_HTML_README], [test "$RST2HTML" != "no"])
    

    in Makefile.am:

    if COND_HTML_README
    dist_doc_DATA += README.html
    endif
    ...
    README.html: README.rst
      $(RST2HTML) --initial-header-level=3 --exit-status=3 $< >$@
    

    Often justification for such situation is to avoid hard dependency on rst2html but still be able to provide README.html in tarball. Now, if one clones the repository, calls autogen.sh to generate build system, then calls configure with --with-rst2html=no (or just configure, if rst2html is not installed) to generate Makefiles and then calls make && make dist to build project and generate the tarball then the build will fail during executing dist target. That is because make wants to put README.html into tarball thus it executes a rule generating it. But RST2HTML variable is 'no'. This often goes unnoticed for long time. Why is that? Because automake strives to make tarballs having always the same content, regardless of flags passed to configure, regardless of existence of some installed software. Tarballs have to be always the same. With this in mind there are two clean solutions to this situation: either never distribute README.html or always do it. For the former changing the line:

    dist_doc_DATA += README.html
    
    into:
    nodist_doc_DATA += README.html
    
    should be enough. For the latter - make rst2html a hard dependency and thus remove the
    COND_HTML_README
    conditional.

    There is also sort of solution for having README.html always distributed:
    dist_doc_DATA += README.html
    
    if COND_HTML_README
    
    README.html: README.rst
      $(RST2HTML) --initial-header-level=3 --exit-status=3 $< >$@
    
    else
    
    README.html:
      if test ! -f README.html ; \
      then \
        echo "no rst2html and README.html is not found!"; \
      exit 1; \
    fi
    
    endif
    
    Understanding of this solution is left to reader as an exercise.
  4. Since non-recursive Automake means that only one Makefile (a toplevel one) is generated that forces developer to be careful when using variables. That is because all .am files are included into toplevel Makefile.am and thus it may happen that some variables are clobbered. To avoid clobbering a variable meant as internal one (say: my_sources) it is good idea to prefix it with escaped path of this .am file, for instance src_dbus_server_my_sources. To avoid clobbering an Automake variable (say: lib_LTLIBRARIES) it is good to initatialize this variable at the beginning of Makefile.am with an empty value and later just append values to it:

    # beginning of Makefile.am
    lib_LTLIBRARIES =
    ...
    # some where later or in another file being included by Makefile.am
    lib_LTLIBRARIES += src/foo/libfoo.la
    src_foo_libfoo_la_SOURCES = ...
    

    Since I had like twenty of such variables (MAINTAINERCLEANFILES, DISTCLEANFILES, bin_PROGRAMS, dist_noinst_DATA and so on) I created a separate setup-variables.am file which contained only such initializations and included it in toplevel Makefile.am.

    Also, with such system not only variables may be clobbered but also some local rules meant to be run as hooks like installcheck-local or some special make variables (.PHONY). My solution was to add lines to setup-variables.am:

    all_dist_hooks =
    all_phonies =
    

    Add lines to toplevel Makefile.am:

    .PHONY: $(all_phonies) ;
    
    dist-hook: $(all_dist_hooks) ;
    

    And to .am file with dist check routine:

    all_dist_hooks += src_dist_hook
    src_dist_hook:
     ...
    ...
    all_phonies += $(TEST_FILES_GENERATED)
    $(TEST_FILES_GENERATED):
     ...
    
  5. Another not documented (or maybe a bug) was that one have to define explicit foo_DEPENDENCIES variable when foo_LIBADD (or foo_LDADD) has AC_SUBSTed variable containing a path to in-project library. Otherwise Automake won't generate dependency on such library and race condition ensues (foo may be linked before the library is build). This is written in detail in bug report I filed.

When I'm filing bug reports to any project I try to at least look where the problem in source code is and to create a patch. But Automake being script of 8k lines of scarcely documented functions scared me away - I suppose that modularisation should be performed earlier instead of writing such a monster. Maybe later I'll try looking at it again.

In the end I must say that I am not happy with the outcome. Probably some generated files does not need to be generated anymore, so they should reside in repository, be visible for newcomers and scripts previously generating them removed. That would for sure improve clarity. Toplevel Makefile.am and src.am in src/ are still a mess. configure.ac is still also a mess. Also non-recursive Automake being less confusing to newcomers is questionable. I suppose that most of people using Automake is used to recursive build system and tend to treat .am files as Makefile.am files. Which is obviously a pitfall, because $(srcdir) and $(builddir) change their behavior. But I suppose that Autotools are in general confusing for newcomers.

tl;dr - I noticed that when converting a recursive complex Autotools based build system (such as Syncevolution had) to non-recursive one, some problems never appearing before may (or rather: will) appear. Some of them appears to be an effect of not being documented clearly in Automake documentation.