Virtual On-Screen Keyboards On Linux
With a tablet awaiting a better operating system I decided I had to evaluate the landscape on on-screen virtual keyboards on Linux. Obviously the ideal would be iPad/iPhone-esque, but that would be really frighteningly good, I had much lower expectations.
I was not sure if the tech was there to even show the keyboards automatically when an input field was selected. GTK are well ahead of the game on this with their IMContext base class to encapsulate input mainly for complex languages or disability accessibility, but a good side-effect is the ability to hook in an on-screen keyboard when a text input is focused. In that case the on-screen keyboard needs to know about GTK in order to know when to show.
I tested all these on a VM that has now run out of hard disk space thanks to having to compile lots of stuff. Most of the keyboards I completely failed to make automatically appear on the screen when a text input was selected, it was either me or them....
So in no particular order....
Matchbox Keyboard
Some people are using this actively. It requires hacking to be able to toggle the display of it which makes it pretty much a no-go as it would always be on top of the screen. The GUI looks all right though. Some good info here.
xvkbd
The venerable old man of the line up. It's been around the block and does exactly what it says on the tin. It has lots of good features and hacks (like making it appear for login windows). As far as I could see I couldn't hook it up to the IMContext but I could be missing something.
Gnome On-Screen Keyboard
The oldest GTK based one and seems to be showing its age thanks to the UI.
onboard
This is the Ubuntu one. I couldn't get it to automatically appear on focus. It looks OK, but seems to be lacking some features. At least it is easily hackable in Python. Some more interesting information is here.
Florence
This is the most recommended one and it is pretty good. The main pain was I had to compile it as their were no packages for it.
fvkbd
This looks fairly promising as it is very skinnable on configurable. Unfortunately I could not work out how to hide or autohide the keyboard. Also this had no packages and requires compiling. This provides the basis for the OLPC virtual keyboard test and also the Meego/Maemo virtual keyboard.
Caribou
This is the virtual keyboard testbed for Gnome 3.0. It is written in Python and needs packaging so I had to compile it. This was the one that got closest to what I want because it worked. It shows a small keyboard by the input with current focus. In my VM I had to disable the Clutter code for whizzy animations before getting it to work.
I've probably missed some but there are only so many not quite ready applications you can test before going completely and utterly bonkers.
Another thing I discovered was Google Chromium's addressbar is accessibility implemented, but none of the content in webpages is. Firefox on the other hand and anything WebkitGTK based (like Epiphany) all work properly with their accessibility. Part of the Chrome problem is the multiprocess rendering and that they have not prioritised accessibility according the bug reports I have read.
coda_network – and update already(!)
After releasing the first version of coda_network over the weekend I have found a few bugfixes (corner cases).
Mainly these involved using an NTLM proxy and attempting to tunnel a socket through it. This makes sense of the proxy case where you are trying to daisy-chain sockets together. This was really an issue with keep-alive proxies where the Request object was not resetting some cached state that would only ever work once.
The main feature now though is I have added mini_proxy.py. A multiprocess proxy server using all my code changes. Only GET, POST and CONNECT are implemented, but that should be sufficient for most tests. This mini proxy can then speak to upstream proxies of no authentication types, as well as basic. digest, and NTLM.
The readme on the github page has been updated with example commandlines for the proxy and I have tested it against all kinds of proxies. Because of all the other work put into httplib and urllib2 the proxy itself is actually only 200 lines. I've got no idea how efficient it is, but it should work fine.
What next for coda_network
You can get coda_network here and read about it in my previous post.
So what am I planning to do next for coda_network? To be honest it is functionally complete from my requirements but there are some less glamorous things that need doing.
- Finish writing up the unit test suite for proxies, downloads and requests.
- Add some more example scripts for downloading and the suchlike.
- Add an example mini_proxy that is half written right now that provides a Python based local proxy that can connect to any of the supported parent proxies.
- Fix any bugs I find and from any feedback.
- Try to egage the Python community to get these fixes or something similar pushed upstream.
- And when I feel really brave - go have a look at Python 3.
Fixing urllib2 and httplib Python networking – coda_network
You can get coda_network from GitHub here.
Originally the networking code at Camvine was based on httplib a very long time ago, then Curl was used (pycurl specifically with changes pushed upstream), but now I have been using normal Python 2.6 standard library functions which is much nicer. The default one distributed was not sufficient so I had to make some fixes through deriving classes and monkeypatching. That is a very basic history of the coda_network package's origins.
I've based these changes on the Python 2.6 maintenance branch so it should in theory work with all versions of 2.6 Python (and I suppose possibly the 2.7 release). I did maintain my changes as patched derived classes, but I decided to integrate the changes into urllib2 and httplib directly in an attempt to eventually get these changes or something similar to them upstream.
Camvine have allowed me to publish the source code for all of these fixes so I am putting them on github with my often strange commit comments. I'll get on to the functionality and fixes that I provide in a moment. This all stemmed from attempting to fix tunnelling HTTPS through proxies and then having to fix it for NTLM.
In the repository there are the new httplib/urllib2 files which can be dropped in. They have been made part of a coda_network package along with a few other helper files. Below is the readme in the github repository.
coda_network
Author: Garry Bodsworth Website: http://www.programmerslog.com
Thanks to Camvine http://www.camvine.com for allowing the source to be published.
The changes are based on the Python 2.6 maintenance branch so it should in theory work with all versions of 2.6 Python (and I suppose possibly the 2.7 release). The changes were maintained as patched derived classes, but now they are integrated into urllib2 and httplib directly in an attempt to eventually get these changes or something similar to them upstream.
In the repository there are the new httplib/urllib2 files which can be dropped in. They have been made part of a coda_network package along with a few other helper files.
The Files
urllib2.py Based off http://svn.python.org/view/python/branches/release26-maint/Lib/urllib2.py?view=markup revision 81637 The only public API change is that Request now has an optional "method" parameter.
httplib.py Based off http://svn.python.org/view/python/branches/release26-maint/Lib/httplib.py?view=markup revision 81688 The only public API change is that the HTTPResponse object has a tunnel True/False parameter so we know if the response is from a normal request or a tunnelling request.
utilities.py Some simple wrappers I use to expose the functionality. This also provides a really basic do_download functions to do downloading with minimum of fuss. This stuff is a bit specific to me.
connect.py This is to allow connecting to a vanilla socket whilst reusing all the proxy auto code (rather than the rather painful process of not being able to reuse it).
ntlm_auth.py This provides the proxy and HTTP authentication handlers.
ntlm subdirectory This is a copy of a portion of the code from python-ntlm http://code.google.com/p/python-ntlm/
Fixes and features provided
- The request object now provides a method. This is handy for doing requests like OPTIONS which are necessary for cross-domain javascript or proper HEAD requests.
- Allow digest proxy authentication to work by retrieving the correct location and the correct password because these were previously designed only for HTTP auto. Also for digest auto of proxies it requires the CONNECT information to generate the digest correctly.
- Remove the cyclic dependency in constructing the response passed to the higher levels thanks to a member function being assigned to a member variable. This is done in the simplest way possible by wrapper the receiver in a simple class adapter, it could have been done by using weakmethods, but it works.
- Tunnelling HTTPS through proxies is fixed.
- Added the optional socket_keepalive in order to do deliberately long-lived connections (for long-poll). This is available for HTTP and HTTPS.
- A very important part with CONNECT errors was to make sure it is exposed at the right time. This then allows for proxies and other things that require retrying to work. Previously a socket error was thrown immediately, whereas now it is exposed through the getresponse() function which allows the call stack to make the right judgements on the error code.
- Allow the specification of a certificate for HTTPS connections such that secure connections to remote servers can be made.
- Rather than using simple socket sends when doing tunnelling it now uses the standard HTTPConnection putrequest(), putheader() and endheaders().
- Improve some of the messaging in the error handling so we can reason about some of the errors.
- Add the ability for connections to be kept alive with Keep-Alive when the connection/proxy connection specifies it. I noticed digest is really not happy with this.
- Add NTLM authentication using the python-ntlm files for generating the hashes which in turn is from NTLMAPS as far as I can see. This works for both proxies and www-authenticate. The connection Keep-Alive is really important for NTLM as the Windows servers require the socket connection to be kept open continually.
What coda_network can do
- Use the following proxy types - normal, basic authentication, digest authentication, NTLM authentication.
- Authenticate websites - basic authentication, digest authentication, NTLM authentication.
- Request types - any you can think of.
Known limitations
Digest authentication of a website when going through a proxy seems broken. This is probably due to the source of the request in generating the hash. It is such corner case that I am not looking into it right now.
FAQ
Long-Lived Processes
One of the things about developing a digital signage system is that the software running the display has to run for a long period of time. From what I have heard some people set their digital signage players (from other companies) to reboot every night which is an odd state of affairs, whereas we just say plug it in and switch it on. That does not mean we are infallible and I am constantly hunting down strange behaviours in the long-lived process.
What I have really found from the reality of having real operating systems out there in the big wide world is that you will always encounter that which you cannot predict. There is however a way to mitigate the risk. You need to take a leaf out of the server world. Your main process should be as simple as possible doing the absolute minimum, then spawn processes to do the donkey work.
The advantage of this is that if anything does go wrong in the real hard work then when the process is closed or is aggressively killed the resources are returned back to the system. In Linux this is particularly good because it is really good at starting processes efficiently. I saw evidence of this when git was spawning lots of perl processes, which in Windows was a lot slower, meaning that git in Linux on a VM in Windows was much faster than on plain Windows.
Even Python has a built-in multiprocessing library (which was really put in place to get around the GIL issues) but it gives a great framework for managing processes and inter-process-communication. This is available from 2.6 up and I must get around to porting more stuff to it rather than rolling my own.
Another great feature of the Linux graphics stack is texture from pixmap so that you can have out-of-process rendering and composite it all together in OpenGL.
Really the lesson I learnt by forgetting to not put something into a separate process is that the unimaginable will happen, from bizarre corruptions, to parsers that should fail but are just too robust, which then means you have memory hanging around forever.
Google’s New Font Service
At Camvine I spent a small amount of time recently making sure TypeKit works, so we can use funky fonts in our WebKit based browser.
Now Google have announced the Google Web Font API and the Google Web Font Directory. The API webpage is here.
You can view the available fonts here. Not a huge amount or anything but f it catches on it could explode very quickly.
What with Google pushing 3D web, video, geolocation, fonts, app store, online collaboration (opening up Wave to all), and god knows what else I think they are heading towards completely closing the loop functionality-wise. As far as I can tell you can actually make commercial use of some of these APIs now whereas you were completely unable to do so legally before.
Video Codecs That Don’t Make You Go Grey
Great news today, Google are open-sourcing and more importantly free patent licensing the VP8 codec. As part of this they are creating the WebM project with a new container format that uses the open-source Vorbis audio codec.
As someone who has to deal with video codecs every day and the horrors they induce (not just from a technical standpoint but legal) this is a very good thing. Unfortunately it does have some way to go to depose the incumbent codecs, and to reach parity with respect to hardware acceleration, but I am sure it will happen swifter than we can imagine. The main reasons for this is that Youtube will be using it which is a large portion of the web's video, and th enames that are all on board to support it in both software and hardware.
I can tell you as soon as gstreamer gets support CODA will be getting WebM.
There is also now a technical analysis from an x264 developer which can be read here.
Good news for video acceleration on Linux
Video acceleration has got a whole lot better on Linux. Splitted Desktop have released gstreamer plugins with VA-API support. gstreamer-vaapi was announced on the gstreamer-devel list here.
There are VA-API backends (in the project libva) for Intel chips, VDPAU for nvidia (and allegedly S3) chips and XvBA for AMD chips (have they ever officially released it?) So this acceleration framework will work on tons of hardware out there. Except for the Intel open-source drivers you do need to use the binary closed source drivers (also for Intel Poulsbo).
The gstreamer-vaapi plugin supplies MPEG-2, MPEG-4, H.264, VC-1, WMV3 acceleration (providing your hardware supports it). Also output is on OpenGL rendering through VA/GLX or GLX texture-from-pixmap + FBO.
Being gstreamer it should be relatively swift for many applications to pick up the acceleration.
WebKit keeps me spinning round
So after all the issues I was having with WebKitGTK having undebuggable crashes somewhere in the nvidia binary blob I decided to have a go at QtWebKit integration. It was fairly quick and easy to do (I cheated and used PySide for brevity). That was good - no crashes, some nice looking fonts, Flash working and generally all was good, so I started to think how to do a proper integration.
This is where it all goes wrong. I find with most things an initial test implementation is quick and gives you a feel for the problem, when you actually have to do the real thing you discover all kinds of creepy crawlies hiding under rocks. QtWebKit has now split out from Qt, but there is no packaging for this new branch that I could find under Ubuntu, not too much of an issue since I am used to packaging stuff up. I found HTML5 playback with the gstreamer Phonon backend simply did not work in any half decent fasion and they were concentrating on QtMultimedia which unfortunately is part of the development release.
The deal breaker was the speed of CSS animations, which I am lucky I actually got around to checking since I was juggling so many variables. It was unfeasibly slow. So I decided to use the new QtWebKit 2.0 as it contains compositing acceleration support for just this problem. Indeed, using raster or opengl graphics backends did speed up the animation to very smooth. What I then discovered was I could have smooth animations and no working Flash or rough animations and a kind of working Flash.
This was all most depressing, so I have gone back to WebkitGTK. It is only this crash issue I have to get to the bottom of but it only shows itself in one of my tests which is very specific, and I am thinking it is related to the way xlib is working in the nvidia driver. I do prefer GTK to Qt, so that is also good for me! I would have given EFL Webkit a go, but unfortunately they seem to have removed features in getting the functionality into the Webkit trunk.
The Many Faces Of WebKit
I've been back into the belly of the beast investigating different implementations of WebKit again. So I thought I'd share some of the things I picked up about them.
Obviously the Apple implementation is the cream of the crop for speed an features. In fact the Mac OSX version isn't top of the tree, it's the iPhone/iPad version that has been furthest ahead. It got the accelerated compositing of web elements first and could do all kinds of funky 3D transforms using OpenGL. The OS X version got those features only in the middle of last year.
Other ports vary in stability, features and usefulness.
WebkitGTK
This is the port of WebKit to the GTK+ widget set. It has been through a lot of changes like where the network backend moved to a new library which meant that proxies could not work for a fair while. The rendering itself uses Cairo (good stuff), but it is just a shame you can't easily switch backend (like swapping from xlib to opengl). It uses Pango for font rendering which I know is very good, but it has a serious limitation, it can't load temporary fonts directly from a file, this means that the new font-face features in HTML will be a long way off.
QtWebKit
The rendering quality does look seriously good, but as usual with Qt I am finding very bad performance metrics. I've tried changing rendering backends where "native" seems to perform best and "opengl" fails then segfaults (which is even more surprising when you consider I am using the nvidia binary drivers on Linux which are pretty full featured).
HTML5support seems a bit flaky at best with one of my sample set completely freezing the process. On the plus side there is integration work already done for compositing, and fonts look seriously good. I got Flash working and it seems OK, but not quite as fast as WebkitGTK.
The biggest advantage though is PySide which are comprehensive generated Python bindings for Qt. It allowed me to a do a very quick integration with the gobject mainloop in Python as well.
They have also broken away from the protracted Qt release process which allow them to release WebKit features much faster. This makes life a lot easier and they are allowing targetting of more than one version (although not officially supported).
EFL Webkit
I have a soft spot for EFL and Enlightenment, it gets a lot of the basics very right (small reusable components and fast and low footprint). EFL Webkit shares a majority of infrastructure with GTK including the Cairo rendering and network backend. The project has also recently picked up sponsorship from Samsung.
There is a really cool demo of the web browser being drawn on a 3D surface using the EFL libraries. You can see it here. It's got gstreamer integration for HTML5 video and plugin support for Flash (oh well).
Now that just covers those main ones available on Linux. There is still the Palm WebOS version, Blackberry, OWB, and tons more. OWB looks the simplest for enabling porting to new platforms and I have built the SDL version in the past. Then there is the Windows version of WebKit and the standalone Mac build...
What I'm Doing...
- http://j.mp/c5NtYQ 2 weeks ago
- Awesome painted scifi. I'm sure the cover is the same as Psygnosis' Blood Money. http://j.mp/aiT6Ta 2 weeks ago
- @kumbunterland got all three books and they rock. Especially surfing a Dracula robot from the moon. in reply to kumbunterland 2 weeks ago
- Oh, and they sold the Samsung Wave as an Android phone when it blatantly isn't. AAAARGH! 3 weeks ago
- And that was in exchange of the hatefully bad LG Viewty Snap they had convinced her to get last time. 3 weeks ago
- I sent my wife into O2 expecting her to get a Palm Pre and she walks out with a Samsung Wave. They're actively discouraging people from Palm 3 weeks ago
- ProgrammersLog Update: Python is beautifully over-engineered http://blog.programmerslog.com/?p=459 3 weeks ago
- Whoops - I need to get a micro-SD card so I can upgrade my Android tablet to the latests 2.2 firmware. An oversight involving yet more money 3 weeks ago
- New toy - cheap Chinese Android tablet. http://twitpic.com/2e2gwl 3 weeks ago
- In web browser drawing http://bit.ly/b2Wg0m 3 weeks ago
- More updates...
Posting tweet...
Powered by Twitter Tools
Pages
Archives
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- February 2010
- December 2009
- June 2009
- May 2009
- April 2009
- March 2009
- February 2009
- January 2009
- December 2008
- November 2008
- October 2008
- September 2008
- August 2008
- July 2008
