sandos

Android Developers Blog: Android Market Welcomes Korea!

2010-07-16T14:38:00.001+02:00

Android Developers Blog: Android Market Welcomes Korea!: "As of today, Android Market is open for business to application buyers in the Republic of Korea. We hope that this will make the outstanding Android devices now available in that nation even more useful and fun. We welcome the people of Korea, acknowledged everywhere as one of the world's most-wired societies, to the world of Android."

This is awesome, really. WHAT ABOUT SWEDEN?

;)

Android Developers Blog: A Note on Google Apps for Android

2009-09-27T21:08:00.000+02:00

I think googles stance on modded ROMs is a bit sad, since I am running one of those, and I would frankly not go back to stock ROM, because of lack of tethering. I think a totally open ROM would be awesome, but I realize it would take probably a year to replace the stuff that Google built. Hopefully I am wrong.

Android Developers Blog: A Note on Google Apps for Android

OpenAndroid

Android Scripting Environment needs Groovy!

2009-06-09T13:35:00.000+02:00

I am wondering why Google choose not to mention Groovy for the ASE, the Android Scripting Environment.

Groovy has no trouble accessing java APIs, for one. Could it be that groovy relies too much on dynamic code to be useable on the dalvik VM? See here for example http://www.nabble.com/Anyone-got-the-groovy-running-on-Android--td15044776.html.

Xmpp server components for java

2009-06-05T09:27:00.003+02:00

Its a bit sad that java is totally lacking any sort of embeddable or componentized server-side XMPP libraries. There is Tigase and Openfire servers, but they are just that, servers and not meant to build new servers or to be embedded, as far as I know.

I have also failed to find any examples or libraries for creating component connections to XMPP servers, although they are meant to be very similar to client connections.

Edit: Actually, seems there are ways to use PubSub from java which should be good enough for me: http://www.igniterealtime.org/community/thread/38292

LuxRender performance on AWS

2009-05-26T10:35:00.002+02:00

Just plyaing around with LuxRender on a small Blender 2.48 testscene. On my laptop, which has a Core 2 Duo 2,2Ghz, I get about 22k samples/s. Using a "High-CPU Extra Large Instance" AWS instance I get between 120 and 300k. Average seems to be about 10 times higher than for my laptop, which is nice enough. These cost 0.8$ per instance-hour, which is pretty decent for shorter jobs. I'm not sure about the bandwidth needed, luxrender does transmit the "film" every now and then but the interval can be tweaked.

Scaling this up by using more instances is also very easy.

Lyrics for Fischerspooner - Megacolon

2009-05-19T16:37:00.000+02:00

Just corrected a few errors in the version that are around on the net:

Document on Google docs

How to add implicit imports to an embedded GroovyConsole

2009-05-06T16:20:00.006+02:00

I wanted to have a graphical GroovyConsole that had my own APIs automatically imported. This proved to take some time to fix up. This is how it ended up looking. I created this in my normal eclipse-project so that I can easily launch this with the correct classpath already done for me.

TestGUI.java:

package groovy.ui;

import java.lang.reflect.Field;
import java.net.URL;
import groovy.lang.GroovyClassLoader;
import groovy.lang.GroovyShell;

public class TestGUI extends Console
{
    public static void main(String[] args)
    {
        TestGUI gui = new TestGUI();

        gui.run();
    }

    public TestGUI()
    {
        super();
        try
        {
            setup();
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
    }

    public void setup() throws Exception
    {
        Field shell = 
           Console.class.getDeclaredField("shell");
        shell.setAccessible(true);
        shell.set(this, new OurGroovyShell());
    }
}

OurGroovyShell.java:

package groovy.ui;

import java.util.List;
import org.codehaus.groovy.control.CompilationFailedException;
import groovy.lang.GroovyShell;

public class OurGroovyShell extends GroovyShell
{
    public Object run(String scriptText, 
           String fileName, List list)
        throws CompilationFailedException
    {
        scriptText = "import whatever.*\n" + scriptText;

        String[] args = new String[list.size()];
        list.toArray(args);
        return run(scriptText, fileName, args);
    }
}

Fairly simple in the end. I was fooling around a bit to get this to work at first. This was done with Groovy 1.6.2.

Using putty (win32) to connect to vino in ubuntu 8.10

2009-03-27T10:46:00.003+01:00

Vino, it seems, has a bug in that it only listens on the localhost ipv6 address in ubuntu 8.10, ie. ::1:5900. I've been trying to connect to this using putty for some quite some time, and googling turns up a few threads and a bug in putty, ipv6-literals, that is not yet fixed even in the snapshots. I finally found the solution, almost all by myself, I should add an hostname to use instead of the numeric ip which has the problematic ":" in it. But actually, there already is in ubuntu: ip6-localhost. So adding the tunnel 4L5900:ip6-localhost:5900 works!

Oh, by the way: I did not want to disable ipv6 in Ubuntu since I occasionally use it.

New sources of world knowledge

2009-03-18T11:27:00.003+01:00

I am trying to develop a natural language parser using a "knowledge intensive" approach. For this I need knowledge for my application. I have a few plans for amassing it:

Various ontologies: Freebase, UMBEL, WordNet, FrameNet, VerbNet, DbPedia etc

Using pattern matching on Wikipedia articles to extract info

When trivially working, using the parser itself to extract more knowledge

For pattern matching I will need to develop patterns. Others have already done this, but I believe I can do even better. In wikipedia one can successfully use links to other articles as tokens as to simplify the structure of patterns when developing/evolving ones pattern to match against. Or at least I hope so.

More Wikipedia statistics

2009-03-18T11:20:00.003+01:00

Doing some statistics on the WEX datadump, and I am once again hitting memory limits. At least unless I filter uncommon stuff out ,which is entirely almost possible, given that you know every common word should appear a number of times if you take a chunk of wikipedia articles. Current stats are:

words 29657715 sentences: 1795296 articles: 13801 words unique: 1182046

Words are the number of whitespace-separated strings I've seen, sentences is sentences according to WEX, which by the way is way off for some articles such as Andre Agassi where it believes sentences ending with "No." means a new sentence when its Agassi's world ranking that comes next. Articles is self-explanatory, and unique words are unique words, this time with all character allowed but all lower-cased. So, from 14K articles the number of words are already more than 1M. This is very different from my older run which ignored anything that was not a-zA-Z, where it took a couple of hundred K articles to reach that number.

Making Firefox 3 play CNN videos

2009-03-04T09:24:00.000+01:00

The fix that worked for me was to add cnn.com to the whitelist in flashblock.

Others have mentioned other fixes, such as installing mediawrap, cleaning the cache, moving from adblock to adblock plus, using ietab, not using ietab, etc. For me it was a bit simpler.

Official Google Reader Blog: Square is the new round.

2009-01-08T13:13:00.000+01:00

I have to say the new Google Reader look is way better than before. I always hated round corners, too! I normally use the microfirefox theme in FF, so that tells you how anal I am about screenspace.

Official Google Reader Blog: Square is the new round.

The best way to browser Flickr to date

2008-12-18T20:51:00.005+01:00

This must be the way to browse Flickr lazily: tag browser. That is the tag browser I've been waiting for, basically. I would guess they are using something like DbPedia?

Novint Falcon

2008-12-18T20:41:00.003+01:00

This thing seems really cool. Not so much as a game controller, since I barely play games, but rather as a simple CNC machine ;)

Imagine building a foam sculpter or a 3D scanner from this. Should be easy, especially the scanner since there are buttons on the attachments already, atleast if you dare modifying the official attachments. Apparently they have PICs in them, too, so there might be a real protocol between them and the falcon making building your own attachments that much more of a challenge.

Android Developers Blog: Android is now Open Source

2008-10-22T11:50:00.000+02:00

So, the andorod source was released. That is awesome, and I never doubted it would happen. On the other hand, symbian is also something I would like to see the source of, with me using it every day! I read this morning that the source would cost money to get at, which is really sad. This is what I found:

"The Symbian Foundation platform will be available to members under a royalty-free license during first half 2009. The Foundation will provide, manage and unify the platform, ultimately releasing it as open source. "

Hopefully this is just a temporary measure, as I understand releasing it in the end means I can look at it for free! We can hope anyway...

Android signing signs..

2008-08-19T17:13:00.000+02:00

I hope this is how signing will be done even in Android devices:

There are no requirements on the key used to sign .apk files;  locally-generated and self-signed keys are allowed.  There is no PKI, and developers will not be required to purchase certificates, or similar.   For developers who use the Eclipse/ADT plugin, application signing will be largely automatic.  Developers who do not use Eclipse/ADT can use the standard Java jarsigner tool to sign .apk files.

Taken from http://code.google.com/android/RELEASENOTES.html. This is the major problem with J2ME in my opinion: no way for developers to self-sign stuff.

Google Knol - Information without semantics

2008-07-28T11:18:00.000+02:00

So I saw gnol mentioned on identi.ca. I checked it out, but oh boy was I disappointed. Absolutely no semantics at all? Wikipedia is more semantic than that! Faviki is more semantic still. And freebase!

Unless they fix that I don't see how it will be succesfull in the the long term. Other "semanticer" things will simply crush it.

Official Google Blog: Encouraging people to contribute knowledge

Using joins to find missing data and blobs

2008-07-06T20:45:00.007+02:00

I just realized, after having read about column based storage, how potentially bad it is to have blobs in a metadata-rich table.

I am currently developing a small project during my sparetime, and we have a table which is roughly: id int, mimetype varchar, data blob, version int, and so on with a few more non-blob columns in it. What I noticed was that selects on this table that does not need the blobs using the indexed mimetype-field was incredibly slow. Relevant is that I am running MySQL in development mode, so it never seems to use more than a few tens of MB of RAM, meaning almost no caching is happening here. This means, in a row-storage database like MySQL I will be doing a tremendous amount of either seeking or reading of unneccesary data, depending on how MySQL plans its disk-reading, how big the blobs are and the effect of any read-ahead in OS or elsewhere.

The other part of this slow query is a outer join, something like this:

select id from c
left outer join b on b.c_id = c.id and b.name = "something"
where b.name is null

This is apparently also fairly slow. So don't do these things too much. In this case, the join to find if data is not in a joined table could be easily fixed by adding data to the db.

More funny java bugs

2008-07-06T13:46:00.000+02:00

So, I get this simple assignment at work. Install a new version of our software on a brand spanking new Intel Xeon quad-core machine. Nice.

Only not so very nice when I notice hard java crashes. This is in EDU.oswego...ConcurrentHashMap's iterators hasNext method, always. I try downgrading to JDK1.5, but that does not help.

I might have to do yet another probing bug hunt in third-party software. Argh.

Also, trying out identi.ca even though blogging, and micro-blogging especially is not really my thing. Finding it fairly good though, just as ping.fm.

HTMLUnit, MultiThreadedHttpConnectionManager and memory leaks

2008-06-10T10:51:00.005+02:00

I have been having this wonderful time at my work. A small coding project involving use of mostly HtmlUnit was almost done, and working properly. But what happens? By chance I notice that it is leaking memory: Perm Gen space, even.

This was coded as a plugin for a larger product, and was dynamically reloaded at every invocation. I had to first remake the plugin CassLoader in this case to become a post-delegating classloader so that I could override the version of HttpClient already in the product. I couldn't really be sure it was not my changes leading to the classloader that gave rise to the leak, but eventually I got to that conclusion. The next step was to narrow down where this happened. Long story short, I found that if I changed HtmlUnit to not use the MultiThreadedHttpConnectionManager from HttpClient, it did not leak. I did not want to really do this though, being unsure of how HtmlUnit actually used this, and also because of the fact that we have multiple threads using HtmlUnit.

The thing that solved the issue was to call shutdownAll in the connection manager. I am not allowed to access that from my code as a user of HtmlUnit though, and I did want to avoid having to recompile anything, so I used reflection to subvert the access checks. Calling shutdown on the one manager did not work, however, nor did closing the connection, which HtmlUnit already did by the way.

I can only assume this is some obscure bug that nobody else ever trips, but now at least if somebody does, they might find this as a reference.

I could not use the latest HtmlUnit because of needing JDK1.4 compatibility, so this was done in HtmlUnit 1.13. Oh, and 1.14 needed CSS stuff that clashed with regular DOM libraries, making classloading not work. Not sure why this does not work when I can safely override HttpClient with a newer version.

Semantic Web is freaking cool! (and on a roll it seems)

2008-06-06T18:19:00.004+02:00

Been surfing around for semantic web websites to find ontologies or datasources to (ab)use for Yet Another AI-Project From Me. This is what I found:

True Knowledge. Incredibly cool question-answering frontend to an incredibly complex datamodel, with a moderately complex and severely boring input process. Not free data, I can not download a dump of their database.

Freebase. Took me some time to dig into this, actually, but I like what I am seeing. Data model seems a lot simpler than True Knowledge, or at least that is what i think (subclassing, transitivity missing?). Inputting stuff is from 2-10 times quicker/easier. For bulk stuff it is infinitely easier, since TK does not support that at all. Free data, but not RDF!

Faviki. Very nice and easy to use semantic social tagging/bookmarking service.

RDFScape. Visualizer for cytoscape. Very nice, have not had time to play with this yet.

Attempto Controlled English. Maybe the least exciting of the bunch, but is useful for my NLP-related project.

I also got access to twine. Oh my god what a bore. I just did not see the idea behind it, and the interface turned me off so much that after my third visit I never came back.

True Knowledge has some awesome NLP parsing going on, but it also fails miserably often. I have a simpe idea to get me atleast started, it pretty much builds upon AIML/patterns to extract meaning from stuff, specifically Wikipedia.

Freebase has a weak model in my mind, there does not seem to be a real inheritance hierarchy and the "upper ontology" is basically missing. The upper ontology not being there is not such a big deal though, I think. There should be a set of "uppermost" classes in Freebase that can be mapped to SUMO/YAGO/DBPedia/Wordnet or whatever to help with any inferencing/analogous thinking.

I can not help but think that in five years from now, "semantic" does not really exist. Everything is then semantic, or gone since long. AGI is not far behind, either. I predict a surge of NLP success in the coming few years, mainly with knowledge-intensive approaches. Common-sense is still the missing piece of the puzzle, the above efforts do not concentrate on this at all, but rather on knowledge that is useful to humans. Remember, common-sense is boring for humans to input and administer, since it is all so basic.

Nokia E51 stability

2008-04-28T14:50:00.003+02:00

I love my phone for several reasons. It is just the right size, fits very nicely in jeans-pockets without wearing them down in no time. My Motorola E398 was much worse in this regard due to being thicker.

I also love how it handles most J2ME apps, has email and so on. I've started using it during my commute every day to surf on my laptop, via bluetooth mostly since USB-cables are not that much fun carrying around. The phone is mostly stable. I've encountered crashes when for example running many apps and playing mp3s at the same time. Nicely enough it reboots automatically most of the time. I think I've found what it does not handle so well though: bluetooth internet sharing. Firstly it gets very hot, thats fine though I guess. But it also seems to become very unstable if I try to actually use it during this time.

Too bad, but I'm not surprised. I think that if I just don't touch anything while using it as a mode, it copes better.

"not in" versus joins

2008-01-29T14:43:00.001+01:00

I have been fighting this HQL query for probably 8 hours in total now, where I have a table Entity and EntityAttribute. Every entity has zero or more attributes, so the EntityAttribute table has Entity_ID row in it. Attributes have a name and value, each in its own column. EntityAttribute is mapped as a map collection from Entity, with the "name" as the index. I want to select entity based on whether they do or do not have an attribute with a particular name and value, although I know in practice that those who attributes with the right name do have the rigth value always, at the moment anyway.

The end of the HQL looks something like this:


SELET DISTINCT e FROM entity
WHERE ...
... AND
'type' in indices(e.attributes)

'attributes ' is the collection of attributes.

I would have guessed this to work, but no. I eventually tried this in plain SQL, where it of course also does not work.

I also tried with this, for the other way around (exclusion):


SELET DISTINCT e FROM entity
WHERE ...
... AND
e.attributes['type'] != 'animation'

I can sort of understand this last construct being wrong. The correct way to do things is apparently to swap these two, basically use "not in" for exclusion and use "join" to include things, which is what e.attributes['type'] = 'animation' uses.

Domain Model - A Tale of Bad (J2EE) Design(And a newbie developer trying to fix it)

2007-12-05T23:06:00.001+01:00

Where I work we have a product that we can call ABC. Now ABC is what I would call "legacy": more than 10k loc (I think it is somewhere around 50-100k), and the project was started more than 5 years ago, in 2001 I think. I have been working on this for about 16 months now, and I am getting more and more comfortable with it every day due to actually getting assigned to develop it at the moment. I am the only developer since a couple of months back.

ABC is in need of a lot of things, this I saw the first week on the job. So, many obvious things are wrong. Where to start? Well, our "service layer" (Note: we do not have clearly defined layers, and nobody seems to have known what to call bigger parts of the system, so if I could say "hey, we have a problem with our service layer" nobody would understand what I was talking about) has code in it that is very, very long-winded, some made up code:


dto.Part p = DAOFactory.getPartsDAO().getPartById(new Integer(1));
validatePart(p);
if(p.getInventoryID() != null) {
 dto.Inventory inv = DAOFactory.getInventoryDAO().getInvetoryById(p.getInventoryID());

 //Do more...
}

Hey, we use a factory! Great. But, this code does very little for being so verbose, and note that this sort of code could go on for hundreds of lines. Can we see spot any problems? I can, at least now, previously I really could not. I am not a very experienced J2EE developer, this is my first job with it fresh from university:

I hate the fact that we have to explicitly do "DAO things". Why is this necessary? It is not, if we for example use hibernate a bit more correctly: ABC is currently not using any mappings between entities for example. Adding this, we could do away with fetching one object in a transaction and then accessing the collection mappings instead of all the "DAOing" as I call it.
^{(This DAO-ing also has severe performance penalties when done this way: it has a select n+1 problem. It can be solved by being "smart", but takes a lot of manual coding, for something that hibernate solves perfectly for us, practically for free. Logic is honestly more compact in a HQL query than in this sort of convoluted code. 20-lines of HQL did in some instances replace 100+ lines of Java for us, especially manually fetching objects, checking them instead of simply using a WHERE clause.)}

We are fetching dtos from a database, by the looks of it. Is this really the intended use for DTO pattern? No (as far as I know, anyway), and it turns out these are not DTOs, firstly, and secondly, they should not be. Our "DTO layer" here is misnamed, and misguided. It is actually our domain model, although an almost completely anemic such^[1]. We do in fact have another DTO layer, that is really a DTO layer, and which is needed and should be a DTO layer. This is used for sending data from webservices, and decoupling this from our domain model is on the contrary not a bad idea.

So we have a, more or less, procedural service layer with 95% of our business logic in it, and a domain model layer with data and only trivial behavior in it. It seems straightforward enough to just move behavior into our domain model, and be done with it. One slight problem here is that I have introduced alternative "DTOs" (entity beans, domain objects) that are properly mapped using hibernate, and adding _Mapped to their classname, inheriting from its non-collection-mapped hibernate class. For example we might have:

Part has a subclass PartMapped. PartMapped has collection mappings, Part does not.

It can also look like so: We have PartBase, PartMapped and Part. Part is the "original" DTO, with a foreign key in the form of a Integer. We don't really want this integer in our hibernate-collection-mapped variant, so we create a base class which does not have it, and extend that while adding the collections for hibernate to populate.

Modifying the domain model to fit hibernate is a bad idea, I realize that. We need to stop depending on hibernate. I have been looking at spring lately (which ABC of course does not use) and I think it could help with a lot of this. One reason for having alternate classes for collection-mapped entities was that I do not yet trust hibernate to actually save those back: we still always use the simple save() call for that on a entity with no collection mappings. Makes it very explicit what is happening. This is simply a matter of education though, I need to learn how hibernate works in this regard and, say, filtered collection.

This inheritance structure makes it not hard, far from, but a bit smelly to add behaviour to these objects. Mostly due to the structure being hibernate induced, but also because of the duplication arising from that: We would need two different implementations of a "public Inventory getInventory()" for a Part and a PartMapped, one using straight accessors (hibernate needs objects to use the accessors for its collection proxies to work) and the other one using old-style DAO-ing to fetch the object. We have a lot of these methods to implement if we were to move them into the domain model proper.

I assume the right course of action is to abstract away the database stuff such as Integer ID; in all domain objects, and let "something" (not specifically hibernate) manage persistence for us, and then start modeling the domain properly. And then document it all and educate everyone that, now, ABC is much more agile and can withstand changes easily.

It cannot today, that is for sure. I draw the conclusion that earlier developers have been "fooled" by EJB patterns or something similar, and was unable to see anything wrong with our anemic domain model. Design patterns are used, but those are localized, and in general there "is no design". No explicit design anyway, there is one that can be sort-of inferred from looking at the code, but it is very vague and lots of boundaries in that are often broken by code. It can still server as a "new vision" for the overall design though.

Amazon EC2 and S3 performance...

2007-07-02T09:57:00.000+02:00

I was playing around with a EC2 instance, and I got this great idea that I should benchmark the disk subsystem. I had expected pretty standard performance for the disk at /mnt, but I was wrong:

Benchmarking /dev/sda1 [1537MB], wait 30 seconds
Results: 1014 seeks/second, 0.99 ms random access time

Benchmarking /dev/sda2 [152704MB], wait 30 seconds
Results: 4494 seeks/second, 0.22 ms random access time

Thats pretty damn fast. To get this sort of performance out of a RAID-5 set I think you need a hole lot of disks, maybe 20 or so? It sure as hell beats any disk I have at home with a factor of about between 15 and 60.

I don't know if this could be a Xen problem with this particular benchmark, but if it is true I actually just found a use for EC2: to run my DB-intensive information extraction jobs on. Though they are still most likely better solved by just buying a new machine with 4 or 8GB of RAM. It is pretty cheap now.

Edit: Oh, yeah, I understand that this disk array is shared by a number of instances. A good question then is by how many? And how likely are people to actually use the disk for any stressful activity, with it being non-persistent and all?

Edit2: Oh yes, I am stupid. This is most likely due to Xen using a sparse Copy-on-Write format disk image? It would explain it all, hitting the same physical sector over and over....

Flash random access write performance

2007-07-01T12:28:00.000+02:00

I just bought a new 2GB USB stick, Sandisk Cruzer Micro. I have a Pretec Tiny 256MB since before. I benchmarked these using h2benchw in windows (and seeker in linux, but only for reads).

Now, these two are actually pretty close together in performance, h2benchw reports 0.77 ms access time for the 2GB stick, and 1.77 ms for the 256MB one. Raw read speeds are, according to linux hdparm -tT about 25MB/s for the smaller one, and slightly lower for the Sandisk.

But the real odd difference comes when I set h2benchw up to test random access _write_ performance. Pretec manages ~10ms access time here, whereas the Sandisk comes in at a whopping 132 ms!! Wooha. There faded my hope somewhat of using flash as a faster substitue for disks. X-bit labs says there are faster write access times though, specifically for Apacer HT202 sticks with a 28ms access time for writes. Thats unfortunately still in line with harddisks, so the only real benefit is the read access times, in other words flash will only be great when the ratio of read/writes is heavily biased towards reads. A typical database-load might not perform so well.

MIDP 2.0 and killing homebrew..

2007-05-22T09:33:00.000+02:00

Why is it that MIDP 2.0 decided to completely lock homebrew developers out of J2ME development? If you read this first you'll notice some people are annoyed at the way root certificates are handled, mainly that there is no certificate that is guaranteed to be available on every device, and the way Java Verified is per-device. This is a big problem.

Another problem that I have been having lately is that of free software and J2ME. Now you may ask why can't you just put up with a few security prompts for your hobby projects? Sure, I am fine with that. As long as they are few. The problem here is of course that they are not few! If you want to, for example, read maps from a memory card you could be reading from hundreds of tiles.

I think the problem here is that the APIs are actually becoming useful, and the security prompts for these sort of applications are becoming a big bottleneck, necessitating signing. But there is no way to sign anything for free! Not as far i know, anyway. You are also not allowed to import your own root-certificate.

I think security is a good thing, but locking out power-users is also bad. This while things is putting me off my long-needed phone-upgrade.

KR tutorial...

2007-04-10T09:14:00.000+02:00

I found this interesting and simple intro to KR and I must say it is good. I don't quite agree that description logics is the way of the future though.

I also fully agree that Google most likely is doing A.I. research. If they are not, I think they are totally misjudging how useful it is and how close we are to useful A.I. applications.

This book that was linked also looks very promising, I have to add it to my Amazon wish list for sure.

Nokia E70 video playback resolution

2007-04-08T11:09:00.000+02:00

I have been trying to find information on what Nokia 5300 and Nokia E70 is capable of playing back in terms of resolution. I would love a E70 if it can playback full-res video, and my girlfriend wants a 5300. This thread seems to imply the 5300 is unable to play back full-res video, and I suspect the E70 has the same problem, given its huge resolution.

SleekXMPP

2007-04-05T14:38:00.000+02:00

Another nice XMPP library that aims to make implementing or testing XEPs easy, something I really long to do.

Some day I am going to start using XMPP "for real": I have been on jabber for a couple of months but I have also noticed that I never really IM anyone anymore now that I work all day long. I am an antisocial creature I guess. Also I only have bots and automated services on jabber, while every human in my roster is on MSN or ICQ.

Canon TX-1

2007-04-05T10:59:00.000+02:00

I would much like a Canon TX1 when I go to Venezuela in December. Or why not for the summer vacation in Paris/Öland as well.

I see many people complaining about the MJPEG of Canon. I like it though: its perfect for editing, no temporal component that makes re-encoding necessary when you cut the movie. Also the quality is very high. 8GB SDHC cards are not expensive enough to make them unachievable in any way. A problem is where to store all the raw footage when you empty the cards though... this will be a serious filler of HDDs.

Linkbacks and blogger/blogspot

2007-04-04T08:50:00.000+02:00

Why, oh why, doesn't Google support pingback? As far as I have understood, blogger does not support any of the linkback protocols at all.

Thankfully there is, as is common, a hack around this. The problem is that the hack this time around is very ugly: Trackback with some help from Greasemonkey:

Manually add links

Trackback as opposed to pingback

I need to do... more m$ stuff.

2007-04-04T08:29:00.000+02:00

Wow, I found this blog really interesting: Vista Smalltalk. It is, you might have guessed it, Smalltalk for Vista, sort of. It's using .NET and Microsoft's "AJAX" WPF/E to do some very cool stuff in the browser. I happen not to like Smalltalk much though, i did a presentation on it when I was at university and I was not impressed. It lacked anything special enough for it to considered useful by me anyway.

The future will be very interesting when it comes to portable applications. Seems Microsoft really has got something going here, and sadly the XUL approach of Mozilla will not be able to compete here, even though it is not really aimed at the same thing. I really need to learn some .NET stuff for real soon, or I fear I will be obsolete in a short time.

Broken NAD 317 amp

2007-03-18T16:54:00.000+01:00

My NAD 317 stereo amp is broken: the right channel is exhibiting some very bad noise regardless of volume and input source. I opened it up but I could see no signs of obvious damage such as open caps or burned areas.

These seem to have a problem with the quality of their capacitors in the power supply, but since my problem is virtually non-existent on the left channel I assume the power supply is fine. I guess caps could still be to blame on the right channel but I have no way of measuring these things unfortunately, and I am not about to jeopardize my speakers just to try and save some money on the repairs for the amp, or buying a new amp. If I could just find a specific reference where somewhere fixed this specific problem, I might be able to fix it myself. As I (almost) did for our Canon G1. Yes, I found the fix, bought the caps but I actually let a professional guy solder it. Nicely enough, he did it for free since I did all the hard work with actually dismantling the camera, which was a bith. :)

More Wikipedia distance!

2007-02-20T20:42:00.000+01:00

I recently mentioned a Wikipedia distance service in this post, and I've now found a much more current one: Six Degrees of Wikipedia.

Number of unique words in the english language...

2007-02-16T10:15:00.000+01:00

I am trying to use statistics, mostly n-gram statistics to peel some useful data off of Wikipedia. This poses a bit a challenge for someone like me who do not own a cluster of machines to do these things on. (hint: Google) The best-speced machine I own is actually a laptop: 1GiB RAM, Core Duo 1.66Ghz. This is fine for almost anything I can think of, but when it comes to Artificial Intelligence its just not anywhere near enough.

Take this scenario: You want to have a frequency of all words occurring in Wikipedia, so that you can use it later to exclude not-so-important data. Now, how many unique words could there really be in Wikipedia? Maybe one million? That sounds like a lot, and should be more than the true number, right? Wrong. Well, this depends on how you count I guess. I do not yet have a useful stemmer to remove different forms of the same word, and I will always include misspellings. Still, I only include words with a-z, A-Z in them, remember this will exclude common things like "it's", "don't" and so on.

I am still finding more than 2 million words, in parsing less than half the articles in Wikipedia. I am also ignoring all other namespaces other than the main wikipedia article namespace so talk and userpages are not included in this.

So why is this 2M words a problem? Well, firstly, Java uses 16 bits per character. Lets ponder that each word is in average 8 characters in length, and means that each word now consumes at least 24 bytes, and probably even more. Add in the storage requirements for the HashMap and Integer that is required to keep track of frequencies and you have a lot of memory usage. This means, that even if I "optimize" my strings to only store a byte[] of ASCII, I can still only store 2M words in 400+M of RAM!

And there is a lot more words than 2 million, unfortunately. If we look at the Google n-gram data we see that they found 13 million words!

That would require me to have something like 2.6 GB of RAM available to Java, and I just don't.

The obvious solution here is of course to use a proper, disk-based database but that is painfully slow! Compare: 1000 articles taking ~2 seconds, or it taking about 25 minutes! This would mean that the 6M+ articles would take approximately 14 weeks to analyze. This was with Apache Derby (durability=test, autocommit=false). HSQLDB is faster, but is unusable. Maybe I will have to use MySQL after all. Also remember that this step was planned to be a simple pre-optimizer stage for my n-gram goodies....

Apache Derby versus HSQLDB

2007-02-14T14:31:00.000+01:00

So I tried to use HSQLDB for a little hobby AI project of mine, that is, mining information from Wikipedia. Now HSQLDB is so fast it is just silly. Thats all good. Now, I wanted to store a histogram in a table because the number of items was too great to store in RAM/HashMap. I convert the code to use the DB instead of the HashMap, and all is fine. Now I want to see the resulting topmost entries in this histogram thing. So I do something along these lines:

SELECT TOP 10 * FROM histo ORDER BY cnt;

Does this work? No. OutOfMemoryError. Why?? It turns out that HSQLDB does not use indexes for ORDER BY, and hence it tries to build a temporary result that consists of the entire database. I had a look at the source, and I was determined to fix this, however ugly the solution would be.

But then I found Apache Derby, and it looks to be all that I want. It does not seem to be as fast as HSQLDB, but on the other hand I should be mostly I/O bound anyway since my databases will be many times my RAM in size. Also, Derby seems to excel at the embedded and PreparedStatement corner, and that is exactly what I'm doing. 100% of my recurring SQL statements are already Prepared.

HSQLDB is not a very young project, but I must say Derby seems to be about 10 times more mature to me.

HSQLDB 0, Apache Derby 1.

How to compile ffmpeg statically

2007-02-13T15:45:00.000+01:00

It took me some time to figure this one out. I've never understood all the details of shared vs. static linking. Just thought I would share my experience. This was to enable AAC encoding in linux, and not having to depend on the shared libraries. You can of course add any necessary libraries that you need.

Build libfaac normally (.tar.gz files are somewhat problematic, (linebreaks?) use CVS instead to get properly formatted files. Centos 4.4 was unable to compile this due to outdated autotools, used Debian testing.)

Build ffmpeg with this:
./configure --prefix=/home/x/ffmpeginstall/ --enable-faac --extra-libs=/home/x/ffmpeginstall/lib/libfaac.a --enable-gpl --extra-cflags=-I/home/x/ffmpeginstall/include --disable-ffplay --disable-ffserver --disable-shared --disable-debug --extra-ldflags=-L/home/x/ffmpeginstall/lib

"Wikipedia Distance"

2006-12-28T14:03:00.000+01:00

I found the website I was talking about earlier, its available here: http://www.omnipelagos.com. It seems very rough around the edges, in addition to having very outdated wikipedia content. I might just make my own version of that site soon, if I find the time for it.

Wikipedia seems incredibly well-linked, since everything seems to have a distance of 4 or 5. This means that measuring links qualitatively becomes important: a very simple first step to doing this is to start considering the language-links as having higher "connectedness" than random linked words in sentences. Next step could be simple template-based recognition of common high-valued links such as "X is a Y", "X is a sort of Y" and so on. Doing more than this though very quickly becomes an academic exercise in implementing general AI.

Oh, oh, this thing would be so much easier if wikipedia started implementing semantic tags a la semantic mediawiki and ontoworld

Exploiting Wikipedia for AI purposes

2006-05-16T23:43:00.000+02:00

Firstly, I'm pretty sure I've seen a reference to a service which could tell you the wikipedia distance between two articles, but I can't for the life of me find it again. If anyone knows what I'm talking about, please leave me a comment telling me what you know about this!

Secondly, I'm perplexed that nobody has yet parsed wikipedia and used it for common-sense or other AI-related tasks such as spreading activation. Think Conceptnet, Wordnet and OpenMind. They all try to build some form of graph between concepts, you could say, and that is exactly what Wikipedia is. Wikipedia currently has 1M+ articles, easily besting both Wordnet and Conceptnet in number of nodes. I'm convinced that the number of internal links also outnumber the others mentioned here, so I think Wikipedia could be a really useful AI resource. You could even start to follow external links to get an even finer-grained link between Wordnet-nodes, although I suspect that would help very little since I suspect very few Wordnet articles references the same external URLs.

If I had the opportunity, I would try to experiment around with these ideas, but unfortunately it seems I won't have the time for that, finishing up my thesis and then going straight to getting a job. Actually, I am already trying to get a job.

Groovy is slow!

2006-04-01T15:09:00.000+02:00

I have been trying to write a prototype for my thesis work in groovy, but I was thinking it seemed a bit slow. Sure, this is to be expected for such a dynamic language, and interpreted at that. But when I wrote a few microbenchmarks I noticed that interpretation didnt seem to be the culprit here.

First, I tried adding things to a hashmap. This did not reveal any big difference between java and groovy though, the difference were about 100% which I find acceptable, so I continued my search on towards creating objects, and here groovy seems to be adding some substantial overhead:


        val = new Integer("0") 
        for(i in 0..<1000000){
            Integer tmp2 = new Integer(43)
            val += tmp2
        }

versus:


        val = new Integer("0") 
        Integer tmp2 = new Integer(43)
        for(i in 0..<1000000){
            val += tmp2
        }

Java: 387ms, 206ms, Groovy: 29873ms, 4527ms respectively. Firstly, even when not creating Integers in the loop, groovy is still much much slower, but adding the object creation just makes it intolerably slow! This was precompiled with groovyc then run with the -server parameter to gain some speed.

I am reading in and parsing the database of Wordnet, and that alone takes some 400 seconds in the current version of my code, which I find a bit too much to sit around and wait for each time I restart my app! Yes, there are ways around this, and I will use them eventually (such as only reloading changed code and not restart the entire JVM, keeping the intialized structures in RAM and ready).

Rm my Mac

2006-02-22T11:52:00.000+01:00

I saw this page http://rm-my-mac.wideopenbsd.org/ which hands out free ssh accounts on a mac mini, for the purpose of hacking it. Not my favorite pastime, for sure, but I'm sure there are others interested out there.

Thesis: Opennlp and groovy..

2006-02-08T09:00:00.000+01:00

Just wanted to write about what my thesis ended up being about: Question classification. No chatbot, no Loebner contest I'm afraid. Seems I will use Opennlp and java, but the problem is I hate Java, I find it way too slow to prototype in. So I decided to use groovy. This did not turn out well, it took me about a day to figure out that groovys error-messages are far from perfect, leaving me to believe that class loading was all broken in groovy.

Anyway, both groovy and opennlp seems like great projects. Ive also downloaded the Stanford parser, might be useful.

Piratpartiet

2006-01-02T14:01:00.000+01:00

Piratpartiet:

Piratpartiet siktar på att ta en vågmästarroll efter valet 2006. Det finns mellan 800 000 och 1 100 000 aktiva fildelare i Sverige, och de är alla trötta på att kallas kriminella. Vi behöver ha 225 000 av dessa med oss för att komma förbi fyraprocentspärren och hamna i vågmästarroll. Den rollen tänker vi sedan använda till att avskaffa upphovsrätten.

Mycket intressant, och det kan bli så att jag röstar på detta parti men är inte helt övertygad ännu.

Filed in: politik, p2p, filesharing

Thesis subject: winning the Loebner contest

2005-12-15T00:12:00.000+01:00

The Open Mind project, together with ThoughtTreasure and Learner are all very interesting projects in and of their own, where Learner and Open Mind try to build up a common sense ontology from voluntary user input on webpages, and TT trying to understand texts. The one thing in common to all these is that some version of their knowledgebase is available for download. There is also data from SUMO and OpenCYC available.

Why all this is interesting to me right now is that I am trying to choose my thesis topic, and I am very much in favor of my idea about entering a chatbot into the Loebner contest right now. The problem is I understand that this is a huge, enourmous, even mind-bogglingly hard task. Firstly there is the three main parts of a chatbot: Input recognition, inference engine/reasoning/knowledge store, and generating output. Now each of these are hard parts, and I will try to make them all in less than 20 weeks, with no deep knowledge about inferencing, knowledge representation or linguistics. Yeah, I would say it is looking dark. But it is so damn fun to atleast try. I might have to restrict myself though.

Filed in: ai,ontology,science,research,cyc

Det kontantlösa samhället

2005-11-28T19:07:00.000+01:00

Det kontantlösa samhället:

En ekonomi utan gråa pengar vore lika svår att skapa som ett internet utan darknets

Varför inte ett tor för elektroniska transaktioner? Där användare från flera olika banker skickar vidare elektroniska överföringar för att dölja mottagare och avsändare. Vet att åtminstone en bank har möjlighet till inloggning med personlig kod, vilket väl är ett krav då varken dosa eller engångs-skrapkoder blir särskilt enkelt för ett script att använda sig av. Jag tror inte på att systemet skulle fungera om det var manuellt. Ett problem är att bankerna säkerligen skulle blockera dessa överföringar ganska snabbt genom att upptäcka typiska mönster.

Dock har väl polisen fortfarande möjlighet att plocka fram uppgifter från vilken bank som helst, så det handlar egentligen mest om att en bank inte ska sitta på för mycket information, och inte om att undkomma lagen.

Filed in: p2p, anonp2p, economy, cash

Freenet Alphatest

2005-11-27T14:59:00.000+01:00

Soo, freenet is alphatesting their next version 0.7. If you want in, go to #freenet-alphatest on the freenode irc network. This version is still far from being usable or anything, but I think they need more testers still. Basically, you can insert files and download them from a single-threaded CLI interface.

Instructions for installation of the node is here.

Filed in: freenet, p2p, filesharing